LibreOperations / Data Archive / Timeseries Store

This LibreCube element lets you quickly set up a REST API database for storing large amounts of timeseries data.

It may be used to store timestamped data, such as computer metrics, satellite telemetry or IoT device measurements.

The database is organized in domain/parameter categories. That means, datasets are accessed eg. as http://localhost:6502/deviceX/measurementY/.

A dataset is composed of two key value pairs: time and value. Example:

{
    "time": "2021-01-01T10:12:00",
    "value": 123.99
}

Getting Started

First make sure you have Docker and docker-compose installed and running on your machine. Then clone (or download) this repository and spin up the docker container:

$ git clone https://gitlab.com/librecube/elements/LC6502.git timeseries-store
$ cd timeseries-store
$ docker-compose up

At first start, the database will initialize itself. You may need to stop and start the docker container again to have it working properly.

Now the database REST API is exposed to http://localhost:6502.

To change to a different port, modify the docker-compose.yml file.

Tutorial

The REST API provides basic CRUD (create, read, update, and delete) operations for managing entries in the database. For this tutorial we assume the scenario of storing temperature measurements of a satellite.

POST

In order to add data, issue a POST request with a single entry or a list of entries in the request body to a domain/parameter endpoint.

Let’s create a new entry in the parameter temperatureY in the domain satelliteX.

POST /satelliteX/temperatureY

{
    "time": "2021-01-01T10:12:00",
    "value": 123.99
}

At this point this will result in a 404 NOT FOUND error. The reason is that you first need to specify the metadata of a parameter. This needs to be done only once per parameter. Currently the supported datatypes are int, smallint, and float.

Let’s define the parameter to store float values:

POST /_meta

{
    "domain": "satelliteX",
    "name": "temperatureY",
    "dtype": "float"
}

Now retry the posting of data to it to see that the data is stored now as expected.

Issuing a POST request to an existing entry, that is, an entry with a timestamp that already is in the database for that parameter, it will be silently ignored, but not updated.

Successful write operations are confirmed with an HTTP 201 status and the following JSON body:

{
    "count": 1,
    "info": "created"
}

To create two or more new datasets post them as a list.

POST /satelliteX/temperatureY

[
    {
        "time": "2021-01-02T10:44:00",
        "value": 13.961
    },
    {
        "time": "2021-01-03T18:19:00",
        "value": 73.01
    }
]

GET

Successful read operations are confirmed with a HTTP 200 status code and have a JSON object in the response body that contains one or more datasets.

Get a list of all defined domains:

GET /

[
    "satelliteX",
    "roverY",
    "droneZ"
]

Get a list of all parameters in a domain:

GET /satelliteX

[
    "temperatureX",
    "temperatureY",
    "temperatureZ"
]

Get all datasets of a model of a domain:

GET /satelliteX/temperatureY

[
  {
    "time": "2021-01-01 10:12:00",
    "value": 123.99
  },
  {
    "time": "2021-01-02 10:44:00",
    "value": 13.961
  },
  {
    "time": "2021-01-03 18:19:00",
    "value": 73.01
  }
]

Filtering

To get the entry of a specific time:

GET /satelliteX/temperatureY/time=2021-01-01T10:12:00

Note that times should be supplied in ISO format to avoid having a space character in the request.

It is also possible to get two or more specific entries using commas:

GET /satelliteX/temperatureY/time=2021-01-01T10:12:00,2021-01-02T10:44:00

Apart from exact matches, one can also use the following operators:

• in : same as exact match
• lt : lower than
• le : lower than or equal
• gt : greater than
• ge : greater than or equal

To get the entries from a specific time onwards, issue the following:

GET /satelliteX/temperatureY?time=ge:2021-01-02T00:00:00

Limiting

To limit the result to n entries, use the _limit option:

GET /satelliteX/temperatureY?_limit=2

Sorting

Sorting of the returned datasets is done via the _sort option. Sorting can be applied on the time and/or the values. To sort in descending order, use the minus sign as prefix.

GET /satelliteX/temperatureY?_sort=-time

DELETE Operations

Not yet implemented.

HTTP Errors

A few HTTP error codes and messages are defined by the REST API.

• Providing a malformed request (for example, a POST request without JSON body) will return an HTTP 400 status.
• Trying to read a non-existing parameter returns an HTTP 404 status.