LibreOperations / Data Archive / Documents Store

This LibreCube element lets you quickly set up a REST API database for storing large amounts of data. It is designed for storing schemaless (NoSQL) data in the form of documents; think of storing JSON data.

It may be used for storing large amounts of documents, whose data structure may only be known during runtime and thus can be created on the fly.

The database is organized in domain/model categories. That means, documents are accessed eg. as http://localhost:6501/my_project/my_tasks/.

Getting Started

First make sure you have Docker and docker-compose installed and running on your machine. Then clone (or download) this repository and spin up the docker container:

$ git clone https://gitlab.com/librecube/elements/LC6501.git documents-store
$ cd documents-store
$ docker-compose up

Now the database REST API is exposed to http://localhost:6501.

To change to a different port, modify the docker-compose.yml file.

If you don’t have Docker or just want to go through the pain of installing everything manually then follow these steps. First install MongoDb on you computer.

Then clone the repository:

$ git clone https://gitlab.com/librecube/elements/LC6501.git datastore
$ cd datastore

Next step is to install the Python dependencies in a virtual environment:

$ python -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Finally, start the application either from Python or as a script:

>>> from application import Application
>>> app = Application()
>>> app.run(port=6501)

Tutorial

The REST API provides basic CRUD (create, read, update, and delete) operations for managing entries in the database. For this tutorial we assume the scenario of booking contact times (refered to as passes) between groundstations and satellites.

POST

In order to add data (in the form of JSON documents) to the database, issue a POST request with a single document or a list of documents in the request body to a domain/model endpoint.

Let’s create a new entry in the model passes in the domain missionA.

POST /missionA/passes

{
    "satellite": "SatX",
    "groundstation": "AntennaY",
    "start": "2020-01-01T10:00:00.000000",
    "end": "2020-01-01T11:30:00.000000"
}

Values are stored in the datatype that they are provided. An integer is saved as integer, float as float, string as string etc. One exception is for datetime objects because they cannot be natively expressed as JSON objects. Commonly timestamps, that is integer numbers, are used, but those are not readable to humans. Thus, as a convention, it is better to convert them to strings. The recommended format is ISO datetime format: “2018-08-30T11:09:28.411410”. The reason to use microsecond format is to allow for correct filtering (because filtering is applied on strings, so they must have same length).

Successful write operations are confirmed with an HTTP 201 status and the following JSON body:

{
    "count": 1,
    "info": "created"
}

Writing to domains and/or models that do not exist will create them.

An _id field is generated automatically when new documents are added to a model. It is a string of hex values that is unique within that model. You must not supply a value for _id in the document.

Create two new documents in a domain model.

POST /missionA/passes

[
    {
        "satellite": "SatX",
        "groundstation": "AntennaZ",
        "start": "2020-01-02T11:00:00.000000",
        "end": "2020-01-02T11:30:00.000000"
    },
    {
        "satellite": "SatX",
        "groundstation": "AntennaZ",
        "start": "2020-02-01T10:00:00.000000",
        "end": "2020-02-01T11:30:00.000000"
    }
]

Add yet another document, this time supplying adding another field and leaving one out.

POST /missionA/passes

{
    "satellite": "SatX",
    "groundstation": "AntennaZ",
    "start": "2020-02-01T10:00:00.000000",
    "end": "2020-02-01T11:30:00.000000",
    "note": "This pass may be cancelled."
}

As shown, documents within a domain can have different fields.

PUT

To replace an existing document, use a PUT request with the new document in the request body. You can only replace a document that exists and you must know its _id value to form the URL endpoint.

PUT /missionA/passes/25b87bfef2ce90e0108668ba5

{
    "title": "Just a test",
    "hello": "world"
}

This replaces the document with the _id = 25b87bfef2ce90e0108668ba5 and gives an status code of 200, providing that the document existed already. If not, a not found error (404) will be returned.

You must not supply a value for _id in the document.

PATCH

To modify an existing document (that is, to add/change selected fields), use a PATCH request with a document that encompasses the fields to be modified in the request body. You can only modify a document that exists and you must know its _id value to form the URL endpoint.

PATCH /cinema/best-movies/25b87bfef2ce90e0108661ba9

{
    "note": "This pass is confirmed."
}

This modifies the document with the _id = 25b87bfef2ce90e0108661ba9 and gives an status code of 200, providing that the document existed already. If not, a not found error (404) will be returned.

You must not supply a value for _id in the document.

The result of this operation is that the provided fields will be added to or overwrite fields in the existing document, while keeping all other fields.

GET

Successful read operations are confirmed with a HTTP 200 status code and have a JSON object in the response body that contains one or more documents.

Get a list of all defined domains:

GET /

[
    "missionA",
    "missionB",
    "missionC"
]

Get a list of all models in a domain:

GET /missionA

[
    "passes",
    "eclipses",
    "schedules"
]

Get a specific document:

GET /missionA/passes/5e7b96d8a9d98e6f63ad7fa7

{
    "_id": "5e7b96d8a9d98e6f63ad7fa7",
    "end": "2020-01-02T11:30:00.000000",
    "groundstation": "AntennaZ",
    "satellite": "SatX",
    "start": "2020-01-02T11:00:00.000000"
}

Documents are identified through their unique _id field.

Get all documents of a model of a domain:

GET /missionA/passes

[
    {
        "_id": "5e7b96d8a9d98e6f63ad7fa7",
        "end": "2020-01-02T11:30:00.000000",
        "groundstation": "AntennaX",
        "satellite": "SatX",
        "start": "2020-01-02T11:00:00.000000"
    },
    {
        "_id": "5e7b96d8a9d98e6f63ad7fa8",
        "end": "2020-02-01T11:30:00.000000",
        "groundstation": "AntennaY",
        "satellite": "SatX",
        "start": "2020-02-01T10:00:00.000000"
    },
    {
        "_id": "5e7b96e0a9d98e6f63ad7fa9",
        "end": "2020-02-01T11:30:00.000000",
        "groundstation": "AntennaZ",
        "note": "This pass is confirmed.",
        "satellite": "SatX",
        "start": "2020-02-01T10:00:00.000000"
    },
    {
        "_id": "5e7b96e0a9d98e6f63ad7faa",
        "end": "2020-02-01T11:30:00.000000",
        "groundstation": "AntennaZ",
        "note": "This pass is confirmed.",
        "satellite": "SatX",
        "start": "2020-02-01T10:00:00.000000"
    }
]

Filtering

Search for entries with a field of specific value(s):

GET /missionA/passes?groundstation=AntennaX,AntennaY

Filters can be concatenated:

GET /missionA/passes?groundstation=AntennaX,AntennaY&satellite=SatX

Next to these exact matches, one can also use the following operators:

• in : same as exact match
• lt : lower than
• le : lower than or equal
• gt : greater than
• ge : greater than or equal
• nin: not in
• re: a regular expression, see here

To get the entries from a specific time onwards, issue the following:

GET /missionA/passes?start=ge:2020-01-01T00:00:00.000000

Paging and Limiting

To obtain only a subset of the queryed results, pagination can be used. The number of documents to return is controlled via the _pagesize parameter, whose default is 100. The page to return is specified with the _page parameter.

Page numbering starts from page 1.

For instance, to return entries from 20 to 29 (page 3):

GET /missionA/passes?_page=3&_pagesize=10

To simply limit the result to n entries, use the _limit parameter:

GET /missionA/passes?_limit=10

Projection

Projection limits the fields to return for all matching documents, specifying which fields to be returned. This is done via the _fields parameter:

GET /missionA/passes?_fields=satellite,groundstation

The _id field is always included in the returned documents.

Sorting

Sorting of the returned documents is done via the _sort parameter. To sort for several fields, pass them as a list. To sort in descending order (from large to small), prefix the field name with a minus sign. Default is ascending order, for which a plus sign can be used optionally.

GET /missionA/passes?_sort=-satellite,groundstation

DELETE Operations

The delete operations can be applied to individual documents, a range of documents, the full model, and even the entire domain.

To delete a single document, e.g. document with _id = 5b7eb2b348e7552fdc54a31b:

DELETE /missionA/passes/5b7eb2b348e7552fdc54a31b

Successful delete operations are confirmed with an HTTP 200 status and the following JSON body:

{
    "count": 1,
    "info": "deleted"
}

Filtering can be used to deleted several entries at once:

DELETE /missionA/passes?groundstation=in:AntennaX,AntennaY

To delete a model and domain, issue the following statements, respectively:

DELETE /missionA/passes
DELETE /missionA

HTTP Errors

A few HTTP error codes and messages are defined by the REST API.

• Providing a malformed request (for example, a POST request without JSON body) will return an HTTP 400 status.
• Providing a request with unexpected/forbidden values (for example, providing an _id field in POST/PUT/PATCH requests) will return an HTTP 422 status.
• Trying to read a non-existing resource returns an HTTP 404 status.

Database Dumps

To export a dump of the entire database content, run in the command line:

curl http://localhost:6501/_export --output dump.zip

To export contents from a certain domain only:

curl http://localhost:6501/_export/mydomain --output dump.zip

To export contents from a certain domain model only:

curl http://localhost:6501/_export/mydomain/mymodel --output dump.zip

Then to import from this dump file run:

curl -F 'dump=@./dump.zip' http://localhost:6501/_import