Datacube API

Datacube API is a system for storing, querying and filtering data-point time series.

Concept

Datacube serves a “data lake” of data-points, an ever increasing storage of versioned, multi-dimensional time-series data. New data from various sources, including but not limited to SpaceKnow algorithms applied on satellite imagery, are constantly streamed to Datacube.

A data-point is a scalar (for example a result of a specific algorithm, e.g. mean NDVI value, or number of detected cars) that can be computed on a given (multi)polygon AoI and one source data sample (usually one satellite image shot or a imagery mosaic). Datacube is a query-able, multi-dimensional data-point time-series storage and API.

Data-points have the following properties (i.e. dimensions of the datacube):

  • version (str) – version of the data-point. For example data can be re-computed when the source algorithm changes or when new interpolation could be computed after new satellite images become available. Once a version of a data-point is uploaded it remains forever available.

    Alphabetically greater versions are considered as newer.For example version ab is considered as newer as version aa.

  • startDatetime, endDatetime (datetime) – a time-range (possible 0 seconds wide) of the source data acquisition (for example time an underlying satellite image was taken).

  • algorithm (str) – a string identification of the underlying algorithm(s) which produced the particular data-point. For example ndmi_mean.

  • project (str) – For example aluminium or coal (but not china-coal).

  • aoi (GeoJSON) – point AoI, this information is not retrievable with the API but data-points can be filtered to a region.

  • aoiId (str) – a unique identifier of the data-point AoI.

  • source (str) – underlying data source identification, for example landsat_8, viirs_nl_monthly.

  • firstSeen (datetime) – date of the initial availability of the data to us, or date when we’ve first seen the data when the former is not available. This value is optional.

  • ingestDatetime (datetime) – time at which the data-point was inserted into Datacube.

  • cloudCover (float) – expected cloudiness in range [0, 1], 0 being cloud-free. Satellite-derived data-points may have diminished quality due to high cloud cover. This value is optional.

  • intersectionRatio (float) – a value in range [0, 1]. Ratio of site’s area that has been analyzed in given sample and total site’s area. This value is optional.

Get Data

This API end-point returns a link to a CSV on Google Cloud Storage with column for each Datacube dimension and another column with values.

Filters

Data can be filtered with so-called filters. Filter on project and algorithm is mandatory for each query. Each filter is a JSON object in the following form.

  • type (str) – type of filter, see list of types below.
  • field (str) – field / axis on which the filter should be applied. See available fields in Concept.
  • params (object) – object with filter parameters which is specific for each filter type.

The following filter-types are available.

  • time-range – keeps only data-points inside a date-range. Filter bounds can be half-open if one of the bounding parameters is left out.

    Parameters:

    • from – a date-time in the form YYYY-mm-dd HH:MM:SS, inclusive.
    • to – a date-time in the form YYYY-mm-dd HH:MM:SS, inclusive.
  • value-range – keeps only data-points inside a value-range. Filter bounds can be half-open if one of the bounding parameters is left out. Missing / null values are filtered out by this filter.

    Parameters:

    • min – a minimum allowed value, inclusive.
    • max – a maximum allowed value, inclusive.
  • value-list – keeps only data-points which have given field set to one of provided values. Needs to be used for project and algorithm. Other possible fields to filter are version, aoiId, source and tag. Behaviour of tag filter is a little different than the rest - all provided values must be matched by given data-point to satisfy the filter.

    Parameters:

    • values (list) – list of allowed string values.
  • geo-intersects – keeps only data-points with AoI within a given area. Parameters:

    • geometryLabel – label of the filtering geometry. For countries lower-case ISO identifier is used. Examples:
      • cz
      • cn

Permission Packages

Each user can be given (multiple) permission packages. Each permission package allows the user to access some data-point subset. A permission package is a list of mandatory filters (see above). The user is allowed to perform a query if she uses all filters from some of her permission packages. In other words, the user must use at least as restrictive filter as one of her permission packages.

Note

Users need regular datacube.get permission to have access to the API endpoint. Permission packages work on-top of that.

All users are automatically granted permission package with value-range filter on field project set to values ['test']. Project test contains test values. This implicit “free” permission package serves for development purposes.

Aggregation

It is possible to retrieve periodic aggregates of the data. In such a case, the original data-points are replaced by average values computed over multiple data-points falling into the same time-period. The aggregation preserves version, algorithm, project, aoiId and source, id est data-points with different values in one of the mentioned fields are treated as separate.

When aggregated data are requested, original start and end date-times are replaced with formatted dates of the time periods. daily (%Y-%m-%d), weekly (%G-W%V, i.e. ISO 8601 week), monthly (%Y-%m), yearly (%Y) aggregations are allowed.

Aggregated datacube output deduplicates data with the same properties. If there are duplicates present, datacube selects only datapoints with newest firstSeen and ingestDatetime. This ensures that average values are calculated from unique values only.

Postprocessing

Datacube API can provide a simple view on top of the query result. This option is available for the asynchronous endpoint only. Postprocessing can be done by adding 'postprocess':'postprocess_name' to request json. The list of supported post-processing operations follows:

  • pivot – groups values by datetime transposes aoi_ids into columns and averages values that were grouped.
  • columns – allows to filter resulting view based on given columns. Requested columns are provided via columns property.

Warning

Important metadata will be lost when using postprocessing.

Needed Permissions: datacube.get

POST /datacube/datapoints/get
Request JSON Object:
 
  • filters (list) – filter on single project and single algorithm is required; list of filters of data. Only data-points matching all filters are returned.
  • aggregate (str) – optional; see Aggregation.

Example request:

{
    "filters": [
        {
            "type": "time-range",
            "field": "startDatetime",
            "params": {
                "from": "2018-01-01 00:10:00"
            }
        },
        {
            "type": "value-range",
            "field": "cloudCover",
            "params": {
                "max": 0.05
            }
        },
        {
            "type": "value-list",
            "field": "project",
            "params": {
                "values": ["industrial_production"]
            }
        },
        {
            "type": "value-list",
            "field": "algorithm",
            "params": {
                "values": ["ndmi_mean"]
            }
        },
        {
            "type": "value-list",
            "field": "tag",
            "params": {
                "values": ["mining", "mine", "ore"]
            }
        },
        {
            "type": "geo-intersects",
            "field": "aoi",
            "params": {
                "geometryLabel": "cz"
            }
        }
    ]
}

Example response:

{
    "csvLink": "https://storage.googleapis.com/spaceknow-devel-datacube.appspot.com/cache/4a405b16348ae606.csv",
    "totalRows": 2648
}

Get Large Amount of Data-points

Some queries to Datacube API can match a larger number of data-points than it is possible to query and return in time span of one HTTP request. Pass the same payload as to /datacube/datapoints/get to the asynchronous endpoint to initiate the query.

POST /datacube/datapoints/get/initiate

Example request:

See the synchronous endpoint.

Example response:

See pipeline initiation.

Needed Permissions: datacube.get

The Tasking API should be used to check whether the result is ready.

Once the result is ready, retrieve it from the retrieve endpoint.

POST /datacube/datapoints/get/retrieve

Example request:

See pipeline retrieve.

Example response:

{
    "csvLink": "https://storage.googleapis.com/spaceknow-devel-datacube.appspot.com/cache/4a405b16348ae606.csv",
    "totalRows": 2648
}

Needed Permissions: datacube.get

List Available AoIs

This endpoint lists all AoIs available for filtering of data-points in Datacube. You may be limited to a subset of these AoIs, please contact SpaceKnow representatives to get access to more.

Needed Permissions: datacube.get

POST /datacube/aois/get

Example request:

{}

Example response:

[
    {
        "label": "cz",
        "name": "Czech Republic"
    },
    {
        "label": "cn",
        "name": "China"
    }
]

List Catalogue of Available Data-points

This endpoint lists catalogue of data-points available in Datacube. Different algorithms are available for different regions, date ranges, and their inputs come from different sources.

Needed Permissions: datacube.get

POST /datacube/catalogue/get

Example request:

{}

Example response:

{
    "ndmi_mean": [
        {
            "version": "160",
            "aoi_label": "cz",
            "aoi_name": "Czech Republic",
            "source": "LANDSAT/LC08/C01/T1",
            "project": "coal",
            "last_update": "2019-02-27 17:39:56",
            "date_ranges": [{
                "from": "2010-01-01",
                "to": "2019-02-28"
            }]
        },
        {
            "version": "160",
            "aoi_label": "cz",
            "aoi_name": "Czech Republic",
            "source": "COPERNICUS/S2",
            "project": "steel",
            "last_update": "2019-02-27 17:39:56",
            "date_ranges": [{
                "from": "2010-01-01",
                "to": "2016-10-12"
            },
            {
                "from": "2017-05-01",
                "to": "2019-02-28"
            }]
        }
    ],
    "radiance_mean": [
        {
            "version": "162",
            "aoi_label": "de",
            "aoi_name": "Germany",
            "source": "VIIRS/Monthly",
            "project": "night_lights",
            "last_update": "2019-02-03 12:19:01",
            "date_ranges": [{
                "from": "2018-01-01",
                "to": "2019-02-28"
            }]
        }
    ]
}