add support for feature/record domain queries (#1988)

* add support for feature/record domain queries

* update parameter values based on OGC discussion

* add ES test

Author: tomkralidis

docs/source/data-publishing/ogcapi-features.rst

@@ -16,24 +16,24 @@ parameters.
 
 
 .. csv-table::
-   :header: Provider, property filters/display, resulttype, bbox, datetime, sortby, skipGeometry, CQL, transactions, crs
+   :header: Provider, property filters/display, resulttype, bbox, datetime, sortby, skipGeometry, domains, CQL, transactions, crs
    :align: left
 
-   `CSV`_,✅/✅,results/hits,❌,❌,❌,✅,❌,❌,✅
-   `Elasticsearch`_,✅/✅,results/hits,✅,✅,✅,✅,✅,✅,✅
-   `ERDDAP Tabledap Service`_,❌/❌,results/hits,✅,✅,❌,❌,❌,❌,✅
-   `ESRI Feature Service`_,✅/✅,results/hits,✅,✅,✅,✅,❌,❌,✅
-   `GeoJSON`_,✅/✅,results/hits,❌,❌,❌,✅,❌,❌,✅
-   `MongoDB`_,✅/❌,results,✅,✅,✅,✅,❌,❌,✅
-   `OGR`_,✅/❌,results/hits,✅,❌,❌,✅,❌,❌,✅
-   `OpenSearch`_,✅/✅,results/hits,✅,✅,✅,✅,✅,✅,✅
-   `Oracle`_,✅/✅,results/hits,✅,❌,✅,✅,❌,❌,✅
-   `Parquet`_,✅/✅,results/hits,✅,✅,❌,✅,❌,❌,✅
-   `PostgreSQL`_,✅/✅,results/hits,✅,✅,✅,✅,✅,✅,✅
-   `SQLiteGPKG`_,✅/❌,results/hits,✅,❌,❌,✅,❌,❌,✅
-   `SensorThings API`_,✅/✅,results/hits,✅,✅,✅,✅,❌,✅,✅
-   `Socrata`_,✅/✅,results/hits,✅,✅,✅,✅,❌,❌,✅
-   `TinyDB`_,✅/✅,results/hits,✅,✅,✅,✅,❌,✅,✅
+   `CSV`_,✅/✅,results/hits,❌,❌,❌,✅,❌,❌,❌,✅
+   `Elasticsearch`_,✅/✅,results/hits,✅,✅,✅,✅,✅,✅,✅,✅
+   `ERDDAP Tabledap Service`_,❌/❌,results/hits,✅,✅,❌,❌,❌,❌,❌,✅
+   `ESRI Feature Service`_,✅/✅,results/hits,✅,✅,✅,✅,❌,❌,❌,✅
+   `GeoJSON`_,✅/✅,results/hits,❌,❌,❌,✅,❌,❌,❌,✅
+   `MongoDB`_,✅/❌,results,✅,✅,✅,✅,❌,❌,❌,✅
+   `OGR`_,✅/❌,results/hits,✅,❌,❌,✅,❌,❌,❌,✅
+   `OpenSearch`_,✅/✅,results/hits,✅,✅,✅,✅,❌,✅,✅,✅
+   `Oracle`_,✅/✅,results/hits,✅,❌,✅,✅,❌,❌,❌,✅
+   `Parquet`_,✅/✅,results/hits,✅,✅,❌,✅,❌,❌,❌,✅
+   `PostgreSQL`_,✅/✅,results/hits,✅,✅,✅,✅,❌,✅,✅,✅
+   `SQLiteGPKG`_,✅/❌,results/hits,✅,❌,❌,✅,❌,❌,❌,✅
+   `SensorThings API`_,✅/✅,results/hits,✅,✅,✅,✅,❌,❌,✅,✅
+   `Socrata`_,✅/✅,results/hits,✅,✅,✅,✅,❌,❌,❌,✅
+   `TinyDB`_,✅/✅,results/hits,✅,✅,✅,✅,✅,❌,✅,✅
 
 .. note::
 
@@ -755,18 +755,35 @@ Data access examples
 * list all collections
 
   * http://localhost:5000/collections
+
 * overview of dataset
 
   * http://localhost:5000/collections/foo
+
 * queryables
 
   * http://localhost:5000/collections/foo/queryables
+
+* queryables on specific properties
+
+  * http://localhost:5000/collections/foo/queryables?properties=title,type
+
+* queryables with current domain values
+
+  * http://localhost:5000/collections/foo/queryables?profile=actual-domain
+
+* queryables on specific properties with current domain values
+
+  * http://localhost:5000/collections/foo/queryables?profile=actual-domain&properties=title,type
+
 * browse features
 
   * http://localhost:5000/collections/foo/items
+
 * paging
 
   * http://localhost:5000/collections/foo/items?offset=10&limit=10
+
 * CSV outputs
 
   * http://localhost:5000/collections/foo/items?f=csv
@@ -779,24 +796,31 @@ Data access examples
 * query features (attribute)
 
   * http://localhost:5000/collections/foo/items?propertyname=foo
+
 * query features (temporal)
 
   * http://localhost:5000/collections/foo/items?datetime=2020-04-10T14:11:00Z
+
 * query features (temporal) and sort ascending by a property (if no +/- indicated, + is assumed)
 
   * http://localhost:5000/collections/foo/items?datetime=2020-04-10T14:11:00Z&sortby=+datetime
+
 * query features (temporal) and sort descending by a property
 
   * http://localhost:5000/collections/foo/items?datetime=2020-04-10T14:11:00Z&sortby=-datetime
+
 * query features in a given (and supported) CRS
 
   * http://localhost:5000/collections/foo/items?crs=http%3A%2F%2Fwww.opengis.net%2Fdef%2Fcrs%2FEPSG%2F0%2F32633
+
 * query features in a given bounding BBOX and return in given CRS
 
   * http://localhost:5000/collections/foo/items?bbox=120000,450000,130000,460000&bbox-crs=http%3A%2F%2Fwww.opengis.net%2Fdef%2Fcrs%2FEPSG%2F0%2F28992&crs=http%3A%2F%2Fwww.opengis.net%2Fdef%2Fcrs%2FEPSG%2F0%2F32633
+
 * fetch a specific feature
 
   * http://localhost:5000/collections/foo/items/123
+
 * fetch a specific feature in a given (and supported) CRS
 
   * http://localhost:5000/collections/foo/items/123?crs=http%3A%2F%2Fwww.opengis.net%2Fdef%2Fcrs%2FEPSG%2F0%2F32633

docs/source/data-publishing/ogcapi-records.rst

@@ -15,12 +15,12 @@ pygeoapi core record providers are listed below, along with a matrix of supporte
 parameters.
 
 .. csv-table::
-   :header: Provider, properties (filters), resulttype, q, bbox, datetime, sortby, properties (display), CQL, transactions
+   :header: Provider, properties (filters), resulttype, q, bbox, datetime, sortby, properties (display), domains, CQL, transactions
    :align: left
 
    `ElasticsearchCatalogue`_,✅,results/hits,✅,✅,✅,✅,✅,✅,✅
-   `TinyDBCatalogue`_,✅,results/hits,✅,✅,✅,✅,❌,✅,✅
-   `CSWFacade`_,✅,results/hits,✅,✅,✅,❌,❌,❌,❌
+   `TinyDBCatalogue`_,✅,results/hits,✅,✅,✅,✅,❌,✅,✅,✅,✅
+   `CSWFacade`_,✅,results/hits,✅,✅,✅,❌,❌,✅,❌,❌
 
 
 Below are specific connection examples based on supported providers.
@@ -107,6 +107,18 @@ Metadata search examples
 
   * http://localhost:5000/collections/foo/queryables
 
+* queryables on specific properties
+
+  * http://localhost:5000/collections/foo/queryables?properties=title,type
+
+* queryables with current domain values
+
+  * http://localhost:5000/collections/foo/queryables?profile=actual-domain
+
+* queryables on specific properties with current domain values
+
+  * http://localhost:5000/collections/foo/queryables?profile=actual-domain&properties=title,type
+
 * browse records
 
   * http://localhost:5000/collections/foo/items

pygeoapi/api/itemtypes.py

@@ -7,7 +7,7 @@
 #          Colin Blackburn <colb@bgs.ac.uk>
 #          Ricardo Garcia Silva <ricardo.garcia.silva@geobeyond.it>
 #
-# Copyright (c) 2024 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 # Copyright (c) 2025 Francesco Bartoli
 # Copyright (c) 2022 John A Stevenson and Colin Blackburn
 # Copyright (c) 2023 Ricardo Garcia Silva
@@ -111,6 +111,7 @@ def get_collection_queryables(api: API, request: Union[APIRequest, Any],
     :returns: tuple of headers, status code, content
     """
 
+    domains = {}
     headers = request.get_response_headers(**api.api_headers)
 
     if any([dataset is None,
@@ -135,16 +136,39 @@ def get_collection_queryables(api: API, request: Union[APIRequest, Any],
     if p is None:
         msg = 'queryables not available for this collection'
         return api.get_exception(
+
             HTTPStatus.BAD_REQUEST, headers, request.format,
             'NoApplicableError', msg)
 
+    LOGGER.debug('Processing profile')
+    profile = request.params.get('profile', '')
+
+    LOGGER.debug('Processing properties')
+    val = request.params.get('properties')
+    if val is not None:
+        properties = [x for x in val.split(',') if x]
+        properties_to_check = set(p.properties) | set(p.fields.keys())
+
+        if len(list(set(properties) - set(properties_to_check))) > 0:
+            msg = 'unknown properties specified'
+            return api.get_exception(
+                HTTPStatus.BAD_REQUEST, headers, request.format,
+                'InvalidParameterValue', msg)
+    else:
+        properties = []
+
+    queryables_id = f'{api.get_collections_url()}/{dataset}/queryables'
+
+    if request.params:
+        queryables_id += '?' + urllib.parse.urlencode(request.params)
+
     queryables = {
         'type': 'object',
         'title': l10n.translate(
             api.config['resources'][dataset]['title'], request.locale),
         'properties': {},
         '$schema': 'http://json-schema.org/draft/2019-09/schema',
-        '$id': f'{api.get_collections_url()}/{dataset}/queryables'
+        '$id': queryables_id
     }
 
     if p.fields:
@@ -153,8 +177,17 @@ def get_collection_queryables(api: API, request: Union[APIRequest, Any],
             'x-ogc-role': 'primary-geometry'
         }
 
+    if profile == 'actual-domain':
+        try:
+            domains, _ = p.get_domains(properties)
+        except NotImplementedError:
+            LOGGER.debug('Domains are not suported by this provider')
+            domains = {}
+
     for k, v in p.fields.items():
         show_field = False
+        if properties and k not in properties:
+            continue
         if p.properties:
             if k in p.properties:
                 show_field = True
@@ -175,6 +208,8 @@ def get_collection_queryables(api: API, request: Union[APIRequest, Any],
                 queryables['properties'][k]['x-ogc-role'] = 'id'
             if k == p.time_field:
                 queryables['properties'][k]['x-ogc-role'] = 'primary-instant'  # noqa
+            if domains.get(k):
+                queryables['properties'][k]['enum'] = domains[k]
 
     if request.format == F_HTML:  # render
         tpl_config = api.get_dataset_templates(dataset)
@@ -1057,6 +1092,19 @@ def get_oas_30(cfg: dict, locale: str) -> tuple[list[dict[str, str]], dict[str,
         }
     }
 
+    profile = {
+        'name': 'profile',
+        'in': 'query',
+        'description': 'The profile to be applied to a given request',
+        'required': False,
+        'style': 'form',
+        'explode': False,
+        'schema': {
+            'type': 'string',
+            'enum': ['actual-domain', 'valid-domain']
+        }
+    }
+
     LOGGER.debug('setting up collection endpoints')
     paths = {}
 
@@ -1190,7 +1238,9 @@ def get_oas_30(cfg: dict, locale: str) -> tuple[list[dict[str, str]], dict[str,
                         'tags': [k],
                         'operationId': f'get{k.capitalize()}Queryables',
                         'parameters': [
+                            coll_properties,
                             {'$ref': '#/components/parameters/f'},
+                            profile,
                             {'$ref': '#/components/parameters/lang'}
                         ],
                         'responses': {

pygeoapi/django_/urls.py

@@ -8,7 +8,7 @@
 # Copyright (c) 2025 Francesco Bartoli
 # Copyright (c) 2022 Luca Delucchi
 # Copyright (c) 2022 Krishna Lodha
-# Copyright (c) 2024 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 #
 # Permission is hereby granted, free of charge, to any person
 # obtaining a copy of this software and associated documentation

pygeoapi/flask_app.py

@@ -3,7 +3,7 @@
 # Authors: Tom Kralidis <tomkralidis@gmail.com>
 #          Norman Barker <norman.barker@gmail.com>
 #
-# Copyright (c) 2024 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 #
 # Permission is hereby granted, free of charge, to any person
 # obtaining a copy of this software and associated documentation

pygeoapi/provider/base.py

@@ -2,7 +2,7 @@
 #
 # Authors: Tom Kralidis <tomkralidis@gmail.com>
 #
-# Copyright (c) 2022 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 #
 # Permission is hereby granted, free of charge, to any person
 # obtaining a copy of this software and associated documentation
@@ -145,6 +145,20 @@ def get_metadata(self):
 
         raise NotImplementedError()
 
+    def get_domains(self, properties=[], current=False):
+        """
+        Get domains from dataset
+
+        :param properties: `list` of property names
+        :param current: `bool` of whether to provide list of live
+                        values (default `False`)
+
+        :returns: `tuple` of domains and whether they are based on the
+                  current/live dataset
+        """
+
+        raise NotImplementedError()
+
     def query(self):
         """
         query the provider

pygeoapi/provider/csw_facade.py

@@ -2,7 +2,7 @@
 #
 # Authors: Tom Kralidis <tomkralidis@gmail.com>
 #
-# Copyright (c) 2023 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 #
 # Permission is hereby granted, free of charge, to any person
 # obtaining a copy of this software and associated documentation
@@ -91,6 +91,35 @@ def get_fields(self):
 
         return self._fields
 
+    def get_domains(self, properties=[], current=False) -> tuple:
+        """
+        Get domains from dataset
+
+        :param properties: `list` of property names
+        :param current: `bool` of whether to provide list of live
+                        values (default `False`)
+
+        :returns: `tuple` of domains and whether they are based on the
+                  current/live dataset
+        """
+
+        LOGGER.debug(f'Querying CSW: {self.data}')
+        records = self.query()
+        domains = {}
+
+        if properties:
+            keys = properties
+        else:
+            keys = records['features'][0]['properties'].keys()
+
+        csw = self._get_csw()
+
+        for key in keys:
+            csw.getdomain(key, dtype='property')
+            domains[key] = csw.results['values']
+
+        return domains, True
+
     @crs_transform
     def query(self, offset=0, limit=10, resulttype='results',
               bbox=[], datetime_=None, properties=[], sortby=[],