Skip to content
This repository was archived by the owner on Dec 14, 2022. It is now read-only.

Commit 3b55914

Browse files
author
Chris Wiechmann
authored
Merge pull request #201 from Axway-API-Management-Plus/path-params-support
Path params support
2 parents aa71275 + ea2c47f commit 3b55914

File tree

8 files changed

+61
-5
lines changed

8 files changed

+61
-5
lines changed

.github/workflows/logstash.yml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ jobs:
4444
# echo Using API_BUILDER_URL: $API_BUILDER_URL
4545
# echo Using MEMCACHED: $MEMCACHED
4646
# echo Using GEOIP_ENABLED: $GEOIP_ENABLED, GEOIP_CUSTOM_ATTRIBUTE: ${GEOIP_CUSTOM_ATTRIBUTE}, GEOIP_CACHE_SIZE: ${GEOIP_CACHE_SIZE}
47-
# ./logstash-filter-verifier --diff-command="diff -y" --logstash-output --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED --keep-env=GEOIP_ENABLED --keep-env=GEOIP_CUSTOM_ATTRIBUTE --keep-env=GEOIP_CACHE_SIZE --keep-env=EVENTLOG_CUSTOM_ATTR ./logstash/test/http/test-events.json ./logstash/pipelines/EventsPipeline.conf
47+
# ./logstash-filter-verifier --diff-command="diff -y" --logstash-output --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED --keep-env=CACHE_API_PATHS ./logstash/test/http/test-opentrafficlog.json ./logstash/pipelines/OpenTrafficPipeline.conf
4848

4949
run: |
5050
echo Using API_BUILDER_URL: $API_BUILDER_URL
@@ -54,7 +54,7 @@ jobs:
5454
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED --keep-env=DROP_TRACE_MESSAGE_LEVELS ./logstash/test/http/test-tracemessages.json ./logstash/pipelines/TraceMessagesPipeline.conf
5555
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED ./logstash/test/http/test-tracemessages-gmt-4.json ./logstash/pipelines/TraceMessagesPipeline.conf
5656
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED --keep-env=GEOIP_ENABLED --keep-env=GEOIP_CUSTOM_ATTRIBUTE --keep-env=GEOIP_CACHE_SIZE --keep-env=EVENTLOG_CUSTOM_ATTR ./logstash/test/http/test-events.json ./logstash/pipelines/EventsPipeline.conf
57-
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED ./logstash/test/http/test-opentrafficlog.json ./logstash/pipelines/OpenTrafficPipeline.conf
57+
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED --keep-env=CACHE_API_PATHS./logstash/test/http/test-opentrafficlog.json ./logstash/pipelines/OpenTrafficPipeline.conf
5858
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED ./logstash/test/filetransfer/test-opentrafficlog-filetransfer.json ./logstash/pipelines/OpenTrafficPipeline.conf
5959
./logstash-filter-verifier --diff-command="diff -y" --keep-env=API_BUILDER_SSL_CERT --keep-env=API_BUILDER_URL --keep-env=MEMCACHED ./logstash/test/domainAudit/test-domain-audit-events.json ./logstash/pipelines/DomainAuditPipeline.conf
6060
env:
@@ -70,6 +70,7 @@ jobs:
7070
# This is not relevant as we test with HTTP
7171
# But it's required by Logstash to be a valid certificate
7272
API_BUILDER_SSL_CERT: "./config/certificates/apibuilder4elastic.crt"
73+
CACHE_API_PATHS: "/petstore/v2, /api/v1/banana, /api/v2/banana"
7374
- name: API-Builder Logs
7475
if: ${{ always() }}
7576
run: |

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
99
- Wait 5 instead of 2 seconds, before trying to attach ILM-Policy to Index-Template
1010
- It may take a bit longer to obtain Custom-Properties from API-Manager for the Index-Template
1111

12+
### Added
13+
- Added support to cache APIs with Path-Parameter more efficiently to reduce document ingest latency
14+
- See the new parameter: `CACHE_API_PATHS` for more details.
15+
1216
## [4.3.0] 2022-03-11
1317
### Added
1418
- Added Geo-Location map of API-Requests

UPDATE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ On the other hand, the API builder Docker image, as a central component of the s
3737

3838
| Ver | API-Builder | Logstash | Memcached | Filebeat | ANM-Config | Dashboards | Params |Elastic-Config | ELK-Ver. | Notes |
3939
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |:---: | :---: | :--- |
40+
| 4.4.0 | [X](#api-builderlogstashmemcached) | [X](#api-builderlogstashmemcached) | - | - | - | - | [X](#parameters)|- | [7.17.1](#update-elastic-stack-version) | |
4041
| 4.3.0 | [X](#api-builderlogstashmemcached) | [X](#api-builderlogstashmemcached) | - | - | - | [X](#dashboards)| [X](#parameters)|[X](#elastic-config)| [7.17.1](#update-elastic-stack-version) | |
4142
| 4.2.0 | [X](#api-builderlogstashmemcached) | - | - | - | - | [X](#dashboards)| [X](#parameters)|- | [7.17.0](#update-elastic-stack-version) | |
4243
| 4.1.0 | [X](#api-builderlogstashmemcached) | - | - | - | - | - | [X](#parameters)|[X](#elastic-config)| [7.16.3](#update-elastic-stack-version) | |

docker-compose.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ services:
3030
- GEOIP_CUSTOM_ATTRIBUTE=${GEOIP_CUSTOM_ATTRIBUTE:-xForwardedFor}
3131
- xpack.geoip.download.endpoint=${GEOIP_DOWNLOAD_ENDPOINT}
3232
- EVENTLOG_CUSTOM_ATTR=${EVENTLOG_CUSTOM_ATTR}
33+
- CACHE_API_PATHS=${CACHE_API_PATHS}
3334
ports:
3435
- 5044:5044
3536
volumes:

env-sample

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,24 @@ MEMCACHED=memcached:11211
302302
# Used-By: Logstash, API-Builder
303303
# LOOKUP_CACHE_TTL=1200
304304

305+
# ----------------------------------------------------------------------------------------------
306+
# In case an API contains path parameters (e.g. /api/v2/pet/123456789), it may be advisable to
307+
# configure them here to optimize caching. Especially if the path parameter is very variable
308+
# (e.g. customer IDs). Due to the fact that the API request path is then always different
309+
# Logstash cannot cache the looked up API-Details efficiently.
310+
# Therefore, you configure a list of comma separated API paths here. The Logstash pipeline
311+
# checks if the received API request path starts with one of the configured paths, if so, the
312+
# API details are cached with the configured path (/api/v2/pet) instead of the received
313+
# API path (/api/v2/pet/123456789).
314+
# It is recommended to configure the most frequently called paths first in the list to avoid
315+
# unnecessary iterations.
316+
# You can also configure a shorter path. It is important to understand that based on this
317+
# configured API path the API details are cached and of course there should not be any overlap
318+
# between two actually different APIs.
319+
# Defaults to null
320+
# Used-By: Logstash
321+
# CACHE_API_PATHS=/api/v2/petstore, /api/v1/user, ...
322+
305323
# ----------------------------------------------------------------------------------------------
306324
# The maximum heap memory for ES. Xmx should be configured to 50% of the available memory, when
307325
# running ES on a dedicated node. Both values should be same.

helm/templates/elasticApimLogstash/logstash-config.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ data:
3737
{{- if .Values.logstash.lookupCacheTTL }}
3838
LOOKUP_CACHE_TTL: {{ default "" .Values.logstash.lookupCacheTTL | quote }}
3939
{{- end }}
40+
CACHE_API_PATHS: {{ default "" .Values.logstash.cacheAPIPaths | quote }}
4041
GEOIP_ENABLED: {{ default "true" .Values.logstash.geoip.enabled | quote }}
4142
GEOIP_CACHE_SIZE: {{ default "1000" .Values.logstash.geoip.cacheSize | quote }}
4243
GEOIP_CUSTOM_ATTRIBUTE: {{ default "true" .Values.logstash.geoip.customAttribute | quote }}

helm/values.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,10 @@ logstash:
270270
# https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html#plugins-filters-geoip-manage_update
271271
# downloadEndpoint: "http://some-custom-endpoint"
272272

273+
# Configures a list of API-Paths that should be used for caching instead of the complete request path.
274+
# Check the env-sample parameter: CACHE_API_PATHS for more details.
275+
# cacheAPIPaths: "/api/v2/petstore, /api/v1/user, ..."
276+
273277
# Injects the environment variables from the ConfigMaps and Secrets into the
274278
# Logstash container. Specify your own ConfigMaps or Secrets if you don't
275279
# provide Configuration and Secrets as part of this values.yaml.

logstash/pipelines/OpenTrafficPipeline.conf

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -139,14 +139,40 @@ filter {
139139
if([transactionSummary] and [transactionSummary][protocol] in ["http", "https"]) {
140140
# Check, if an apiName is given (not the case for native APIs such as /healthcheck) to be used for the lookup
141141
if([transactionSummary][serviceContext][service]) {
142-
mutate { add_field => { "[@metadata][apiName]" => "%{[transactionSummary][serviceContext][service]}" } }
142+
mutate {
143+
add_field => { "[@metadata][apiName]" => "%{[transactionSummary][serviceContext][service]}" }
144+
add_field => { "[@metadata][cacheAPIPaths]" => "${CACHE_API_PATHS:''}" }
145+
}
143146
} else {
144147
# Without an API name, only API path is used for the API lookup.
145148
mutate { add_field => { "[@metadata][apiName]" => "" } }
146149
}
147-
# Create a key for the API
150+
# Create a cache key for the API either based on the received request path or the configured path.
151+
# For example: /v1/get/pet/687687678 --> CACHE_API_PER_NAME = /v1/get/pet makes sure the API is cached based on the API-Name
152+
ruby {
153+
code => '
154+
cacheAPIPaths = event.get("[@metadata][cacheAPIPaths]");
155+
apiRequestPath = event.get("[transactionSummary][path]");
156+
if (cacheAPIPaths.nil? || cacheAPIPaths.empty?)
157+
event.set("[@metadata][apiCacheKeyPrefix]", apiRequestPath);
158+
return;
159+
end
160+
apiName = event.get("[transactionSummary][serviceContext][service]");
161+
logger.info("Configured paths and current API-Request path: ", { "cacheAPIPaths" => cacheAPIPaths, "apiRequestPath" => apiRequestPath } );
162+
event.set("[@metadata][apiCacheKeyPrefix]", apiRequestPath);
163+
for configuredPath in cacheAPIPaths.split(",") do
164+
if(apiRequestPath.start_with?(configuredPath) )
165+
logger.info("Using configured path as primary cache key as it matches to request path. ", { "configuredPath" => configuredPath, "apiRequestPath" => apiRequestPath });
166+
event.set("[@metadata][apiCacheKeyPrefix]", configuredPath);
167+
break;
168+
end
169+
end
170+
'
171+
}
148172
mutate {
149-
add_field => { "apiCacheKey" => "%{[transactionSummary][path]}###%{[processInfo][groupId]}###%{[processInfo][gatewayRegion]}" }
173+
add_field => {
174+
apiCacheKey => "%{[@metadata][apiCacheKeyPrefix]}###%{[processInfo][groupId]}###%{[processInfo][gatewayRegion]}"
175+
}
150176
}
151177
# Lookup the cache with the created API-Key (API-Name---API-Path)
152178
memcached {

0 commit comments

Comments
 (0)