Skip to content

Commit fa782bc

Browse files
joscalvjoscalv
authored
Merge pull request #343 from tinybirdco/341-use-case-drop-column-base
Add drop column base use case
2 parents 199bf7b + 2015528 commit fa782bc

21 files changed

+653
-0
lines changed

‎drop_column/.tinyenv‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
2+
# VERSION format is major.minor.patch-post where major, minor, patch and post are integer numbers
3+
# bump post to deploy to the current live Release, rollback to previous post version is not available
4+
# bump patch or minor to deploy a new Release and auto-promote it to live. Add TB_AUTO_PROMOTE=0 to create the Release in preview status
5+
# bump major to deploy a new Release in preview status
6+
VERSION=0.0.0
7+
8+
9+
10+
##########
11+
# OPTIONAL env vars
12+
# Deploy a new Release in preview status (default is 1)
13+
# TB_AUTO_PROMOTE=0
14+
15+
# Check if deploy requires backfilling on preview (default is 1)
16+
# TB_CHECK_BACKFILL_REQUIRED=0
17+
18+
# Force old Releases deletion on promote (default is 0)
19+
# Setting it to 1 will remove oldest rollback Releases even when some resource is still in use
20+
# TB_FORCE_REMOVE_OLDEST_ROLLBACK=0
21+
22+
# Don't print CLI version warning message if there's a new available version
23+
# TB_VERSION_WARNING=0
24+
25+
# Skip regression tests
26+
# TB_SKIP_REGRESSION=0
27+
28+
# Use `OBFUSCATE_REGEX_PATTERN` and `OBFUSCATE_PATTERN_SEPARATOR` environment variables to define a regex pattern and a separator (in case of a single string with multiple regex) to obfuscate secrets in the CLI output.
29+
# OBFUSCATE_REGEX_PATTERN="https://(www\.)?[^/]+||^Follow these instructions =>"
30+
# OBFUSCATE_PATTERN_SEPARATOR=||
31+
##########

‎drop_column/datasources/analytics_events.datasource‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
TOKEN "tracker" APPEND
2+
3+
DESCRIPTION >
4+
Analytics events landing data source
5+
6+
SCHEMA >
7+
`timestamp` DateTime `json:$.timestamp`,
8+
`session_id` String `json:$.session_id`,
9+
`action` LowCardinality(String) `json:$.action`,
10+
`version` LowCardinality(String) `json:$.version`,
11+
`payload` String `json:$.payload`
12+
13+
ENGINE "MergeTree"
14+
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
15+
ENGINE_SORTING_KEY "timestamp"
16+
ENGINE_TTL "timestamp + toIntervalDay(60)"

‎drop_column/datasources/analytics_pages_mv.datasource‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
SCHEMA >
3+
`date` Date,
4+
`device` String,
5+
`browser` String,
6+
`location` String,
7+
`pathname` String,
8+
`visits` AggregateFunction(uniq, String),
9+
`hits` AggregateFunction(count)
10+
11+
ENGINE "AggregatingMergeTree"
12+
ENGINE_PARTITION_KEY "toYYYYMM(date)"
13+
ENGINE_SORTING_KEY "date, device, browser, location, pathname"

‎drop_column/datasources/analytics_sessions_mv.datasource‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
SCHEMA >
3+
`date` Date,
4+
`session_id` String,
5+
`device` SimpleAggregateFunction(any, String),
6+
`browser` SimpleAggregateFunction(any, String),
7+
`location` SimpleAggregateFunction(any, String),
8+
`first_hit` SimpleAggregateFunction(min, DateTime),
9+
`latest_hit` SimpleAggregateFunction(max, DateTime),
10+
`hits` AggregateFunction(count)
11+
12+
ENGINE "AggregatingMergeTree"
13+
ENGINE_PARTITION_KEY "toYYYYMM(date)"
14+
ENGINE_SORTING_KEY "date, session_id"

‎drop_column/datasources/analytics_sources_mv.datasource‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
SCHEMA >
3+
`date` Date,
4+
`device` String,
5+
`browser` String,
6+
`location` String,
7+
`referrer` String,
8+
`visits` AggregateFunction(uniq, String),
9+
`hits` AggregateFunction(count)
10+
11+
ENGINE "AggregatingMergeTree"
12+
ENGINE_PARTITION_KEY "toYYYYMM(date)"
13+
ENGINE_SORTING_KEY "date, device, browser, location, referrer"

‎drop_column/datasources/fixtures/.gitkeep‎

Whitespace-only changes.

‎drop_column/pipes/analytics_hits.pipe‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
DESCRIPTION >
2+
Parsed `page_hit` events, implementing `browser` and `device` detection logic.
3+
4+
5+
TOKEN "dashboard" READ
6+
7+
NODE parsed_hits
8+
DESCRIPTION >
9+
Parse raw page_hit events
10+
11+
SQL >
12+
13+
SELECT
14+
timestamp,
15+
action,
16+
version,
17+
coalesce(session_id, '0') as session_id,
18+
JSONExtractString(payload, 'locale') as locale,
19+
JSONExtractString(payload, 'location') as location,
20+
JSONExtractString(payload, 'referrer') as referrer,
21+
JSONExtractString(payload, 'pathname') as pathname,
22+
JSONExtractString(payload, 'href') as href,
23+
lower(JSONExtractString(payload, 'user-agent')) as user_agent
24+
FROM analytics_events
25+
where action = 'page_hit'
26+
27+
28+
29+
NODE endpoint
30+
SQL >
31+
32+
SELECT
33+
timestamp,
34+
action,
35+
version,
36+
session_id,
37+
location,
38+
referrer,
39+
pathname,
40+
href,
41+
case
42+
when match(user_agent, 'wget|ahrefsbot|curl|urllib|bitdiscovery|\+https://|googlebot')
43+
then 'bot'
44+
when match(user_agent, 'android')
45+
then 'mobile-android'
46+
when match(user_agent, 'ipad|iphone|ipod')
47+
then 'mobile-ios'
48+
else 'desktop'
49+
END as device,
50+
case
51+
when match(user_agent, 'firefox')
52+
then 'firefox'
53+
when match(user_agent, 'chrome|crios')
54+
then 'chrome'
55+
when match(user_agent, 'opera')
56+
then 'opera'
57+
when match(user_agent, 'msie|trident')
58+
then 'ie'
59+
when match(user_agent, 'iphone|ipad|safari')
60+
then 'safari'
61+
else 'Unknown'
62+
END as browser
63+
FROM parsed_hits
64+
65+

‎drop_column/pipes/analytics_pages.pipe‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
NODE analytics_pages_1
2+
DESCRIPTION >
3+
Aggregate by pathname and calculate session and hits
4+
5+
SQL >
6+
7+
SELECT
8+
toDate(timestamp) AS date,
9+
device,
10+
browser,
11+
location,
12+
pathname,
13+
uniqState(session_id) AS visits,
14+
countState() AS hits
15+
FROM analytics_hits
16+
GROUP BY date, device, browser, location, pathname
17+
18+
TYPE materialized
19+
DATASOURCE analytics_pages_mv
20+
21+

‎drop_column/pipes/analytics_sessions.pipe‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
NODE analytics_sessions_1
2+
DESCRIPTION >
3+
Aggregate by session_id and calculate session metrics
4+
5+
SQL >
6+
7+
SELECT
8+
toDate(timestamp) AS date,
9+
session_id,
10+
anySimpleState(device) AS device,
11+
anySimpleState(browser) AS browser,
12+
anySimpleState(location) AS location,
13+
minSimpleState(timestamp) AS first_hit,
14+
maxSimpleState(timestamp) AS latest_hit,
15+
countState() AS hits
16+
FROM analytics_hits
17+
GROUP BY date, session_id
18+
19+
TYPE materialized
20+
DATASOURCE analytics_sessions_mv
21+
22+

‎drop_column/pipes/analytics_sources.pipe‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
NODE analytics_sources_1
2+
DESCRIPTION >
3+
Aggregate by referral and calculate session and hits
4+
5+
SQL >
6+
7+
WITH (SELECT domainWithoutWWW(href) FROM analytics_hits LIMIT 1) AS currenct_domain
8+
SELECT
9+
toDate(timestamp) AS date,
10+
device,
11+
browser,
12+
location,
13+
referrer,
14+
uniqState(session_id) AS visits,
15+
countState() AS hits
16+
FROM analytics_hits
17+
WHERE domainWithoutWWW(referrer) != currenct_domain
18+
GROUP BY date, device, browser, location, referrer
19+
20+
TYPE materialized
21+
DATASOURCE analytics_sources_mv
22+
23+

0 commit comments

Comments
 (0)