Skip to content

Commit b908dd9

Browse files
himanshu219himanshu219
authored
Merge pull request #123 from SumoLogic/hpal_SUMO-252521
Added support for Virtual Network Flow Logs
2 parents 64bac4f + a0a0293 commit b908dd9

File tree

16 files changed

+332
-98
lines changed

16 files changed

+332
-98
lines changed

‎AppendBlobReader/README.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -35,16 +35,17 @@ This command copies required files in `AppendBlobReader/target` directory
3535

3636
Integrations tests are in `AppendBlobReader/tests` folder and unit tests are in sumo-`function-utils/tests` folder
3737

38+
Modify the run_integration_test.sh file with below parameters
3839
```console
3940

40-
export AZURE_SUBSCRIPTION_ID=`<Your azure subscription id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription>`
41-
export AZURE_CLIENT_ID=`Your application id which you can get after registering application. Refer https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application`
42-
export AZURE_CLIENT_SECRET=`Generate client secret by referring docs https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#add-credentials`
43-
export AZURE_TENANT_ID=`You tenant id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-microsoft-entra-tenant`
44-
export AZURE_DEFAULT_REGION=`eastus`
45-
export SUMO_ACCESS_ID=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
46-
export SUMO_ACCESS_KEY=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
47-
export SUMO_DEPLOYMENT=`Enter one of the allowed values au, ca, de, eu, fed, in, jp, us1 or us2. Visit https://help.sumologic.com/APIs/General-API-Information/Sumo-Logic-Endpoints-and-Firewall-Security`
41+
AZURE_SUBSCRIPTION_ID=`<Your azure subscription id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription>`
42+
AZURE_CLIENT_ID=`Your application id which you can get after registering application. Refer https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application`
43+
AZURE_CLIENT_SECRET=`Generate client secret by referring docs https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#add-credentials`
44+
AZURE_TENANT_ID=`You tenant id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-microsoft-entra-tenant`
45+
AZURE_DEFAULT_REGION=`eastus`
46+
SUMO_ACCESS_ID=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
47+
SUMO_ACCESS_KEY=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
48+
SUMO_DEPLOYMENT=`Enter one of the allowed values au, ca, de, eu, fed, in, jp, us1 or us2. Visit https://help.sumologic.com/APIs/General-API-Information/Sumo-Logic-Endpoints-and-Firewall-Security`
4849
```
4950

5051
Execute below command under `AppendBlobReader/tests` directory

‎AppendBlobReader/tests/run_integration_test.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/bash
2+
3+
export AZURE_SUBSCRIPTION_ID=""
4+
# application id
5+
export AZURE_CLIENT_ID=""
6+
export AZURE_CLIENT_SECRET=""
7+
export AZURE_TENANT_ID=""
8+
export AZURE_DEFAULT_REGION="eastus"
9+
export SUMO_ACCESS_ID=""
10+
export SUMO_ACCESS_KEY=""
11+
export SUMO_DEPLOYMENT="us1"
12+
export TEMPLATE_NAME="appendblobreaderdeploy.json"
13+
python test_appendblobreader.py
14+
# For deleting leftover resources in case of failures
15+
# python ~/git/sumologic-azure-function/deletetestresourcegroups.py

‎BlockBlobReader/README.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -37,19 +37,21 @@ Integrations tests are in `BlockBlobReader/tests` folder and unit tests are in `
3737

3838
Integrations tests are in `BlockBlobReader/tests` folder and unit tests are in sumo-`function-utils/tests` folder
3939

40-
```console
40+
Modify the run_integration_test.sh file with below parameters
4141

42-
export AZURE_SUBSCRIPTION_ID=`<Your azure subscription id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription>`
43-
export AZURE_CLIENT_ID=`Your application id which you can get after registering application. Refer https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application`
44-
export AZURE_CLIENT_SECRET=`Generate client secret by referring docs https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#add-credentials`
45-
export AZURE_TENANT_ID=`You tenant id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-microsoft-entra-tenant`
46-
export AZURE_DEFAULT_REGION=`eastus`
47-
export SUMO_ACCESS_ID=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
48-
export SUMO_ACCESS_KEY=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
49-
export SUMO_DEPLOYMENT=`Enter one of the allowed values au, ca, de, eu, fed, in, jp, us1 or us2. Visit https://help.sumologic.com/APIs/General-API-Information/Sumo-Logic-Endpoints-and-Firewall-Security`
42+
```console
5043

44+
AZURE_SUBSCRIPTION_ID=`<Your azure subscription id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-subscription>`
45+
AZURE_CLIENT_ID=`Your application id which you can get after registering application. Refer https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#register-an-application`
46+
AZURE_CLIENT_SECRET=`Generate client secret by referring docs https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app#add-credentials`
47+
AZURE_TENANT_ID=`You tenant id, to obtain it refer docs https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-microsoft-entra-tenant`
48+
AZURE_DEFAULT_REGION=`eastus`
49+
SUMO_ACCESS_ID=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
50+
SUMO_ACCESS_KEY=`<Generate access id and access key https://help.sumologic.com/docs/manage/security/access-keys/#create-your-access-key>`
51+
SUMO_DEPLOYMENT=`Enter one of the allowed values au, ca, de, eu, fed, in, jp, us1 or us2. Visit https://help.sumologic.com/APIs/General-API-Information/Sumo-Logic-Endpoints-and-Firewall-Security`
5152
```
5253

54+
5355
Execute below command under `BlockBlobReader/tests` directory
5456

5557
`python test_blobreader.py`

‎BlockBlobReader/src/CHANGELOG.md‎

Lines changed: 0 additions & 32 deletions
This file was deleted.

‎BlockBlobReader/src/blobreaderzipdeploy.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -419,7 +419,7 @@
419419
"[variables('BlobReader_resourceId')]"
420420
],
421421
"properties": {
422-
"packageUri": "https://appdev-cloudformation-templates.s3.amazonaws.com/AzureBlobReader/taskproducer4.1.4.zip",
422+
"packageUri": "https://appdev-cloudformation-templates.s3.amazonaws.com/AzureBlobReader/taskproducer4.1.5.zip",
423423
"appOffline": true
424424
}
425425
}
@@ -512,7 +512,7 @@
512512
"[variables('blobreaderconsumer_resourceId')]"
513513
],
514514
"properties": {
515-
"packageUri": "https://appdev-cloudformation-templates.s3.amazonaws.com/AzureBlobReader/taskconsumer4.1.4.zip",
515+
"packageUri": "https://appdev-cloudformation-templates.s3.amazonaws.com/AzureBlobReader/taskconsumer4.1.5.zip",
516516
"appOffline": true
517517
}
518518
}
@@ -609,7 +609,7 @@
609609
"[variables('DLQProcessor_resourceId')]"
610610
],
611611
"properties": {
612-
"packageUri": "https://appdev-cloudformation-templates.s3.amazonaws.com/AzureBlobReader/dlqprocessor4.1.4.zip",
612+
"packageUri": "https://appdev-cloudformation-templates.s3.amazonaws.com/AzureBlobReader/dlqprocessor4.1.5.zip",
613613
"appOffline": true
614614
}
615615
}

‎BlockBlobReader/src/consumer.js‎

Lines changed: 68 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ async function setAppendBlobOffset(context, serviceBusTask, newOffset) {
218218

219219
async function nsgLogsHandler(context, msg, serviceBusTask) {
220220

221-
var jsonArray = [];
221+
let jsonArray = [];
222222
msg = msg.trim().replace(/(^,)|(,$)/g, ""); //removing trailing spaces,newlines and leftover commas
223223

224224
try {
@@ -236,7 +236,7 @@ async function nsgLogsHandler(context, msg, serviceBusTask) {
236236

237237
}
238238

239-
var eventsArr = [];
239+
let eventsArr = [];
240240
jsonArray.forEach(function (record) {
241241
let version = record.properties.Version;
242242
record.properties.flows.forEach(function (rule) {
@@ -282,6 +282,66 @@ async function nsgLogsHandler(context, msg, serviceBusTask) {
282282
return eventsArr;
283283
}
284284

285+
async function networkFlowLogsHandler(context, msg, serviceBusTask) {
286+
287+
let jsonArray = [];
288+
msg = msg.trim().replace(/(^,)|(,$)/g, ""); //removing trailing spaces,newlines and leftover commas
289+
290+
try {
291+
jsonArray = JSON.parse("[" + msg + "]");
292+
} catch(err) {
293+
let response = getParseableJsonArray(msg, context, serviceBusTask);
294+
jsonArray = response[0];
295+
let is_success = response[2];
296+
let newOffset = response[1] + serviceBusTask.startByte;
297+
if (is_success) {
298+
await setAppendBlobOffset(context, serviceBusTask, newOffset);
299+
} else {
300+
return jsonArray;
301+
}
302+
303+
}
304+
// Format: https://learn.microsoft.com/en-us/azure/network-watcher/vnet-flow-logs-overview#log-format
305+
let eventsArr = [];
306+
jsonArray.forEach(function (record) {
307+
record.flowRecords.flows.forEach(function (acl) {
308+
acl.flowGroups.forEach(function (rule) {
309+
rule.flowTuples.forEach(function (tuple) {
310+
let col = tuple.split(",");
311+
let event = {
312+
time: col[0], // Time stamp of when the flow occurred, in UNIX epoch format.
313+
flowLogGUID: record.flowLogGUID,
314+
category: record.category,
315+
flow_log_resource_id: record.flowLogResourceID,
316+
target_resource_id: record.targetResourceID,
317+
event_name: record.operationName,
318+
acl_id: acl.aclID,
319+
rule_name: rule.rule,
320+
mac: record.macAddress,
321+
src_ip: col[1],
322+
dest_IP: col[2],
323+
src_port: col[3],
324+
dest_port: col[4],
325+
protocol: col[5],
326+
flow_direction: col[6],
327+
flow_state: (col[7] === "" || col[7] === undefined) ? null : col[7],
328+
flow_encryption_status: (col[8] === "" || col[8] === undefined) ? null : col[8],
329+
num_packets_sent_src_to_dest: (col[9] === "" || col[9] === undefined) ? null : col[9],
330+
bytes_sent_src_to_dest: (col[10] === "" || col[10] === undefined) ? null : col[10],
331+
num_packets_sent_dest_to_src: (col[11] === "" || col[11] === undefined) ? null : col[11],
332+
bytes_sent_dest_to_src: (col[12] === "" || col[12] === undefined) ? null : col[12],
333+
version: record.flowLogVersion
334+
335+
}
336+
eventsArr.push(event);
337+
});
338+
});
339+
});
340+
});
341+
return eventsArr;
342+
}
343+
344+
285345
function jsonHandler(context,msg) {
286346
// it's assumed that json is well formed {},{}
287347
var jsonArray = [];
@@ -347,13 +407,13 @@ function messageHandler(serviceBusTask, context, sumoClient) {
347407
if (file_ext == serviceBusTask.blobName) {
348408
file_ext = "log";
349409
}
350-
var msghandler = {"log": logHandler, "csv": csvHandler, "json": jsonHandler, "blob": blobHandler, "nsg": nsgLogsHandler};
410+
var msghandler = {"log": logHandler, "csv": csvHandler, "json": jsonHandler, "blob": blobHandler, "nsg": nsgLogsHandler, "vnetflowlogs": networkFlowLogsHandler};
351411
if (!(file_ext in msghandler)) {
352412
context.log.error("Error in messageHandler: Unknown file extension - " + file_ext + " for blob: " + serviceBusTask.blobName);
353413
context.done();
354414
return;
355415
}
356-
if ((file_ext === "json") && (serviceBusTask.containerName === "insights-logs-networksecuritygroupflowevent")) {
416+
if ((file_ext === "json") && (serviceBusTask.containerName === "insights-logs-networksecuritygroupflowevent" || serviceBusTask.containerName === "insights-logs-flowlogflowevent")) {
357417
// because in json first block and last block remain as it is and azure service adds new block in 2nd last pos
358418
if ((serviceBusTask.endByte < JSON_BLOB_HEAD_BYTES + JSON_BLOB_TAIL_BYTES) || (serviceBusTask.endByte == serviceBusTask.startByte)) {
359419
context.done(); //rejecting first commit when no data is there data will always be atleast HEAD_BYTES+DATA_BYTES+TAIL_BYTES
@@ -365,7 +425,7 @@ function messageHandler(serviceBusTask, context, sumoClient) {
365425
} else {
366426
serviceBusTask.startByte -= 1; //to remove comma before json object
367427
}
368-
file_ext = "nsg";
428+
file_ext = serviceBusTask.containerName === "insights-logs-networksecuritygroupflowevent" ? "nsg" : "vnetflowlogs";
369429
}
370430
getBlockBlobService(context, serviceBusTask).then(function (blobService) {
371431
return getData(serviceBusTask, blobService, context).then(async function (msg) {
@@ -385,10 +445,10 @@ function messageHandler(serviceBusTask, context, sumoClient) {
385445
context.done(err);
386446
});
387447
} else {
388-
if (file_ext == "nsg") {
389-
messageArray = await nsgLogsHandler(context, msg, serviceBusTask);
448+
if (file_ext === "nsg" || file_ext === "vnetflowlogs") {
449+
messageArray = await msghandler[file_ext](context, msg, serviceBusTask);
390450
} else {
391-
messageArray = msghandler[file_ext](context,msg);
451+
messageArray = msghandler[file_ext](context, msg);
392452
}
393453
messageArray.forEach(function (msg) {
394454
sumoClient.addData(msg);

‎BlockBlobReader/src/create_zip.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ else
8080
fi
8181

8282
echo "creating zip"
83-
version="4.1.4"
83+
version="4.1.5"
8484
producer_zip_file="taskproducer$version.zip"
8585
consumer_zip_file="taskconsumer$version.zip"
8686
dlqprocessor_zip_file="dlqprocessor$version.zip"

0 commit comments

Comments
 (0)