Skip to content

Commit 33aabec

Browse files
authored
feat: Add new sub-module for AWS flow log (#1241)
* feat: Add new sub-module for AWS flow log * chore: Add notes/docs based on PR review
1 parent b62b28a commit 33aabec

File tree

37 files changed

+1087
-665
lines changed

37 files changed

+1087
-665
lines changed

README.md

Lines changed: 11 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@ module "vpc" {
2727
}
2828
```
2929

30+
> [!WARNING]
31+
> v6.x of the module still supports creating a VPC Flow Log within the root (VPC) module. However, this is deprecated behavior and will be removed in v7.0.0. Please use the [standalone flow log](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/modules/flow-log) module instead.
32+
3033
## External NAT Gateway IPs
3134

3235
By default this module will provision new Elastic IPs for the VPC's NAT Gateways.
@@ -116,24 +119,6 @@ If you need private subnets that should have no Internet routing (in the sense o
116119

117120
Since AWS Lambda functions allocate Elastic Network Interfaces in proportion to the traffic received ([read more](https://docs.aws.amazon.com/lambda/latest/dg/vpc.html)), it can be useful to allocate a large private subnet for such allocations, while keeping the traffic they generate entirely internal to the VPC.
118121

119-
You can add additional tags with `intra_subnet_tags` as with other subnet types.
120-
121-
## VPC Flow Log
122-
123-
VPC Flow Log allows to capture IP traffic for a specific network interface (ENI), subnet, or entire VPC. This module supports enabling or disabling VPC Flow Logs for entire VPC. If you need to have VPC Flow Logs for subnet or ENI, you have to manage it outside of this module with [aws_flow_log resource](https://www.terraform.io/docs/providers/aws/r/flow_log.html).
124-
125-
### VPC Flow Log Examples
126-
127-
By default `file_format` is `plain-text`. You can also specify `parquet` to have logs written in Apache Parquet format.
128-
129-
```
130-
flow_log_file_format = "parquet"
131-
```
132-
133-
### Permissions Boundary
134-
135-
If your organization requires a permissions boundary to be attached to the VPC Flow Log role, make sure that you specify an ARN of the permissions boundary policy as `vpc_flow_log_permissions_boundary` argument. Read more about required [IAM policy for publishing flow logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-iam).
136-
137122
## Conditional creation
138123

139124
Prior to Terraform 0.13, you were unable to specify `count` in a module block. If you wish to toggle the creation of the module's resources in an older (pre 0.13) version of Terraform, you can use the `create_vpc` argument.
@@ -231,19 +216,18 @@ module "vpc_cidr_from_ipam" {
231216

232217
## Examples
233218

234-
- [Complete VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/complete) with VPC Endpoints.
219+
- [Block Public Access](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/block-public-access)
220+
- [Complete VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/complete) w/ VPC Endpoints
221+
- [VPC w/ Flow Log](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/flow-log)
235222
- [VPC using IPAM](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/ipam)
236223
- [Dualstack IPv4/IPv6 VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/ipv6-dualstack)
237-
- [IPv6 only subnets/VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/ipv6-only)
224+
- [IPv6 only subnets VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/ipv6-only)
238225
- [Manage Default VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/manage-default-vpc)
239-
- [Network ACL](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/network-acls)
240-
- [VPC with Outpost](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/outpost)
241-
- [VPC with secondary CIDR blocks](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/secondary-cidr-blocks)
242-
- [VPC with unique route tables](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/separate-route-tables)
226+
- [VPC w/ Network ACL](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/network-acls)
227+
- [VPC w/ Outpost](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/outpost)
228+
- [VPC w/ secondary CIDR blocks](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/secondary-cidr-blocks)
229+
- [VPC w/ unique route tables](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/separate-route-tables)
243230
- [Simple VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/simple)
244-
- [VPC Flow Logs](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/vpc-flow-logs)
245-
- [VPC Block Public Access](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/block-public-access)
246-
- [Few tests and edge case examples](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/issues)
247231

248232
## Contributing
249233

examples/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Examples
2+
3+
The examples provided demonstrate different cluster configurations that users can create with the modules provided.
4+
5+
Please do not mistake the examples provided as "best practices". It is up to users to consult the AWS service documentation for best practices, usage recommendations, etc.

examples/complete/README.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -154,10 +154,6 @@ No inputs.
154154
| <a name="output_vpc_endpoints"></a> [vpc\_endpoints](#output\_vpc\_endpoints) | Array containing the full resource object and attributes for all endpoints created |
155155
| <a name="output_vpc_endpoints_security_group_arn"></a> [vpc\_endpoints\_security\_group\_arn](#output\_vpc\_endpoints\_security\_group\_arn) | Amazon Resource Name (ARN) of the security group |
156156
| <a name="output_vpc_endpoints_security_group_id"></a> [vpc\_endpoints\_security\_group\_id](#output\_vpc\_endpoints\_security\_group\_id) | ID of the security group |
157-
| <a name="output_vpc_flow_log_cloudwatch_iam_role_arn"></a> [vpc\_flow\_log\_cloudwatch\_iam\_role\_arn](#output\_vpc\_flow\_log\_cloudwatch\_iam\_role\_arn) | The ARN of the IAM role used when pushing logs to Cloudwatch log group |
158-
| <a name="output_vpc_flow_log_destination_arn"></a> [vpc\_flow\_log\_destination\_arn](#output\_vpc\_flow\_log\_destination\_arn) | The ARN of the destination for VPC Flow Logs |
159-
| <a name="output_vpc_flow_log_destination_type"></a> [vpc\_flow\_log\_destination\_type](#output\_vpc\_flow\_log\_destination\_type) | The type of the destination for VPC Flow Logs |
160-
| <a name="output_vpc_flow_log_id"></a> [vpc\_flow\_log\_id](#output\_vpc\_flow\_log\_id) | The ID of the Flow Log resource |
161157
| <a name="output_vpc_id"></a> [vpc\_id](#output\_vpc\_id) | The ID of the VPC |
162158
| <a name="output_vpc_instance_tenancy"></a> [vpc\_instance\_tenancy](#output\_vpc\_instance\_tenancy) | Tenancy of instances spin up within VPC |
163159
| <a name="output_vpc_ipv6_association_id"></a> [vpc\_ipv6\_association\_id](#output\_vpc\_ipv6\_association\_id) | The association ID for the IPv6 CIDR block |

examples/complete/main.tf

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -72,14 +72,6 @@ module "vpc" {
7272
dhcp_options_domain_name = "service.consul"
7373
dhcp_options_domain_name_servers = ["127.0.0.1", "10.10.0.2"]
7474

75-
# VPC Flow Logs (Cloudwatch log group and IAM role will be created)
76-
vpc_flow_log_iam_role_name = "vpc-complete-example-role"
77-
vpc_flow_log_iam_role_use_name_prefix = false
78-
enable_flow_log = true
79-
create_flow_log_cloudwatch_log_group = true
80-
create_flow_log_cloudwatch_iam_role = true
81-
flow_log_max_aggregation_interval = 60
82-
8375
tags = local.tags
8476
}
8577

examples/complete/outputs.tf

Lines changed: 0 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -513,27 +513,6 @@ output "elasticache_network_acl_arn" {
513513
value = module.vpc.elasticache_network_acl_arn
514514
}
515515

516-
# VPC flow log
517-
output "vpc_flow_log_id" {
518-
description = "The ID of the Flow Log resource"
519-
value = module.vpc.vpc_flow_log_id
520-
}
521-
522-
output "vpc_flow_log_destination_arn" {
523-
description = "The ARN of the destination for VPC Flow Logs"
524-
value = module.vpc.vpc_flow_log_destination_arn
525-
}
526-
527-
output "vpc_flow_log_destination_type" {
528-
description = "The type of the destination for VPC Flow Logs"
529-
value = module.vpc.vpc_flow_log_destination_type
530-
}
531-
532-
output "vpc_flow_log_cloudwatch_iam_role_arn" {
533-
description = "The ARN of the IAM role used when pushing logs to Cloudwatch log group"
534-
value = module.vpc.vpc_flow_log_cloudwatch_iam_role_arn
535-
}
536-
537516
# VPC endpoints
538517
output "vpc_endpoints" {
539518
description = "Array containing the full resource object and attributes for all endpoints created"

examples/flow-log/README.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# AWS Flow Logs to S3 and CloudWatch logs
2+
3+
Configuration in this directory creates a set of VPC resources with VPC Flow Logs enabled in different configurations:
4+
5+
- Flow log to CloudWatch logs using module created CloudWatch log group and IAM role
6+
- Flow log to CloudWatch logs using external CloudWatch log group and IAM role
7+
- Flow log to S3 bucket in text format
8+
- Flow log to S3 bucket in Parquet format
9+
10+
## Usage
11+
12+
To run this example you need to execute:
13+
14+
```bash
15+
$ terraform init
16+
$ terraform plan
17+
$ terraform apply
18+
```
19+
20+
Note that this example may create resources which can cost money (AWS Elastic IP, for example). Run `terraform destroy` when you don't need these resources.
21+
22+
<!-- BEGIN_TF_DOCS -->
23+
## Requirements
24+
25+
| Name | Version |
26+
|------|---------|
27+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.5.7 |
28+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 6.5 |
29+
30+
## Providers
31+
32+
| Name | Version |
33+
|------|---------|
34+
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 6.5 |
35+
36+
## Modules
37+
38+
| Name | Source | Version |
39+
|------|--------|---------|
40+
| <a name="module_disabled"></a> [disabled](#module\_disabled) | ../../modules/flow-log | n/a |
41+
| <a name="module_flow_log"></a> [flow\_log](#module\_flow\_log) | ../../modules/flow-log | n/a |
42+
| <a name="module_flow_log_cloudwatch_external"></a> [flow\_log\_cloudwatch\_external](#module\_flow\_log\_cloudwatch\_external) | ../../modules/flow-log | n/a |
43+
| <a name="module_flow_log_s3"></a> [flow\_log\_s3](#module\_flow\_log\_s3) | ../../modules/flow-log | n/a |
44+
| <a name="module_flow_log_s3_parquet"></a> [flow\_log\_s3\_parquet](#module\_flow\_log\_s3\_parquet) | ../../modules/flow-log | n/a |
45+
| <a name="module_s3_bucket"></a> [s3\_bucket](#module\_s3\_bucket) | terraform-aws-modules/s3-bucket/aws | ~> 5.0 |
46+
| <a name="module_vpc"></a> [vpc](#module\_vpc) | ../../ | n/a |
47+
48+
## Resources
49+
50+
| Name | Type |
51+
|------|------|
52+
| [aws_cloudwatch_log_group.flow_log](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource |
53+
| [aws_iam_role.flow_log_cloudwatch](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
54+
| [aws_iam_role_policy.flow_log_cloudwatch](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource |
55+
| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
56+
57+
## Inputs
58+
59+
No inputs.
60+
61+
## Outputs
62+
63+
| Name | Description |
64+
|------|-------------|
65+
| <a name="output_arn"></a> [arn](#output\_arn) | The ARN of the Flow Log |
66+
| <a name="output_cloudwatch_log_group_arn"></a> [cloudwatch\_log\_group\_arn](#output\_cloudwatch\_log\_group\_arn) | ARN of CloudWatch log group created |
67+
| <a name="output_cloudwatch_log_group_name"></a> [cloudwatch\_log\_group\_name](#output\_cloudwatch\_log\_group\_name) | Name of CloudWatch log group created |
68+
| <a name="output_iam_role_arn"></a> [iam\_role\_arn](#output\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
69+
| <a name="output_iam_role_name"></a> [iam\_role\_name](#output\_iam\_role\_name) | The name of the IAM role |
70+
| <a name="output_iam_role_unique_id"></a> [iam\_role\_unique\_id](#output\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
71+
| <a name="output_id"></a> [id](#output\_id) | The ID of the Flow Log |
72+
<!-- END_TF_DOCS -->

examples/flow-log/main.tf

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
provider "aws" {
2+
region = local.region
3+
}
4+
5+
data "aws_availability_zones" "available" {}
6+
7+
locals {
8+
name = "ex-${basename(path.cwd)}"
9+
region = "eu-west-1"
10+
11+
vpc_cidr = "10.0.0.0/16"
12+
azs = slice(data.aws_availability_zones.available.names, 0, 3)
13+
14+
tags = {
15+
Example = local.name
16+
GithubRepo = "terraform-aws-vpc"
17+
GithubOrg = "terraform-aws-modules"
18+
}
19+
}
20+
21+
################################################################################
22+
# Flow Log
23+
################################################################################
24+
25+
module "flow_log" {
26+
source = "../../modules/flow-log"
27+
28+
name = local.name
29+
vpc_id = module.vpc.vpc_id
30+
31+
tags = local.tags
32+
}
33+
34+
module "flow_log_cloudwatch_external" {
35+
source = "../../modules/flow-log"
36+
37+
name = "${local.name}-cloudwatch-external"
38+
vpc_id = module.vpc.vpc_id
39+
40+
create_cloudwatch_log_group = false
41+
log_destination = aws_cloudwatch_log_group.flow_log.arn
42+
43+
create_iam_role = false
44+
iam_role_arn = aws_iam_role.flow_log_cloudwatch.arn
45+
46+
tags = local.tags
47+
}
48+
49+
module "flow_log_s3" {
50+
source = "../../modules/flow-log"
51+
52+
name = "${local.name}-s3"
53+
vpc_id = module.vpc.vpc_id
54+
55+
log_destination_type = "s3"
56+
log_destination = module.s3_bucket.s3_bucket_arn
57+
58+
tags = local.tags
59+
}
60+
61+
module "flow_log_s3_parquet" {
62+
source = "../../modules/flow-log"
63+
64+
name = "${local.name}-s3-parquet"
65+
vpc_id = module.vpc.vpc_id
66+
67+
log_destination_type = "s3"
68+
log_destination = module.s3_bucket.s3_bucket_arn
69+
destination_options = {
70+
file_format = "parquet"
71+
hive_compatible_partitions = true
72+
per_hour_partition = true
73+
}
74+
75+
tags = local.tags
76+
}
77+
78+
module "disabled" {
79+
source = "../../modules/flow-log"
80+
81+
create = false
82+
}
83+
84+
################################################################################
85+
# Supporting Resources
86+
################################################################################
87+
88+
module "vpc" {
89+
source = "../../"
90+
91+
name = local.name
92+
cidr = local.vpc_cidr
93+
94+
azs = local.azs
95+
private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k)]
96+
public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 4)]
97+
98+
tags = local.tags
99+
}
100+
101+
module "s3_bucket" {
102+
source = "terraform-aws-modules/s3-bucket/aws"
103+
version = "~> 5.0"
104+
105+
bucket_prefix = "${local.name}-"
106+
force_destroy = true
107+
108+
# Policy works for flow logs as well
109+
attach_waf_log_delivery_policy = true
110+
111+
tags = local.tags
112+
}
113+
114+
resource "aws_cloudwatch_log_group" "flow_log" {
115+
name_prefix = "/aws/flow-log/vpc/${module.vpc.vpc_id}/${local.name}-external-"
116+
117+
retention_in_days = 7
118+
119+
tags = local.tags
120+
}
121+
122+
resource "aws_iam_role" "flow_log_cloudwatch" {
123+
name_prefix = "${local.name}-external-"
124+
125+
assume_role_policy = jsonencode({
126+
Version = "2012-10-17"
127+
Statement = [
128+
{
129+
Action = "sts:AssumeRole"
130+
Effect = "Allow"
131+
Sid = "VPCFlowLogsAssume"
132+
Principal = {
133+
Service = "vpc-flow-logs.amazonaws.com"
134+
}
135+
},
136+
]
137+
})
138+
139+
tags = local.tags
140+
}
141+
142+
resource "aws_iam_role_policy" "flow_log_cloudwatch" {
143+
name_prefix = "${local.name}-external-"
144+
role = aws_iam_role.flow_log_cloudwatch.id
145+
146+
policy = jsonencode({
147+
Version = "2012-10-17"
148+
Statement = [
149+
{
150+
Action = [
151+
"logs:CreateLogGroup",
152+
"logs:CreateLogStream",
153+
"logs:PutLogEvents",
154+
"logs:DescribeLogGroups",
155+
"logs:DescribeLogStreams",
156+
]
157+
Effect = "Allow"
158+
Resource = aws_cloudwatch_log_group.flow_log.arn
159+
},
160+
]
161+
})
162+
}

0 commit comments

Comments
 (0)