Update testing to CloudBuild and general linting #17

Author: averbuks

CONTRIBUTING.md

@@ -0,0 +1,99 @@
+# Contributing
+
+This document provides guidelines for contributing to the module.
+
+## Dependencies
+
+The following dependencies must be installed on the development system:
+
+- [Docker Engine][docker-engine]
+- [Google Cloud SDK][google-cloud-sdk]
+- [make]
+
+## Generating Documentation for Inputs and Outputs
+
+The Inputs and Outputs tables in the READMEs of the root module,
+submodules, and example modules are automatically generated based on
+the `variables` and `outputs` of the respective modules. These tables
+must be refreshed if the module interfaces are changed.
+
+### Execution
+
+Run `make generate_docs` to generate new Inputs and Outputs tables.
+
+## Integration Testing
+
+Integration tests are used to verify the behaviour of the root module,
+submodules, and example modules. Additions, changes, and fixes should
+be accompanied with tests.
+
+The integration tests are run using [Kitchen][kitchen],
+[Kitchen-Terraform][kitchen-terraform], and [InSpec][inspec]. These
+tools are packaged within a Docker image for convenience.
+
+The general strategy for these tests is to verify the behaviour of the
+[example modules](./examples/), thus ensuring that the root module,
+submodules, and example modules are all functionally correct.
+
+### Test Environment
+The easiest way to test the module is in an isolated test project. The setup for such a project is defined in [test/setup](./test/setup/) directory.
+
+To use this setup, you need a service account with Project Creator access on a folder. Export the Service Account credentials to your environment like so:
+
+```
+export SERVICE_ACCOUNT_JSON=$(< credentials.json)
+```
+
+You will also need to set a few environment variables:
+```
+export TF_VAR_org_id="your_org_id"
+export TF_VAR_folder_id="your_folder_id"
+export TF_VAR_billing_account="your_billing_account_id"
+```
+
+With these settings in place, you can prepare a test project using Docker:
+```
+make docker_test_prepare
+```
+
+### Noninteractive Execution
+
+Run `make docker_test_integration` to test all of the example modules
+noninteractively, using the prepared test project.
+
+### Interactive Execution
+
+1. Run `make docker_run` to start the testing Docker container in
+   interactive mode.
+
+1. Run `kitchen_do create <EXAMPLE_NAME>` to initialize the working
+   directory for an example module.
+
+1. Run `kitchen_do converge <EXAMPLE_NAME>` to apply the example module.
+
+1. Run `kitchen_do verify <EXAMPLE_NAME>` to test the example module.
+
+1. Run `kitchen_do destroy <EXAMPLE_NAME>` to destroy the example module
+   state.
+
+## Linting and Formatting
+
+Many of the files in the repository can be linted or formatted to
+maintain a standard of quality.
+
+### Execution
+
+Run `make docker_test_lint`.
+
+[docker-engine]: https://www.docker.com/products/docker-engine
+[flake8]: http://flake8.pycqa.org/en/latest/
+[gofmt]: https://golang.org/cmd/gofmt/
+[google-cloud-sdk]: https://cloud.google.com/sdk/install
+[hadolint]: https://github.com/hadolint/hadolint
+[inspec]: https://inspec.io/
+[kitchen-terraform]: https://github.com/newcontext-oss/kitchen-terraform
+[kitchen]: https://kitchen.ci/
+[make]: https://en.wikipedia.org/wiki/Make_(software)
+[shellcheck]: https://www.shellcheck.net/
+[terraform-docs]: https://github.com/segmentio/terraform-docs
+[terraform]: https://terraform.io/

LICENSE

@@ -199,4 +199,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

Makefile

@@ -82,4 +82,4 @@ docker_generate_docs:
 
 # Alias for backwards compatibility
 .PHONY: generate_docs
-generate_docs: docker_generate_docs
+generate_docs: docker_generate_docs

README.md

@@ -69,92 +69,60 @@ Then perform the following commands on the root folder:
 <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 
 ## Requirements
-### Terraform plugins
-- [Terraform](https://www.terraform.io/downloads.html) 0.12.x
-- [terraform-provider-google](https://github.com/terraform-providers/terraform-provider-google) plugin v2.1
+
+These sections describe requirements for using this module.
+
+### Software
+
+The following dependencies must be available:
+
+- [Terraform][terraform] v0.12
+- [Terraform Provider for GCP][terraform-provider-gcp] plugin v2.14
 
 ### App Engine
 Note that this module requires App Engine being configured in the specified project/region.
 This is because Google Cloud Scheduler is dependent on the project being configured with App Engine.
-Refer to the [Google Cloud Scheduler documentation](https://cloud.google.com/scheduler/docs/) for more
+Refer to the [Google Cloud Scheduler documentation][cloud-scheduler-documentation]
 information on the App Engine dependency.
 
 The recommended way to create projects with App Engine enabled is via the [Project Factory module](https://github.com/terraform-google-modules/terraform-google-project-factory).
 There is an example of how to create the project [within that module](https://github.com/terraform-google-modules/terraform-google-project-factory/tree/master/examples/app_engine)
 
-### Configure a Service Account
-In order to execute this module you must have a Service Account with the following roles.
-
-- roles/storage.admin
-- roles/pubsub.editor
-- roles/cloudscheduler.admin
-- roles/cloudfunctions.developer
-- roles/iam.serviceAccountUser
-
-
-### Enable API's
-In order to operate with the Service Account you must activate the following API on the project where the Service Account was created:
-
-- Cloud Scheduler API - cloudscheduler.googleapis.com
-- Cloud PubSub API - pubsub.googleapis.com
-- Cloud Functions API - cloudfunctions.googleapis.com
+### Service Account
 
-## Install
+A service account with the following roles must be used to provision
+the resources of this module:
 
-### Terraform
-Be sure you have the correct Terraform version (0.12.x), you can choose the binary here:
-- https://releases.hashicorp.com/terraform/
+- Storage Admin: `roles/storage.admin`
+- PubSub Editor: `roles/pubsub.editor`
+- Cloudscheduler Admin: `roles/cloudscheduler.admin`
+- Cloudfunctions Developer: `roles/cloudfunctions.developer`
+- IAM ServiceAccount User: `roles/iam.serviceAccountUser`
 
-## Testing and documentation generation
+The [Project Factory module][project-factory-module] and the
+[IAM module][iam-module] may be used in combination to provision a
+service account with the necessary roles applied.
 
-### Requirements
-- [docker](https://docker.com)
-- [terraform-docs](https://github.com/segmentio/terraform-docs/releases) 0.6.0
+### APIs
 
-### Integration test
-##### Terraform integration tests
-It is recommended to to run the integration tests via docker. To do so, run `make test_integration_docker`. In containers, this will
-- Perform `terraform init` command
-- Perform `terraform get` command
-- Perform `terraform validate` command
-- Perform `terraform apply -auto-approve` command and check that it has created the appropriate resources
-- Perform `terraform destroy -force` command and check that it has destroyed the appropriate resources
+A project with the following APIs enabled must be used to host the
+resources of this module:
 
-### Autogeneration of documentation from .tf files
-Run
-```
-make generate_docs
-```
-
-### Linting
-The makefile in this project will lint or sometimes just format any shell,
-Python, golang, Terraform, or Dockerfiles. The linters will only be run if
-the makefile finds files with the appropriate file extension.
-
-All of the linter checks are in the default make target, so you just have to
-run
+- Cloud Scheduler API: `cloudscheduler.googleapis.com`
+- Cloud PubSub API: `pubsub.googleapis.com`
+- Cloud Functions API: `cloudfunctions.googleapis.com`
+- App Engine Admin API: `appengine.googleapis.com`
 
-```
-make -s
-```
+The [Project Factory module][project-factory-module] can be used to
+provision a project with the necessary APIs enabled.
 
-The -s is for 'silent'. Successful output looks like this
+## Contributing
 
-```
-Running shellcheck
-Running flake8
-Running gofmt
-Running terraform validate
-Running hadolint on Dockerfiles
-Test passed - Verified all file Apache 2 headers
-```
+Refer to the [contribution guidelines](./CONTRIBUTING.md) for
+information on contributing to this module.
 
-The linters
-are as follows:
-* Shell - shellcheck. Can be found in homebrew
-* Python - flake8. Can be installed with 'pip install flake8'
-* Golang - gofmt. gofmt comes with the standard golang installation. golang
-is a compiled language so there is no standard linter.
-* Terraform - terraform has a built-in linter in the 'terraform validate'
-command.
-* Dockerfiles - hadolint. Can be found in homebrew
+[iam-module]: https://registry.terraform.io/modules/terraform-google-modules/iam/google
+[project-factory-module]: https://registry.terraform.io/modules/terraform-google-modules/project-factory/google
+[terraform-provider-gcp]: https://www.terraform.io/docs/providers/google/index.html
+[terraform]: https://www.terraform.io/downloads.html
+[cloud-scheduler-documentation]: https://cloud.google.com/scheduler/docs/

build/int.cloudbuild.yaml

@@ -0,0 +1,41 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+timeout: 3600s
+steps:
+- id: prepare
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && prepare_environment']
+  env:
+  - 'TF_VAR_org_id=$_ORG_ID'
+  - 'TF_VAR_folder_id=$_FOLDER_ID'
+  - 'TF_VAR_billing_account=$_BILLING_ACCOUNT'
+- id: create
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do create']
+- id: converge
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do converge']
+- id: verify
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do verify']
+- id: destroy
+  name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && kitchen_do destroy']
+tags:
+- 'ci'
+- 'integration'
+substitutions:
+  _DOCKER_IMAGE_DEVELOPER_TOOLS: 'cft/developer-tools'
+  _DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '0.1.0'

test/boilerplate/boilerplate.yaml.txt renamed to build/lint.cloudbuild.yaml

@@ -11,3 +11,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+steps:
+- name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
+  id: 'lint'
+  args: ['/usr/local/bin/test_lint.sh']
+tags:
+- 'ci'
+- 'lint'
+substitutions:
+  _DOCKER_IMAGE_DEVELOPER_TOOLS: 'cft/developer-tools'
+  _DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '0.1.0'

examples/logs-slack-alerts/README.md

@@ -4,7 +4,7 @@ Disclaimer (8/1/2019): Test Coverage has currently not been added to this exampl
 
 This logging slack alerts example module schedules a job to run hourly queries of any errors which have occurred in logs which have been ingested into BigQuery. If any errors are found, the errors are sent as alerts to a slack webhook.
 
-Running this module requires log exports into BigQuery in the specified project/region, which is not handled by this example. 
+Running this module requires log exports into BigQuery in the specified project/region, which is not handled by this example.
 A good example of exported logging in BigQuery can be found in [Stackdriver Logging](https://cloud.google.com/logging/docs/export/).
 
 ## Configure a Service Account

examples/logs-slack-alerts/function_source/main.py

@@ -1,18 +1,16 @@
-"""
- * Copyright 2019 Google LLC
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *      http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
-"""
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 import os
 import logging
@@ -54,12 +52,13 @@
   1,2
 """.format(**VARIABLES)
 
+
 def query_for_errors(pubsub_event, pubsub_context):
     """
     Cloud Function to query audit logs for errors
     and send alerts to Slack Webhook
     """
-    
+
     logging.info("Running: %s", QUERY)
     query_job = BQ_CLIENT.query(QUERY)
 
@@ -76,5 +75,6 @@ def query_for_errors(pubsub_event, pubsub_context):
                                 data=str({"text": text}))
             logging.info(req.text)
 
+
 if __name__ == "__main__":
     query_for_errors(None, None)

examples/logs-slack-alerts/function_source/requirements.txt

@@ -1,2 +1,2 @@
 google-cloud-bigquery==1.14.0
-requests==2.21.0
+requests==2.21.0