diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000000..30e551a122 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,60 @@ +--- +name: Report a Bug +about: Found an issue? Let us fix it. +--- + +Please ensure you do the following when reporting a bug: + +- [ ] Provide a concise description of what the bug is. +- [ ] Provide information about your environment. +- [ ] Provide clear steps to reproduce the bug. +- [ ] Attach applicable logs. Please do not attach screenshots showing logs unless you are unable to copy and paste the log data. +- [ ] Ensure any code / output examples are [properly formatted](https://docs.github.com/en/github/writing-on-github/basic-writing-and-formatting-syntax#quoting-code) for legibility. + +Note that some logs needed to troubleshoot may be found in the `/pgdata//pg_log` directory on your Postgres instance. + +An incomplete bug report can lead to delays in resolving the issue or the closing of a ticket, so please be as detailed as possible. + +If you are looking for [general support](https://access.crunchydata.com/documentation/postgres-operator/latest/support/), please view the [support](https://access.crunchydata.com/documentation/postgres-operator/latest/support/) page for where you can ask questions. + +Thanks for reporting the issue, we're looking forward to helping you! + +## Overview + +Add a concise description of what the bug is. + +## Environment + +Please provide the following details: + +- Platform: (`Kubernetes`, `OpenShift`, `Rancher`, `GKE`, `EKS`, `AKS` etc.) +- Platform Version: (e.g. `1.20.3`, `4.7.0`) +- PGO Image Tag: (e.g. `ubi8-5.x.y-0`) +- Postgres Version (e.g. `15`) +- Storage: (e.g. `hostpath`, `nfs`, or the name of your storage class) + +## Steps to Reproduce + +### REPRO + +Provide steps to get to the error condition: + +1. Run `...` +1. Do `...` +1. Try `...` + +### EXPECTED + +1. Provide the behavior that you expected. + +### ACTUAL + +1. Describe what actually happens + +## Logs + +Please provided appropriate log output or any configuration files that may help troubleshoot the issue. **DO NOT** include sensitive information, such as passwords. + +## Additional Information + +Please provide any additional information that may be helpful. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000000..4de2077c77 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,42 @@ +--- +name: Feature Request +about: Help us improve PGO! +--- + +Have an idea to improve PGO? We'd love to hear it! We're going to need some information from you to learn more about your feature requests. + +Please be sure you've done the following: + +- [ ] Provide a concise description of your feature request. +- [ ] Describe your use case. Detail the problem you are trying to solve. +- [ ] Describe how you envision that the feature would work. +- [ ] Provide general information about your current PGO environment. + +## Overview + +Provide a concise description of your feature request. + +## Use Case + +Describe your use case. Why do you want this feature? What problem will it solve? Why will it help you? Why will it make it easier to use PGO? + +## Desired Behavior + +Describe how the feature would work. How do you envision interfacing with it? + +## Environment + +Tell us about your environment: + +Please provide the following details: + +- Platform: (`Kubernetes`, `OpenShift`, `Rancher`, `GKE`, `EKS`, `AKS` etc.) +- Platform Version: (e.g. `1.20.3`, `4.7.0`) +- PGO Image Tag: (e.g. `ubi8-5.x.y-0`) +- Postgres Version (e.g. `15`) +- Storage: (e.g. `hostpath`, `nfs`, or the name of your storage class) +- Number of Postgres clusters: (`XYZ`) + +## Additional Information + +Please provide any additional information that may be helpful. diff --git a/.github/ISSUE_TEMPLATE/support---question-and-answer.md b/.github/ISSUE_TEMPLATE/support---question-and-answer.md new file mode 100644 index 0000000000..271caa9029 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/support---question-and-answer.md @@ -0,0 +1,35 @@ +--- +name: Support +about: "Learn how to interact with the PGO community" +--- + +If you believe you have found have found a bug, please open up [Bug Report](https://github.com/CrunchyData/postgres-operator/issues/new?template=bug_report.md) + +If you have a feature request, please open up a [Feature Request](https://github.com/CrunchyData/postgres-operator/issues/new?template=feature_request.md) + +You can find information about general PGO [support](https://access.crunchydata.com/documentation/postgres-operator/latest/support/) at: + +[https://access.crunchydata.com/documentation/postgres-operator/latest/support/](https://access.crunchydata.com/documentation/postgres-operator/latest/support/) + +## Questions + +For questions that are neither bugs nor feature requests, please be sure to + +- [ ] Provide information about your environment (see below for more information). +- [ ] Provide any steps or other relevant details related to your question. +- [ ] Attach logs, where applicable. Please do not attach screenshots showing logs unless you are unable to copy and paste the log data. +- [ ] Ensure any code / output examples are [properly formatted](https://docs.github.com/en/github/writing-on-github/basic-writing-and-formatting-syntax#quoting-code) for legibility. + +Besides Pod logs, logs may also be found in the `/pgdata/pg/log` directory on your Postgres instance. + +If you are looking for [general support](https://access.crunchydata.com/documentation/postgres-operator/latest/support/), please view the [support](https://access.crunchydata.com/documentation/postgres-operator/latest/support/) page for where you can ask questions. + +### Environment + +Please provide the following details: + +- Platform: (`Kubernetes`, `OpenShift`, `Rancher`, `GKE`, `EKS`, `AKS` etc.) +- Platform Version: (e.g. `1.20.3`, `4.7.0`) +- PGO Image Tag: (e.g. `ubi8-5.x.y-0`) +- Postgres Version (e.g. `15`) +- Storage: (e.g. `hostpath`, `nfs`, or the name of your storage class) diff --git a/.github/actions/k3d/action.yaml b/.github/actions/k3d/action.yaml new file mode 100644 index 0000000000..b6e6ed5c2b --- /dev/null +++ b/.github/actions/k3d/action.yaml @@ -0,0 +1,94 @@ +name: k3d +description: Start k3s using k3d +inputs: + k3d-tag: + default: latest + required: true + description: > + Git tag from https://github.com/k3d-io/k3d/releases or "latest" + k3s-channel: + default: latest + required: true + description: > + https://docs.k3s.io/upgrades/manual#release-channels + prefetch-images: + required: true + description: > + Each line is the name of an image to fetch onto all Kubernetes nodes + prefetch-timeout: + default: 3m + required: true + description: > + Amount of time to wait for images to be fetched + +outputs: + k3d-version: + value: ${{ steps.k3d.outputs.k3d }} + description: > + K3d version + kubernetes-version: + value: ${{ steps.k3s.outputs.server }} + description: > + Kubernetes server version, as reported by the Kubernetes API + pause-image: + value: ${{ steps.k3s.outputs.pause-image }} + description: > + Pause image for prefetch images DaemonSet + +runs: + using: composite + steps: + - id: k3d + name: Install k3d + shell: bash + env: + K3D_TAG: ${{ inputs.k3d-tag }} + run: | + curl --fail --silent https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | + TAG="${K3D_TAG#latest}" bash + k3d version | awk '{ print "${tolower($1)}=${$3}" >> $GITHUB_OUTPUT }' + + - id: k3s + name: Start k3s + shell: bash + run: | + k3d cluster create --image '+${{ inputs.k3s-channel }}' --no-lb --timeout=2m --wait + kubectl version | awk '{ print "${tolower($1)}=${$3}" >> $GITHUB_OUTPUT }' + + PAUSE_IMAGE=$(docker exec $(k3d node list --output json | jq --raw-output 'first.name') \ + k3s agent --help | awk '$1 == "--pause-image" { + match($0, /default: "[^"]*"/); + print substr($0, RSTART+10, RLENGTH-11) + }') + echo "pause-image=${PAUSE_IMAGE}" >> $GITHUB_OUTPUT + + - name: Prefetch container images + shell: bash + env: + INPUT_IMAGES: ${{ inputs.prefetch-images }} + INPUT_TIMEOUT: ${{ inputs.prefetch-timeout }} + run: | + jq <<< "$INPUT_IMAGES" --raw-input 'select(. != "")' | + jq --slurp \ + --arg pause '${{ steps.k3s.outputs.pause-image }}' \ + --argjson labels '{"name":"image-prefetch"}' \ + --argjson name '"image-prefetch"' \ + '{ + apiVersion: "apps/v1", kind: "DaemonSet", + metadata: { name: $name, labels: $labels }, + spec: { + selector: { matchLabels: $labels }, + template: { + metadata: { labels: $labels }, + spec: { + initContainers: to_entries | map({ + name: "c\(.key)", image: .value, command: ["true"], + }), + containers: [{ name: "pause", image: $pause }] + } + } + } + }' | + kubectl create --filename=- + kubectl rollout status daemonset.apps/image-prefetch --timeout "$INPUT_TIMEOUT" || + kubectl describe daemonset.apps/image-prefetch diff --git a/.github/actions/trivy/action.yaml b/.github/actions/trivy/action.yaml new file mode 100644 index 0000000000..bcc67421cb --- /dev/null +++ b/.github/actions/trivy/action.yaml @@ -0,0 +1,138 @@ +# Copyright 2024 - 2025 Crunchy Data Solutions, Inc. +# +# SPDX-License-Identifier: Apache-2.0 +# +# schema documentation: https://docs.github.com/actions/sharing-automations/creating-actions/metadata-syntax-for-github-actions +# yaml-language-server: $schema=https://json.schemastore.org/github-action.json + +name: Trivy +description: Scan this project using Trivy + +# The Trivy team maintains an action, but it has trouble caching its vulnerability data: +# https://github.com/aquasecurity/trivy-action/issues/389 +# +# 1. It caches vulnerability data once per calendar day, despite Trivy wanting +# to download more frequently than that. +# 2. When it fails to download the data, it fails the workflow *and* caches +# the incomplete data. +# 3. When (1) and (2) coincide, every following run that day *must* update the data, +# producing more opportunities for (2) and more failed workflows. +# +# The action below uses any recent cache matching `cache-prefix` and calculates a cache key +# derived from the data Trivy downloads. An older database is better than no scans at all. +# When a run successfully updates the data, that data is cached and available to other runs. + +inputs: + cache: + default: restore,success,use + description: >- + What Trivy data to cache; one or more of restore, save, success, or use. + The value "use" instructs Trivy to read and write to its cache. + The value "restore" loads the Trivy cache from GitHub. + The value "success" saves the Trivy cache to GitHub when Trivy succeeds. + The value "save" saves the Trivy cache to GitHub regardless of Trivy. + + database: + default: update + description: >- + How Trivy should handle its data; one of update or skip. + The value "skip" fetches no Trivy data at all. + + setup: + default: v0.65.0,cache + description: >- + How to install Trivy; one or more of version, none, or cache. + The value "none" does not install Trivy at all. + + cache-directory: + default: ${{ github.workspace }}/.cache/trivy + description: >- + Directory where Trivy should store its data + + cache-prefix: + default: cache-trivy + description: >- + Name (key) where Trivy data should be stored in the GitHub cache + + scan-target: + default: . + description: >- + What Trivy should scan + + scan-type: + default: repository + description: >- + How Trivy should interpret scan-target; one of filesystem, image, repository, or sbom. + +runs: + using: composite + steps: + # Parse list inputs as separated by commas and spaces. + # Select the maximum version-looking string from `inputs.setup`. + - id: parsed + shell: bash + run: | + # Validate inputs + ( + <<< '${{ inputs.cache }}' jq -rRsS '"cache=\(split("[,\\s]+"; "") - [""])"' + <<< '${{ inputs.setup }}' jq -rRsS ' + "setup=\(split("[,\\s]+"; "") - [""])", + "version=\(split("[,\\s]+"; "") | max_by(split("[v.]"; "") | map(tonumber?)))" + ' + ) | tee --append "${GITHUB_OUTPUT}" + + # Install Trivy as requested. + # NOTE: `setup-trivy` can download a "latest" version but cannot cache it. + - if: ${{ ! contains(fromJSON(steps.parsed.outputs.setup), 'none') }} + uses: aquasecurity/setup-trivy@v0.2.4 + with: + cache: ${{ contains(fromJSON(steps.parsed.outputs.setup), 'cache') }} + version: ${{ steps.parsed.outputs.version }} + + # Restore a recent cache beginning with the prefix. + - id: restore + if: ${{ contains(fromJSON(steps.parsed.outputs.cache), 'restore') }} + uses: actions/cache/restore@v4 + with: + path: ${{ inputs.cache-directory }} + key: ${{ inputs.cache-prefix }}- + + - id: trivy + shell: bash + env: + TRIVY_CACHE_DIR: >- + ${{ contains(fromJSON(steps.parsed.outputs.cache), 'use') && inputs.cache-directory || '' }} + TRIVY_SKIP_CHECK_UPDATE: ${{ inputs.database == 'skip' }} + TRIVY_SKIP_DB_UPDATE: ${{ inputs.database == 'skip' }} + TRIVY_SKIP_JAVA_DB_UPDATE: ${{ inputs.database == 'skip' }} + TRIVY_SKIP_VEX_REPO_UPDATE: ${{ inputs.database == 'skip' }} + run: | + # Run Trivy + trivy '${{ inputs.scan-type }}' '${{ inputs.scan-target }}' || result=$? + + checksum=$([[ -z "${TRIVY_CACHE_DIR}" ]] || cat "${TRIVY_CACHE_DIR}/"*/metadata.json | sha256sum) + echo 'cache-key=${{ inputs.cache-prefix }}-'"${checksum%% *}" >> "${GITHUB_OUTPUT}" + + exit "${result-0}" + + # Save updated data to the cache when requested. + - if: >- + ${{ + steps.restore.outcome == 'success' && + steps.restore.outputs.cache-matched-key == steps.trivy.outputs.cache-key + }} + shell: bash + run: | + # Cache hit on ${{ steps.restore.outputs.cache-matched-key }} + - if: >- + ${{ + steps.restore.outputs.cache-matched-key != steps.trivy.outputs.cache-key && + ( + (contains(fromJSON(steps.parsed.outputs.cache), 'save') && !cancelled()) || + (contains(fromJSON(steps.parsed.outputs.cache), 'success') && success()) + ) + }} + uses: actions/cache/save@v4 + with: + key: ${{ steps.trivy.outputs.cache-key }} + path: ${{ inputs.cache-directory }} diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000000..8a16fc8d6f --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,63 @@ +# Copyright 2024 - 2025 Crunchy Data Solutions, Inc. +# +# SPDX-License-Identifier: Apache-2.0 +# +# documentation: https://docs.github.com/code-security/dependabot/dependabot-version-updates +# schema documentation: https://docs.github.com/code-security/dependabot/working-with-dependabot/dependabot-options-reference +# yaml-language-server: $schema=https://json.schemastore.org/dependabot-2.0.json +# +# Dependabot allows only one schedule per package-ecosystem, directory, and target-branch. +# Configurations that lack a "target-branch" field also affect security updates. +# +# There is a hack to have *two* schedules: https://github.com/dependabot/dependabot-core/issues/1778#issuecomment-1988140219 +--- +version: 2 + +updates: + - package-ecosystem: github-actions + directories: + # "/" is a special case that includes ".github/workflows/*" + - '/' + - '.github/actions/*' + registries: '*' + schedule: + interval: weekly + day: tuesday + labels: + - dependencies + groups: + # Group security updates into one pull request + action-vulnerabilities: + applies-to: security-updates + patterns: ['*'] + + # Group version updates into one pull request + github-actions: + applies-to: version-updates + patterns: ['*'] + + - package-ecosystem: gomod + directory: '/' + registries: '*' + schedule: + interval: weekly + day: wednesday + labels: + - dependencies + groups: + # Group security updates into one pull request + go-vulnerabilities: + applies-to: security-updates + patterns: ['*'] + + # Group Kubernetes and OpenTelemetry version updates into separate pull requests + kubernetes: + patterns: ['k8s.io/*', 'sigs.k8s.io/*'] + opentelemetry: + patterns: ['go.opentelemetry.io/*'] + go-dependencies: + patterns: ['*'] + exclude-patterns: + - 'k8s.io/*' + - 'sigs.k8s.io/*' + - 'go.opentelemetry.io/*' diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000..b03369bf09 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,30 @@ +**Checklist:** + + + - [ ] Have you added an explanation of what your changes do and why you'd like them to be included? + - [ ] Have you updated or added documentation for the change, as applicable? + - [ ] Have you tested your changes on all related environments with successful results, as applicable? + - [ ] Have you added automated tests? + + + +**Type of Changes:** + + + - [ ] New feature + - [ ] Bug fix + - [ ] Documentation + - [ ] Testing enhancement + - [ ] Other + + +**What is the current behavior (link to any open issues here)?** + + + +**What is the new behavior (if this is a feature change)?** +- [ ] Breaking change (fix or feature that would cause existing functionality to change) + + + +**Other Information**: diff --git a/.github/workflows/codeql-analysis.yaml b/.github/workflows/codeql-analysis.yaml new file mode 100644 index 0000000000..16a6d85e12 --- /dev/null +++ b/.github/workflows/codeql-analysis.yaml @@ -0,0 +1,37 @@ +# https://codeql.github.com +name: CodeQL + +on: + pull_request: + branches: + - REL_5_8 + push: + branches: + - REL_5_8 + schedule: + - cron: '10 18 * * 2' + +jobs: + analyze: + if: ${{ github.repository == 'CrunchyData/postgres-operator' }} + permissions: + actions: read + contents: read + security-events: write + + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + + - name: Initialize CodeQL + uses: github/codeql-action/init@v4 + with: { languages: go } + + - name: Autobuild + # This action calls `make` which runs our "help" target. + uses: github/codeql-action/autobuild@v4 + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v4 diff --git a/.github/workflows/govulncheck.yaml b/.github/workflows/govulncheck.yaml new file mode 100644 index 0000000000..6721104401 --- /dev/null +++ b/.github/workflows/govulncheck.yaml @@ -0,0 +1,50 @@ +# https://go.dev/security/vuln +name: govulncheck + +on: + pull_request: + branches: + - REL_5_8 + push: + branches: + - REL_5_8 + +env: + # Use the Go toolchain installed by setup-go + # https://github.com/actions/setup-go/issues/457 + # + # TODO(govulncheck): Remove when "golang/govulncheck-action" uses "actions/setup-go" v6 or newer + GOTOOLCHAIN: local + +jobs: + vulnerabilities: + if: ${{ github.repository == 'CrunchyData/postgres-operator' }} + permissions: + security-events: write + + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + + # Install Go and produce a SARIF report. This fails only when the tool is + # unable to scan. + - name: Prepare report + uses: golang/govulncheck-action@v1 + with: + output-file: 'govulncheck-results.sarif' + output-format: 'sarif' + repo-checkout: false + + # Submit the SARIF report to GitHub code scanning. Pull request checks + # succeed or fail according to branch protection rules. + # - https://docs.github.com/en/code-security/code-scanning + - name: Upload results to GitHub + uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: 'govulncheck-results.sarif' + + # Print any detected vulnerabilities to the workflow log. This step fails + # when the tool detects a vulnerability in code that is called. + # - https://go.dev/blog/govulncheck + - name: Log results + run: govulncheck --format text --show verbose ./... diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml new file mode 100644 index 0000000000..230e2a7fae --- /dev/null +++ b/.github/workflows/lint.yaml @@ -0,0 +1,32 @@ +name: Linters + +on: + pull_request: + branches: + - REL_5_8 + +jobs: + golangci-lint: + runs-on: ubuntu-24.04 + permissions: + contents: read + checks: write + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + + - uses: golangci/golangci-lint-action@v9 + with: + version: latest + args: --timeout=5m + + # Count issues reported by disabled linters. The command always + # exits zero to ensure it does not fail the pull request check. + - name: Count non-blocking issues + run: | + golangci-lint run --config .golangci.next.yaml --show-stats >> "${GITHUB_STEP_SUMMARY}" \ + --max-issues-per-linter=0 \ + --max-same-issues=0 \ + --uniq-by-line=0 \ + --output.text.path=/dev/null ||: diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml new file mode 100644 index 0000000000..3e0629dd5f --- /dev/null +++ b/.github/workflows/test.yaml @@ -0,0 +1,199 @@ +name: Tests + +on: + pull_request: + branches: + - REL_5_8 + push: + branches: + - REL_5_8 + +jobs: + go-test: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + + - name: Ensure go.mod is tidy + run: go mod tidy --diff + - name: Ensure generated files are committed + run: make check-generate + - run: make check + + kubernetes-api: + runs-on: ubuntu-24.04 + needs: [go-test] + strategy: + fail-fast: false + matrix: + kubernetes: ['default'] + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + + - run: go mod download + - run: ENVTEST_K8S_VERSION="${KUBERNETES#default}" make check-envtest + env: + KUBERNETES: "${{ matrix.kubernetes }}" + GO_TEST: go test --coverprofile 'envtest.coverage' --coverpkg ./internal/... + + # Upload coverage to GitHub + - run: gzip envtest.coverage + - uses: actions/upload-artifact@v5 + with: + name: "~coverage~kubernetes-api=${{ matrix.kubernetes }}" + path: envtest.coverage.gz + retention-days: 1 + + kubernetes-k3d: + if: "${{ github.repository == 'CrunchyData/postgres-operator' }}" + runs-on: ubuntu-24.04 + needs: [go-test] + strategy: + fail-fast: false + matrix: + kubernetes: [v1.30, v1.34] + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + + - name: Start k3s + uses: ./.github/actions/k3d + with: + k3s-channel: "${{ matrix.kubernetes }}" + prefetch-images: | + registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi9-2.56.0-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-pgbouncer:ubi9-1.24-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi9-16.11-2547 + + - run: make createnamespaces check-envtest-existing + env: + PGO_TEST_TIMEOUT_SCALE: 1.2 + GO_TEST: go test --coverprofile 'envtest-existing.coverage' --coverpkg ./internal/... + + # Upload coverage to GitHub + - run: gzip envtest-existing.coverage + - uses: actions/upload-artifact@v5 + with: + name: "~coverage~kubernetes-k3d=${{ matrix.kubernetes }}" + path: envtest-existing.coverage.gz + retention-days: 1 + + e2e-k3d-kuttl: + runs-on: ubuntu-24.04 + needs: [go-test] + strategy: + fail-fast: false + matrix: + kubernetes: [v1.30, v1.34] + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + + - name: Start k3s + uses: ./.github/actions/k3d + with: + k3s-channel: "${{ matrix.kubernetes }}" + prefetch-timeout: 5m + prefetch-images: | + registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi9-2.56.0-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi9-17.7-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi9-16.11-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-pgbouncer:ubi9-1.24-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:ubi9-0.18.1-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-upgrade:ubi9-18.1-2547 + registry.developers.crunchydata.com/crunchydata/crunchy-pgadmin4:ubi9-9.8-2547 + + - name: Get pgMonitor files. + run: make get-pgmonitor + env: + PGMONITOR_DIR: "${{ github.workspace }}/hack/tools/pgmonitor" + QUERIES_CONFIG_DIR: "${{ github.workspace }}/hack/tools/queries" + + - run: go mod download + - name: Build executable + run: PGO_VERSION='${{ github.sha }}' make build-postgres-operator + + # Start a Docker container with the working directory mounted. + - name: Start PGO + run: | + kubectl apply --server-side -k ./config/namespace + kubectl apply --server-side -k ./config/dev + hack/create-kubeconfig.sh postgres-operator pgo + docker run --detach --network host --read-only \ + --volume "$(pwd):/mnt" --workdir '/mnt' --env 'PATH=/mnt/bin' \ + --env 'QUERIES_CONFIG_DIR=/mnt/hack/tools/queries' \ + --env 'KUBECONFIG=hack/.kube/postgres-operator/pgo' \ + --env 'RELATED_IMAGE_PGBACKREST=registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi9-2.56.0-2547' \ + --env 'RELATED_IMAGE_PGBOUNCER=registry.developers.crunchydata.com/crunchydata/crunchy-pgbouncer:ubi9-1.24-2547' \ + --env 'RELATED_IMAGE_PGEXPORTER=registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:ubi9-0.18.1-2547' \ + --env 'RELATED_IMAGE_PGUPGRADE=registry.developers.crunchydata.com/crunchydata/crunchy-upgrade:ubi9-18.1-2547' \ + --env 'RELATED_IMAGE_POSTGRES_16=registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi9-16.11-2547' \ + --env 'RELATED_IMAGE_POSTGRES_17=registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi9-17.7-2547' \ + --env 'RELATED_IMAGE_STANDALONE_PGADMIN=registry.developers.crunchydata.com/crunchydata/crunchy-pgadmin4:ubi9-9.8-2547' \ + --env 'RELATED_IMAGE_COLLECTOR=registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi9-5.8.5-0' \ + --env 'PGO_FEATURE_GATES=TablespaceVolumes=true,OpenTelemetryLogs=true,OpenTelemetryMetrics=true' \ + --name 'postgres-operator' ubuntu \ + postgres-operator + + - run: make generate-kuttl + env: + KUTTL_PG_UPGRADE_FROM_VERSION: '16' + KUTTL_PG_UPGRADE_TO_VERSION: '17' + KUTTL_PG_VERSION: '16' + KUTTL_POSTGIS_VERSION: '3.4' + KUTTL_PSQL_IMAGE: 'registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi9-16.11-2547' + - run: | + make check-kuttl && exit + failed=$? + echo '::group::PGO logs'; docker logs 'postgres-operator'; echo '::endgroup::' + exit $failed + + - name: Stop PGO + run: docker stop 'postgres-operator' || true + + coverage-report: + if: ${{ success() || contains(needs.*.result, 'success') }} + runs-on: ubuntu-24.04 + needs: + - kubernetes-api + - kubernetes-k3d + - e2e-k3d-kuttl + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-go@v6 + with: { go-version: stable } + - uses: actions/download-artifact@v6 + with: { path: download } + + # Combine the coverage profiles by taking the mode line from any one file + # and the data from all files. Write a list of functions with less than + # 100% coverage to the job summary, and upload a complete HTML report. + - name: Generate report + run: | + gunzip --keep download/*/*.gz + ( sed -e '1q' download/*/*.coverage + tail -qn +2 download/*/*.coverage ) > total.coverage + go tool cover --func total.coverage -o total-coverage.txt + go tool cover --html total.coverage -o total-coverage.html + + awk < total-coverage.txt ' + END { print "
Total Coverage: " $3 " " $2 "" } + ' >> "${GITHUB_STEP_SUMMARY}" + + sed < total-coverage.txt -e '/100.0%/d' -e "s,$(go list -m)/,," | column -t | awk ' + NR == 1 { print "\n\n```" } { print } END { if (NR > 0) print "```\n\n"; print "
" } + ' >> "${GITHUB_STEP_SUMMARY}" + + # Upload coverage to GitHub + - run: gzip total-coverage.html + - uses: actions/upload-artifact@v5 + with: + name: coverage-report=html + path: total-coverage.html.gz + retention-days: 15 diff --git a/.github/workflows/trivy.yaml b/.github/workflows/trivy.yaml new file mode 100644 index 0000000000..e3fd63b2ee --- /dev/null +++ b/.github/workflows/trivy.yaml @@ -0,0 +1,127 @@ +# https://aquasecurity.github.io/trivy +name: Trivy + +on: + pull_request: + branches: + - REL_5_8 + push: + branches: + - REL_5_8 + +env: + # Use the committed Trivy configuration files. + TRIVY_IGNOREFILE: .trivyignore.yaml + TRIVY_SECRET_CONFIG: trivy-secret.yaml + +jobs: + cache: + # Run only one of these jobs at a time across the entire project. + concurrency: { group: trivy-cache } + # Do not fail this workflow when this job fails. + continue-on-error: true + + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + - name: Download Trivy + uses: ./.github/actions/trivy + env: + TRIVY_DEBUG: true + TRIVY_DOWNLOAD_DB_ONLY: true + TRIVY_NO_PROGRESS: true + TRIVY_SCANNERS: license,secret,vuln + with: + cache: restore,success,use + database: update + + licenses: + # Run this job after the cache job regardless of its success or failure. + needs: [cache] + if: >- + ${{ !cancelled() }} + + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + + # Trivy needs a populated Go module cache to detect Go module licenses. + - uses: actions/setup-go@v6 + with: { go-version: stable } + - run: go mod download + + # Report success only when detected licenses are listed in [.trivyignore.yaml]. + - name: Scan licenses + uses: ./.github/actions/trivy + env: + TRIVY_DEBUG: true + TRIVY_EXIT_CODE: 1 + TRIVY_SCANNERS: license + with: + cache: restore,use + database: skip + + secrets: + # Run this job after the cache job regardless of its success or failure. + needs: [cache] + if: >- + ${{ !cancelled() }} + + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + + # Report success only when detected secrets are listed in [.trivyignore.yaml]. + - name: Scan secrets + uses: ./.github/actions/trivy + env: + TRIVY_EXIT_CODE: 1 + TRIVY_SCANNERS: secret + with: + cache: restore,use + database: skip + + vulnerabilities: + # Run this job after the cache job regardless of its success or failure. + needs: [cache] + if: >- + ${{ github.repository == 'CrunchyData/postgres-operator' && !cancelled() }} + permissions: + security-events: write + + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v5 + + # Print any detected secrets or vulnerabilities to the workflow log for + # human consumption. This step fails only when Trivy is unable to scan. + # A later step uploads results to GitHub as a pull request check. + - name: Log detected vulnerabilities + uses: ./.github/actions/trivy + env: + TRIVY_SCANNERS: secret,vuln + with: + cache: restore,use + database: skip + + # Produce a SARIF report of actionable results. This step fails only when + # Trivy is unable to scan. + - name: Report actionable vulnerabilities + uses: ./.github/actions/trivy + env: + TRIVY_IGNORE_UNFIXED: true + TRIVY_FORMAT: 'sarif' + TRIVY_OUTPUT: 'trivy-results.sarif' + TRIVY_SCANNERS: secret,vuln + with: + cache: use + database: skip + setup: none + + # Submit the SARIF report to GitHub code scanning. Pull requests checks + # succeed or fail according to branch protection rules. + # - https://docs.github.com/en/code-security/code-scanning + - name: Upload results to GitHub + uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: 'trivy-results.sarif' diff --git a/.golangci.next.yaml b/.golangci.next.yaml index 6b76d7b1d2..2aa389e841 100644 --- a/.golangci.next.yaml +++ b/.golangci.next.yaml @@ -4,39 +4,95 @@ # Rules that should be enforced immediately belong in [.golangci.yaml]. # # Both files are used by [.github/workflows/lint.yaml]. +version: "2" +# https://golangci-lint.run/usage/linters linters: - disable-all: true - enable: - - contextcheck - - err113 - - gocritic - - godot - - godox - - gofumpt - - gosec # exclude-use-default - - nilnil + default: all + disable: + - asasalint + - asciicheck + - bidichk + - bodyclose + - copyloopvar + - depguard + - dupword + - durationcheck + - errchkjson + - errname + - errorlint + - exhaustive + - exptostd + - fatcontext + - forbidigo + - ginkgolinter + - gocheckcompilerdirectives + - gochecksumtype + - goheader + - gomoddirectives + - gomodguard + - goprintffuncname + - gosmopolitan + - grouper + - iface + - importas + - interfacebloat + - intrange + - loggercheck + - makezero + - mirror + - misspell + - musttag + - nilerr + - nilnesserr + - noctx - nolintlint - - predeclared - - revive - - staticcheck # exclude-use-default - - tenv - - thelper - - tparallel + - nosprintfhostport + - prealloc + - promlinter + - protogetter + - reassign + - recvcheck + - rowserrcheck + - sloglint + - spancheck + - sqlclosecheck + - tagalign + - testifylint + - unconvert + - unparam + - usestdlibvars + - usetesting - wastedassign + - wsl + - zerologlint -issues: - exclude-rules: - # We call external linters when they are installed: Flake8, ShellCheck, etc. - - linters: [gosec] - path: '_test[.]go$' - text: 'G204: Subprocess launched with variable' + settings: + thelper: + # https://github.com/kulti/thelper/issues/27 + tb: { begin: true, first: true } + test: { begin: true, first: true, name: true } + + exclusions: + warn-unused: true + # Ignore built-in exclusions + presets: [] + rules: + # We call external linters when they are installed: Flake8, ShellCheck, etc. + - linters: [gosec] + path: '_test[.]go$' + text: 'G204: Subprocess launched with variable' - # https://github.com/golangci/golangci-lint/issues/2239 - exclude-use-default: false +# https://golangci-lint.run/usage/formatters +formatters: + enable: + - gofumpt + +issues: + # Fix only when requested + fix: false -linters-settings: - thelper: - # https://github.com/kulti/thelper/issues/27 - tb: { begin: true, first: true } - test: { begin: true, first: true, name: true } + # Show all issues at once + max-issues-per-linter: 0 + max-same-issues: 0 + uniq-by-line: false diff --git a/.golangci.yaml b/.golangci.yaml index da19e26976..55a54549f6 100644 --- a/.golangci.yaml +++ b/.golangci.yaml @@ -1,121 +1,201 @@ # https://golangci-lint.run/usage/configuration/ +version: "2" +# https://golangci-lint.run/usage/linters linters: - disable: - - contextcheck - - gci - - gofumpt + default: standard enable: + - asasalint + - asciicheck + - bidichk + - bodyclose + - copyloopvar - depguard + - dupword + - durationcheck + - errchkjson + - errname + - errorlint + - exhaustive + - exptostd + - fatcontext + - forbidigo + - ginkgolinter + - gocheckcompilerdirectives + - gochecksumtype - goheader + - gomoddirectives - gomodguard - - gosimple + - goprintffuncname + - gosec + - gosmopolitan + - grouper + - iface - importas + - interfacebloat + - intrange + - loggercheck + - makezero + - mirror - misspell + - musttag + - nilerr + - nilnesserr + - noctx + - nolintlint + - nosprintfhostport + - prealloc + - promlinter + - protogetter + - reassign + - recvcheck + - rowserrcheck + - sloglint + - spancheck + - sqlclosecheck + - tagalign + - testifylint - unconvert - presets: - - bugs - - format - - unused - -linters-settings: - depguard: + - unparam + - usestdlibvars + - usetesting + - wastedassign + - zerologlint + + settings: + dupword: + ignore: + # We might see duplicate instances of 'fi' if we end two bash 'if' statements + - fi + + depguard: + rules: + everything: + files: ['$all'] + list-mode: lax + allow: + - go.opentelemetry.io/otel/semconv/v1.27.0 + deny: + - pkg: go.opentelemetry.io/otel/semconv + desc: Use "go.opentelemetry.io/otel/semconv/v1.27.0" instead. + - pkg: io/ioutil + desc: Use the "io" and "os" packages instead. See https://go.dev/doc/go1.16#ioutil + - pkg: math/rand$ + desc: Use the "math/rand/v2" package instead. See https://go.dev/doc/go1.22#math_rand_v2 + not-tests: + files: ['!$test','!**/internal/testing/**'] + list-mode: lax + deny: + - pkg: net/http/httptest + desc: Should be used only in tests. + - pkg: testing/* + desc: The "testing" packages should be used only in tests. + - pkg: github.com/crunchydata/postgres-operator/internal/crd/* + desc: The "internal/crd" packages should be used only in tests. + - pkg: github.com/crunchydata/postgres-operator/internal/testing/* + desc: The "internal/testing" packages should be used only in tests. + - pkg: k8s.io/client-go/discovery + desc: Use the "internal/kubernetes" package instead. + tests: + files: ['$test'] + list-mode: lax + deny: + - pkg: github.com/pkg/errors + desc: Use the "errors" package unless you are interacting with stack traces. + + errchkjson: + check-error-free-encoding: true + + goheader: + template: |- + Copyright {{ DATES }} Crunchy Data Solutions, Inc. + + SPDX-License-Identifier: Apache-2.0 + values: + regexp: + DATES: ((201[7-9]|202[0-4]) - 2025|2025) + + gomodguard: + blocked: + modules: + - go.yaml.in/yaml/v2: { recommendations: [sigs.k8s.io/yaml] } + - go.yaml.in/yaml/v3: { recommendations: [sigs.k8s.io/yaml] } + - gopkg.in/yaml.v2: { recommendations: [sigs.k8s.io/yaml] } + - gopkg.in/yaml.v3: { recommendations: [sigs.k8s.io/yaml] } + - gotest.tools: { recommendations: [gotest.tools/v3] } + - k8s.io/kubernetes: + reason: k8s.io/kubernetes is for building kubelet, kubeadm, etc. + + importas: + no-unaliased: true + alias: + - pkg: k8s.io/api/(\w+)/(v[\w\w]+) + alias: $1$2 + - pkg: k8s.io/apimachinery/pkg/apis/(\w+)/(v[\w\d]+) + alias: $1$2 + - pkg: k8s.io/apimachinery/pkg/api/errors + alias: apierrors + + spancheck: + checks: [end, record-error] + extra-start-span-signatures: + - github.com/crunchydata/postgres-operator/internal/tracing.Start:opentelemetry + ignore-check-signatures: + - tracing.Escape + + exclusions: + warn-unused: true + presets: + - common-false-positives + - legacy + - std-error-handling rules: - everything: - list-mode: lax - allow: - - go.opentelemetry.io/otel/semconv/v1.27.0 - deny: - - pkg: go.opentelemetry.io/otel/semconv - desc: Use "go.opentelemetry.io/otel/semconv/v1.27.0" instead. - - - pkg: io/ioutil - desc: > - Use the "io" and "os" packages instead. - See https://go.dev/doc/go1.16#ioutil - - not-tests: - files: ['!$test'] - deny: - - pkg: net/http/httptest - desc: Should be used only in tests. - - - pkg: testing/* - desc: The "testing" packages should be used only in tests. - - - pkg: github.com/crunchydata/postgres-operator/internal/testing/* - desc: The "internal/testing" packages should be used only in tests. - - - pkg: k8s.io/client-go/discovery - desc: Use the "internal/kubernetes" package instead. - - tests: - files: ['$test'] - deny: - - pkg: github.com/pkg/errors - desc: Use the "errors" package unless you are interacting with stack traces. - - errchkjson: - check-error-free-encoding: true - - exhaustive: - default-signifies-exhaustive: true - - goheader: - template: |- - Copyright {{ DATES }} Crunchy Data Solutions, Inc. - - SPDX-License-Identifier: Apache-2.0 - values: - regexp: - DATES: '((201[7-9]|202[0-4]) - 2025|2025)' - - goimports: - local-prefixes: github.com/crunchydata/postgres-operator - - gomodguard: - blocked: - modules: - - gopkg.in/yaml.v2: { recommendations: [sigs.k8s.io/yaml] } - - gopkg.in/yaml.v3: { recommendations: [sigs.k8s.io/yaml] } - - gotest.tools: { recommendations: [gotest.tools/v3] } - - k8s.io/kubernetes: - reason: > - k8s.io/kubernetes is for managing dependencies of the Kubernetes - project, i.e. building kubelet and kubeadm. - - gosec: - excludes: - # Flags for potentially-unsafe casting of ints, similar problem to globally-disabled G103 - - G115 - - importas: - alias: - - pkg: k8s.io/api/(\w+)/(v[\w\w]+) - alias: $1$2 - - pkg: k8s.io/apimachinery/pkg/apis/(\w+)/(v[\w\d]+) - alias: $1$2 - - pkg: k8s.io/apimachinery/pkg/api/errors - alias: apierrors - no-unaliased: true - - spancheck: - checks: [end, record-error] - extra-start-span-signatures: - - 'github.com/crunchydata/postgres-operator/internal/tracing.Start:opentelemetry' - ignore-check-signatures: - - 'tracing.Escape' + # It is fine for tests to use "math/rand" packages. + - linters: [gosec] + path: '(.+)_test[.]go' + text: weak random number generator + + # This internal package is the one place we want to do API discovery. + - linters: [depguard] + path: internal/kubernetes/discovery.go + text: k8s.io/client-go/discovery + + # Postgres HBA rules often include "all all all" + - linters: [dupword] + path: /(hba|postgres)[^/]+$ + text: words \(all\) found + + # These value types have unmarshal methods. + # https://github.com/raeperd/recvcheck/issues/7 + - linters: [recvcheck] + path: internal/pki/pki.go + text: methods of "(Certificate|PrivateKey)" + + - linters: [staticcheck] + text: corev1.(Endpoints|EndpointSubset) is deprecated + + - linters: [staticcheck] + path: internal/controller/ + text: >- + deprecated: Use `RequeueAfter` instead + +# https://golangci-lint.run/usage/formatters +formatters: + enable: + - gci + - gofmt + settings: + gci: + sections: + - standard + - default + - localmodule issues: - exclude-generated: strict - exclude-rules: - # This internal package is the one place we want to do API discovery. - - linters: [depguard] - path: internal/kubernetes/discovery.go - text: k8s.io/client-go/discovery - - # These value types have unmarshal methods. - # https://github.com/raeperd/recvcheck/issues/7 - - linters: [recvcheck] - path: internal/pki/pki.go - text: 'methods of "(Certificate|PrivateKey)"' + # Fix only when requested + fix: false + + # Show all issues at once + max-issues-per-linter: 0 + max-same-issues: 0 + uniq-by-line: false diff --git a/Makefile b/Makefile index 1ec77512ea..ad32ad2f86 100644 --- a/Makefile +++ b/Makefile @@ -15,7 +15,7 @@ BUILDAH_BUILD ?= buildah bud GO ?= go GO_BUILD = $(GO) build GO_TEST ?= $(GO) test -KUTTL ?= kubectl-kuttl +KUTTL ?= $(GO) run github.com/kudobuilder/kuttl/cmd/kubectl-kuttl@latest KUTTL_TEST ?= $(KUTTL) test ENVTEST_K8S_VERSION ?= 1.34