diff --git a/.github/workflows/gradle.yml b/.github/workflows/gradle.yml index 8409190cd1..cb680f644c 100644 --- a/.github/workflows/gradle.yml +++ b/.github/workflows/gradle.yml @@ -66,7 +66,9 @@ jobs: ./gradlew check sourceTarball distTar distZip publishToMavenLocal \ -x :polaris-runtime-service:test \ -x :polaris-admin:test \ - -x intTest --continue + -x intTest \ + -x :polaris-helm:check \ + --continue - name: Save partial Gradle build cache uses: ./.github/actions/ci-incr-build-cache-save if: github.event_name == 'push' && github.ref == 'refs/heads/main' @@ -213,6 +215,7 @@ jobs: ./gradlew \ intTest \ -x :polaris-runtime-service:intTest \ + -x :polaris-helm:intTest \ --continue env: DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} diff --git a/.github/workflows/helm.yml b/.github/workflows/helm.yml index 4c31fb39f8..759494e4b7 100644 --- a/.github/workflows/helm.yml +++ b/.github/workflows/helm.yml @@ -98,8 +98,6 @@ jobs: ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ - :polaris-admin:assemble \ - :polaris-admin:quarkusAppPartsBuild --rerun \ -Dquarkus.container-image.build=true minikube image ls diff --git a/.github/workflows/spark_client_regtests.yml b/.github/workflows/spark_client_regtests.yml index 1ec0082c31..ae3f207b67 100644 --- a/.github/workflows/spark_client_regtests.yml +++ b/.github/workflows/spark_client_regtests.yml @@ -58,7 +58,7 @@ jobs: # publishToMavenLocal causes a GH API requests, use the token for those requests GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }} - run: ./gradlew assemble publishToMavenLocal + run: ./gradlew assemble publishToMavenLocal -x :polaris-helm:assemble - name: Image build env: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dbacefaa59..273f886335 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -119,12 +119,86 @@ Tips: * Keep in mind that the Git commit subject and message is going to be read by other people, potentially even after years. The Git commit subject and message will appear "as is" in release notes. * Make sure the subject and message are properly formatted and contains a concise description of the changes in way that someone who has no prior knowledge can understand the rationale of the change and the change itself. Remove information that's of no use for someone reading the Git commit log, for example single intermediate commit messages like `formatting` or `fix test`. -## Java version requirements +## Build Prerequisites and Required Tools -The Apache Polaris build currently requires Java 21 or later. There are a few tools that help you running the right Java version: +Apache Polaris (incubating) requires several tools to run a full build, including tests and documentation generation. -* [SDKMAN!](https://sdkman.io/) follow the installation instructions, then run `sdk list java` to see the available distributions and versions, then run `sdk install java ` using the identifier for the distribution and version (>= 21) of your choice. -* [jenv](https://www.jenv.be/) If on a Mac you can use jenv to set the appropriate SDK. +### Core Build Requirements + +These tools are required for basic building and testing: + +* **Git**: Required for cloning the repository and version control. + +* **Java 21+**: The build requires Java 21 or later. Suggested installers: + * [SDKMAN!](https://sdkman.io/) - Run `sdk list java` to see available distributions, then `sdk install java ` to install. + * [jEnv](https://www.jenv.be/) - You can also use jEnv to manage Java versions. + +* **Container Runtime**: A container runtime like [Docker] or [Podman] is required for integration tests, regression tests + and building container images. + +[Docker]: https://www.docker.com/ +[Podman]: https://podman.io/ + +### Helm Chart Testing Requirements + +These tools are required to run Helm chart tests (part of `./gradlew test` and `./gradlew intTest`): + +* **Helm**: Kubernetes package manager, required for template validation and unit tests. + * macOS: `brew install helm` + * See [Helm installation instructions](https://helm.sh/docs/intro/install/) + +* **Helm Unittest Plugin**: Required for running Helm unit tests. + * Install: `helm plugin install https://github.com/helm-unittest/helm-unittest.git` + +* **helm-docs**: Required for generating Helm chart documentation. + * macOS: `brew install norwoodj/tap/helm-docs` + * See [helm-docs installation instructions](https://github.com/norwoodj/helm-docs) + +* **chart-testing (ct)**: Required for linting and testing Helm charts. + * macOS: `brew install chart-testing` + * See [chart-testing installation instructions](https://github.com/helm/chart-testing) + +* **Minikube**: Required for running Helm chart integration tests. + * macOS: `brew install minikube` + * See [Minikube installation instructions](https://minikube.sigs.k8s.io/docs/start/) + +* **kubectl**: Kubernetes command-line tool, required for Helm integration tests. + * macOS: `brew install kubectl` + * See [kubectl installation instructions](https://kubernetes.io/docs/tasks/tools/) + +### Other Tools + +These tools are helpful but not strictly required: + +* **jq**: JSON processor, useful for working with Polaris APIs and scripts. + * macOS: `brew install jq` + * See [jq installation instructions](https://jqlang.github.io/jq/download/) + +### Quick Installation (macOS) + +For macOS users with Homebrew, you can install all dependencies at once using the provided Makefile: + +```bash +make install-dependencies-brew +make install-optional-dependencies-brew +``` + +To check which dependencies are installed: +```bash +make check-dependencies +``` + +### Skipping Specific Tests + +If you don't have all the Helm-related tools installed, you can skip those tests: + +```bash +# Run checks excluding Helm tests +./gradlew check -x :polaris-helm:test + +# Run build excluding Helm integration tests +./gradlew build -x :polaris-helm:intTest +``` ## Code Contribution Guidelines diff --git a/gradle/projects.main.properties b/gradle/projects.main.properties index 5869bc5bc7..92264c6a0b 100644 --- a/gradle/projects.main.properties +++ b/gradle/projects.main.properties @@ -50,6 +50,8 @@ polaris-config-docs-annotations=tools/config-docs/annotations polaris-config-docs-generator=tools/config-docs/generator polaris-config-docs-site=tools/config-docs/site +polaris-helm=helm/polaris + # executor abstraction polaris-async-api=persistence/nosql/async/api polaris-async-java=persistence/nosql/async/java diff --git a/helm/polaris/README.md b/helm/polaris/README.md index bbfc8557df..9d12fa6b57 100644 --- a/helm/polaris/README.md +++ b/helm/polaris/README.md @@ -28,6 +28,7 @@ weight: 675 Do not modify the README.md file directly, please modify README.md.gotmpl instead. To re-generate the README.md file, install helm-docs then run from the repo root: helm-docs --chart-search-root=helm + Alternatively, run `./gradlew helmDocs` or `make helm-doc-generate` from the repo root. --> ![Version: 1.2.0-incubating-SNAPSHOT](https://img.shields.io/badge/Version-1.2.0--incubating--SNAPSHOT-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.2.0-incubating-SNAPSHOT](https://img.shields.io/badge/AppVersion-1.2.0--incubating--SNAPSHOT-informational?style=flat-square) @@ -55,8 +56,6 @@ eval $(minikube docker-env) ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ - :polaris-admin:assemble \ - :polaris-admin:quarkusAppPartsBuild --rerun \ -Dquarkus.container-image.build=true ``` @@ -149,6 +148,13 @@ helm plugin install https://github.com/helm-unittest/helm-unittest.git brew install chart-testing ``` +Alternatively, on macOS, you can also run the following make targets: + +```bash +make install-dependencies-brew +make install-optional-dependencies-brew +``` + The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory contains the required resources. To deploy them, run the following command: ```bash @@ -183,6 +189,15 @@ Integration tests are run with the Chart Testing tool: ct install --namespace polaris --charts ./helm/polaris ``` +### Running tests with Gradle + +Both unit and integration tests can be run with Gradle. From the Polaris repo root: + +```bash +./gradlew :polaris-helm:test +./gradlew :polaris-helm:intTest +``` + ## Values | Key | Type | Default | Description | diff --git a/helm/polaris/README.md.gotmpl b/helm/polaris/README.md.gotmpl index 99feabb101..cfae9b9b13 100644 --- a/helm/polaris/README.md.gotmpl +++ b/helm/polaris/README.md.gotmpl @@ -28,6 +28,7 @@ weight: 675 Do not modify the README.md file directly, please modify README.md.gotmpl instead. To re-generate the README.md file, install helm-docs then run from the repo root: helm-docs --chart-search-root=helm + Alternatively, run `./gradlew helmDocs` or `make helm-doc-generate` from the repo root. --> {{ template "chart.deprecationWarning" . }} @@ -57,8 +58,6 @@ eval $(minikube docker-env) ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ - :polaris-admin:assemble \ - :polaris-admin:quarkusAppPartsBuild --rerun \ -Dquarkus.container-image.build=true ``` @@ -151,6 +150,13 @@ helm plugin install https://github.com/helm-unittest/helm-unittest.git brew install chart-testing ``` +Alternatively, on macOS, you can also run the following make targets: + +```bash +make install-dependencies-brew +make install-optional-dependencies-brew +``` + The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory contains the required resources. To deploy them, run the following command: ```bash @@ -185,4 +191,13 @@ Integration tests are run with the Chart Testing tool: ct install --namespace polaris --charts ./helm/polaris ``` +### Running tests with Gradle + +Both unit and integration tests can be run with Gradle. From the Polaris repo root: + +```bash +./gradlew :polaris-helm:test +./gradlew :polaris-helm:intTest +``` + {{ template "chart.valuesSection" . }} diff --git a/helm/polaris/build.gradle.kts b/helm/polaris/build.gradle.kts new file mode 100644 index 0000000000..06933f6a45 --- /dev/null +++ b/helm/polaris/build.gradle.kts @@ -0,0 +1,247 @@ +import java.io.OutputStream + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +plugins { + id("base") + id("polaris-spotless") +} + +description = "Apache Polaris (incubating) Helm Chart" + +val runtimeServerDistribution by + configurations.creating { + isCanBeConsumed = false + isCanBeResolved = true + } + +dependencies { runtimeServerDistribution(project(":polaris-server", "distributionElements")) } + +val helmTestReportsDir = layout.buildDirectory.dir("reports") + +val helmTemplateValidation by + tasks.registering(Exec::class) { + group = "verification" + description = "Run 'helm template' validation with all values files" + + val outputFile = helmTestReportsDir.get().file("helm-template/validation.log").asFile + + runShellScript( + """ + set -e + for f in values.yaml ci/*.yaml; do + echo "Validating helm template with ${'$'}f" + helm template --debug --namespace polaris-ns --values ${'$'}f . + done + """, + outputFile, + ) + + inputs.files( + fileTree(projectDir) { include("Chart.yaml", "values.yaml", "ci/*.yaml", "templates/**/*") } + ) + + outputs.cacheIf { true } + } + +val helmUnitTest by + tasks.registering(Exec::class) { + group = "verification" + description = "Run Helm unit tests" + + val outputFile = helmTestReportsDir.get().file("helm-unit/test.log").asFile + + runShellScript( + """ + set -e + echo "====== Install helm-unittest plugin ======" + helm plugin install https://github.com/helm-unittest/helm-unittest.git || true + + echo "====== Run helm unit tests ======" + helm unittest . + """, + outputFile, + ) + + inputs.files( + fileTree(projectDir) { include("Chart.yaml", "values.yaml", "templates/**/*", "tests/**/*") } + ) + + outputs.cacheIf { true } + } + +val chartTestingLint by + tasks.registering(Exec::class) { + group = "verification" + description = "Run chart-testing (lint)" + + val outputFile = helmTestReportsDir.get().file("ct/lint.log").asFile + + runShellScript( + """ + set -e + ct lint --debug --charts . + """, + outputFile, + ) + + inputs.files( + fileTree(projectDir) { include("Chart.yaml", "values.yaml", "ci/*.yaml", "templates/**/*") } + ) + + outputs.cacheIf { true } + } + +// Task to build Docker images in minikube's Docker environment +val buildMinikubeImages by + tasks.registering(Exec::class) { + group = "build" + description = "Build Polaris Docker images in minikube's Docker environment" + + // Must be run from root project directory because Dockerfile.jvm must be + // contained within the build context + workingDir = rootProject.projectDir + + val outputFile = helmTestReportsDir.get().file("minikube-images/build.log").asFile + val digestFile = helmTestReportsDir.get().file("minikube-images/build.sha256").asFile + val dockerFile = project(":polaris-server").projectDir.resolve("src/main/docker/Dockerfile.jvm") + + digestFile.parentFile.mkdirs() + + runShellScript( + """ + set -e + + echo "====== Set up docker environment and build images ======" + eval $(minikube -p minikube docker-env) + + echo "====== Build server image ======" + docker build -t apache/polaris:latest \ + -f ${dockerFile.relativeTo(workingDir)} \ + ${project(":polaris-server").projectDir.relativeTo(workingDir)} + + { + docker inspect --format='{{.Id}}' apache/polaris:latest + } > ${digestFile.relativeTo(workingDir)} + """, + outputFile, + ) + + dependsOn(":polaris-server:quarkusBuild") + + inputs.files(runtimeServerDistribution) + inputs.file(dockerFile) + + outputs.cacheIf { true } + } + +// Task to run chart-testing install on minikube +val chartTestingInstall by + tasks.registering(Exec::class) { + group = "verification" + description = "Run chart-testing (install) on minikube" + + val outputFile = helmTestReportsDir.get().file("ct/install.log").asFile + + runShellScript( + """ + set -eo pipefail + + echo "====== Create namespace if it doesn't exist ======" + kubectl create namespace polaris-ns --dry-run=client -o yaml | kubectl apply -f - + + echo "===== Install fixtures ======" + kubectl apply --namespace polaris-ns -f ci/fixtures + + echo "===== Run chart-testing install ======" + ct install --namespace polaris-ns --debug --charts . + """, + outputFile, + ) + + dependsOn(buildMinikubeImages) + + inputs.files( + fileTree(projectDir) { include("Chart.yaml", "values.yaml", "ci/**/*", "templates/**/*") } + ) + + outputs.cacheIf { true } + } + +val helmDocs by + tasks.registering(Exec::class) { + group = "documentation" + description = "Generate Helm chart documentation using helm-docs" + + val outputFile = helmTestReportsDir.get().file("helm-docs/build.log").asFile + + runShellScript( + """ + set -e + echo "====== Generate Helm documentation ======" + helm-docs --chart-search-root=. + """, + outputFile, + ) + + inputs.files(fileTree(projectDir) { include("Chart.yaml", "values.yaml", "README.md.gotmpl") }) + + outputs.cacheIf { true } + } + +val test by + tasks.registering { + group = "verification" + description = "Run all Helm chart tests" + dependsOn(helmTemplateValidation, helmUnitTest, chartTestingLint) + } + +val intTest by + tasks.registering { + group = "verification" + description = "Run Helm chart integration tests on minikube" + dependsOn(chartTestingInstall) + mustRunAfter(test) + } + +tasks.named("check") { dependsOn(test, intTest) } + +tasks.named("assemble") { dependsOn(helmDocs) } + +fun Exec.runShellScript(script: String, outputFile: File) { + var outStream: OutputStream? = null + doFirst { + outputFile.parentFile.mkdirs() + outStream = outputFile.outputStream() + standardOutput = outStream + errorOutput = outStream + } + commandLine = listOf("bash", "-c", script.trimIndent()) + this.isIgnoreExitValue = true + doLast { + outStream?.close() + val exitValue = executionResult.get().exitValue + if (exitValue != 0) { + this.logger.error("Shell script failed with exit code $exitValue:\n") + outputFile.readLines().forEach { this.logger.error(it) } + throw GradleException("Shell script failed with exit code $exitValue") + } + } +} diff --git a/site/content/in-dev/unreleased/helm.md b/site/content/in-dev/unreleased/helm.md index ef82e8e675..17e57eefa1 100644 --- a/site/content/in-dev/unreleased/helm.md +++ b/site/content/in-dev/unreleased/helm.md @@ -28,6 +28,7 @@ weight: 675 Do not modify the README.md file directly, please modify README.md.gotmpl instead. To re-generate the README.md file, install helm-docs then run from the repo root: helm-docs --chart-search-root=helm + Alternatively, run `./gradlew helmDocs` or `make helm-doc-generate` from the repo root. --> ![Version: 1.2.0-incubating-SNAPSHOT](https://img.shields.io/badge/Version-1.2.0--incubating--SNAPSHOT-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.2.0-incubating-SNAPSHOT](https://img.shields.io/badge/AppVersion-1.2.0--incubating--SNAPSHOT-informational?style=flat-square) @@ -55,8 +56,6 @@ eval $(minikube docker-env) ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ - :polaris-admin:assemble \ - :polaris-admin:quarkusAppPartsBuild --rerun \ -Dquarkus.container-image.build=true ``` @@ -151,6 +150,13 @@ helm plugin install https://github.com/helm-unittest/helm-unittest.git brew install chart-testing ``` +Alternatively, on macOS, you can also run the following make targets: + +```bash +make install-dependencies-brew +make install-optional-dependencies-brew +``` + The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory contains the required resources. To deploy them, run the following command: ```bash @@ -185,6 +191,15 @@ Integration tests are run with the Chart Testing tool: ct install --namespace polaris --charts ./helm/polaris ``` +### Running tests with Gradle + +Both unit and integration tests can be run with Gradle. From the Polaris repo root: + +```bash +./gradlew :polaris-helm:test +./gradlew :polaris-helm:intTest +``` + ## Values | Key | Type | Default | Description | @@ -312,6 +327,11 @@ ct install --namespace polaris --charts ./helm/polaris | persistence.relationalJdbc.secret.username | string | `"username"` | The secret key holding the database username for authentication | | persistence.type | string | `"in-memory"` | The type of persistence to use. Two built-in types are supported: in-memory and relational-jdbc. The eclipse-link type is also supported but is deprecated. | | podAnnotations | object | `{}` | Annotations to apply to polaris pods. | +| podDisruptionBudget | object | `{"annotations":{},"enabled":false,"maxUnavailable":null,"minAvailable":null}` | Pod disruption budget settings. | +| podDisruptionBudget.annotations | object | `{}` | Annotations to add to the pod disruption budget. | +| podDisruptionBudget.enabled | bool | `false` | Specifies whether a pod disruption budget should be created. | +| podDisruptionBudget.maxUnavailable | string | `nil` | The maximum number of pods that can be unavailable during disruptions. Can be an absolute number (ex: 5) or a percentage of desired pods (ex: 50%). IMPORTANT: Cannot be used simultaneously with minAvailable. | +| podDisruptionBudget.minAvailable | string | `nil` | The minimum number of pods that should remain available during disruptions. Can be an absolute number (ex: 5) or a percentage of desired pods (ex: 50%). IMPORTANT: Cannot be used simultaneously with maxUnavailable. | | podLabels | object | `{}` | Additional Labels to apply to polaris pods. | | podSecurityContext | object | `{"fsGroup":10001,"seccompProfile":{"type":"RuntimeDefault"}}` | Security context for the polaris pod. See https://kubernetes.io/docs/tasks/configure-pod-container/security-context/. | | podSecurityContext.fsGroup | int | `10001` | GID 10001 is compatible with Polaris OSS default images; change this if you are using a different image. |