[TEP-0100] Fields/flags/docs for embedded TaskRun and Run statuses in PipelineRuns

See https://github.com/tektoncd/community/blob/main/teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md

This adds new choices for how `TaskRun` and `Run` statuses are stored/referenced in `PipelineRun`
statuses, depending on how the new feature flag, `embedded-status` is set:
* `full` - the current default, with the `TaskRun` and `Run` statuses embedded in full.
* `minimal` - instead of storing the full embedded statuses, information is stored about the task's
   version, kind, `PipelineTask` name, and the underlying name of the `TaskRun` or `Run`. This can be
   used to look up the `TaskRun` or `Run` itself and get the full status from there. Information
   which is specific to the `PipelineRun`, namely condition checks status and when expressions which
   may have blocked creation of an actual `TaskRun` or `Run` are stored as well.
* `both` - both the full embedded `TaskRun` and `Run` statuses and the minimal references are stored.

The condition check information will be removed from the minimal references once the now-deprecated
condition check functionality is removed from Tekton Pipeline.

As described in https://github.com/tektoncd/community/blob/main/teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md#beta-api,
9 months after this change has been released in `v1beta1`, the default value for the
`embedded-status` feature flag will be changed from `full` to `minimal`, and a few months after
that, the `embedded-status` feature flag will be removed entirely and only the `minimal` behavior
will remain.

Note that this also may address tektoncd#4657, since I had to add documentation for `PipelineRunStatus`,
and while I was there, I added some missing top-level fields to the `PipelineRun` definition.

The actual implementation will follow this PR.

Signed-off-by: Andrew Bayer <andrew.bayer@gmail.com>

Author: abayer

docs/install.md

@@ -390,6 +390,11 @@ features](#alpha-features) to be used.
   to "false" to guard a `Task` and its dependent `Tasks`. It defaults to "true". For more information, see [guarding
   `Task` execution using `when` expressions](pipelines.md#guard-task-execution-using-whenexpressions).
 
+- `embedded-status`: set this flag to "full" to enable full embedding of `TaskRun` and `Run` statuses in the 
+ `PipelineRun` status. Set it to "minimal" to populate the `ChildReferences` field in the `PipelineRun` status with
+  name, kind, and API version information for each `TaskRun` and `Run` in the `PipelineRun` instead. Set it to "both" to 
+  do both. For more information, see [Configuring usage of `TaskRun` and `Run` embedded statuses](pipelineruns.md#configuring-usage-of-taskrun-and-run-embedded-statuses). **NOTE**: This functionality is not yet active.
+
 For example:
 
 ```yaml
@@ -423,6 +428,7 @@ Features currently in "alpha" are:
 | [Debug](./debug.md)                                                                                   | [TEP-0042](https://github.com/tektoncd/community/blob/main/teps/0042-taskrun-breakpoint-on-failure.md)               | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0) |                             |
 | [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars)                        | [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) |                                                                      |                             |
 | [Matrix](./matrix.md)                                                                                 | [TEP-0090](https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md)                                      |                                                                      |                             |
+| [Embedded Statuse](pipelineruns.md#configuring-usage-of-taskrun-and-run-embedded-statuses)                                                                                  | [TEP-0100](https://github.com/tektoncd/community/blob/main/teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md)                                      |                                                                      |                             |
 
 ## Configuring High Availability
 

docs/pipelineruns.md

@@ -23,7 +23,10 @@ weight: 500
     - [Specifying <code>Workspaces</code>](#specifying-workspaces)
     - [Specifying <code>LimitRange</code> values](#specifying-limitrange-values)
     - [Configuring a failure timeout](#configuring-a-failure-timeout)
-  - [Monitoring execution status](#monitoring-execution-status)
+  - [<code>PipelineRun</code> status](#pipelinerun-status)
+    - [The <code>status</code> field](#the-status-field) 
+    - [Configuring usage of <code>TaskRun</code> and <code>Run</code> embedded statuses](#configuring-usage-of-taskrun-and-run-embedded-statuses)
+    - [Monitoring execution status](#monitoring-execution-status)
   - [Cancelling a <code>PipelineRun</code>](#cancelling-a-pipelinerun)
   - [Gracefully cancelling a <code>PipelineRun</code>](#gracefully-cancelling-a-pipelinerun)
   - [Gracefully stopping a <code>PipelineRun</code>](#gracefully-stopping-a-pipelinerun)
@@ -67,10 +70,11 @@ A `PipelineRun` definition supports the following fields:
     object that supplies specific execution credentials for the `Pipeline`.
   - [`serviceAccountNames`](#mapping-serviceaccount-credentials-to-tasks) - Maps specific `serviceAccountName` values
     to `Tasks` in the `Pipeline`. This overrides the credentials set for the entire `Pipeline`.
-  - [`taskRunSpec`](#specifying-taskrunspecs) - Specifies a list of `PipelineRunTaskSpec` which allows for setting `ServiceAccountName` and [`Pod` template](./podtemplates.md) for each task. This overrides the `Pod` template set for the entire `Pipeline`.
-  - [`timeout`](#configuring-a-failure-timeout) - Specifies the timeout before the `PipelineRun` fails.
-  - [`podTemplate`](#specifying-a-pod-template) - Specifies a [`Pod` template](./podtemplates.md) to use as the basis
-    for the configuration of the `Pod` that executes each `Task`.
+  - [`taskRunSpecs`](#specifying-taskrunspecs) - Specifies a list of `PipelineRunTaskSpec` which allows for setting `ServiceAccountName` and [`Pod` template](./podtemplates.md) for each task. This overrides the `Pod` template set for the entire `Pipeline`.
+  - [`timeout`](#configuring-a-failure-timeout) - Specifies the timeout before the `PipelineRun` fails. `timeout` is deprecated and will eventually be removed, so consider using `timeouts` instead.
+  - [`timeouts`](#configuring-a-failure-timeout) - Specifies the timeout before the `PipelineRun` fails. `timeouts` allows more granular timeout configuration, at the pipeline, tasks, and finally levels
+  - [`podTemplate`](#specifying-a-pod-template) - Specifies a [`Pod` template](./podtemplates.md) to use as the basis for the configuration of the `Pod` that executes each `Task`.
+  - [`workspaces`](#specifying-workspaces) - Specifies a set of workspace bindings which must match the names of workspaces declared in the pipeline being used. 
 
 [kubernetes-overview]:
   https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
@@ -560,7 +564,43 @@ The `timeout` value is a `duration` conforming to Go's
 values are `1h30m`, `1h`, `1m`, and `60s`. If you set the global timeout to 0, all `PipelineRuns`
 that do not have an individual timeout set will fail immediately upon encountering an error.
 
-## Monitoring execution status
+## `PipelineRun` status
+
+### The `status` field
+
+Your `PipelineRun`'s `status` field can contain the following fields:
+
+- Required:
+  - `status` - Most relevant, `status.conditions`, which contains the latest observations of the `PipelineRun`'s state. [See here](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties) for information on typical status properties. 
+  - `startTime` - The time at which the `PipelineRun` began executing, in [RFC3339](https://tools.ietf.org/html/rfc3339) format.
+  - `completionTime` - The time at which the `PipelineRun` finished executing, in [RFC3339](https://tools.ietf.org/html/rfc3339) format.
+  - [`pipelineSpec`](pipelines.md#configuring-a-pipeline) - The exact `PipelineSpec` used when starting the `PipelineRun`.
+- Optional:
+  - `taskRuns` - A map of `TaskRun` names to detailed information about the status of that `TaskRun`. This is deprecated and will be removed in favor of using `childReferences`.
+  - `runs` - A map of custom task `Run` names to detailed information about the status of that `Run`. This is deprecated and will be removed in favor of using `childReferences`.
+  - [`pipelineResults`](pipelines.md#emitting-results-from-a-pipeline) - Results emitted by this `PipelineRun`.
+  - `skippedTasks` - A list of `Task`s which were skipped when running this `PipelineRun` due to [when expressions](pipelines.md#guard-task-execution-using-when-expressions), including the when expressions applying to the skipped task.
+  - `childReferences` - A list of references to each `TaskRun` or `Run` in this `PipelineRun`, which can be used to look up the status of the underlying `TaskRun` or `Run`. Each entry contains the following:
+    - [`kind`][kubernetes-overview] - Generally either `TaskRun` or `Run`.
+    - [`apiVersion`][kubernetes-overview] - The API version for the underlying `TaskRun` or `Run`.
+    - `conditionChecks` - A list of [condition checks](conditions.md) performed for this `TaskRun`. `conditions` are deprecated and this will be removed in the future.
+    - [`whenExpressions`](pipelines.md#guard-task-execution-using-when-expressions) - The list of when expressions guarding the execution of this task.
+
+### Configuring usage of `TaskRun` and `Run` embedded statuses
+
+Currently, the default behavior is for the statuses of `TaskRun`s and `Run`s within this `PipelineRun`
+to be embedded in the `status.taskRuns` and `status.runs` fields. This will change in the future to
+instead default to `status.childReferences` being populated with references to the `TaskRun`s and
+`Run`s, which can be used to look up their statuses.
+
+This behavior can be controlled by changing the `embedded-status` feature flag in the `feature-flags`
+config map. See [`install.md`](./install.md#customizing-the-pipelines-controller-behavior) for more
+information on feature flags. The possible values for `embedded-status` are:
+- `full` - The current default behavior of populating `status.taskRuns` and `status.runs`, without populating `status.childReferences`.
+- `minimal` - Just populate `status.childReferences`, not `status.taskRuns` or `status.runs`.
+- `both` - Populate `status.childReferences` as well as `status.taskRuns` and `status.runs`.
+
+### Monitoring execution status
 
 As your `PipelineRun` executes, its `status` field accumulates information on the execution of each `TaskRun`
 as well as the `PipelineRun` as a whole. This information includes the name of the pipeline `Task` associated

pkg/apis/config/feature_flags.go

@@ -30,6 +30,15 @@ const (
 	StableAPIFields = "stable"
 	// AlphaAPIFields is the value used for "enable-api-fields" when alpha APIs should be usable as well.
 	AlphaAPIFields = "alpha"
+	// FullEmbeddedStatus is the value used for "embedded-status" when the full statuses of TaskRuns and Runs should be
+	// embedded in PipelineRunStatusFields, but ChildReferences should not be used.
+	FullEmbeddedStatus = "full"
+	// BothEmbeddedStatus is the value used for "embedded-status" when full embedded statuses of TaskRuns and Runs as
+	// well as ChildReferences should be used in PipelineRunStatusFields.
+	BothEmbeddedStatus = "both"
+	// MinimalEmbeddedStatus is the value used for "embedded-status" when only ChildReferences should be used in
+	// PipelineRunStatusFields.
+	MinimalEmbeddedStatus = "minimal"
 	// DefaultDisableAffinityAssistant is the default value for "disable-affinity-assistant".
 	DefaultDisableAffinityAssistant = false
 	// DefaultDisableCredsInit is the default value for "disable-creds-init".
@@ -48,6 +57,8 @@ const (
 	DefaultEnableAPIFields = StableAPIFields
 	// DefaultSendCloudEventsForRuns is the default value for "send-cloudevents-for-runs".
 	DefaultSendCloudEventsForRuns = false
+	// DefaultEmbeddedStatus is the default value for "embedded-status".
+	DefaultEmbeddedStatus = FullEmbeddedStatus
 
 	disableAffinityAssistantKey         = "disable-affinity-assistant"
 	disableCredsInitKey                 = "disable-creds-init"
@@ -58,6 +69,7 @@ const (
 	enableAPIFields                     = "enable-api-fields"
 	scopeWhenExpressionsToTask          = "scope-when-expressions-to-task"
 	sendCloudEventsForRuns              = "send-cloudevents-for-runs"
+	embeddedStatus                      = "embedded-status"
 )
 
 // FeatureFlags holds the features configurations
@@ -72,6 +84,7 @@ type FeatureFlags struct {
 	ScopeWhenExpressionsToTask       bool
 	EnableAPIFields                  string
 	SendCloudEventsForRuns           bool
+	EmbeddedStatus                   string
 }
 
 // GetFeatureFlagsConfigName returns the name of the configmap containing all
@@ -120,6 +133,9 @@ func NewFeatureFlagsFromMap(cfgMap map[string]string) (*FeatureFlags, error) {
 	if err := setFeature(sendCloudEventsForRuns, DefaultSendCloudEventsForRuns, &tc.SendCloudEventsForRuns); err != nil {
 		return nil, err
 	}
+	if err := setEmbeddedStatus(cfgMap, DefaultEmbeddedStatus, &tc.EmbeddedStatus); err != nil {
+		return nil, err
+	}
 
 	// Given that they are alpha features, Tekton Bundles and Custom Tasks should be switched on if
 	// enable-api-fields is "alpha". If enable-api-fields is not "alpha" then fall back to the value of
@@ -157,6 +173,22 @@ func setEnabledAPIFields(cfgMap map[string]string, defaultValue string, feature
 	return nil
 }
 
+// setEmbeddedStatus sets the "embedded-status" flag based on the content of a given map.
+// If the feature gate is invalid or missing then an error is returned.
+func setEmbeddedStatus(cfgMap map[string]string, defaultValue string, feature *string) error {
+	value := defaultValue
+	if cfg, ok := cfgMap[embeddedStatus]; ok {
+		value = strings.ToLower(cfg)
+	}
+	switch value {
+	case FullEmbeddedStatus, BothEmbeddedStatus, MinimalEmbeddedStatus:
+		*feature = value
+	default:
+		return fmt.Errorf("invalid value for feature flag %q: %q", embeddedStatus, value)
+	}
+	return nil
+}
+
 // NewFeatureFlagsFromConfigMap returns a Config for the given configmap
 func NewFeatureFlagsFromConfigMap(config *corev1.ConfigMap) (*FeatureFlags, error) {
 	return NewFeatureFlagsFromMap(config.Data)

pkg/apis/config/feature_flags_test.go

@@ -38,6 +38,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
 				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 				EnableAPIFields:                  "stable",
+				EmbeddedStatus:                   config.DefaultEmbeddedStatus,
 			},
 			fileName: config.GetFeatureFlagsConfigName(),
 		},
@@ -51,6 +52,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				ScopeWhenExpressionsToTask:       true,
 				EnableAPIFields:                  "alpha",
 				SendCloudEventsForRuns:           true,
+				EmbeddedStatus:                   "both",
 			},
 			fileName: "feature-flags-all-flags-set",
 		},
@@ -64,6 +66,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
 				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
+				EmbeddedStatus:                   config.DefaultEmbeddedStatus,
 			},
 			fileName: "feature-flags-enable-api-fields-overrides-bundles-and-custom-tasks",
 		},
@@ -75,6 +78,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
 				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
+				EmbeddedStatus:                   config.DefaultEmbeddedStatus,
 			},
 			fileName: "feature-flags-bundles-and-custom-tasks",
 		},
@@ -95,6 +99,7 @@ func TestNewFeatureFlagsFromEmptyConfigMap(t *testing.T) {
 		RunningInEnvWithInjectedSidecars: true,
 		ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 		EnableAPIFields:                  "stable",
+		EmbeddedStatus:                   config.DefaultEmbeddedStatus,
 	}
 	verifyConfigFileWithExpectedFeatureFlagsConfig(t, FeatureFlagsConfigEmptyName, expectedConfig)
 }
@@ -137,6 +142,8 @@ func TestNewFeatureFlagsConfigMapErrors(t *testing.T) {
 		fileName: "feature-flags-invalid-boolean",
 	}, {
 		fileName: "feature-flags-invalid-enable-api-fields",
+	}, {
+		fileName: "feature-flags-invalid-embedded-status",
 	}, {
 		fileName: "feature-flags-invalid-scope-when-expressions-to-task",
 	}} {

pkg/apis/config/testdata/feature-flags-all-flags-set.yaml

@@ -26,3 +26,4 @@ data:
   scope-when-expressions-to-task: "true"
   enable-api-fields: "alpha"
   send-cloudevents-for-runs: "true"
+  embedded-status: "both"

pkg/apis/config/testdata/feature-flags-invalid-embedded-status.yaml

@@ -0,0 +1,21 @@
+# Copyright 2021 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: feature-flags
+  namespace: tekton-pipelines
+data:
+  embedded-status: "im-not-a-valid-feature-gate"