feat: always use a default workgroup config, instead of retrieving a workgroup config from the AWS API

[inbar-beehero]

Feature or Bugfix

• Feature

Detail

Remove redundant call to AWS API in order to get the workgroup config, as it will take place anyways if not overridden by the client

Relates

Issue #3233

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 39570b4
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 39570b4
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[kukushking]

Looks like a few test cases failed with the error:

botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the
 StartQueryExecution operation: The Create Table As Select query failed because it was submitted with an
  'external_location' property to an Athena Workgroup that enforces a centralized output location for all queries. 
  Please remove the 'external_location' property and resubmit the query.

From my analysis, the issue is in how we override s3_output:

def _get_s3_output(
    s3_output: str | None, wg_config: _WorkGroupConfig, boto3_session: boto3.Session | None = None
) -> str:
    if wg_config.enforced and wg_config.s3_output is not None:
        return wg_config.s3_output
    if s3_output is not None:
        return s3_output
    if wg_config.s3_output is not None:
        return wg_config.s3_output
    return create_athena_bucket(boto3_session=boto3_session)

Since we are not fetching wg_config anymore, we do not know if the workgroup config is enforced until we submit the query. We are however overriding s3_output, that might confict with enforced workgroup config.

I think we might want to deprecate _get_workgroup_config/_get_default_workgroup_config, and assume if the user passed s3_output, then it means they want to override it (and assume workgroup allows it).

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 0d10c7f
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 53c05b3
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 53c05b3
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 0d10c7f
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[inbar-beehero]

@kukushking agreed - if the user passed an S3 output, it should be included in start query execution, otherwise nothing should be passed for S3 output (so the workgroup definitions will take place). I'll make the code change

[inbar-beehero]

Unload requires a TO location, so keeping the previous implementation here, but added an exception in case we don't have S3 output from either user or workgroup

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 4b79b76
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
• Commit ID: 535b8dc
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 4b79b76
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 535b8dc
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository