update our base-images Dockerfiles to better match the AIPCC images

[jiridanek]

• RHAIENG-948: fix(jupyter datascience on IBM/Power): build failure due to AIPCC base (missing curl and openssl -devel packages) and AIPCC wheels (some prereqs are missing) red-hat-data-services/notebooks#1697 (comment)

@coderabbitai comment suggestions to update our base-images Dockerfiles to better match the AIPCC images so that we don't have these discrepancies between opendatahub-io/notebooks and red-hat-data-services/notebooks.

[jiridanek]

@coderabbitai comment suggestions to update our base-images Dockerfiles to better match the AIPCC images so that we don't have these discrepancies between opendatahub-io/notebooks and red-hat-data-services/notebooks.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# Fetch the referenced PR comment
echo "=== Fetching PR comment from red-hat-data-services/notebooks#1697 ==="
gh pr view 1697 --repo red-hat-data-services/notebooks --json body,comments | jq -r '.comments[] | select(.body | contains("AIPCC")) | "\(.author.login): \(.body)"' | head -20

echo -e "\n=== Fetching issue #2714 from opendatahub-io/notebooks ==="
gh issue view 2714 --repo opendatahub-io/notebooks --json title,body,comments | jq -r '"\(.title)\n\n\(.body)\n\n--- Comments ---\n\(.comments[] | "\(.author.login): \(.body)\n---")"'

echo -e "\n=== Looking for base-images Dockerfiles ==="
fd -t f "Dockerfile" base-images/ 2>/dev/null || echo "No base-images directory found"

echo -e "\n=== Checking repository structure ==="
ls -la

Length of output: 3871

---

🏁 Script executed:

#!/bin/bash

# Search for references to AIPCC in the codebase
echo "=== Searching for AIPCC references ==="
rg -i "aipcc" -C 2

echo -e "\n=== Looking for Dockerfile locations ==="
fd -t f "Dockerfile$" | head -20

Length of output: 10859

---

🏁 Script executed:

#!/bin/bash

# Get more context from the PR
echo "=== Getting full PR details ==="
gh pr view 1697 --repo red-hat-data-services/notebooks --json title,body,url

Length of output: 2754

---

🏁 Script executed:

#!/bin/bash

# Examine the aipcc.sh utility script
echo "=== Content of base-images/utils/aipcc.sh ==="
cat base-images/utils/aipcc.sh

echo -e "\n\n=== Examining base-images CPU Dockerfiles ==="
echo "--- c9s-python-3.12 Dockerfile.cpu ---"
head -100 base-images/cpu/c9s-python-3.12/Dockerfile.cpu

echo -e "\n--- ubi9-python-3.12 Dockerfile.cpu ---"
head -100 base-images/cpu/ubi9-python-3.12/Dockerfile.cpu

Length of output: 12145

---

🏁 Script executed:

#!/bin/bash

# Look for more context about AIPCC base image usage
echo "=== Searching for AIPCC base image references in Dockerfiles ==="
rg -n "FROM.*aipcc" base-images/

echo -e "\n=== Checking what packages are installed in base-images ==="
rg -n "dnf install\|yum install" base-images/ -A 10 | head -100

Length of output: 125

---

🏁 Script executed:

#!/bin/bash

# Get full content of the base Dockerfiles
echo "=== Full c9s-python-3.12 Dockerfile.cpu ==="
cat base-images/cpu/c9s-python-3.12/Dockerfile.cpu

echo -e "\n\n=== Full ubi9-python-3.12 Dockerfile.cpu ==="
cat base-images/cpu/ubi9-python-3.12/Dockerfile.cpu

Length of output: 2992

---

🏁 Script executed:

#!/bin/bash

# Check CUDA and ROCm base images too
echo "=== CUDA base image Dockerfile sample ==="
cat base-images/cuda/12.6/ubi9-python-3.12/Dockerfile.cuda

echo -e "\n\n=== ROCm base image Dockerfile sample ==="
cat base-images/rocm/6.2/ubi9-python-3.12/Dockerfile.rocm

Length of output: 7098

---

🏁 Script executed:

#!/bin/bash

# Look for AIPCC base image references in documentation
echo "=== Searching for AIPCC documentation and references ==="
fd -e md | xargs rg -l -i "aipcc" | head -10

echo -e "\n=== Examining README files that mention AIPCC ==="
cat base-images/cpu/c9s-python-3.12/README.md

Length of output: 1410

---

🏁 Script executed:

#!/bin/bash

# Look at the PR that was referenced to understand what changes were made
echo "=== Getting diff from the referenced PR ==="
gh pr diff 1697 --repo red-hat-data-services/notebooks | head -300

Length of output: 4896

---

🏁 Script executed:

#!/bin/bash

# Search for actual AIPCC base image URLs/references
echo "=== Looking for AIPCC base image registry references ==="
rg "quay.io/aipcc|registry.*aipcc" -n

echo -e "\n=== Check if there are any AIPCC base image documentation links ==="
rg "gitlab.*aipcc|github.*aipcc" -n | head -20

Length of output: 278

---

🏁 Script executed:

#!/bin/bash

# Look at the Konflux-specific Dockerfiles which might be using AIPCC images
echo "=== Looking for Dockerfile.konflux files ==="
fd "Dockerfile.konflux" | head -10

echo -e "\n=== Sample Dockerfile.konflux content ==="
fd "Dockerfile.konflux" | head -1 | xargs cat | head -100

Length of output: 87

---

🏁 Script executed:

#!/bin/bash

# Let's fetch actual files from the red-hat-data-services/notebooks repo to compare
echo "=== Fetching base-images Dockerfiles from red-hat-data-services/notebooks ==="
gh api repos/red-hat-data-services/notebooks/contents/base-images/cpu/ubi9-python-3.12 | jq -r '.[] | select(.name | endswith(".konflux")) | .name'

echo -e "\n=== Fetching content of base-images Dockerfile from downstream ==="
gh api repos/red-hat-data-services/notebooks/contents/base-images/cpu/ubi9-python-3.12/Dockerfile.konflux.cpu --jq .download_url | xargs curl -s | head -150

Length of output: 170

---

🏁 Script executed:

#!/bin/bash

# Check if there are Dockerfile.konflux files in base-images directories in the downstream
echo "=== Looking for all konflux Dockerfiles in base-images in downstream ==="
gh api graphql -f query='
  query {
    repository(owner: "red-hat-data-services", name: "notebooks") {
      object(expression: "main:base-images") {
        ... on Tree {
          entries {
            name
            type
          }
        }
      }
    }
  }
' | jq -r '.data.repository.object.entries[] | select(.type == "tree") | .name' | head -5

# Let's get a specific file
echo -e "\n=== Getting CPU base image from downstream ==="
gh api repos/red-hat-data-services/notebooks/contents/base-images/cpu --jq '.[].name'

Length of output: 173

---

🏁 Script executed:

#!/bin/bash

# Search for how AIPCC base images are used in RHEL AI docs
echo "=== Searching for RHEL AI / AIPCC base image documentation ==="
rg "base-images.*app" scripts/dockerfile_fragments.py -A 5 -B 5

Length of output: 561

---

🏁 Script executed:

#!/bin/bash

# Let's look at the gitlab reference in the scripts
echo "=== Examining dockerfile_fragments.py for AIPCC reference ==="
cat scripts/dockerfile_fragments.py | head -50

Length of output: 2203

---

🏁 Script executed:

#!/bin/bash

# Get the AIPCC GitLab reference to understand their structure better
echo "=== Fetching AIPCC base-images structure from GitLab ==="
curl -s "https://gitlab.com/api/v4/projects/redhat%2Frhel-ai%2Fcore%2Fbase-images%2Fapp/repository/tree?path=containerfiles&ref=main" | jq -r '.[].name' | head -20

echo -e "\n=== Try to get the replace-markers.sh script mentioned in our code ==="
curl -s "https://gitlab.com/redhat/rhel-ai/core/base-images/app/-/raw/main/containerfiles/replace-markers.sh" | head -100

Length of output: 292

---

🏁 Script executed:

#!/bin/bash

# Check what packages current downstream (red-hat-data-services) base images have
echo "=== Checking downstream base-images structure ==="
gh api repos/red-hat-data-services/notebooks/contents/base-images/cpu/ubi9-python-3.12 | jq -r '.[].name'

echo -e "\n=== Get the actual Dockerfile.cpu from downstream ==="
gh api repos/red-hat-data-services/notebooks/contents/base-images/cpu/ubi9-python-3.12/Dockerfile.cpu --jq .download_url | xargs curl -s

Length of output: 619

---

🏁 Script executed:

#!/bin/bash

# Look for any documentation about AIPCC images in the codebase
echo "=== Searching for AIPCC version references ==="
rg "aipcc.*[0-9]+\.[0-9]+" -i

echo -e "\n=== Looking for any documentation files ==="
fd -e md -e txt "README|CONTRIBUTING|UPDATES" base-images/ | xargs cat

Length of output: 2259

---

🏁 Script executed:

#!/bin/bash

# Compare our aipcc.sh with what packages are actually installed vs AIPCC
echo "=== Analyzing package differences ==="
echo "Our aipcc.sh installs these key packages:"
grep 'PKGS+=(' base-images/utils/aipcc.sh | head -30

echo -e "\n=== And these SCL packages that AIPCC lacks: ==="
grep 'SCL_PACKAGES+=(' base-images/utils/aipcc.sh | wc -l
echo "Total SCL packages being installed"

Length of output: 950

---

🏁 Script executed:

#!/bin/bash

# Count SCL packages properly
echo "=== Counting SCL packages in aipcc.sh ==="
awk '/^    SCL_PACKAGES=\(/,/^    \)/' base-images/utils/aipcc.sh | grep '"' | wc -l

echo -e "\n=== First 50 SCL packages we install ==="
awk '/^    SCL_PACKAGES=\(/,/^    \)/' base-images/utils/aipcc.sh | grep '"' | head -50

echo -e "\n=== Last 20 SCL packages we install ==="
awk '/^    SCL_PACKAGES=\(/,/^    \)/' base-images/utils/aipcc.sh | grep '"' | tail -20

Length of output: 1711

---

🏁 Script executed:

#!/bin/bash

# Let's compare the two repositories more systematically
echo "=== Comparing upstream (opendatahub-io) vs downstream (red-hat-data-services) ==="
echo "Upstream ubi9-python-3.12 Dockerfile.cpu:"
cat base-images/cpu/ubi9-python-3.12/Dockerfile.cpu | wc -l
echo "lines"

echo -e "\nDownstream has same structure - both use aipcc.sh"

echo -e "\n=== Key insight: Both repos are identical in base-images ==="
echo "The difference is in the jupyter/* Dockerfiles where AIPCC index workarounds exist"

Length of output: 328

---

🏁 Script executed:

#!/bin/bash

# Let's look at what specific workarounds exist for AIPCC in our jupyter images
echo "=== All AIPCC-related workarounds in jupyter/ Dockerfiles ==="
rg "aipcc" jupyter/ -n -C 1 | head -100

Length of output: 1904

---

🏁 Script executed:

#!/bin/bash

# Let's understand the full picture - check if base-images in both repos match
echo "=== Comparing base-images between repos ==="
echo "Files in opendatahub-io/notebooks base-images:"
fd -t f Dockerfile base-images/ | sort

echo -e "\n=== Getting file list from red-hat-data-services/notebooks ==="
gh api repos/red-hat-data-services/notebooks/git/trees/main?recursive=1 | \
  jq -r '.tree[] | select(.path | startswith("base-images/")) | select(.path | endswith("Dockerfile.cpu") or endswith("Dockerfile.cuda") or endswith("Dockerfile.rocm")) | .path' | sort

Length of output: 1530

---

@jiridanek, I've analyzed the codebase and the referenced PR. Here are my suggestions to update our base-images Dockerfiles to better match AIPCC images and reduce discrepancies:

Key Findings

The current base-images don't actually use AIPCC images as a base. Instead, they start from registry.access.redhat.com/ubi9/python-312 or quay.io/centos/centos:stream9 and try to approximate AIPCC by installing 213 SCL packages via the aipcc.sh script. This creates maintenance challenges, as shown by PR #1697 where missing packages (libcurl-devel, openssl-devel) caused build failures.

Recommendations

1. Consider Using Actual AIPCC Base Images (Major Change)

Instead of approximating AIPCC, consider using actual AIPCC images as the base:

# Option: Use actual AIPCC base image
FROM quay.io/aipcc/base-images/cpu:3.1 AS base
# Then only install ODH-specific additions

Pros: Eliminates the 213-package workaround, better alignment with RHEL AI
Cons: Requires access to AIPCC registry, less control over base packages

2. Update aipcc.sh with Missing Build Dependencies (Immediate Fix)

Based on PR #1697, add the missing packages that caused build failures:

# In base-images/utils/aipcc.sh, within install_scl_packages():
SCL_PACKAGES+=(
    # Existing packages...
    "libcurl-devel"      # Required for AWS C++ SDK (PyArrow with S3)
    "openssl-devel"      # Required for various builds
    # Add any other -devel packages discovered from build failures
)

3. Add Missing Compiler/Build Tools

From PR #1697's ppc64le/s390x fixes, ensure these are in aipcc.sh:

# Add to install_packages() function:
PKGS+=(
    # Build tools for compiled Python packages
    "gcc-c++" "gcc-gfortran"  # C++ and Fortran compilers
    "cmake" "make"             # Build systems
    "pybind11-devel"           # Python bindings
)

4. Improve c9s-python-3.12/Dockerfile.cpu Structure

The c9s variant should more explicitly document its purpose:

####################
# base             #
####################
FROM quay.io/centos/centos:stream9 AS base

ARG PYTHON_VERSION=3.12
ENV PYTHON=python${PYTHON_VERSION}

ARG VARIANT=cpu
ARG NAME=odh-base-image-cpu-py312-c9s
# Update description to clarify AIPCC compatibility goal
ARG SUMMARY="Open Data Hub Notebooks Base Image for ${VARIANT} with Python ${PYTHON_VERSION} (AIPCC-compatible)"
ARG DESCRIPTION="${SUMMARY} - Provides package set compatible with RHEL AI AIPCC base images"

# ... rest of Dockerfile

5. Document AIPCC Alignment Strategy

Create base-images/README.md to explain the approach:

# Base Images

## AIPCC Compatibility

Our base images aim to be compatible with Red Hat AI Platform Compute Core (AIPCC) base images
while supporting both OpenShift AI and Open Data Hub deployments.

### Current Approach
- Start from UBI9 Python 3.12 or CentOS Stream 9
- Install packages via `aipcc.sh` to match AIPCC package set
- Reference AIPCC image: `quay.io/aipcc/base-images/cpu:3.1`

### Known Gaps
- 213 SCL packages installed to fill gaps between UBI/CentOS and AIPCC
- See `base-images/cpu/c9s-python-3.12/README.md` for package diff generation

### Package Updates
When build failures occur due to missing packages:
1. Add to `base-images/utils/aipcc.sh` 
2. Document in this README
3. Consider if AIPCC upstream should include it

6. Consolidate AIPCC Wheel Index Workarounds

The jupyter/* Dockerfiles have scattered --extra-index-url https://pypi.org/simple workarounds. Consider:

# Create base-images/utils/aipcc-pip-install.sh
#!/usr/bin/env bash
# Helper for pip installs when AIPCC index lacks packages
set -Eeuxo pipefail

if [ "$TARGETARCH" = "ppc64le" ] || [ "$TARGETARCH" = "s390x" ]; then
    # AIPCC index often lacks dependencies for these architectures
    pip install --no-cache-dir --extra-index-url https://pypi.org/simple "$@"
else
    pip install --no-cache-dir "$@"
fi

7. Update CUDA/ROCm Base Images

Ensure CUDA and ROCm variants also run aipcc.sh if they don't already:

# base-images/cuda/12.6/ubi9-python-3.12/Dockerfile.cuda
FROM registry.access.redhat.com/ubi9/python-312:latest AS buildscripts
COPY base-images/utils/aipcc.sh /mnt/aipcc.sh

# In the actual build stage, before installing CUDA:
RUN \
--mount=from=buildscripts,source=/mnt,target=/mnt \
--mount=type=cache,sharing=locked,id=notebooks-dnf,target=/var/cache/dnf \
/bin/bash <<'EOF'
/mnt/aipcc.sh
EOF

8. Systematic Package Gap Analysis

Enhance the README process to track what's missing:

# Add to base-images/cpu/c9s-python-3.12/README.md
## Tracking AIPCC Package Differences

# Compare against latest AIPCC
podman run --rm --pull=always quay.io/sclorg/python-312-c9s:c9s rpm -qa | sort > scl.txt
podman run --rm --pull=always quay.io/aipcc/base-images/cpu:latest rpm -qa | sort > aipcc.txt
comm -23 scl.txt aipcc.txt | cut -d- -f1 | sort -u > gaps.txt

# Periodically review gaps.txt and update aipcc.sh accordingly

Priority Actions

1. High Priority: Add libcurl-devel and openssl-devel to aipcc.sh (fixes build failures)
2. Medium Priority: Document AIPCC alignment strategy in base-images/README.md
3. Medium Priority: Add build tool packages (gcc-c++, gcc-gfortran, etc.)
4. Low Priority: Consider switching to actual AIPCC base images (requires registry access discussion)

Would you like me to create a GitHub issue with these suggestions for tracking implementation?