Skip to content

Conversation

@adwiteeymauriya
Copy link

Summary

This PR introduces a complete Go implementation of a Docker image push CLI for the serverless-registry, specifically designed to handle large container images that exceed Cloudflare Workers' 500 MB request body limit.

Key Features

  • Chunked Upload Support: Automatically reads the oci-chunk-max-length header from the registry (default 470MB) to determine optimal chunk sizes, enabling uploads of arbitrarily large image layers
  • Concurrent Processing: Configurable concurrent uploads (default 5) using goroutines with semaphore pattern for faster performance
  • Layer Caching: Smart caching system that stores compressed layers with pointer files to avoid recompression on subsequent pushes
  • Retry Logic: Configurable retry attempts (default 3) for handling transient network failures
  • Authentication: Secure Basic authentication with password via stdin to prevent exposure in process lists or command history
  • Protocol Flexibility: Support for both HTTPS and HTTP (for local/dev registries) via --insecure flag

Implementation Details

The tool follows the complete OCI Distribution Spec workflow:

  1. Exports Docker image using docker save
  2. Extracts and compresses each layer to gzip format
  3. Calculates SHA256 digests for all blobs (layers + config)
  4. Uploads blobs in chunks respecting registry limits
  5. Constructs and uploads OCI-compliant manifest

Technical Highlights

  • Zero external dependencies except golang.org/x/term for secure password input
  • Uses Go's native net/http client and standard library for all operations
  • Type-safe implementation with proper error handling
  • Cross-platform binary support (no runtime required)
  • Memory-efficient streaming for large files

Usage

# Build the tool
cd push-go && go build -o docker-push

# Push an image
echo "password" | ./docker-push -username myuser my-registry.com/image:tag

# With custom settings
echo "password" | ./docker-push \
  -username myuser \
  -concurrency 10 \
  -max-retries 5 \
  my-registry.com/large-image:latest

Files Added

  • push-go/main.go - Complete implementation (699 lines)
  • push-go/README.md - Comprehensive documentation with examples
  • push-go/go.mod & push-go/go.sum - Go module definition
  • push-go/.gitignore - Ignore cache, build artifacts, and temporary files

Testing

Tested with:

  • Multi-GB PyTorch images with layers exceeding 500 MB
  • Concurrent uploads of 10+ layers
  • Retry logic under network disruption
  • Both authenticated and anonymous (local registry) scenarios

Test plan

  • Build the binary successfully
  • Push a small image (<100 MB) to verify basic functionality
  • Push a large image with layers >500 MB to test chunking
  • Verify layer deduplication works (push same image twice)
  • Test concurrent uploads with multiple layers
  • Verify retry logic handles transient failures
  • Test with both HTTP and HTTPS registries
  • Confirm authentication works correctly
  • Integration test with actual Cloudflare serverless-registry deployment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant