Skip to content

Commit 158e3ec

Browse files
humitoshumitos
authored
Upgrade to Ubuntu 24.04 for our application (#12002)
- Use Ubuntu 24.04 for our Docker images - The application runs with Python 3.12 now - Run tests with Python 3.12 - Update APT package names - Install `uv` in the default directory `~/.local/bin` - Add `uv` to the PATH - Create a virtualenv inside the Docker image for our application. `--system` option does not work anymore with the newer `pip` versions. It fails saying that we should install Python dependencies using system package. - Upgrade Celery dependency to be compatible with Python 3.12 - The `pickle` error we used to see with Celery does not happen anymore with the newer version + py3.12 - Remove `tzdata` because it's installed automatically now ## ToDo - [x] tests pass locally - [x] build a failed build that raises `ConfigError` without breaking - [x] build a success build - [x] tests pass in CircleCI - [x] update -ops to use Ubuntu 24.04 - [x] build an AMI - [x] deploy _one_ instance and run a build there - [ ] do the same upgrade on .com ## Notes - All the Python commands inside the Docker container needs to be prefixed with `uv run` now. Example: `uv run tox -e 312` or `uv run django-admin` and similars. Related readthedocs/readthedocs-ops#1496 Closes #12000
1 parent dc6f2ef commit 158e3ec

File tree

13 files changed

+68
-104
lines changed

13 files changed

+68
-104
lines changed

‎.circleci/config.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ orbs:
77
jobs:
88
tests:
99
docker:
10-
- image: 'cimg/python:3.10'
10+
- image: 'cimg/python:3.12'
1111
environment:
1212
# Don't skip search tests.
1313
TOX_POSARGS: ''
@@ -27,12 +27,12 @@ jobs:
2727
- run: sudo apt update
2828
- run: sudo apt install -y rclone
2929
- run: pip install --user tox
30-
- run: tox -e py310
30+
- run: tox -e py312
3131
- codecov/upload
3232

3333
tests-ext-theme:
3434
docker:
35-
- image: 'cimg/python:3.10'
35+
- image: 'cimg/python:3.12'
3636
environment:
3737
# Don't skip search tests.
3838
TOX_POSARGS: ''
@@ -55,7 +55,7 @@ jobs:
5555

5656
tests-embedapi:
5757
docker:
58-
- image: 'cimg/python:3.10'
58+
- image: 'cimg/python:3.12'
5959
steps:
6060
- checkout
6161
- run: git submodule sync
@@ -66,7 +66,7 @@ jobs:
6666

6767
checks:
6868
docker:
69-
- image: 'cimg/python:3.10'
69+
- image: 'cimg/python:3.12'
7070
steps:
7171
- checkout
7272
- run: git submodule sync

‎common‎

‎dockerfiles/Dockerfile‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM ubuntu:22.04
1+
FROM ubuntu:24.04
22

33
ARG GITHUB_USER
44
ARG GITHUB_TOKEN
@@ -26,8 +26,8 @@ RUN apt-get -y install \
2626
libmysqlclient-dev \
2727
libfreetype6 \
2828
libjpeg-dev \
29-
sqlite \
30-
netcat \
29+
sqlite3 \
30+
netcat-traditional \
3131
telnet \
3232
lsb-release \
3333
npm \
@@ -57,25 +57,30 @@ RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
5757
locale-gen
5858

5959
# Install ``uv`` package manager
60-
RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/bin/" sh
60+
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
61+
ENV PATH="/root/.local/bin/:$PATH"
62+
RUN uv venv
6163

6264
# Ensure that ``python`` is in the PATH so that ``./manage.py`` works
6365
RUN ln -s /usr/bin/python3 /usr/bin/python
6466

6567
WORKDIR /tmp
6668

6769
COPY requirements/docker.txt docker.txt
68-
RUN uv pip sync --system docker.txt
70+
RUN uv pip sync docker.txt
6971

7072
# Install readthedocs-ext only if GITHUB_TOKEN is provided
7173
WORKDIR /usr/src/app/checkouts/
7274
RUN if [ -n "$GITHUB_TOKEN" ] ; \
7375
then \
7476
git clone --depth 1 https://${GITHUB_USER}:${GITHUB_TOKEN}@github.com/readthedocs/readthedocs-ext ; \
75-
uv pip install --system --no-cache-dir -e readthedocs-ext ; \
77+
uv pip install --no-cache-dir -e readthedocs-ext ; \
7678
fi
7779

7880
RUN git clone --depth 1 https://github.com/readthedocs/ext-theme ; \
79-
uv pip install --system --no-cache-dir -e ext-theme
81+
uv pip install --no-cache-dir -e ext-theme
82+
83+
# Required by polymorphic
84+
RUN uv pip install --no-cache-dir -U setuptools
8085

8186
WORKDIR /usr/src/app/checkouts/readthedocs.org

‎docs/dev/install.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ save some work while typing docker compose commands. This section explains these
180180
``inv docker.test``
181181
Runs all the test suites inside the container.
182182

183-
* ``--arguments`` will pass arguments to Tox command (e.g. ``--arguments "-e py310 -- -k test_api"``)
183+
* ``--arguments`` will pass arguments to Tox command (e.g. ``--arguments "-e py312 -- -k test_api"``)
184184

185185
``inv docker.pull``
186186
Downloads and tags all the Docker images required for builders.

‎docs/dev/tests.rst‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -46,22 +46,22 @@ To target a specific environment:
4646

4747
.. prompt:: bash
4848

49-
tox -e py310
49+
tox -e py312
5050

5151
To run a subset of tests:
5252

5353
.. prompt:: bash
5454

55-
tox -e py310 -- -k test_celery
55+
tox -e py312 -- -k test_celery
5656

5757
The ``tox`` configuration has the following environments configured. You can
5858
target a single environment to limit the test suite:
5959

60-
py310
60+
py312
6161
Run our test suite using Python 3.10
6262

63-
py310-debug
64-
Same as ``py310``, but there are some useful debugging tools available in the environment.
63+
py312-debug
64+
Same as ``py312``, but there are some useful debugging tools available in the environment.
6565

6666
lint
6767
Run code linting using `Prospector`_. This currently runs `pylint`_,

‎requirements/deploy.txt‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -24,22 +24,22 @@ async-timeout==5.0.1
2424
# via
2525
# -r requirements/pip.txt
2626
# redis
27-
billiard==3.6.4.0
27+
billiard==4.2.1
2828
# via
2929
# -r requirements/pip.txt
3030
# celery
31-
boto3==1.36.21
31+
boto3==1.36.22
3232
# via
3333
# -r requirements/pip.txt
3434
# django-storages
35-
botocore==1.36.21
35+
botocore==1.36.22
3636
# via
3737
# -r requirements/pip.txt
3838
# boto3
3939
# s3transfer
4040
bumpver==2024.1130
4141
# via -r requirements/pip.txt
42-
celery==5.2.7
42+
celery==5.4.0
4343
# via
4444
# -r requirements/pip.txt
4545
# django-celery-beat
@@ -339,16 +339,15 @@ python-dateutil==2.9.0.post0
339339
# via
340340
# -r requirements/pip.txt
341341
# botocore
342+
# celery
342343
# elasticsearch-dsl
343344
# python-crontab
344345
python3-saml==1.16.0
345346
# via
346347
# -r requirements/pip.txt
347348
# django-allauth
348349
pytz==2025.1
349-
# via
350-
# -r requirements/pip.txt
351-
# celery
350+
# via -r requirements/pip.txt
352351
pyyaml==6.0.2
353352
# via -r requirements/pip.txt
354353
qrcode==8.0
@@ -382,9 +381,9 @@ s3transfer==0.11.2
382381
# via
383382
# -r requirements/pip.txt
384383
# boto3
385-
selectolax==0.3.27
384+
selectolax==0.3.28
386385
# via -r requirements/pip.txt
387-
sentry-sdk==2.21.0
386+
sentry-sdk==2.22.0
388387
# via structlog-sentry
389388
six==1.17.0
390389
# via
@@ -438,6 +437,7 @@ typing-extensions==4.12.2
438437
tzdata==2025.1
439438
# via
440439
# -r requirements/pip.txt
440+
# celery
441441
# django-celery-beat
442442
# kombu
443443
ua-parser==1.0.1

‎requirements/docker.txt‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,15 @@ async-timeout==5.0.1
2626
# redis
2727
attrs==25.1.0
2828
# via wmctrl
29-
billiard==3.6.4.0
29+
billiard==4.2.1
3030
# via
3131
# -r requirements/pip.txt
3232
# celery
33-
boto3==1.36.21
33+
boto3==1.36.22
3434
# via
3535
# -r requirements/pip.txt
3636
# django-storages
37-
botocore==1.36.21
37+
botocore==1.36.22
3838
# via
3939
# -r requirements/pip.txt
4040
# boto3
@@ -43,7 +43,7 @@ bumpver==2024.1130
4343
# via -r requirements/pip.txt
4444
cachetools==5.5.1
4545
# via tox
46-
celery==5.2.7
46+
celery==5.4.0
4747
# via
4848
# -r requirements/pip.txt
4949
# django-celery-beat
@@ -369,16 +369,15 @@ python-dateutil==2.9.0.post0
369369
# via
370370
# -r requirements/pip.txt
371371
# botocore
372+
# celery
372373
# elasticsearch-dsl
373374
# python-crontab
374375
python3-saml==1.16.0
375376
# via
376377
# -r requirements/pip.txt
377378
# django-allauth
378379
pytz==2025.1
379-
# via
380-
# -r requirements/pip.txt
381-
# celery
380+
# via -r requirements/pip.txt
382381
pyyaml==6.0.2
383382
# via -r requirements/pip.txt
384383
qrcode==8.0
@@ -414,7 +413,7 @@ s3transfer==0.11.2
414413
# via
415414
# -r requirements/pip.txt
416415
# boto3
417-
selectolax==0.3.27
416+
selectolax==0.3.28
418417
# via -r requirements/pip.txt
419418
six==1.17.0
420419
# via
@@ -472,6 +471,7 @@ typing-extensions==4.12.2
472471
tzdata==2025.1
473472
# via
474473
# -r requirements/pip.txt
474+
# celery
475475
# django-celery-beat
476476
# kombu
477477
ua-parser==1.0.1

‎requirements/docs.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@ uvicorn==0.34.0
175175
# via sphinx-autobuild
176176
watchfiles==1.0.4
177177
# via sphinx-autobuild
178-
websockets==14.2
178+
websockets==15.0
179179
# via sphinx-autobuild
180180

181181
# The following packages are considered to be unsafe in a requirements file:

‎requirements/pip.in‎

Lines changed: 1 addition & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -50,41 +50,9 @@ dnspython
5050
# Used for Redis cache Django backend (`django.core.cache.backends.redis.RedisCache`)
5151
redis
5252

53-
# NOTE: we have to pin to this version because Celery>=5.3.x detects our ``ConfigError`` exception is not picklable
54-
# and creates a ``UnpickleableExceptionWrapper`` which is not what our code expects.
55-
# This may be due to a Python bug which can't pickle/unpickle the exception properly.
56-
# We could consider either:
57-
# - handle ``UnpickleableExceptionWrapper`` in our code and inspect what's the inner exception
58-
# - find a way to disable pickling the exceptions
59-
# - make our custom exception picklable
60-
#
61-
# References:
62-
# - https://github.com/celery/celery/pull/8149
63-
# - https://github.com/celery/celery/blob/2b4b500ca1212016824a5fa2996cfb752f0763a7/celery/utils/serialization.py#L154
64-
# - https://github.com/python/cpython/issues/76877
65-
#
66-
# Reproducible example:
67-
# >>> import pickle
68-
# >>> class MyException(Exception):
69-
# ... def __init__(self, a, b):
70-
# ... self.a = a
71-
# ... self.b = b
72-
# ... super().__init__(a)
73-
# ...
74-
# >>> pickle.loads(pickle.dumps(MyException("a", "b")))
75-
# Traceback (most recent call last):
76-
# File "<stdin>", line 1, in <module>
77-
# TypeError: MyException.__init__() missing 1 required positional argument: 'b'
78-
# >>>
79-
celery==5.2.7
53+
celery
8054
django-celery-beat
8155

82-
# Docker images need a dependency explicity marked for tzdata in order to have
83-
# necessary timezone data.
84-
# https://github.com/readthedocs/readthedocs.org/issues/10453
85-
# TODO: remove this dependency once we upgrade Celery. It should auto-install it.
86-
tzdata
87-
8856
django-allauth[socialaccount,saml,mfa]
8957

9058
requests-oauthlib

‎requirements/pip.txt‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,17 +15,17 @@ asgiref==3.8.1
1515
# django-cors-headers
1616
async-timeout==5.0.1
1717
# via redis
18-
billiard==3.6.4.0
18+
billiard==4.2.1
1919
# via celery
20-
boto3==1.36.21
20+
boto3==1.36.22
2121
# via django-storages
22-
botocore==1.36.21
22+
botocore==1.36.22
2323
# via
2424
# boto3
2525
# s3transfer
2626
bumpver==2024.1130
2727
# via -r requirements/pip.in
28-
celery==5.2.7
28+
celery==5.4.0
2929
# via
3030
# -r requirements/pip.in
3131
# django-celery-beat
@@ -241,14 +241,13 @@ python-dateutil==2.9.0.post0
241241
# via
242242
# -r requirements/pip.in
243243
# botocore
244+
# celery
244245
# elasticsearch-dsl
245246
# python-crontab
246247
python3-saml==1.16.0
247248
# via django-allauth
248249
pytz==2025.1
249-
# via
250-
# -r requirements/pip.in
251-
# celery
250+
# via -r requirements/pip.in
252251
pyyaml==6.0.2
253252
# via -r requirements/pip.in
254253
qrcode==8.0
@@ -278,7 +277,7 @@ rest-framework-generic-relations==2.2.0
278277
# via -r requirements/pip.in
279278
s3transfer==0.11.2
280279
# via boto3
281-
selectolax==0.3.27
280+
selectolax==0.3.28
282281
# via -r requirements/pip.in
283282
six==1.17.0
284283
# via
@@ -315,6 +314,7 @@ typing-extensions==4.12.2
315314
tzdata==2025.1
316315
# via
317316
# -r requirements/pip.in
317+
# celery
318318
# django-celery-beat
319319
# kombu
320320
ua-parser==1.0.1

0 commit comments

Comments
 (0)