How to cache poetry environment in GitLab CI/CD

Table of Contents

Introduction
#

By setting poetry config virtualenvs.in-project true, Poetry creates a .venv directory within the project to isolate the environment.

$ tree -L 1 -a
.
├── app/
├── dockerfiles/
├── .dockerignore
├── .gitlab-ci.yml
├── poetry.lock
├── pyproject.toml
├── README.md
└── .venv/

In GitLab CI/CD, caching this .venv directory can avoid reinstalling external libraries (pip) every time, thus speeding up the process.

Reusing .venv in different jobs (different directories) is not always guaranteed to work correctly. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/93367

1. Creating the Dockerfile
#

Since we don’t want to create a virtual environment inside the image, we only make Poetry commands available without running poetry install.

FROM python:3.12

ENV PYTHONUNBUFFERED=1
ENV PATH="/root/.local/bin:$PATH"

RUN curl -sSL https://install.python-poetry.org | python3 -

RUN poetry config virtualenvs.in-project true

You can also set this externally using the POETRY_VIRTUALENVS_IN_PROJECT environment variable. docs

2. Creating the `.gitlab-ci.yml`
#

Save the created image as $CI_REGISTRY_IMAGE/ci/python312:latest in GitLab’s container registry and use it in GitLab CI/CD.

By preparing a dedicated job prepare_cache to run poetry install, subsequent jobs can use the cached .venv.

If you need to use different .venv for testing multiple Python versions, you can change prefix: ci-python312 to separate them. For dynamic usage per branch, refer to the docs.

.python_cache:
  image: $CI_REGISTRY_IMAGE/ci/python312:latest
  cache:
    key:
      files:
        - poetry.lock
      prefix: ci-python312 # Want to change the scope of the cache -> https://docs.gitlab.com/ee/ci/caching/#common-use-cases-for-caches
    policy: pull
    paths:
      - .venv
      # - mypy_cache # Uncomment if you use mypy

prepare_cache:
  extends: .python_cache
  stage: cache
  cache:
    policy: pull-push
  script:
    - poetry install -v

small_test:
  extends: .python_cache
  stage: test
  script:
    - poetry run python -m pytest
  needs:
    - prepare_cache