Python 3rdparty Pattern

In general, we use the 3rdparty idiom to organize dependencies on code from outside the source tree. This document describes how to make this work for Python code.

Your Python code can pull in code written elsewhere. Pants fetches code via a library that uses pip-style specifications (name and version-range).

3rdparty/python

To keep all of your code depending on the same versions of third-party Python artifacts, you might use the idiom of keeping them in a directory tree under 3rdparty/python. If your organization has many such dependencies, you might arrange them in several directories: this can ease later "git detective work" when finding out who changed a version. (Pants itself doesn't have many Python dependencies; thus, we haven't split its 3rdparty tree into many directories.)

pip-style requirements.txt:

To define some third-party dependencies, use a python_requirements in your BUILD file and make a pip requirements.txt file in the same directory.

E.g, your 3rdparty/python/BUILD file might look like:

python_requirements(
  module_mapping={
    "ansicolors": ["colors"],
    "beautifulsoup4": ["bs4"],
    "packaging": ["pkgutil"],
    "python-Levenshtein": ["Levenshtein"],
    "PyYAML": ["yaml"],
    "setuptools": ["pkg_resources"],
  }
)

...with 3rdparty/python/requirements.txt like:

ansicolors==1.1.8
beautifulsoup4>=4.6.0,<4.7
coverage>=4.5,<4.6
dataclasses==0.6
docutils>=0.15,<0.17
fasteners==0.15.0

# The MyPy requirement should be maintained in lockstep with the requirement the Pants repo uses
# for the mypy task since it configures custom MyPy plugins. That requirement can be found via:
#
#   ./pants \
#     --backend-packages=pants.contrib.mypy \
#       options \
#         --output-format=json \
#         --scope=mypy \
#         --name=version \
#     | jq -r '."mypy.version".value'
#
mypy==0.780

Markdown==2.6.11
packaging==20.3

python_requirements defines a named target for each line in the requirements.txt line. For example, a line like ansicolors==1.0.2 in requirements.txt defines a target named ansicolors that pulls in ansicolors version 1.0.2.

python_requirement_library and python_requirement:

A BUILD file can also define requirements without a requirements.txt file. Set up a python_requirement_library with one or more python_requirements like:

python_requirement_library(
  name='beautifulsoup',
  requirements=[
    python_requirement(name='beautifulsoup',
                       requirement='BeautifulSoup==3.2.0'),
  ])

Your Code's BUILD File

In your code's BUILD file, introduce a dependency on the 3rdparty target:

python_library(
  dependencies=[
    '3rdparty/python:ansicolors',
  ],
)

Then in your Python code, you can import from that package:

from colors import green

Managing dependencies for multiple platforms

If you're building a python binary for use on multiple platforms, you might have 3rd-party dependencies that rely on platform-specific code. In addition to specifying the platforms with which your binary is intended to be compatible in the platforms field of your python_binary target, you will need to make wheel files for each package and platform available at build time.

Pants will use the explicitly specified platforms field of your python_binary target if set for both itself and its dependencies, or will otherwise fall back to the python-setup.platforms option value.

Pants will look for those files in the location specified in the python-repos field in pants.toml. It can understand either a simple local directory of .whl files or a "find links"-friendly webpage of links formatted like so:

<a href="x.whl">x.whl</a>

If you opt for the local directory method under version control, you may want to use git-lfs or similar to avoid storing large binaries in your repository. If you opt for a hosted solution, Github pages may be helpful.