The Python community is set to adopt PEP 751, an enhancement aimed at improving dependency management and installation reproducibility.
PEP 751 introduces a standardised file format, pylock.toml, designed to create an immutable record of both direct and indirect dependencies for Python environments. Recently marked as ‘Accepted,’ the proposal promises to streamline workflows, enhance security, and foster greater interoperability between various Python packaging tools.
Addressing fragmentation and security
For years, Python developers have relied on various methods to manage project dependencies, often leading to inconsistencies and the infamous “it works on my machine” problem.
While solutions like pip freeze, pip-tools, PDM, Poetry, and uv have emerged – offering ways to ‘lock’ dependencies to specific versions – “currently, no standard exists to create an immutable record, such as a lock file, which specifies what direct and indirect dependencies should be installed into a virtual environment.”
This lack of standardisation presents several challenges. Tooling vendors face choices about which non-standard format to support, potentially excluding users. Developers find switching between tools costly due to incompatible lock file formats, leading to vendor lock-in and fractured tooling ecosystems.
Furthermore, security concerns linger, particularly regarding “the lack of secure defaults in the face of supply chain attacks by some tools (e.g. including hashes for files)”. While requirements.txt files are common, they are supported merely by convention, lack flexibility, and are not secure by default.
PEP 751, authored by Brett Cannon, directly addresses these issues by proposing a single, well-defined format. The goal is a format that is both “human-readable” for auditing and “machine-generated” for reliability. Installers using this format should determine precisely what to install without needing complex dependency resolution at install time, simplifying the process and likely speeding up installations.
Introducing pylock.toml
The proposed standard uses the TOML format for its lock files, promoting readability and easier syntax highlighting in editors. Files will be named pylock.toml by default, or pylock.{name}.toml if a specific name is needed or multiple lock files exist (e.g., pylock.production.toml).
Key features of the pylock.toml format:
- Human-readable and machine-generated: Designed for easy auditing by humans and reliable creation by tools (‘lockers’).
- No install-time resolution: Specifies exact dependencies, simplifying and speeding up the installation process handled by ‘installers’.
- Enhanced security: Promotes secure defaults, including mandatory file hashes. At least one secure hash (like SHA-256) is required for files referenced via archives, sdists, or wheels.
- Interoperability: Aims to replace most uses of requirements.txt as lock files, acting as a common export target for tools with internal formats, reducing vendor lock-in.
- Multi-use support: Supports defining dependencies for different scenarios (like ‘extras’ or ‘dependency groups’) within a single file using enhanced environment markers, reducing the need for multiple lock files.
- Tooling agnostic installation: Any installer implementing the specification can install from a pylock.toml file, regardless of the tool used to create it.
- Detailed metadata: Records essential information like the creating tool, required Python version, compatible environments, package versions, sources (VCS, directory, archive, index), dependencies (for informational purposes), file sizes, upload times, and attestations.
The specification mandates a lock-version (initially “1.0”) and the name of the created-by tool. It includes detailed sections for each package specifying its name, version (where stable), environment markers, Python version requirements, and source details.
Sources can be Version Control Systems, local directories, direct archive URLs, sdists, or wheels. Crucially, wheel support is mandatory for tools.
A significant addition is the extension of environment marker syntax specifically for lock files, introducing extras and dependency_groups markers to precisely control installation based on requested optional dependencies.
Installing from a pylock.toml file follows a defined process:
- Determine the extras and dependency groups required.
- Validate the lock file version, overall Python requirements, and environment compatibility.
- Iterate through packages, checking markers and Python compatibility for each.
- Ensure no conflicting versions of the same package are selected.
- Verify the package source details are unambiguous.
- For each package slated for installation, fetch/clone the source, validate hashes/sizes where applicable, build if necessary (for sdists, VCS, directories, archives), and install. Installers must validate hashes and should validate sizes when provided.
What PEP 751 means for Python developers
The adoption of PEP 751 signals a move towards more robust, reliable, and secure dependency management in the Python ecosystem.
Developers can expect:
- Improved reproducibility: Lock files ensure consistent environments across different machines and collaborators.
- Enhanced security: Standardised inclusion of hashes makes validating dependencies easier.
- Greater tool choice: Reduced friction when switching between packaging tools that support the standard.
- Simplified auditing: The human-readable format and inclusion of metadata facilitate easier review of dependencies.
While PEP 751 promotes a “security-first posture” by mandating hashes and recording provenance details like attestations, it acknowledges it doesn’t solve all security issues.
Tampering with lock files remains a risk if they aren’t audited or managed securely (e.g., via source control). Signing lock files is suggested as a mitigation. The PEP also notes it doesn’t prevent issues like typosquatting, although tooling UX could potentially help.
While existing tools will need to decide whether and how to adopt PEP 751 (either as an export target or primary format,) its acceptance addresses long-standing community needs for a standardised way to lock dependencies—paving the way for more cohesive and secure packaging.
See also: KAI Scheduler: NVIDIA open-sources Kubernetes GPU scheduler
Want to learn more about cybersecurity and the cloud from industry leaders? Check out Cyber Security & Cloud Expo taking place in Amsterdam, California, and London. The comprehensive event is co-located with other leading events including Digital Transformation Week, IoT Tech Expo, Blockchain Expo, and AI & Big Data Expo.
Explore other upcoming enterprise technology events and webinars powered by TechForge here.
