At prefix.dev we're building a cross-platform, system-level package and workflow manager called Pixi. We use Conda packages and recently added support for grabbing them straight from an S3 bucket with all the standard credential configuration. Vendor neutrality FTW!
We recently merged S3 support in rattler, pixi and rattler-build – all of our tools now support the most common cloud storage standard. The contribution was made by our friends at QuantCo, across multiple PRs and we were lucky to be joined by
where he explained all the details on how to use S3 support across our tools.
At prefix.dev we are excited about S3 support because we strongly believe in the versatility of Conda packages and having vendor agnostic ways of distributing Conda packages is a huge benefit to the ecosystem that will allow it to grow further and get more adoption. At the same time, we obviously hope to offer the superior experience on prefix.dev. Our goal is to make Conda packages as popular and widely used as Docker containers and it's an absolute must to have vendor-neutral options for hosting.
Info
Did you know – the original Conda package manager has supported the S3 protocol since a long time. You just need to install boto3 alongside Conda to enable s3:// channel URLs for Conda. Unfortunately there does not seem to be a lot of documentation for the feature.
The new S3 support in Pixi offers a compelling alternative with several advantages:
Open standard implemented by numerous cloud providers (AWS, Cloudflare R2, Backblaze B2, Hetzner Object Storage, etc.)
Built-in authentication through the S3 protocol
Reduced vendor lock-in due to wide compatibility
Bare-bones, universal solution that works in virtually any environment
We hope that this makes it easier for companies to decide to go all in on pixi & rattler-build.
How Conda Channels Work
A conda channel is essentially a collection of files hosted on a web server. It includes:
Platform-specific directories (like noarch) that contain all the packages
A repodata.json file containing metadata about packages, versions, and dependencies
Theoretically, even a "flat file server" can run a Conda channel - there is really nothing to it. Keeping the index up to date is the main challenge.
Setting Up S3 Support in Pixi
... and for your own packages. The implementation is straightforward.
1. Add the S3 bucket in your Pixi.toml
In your pixi.toml file you can simply use the s3:// protocol to select a S3 bucket as channel. By default, this will use the various ways of finding standard AWS credentials on your machine.
There are a number of additional configuration options that you can supply for broad compatibility with many different S3-storage vendors. All configuration options are listed in the extensive documentation.
2. Upload packages
You can use your favorite tool to upload or mirror packages to your S3 bucket. But you can also build a new package with rattler-build and upload it directly with rattler-build itself.
3. Generate the index with rattler-index
Uploading the package(s) was easy – but a working conda channel needs an index. For each subdir (platform) like linux-64, osx-arm64, or noarch, a repodata.json file needs to be created that contains all package file names, dependencies, SHA256 hashes, and so on. This information will be used by the solver to determine which versions to install, and what dependencies to fetch. There are a number of variations to this file – such as zstd compressed or "sharded" across a number of smaller files, and the indexing tool will take care of that.
The rattler-index standalone tool builds on top of the solid Rust foundations of rattler and offers a full-fledged conda package indexing workflow. The latest version supports sharded repodata and repodata patching, which makes it compatible with the latest conda standards. It's a newer alternative to conda-index.
You can install and run rattler-index from conda-forge using the following commands:
If you want to index packages with Python, we also have Python bindings to the functionality in rattler-index as part of py-rattler: Documentation
Watch the show & tell
Pavel presented this information live - and in color at our community show & tell. You can watch it on Youtube:
Come talk to us
The addition of S3 support to Pixi and Rattler represents a significant step forward in package distribution flexibility. Thanks to the ubiquitous S3 protocol, teams can now host conda packages in virtually any environment with robust authentication and minimal vendor lock-in.
# to install rattler-index with pixipixi global install rattler-index# or with condaconda install rattler-index -c conda-forge# and then run it usingrattler-index s3 s3://my-bucket/channel --endpoint-url=https://...
:focal(smart))
