This document provides a comprehensive reference on using the cports
system,
more specifically its cbuild
component.
Table of Contents
The cports
collection comes with a specialized build system, cbuild
. The
system provides a way for people to build their own binary packages from
special templates.
If you are looking for instructions on how to write templates, refer to the
Packaging.md
instead.
In order to get started with the system, your operating system environment must satisfy some requirements. After that, you can use it to build and manage packages, assuming you have bootstrapped the system.
TL;DR: You need a handful of tools, mainly Python and a few binaries mentioned
in the list below. You need a 3.8+ kernel with support for namespaces, including
user namespaces, and cgroups. You need to run as a regular user, and not in a
chroot
. At least 2GB of RAM per each CPU thread is recommended (all threads are
used by default).
The cbuild
tool has relatively few dependencies. You can usually find all of
them in any Linux distribution. Additionally, it imposes some requirements on
the Linux kernel you are running.
The userland dependencies are the following:
apk
(from apk-tools
)openssl
(key generation only; not needed otherwise)git
(optional; required for reproducibility)bwrap
(from bubblewrap
)tee
If running a Chimera system, these tools can all be installed with the
base-cbuild-host
metapackage.
You need a recent Git snapshot of apk-tools
at this point. It is your
responsibility to ensure that your apk
is new enough (cbuild
does some
rudimentary testing that it's 3.x) and compatible with cbuild
. Your best
bet is to use the same version as is packaged.
You also need Linux kernel 3.8 or newer, with namespaces and cgroups enabled. Notably the following options must be enabled:
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_CGROUPS=y
You can check for those with something like zgrep /proc/config.gz
or
alternatively grep /boot/config-$(uname -r)
.
Most distribution kernels should have the options enabled by default.
In addition to these, you must run the system under a normal user. Running as
the root
user will result in early failure.
The environment used to run cbuild
must not be a chroot
. Running inside
of a chroot
interferes with the sandbox/namespaces. If you really need to
use a custom root, you can use bwrap
to provide functionality equivament
to chroot
, as there is nothing preventing nesting namespaces. The command
would be something like the following:
$ bwrap --unshare-user --bind /path/to/my/root / --dev /dev --proc /proc --tmpfs /tmp /bin/sh
You will also want to ensure you have sufficient RAM available. The cbuild
system will by default use all CPU threads it can, unless you manually restrict
it.
If you satisfy all this, you should be good to go.
TL;DR: Packages are built in a sandboxed container with limited access to the outside environment. The system automatically manages a local repository for you, including package signing. Dependencies are installed in the sandbox, software is built, packages are created, and cleanup is performed. The system can build software recursively, so you can give it a metapackage to build and it will do the whole dependency tree.
If you are familiar with Void Linux's xbps-src
, the system should immediately
appear familiar to you. You should not consider it a clone, since it was written
from scratch in a completely different language and does a lot of things in a
different way, but you will notice a lot of similarities.
When building packages with cbuild
, the build process happens in a minimal
container. This is what you need namespaces for; they are the building blocks
of this container.
This container is made up of a minimal collection of Chimera packages, which provide the initial environment. We call it the build root. It is essentially a sandbox with different restrictions depending on the phase of the build.
Most of the time, the build root is:
When building a package, the following happens, in simplified terms:
destdir
. Outside of this, the directory
where files are extracted, the /tmp
directory in the sandbox and potential
cache directories (e.g. for ccache
), the entire sandbox is read only during
all steps after installing dependencies.If you are familiar with xbps-src
, these are the main conceptual differences:
cbuild
code is run outside the sandbox. Only specific commands are run
within, which includes dependency installation, sources extraction, patching,
and the build itself. Once files are installed, cbuild
handles the rest on
its own without involving the container. In contrast, xbps-src
will reexec
itself inside its sandbox and run everything there.xbps-src
you don't get any warranty that the build container is intact after anything
is run within. In contrast, cbuild
guarantees that the sandbox is exactly
the same before and after building something in it.cbuild
system has no concept of hostdir
, instead preferring fine
grained control over every directory.xbps-src
provides a "temporary build root" functionality, cbuild
does not. This is because doing so would introduce reliance on overlayfs
and a custom suid
binary. This would prevent us from sandboxing properly.
However, this functionality is not needed, since we guarantee consistency of
the sandbox at all times. For parallel building of several packages at once,
the -t
flag still exists. Instead of using an overlay, it will simply
bootstrap a fresh temporary root. Unlike with xbps-src
, this does not
create a performance problem, as everything is much faster.xbps-src
this would be a
potential security hazard, but cbuild
can guarantee no malicious process
can get access to your signing keys. That means repositories generated with
cbuild
are ready to be deployed in remote locations.You will need to generate a signing key. You can do that like this:
$ ./cbuild keygen
You can optionally pass your own private key name or path as an argument. If
you don't do that, it will be assigned an automatic name consisting of your
Git email address (or system username if not available) and a timestamp, plus
the .rsa
extension. Your public key will be stored in the same location,
but the extension will be .pub
.
An optional second argument can specify the key size, which is 2048 by default.
Keys are by default stored in etc/keys
.
Once generated, the tool will automatically update the configuration file you
have, which is etc/config.ini
by default, with the correct key path.
If you don't have a key generated and set, you will not be able to build
packages. It is possible to override this with --allow-unsigned
, but it
is not recommended to do that.
The easiest way to bring up a build container is from binary packages, like this:
$ ./cbuild binary-bootstrap
By default, this will be bldroot
inside your cports
directory. If you have
just done a source bootstrap, there is a chance you don't need to run this as
the source bootstrap does it for you as the last step. You will need to do this
if you ever need to re-create it.
Then, the only thing left to do is to pick a package to build. Let's say, awk
from the main
category. You need to run this:
$ ./cbuild pkg main/awk
This will parse main/awk/template.py
and build it according to the metadata
and routines declared in the template.
That's it!
By this, it is meant bootstrapping from source. This is an alternative to binary bootstrap, if you wish to compile the whole system from source. Keep in mind that this takes a long time, because it has to rebuild the whole bootstrap path 4 times.
Bootstrapping has more requirements than simply using the system.
The base requirements of cbuild
still apply. You also need to be running a
system based on the musl
C library. This can be for example Void Linux or
Chimera itself. Alpine Linux is not supported for direct bootstrapping because
of its patched musl SONAME (which would be more effort to work around) and
lack of libc++
in repositories.
The system must contain an initial toolchain. It consists of these:
clang
with lld
, libc++
, compiler-rt
and LLVM libunwind
cmake
meson
patch
pkg-config
(pkgconf
or the regular one)make
(called gmake
or make
)make
(called bmake
or make
)ninja
strip
byacc
or bison
(either with yacc
symlink)flex
perl
m4
These can all be found in most distributions' package collections. If running
a Chimera system, these tools can all be installed with the base-cbuild-bootstrap
metapackage.
It is possible to do an almost full source bootstrap on an incompatible system, provided that Chimera ships binary packages for the given architecture. See below for an example.
Chimera uses a 4-stage bootstrap process. It is largely automatic and hidden from you. You can invoke it like:
$ ./cbuild bootstrap
Optionally you can stop the process at a specific stage by passing its number
as an argument (not when using bootstrap.sh
).
To explain what's going on:
The initial stage is raw and intentionally stripped down. Its purpose is to get a minimal environment going, to free further builds of the host system's influence and narrow down the dependencies. This stage will likely not be reproducible between different systems.
Stage 1 resembles a final container. Unlike stage 0 build, it uses its own host tools. The feature set of the packages may not be complete, with some subpackages (e.g. LLVM debugger) not being built. LTO is also not applied for this stage yet.
Stage 2 is considered almost final, being built with all of the features of a final system within a Chimera container, including full LTO. Unit tests are not run yet as they are not considered reliable.
Stage 3 is the final stage, which is a clean rebuild of every bootstrap package using a "good" toolchain. There is no distinction from regular package builds (these are considered stage 3 as well) and unit tests and so on are run normally.
Templates should in general not make any distinction between stage 2 and 3 builds, as they are to be considered feature-equivalent.
You will have the following artifacts:
bldroot-stage0
is the build root that was assembled from packages originally
built on the host system.bldroot-stage1
is the build root assembled from stage 1 packages.bldroot-stage2
is the build root assembled from stage 2 packages.bldroot
is the final build root; if you remove it and binary-bootstrap
,
you will get the same thing.packages-stage0
is the repository of packages bldroot-stage0
is created
from.packages-stage1
is the repository of packages bldroot-stage1
is created
from.packages-stage2
is the repository of packages bldroot-stage2
is created
from.packages
is the final repository.sources
is the sources cache, shared for all.You can remove all the *-stage*
directories if you want. They are present
mostly for inspection and possibly debugging.
If the bootstrap fails at any point, you can start it again and it will continue where it left off. No things already built will be built again.
If you have an incompatible system and wish to do a source bootstrap, you can run most of the process provided that Chimera already has existing binary packages for the architecture. In this case, the host system requirements are identical to regular builds without source bootstrap.
This is done by pre-bootstrapping a stage 0 environment from binaries:
$ ./cbuild -b bldroot-stage0 binary-bootstrap
After that, you can run the bootstrap
command as usual. The stage 0 will be
skipped (but it's largely unnecessary due to the environment already being
a Chimera environment and not dependent on host toolchain) but every other
stage will build.
Every cbuild
action consists of the following:
$ ./cbuild [optional arguments] COMMAND [command arguments]
The order of reading settings is the following:
1) Optional arguments or command arguments 2) Configuration file 3) Default value
That is, if you pass a setting on the command line, it is always prefered. Otherwise, it is read from the configuration file. If this is not possible, the default value is used.
Optional arguments are global, separate from the command. However, some of them only have an effect with specific commands.
--allow-unsigned
Do not sign packages and allow building without a signing
key set up.
-A ARCH
, --host-arch ARCH
Override the host architecture. The given host
arch must be runnable on the current kernel. This is typically useful for e.g.
32-bit builds on 64-bit architectures, or for emulated targets. Note that once
a build root is bootstrapped, it decides the host architecture exclusively, so
this is mostly useful for actions that bootstrap a new root.-a ARCH
, --arch ARCH
Build for architecture ARCH
, possibly cross compiling.-b ROOT
, --build-root ROOT
(default: bldroot
) Set the path to the build
root to use.-B PATH
, --build-dir PATH
(default: empty) Set the path to the directory
where builds will happen. If not set, builddir
inside the build root will be
used as is. Otherwise, it will be bound to the given path (which will be created
if necessary).-c PATH
, --config PATH
(default: etc/config.ini
) The path to the config
file that cbuild
reads configuration data from. If relative, it is to cports.-C
, --skip-check
Never attempt to run the check
phase.-D
, --dirty-build
Skip installation of dependencies in the bldroot
,
as well as removal of automatic dependencies after successful build, and
do not clean the remains of a previous build of the template from builddir
and destdir
. This is mostly useful to continue previous failed builds.--dry-run
Do not perform any changes on the file system. This applies to only
specific commands, notably the prune-
commands.-f
, --force
Packages will be created and overwritten even if one already
exists in the local repository.-G
, --no-dbg
Do not build -dbg
packages.-j JOBS
, --jobs JOBS
(default: thread count) The number of build jobs to
use. By default uses the number of CPUs the cbuild run is restricted to (which
is usually the number of CPU threads you have). If you have insufficient RAM
(at least 2GB per thread is recommended), you will want to lower this. Setting
to 0 just uses the default.-K
, --keep-temporary
Keep temporary build files after a successful build,
this includes the builddir
and destdir
. If using a temporary build root,
it will not be removed.-L
, --no-color
Color output will be suppressed. By default color output
is used, unless NO_COLOR
is set in the environment or the output is being
redirected/piped.-N
, --no-remote
Never use remote repositories to fetch dependencies.-r REPO
, --repository-path REPO
(default: packages
) Set the path to the
local repository to build packages in.-R REPO
, --alt-repository REPO
(default: None) Create packages into an
alternative repository. This is a completely separate repository path. When
installing dependencies, both repositories are considered; when checking for
whether to build at all, only the alternative repository is considered. This
is useful for doing various quick tests and so on without messing up your
main repo, while still pulling build dependencies from the primary one.-s SOURCES
, --sources-path SOURCES
(default: sources
) Set the path to the
sources cache.--stage
Keep newly built packages staged. They will get unstaged either with
the next build or by explicitly doing so.--stage-path REPO
(default: pkgstage
) Packages are staged into a separate
location before being migrated into the primary repository. This separate location
mirrors the primary repository's layout. This allows one to "hide" changes until
they are ready, for example until all shlibs are properly bumped.-t
, --temporary
Create a temporary bldroot
for the build. The -b
argument
is used as a base path as well as the name prefix for the temporary root if
provided. The temporary root is removed at the end (whether the build succeeded
or failed) unless --keep-temporary
is passed.The following commands are recognized:
binary-bootstrap
Create a build root from local packages. The local
repository must be populated, or a sufficient remote repository must be
available.bootstrap [STAGE]
Bootstrap from source. If STAGE
is passed, stop at that
stage (number). By default, that is 2
. Stage 0 bootstrap must be run in a
compatible host system.bootstrap-update
Update the packages in your build root to latest.chroot
Enter the build root with an interactive shell. In this environment,
the root is mostly unsandboxed, i.e. writable and with network access. You
can use this kind of environment for quick testing, as well as entering failed
builds and inspecting them.clean
Clean up the build root. This means removing automatic dependencies
and removing builddir
and destdir
within.keygen [KEYPATH [KEYSIZE]]
Generate your signing key. You can optionally
specify the key name (if not a path, will be stored in the default location
of etc/keys
), key path, and key size (2048 by default). The configuration
file will automatically be updated. You can also pre-specify the key path
or name in the configuration file ahead of time, in which case it will use
those, unless overridden on the command line. The system will not overwrite
keys that already exist (i.e. if a valid key is specified in configuration,
this will fail).prune-obsolete
Prune obsolete packages within all repositories for the
current architecture (can be set with -a
). This works for recursively
searching for APKINDEX.tar.gz
within the repository path (-r
or default)
and using those paths as repositories.prune-removed
Prune removed packages within all repositories for the
current architecture (can be set with -a
). This works for recursively
searching for APKINDEX.tar.gz
within the repository path (-r
or default)
and using those paths as repositories. The affected repositories are
reindexed afterwards.relink-subpkgs
Recreate subpackage symlinks for a template. If not
given any arguments, it will do it for all available templates. Otherwise,
it will do it for the given template. Invalid symlinks will be deleted
when the global action is performed, otherwise symlinks will only be
created or replaced. For the global action, passing prune
as an
argument will result in the command also removing invalid directories
(not containing templates) and files.remove-autodeps
Remove automatic dependencies possibly installed in the
build root.zap
Remove the build root.lint
Read and parse the template, and do lint checks on it. Do nothing
else. Error on failures.cycle-check
Scan all templates or a single template for build-time
dependency cycles. Only one cycle at a time is printed. The goal is to
keep the tree free of cycles at all times. Therefore, if you encounter
a cycle, resolve it and check again.index
When not given a path, reindex all known repositories. When given
a path, reindex a specific repository. Only either the host architecture or
the -a
architecture are indexed, and the path should not include the
architecture.fetch
, extract
, prepare
, patch
, configure
, build
, check
,
install
, pkg
Given an argument of template path (category/name
) this
will invoke the build process for the given template up until the given phase.
The pkg
phase contains all of the others. For example, configure
will
invoke all of fetch
, extract
, prepare
, patch
and configure
phases
before stopping there. A complete pkg
will also take care of automatically
cleaning up afterwards, unless overridden. The build will not run if an up to
date version of the package already exists in the local repository, unless
overridden with -f
or --force
, when using the "pkg" target. Other
targets will run always unless already finished in builddir (you can
make them always run regardless by passing -f
or --force
).unstage
Attempt unstaging the repositories if possible. If conflicts
prevent it from doing so (i.e. missing rebuilds and so on) you will get
a warning instead, and nothing will happen.Most options can be specified in the configuration file as well. The system
reads etc/config.ini
by default (can be changed with -c
). It follows a
standard ini
format of Python configparser
.
There is a sample configuration file in etc/config.ini.example
. It contains
every option that can be specified, with its default value. You do not need
to specify every option in your own configuration file, this file is only
for reference.
The cbuild
system is fully capable of cross compiling. The same architecture
profile can be used for both native and cross builds, and in a lot of cases
the process can be entirely transparent.
Unlike native builds, cross builds are not capable of running the check
phase
so it is always skipped.
Cross compiling is nearly identical to compiling natively. You just need to do something like this:
$ ./cbuild -a aarch64 pkg main/zlib
The system will automatically take care of setting up an architecture sysroot
within the build root and preparing it for installing makedepends
. If the
necessary toolchain packages for the cross architecture do not exist, they are
built first. Cross sysroots are persistent, i.e. they are permanently set up
in your build root, but have the same guarantees as the rest of the root, so
once they are set up they should never get corrupt.
The builds will transparently use ccache
to speed things up if enabled. This
does not apply to bootstrap
, which never uses the cache.
You can enable this in your config.ini
, simply by setting ccache = yes
in
the build
section. The cache will be stored in the ccache
subdirectory of
the cbuild caches path (by default cbuild_cache
, see config.ini.example
for how to change it).
If you still need help, you should be able to get your answers in our
IRC channel (#chimera-linux
on irc.oftc.net
) or our Matrix channel
(#chimera-linux:matrix.org
). The two are linked, so use whichever
you prefer.