gnu-guile-gnu-guix-packaging-guide/guide.org

About

This document is supposed to be a guide for packaging GNU Guile software for GNU Guix. The process of packaging depends on the GNU Guile software and its dependencies.

Prerequisites

Creating a GNU Guix environment

For offering a reproducible experience of running your code can create a GNU Guix environment as described in the following.

Environment files

channels.scm

mkdir guix-env # in your project root directory pushd guix-env guix describe --format=channels channels.scm popd

The channels.scm file at guix-env/channels.scm looks something like the following:

(list (channel (name 'guix) (url "https://git.savannah.gnu.org/git/guix.git") (commit "df0447be61") (introduction (make-channel-introduction "9edb3f66fd" (openpgp-fingerprint "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA")))))

It lists the exact commit of the repository at ~https://git.savannah.gnu.org/git/guix.git~, which the environment is based on. Unless someone retroactively changes the specified commit, which is best practice not to ever do, this makes sure, that the packages your project depends on will be exactly the same, every time you use the GNU Guix environment.

Environment activation script

Once you have this, add the following GNU Bash script at ~guix-env/env.sh~:

#!/usr/bin/env bash

set -Eeuxo pipefail

DIR=$(dirname "$(readlink -f "${0}")")

guix time-machine \ --channels="${DIR}/channels.scm" -- \ environment \ --manifest="${DIR}/manifest.scm"

Here we are making use of the time-machine facility of GNU Guix, which allows us to specify a channels file, which contains the reference to the exact commit of the GNU Guix repository. Using that commit, we further make use of the ~environment~ facility, which allows us to install dependencies specified inside manifest file.

There are even more strict ways to create an environment, which also exclude environment variables set in the current shell, but for now we will leave it at this.

manifest.scm

Next create a manifest.scm file at guix-env/manifest.scm, which specifies GNU Guix packages, which your project depends on. It looks something like the following example:

(specifications->manifest '("guile" "guile-hall"))

In this example case the GNU Guile project requires:

• guile, which is GNU Guile, currently in version 3.x.y
• guile-hall, which helps with packaging GNU Guile projects -- Only include
this dependency, if you are trying to package your project using guile-hall.

The manifest.scm file in this case contains runtime and development dependencies. Note, that we do not need to specify versions in this file, as they are already indirectly specified by specifying the GNU Guix commit inside the channels.scm file.

Updating the GNU Guix environment

If we ever want to update our GNU Guix environment, to work with newer package versions, we can update our GNU Guix profile as follows:

guix pull && guix package -u

This will bring the GNU Guix profile to the current commit of the original GNU Guix repository.

Then we can run the describe command again:

guix describe --format=channels guix-env/channels.scm

This will write the new commit id into the channels.scm file. Then we must test our new GNU Guix environment by trying to activate it using:

guix time-machine \ --channels="guix-env/channels.scm" -- \ environment \ --manifest="guix-env/manifest.scm"

It could happen, that packages from our manifest.scm file are no longer available on a newer GNU Guix commit and that the environment creation fails. Therefore it is important to run this test before committing the new environment files to version control.

Activate the GNU Guix environment

Finally you will be able to use the environment, which was prepared here by running the following command:

bash guix-env/env.sh

This will drop you into a new shell, which has the packages specified in ~manifest.scm~ installed.

Why make a project specific environment?

"Why make a project specific environment, and not install the packages into the default profile?" you may ask. The reasons are the following:

1. It helps to keep the default profile thin. When we update your default
profile, we probably do not want to wait for all the packages of all the projects we have to update. Instead we can update those project specific packages in their own environment specifications, inside those projects.

1. This approach also helps us to avoid installing development packages like
~guile-hall~ into the global scope of the default GNU Guix profile.

1. It helps us to keep results reproducible. The less interference we get from
other contexts, especially packages in a global scope like the GNU Guix default profile, the closer we get to a completely reproducible result.

Using guile-hall

Introduction

This guide walks the reader through creating a GNU Guix package from a not yet packaged GNU Guile project using the tool guile-hall. Follow the steps in the guide from top to bottom to create a package. For a short refresh of what needs to be done to create a package, the TLDR section could be helpful.

Run hall init

guile-hall is specified inside the manifest.scm file. After activating the environment specified in guix-env, the hall command should be available.

guile-hall works in 2 modes. One is creating a new hall project, to which one then adds more and more files, developing the actual project. Such a project should be a valid GNU Guix package. The second mode is to convert an existing project into a hall project, making it a valid GNU Guix package.

This guide will focus on converting an existing project into a GNU Guix package using hall.

Running

The next step is to actually run hall as follows:

hall init \ --convert \ --author 'AUTHOR NAME' \ --prefix 'guile' \ --license='LICENSE_NAME' \ 'PROJECT_NAME' \ --execute

With this we tell hall to do the following things:

• ~init~: Initialize a new hall project.

• ~--convert~: Tell hall that we are transforming an existing project into a
hall project, instead of creating a project from scratch.

• ~--author~: Ascribe authorship of the project to the specified name.

• ~--prefix~: Define a prefix, which the package name shall have in GNU
Guix. This should be ~guile~ by convention, for GNU Guile packages.

• --license~: Tell hall to use the specified license to create the ~COPYING
file, in which it will put the license text. Examples for valid values are: ~gpl3+~ or ~agpl3+~. A complete list of valid license names can be found by browsing the GNU Guix sources at [[https://git.savannah.gnu.org/cgit/guix.git/tree/guix/licenses.scm]].

• project name: For this you need to insert the name of your project. Often a
GNU Guile project is named something like ~guile-hall~. In this case, leave the prefix ~guile-~ away, because it is already specified using the ~--prefix~ argument. hall will assume, that the source code of the project is in a folder, that has the project name as folder name. If that is the case, it will discover the source code files within it.

• ~--execute~: Without this flag being set, hall would merely preview the
changes to the project, which it would perform. Generally this is true for all hall commands. The show a preview, if you do not add ~--execute~. This is called a "dry run". It can be very helpful to avoid mistakes.

Result

guile-hall will create the following files:

• directory with the name of the project

• ~COPYING~: This file by GNU convention contains the license of a project,
depending on the command line argument.

Note: If your project used a different file for the license, for example LICENSE, it is advisable to change this and use the COPYING file from now on, because hall will generate the license in COPYING, when running some commands, when it does not find the license there and will also use the COPYING file for building the distribution tarball.

• ~HACKING~: Also seems to be a GNU convention. Contains information for other
developers about how to make changes to the project. hall will create a dummy file, which contains general information matching the structure created by hall. You might want to edit it later.

• README~: Merely a symlink to ~README.org.

• ~README.org~: org-mode file containing more information about the project and
its usage.

• ~doc/PROJECT_NAME.texi~: It seems to be convention in GNU projects to use the
Texinfo file format for software project documentation. Texinfo files have the file extension ~texi~. The Texinfo format is a set of macros for Tex, which simplify writing technical documentation.

• scripts directory and files therein: In this directory hall organizes
executable scripts, which are included in the distribution tarball. This can be useful, if the project, which is being packaged aims to provide a command line interface when installed via GNU Guix.

If the project does not have need for such a command line interface, for example in case of a GNU Guile library, which will only be used inside other GNU Guile programs, then this directory is not needed and can be removed. If removed, hall.scm should be adapted correspondingly by changing the programs category of the files attribute. HACKING might need to be adapted as well, to avoid having misleading information in it.

• guix.scm~: See [[#using-guile-hall-guix-package-file-guix-scm][Guix package file ~guix.scm]]

• ~hall.scm~: Contains information for hall. Using this information hall
generates various files, which are used for building the project using the GNU build system. This is the file that will be managed by hall and potentially you, in order to control, how the project is build. It contains some of the same configuration attributes that ~guix.scm~ contains and probably delegates them to the ~guix.scm~ file. For more information see [[#using-guile-hall-hall-project-definition-file-hall-scm][Hall project definition file ~hall.scm~]].

Guix package file guix.scm

This file contains the actual specification of your GNU Guix package. It contains information about:

• the name of the package

• the version of the package

• the tarball, which will be created when a distribution is made and which
will contain the sources GNU Guix needs to install the package on a system

• the specification of which build system shall be used to install the
package. Using hall the build system will be the GNU build system, which relates to GNU autotools, GNU Make and others.

• specification of dependencies

• ~inputs~: Inputs are dependencies, which are build for the target
architecture. This includes the runtime dependencies of the project. Put libraries the project depends on at runtime here.

• ~native-inputs~: Seem to be all tools, which are required to build the
project contained in the distribution source tarball.

• propagated-inputs See the package reference of GNU Guix for an
explanation.

• a synopsis describing the project

• a description, describing the project

• a URL to the home page of the project, if such exists, otherwise empty
string

• the license name

More information about the attributes can be found in the package reference of GNU Guix.

Hall project definition file hall.scm

In this section, we take a closer look at what we find inside hall.scm. With this knowledge it should be possible to edit the hall.scm file by hand, if that is needed.

There are some attributes, which we have already seen in guix.scm like name~ and ~prefix, although now separately, version, synopsis, description, ~home-page~ and license. We will look at the other attributes.

All attributes are under a top level expression ~hall-description~:

• ~author~: This one is simple: The name of the author of the package.
• ~copyright~: Seems to constitute some copyright notice together with the
author name.
• ~dependencies~: Contains a list of dependencies. (TODO: Give an example.)
• ~files~: This one is the most important one. It tells hall, which files to
include in the distribution tarball, which enables hall to make a working distribution.

Files

There is a definition files inside hall.scm for all source code files of a project. files contains a multi-element nested definition of the files and directories of the project. Some of the nested definitions can be scheme-file, ~directory~, org-file, symlink, text-file and probably others (TODO: Where to find a list of all supported file types?). The files are sorted into separate categories.

The actual files are then specified using either directory as a container, or one of the supported file types.

• scheme-file for example, like the other *-file specifications, expects a
string as argument, which is the filename of the Scheme file, without the file extension.

• directory expects a string as argument, which is the directory name, and
can recursively contain more directories or Scheme files. So any directory structure can be specified using ~scheme-file~ and ~directory~.

File categories

* Libraries

Indicated by libraries. Contains Scheme a nested list of files and directories indicated by scheme-file or directory and recursively their files or directories.

* Tests

Indicated by tests. Contains a nested list of directories and files, which contain tests.

hall will assume, that the tests are in the subdirectory tests. If the tests are not in that directory, the attribute needs to be changed accordingly.

* Programs

Indicated by programs.

TODO: What are programs in this context?

* Documentation

Indicated by documentation. Contains a nested list of the files or directories, which contain the documentation of the package.

The following files and directories go here:

• readme file and symlink to readme file
• to-do files
• notes files
• HACKING
• COPYING
• AUTHORS
• doc directory
• texi files of the documentation

* Infrastructure

Indicated by infrastructure. This category contains the following kinds of files:

• The GNU Guix package file guix.scm.
• Version control system specific files like ~.gitignore~ for git.
• The hall project file hall.scm.

Example

Here is an example for a hall.scm file:

(hall-description (name "fslib") (prefix "guile") (version "0.2.0") (author "Zelphir Kaltstahl") (copyright (2021)) (synopsis "file system utils for GNU Guile") (description "This library provides procedures for working with GNU/Linux file systems from GNU Guile.") (home-page "https://notabug.org/ZelphirKaltstahl/guile-fslib") (license agpl3+) (dependencies `()) (files (libraries ((scheme-file "fslib") (scheme-file "logging") (scheme-file "list-utils") (scheme-file "file-system") (scheme-file "file-reader") (scheme-file "string-utils"))) (tests ((directory "test" ;; hall automatically knows to add .scm to scheme files ((scheme-file "test-string-utils") (scheme-file "test-list-utils") (scheme-file "test-fslib"))))) (programs ()) (documentation ((org-file "todo") (org-file "README") (symlink "README" "README.org") (text-file "HACKING") (text-file "COPYING") (text-file "AUTHORS") (directory "doc" ((texi-file "fslib"))))) (infrastructure ((scheme-file "guix") (text-file ".gitignore") (scheme-file "hall")))))

Correction of files

hall generated some documentation in HACKING and inside the doc folder. They should be adapted to the individual project.

hall also generated a script in the scripts folder, which might not be needed. For example if the project that is being packaged is a library, then it does not need to have an executable script in the distributed tarball. If the project does not need scripts like that, the scripts folder can be removed and removed from the hall.scm. There are also parts inside the HACKING file, which should be adapted, in case the scripts directory and its contents are not needed.

Correct version

hall by default specifies version "0.1" for the version attribute in ~guix.scm~ and in hall.scm, as it would for a new and not converted project. However, if we are converting a project, this version specification might not be correct. It should be adapted to the correct version.

With the version attribute being updated, we also need to change the source~ in ~guix.scm attribute, to point to the correct distribution tarball filename.

Correct tests location

Depending on the projects original structure, the location of tests, which is specified in hall.scm needs to be changed to the correct directory.

Add GNU Guile files

hall might not have added all GNU Guile files of the project automatically in ~hall.scm~. It depends on where the files are. hall seems to find all files in the subdirectory which goes by the project name. However, if the files are on the top level of the project, hall will not pick up those files. In such cases, when hall does not pick up the files, they need to be added to hall.scm~ manually. See the ~hall.scm example above.

The usual way to add files to hall.scm, the configuration file hall makes use of, is to run a scan for not yet added files as follows:

hall scan

Check, whether the dry run output contains the modifications you wish and, if that is the case, run it with ~--execute~:

hall scan --execute

Only if hall scan does not what you want it to do, you need to edit ~hall.scm~ manually.

The hall scan utility is useful, because we do not need to manually arrange the file structure in hall.scm, even, if the structure of source code files inside the directory named after the project changes a lot. At the same time it is useful, that hall.scm is designed to be human-readable and human-editable.

Last checks

Before building the distribution tarball, check critical things like the following:

• Is the license file correct?
• Does hall.scm contains all relevant GNU Guile files?
• Is the documentation correct?

Fix any mistakes you find.

Create distribution tarball

hall created the guix.scm file.

TODO: How does loading the guix.scm file influence the guix environment, which is activated? What does it add?

The guix.scm file needs to be loaded inside a GNU Guix environment. To make sure, that a clean environment is used, activate a new environment by running:

guix environment --load='guix.scm'

Clean the directory of old files that may be around from previous

Then create the distribution by running the following command in the new shell:

hall dist --execute autoreconf --verbose --install --force sh configure make dist

Testing the package

After creating a distribution, it should be tested.

Inside a GNU Guix environment try installing it using:

guix package --install-from-file='guix.scm'

If it succeeds, the package should be listed in the output of:

guix package --list-installed

Cleanup

The cleanup step is only needed, if there happen to be artifacts from previous packaging. This can for example be the case, when you are publishing a new version of your project, after already having packaged it before.

The cleanup step of hall will remove all files, which are not specified in ~hall.scm~. It is a possibly destructive action and one should use the dry run functionality of hall to ensure, that it is only performing the intended actions:

hall clean

If you notice it listing any files up for deletion, which you need and which should be part of the project, you should add them to hall.scm, rerun the dry run and check again. Do this until the actions announced by the dry run fit what you intend and then execute the cleanup step:

hall clean --execute

The clean command will only delete things, which are not specified in your ~hall.scm~ file. If you specified files in hall.scm generated by hall during previous packaging, for example a Makefile.am or a configure, sometimes it can lead to problems, when trying to make a distribution. This is the case, if those specified files contain obsolete information, such as a wrong file name of a license file. Make sure to use fresh files when appropriate. Keeping files like Makefile.am or configure might make sense, when the structure of the project does not change, as they enable people without hall installed to build the project and then install it in their GNU Guix environment.

TLDR

In short you need to do the following:

1. activate the GNU Guix development environment as described in Activate the GNU Guix environment
2. run hall init as described in Running
3. adapt hall's files appropriately

See:
• Hall project definition file hall.scm
• Correction of files
• Correct version
• Correct tests location
• Add GNU Guile files

1. create the distribution tarball as described in Create distribution tarball
2. package installation as described in Testing the package

Packaging pure GNU Guile projects

For pure GNU Guile packages, it should be possible to follow the guide at https://guix.gnu.org/cookbook/en/guix-cookbook.html. The following tries to elaborate a bit on the involved steps.

Adapting the example

The example for the hello package makes use of the gnu-build-system. However, the gnu-build-system adds an unnecessary amount of complexity to the packaging process, such as probably having to use guile-hall, to create the required files and creating more points in the process, where things can go wrong or simply fail. For pure GNU Guile projects it is therefore easier to use the ~guile-build-system~ instead.

When using the guile-build-system, the method to fetch the sources of a package is git-fetch, instead of url-fetch.

For ways of calculating the hashsum see Calculate project hashsums.

The adapted example for guile-fslib using guile-build-system looks as follows:

(use-modules (guix packages) (guix download) (guix git-download) (guix build-system gnu) (guix build-system guile) (gnu packages guile) ((guix licenses) #:prefix license:))

(define-public guile-fslib (package (name "guile-fslib") (version "0.2.0") (source (origin (method git-fetch) (uri (git-reference (url "https://notabug.org/ZelphirKaltstahl/guile-fslib/") (commit version))) (file-name (git-file-name name version)) (sha256 (base32 "118d84p443w7hrslv8hjyhgws631ia08mggiyklkmk0b9plfdsvz")))) (build-system guile-build-system) (inputs `(("guile" ,guile-3.0))) (home-page "https://notabug.org/ZelphirKaltstahl/guile-fslib") (synopsis "File system utilities") (description "This package contains like super cool file system utilities and stuff. It's really good and so easy to install!") (license license:agpl3+)))

guile-fslib

TODO Trying installation