Guix and Nix: Functional package management

Guix and Nix

Functional package management

What's in it for me?

Meta

Claes Wallin (acgg hos clacke.greatsinodevelopment.com)

2016-07-16 ACGG Sommarhack

© 2016 Claes Wallin

CC-by-SA 4.0 except where noted.

Stow

Stow

GNU gnu

Version 1.3.2 perfected in 1996

Simplest possible package manager

(...yet does things dpkg/rpm can't!)

$ ./configure --prefix=~/.local
$ make
$ make install prefix=~/.local/stow/min_app-2.3.1
$ cd ~/.local/stow
$ ln -s min_app-2.3.1 min_app
$ stow min_app

Now what?

  • Upgrades
    $ rm min_app && ln -s min_app_9.3.4 min_app &&
    stow -R min_app
  • Dependencies
    • Manual
  • Upgrades of dependencies
    • Are you kidding me?!
  • Atomic install of package and dependencies
    • I did say "simplest possible!"

The Merkle

A hash tree

  • Looser and stricter definitions. Loose definition here.
  • Nodes have hashes. Parent hashes involve child hashes, validating the whole chain.
  • Doesn't even have to be a tree, a DAG is good enough.
  • BitTorrent, ZFS, git, BitCoin, IPFS, MaidSafe, Nix, Guix

Immutability

  • The hash is the ID of the data.
  • Immutable data, mutable references.
  • Git refs, IPFS ipns, Nix/Guix profiles
  • Live distributed systems seem to inevitably involve a DHT.
  • Hashed, immutable data is a cache, if derivative / buildable.
  • Garbage collection

The hash DAG of Guix / Nix

  • Files and directories in /gnu/store, /nix/store
  • /gnu/store/zby49a...-hello-2.10/bin/hello
  • Immutable data: store
  • Mutable reference: profile symlink
$ guix environment --ad-hoc hello -- which hello
/gnu/store/fc71rxim411dn41gcnnrhlvv4zy1qjah-profile/bin/hello
$ guix environment --ad-hoc hello -- bash -c 'readlink -m $(which hello)'
/gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/bin/hello

The hash DAG of Guix / Nix

$ ls -l ~/.guix-profile /var/guix/profiles/per-user/clacke/
lrwxrwxrwx 1 clacke clacke   47 Mar 27 05:40 /home/clacke/.guix-profile -> /var/guix/profiles/per-user/clacke/guix-profile

/var/guix/profiles/per-user/clacke/:
total 0
lrwxrwxrwx 1 clacke clacke 55 Jul 14 15:26 guix-profile -> /var/guix/profiles/per-user/clacke/guix-profile-30-link
lrwxrwxrwx 1 clacke clacke 51 Jun 24 17:21 guix-profile-28-link -> /gnu/store/f9i1pmzpk9s1m7bpgjrprh7mw88jcbcr-profile
lrwxrwxrwx 1 clacke clacke 51 Jul 14 15:22 guix-profile-29-link -> /gnu/store/6rlcrh78ljy6j4hbji4lqxnpfbd23835-profile
lrwxrwxrwx 1 clacke clacke 51 Jul 14 15:26 guix-profile-30-link -> /gnu/store/abdfzdkvvv1fsjr44rxxikw8yzsyiyvp-profile

Hash all the things!

$ ls -U /gnu/store/.links/ | head -n 5 | xargs file
12m47ggncih2sxfcjyajaad7j04bg8vr4jbch8rv7s5qmw13rmia: UTF-8 Unicode text
1pfixdyxlv2qvd8lzpp23lfbqqf0l7zwhjbfwaca1xsqq1i8pgis: Compiled terminfo entry
1mbcry0fi6sg8nz9yb8dbksh3lwyvc1jkg8f3nljm63crh9p11w3: ASCII text, with very long lines, with no line terminators
1rn7ra2pnc0qigy30ykk51m4i17jl52ijzvc5k4kqwjaipgyfp8k: Guile Object, little endian, 64bit, bytecode v2.0
0c0370icr2byr50zvxf9nqd2qikphxyyg08kp93kjq1ksxwpbdmz: Perl POD document, ASCII text

Functional package management

Functional programming

  • No side-effects
  • Data in, data out
  • Same calculation twice, same data out
  • Immutable data
  • Mutable references (at some point!)

How does this translate to package management?

  • No network, no irrelevant data
  • Source, dependencies, tools in, package out
  • chroot with only these things available
  • Eliminate variables like hostname, time
  • Package build functional -> reproducible -> cacheable, hashable
  • System state = set of packages / services = top of hash DAG

What does this enable?

  • Unprivileged builds
  • Per-user package sets
  • Per-process package sets
  • Atomic install / config / rollback
  • Mixed dependency versions
  • ... which allows hard-coded dependency paths

Nix

Nix

NixOS logo

Nix is the package manager

NixOS is the OS

Quick history

  • 2003: First release
  • 2004: NixPkgs on Darwin/OSX (!)
  • 2013: NixOps

A typical package

{ stdenv, fetchurl }:

stdenv.mkDerivation rec {
  name = "hello-2.10";

  src = fetchurl {
    url = "mirror://gnu/hello/${name}.tar.gz";
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };

  doCheck = false;

  meta = {
    description = "A program that produces a familiar, friendly greeting";
    longDescription = ''
      GNU Hello is a program that prints "Hello, world!" when you run it.
      It is fully customizable.
    '';
    homepage = http://www.gnu.org/software/hello/manual/;
    license = stdenv.lib.licenses.gpl3Plus;
    maintainers = [ stdenv.lib.maintainers.eelco ];
    platforms = stdenv.lib.platforms.all;
  };
}

Imperative

$ hello
-bash: hello: command not found
$ nix-env -i hello
installing ‘hello-2.10’
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10
fetching path ‘/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10’...

*** Downloading ‘http://cache.nixos.org/nar/0b8p13mywvl0d151xmil13fggl4kgjmjz8lz7yv0yjrw3pl0bm4r.nar.xz’ to ‘/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24052  100 24052    0     0   6280      0  0:00:03  0:00:03 --:--:--  9439

building path(s) ‘/nix/store/rzpafr39rnia76a8lgc72is7r5svr5jm-user-environment’
created 3108 symlinks in user environment
$ hello
Hello, world!
$ which hello
/Users/cw/.nix-profile/bin/hello

Ad-hoc

$ hello
-bash: hello: command not found
$ nix-shell -p hello
[nix-shell]$ hello
Hello, world!
[nix-shell]$ which hello
/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin/hello
[nix-shell]$ exit
$ hello
-bash: hello: command not found

Manifest

$ hello
-bash: hello: command not found
$ cat default.nix
{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "dummy";
  src = ./.;
  buildInputs = with pkgs; [
    hello
  ];
}
$ nix-shell
[nix-shell]$ hello
Hello, world!
[nix-shell]$ which hello
/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin/hello

Profile

$ nix-env -p profile -i hello
installing ‘hello-2.10’
building path(s) ‘/nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment’
created 2 symlinks in user environment
$ ls -l
total 0
lrwxr-xr-x 1 cw staff 19 Jul 14 13:21 profile -> profile-1-link
lrwxr-xr-x 1 cw staff 76 Jul 14 13:21 profile-1-link -> /nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
$ tree /nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
/nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
├── bin -> /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin
├── manifest.nix -> /nix/store/akxg765a7fj3s9a4wjx1bfk02ya5fx8x-env-manifest.nix
└── share -> /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/share

Language-specific dependencies

  • package.json
  • requirements.txt
  • Gemfile
  • *.cabal
  • pom.xml / build.gradle
  • *.nuspec

Language-specific dependencies

*.nix:

  • Get not only your Python/Ruby/JVM/CLR package, but also the system libraries required.

Language-specific dependencies

This is for a bash script and a python script:

{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "build-env";
  src = ./.;
  buildInputs = with pkgs; [
    wget
    python
    pythonPackages.lxml
    git
    gitAndTools.git-annex
  ];
}

Guix

Guix

Guix logo GuixSD logo

Guix is the package manager

GuixSD is the OS

Based on Nix

  • Same daemon, but ...
  • package and system definitions in scheme! (internal DSL)
  • Less mature (announced 2012)
  • Fewer packages (~3400 vs ~7500) (Debian ~49000!)
  • More work on:
    • Reproducibility
    • Containers
  • Systemd vs Shepherd
  • Service dependencies

Differences from Nix language

  • Not purely functional (but derivations are!)
  • No definition-composition separation
  • Scheme!
    • "No syntax", macros
    • Advanced bytecode compiler and VM (GNU Guile)
  • No inline shell, derivations pure scheme
    • Even some tools reimplemented, e.g. cpio
    • Calls to external shell/tools of course possible

A familiar package

(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source
      (origin
        (method url-fetch)
        (uri (string-append "mirror://gnu/hello/hello-" version
                            ".tar.gz"))
          (sha256
           (base32
            "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.
It serves as an example of standard GNU coding practices.  As such,
it supports command-line arguments, multiple languages, and so on.")
    (home-page "http://www.gnu.org/software/hello/")
    (license gpl3+)))

Imperative

$ hello
-bash: hello: command not found
$ guix package -i hello
The following package will be installed:
   hello        2.10    /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10

updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%g'...  66.7%
The following derivations will be built:
   /gnu/store/44zv48y4fnrsx0prj1pxjw09h7rabzbg-profile.drv
   /gnu/store/d18rgr2ngnw09sq9pbb9k40s3l6lx2d5-info-dir.drv
   /gnu/store/4l7qyp4dng59zfwwhxk5j2ds2mmagdfj-ca-certificate-bundle.drv
7 packages in profile
$ hello
Hello, world!
$ which hello
/home/clacke/.guix-profile/bin/hello

Ad-hoc

$ hello
-bash: hello: command not found
$ guix environment --ad-hoc hello
updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%g'...  66.7%
The following derivations will be built:
   /gnu/store/05jqsyrzgzdfgp12ww6ljm8q1yizj63c-profile.drv
   /gnu/store/gy3942fm5sdgiij5y7s8mkymcfg2ghrx-info-dir.drv
   /gnu/store/c1dnn48gw2w6c3xkk6qbma3jkavyw8dc-ca-certificate-bundle.drv
$ hello
Hello, world!
$ which hello
/gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile/bin/hello
$ exit
exit
$ hello
-bash: hello: command not found

Profile

$ guix package -p profile -i hello
The following package will be installed:
   hello        2.10    /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10

1 package in profile
The following environment variable definitions may be needed:
   export PATH="profile/bin"
clacke@luk:~/tmp/guix$ ls -l
total 0
lrwxrwxrwx 1 clacke clacke 14 Jul 14 11:05 profile -> profile-1-link
lrwxrwxrwx 1 clacke clacke 51 Jul 14 11:05 profile-1-link -> /gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile
$ tree /gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile
/gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile
├── bin -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/bin
├── etc
│   └── profile
├── manifest
└── share
    ├── info
    │   ├── dir -> /gnu/store/6w898pknyrndnaf9syrdak85kqv8v5bg-info-dir/share/info/dir
    │   └── hello.info.gz -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/share/info/hello.info.gz
    ├── locale -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/share/locale
    └── man -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/share/man

Manifest

(define-module (_)
  #:use-module ((gnu packages base) #:select (hello))
  #:use-module ((guix packages) #:select (package origin))
  #:use-module (guix build-system trivial)

(package
  (inputs `(
    ("hello" ,hello)))
  (build-system trivial-build-system)
  (name "")
  (version "")
  (source (origin (uri #f) (method #f) (sha256 #f)))
  (description #f)
  (synopsis #f)
  (home-page #f)
  (license #f))

Manifest

$ hello
-bash: hello: command not found
$ guix environment -l default.scm  -- hello
Hello, world!
$ hello
-bash: hello: command not found

Closing words

Try it out!

  • Put Nix on your OSX today!
  • Put both on your GNU/Linux! (... and Stow!)
  • NixOS works great in a VM, GuixSD getting there.
  • Use Guix instead of Docker for the services on your VPS
  • Try NixOps for your VMs! Nudge, nudge ... ;-)
  • (... what's it like?)
  • Use Guix to sneak tools into your minimal Docker instances

When you develop your next Python / Ruby / Guile / Node package

  • Put a default.scm and a default.nix in there
  • Never use Bundler / VirtualEnv during development again
  • Discover the horrors of Node packaging ;-)

Avoid packaging hell

A moment of nostalgia ...

(license: "any non-commercial purpose")

http://www.commitstrip.com/en/2016/05/10/a-moment-of-nostalgia/

Acknowledgements

Resources

Sources

Making of

$ cat default.nix
{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "build-env";
  src = ./.;
  buildInputs = with pkgs; [
    gnumake
    pandoc
  ];
}
$ nix-shell --pure --run make