Guix and Nix: Functional package management

Guix and Nix

Functional package management

What's in it for me?

Stow

GNU gnu

Version 1.3.2 perfected in 1996

Simplest possible package manager

(...yet does things dpkg/rpm can't!)

$ ./configure --prefix=~/.local
$ make
$ make install prefix=~/.local/stow/min_app-2.3.1
$ cd ~/.local/stow
$ ln -s min_app-2.3.1 min_app
$ stow min_app

Symlink tree

.local/
├── bin
│   ├── din_app -> ../stow/din_app/bin/din_app
│   └── min_app -> ../stow/min_app/bin/min_app
├── lib -> stow/ett_lib/lib
├── share -> stow/min_app/share
└── stow
    ├── din_app -> din_app-5.7.3
    ├── din_app-5.7.3
    │   └── bin
    │       └── din_app
    ├── ett_lib -> ett_lib-4.3.2
    ├── ett_lib-4.3.2
    │   └── lib
    │       ├── libett_lib.so.4 -> libett_lib.so.4.3.2
    │       └── libett_lib.so.4.3.2
    ├── min_app -> min_app-2.3.1
    └── min_app-2.3.1
        ├── bin
        │   └── min_app
        └── share
            └── man
                └── man1
                    └── min_app.1

Now what?

Upgrades

$ rm min_app && ln -s min_app_9.3.4 min_app &&
stow -R min_app

Dependencies
- Manual
Upgrades of dependencies
- Are you kidding me?!
Atomic install of package and dependencies
- I did say "simplest possible!"

The Merkle

A hash tree

Looser and stricter definitions. Loose definition here.
Nodes have hashes. Parent hashes involve child hashes, validating the whole chain.
Doesn't even have to be a tree, a DAG is good enough.
BitTorrent, ZFS, git, BitCoin, IPFS, MaidSafe, Nix, Guix

Immutability

The hash is the ID of the data.
Immutable data, mutable references.
Git refs, IPFS ipns, Nix/Guix profiles
Live distributed systems seem to inevitably involve a DHT.
Hashed, immutable data is a cache, if derivative / buildable.
Garbage collection

The hash DAG of Guix / Nix

Files and directories in /gnu/store, /nix/store
/gnu/store/zby49a...-hello-2.10/bin/hello
Immutable data: store
Mutable reference: profile symlink

$ guix environment --ad-hoc hello -- which hello
/gnu/store/fc71rxim411dn41gcnnrhlvv4zy1qjah-profile/bin/hello
$ guix environment --ad-hoc hello -- bash -c 'readlink -m $(which hello)'
/gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/bin/hello

The hash DAG of Guix / Nix

$ ls -l ~/.guix-profile /var/guix/profiles/per-user/clacke/
lrwxrwxrwx 1 clacke clacke   47 Mar 27 05:40 /home/clacke/.guix-profile -> /var/guix/profiles/per-user/clacke/guix-profile

/var/guix/profiles/per-user/clacke/:
total 0
lrwxrwxrwx 1 clacke clacke 55 Jul 14 15:26 guix-profile -> /var/guix/profiles/per-user/clacke/guix-profile-30-link
lrwxrwxrwx 1 clacke clacke 51 Jun 24 17:21 guix-profile-28-link -> /gnu/store/f9i1pmzpk9s1m7bpgjrprh7mw88jcbcr-profile
lrwxrwxrwx 1 clacke clacke 51 Jul 14 15:22 guix-profile-29-link -> /gnu/store/6rlcrh78ljy6j4hbji4lqxnpfbd23835-profile
lrwxrwxrwx 1 clacke clacke 51 Jul 14 15:26 guix-profile-30-link -> /gnu/store/abdfzdkvvv1fsjr44rxxikw8yzsyiyvp-profile

Hash all the things!

$ ls -U /gnu/store/.links/ | head -n 5 | xargs file
12m47ggncih2sxfcjyajaad7j04bg8vr4jbch8rv7s5qmw13rmia: UTF-8 Unicode text
1pfixdyxlv2qvd8lzpp23lfbqqf0l7zwhjbfwaca1xsqq1i8pgis: Compiled terminfo entry
1mbcry0fi6sg8nz9yb8dbksh3lwyvc1jkg8f3nljm63crh9p11w3: ASCII text, with very long lines, with no line terminators
1rn7ra2pnc0qigy30ykk51m4i17jl52ijzvc5k4kqwjaipgyfp8k: Guile Object, little endian, 64bit, bytecode v2.0
0c0370icr2byr50zvxf9nqd2qikphxyyg08kp93kjq1ksxwpbdmz: Perl POD document, ASCII text

Functional package management

Functional programming

No side-effects
Data in, data out
Same calculation twice, same data out
Immutable data
Mutable references (at some point!)

How does this translate to package management?

No network, no irrelevant data
Source, dependencies, tools in, package out
chroot with only these things available
Eliminate variables like hostname, time
Package build functional -> reproducible -> cacheable, hashable
System state = set of packages / services = top of hash DAG

What does this enable?

Unprivileged builds
Per-user package sets
Per-process package sets
Atomic install / config / rollback
Mixed dependency versions
... which allows hard-coded dependency paths

Nix

Nix is the package manager

NixOS is the OS

Quick history

2003: First release
2004: NixPkgs on Darwin/OSX (!)
2013: NixOps

A typical package

{ stdenv, fetchurl }:

stdenv.mkDerivation rec {
  name = "hello-2.10";

  src = fetchurl {
    url = "mirror://gnu/hello/${name}.tar.gz";
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };

  doCheck = false;

  meta = {
    description = "A program that produces a familiar, friendly greeting";
    longDescription = ''
      GNU Hello is a program that prints "Hello, world!" when you run it.
      It is fully customizable.
    '';
    homepage = http://www.gnu.org/software/hello/manual/;
    license = stdenv.lib.licenses.gpl3Plus;
    maintainers = [ stdenv.lib.maintainers.eelco ];
    platforms = stdenv.lib.platforms.all;
  };
}

Imperative

$ hello
-bash: hello: command not found
$ nix-env -i hello
installing ‘hello-2.10’
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10
fetching path ‘/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10’...

*** Downloading ‘http://cache.nixos.org/nar/0b8p13mywvl0d151xmil13fggl4kgjmjz8lz7yv0yjrw3pl0bm4r.nar.xz’ to ‘/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24052  100 24052    0     0   6280      0  0:00:03  0:00:03 --:--:--  9439

building path(s) ‘/nix/store/rzpafr39rnia76a8lgc72is7r5svr5jm-user-environment’
created 3108 symlinks in user environment
$ hello
Hello, world!
$ which hello
/Users/cw/.nix-profile/bin/hello

Ad-hoc

$ hello
-bash: hello: command not found
$ nix-shell -p hello
[nix-shell]$ hello
Hello, world!
[nix-shell]$ which hello
/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin/hello
[nix-shell]$ exit
$ hello
-bash: hello: command not found

Manifest

$ hello
-bash: hello: command not found
$ cat default.nix
{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "dummy";
  src = ./.;
  buildInputs = with pkgs; [
    hello
  ];
}
$ nix-shell
[nix-shell]$ hello
Hello, world!
[nix-shell]$ which hello
/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin/hello

Profile

$ nix-env -p profile -i hello
installing ‘hello-2.10’
building path(s) ‘/nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment’
created 2 symlinks in user environment
$ ls -l
total 0
lrwxr-xr-x 1 cw staff 19 Jul 14 13:21 profile -> profile-1-link
lrwxr-xr-x 1 cw staff 76 Jul 14 13:21 profile-1-link -> /nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
$ tree /nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
/nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
├── bin -> /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin
├── manifest.nix -> /nix/store/akxg765a7fj3s9a4wjx1bfk02ya5fx8x-env-manifest.nix
└── share -> /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/share

Language-specific dependencies

package.json
requirements.txt
Gemfile
*.cabal
pom.xml / build.gradle
*.nuspec

Language-specific dependencies

*.nix:

Get not only your Python/Ruby/JVM/CLR package, but also the system libraries required.

Language-specific dependencies

This is for a bash script and a python script:

{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "build-env";
  src = ./.;
  buildInputs = with pkgs; [
    wget
    python
    pythonPackages.lxml
    git
    gitAndTools.git-annex
  ];
}

Guix

GuixSD logo

Guix is the package manager

GuixSD is the OS

Based on Nix

Same daemon, but ...
package and system definitions in scheme! (internal DSL)
Less mature (announced 2012)
Fewer packages (~3400 vs ~7500) (Debian ~49000!)
More work on:
- Reproducibility
- Containers
Systemd vs Shepherd
Service dependencies

Differences from Nix language

Not purely functional (but derivations are!)
No definition-composition separation
Scheme!
- "No syntax", macros
- Advanced bytecode compiler and VM (GNU Guile)
No inline shell, derivations pure scheme
- Even some tools reimplemented, e.g. cpio
- Calls to external shell/tools of course possible

A familiar package

(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source
      (origin
        (method url-fetch)
        (uri (string-append "mirror://gnu/hello/hello-" version
                            ".tar.gz"))
          (sha256
           (base32
            "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.
It serves as an example of standard GNU coding practices.  As such,
it supports command-line arguments, multiple languages, and so on.")
    (home-page "http://www.gnu.org/software/hello/")
    (license gpl3+)))

Imperative

$ hello
-bash: hello: command not found
$ guix package -i hello
The following package will be installed:
   hello        2.10    /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10

updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%g'...  66.7%
The following derivations will be built:
   /gnu/store/44zv48y4fnrsx0prj1pxjw09h7rabzbg-profile.drv
   /gnu/store/d18rgr2ngnw09sq9pbb9k40s3l6lx2d5-info-dir.drv
   /gnu/store/4l7qyp4dng59zfwwhxk5j2ds2mmagdfj-ca-certificate-bundle.drv
7 packages in profile
$ hello
Hello, world!
$ which hello
/home/clacke/.guix-profile/bin/hello

Ad-hoc

$ hello
-bash: hello: command not found
$ guix environment --ad-hoc hello
updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%g'...  66.7%
The following derivations will be built:
   /gnu/store/05jqsyrzgzdfgp12ww6ljm8q1yizj63c-profile.drv
   /gnu/store/gy3942fm5sdgiij5y7s8mkymcfg2ghrx-info-dir.drv
   /gnu/store/c1dnn48gw2w6c3xkk6qbma3jkavyw8dc-ca-certificate-bundle.drv
$ hello
Hello, world!
$ which hello
/gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile/bin/hello
$ exit
exit
$ hello
-bash: hello: command not found

Profile

$ guix package -p profile -i hello
The following package will be installed:
   hello        2.10    /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10

1 package in profile
The following environment variable definitions may be needed:
   export PATH="profile/bin"
clacke@luk:~/tmp/guix$ ls -l
total 0
lrwxrwxrwx 1 clacke clacke 14 Jul 14 11:05 profile -> profile-1-link
lrwxrwxrwx 1 clacke clacke 51 Jul 14 11:05 profile-1-link -> /gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile
$ tree /gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile
/gnu/store/3f7ilcf1cn0chk3jaicdbr4wpc4z3c63-profile
├── bin -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/bin
├── etc
│   └── profile
├── manifest
└── share
    ├── info
    │   ├── dir -> /gnu/store/6w898pknyrndnaf9syrdak85kqv8v5bg-info-dir/share/info/dir
    │   └── hello.info.gz -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/share/info/hello.info.gz
    ├── locale -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/share/locale
    └── man -> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10/share/man

Manifest

(define-module (_)
  #:use-module ((gnu packages base) #:select (hello))
  #:use-module ((guix packages) #:select (package origin))
  #:use-module (guix build-system trivial)

(package
  (inputs `(
    ("hello" ,hello)))
  (build-system trivial-build-system)
  (name "")
  (version "")
  (source (origin (uri #f) (method #f) (sha256 #f)))
  (description #f)
  (synopsis #f)
  (home-page #f)
  (license #f))

Manifest

$ hello
-bash: hello: command not found
$ guix environment -l default.scm  -- hello
Hello, world!
$ hello
-bash: hello: command not found

Closing words

Try it out!

Put Nix on your OSX today!
Put both on your GNU/Linux! (... and Stow!)
NixOS works great in a VM, GuixSD getting there.
Use Guix instead of Docker for the services on your VPS
Try NixOps for your VMs! Nudge, nudge ... ;-)
(... what's it like?)
Use Guix to sneak tools into your minimal Docker instances

When you develop your next Python / Ruby / Guile / Node package

Put a default.scm and a default.nix in there
Never use Bundler / VirtualEnv during development again
Discover the horrors of Node packaging ;-)

Avoid packaging hell

A moment of nostalgia ...

(license: "any non-commercial purpose")

http://www.commitstrip.com/en/2016/05/10/a-moment-of-nostalgia/

Acknowledgements

Resources

Nix and NixOS: https://nixos.org/
Guix and GuixSD: https://www.gnu.org/software/guix/
Guile Scheme: https://www.gnu.org/software/guile/
Guix whitepaper: http://arxiv.org/pdf/1305.4584v1.pdf
Let's Package jQuery: A Javascript Packaging Dystopian Novella
hpr2198 :: How awesome is Guix and why will it take over the world
Sander van der Burg's blog on buildEnv
SvdB: Challenges with npm and npm2nix (prelude to node2nix)
SvdB: Making flat nix apps from node packages with node2nix

Sources

GNU logo: Original from https://www.gnu.org/graphics/heckert_gnu.html, modified by me. (CC-by-SA 2.0 | GFDL 1.3 | Free Art License)
NixOS logo: Original from https://github.com/NixOS/nixos-artwork/tree/master/logo/, modified by me. (License: dunno handwave fair use)
Guix logo: Original from https://www.gnu.org/software/guix/graphics/, modified by me. (GPLv3+ | GFDL 1.3 with exceptions, see source)
GuixSD logo: Original from https://www.gnu.org/software/guix/graphics/, modified by me. (CC-by-SA 4.0 Intl)

Making of

$ cat default.nix
{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "build-env";
  src = ./.;
  buildInputs = with pkgs; [
    gnumake
    pandoc
  ];
}
$ nix-shell --pure --run make

Guix and Nix: Functional package management

Guix and Nix

Functional package management

What's in it for me?

Meta

Stow

Stow

Simplest possible package manager

Symlink tree

Now what?

The Merkle

A hash tree

Immutability

The hash DAG of Guix / Nix

The hash DAG of Guix / Nix

Hash all the things!

Functional package management

Functional programming

How does this translate to package management?

What does this enable?

Nix

Nix

Nix is the package manager

NixOS is the OS

Quick history

A typical package

Imperative

Ad-hoc

Manifest

Profile

Language-specific dependencies

Language-specific dependencies

Language-specific dependencies

Guix

Guix

Guix is the package manager

GuixSD is the OS

Based on Nix

Differences from Nix language

A familiar package

Imperative

Ad-hoc

Profile

Manifest

Manifest

Closing words

Try it out!

When you develop your next Python / Ruby / Guile / Node package

Avoid packaging hell

Acknowledgements

Resources

Sources

Making of