Nix: Functional package management

Nix

Functional package management

What’s in it for me?

Stow

GNU gnu

Version 1.3.2 perfected in 1996

Simplest possible package manager

(…yet does things dpkg/rpm can’t!)

$ ./configure --prefix=~/.local
$ make
$ make install prefix=~/.local/stow/min_app-2.3.1
$ cd ~/.local/stow
$ ln -s min_app-2.3.1 min_app
$ stow min_app

Symlink tree

.local/
├── bin
│   ├── din_app -> ../stow/din_app/bin/din_app
│   └── min_app -> ../stow/min_app/bin/min_app
├── lib -> stow/ett_lib/lib
├── share -> stow/min_app/share
└── stow
    ├── din_app -> din_app-5.7.3
    ├── din_app-5.7.3
    │   └── bin
    │       └── din_app
    ├── ett_lib -> ett_lib-4.3.2
    ├── ett_lib-4.3.2
    │   └── lib
    │       ├── libett_lib.so.4 -> libett_lib.so.4.3.2
    │       └── libett_lib.so.4.3.2
    ├── min_app -> min_app-2.3.1
    └── min_app-2.3.1
        ├── bin
        │   └── min_app
        └── share
            └── man
                └── man1
                    └── min_app.1

Now what?

Upgrades

$ rm min_app && ln -s min_app_9.3.4 min_app &&
stow -R min_app

Dependencies
- Manual
Upgrades of dependencies
- Are you kidding me?!
Atomic install of package and dependencies
- I did say “simplest possible!”

Consistency models of infrastructure

Divergent

Divergent systems drift from the known state

Convergent

Convergent systems move toward the known state

Congruent

Congruent systems manifest the defined state

The Merkle

A hash tree

Looser and stricter definitions. Loose definition here.
Nodes have hashes. Parent hashes involve child hashes, validating the whole chain.
Doesn’t even have to be a tree, a DAG is good enough.
BitTorrent, ZFS, git, BitCoin, IPFS, MaidSafe, Nix, Guix

Immutability

The hash is the ID of the data.
Immutable data, mutable references.
Git refs, IPFS ipns, Nix/Guix profiles
Hashed, immutable data is a cache, if derivative / buildable.
Garbage collection

The hash DAG of Nix

Files and directories in /nix/store
/nix/store/zby49a...-hello-2.10/bin/hello
Immutable data: store
Mutable reference: profile symlink

$ nix-shell -p hello --run 'which hello'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/sgqxq0c9wfjsa7lz9bzjq4amf4lfd8ja-hello-2.10
copying path '/nix/store/sgqxq0c9wfjsa7lz9bzjq4amf4lfd8ja-hello-2.10' from 'https://cache.nixos.org'...
/nix/store/sgqxq0c9wfjsa7lz9bzjq4amf4lfd8ja-hello-2.10/bin/hello

The hash DAG of Nix

$ ls -l ~/.nix-profile /nix/var/nix/profiles/per-user/cw/profile /nix/var/nix/profiles/per-user/cw/profile/
lrwxr-xr-x  1 cw   staff   41 Mar  9  2016 /Users/cw/.nix-profile -> /nix/var/nix/profiles/per-user/cw/profile
lrwxr-xr-x  1 cw   nixbld  16 Jun 15 12:42 /nix/var/nix/profiles/per-user/cw/profile -> profile-268-link

/nix/var/nix/profiles/per-user/cw/profile/:
total 8
dr-xr-xr-x 380 root wheel 12920 Jan  1  1970 bin
dr-xr-xr-x  12 root wheel   408 Jan  1  1970 etc
dr-xr-xr-x  12 root wheel   408 Jan  1  1970 include
dr-xr-xr-x  43 root wheel  1462 Jan  1  1970 lib
dr-xr-xr-x  20 root wheel   680 Jan  1  1970 libexec
lrwxr-xr-x   1 root wheel    60 Jan  1  1970 manifest.nix -> /nix/store/96ibr8p0cvn78dmisivdxa8m3sdv4wwh-env-manifest.nix
lrwxr-xr-x   1 root wheel    57 Jan  1  1970 sbin -> /nix/store/1ylyis17lxhxwxvhrnjiqb2jy2yl7i4p-mtr-0.92/sbin
dr-xr-xr-x  28 root wheel   952 Jan  1  1970 share

Hash all the things!

$ ls -U /nix/store/.links/ | head -n 5 | xargs file
12m47ggncih2sxfcjyajaad7j04bg8vr4jbch8rv7s5qmw13rmia: UTF-8 Unicode text
1pfixdyxlv2qvd8lzpp23lfbqqf0l7zwhjbfwaca1xsqq1i8pgis: Compiled terminfo entry
1mbcry0fi6sg8nz9yb8dbksh3lwyvc1jkg8f3nljm63crh9p11w3: ASCII text, with very long lines, with no line terminators
1rn7ra2pnc0qigy30ykk51m4i17jl52ijzvc5k4kqwjaipgyfp8k: Guile Object, little endian, 64bit, bytecode v2.0
0c0370icr2byr50zvxf9nqd2qikphxyyg08kp93kjq1ksxwpbdmz: Perl POD document, ASCII text

Functional package management

Functional programming

No side-effects
Data in, data out
Same calculation twice, same data out
Immutable data
Mutable references (at some point!)

How does this translate to package management?

No network, no irrelevant data
Source, dependencies, tools in, package out
chroot with only these things available
Eliminate variables like hostname, time
Package build functional -> reproducible -> cacheable, hashable
System state = set of packages / services = top of hash DAG

What does this enable?

Unprivileged builds
Per-user package sets
Per-process package sets
Atomic install / config / rollback
Mixed dependency versions
… which allows hard-coded dependency paths

Nix

NixOS logo

Nix is the package manager

NixOS is the OS

Quick history

2003: First release
2004: NixPkgs on Darwin/OSX (!)
2013: NixOps

A typical package

{ stdenv, fetchurl }:

stdenv.mkDerivation rec {
  name = "hello-2.10";

  src = fetchurl {
    url = "mirror://gnu/hello/${name}.tar.gz";
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };

  doCheck = false;

  meta = {
    description = "A program that produces a familiar, friendly greeting";
    longDescription = ''
      GNU Hello is a program that prints "Hello, world!" when you run it.
      It is fully customizable.
    '';
    homepage = http://www.gnu.org/software/hello/manual/;
    license = stdenv.lib.licenses.gpl3Plus;
    maintainers = [ stdenv.lib.maintainers.eelco ];
    platforms = stdenv.lib.platforms.all;
  };
}

Imperative

$ hello
-bash: hello: command not found
$ nix-env -i hello
installing ‘hello-2.10’
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10
fetching path ‘/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10’...

*** Downloading ‘http://cache.nixos.org/nar/0b8p13mywvl0d151xmil13fggl4kgjmjz8lz7yv0yjrw3pl0bm4r.nar.xz’ to ‘/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24052  100 24052    0     0   6280      0  0:00:03  0:00:03 --:--:--  9439

building path(s) ‘/nix/store/rzpafr39rnia76a8lgc72is7r5svr5jm-user-environment’
created 3108 symlinks in user environment
$ hello
Hello, world!
$ which hello
/Users/cw/.nix-profile/bin/hello

Ad-hoc

$ hello
-bash: hello: command not found
$ nix-shell -p hello
[nix-shell]$ hello
Hello, world!
[nix-shell]$ which hello
/nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin/hello
[nix-shell]$ exit
$ hello
-bash: hello: command not found

Manifest

$ hello
bash: hello: command not found

$ cat default.nix
{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.buildEnv rec {
  name = "hello-env";
  paths = with pkgs; [
    hello
  ];
  buildInputs = paths;
}

$ nix-shell

$ hello
Hello, world!

$ which hello
/nix/store/sgqxq0c9wfjsa7lz9bzjq4amf4lfd8ja-hello-2.10/bin/hello

Profile

$ nix-env -p profile -i hello
installing ‘hello-2.10’
building path(s) ‘/nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment’
created 2 symlinks in user environment
$ ls -l
total 0
lrwxr-xr-x 1 cw staff 19 Jul 14 13:21 profile -> profile-1-link
lrwxr-xr-x 1 cw staff 76 Jul 14 13:21 profile-1-link -> /nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
$ tree /nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
/nix/store/1in3spnpk55b7gisds2klp92rz5bzvhl-user-environment
├── bin -> /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/bin
├── manifest.nix -> /nix/store/akxg765a7fj3s9a4wjx1bfk02ya5fx8x-env-manifest.nix
└── share -> /nix/store/20fyp0s5l59ixq7a8y02r9f5837plsiw-hello-2.10/share

Language-specific dependencies

package.json
requirements.txt
Gemfile
*.cabal
pom.xml / build.gradle
*.nuspec

Language-specific dependencies

*.nix:

Get not only your Python/Ruby/JVM/CLR package, but also the system libraries required.

Language-specific dependencies

This is for a bash script and a python script:

{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.stdenv.mkDerivation rec {
  name = "build-env";
  src = ./.;
  buildInputs = with pkgs; [
    wget
    python
    pythonPackages.lxml
    git
    gitAndTools.git-annex
  ];
}

Guix

Guix logo GuixSD logo

Guix is the package manager

GuixSD is the OS

Derived from Nix

Same daemon, but …
package and system definitions in scheme! (internal DSL)
Less mature (announced 2012)
Fewer packages (10,583 vs 49,681) (Debian 89,180!)
More work on:
- Reproducibility
- Containers
Systemd vs Shepherd
Service dependencies

Differences from Nix language

Not purely functional (but derivations are!)
No definition-composition separation
Scheme!
- “No syntax”, macros
- Advanced bytecode compiler and VM (GNU Guile)
No inline shell, derivations pure scheme
- Even some tools reimplemented, e.g. cpio
- Calls to external shell/tools of course possible

Closing words

Try it out!

Put Nix on your OSX today!
Put Nix and Guix on your GNU/Linux! (… and Stow!)
NixOS works great in a VM, GuixSD too.
Use Nix instead of Docker for the services on your VPS
Try NixOps for your VMs!
(… what’s it like?)
Use Nix to sneak tools into your minimal Docker instances

When you develop your next Python / Ruby / Guile / Node package

Put a default.nix in there
Never use Bundler / VirtualEnv during development again
Discover the horrors of Node packaging ;-)

Avoid packaging hell

A moment of nostalgia …

(license: “any non-commercial purpose”)

http://www.commitstrip.com/en/2016/05/10/a-moment-of-nostalgia/

Acknowledgements

Resources

Nix and NixOS: https://nixos.org/
Guix and GuixSD: https://www.gnu.org/software/guix/
Guile Scheme: https://www.gnu.org/software/guile/
Guix whitepaper: http://arxiv.org/pdf/1305.4584v1.pdf
Let’s Package jQuery: A Javascript Packaging Dystopian Novella
hpr2198 :: How awesome is Guix and why will it take over the world
Sander van der Burg’s blog on buildEnv
SvdB: Challenges with npm and npm2nix (prelude to node2nix)
SvdB: Making flat nix apps from node packages with node2nix

Sources

GNU logo: Original from https://www.gnu.org/graphics/heckert_gnu.html, modified by me. (CC-by-SA 2.0 | GFDL 1.3 | Free Art License)
NixOS logo: Original from https://github.com/NixOS/nixos-artwork/tree/master/logo/, modified by me. (License: dunno handwave fair use)
Guix logo: Original from https://www.gnu.org/software/guix/graphics/, modified by me. (GPLv3+ | GFDL 1.3 with exceptions, see source)
GuixSD logo: Original from https://www.gnu.org/software/guix/graphics/, modified by me. (CC-by-SA 4.0 Intl)

Making of

$ cat default.nix
{ }:

let
  pkgs = import <nixpkgs> {};

in pkgs.buildEnv rec {
  name = "slides-env";
  buildInputs = with pkgs; [
    gnumake
    pandoc
  ];
  paths = buildInputs;
}

$ nix-shell --pure --run make