__structuredAttrs in Nix

In Nix 2 a new parameter to the derivation primitive was added. It changes how information is passed to the derivation builder.

Current State

In order to show how it changes the handling of parameters to derivation, the first example will show the current state with __structuredAttrs set to false and the stdenv.mkDerivation wrapper around derivation. All parameters are passed to the builder as environment variables, canonicalised by Nix in imitation of shell script conventions:

 1$ cat <<'EOF' | nix-build - with import (builtins.fetchTarball https://github.com/nixos/nixpkgs/archive/nixos-unstable.tar.gz) {};
 2
 3stdenv.mkDerivation {
 4  name = "foo-1.2.3";
 5  dontUnpack = true;
 6
 7  EXAMPLE_BOOL_TRUE = true;
 8  EXAMPLE_BOOL_FALSE = false;
 9  EXAMPLE_INT = 123;
10  EXAMPLE_INT_NEG = -123;
11  EXAMPLE_STR = "foo bar";
12  EXAMPLE_LIST = [ "foo" "bar" ];
13  # EXAMPLE_ATTRS = { foo = "bar"; };  # currently not possible
14
15  installPhase = ''
16    env | grep "EXAMPLE" | grep -v installPhase > $out
17  '';
18}
19EOF
20
21$ cat result
22EXAMPLE_BOOL_TRUE=1
23EXAMPLE_INT_NEG=-123
24EXAMPLE_INT=123
25EXAMPLE_LIST=foo bar
26EXAMPLE_STR=foo bar
27EXAMPLE_BOOL_FALSE=

From the above, it can be seen than Nix lists are concatenated to space-seperated strings, and booleans are represnted by 1 and '', in accordance with the way bash conditionals work. (Actually bash has two conventions: 1 vs 0 or non-empty string vs empty-string; Nix tries to appease both). Attribute sets cannot be passed at all, resulting in an error:

error: cannot coerce a set to a string

This means that potentially relevant information cannot be passed faithfully. An example is quoting of flags:

1configureFlags = [ "--foo=bar baz" "--qux" ];
2configureFlags2 = [ "--foo=bar" "baz" "--qux" ];

These two parameters would evaluate to the same contents in the respective environment variables, even though the quoting information is essential for the configure script to work properly. For these cases there are workarounds in nixpkgs similar to this:

1preConfigure = ''
2  configureFlagsArray=("--foo=bar baz" "--qux")
3'';

This configureFlagsArray then gets appended to configureFlags in the stdenv bash code. This obviously is a non-ideal solution as the list cannot be passed in Nix but only in bash hooks.

stdenv with __structuredAttrs

We have created an experimental branch enabling __structuredAttrs and fixing up support for it.

 1$ cat <<'EOF' | nix-build - with import (builtins.fetchTarball https://github.com/nixos/nixpkgs/archive/structured-attrs.tar.gz) {};
 2
 3stdenv.mkDerivation {
 4  name = "foo-1.2.3";
 5  dontUnpack = true;
 6
 7  EXAMPLE_BOOL_TRUE = true;
 8  EXAMPLE_BOOL_FALSE = false;
 9  EXAMPLE_INT = 123;
10  EXAMPLE_INT_NEG = -123;
11  EXAMPLE_STR = "foo bar";
12  EXAMPLE_LIST = [ "foo" "bar" ];
13  EXAMPLE_NESTED_LIST = [ [ "foo" "bar"] [ "baz" ] ];
14  EXAMPLE_ATTRS = { foo = "bar"; };
15  EXAMPLE_NESTED_ATTRS = { foo.bar = "baz"; };
16
17  installPhase = ''
18    mkdir -p $out
19    cat .attrs.sh | grep "EXAMPLE" | grep -v installPhase > $out/sh
20    cat .attrs.json > $out/json
21  '';
22}
23EOF
24
25$ cat result/sh
26declare -A EXAMPLE_ATTRS=(['foo']='bar' )
27declare EXAMPLE_BOOL_FALSE=
28declare EXAMPLE_BOOL_TRUE=1
29declare EXAMPLE_INT=123
30declare EXAMPLE_INT_NEG=-123
31declare -a EXAMPLE_LIST=('foo' 'bar' )
32declare EXAMPLE_STR='foo bar'
33
34$ cat result/json | jq
35{
36  ...
37  "EXAMPLE_ATTRS": { "foo": "bar" },
38  "EXAMPLE_BOOL_FALSE": false,
39  "EXAMPLE_BOOL_TRUE": true,
40  "EXAMPLE_INT": 123,
41  "EXAMPLE_INT_NEG": -123,
42  "EXAMPLE_LIST": [ "foo", "bar" ],
43  "EXAMPLE_NESTED_ATTRS": { "foo": { "bar": "baz" } },
44  "EXAMPLE_NESTED_LIST": [ [ "foo", "bar" ], [ "baz" ] ],
45  "EXAMPLE_STR": "foo bar",
46  ...
47}

As can be seen from this example, two files—.attrs.sh and .attrs.json—are created if __structuredAttrs is set to true. The JSON file contains all attributes passed to derivation serialised by Nix in the same manner the builtins.toJSON built-in function works; the shell file contains all parameters that are at most one level deep, with Nix lists being converted to Bash arrays and Nix attribute sets to Bash associative arrays. All data nested deeper than one level is only included in the JSON file.

Another difference to the current state is that the variables in .attrs.sh are not exported—they are just shell variables, not environment variables. This is to be consistent with the other parameters passed to derivation, as Bash arrays and associative arrays cannot be exported. Therefore a new parameter env is added to stdenv.mkDerivation which only allows primitive types and includes logic to export these variables. This should be used in the future for env.NIX_CFLAGS_COMPILE, etc.

Enabling this feature also allows us to clean up and refactor some things in stdenv. Due to the constraints that the old method imposed, we decided to use hardeningEnable = [ ... ] and hardeningDisable = [ ... ], instead of hardening.feature = true which better represents that there is a single association between each feature and one of {true, false, default}. This would be no problem with __structuredAttrs, however. Also, the *FlagsArray parameters can be removed and replaced by the simple *Flags as they are escaped and split properly now. passAsFile is removed as the data can always be retrieved from the JSON file.

New for shell builders is that they have to source .attrs.sh mentioned above; builders in other languages are even easier, as they can simply read the structured .attrs.json and take advantage of all the nested structure rather than contorting to use the bare-bones ENV.

A few issues still need work on in Nix itself before this can be merged, though. e.g. nix-shell -A does not work yet, but generally while experimenting, the experience has been positive and seems that it will result in a cleaner and more consistent stdenv.