In the past year some Mayflower colleagues have started using and contributing to NixOS, a purely functional GNU/Linux distribution that combines package and configuration management. We decided that we would give it a try in production but stumbled upon some issues that had to be resolved first. We have added new packages, services and fixed up some internal. Due to this work two colleagues have even gained commit access in the process.

One of these fixes is adding hardening compiler flags. Currently packages in NixOS and specifically nixpkgs (the package set) are not built with standard compiler hardening flags by default. In contrast other Linux distributions have been adding those for years. These flags provide some basic safety guarantees like stack smashing protection and runtime bounds checking to detect classic memory attacks like buffer overflows that could lead to code execution.

We modified the nixpkgs standard environment, specifically the cc-wrapper, to include some hardening flags by default back in December 2015. After discovering that a lot of packages would fail to build, we started to either disable specific flags for each failing program or add patches to fix underlying problems. In February 2016 we opened a work in progress pull request and started using the changes on our laptops and servers. Until now the pull request has not been merged due to a lot of packages failing to build, that gradually had to be fixed one at a time. We are right now in the process of fixing the few remaining issues to have it merged for the upcoming 16.09 NixOS release.

As this change triggers a complete rebuild of all packages, we could not use the official package cache anymore. Thus we set up our own Hydra, the Nix continuous build platform, and as we were already rebuilding the complete package set ourselves, we switched to building all packages requiring OpenSSL with LibreSSL instead.

Hardening flags enabled by default

format
Adds the -Wformat -Wformat-security -Werror=format-security compiler options. At present, this warns about calls to printf and scanf functions where the format string is not a string literal and there are no format arguments, as in printf(foo). This may be a security hole if the format string came from untrusted input and contains %n.

stackprotector
Adds the -fstack-protector-strong --param ssp-buffer-size=4 compiler options. This adds safety checks against stack overwrites rendering many potential code injection attacks into aborting situations. In the best case this turns code injection vulnerabilities into denial of service or into non-issues (depending on the application).

fortify
Adds the -O2 -D_FORTIFY_SOURCE=2 compiler options. During code generation the compiler knows a great deal of information about buffer sizes (where possible), and attempts to replace insecure unlimited length buffer function calls with length-limited ones. This is especially useful for old, crufty code. Additionally, format strings in writable memory that contain ‚%n‘ are blocked. If an application depends on such a format string, it will need to be worked around.

Additionally, some warnings are enabled which might trigger build failures if compiler warnings are treated as errors in the package build.

pic
Adds the -fPIC compiler options. This options adds support for position independent code in shared libraries and thus making Address Space Layout Randomization (ASLR) possible.

Most notably, the Linux kernel, kernel modules and other code not running in an operating system environment like boot loaders will not build with PIC enabled. The compiler will is most cases complain that PIC is not supported for a specific build.

strictoverflow
Signed integer overflow is undefined behaviour according to the C standard. If it happens, it is an error in the program as it should check for overflow before it can happen, not afterwards. GCC provides built-in functions to perform arithmetic with overflow checking, which are correct and faster than any custom implementation. As a workaround, the option -fno-strict-overflow makes gcc behave as if signed integer overflows were defined.

This flag should not trigger any build or runtime errors.

relro
Adds the -z relro linker option. During program load, several ELF memory sections need to be written to by the linker, but can be turned read-only before turning over control to the program. This prevents some GOT (and .dtors) overwrite attacks, but at least the part of the GOT used by the dynamic linker (.got.plt) is still vulnerable. This flag can break dynamic shared object loading.

For instance, the module systems of Xorg and OpenCV are incompatible with this flag. In almost all cases the bindnow flag must also be disabled and incompatible programs typically fail with similar errors at runtime.

bindnow
Adds the -z bindnow linker option. During program load, all dynamic symbols are resolved, allowing for the complete GOT to be marked read-only (due to relro). This prevents GOT overwrite attacks. For very large applications, this can incur some performance loss during initial load while symbols are resolved, but this should not be an issue for daemons.This flag can break dynamic shared object loading.

For instance, the module systems of Xorg and PHP are incompatible with this flag. Programs incompatible with this flag often fail at runtime due to missing symbols.

Hardening flags disabled by default

pie
Adds the -fPIE compiler and -pie linker options. Position Independent Executables (PIE) are needed to take advantage of Address Space Layout Randomization (ASLR), supported by modern kernel versions. While ASLR can already be enforced for data areas in the stack and heap (brk and mmap), the code areas must be compiled as position-independent. Shared libraries already do this with the pic flag, so they gain ASLR automatically, but binary .text regions need to be build with pie to gain ASLR. When this happens, ROP attacks are much harder since there are no static locations to bounce off of during a memory corruption attack.

How to toggle flags in package definitions

The hardening flags above can be toggled using the stdenv.mkDerivation parameters hardeningDisable and hardeningEnable.

For instance, to enable PIE hardening for the GNU hello package:

 1{ stdenv, fetchurl }:
 2
 3stdenv.mkDerivation rec {
 4  name = "hello-2.10";
 5
 6  src = fetchurl {
 7    url = "mirror://gnu/hello/${name}.tar.gz";
 8    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
 9  };
10
11  hardeningEnable = [ "pie" ];
12}

libdwg needs format hardening disabled because it uses unsafe format strings:

 1{ stdenv, fetchurl, indent }:
 2
 3stdenv.mkDerivation {
 4  name = "libdwg-0.6";
 5
 6  src = fetchurl {
 7    url = "mirror://sourceforge/libdwg/${name}.tar.bz2";
 8    sha256 = "0l8ks1x70mkna1q7mzy1fxplinz141bd24qhrm1zkdil74mcsryc";
 9  };
10
11  nativeBuildInputs = [ indent ];
12
13  hardeningDisable = [ "format" ];
14}

There is also a special all flag for hardeningDisable to disable all hardening flags but it is strongly recommended to only disable incompatible hardening flags one by one.

Moving forward

In the next weeks we will continue to fix all remaining issues to get the changes into the 16.09 release. Even though we have fixed all build failures, we probably have not detected all runtime issues yet. Most of the time the runtime issues are indicators of actual security flaws in the packaged software.

There is more hardening in NixOS to be done. There is a general hardening issue on Github. In particular we will look at service hardening and establishing processes to improve security updates in the near future.