From Vagrant to NixOps
By Hendrik Schaeidt | Fri, 10 Mar 2017
I have been following the development of NixOps for some months. NixOps is a cloud deployment tool using nix, the functional package manager for unix systems. Nix makes it very intuitive to define absolute package dependencies. No more thinking and guessing about required runtime dependencies.
NixOps supports deploying to different platforms. Bare-metal, cloud, and even virtual environments like virtualbox work out of the box. I have worked in many projects using vagrant. Out of curiosity I migrated an existing vagrant project using wasted (Web Application STack for Extreme Development) to nix and NixOps.
This post is a walkthrough to configure a symfony2 project with nginx, mysql, and php-fpm from scratch.
Demo
In the demo project you can see the final setup using the code from this blog post.
Demo link: https://github.com/hschaeidt/cbase
Preparations
Before we get started we need following tools:
Also ensure the following settings in virtualbox:
- open general settings
- navigate to „Network“ section
- in „Network“ switch to „Host-only Networks“ tab
- if no network with the name „vboxnet0“ exists, create one
We are ready to go now.
Overview
The first setup we want to achieve is development machine for developers. The next goal will be to have the exact same configuration for production deployments using the exact same tool, NixOps, but with another provider – maybe ec2, maybe hetzner bare-metal, maybe azure, … – that depends on your use case. But this will be too much for this post, so let’s first focus on our development machine using virtualbox.
What we have:
- a local symfony project on the host machine (the computer you are hacking on right now)
- NixOps installed
Well, that’s all we actually need.
What our final setup will look like:
- the host machine’s symfony2 project is mounted in the virtual machine provisioned by NixOps
- all the project’s dependencies are available within the virtualbox development machine
- php
- composer
- mysql
- nginx
- it is not required to install those tools on the host machine
Hands on
Now let’s start by creating a new folder for our new configuration. Let’s say ./server.
We start by declaring a virtualbox configuration file in nix. I will paste the entire configuration in here as it is pretty easy to follow up. If you are not yet familiar enough with the nix language, try out a tour of nix: https://nixcloud.io/tour/?id=1 – it’s concise, straightforward, and afterwards there are no more surprises in the config syntax.
Let’s say we have a file called ./server/cbase-vbox.nix
1let
2 cbase = # section 1
3 { config, pkgs, ... }:
4 { deployment.targetEnv = "virtualbox"; # section 2
5 deployment.virtualbox = {
6 memorySize = 1024;
7 headless = true;
8 };
9
10 virtualisation.virtualbox.guest.enable = true; # section 3
11
12 deployment.virtualbox.sharedFolders = { # section 4
13 cbase = {
14 hostPath = "/Users/hschaeidt/Projects/github/hschaeidt/cbase";
15 readOnly = false;
16 };
17 };
18
19 fileSystems."/var/www/cbase" = { # section 5
20 device = "cbase";
21 fsType = "vboxsf";
22 options = [ "uid=33" "gid=33" ];
23 };
24 };
25in
26{
27 network.enableRollback = true; # section 6
28 inherit cbase; # section 7
29}
From here on I will call my virtual development machine the „target machine“. Also my local laptop will be called „host machine“ from here on.
I split the configuration in 7 sections I will describe below:
section 1
1cbase = { config, pkgs, ... }:
Defines the server name exposed to NixOps. This name will be used to ssh into the target machine later on.
section 2
1deployment.targetEnv = "virtualbox";
2deployment.virtualbox = {
3 memorySize = 1024;
4 headless = true;
5};
Defines the deployment target – here virtualbox – below some virtualbox specific settings like the memory available to the target machine.
section 3
1virtualisation.virtualbox.guest.enable = true;
Virtualbox guest additions are required on the target machine in order to properly mount the folder in section 4 and section 5.
section 4
1deployment.virtualbox.sharedFolders = {
2 cbase = {
3 hostPath = "/Users/hschaeidt/Projects/github/hschaeidt/cbase";
4 readOnly = false;
5 };
6};
Declaring the virtualbox shared folders. The key of the set is cbase, note that this must equal the device name used in section 5.
The host machine path points to the checked out git project, change it to your needs.
Read-only is set to false because we want to do some operations like composer install within the target machine etc.
section 5
1fileSystems."/var/www/cbase" = {
2 device = "cbase";
3 fsType = "vboxsf";
4 options = [ "uid=33" "gid=33" ];
5};
Defining the filesystem mount from the previously declared sharedFolder from the host machine.
/var/www/cbase will be mounted on the target machine using the virtualbox shared folder containing the git symfony project.
Note: The uid and gid will be the one of the www-data user that will be created in the next step. This makes sure symfony can write its caches.
section 6
1network.enableRollback = true;
Enabling the possibility to roll back to a previous configuration using nixops rollback command. This can also be omitted for the development machine.
section 7
1inherit cbase;
Basically this is the same as writing cbase = cbase;
Apply the declared variable cbase to the nix config.
Now we have defined our target machine we want to use for development. Let’s keep going by creating another file configuring this machine.
Setting up php, nginx and mysql
Let’s get started by creating a new file called ./server/cbase.nix. This file is way bigger, but I think it’s important to paste it entirely first and going through it in a second step.
1{
2 network.description = "mtg card database";
3
4 cbase = { config, pkgs, ... }: # section 1
5 let
6 fcgiSocket = "/run/phpfpm/nginx";
7 projectName = "cbase";
8 realPathRoot = "/var/www/cbase";
9 runUser = "www-data";
10 runGroup = "www-data";
11 in
12 {
13 networking.firewall.allowedTCPPorts = [ 80 443 ]; # section 2
14
15 environment.systemPackages = with pkgs; [ # section 3
16 ag
17 vim
18 php
19 phpPackages.composer
20 ];
21
22 services.phpfpm.poolConfigs.nginx = '' # section 4
23 listen = ${fcgiSocket}
24 listen.owner = ${runUser}
25 listen.group = ${runGroup}
26 listen.mode = 0660
27 user = ${runUser}
28 pm = dynamic
29 pm.max_children = 75
30 pm.start_servers = 10
31 pm.min_spare_servers = 5
32 pm.max_spare_servers = 20
33 pm.max_requests = 500
34 '';
35
36 services.nginx = { # section 5
37 enable = true;
38 recommendedOptimisation = true;
39 recommendedTlsSettings = true;
40 recommendedGzipSettings = true;
41 recommendedProxySettings = true;
42 user = runUser;
43 group = runGroup;
44 virtualHosts."localhost" = {
45 extraConfig = ''
46 root ${realPathRoot}/web;
47
48 location / {
49 try_files $uri /app.php$is_args$args;
50 }
51
52 # DEV
53 location ~ ^/(app_dev|config)\.php(/|$) {
54 # this links to the defined upstream in 'appendHttpConfig'
55 fastcgi_pass phpfcgi;
56 fastcgi_split_path_info ^(.+\.php)(/.*)$;
57 include ${pkgs.nginx}/conf/fastcgi_params;
58 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
59 fastcgi_param DOCUMENT_ROOT $document_root;
60 }
61
62 # PROD
63 location ~ ^/app\.php(/|$) {
64 # this links to the defined upstream in 'appendHttpConfig'
65 fastcgi_pass phpfcgi;
66 fastcgi_split_path_info ^(.+\.php)(/.*)$;
67 include ${pkgs.nginx}/conf/fastcgi_params;
68 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
69 fastcgi_param DOCUMENT_ROOT $document_root;
70 internal;
71 }
72
73 location ~ \.php$ {
74 return 404;
75 }
76 '';
77 };
78 appendHttpConfig = ''
79 upstream phpfcgi {
80 server unix:${fcgiSocket};
81 }
82 '';
83 };
84
85 services.mysql = { # section 6
86 enable = true;
87 package = pkgs.mysql;
88 initialDatabases = [ { name = "cbase"; schema = ./cbase.sql; } ];
89 };
90
91 users.extraUsers."${runUser}" = { # section 7
92 uid = 33;
93 group = "${runGroup}";
94 home = "/var/www";
95 createHome = true;
96 useDefaultShell = true;
97 };
98 users.extraGroups."${runUser}".gid = 33;
99 };
100}
The configuration is again split into 7 sections we will go through now.
section 1
1cbase = { config, pkgs, ... }:
2let
3 fcgiSocket = "/run/phpfpm/nginx";
4 projectName = "cbase";
5 realPathRoot = "/var/www/cbase";
6 runUser = "www-data";
7 runGroup = "www-data";
8in
9{
10 ...
11}
The cbase variable here again corresponds to the server name exposed to nixops. It has to correspond to the one defined in the previous configuration file (see section 1 from the previous configuration).
Additionally, we define some other variables we will use for convenience within the let in block.
section 2
1networking.firewall.allowedTCPPorts = [ 80 443 ];
We open the ports 80 and 443 on our application server. This is necessary for the nginx to be available to the host machine.
section 3
1environment.systemPackages = with pkgs; [
2 ag
3 vim
4 php
5 phpPackages.composer
6];
Declaring the packages available for execution from all target machines users (including www-data user). We actually only need the php and composer packages for development. Adapt to your needs on the target machine.
section 4
1services.phpfpm.poolConfigs.nginx = ''
2 listen = ${fcgiSocket}
3 listen.owner = ${runUser}
4 listen.group = ${runGroup}
5 listen.mode = 0660
6 user = ${runUser}
7 ...
8'';
Enables the php-fpm service on the target machine. The user should be the same as for nginx. For detailed pool configuration options refer to http://php.net/manual/en/install.fpm.configuration.php
section 5
1services.nginx = {
2 enable = true;
3 recommendedOptimisation = true;
4 recommendedTlsSettings = true;
5 recommendedGzipSettings = true;
6 recommendedProxySettings = true;
7 user = "${runUser}";
8 group = "${runGroup}";
9 virtualHosts."localhost" = {
10 extraConfig = ''
11 root ${realPathRoot}/web;
12
13 location / {
14 try_files $uri /app.php$is_args$args;
15 }
16
17 # DEV
18 location ~ ^/(app_dev|config)\.php(/|$) {
19 # this links to the defined upstream in 'appendHttpConfig'
20 fastcgi_pass phpfcgi;
21 fastcgi_split_path_info ^(.+\.php)(/.*)$;
22 include ${pkgs.nginx}/conf/fastcgi_params;
23 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
24 fastcgi_param DOCUMENT_ROOT $document_root;
25 }
26
27 # PROD
28 location ~ ^/app\.php(/|$) {
29 # this links to the defined upstream in 'appendHttpConfig'
30 fastcgi_pass phpfcgi;
31 fastcgi_split_path_info ^(.+\.php)(/.*)$;
32 include ${pkgs.nginx}/conf/fastcgi_params;
33 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
34 fastcgi_param DOCUMENT_ROOT $document_root;
35 internal;
36 }
37
38 location ~ \.php$ {
39 return 404;
40 }
41 '';
42 };
43 appendHttpConfig = ''
44 upstream phpfcgi {
45 server unix:${fcgiSocket};
46 }
47 '';
48};
Enables and configures the nginx service. Please note the variables used within the configuration file. It is mostly the recommended minimal nginx configuration file as described in https://www.nginx.com/resources/wiki/start/topics/recipes/symfony/
section 6
1services.mysql = {
2 enable = true;
3 package = pkgs.mysql;
4 initialDatabases = [ { name = "cbase"; schema = ./cbase.sql; } ];
5};
Enables the mysql service. Note the inital database configuration. The minimal required configuration in order to get the databases available after deployment is:
1--
2-- Current Database: `cbase`
3--
4
5/*!40000 DROP DATABASE IF EXISTS `cbase`*/;
6
7CREATE DATABASE /*!32312 IF NOT EXISTS*/ `cbase` /*!40100 DEFAULT CHARACTER SET utf8 */;
section 7
1users.extraUsers."${runUser}" = {
2 uid = 33;
3 group = "${runGroup}";
4 home = "/var/www";
5 createHome = true;
6 useDefaultShell = true;
7};
8users.extraGroups."${runUser}".gid = 33;
Creates the www-data user on the target machine. We enable the default shell here, because we will work with this user within our target machine. Note that the gid and uid should correspond to the ones defined in section 5 of the previous configuration file.
That covers all we need to do to describe our server setup.
Deploying the development machine
nixops deployment
Okay, here we go. We just finished our minimal symfony configuration file. It’s time to test this setup now. Just a few steps to go.
First we have to create the nixops deployment configuration.
1nixops create --deployment cbase ./server/cbase-vbox.nix ./server/cbase.nix
Now we can test if the machine was created succesfully.
1nixops list
2
3
4# Should output following similar output
5# +------+---------+------------------------+------------+------------+
6# | UUID | Name | Description | # Machines | Type |
7# +------+---------+------------------------+------------+------------+
8# | ... | cbase | mtg card database | 1 | virtualbox |
9# +------+---------+------------------------+------------+------------+
And finally deploying it to virtualbox.
1nixops deploy --deployment cbase
Now we have a virtualbox able to serve our symfony application.
1nixops info --deployment cbase
2
3# Should output following similar output
4# +-------+-----------------------+------------+------------------+----------------+
5# | Name | Status | Type | Resource Id | IP address |
6# +-------+-----------------------+------------+------------------+----------------+
7# | cbase | Starting / Up-to-date | virtualbox | nixops-...-cbase | 192.168.56.101 |
8# +-------+-----------------------+------------+------------------+----------------+
symfony setup
We need to execute composer install from within the target machine to download all dependencies.
1# ssh into the target machine
2nixops ssh --deployment cbase cbase
3
4# change to www-data user
5su - www-data
6
7# navigate to document root
8cd /var/www/cbase
9
10# install dependencies
11composer install
Testing the setup
Now all we have to do is look up the IP-address from the previously executed nixops info --deployment cbase command, which in my case is 192.168.56.101.
And navigating to http://192.168.56.101/app_dev.php we can see our symfony application running.
Happy hacking!