I finally had time to setup a puppet master on my new root server (again). Puppetlabs provides great apt repositories for Ubuntu and Debian. Or you can use the older packages provided by the Debian main repository.

I’ve first added the repository and installed the puppetmaster package.

root@puppet:~# wget http://apt.puppetlabs.com/puppetlabs-release-wheezy.deb
root@puppet:~# dpkg -i puppetlabs-release-wheezy.deb
root@puppet:~# apt-get update
root@puppet:~# apt-get install puppetmaster

The package comes with a SysVinit script running puppet master with Webrick. As Webrick is only intended to be a simple server while developing Ruby projects it is not recommended for production systems. My favored choice is unicorn as a application web server, nginx as a reverse proxy and SSL endpoint and all powered by systemd as the init system. I will not cover how to setup systemd on Wheezy, I assume you have one running.

But let’s start with the puppet master. First we can disable the default SysVinit service in /etc/default/puppetmaster. Set START to no:

# Defaults for puppetmaster - sourced by /etc/init.d/puppetmaster

# Enable puppetmaster service?
# Setting this to "yes" allows the puppet master service to run.
# Setting this to "no" keeps the puppet master service from running.
#
# If you are using Passenger, you should have this set to "no."
START=no

# Startup options
DAEMON_OPTS=""

# On what port should the puppet master listen? (default: 8140)
PORT=8140

Also make sure to stop a running puppet master by invoke-rc.d puppetmaster stop.

Now we can install unicorn. Instead of installing it via Rubygems we can use the Debian provided package and therefore profit from security updates and stability from Debian:

root@puppet:~# apt-get install unicorn

Next we need a unicorn configuration file. Paste the following to /etc/puppet/unicorn.rb:

worker_processes 4

# TODO: Check if /run/puppet exists if SysVinit script was never run
listen "/run/puppet/puppetmaster.sock", :backlog => 512
pid "/run/puppet/puppetmaster.pid"

timeout 120

# TODO: Check if master could be started as puppet use by systemd
user 'puppet'

preload_app true

# TODO: Reload not working, no access or missing HOME
before_fork do |server, worker|
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

The worker_processes defines the number of worker processes unicorn will spawn. As my setup is quite tiny even one process would be enough but I like to have more ;).

Next big thing is the systemd service description. We can create our custom service in /etc/systemd/system/puppetmaster.service:

[Unit]
Description=Puppet master unicorn server

[Service]
WorkingDirectory=/var/lib/puppet
SyslogIdentifier=puppetmaster-unicorn
PIDFile=/run/puppet/puppetmaster.pid

ExecStart=/usr/bin/unicorn -c /etc/puppet/unicorn.rb /usr/share/puppet/ext/rack/config.ru
ExecStop=/bin/kill -QUIT $MAINPID
#ExecReload=/bin/kill -USR1 $MAINPID

[Install]
WantedBy=multi-user.target

We first provide a service description and define the basics, working directory, syslog identifier and PID file. Make sure the PID file matches the one in unicorn.rb.

The ExecStart defines the command systemd uses to launch unicorn. We provide the full path to unicorn, point to our configuration file and finally the path to the puppet master rack configuration that is shipped with the puppet master packages. You may need to fetch the config.ru from Puppetlabs when using Debian’s puppet packages as they may not ship this file.

The ExecStop and ExecReload options specify the commands used to send unicorn the shutdown or reload signals.

Now we can start our puppet master. First reload service descriptions systemctl --system daemon-reload and then systemctl start puppetmaster.service.

We now should have a puppet master running.

root@puppet:~# systemctl status puppetmaster.service
puppetmaster.service - Puppet master unicorn server
          Loaded: loaded (/etc/systemd/system/puppetmaster.service; disabled)
          Active: active (running) since Sun, 23 Feb 2014 15:00:33 +0000; 29s ago
        Main PID: 14201 (ruby)
          CGroup: name=systemd:/system/puppetmaster.service
                  ├ 14201 unicorn master -c /etc/puppet/unicorn.rb /usr/share/puppet/ext/rack/config.ru
                  ├ 14210 unicorn worker[0] -c /etc/puppet/unicorn.rb /usr/share/puppet/ext/rack/config.ru
                  ├ 14213 unicorn worker[1] -c /etc/puppet/unicorn.rb /usr/share/puppet/ext/rack/config.ru
                  ├ 14216 unicorn worker[2] -c /etc/puppet/unicorn.rb /usr/share/puppet/ext/rack/config.ru
                  └ 14219 unicorn worker[3] -c /etc/puppet/unicorn.rb /usr/share/puppet/ext/rack/config.ru

Feb 23 15:00:34 puppet puppet-master[14201]: Starting Puppet master version 3.4.3
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.407452 #14210]  INFO -- : worker=0 spawned pid=14210
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.407762 #14213]  INFO -- : worker=1 spawned pid=14213
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.411553 #14210]  INFO -- : worker=0 ready
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.411553 #14213]  INFO -- : worker=1 ready
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.412683 #14201]  INFO -- : master process ready
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.413157 #14216]  INFO -- : worker=2 spawned pid=14216
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.413400 #14219]  INFO -- : worker=3 spawned pid=14219
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.417493 #14216]  INFO -- : worker=2 ready
Feb 23 15:00:34 puppet puppetmaster-unicorn[14201]: I, [2014-02-23T15:00:34.417489 #14219]  INFO -- : worker=3 ready

Now we need to install and configure nginx. First install, nginx-light should be enough:

root@puppet:~# apt-get install nginx-light

We now have to add our own site, as I dedicate the whole virtual machine to puppet I will just replace nginx’s default site at /etc/nginx/sites-available/default with the following:

upstream puppetmaster {
        server unix:/var/run/puppet/puppetmaster.sock fail_timeout=0;
}

server {
        listen 8140;
        server_name _ puppet puppet.altimos.de;

        ssl on;
        ssl_session_timeout 5m;
        ssl_certificate /var/lib/puppet/ssl/certs/puppet.altimos.de.pem;
        ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppet.altimos.de.pem;
        ssl_client_certificate /var/lib/puppet/ssl/ca/ca_crt.pem;
        ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
        ssl_prefer_server_ciphers on;
        ssl_verify_client optional;
        ssl_verify_depth 1;
        root /usr/share/empty;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Client-Verify $ssl_client_verify;
        proxy_set_header X-Client-DN $ssl_client_s_dn;
        proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
        proxy_read_timeout 120;

        location / {
            proxy_pass http://puppetmaster;
            proxy_redirect off;
        }
}

The upstream must match the socket configure in unicorn.rb. I only listen on IPv4 listen 8140, and not IPv6, as the machine is public IPv6 reachable, but in a private IPv4 virtual network.

The SSL certificates and private keys are generated by puppet and named by the hostname. They should already exists as our puppet master is already running. The root directory can point to any directory and does not need to exist.

Last but not least we are redirecting all request to our puppet master upstream.

Now reload nginx and try to run a puppet agent:

root@puppet:~# invoke-rc.d nginx reload
root@puppet:~# puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /node/puppet.altimos.de [find] at :115
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /file_metadata/plugins [search] at :115
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /file_metadata/plugins [find] at :115 Could not retrieve file metadata for puppet://puppet/plugins: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /file_metadata/plugins [find] at :115
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /catalog/puppet.altimos.de [find] at :115
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /report/puppet.altimos.de [save] at :115

Oh no! Why that? I took me some time but the solution is simple. Remove the following two lines from /etc/puppet/puppet.conf:

ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

They are shipped by default and seems to be needed for passenger (another application server) but interfere with nginx and unicorn. Restart puppet master and everything should work.

root@puppet:~# systemctl restart puppetmaster.service
root@puppet:~# puppet agent --test
Info: Retrieving plugin
Info: Caching catalog for puppet.altimos.de
Info: Applying configuration version '1393168781'
Notice: Finished catalog run in 0.08 seconds

Congratulations. You now have a running puppet master using nginx, unicorn and systemd.

There are several things that can be improved:

• Unicorn hot reload (USR2) fails with Ruby not able to expand ~. Somewhat missing HOME environment in new master process.
• I haven’t tried anything with systemd service recovery.

Next step is running puppet dashboard with unicorn and nginx. See you later.