Step-By-Step Shipyard 2 Docker Cluster
I’ve seen a few other guides written, but they are either dated or missing very important instructions.
This guide makes a few assumtions about your setup. Since we’re going to be creating a cluster, you’ll need more than 2 servers available to run as nodes. This guide will have 3 servers running as nodes, and 1 dedicated master server. You should also have the latest version of docker installed on all your servers (master and nodes) as of right now, that is Docker 1.7. This guide is written to be used with Ubuntu 14.04 but should work on any linux system with little to no changes.
In the following guide, I’ll make use of the following variables. If you see them in any instruction you should replace it with the applicable value
- $MASTER_HOSTNAME - the DNS name/IP Addresss of your Shipyard master server
- $NODE1_HOSTNAME - the DNS name/IP Addresss of your Shipyard node1 server
- $NODE2_HOSTNAME - the DNS name/IP Addresss of your Shipyard node2 server
- $NODE3_HOSTNAME - the DNS name/IP Addresss of your Shipyard node3 server
As recommended by the Docker Security Guide we’re going to want to Protect the Docker daemon socket as Shipyard will need to remotely access the Docker daemon on each node. Don’t worry about blindly following those instructions. All the relavent steps are included below with descriptions and a few tweaks to get everything working for a Shipyard cluster.
By default, Docker runs via a non-networked Unix socket. It can also optionally communicate using a HTTP socket. If you need Docker to be reachable via the network in a safe manner, you can enable TLS by specifying the tlsverify flag and pointing Docker’s tlscacert flag to a trusted CA certificate.
At the end of this section we’ll have protected the Docker daemon on each of our servers such that it will only allow connections from clients authenticated by a certificate signed by that CA. We can then provide Shipyard with the connection information and associated client certificates so that Shipyard can securely communicate with the Docker daemons on the nodes.
Here’s a quick rundown of what we’re going to be doing:
- Create a minimal openssl.cnf file with required settings
- Create a Certificate Authority keypair
- Use the CA to create a keypair for each of your Nodes
- Transfer the keys to the correct nodes
- Configure the Docker daemon on the Nodes to use TLS by default
- Setup IPTables firewall rules
Create a minimal openssl.cnf file on Master
$ ssh root@$MASTER_HOSTNAME $ nano /etc/docker/openssl.cnf [ req ] default_bits = 4096 default_keyfile = privkey.pem distinguished_name = req_distinguished_name x509_extensions = v3_ca default_md = sha1 string_mask = nombstr req_extensions = v3_req prompt = no [req_distinguished_name] countryName = US stateOrProvinceName = CA localityName = San Francisco organizationName = SparkTree Inc organizationalUnitName = Shipyard emailAddress = email@example.com [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth,serverAuth subjectKeyIdentifier = hash [ v3_ca ] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always basicConstraints = CA:true [ crl_ext ] authorityKeyIdentifier=keyid:always
Create a Certificate Authority keypair on Master
First generate CA private and public keys:
$ cd /etc/docker/ $ openssl genrsa -aes256 -out ca-key.pem 2048 $ openssl req -config openssl.cnf -new -x509 -days 3650 -key ca-key.pem -sha256 -out ca.pem
In order to protect your keys from accidental damage, you will want to remove their write permissions.
$ chmod -v 0400 ca-key.pem $ chmod -v 0444 ca.pem
Use the CA to create a keypair for your Nodes on Master
Now that we have a CA, you can create a server key and certificate signing request (CSR) for each Node. We’ll create individual folders to contain the server and client key pairs for each Node
$ mkdir /etc/docker/node1 $ mkdir /etc/docker/node2 $ mkdir /etc/docker/node3
Make sure that “Common Name” (i.e., server FQDN or YOUR name) matches the hostname you will use to connect to your Node1:
$ cd /etc/docker/node1 $ openssl genrsa -out server-key.pem 2048 $ openssl req -subj "/CN=$NODE1_HOSTNAME" -new -key server-key.pem \ -out server.csr
Next, we’re going to sign the Node1 public key with our CA:
$ openssl x509 -req -days 3650 -in server.csr -CA /etc/docker/ca.pem -CAkey /etc/docker/ca-key.pem \ -CAcreateserial -out server-cert.pem -extensions v3_req -extfile /etc/docker/openssl.cnf
For the Shipyard master to communicate with Node1 using client authentication we’ll need to create a client key and certificate signing request:
$ openssl genrsa -out key.pem 2048 $ openssl req -subj '/CN=node1_client' -new -key key.pem -out client.csr
Then sign the Node1 client key as we did the server key.
$ openssl x509 -req -days 3650 -in client.csr -CA /etc/docker/ca.pem -CAkey /etc/docker/ca-key.pem \ -CAcreateserial -out cert.pem -extensions v3_req -extfile /etc/docker/openssl.cnf
After generating cert.pem and server-cert.pem for Node1 you can safely remove the two certificate signing requests:
$ rm -v client.csr server.csr
In order to protect your keys from accidental damage, you will want to remove their write permissions. To make them only readable by you, change file modes as follows:
$ chmod -v 0400 key.pem server-key.pem
Now that we have a server and client key pair for Node1 we have to follow the above instructions 2 more times, for Node2 and Node3 inside their respective folders.
Transfer the keys to the correct nodes
Now that we have all the key pairs that we need for each Node, we need to move them to the correct servers.
$ cd /etc/docker/node1 && \ scp /etc/docker/ca.pem server-cert.pem server-key.pem root@$NODE1_HOSTNAME:/etc/docker/ $ cd /etc/docker/node2 && \ scp /etc/docker/ca.pem server-cert.pem server-key.pem root@$NODE2_HOSTNAME:/etc/docker/ $ cd /etc/docker/node3 && \ scp /etc/docker/ca.pem server-cert.pem server-key.pem root@$NODE3_HOSTNAME:/etc/docker/
Configure the Docker daemon on the Nodes to use TLS by default
Now that the server key pairs have been copied to the Nodes, we need to configure Docker to use them. First lets stop the Docker daemon on the Node.
$ ssh root@$NODE1_HOSTNAME $ service docker stop
Update your Docker daemon settings to use TLS. I’m using Ubuntu so my file is at
$ nano /etc/default/docker
And add the following line to the bottom of the file.
DOCKER_OPTS="--tlsverify -H=unix:///var/run/docker.sock -H=$NODE1_HOSTNAME:2376 --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem --label name=node1"
Now we can restart the Docker daemon on the Node
$ service docker start
Now follow the above instructions for Node2 and Node3 by replacing all instances of
$NODE1_HOSTNAME with the appropriate Node variables.
Setup IPTables firewall rules
Now that the Nodes are all correctly configured, its time to deploy Shipyard on the Master. There is a very small Docker image that will deploy and manage an entire Shipyard stack called Deploy.
$ ssh root@$MASTER_HOSTNAME $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ shipyard/deploy start
Once that’s complete lets also deploy the Shipyard CLI container
$ docker run -it --rm shipyard/shipyard-cli shipyard help
You should now be able to access your Shipyard web UI by visiting
http://$MASTER_HOSTNAME:8080 and logging in with:
username: admin password: shipyard
Register Nodes as Shipyard Engines
Now that we have a working Shipyard container, lets register our Nodes as Shipyard Engines, after logging into the Shipyard CLI.
$ docker run -it -v /etc/docker/:/home/ shipyard/shipyard-cli shipyard cli> shipyard login URL:http://$MASTER_HOSTNAME:8080 Username: admin Password: shipyard
Before we do anything lets change the insecure default Shipyard Admin account password
shipyard cli> shipyard change-password Password: <enter a new password> Confirm: <re-enter a new password>
Now lets register our Nodes
shipyard cli> shipyard add-engine --id node1 \ --addr https://$NODE1_HOSTNAME:2376 \ --label node1 \ --ssl-cert /home/node1/cert.pem \ --ssl-key /home/node1/key.pem \ --ca-cert /home/ca.pem \ --cpus 4.0 \ --memory 2048 shipyard cli> shipyard add-engine --id node2 \ --addr https://$NODE2_HOSTNAME:2376 \ --label node2 \ --ssl-cert /home/node2/cert.pem \ --ssl-key /home/node2/key.pem \ --ca-cert /home/ca.pem \ --cpus 4.0 \ --memory 2048 shipyard cli> shipyard add-engine --id node3 \ --addr https://$NODE3_HOSTNAME:2376 \ --label node3 \ --ssl-cert /home/node3/cert.pem \ --ssl-key /home/node3/key.pem \ --ca-cert /home/ca.pem \ --cpus 4.0 \ --memory 2048
Once the operation is complete, use ctrl+d to exit the CLI.
At this point you should have a working Shipyard Cluster. You should now be able to view and manage all containers deployed on your various Nodes under the Containers tab:
- https://www.ovh.com/us/g1762.orchestrating_a_cluster_of_docker_servers_with_shipyard </re-enter></enter>