We’re finally beginning to build out our production Kubernetes infrastructure at work, after some extensive testing in dev. Kubernetes relies heavily on TLS for securing communications between all of the components (quite understandably) and while you can disable TLS on many components, obviously once you get to production, you don’t really want to be doing that.
Most of the documentation shows you how to generate a self signed certficate using a CA certificate you create especially for kubernetes. Even Kelsey Hightower’s excellent “Kubernetes the Hard Way” post shows you how to generate the TLS components using a self signed CA. One of the nicest things about using Puppet is that you already have a CA set up and best of all, there’s some really nice APIs inside the puppet master/server meaning provisioning new certs for hosts is relatively straightforward. I really wanted to take advantage of this with our kubernetes setup, so I made sure etcd was using Puppet’s certs:
This works out of the box, because the certs for all 3 etcd hosts have been signed by the same CA.
Securing Kubernetes with Puppet’s Certs.
I figured it would be easy to use these certs for Kubernetes also. I set the following parameters in the API server config:
but there were a multitude of problems, the main one being that when a pod starts up, it connects to the API using the kubernetes service cluster IP. You can see this in the log messages when starting a pod:
I figured it would be easy enough to fix, I’ll just add a SAN for the puppet cert using the dns_alt_names configuration option. Unfortunately, this didn’t work, and I got the following error message:
Puppet doesn’t have an option to set IP SANS in the SSL certificate, so I had to generate the cert manually and sign it by the Puppet CA. Thankfully, this is fairly straightforward (albeit manual)
Generating Certs Manually
First, create a Kubernetes config file for OpenSSL on your puppetmaster. I created a directory /var/lib/puppet/ssl/manual_ca to do all this.
Note the two IPs here. The first is the cluster IP from the kubernetes service, you can retrieve it like so:
I also added the actual IP of the kubernetes host for some future proofing. The DNS names have been generated from the Kube DNS config, so also make sure you match that to your kube-dns name.
Next, we need to generate a CSR and a key:
Verify that your CSR has the IP SANS in it:
Now, we need to sign the cert with the Puppet CA:
This will create a cert in certs/kube-api.pem. Now verify it to ensure it looks okay:
We now have the a cert we can use for our kube-apiserver, so we just need to configure kubernetes to use it.
Configuring Kubernetes to use the certs.
Assuming you’ve copied the certs to your kubernetes master, we now need to configure k8s to use it. First, make sure you have the following config set in the apiserver:
And then configure the controller manager like so:
Restart all the k8s components, and you’re almost set.
Regenerate service account secrets
The final thing you’ll need to do is delete the service account secrets kubernetes generates on launch. The reason for this is because it use the service-account-private-key-file to generate them, and if you don’t do this you’ll get all manner of permission denied errors when launching pods. It’s easy to do this:
NOTE if you’re already running pods in your kubernetes system, this may affect them and you may want to be careful doing this. YMMV.
From here, you’re using Puppet’s SSL Certs for kubernetes.