A while back I wrote about how we use Vault in Kubernetes and recently a good samaritan brought it to my attention that so much has changed with our implementation that I should update/rewrite a post about our current setup.
Again congrats to Martin Devlin for all the effort he has put in. Amazing engineer.
So here goes. Please keep in mind, I’ve intentionally abstracted various things out of these files. You won’t be able to copy and paste to stand up your own. This is meant to provide insight into how you could go about it.
If it has ###SOMETHING### its been abstracted.
If it has %%something%%, we use another script that replaces those for real values. This will be far less necessary in Kubernetes 1.3 when we can begin using variables in config files. NICE!
Also understand, I am not providing all of the components we use to populate policies, create tokens, initialize Vault, load secrets etc etc. Those are things I’m not comfortable providing at this time.
Here is our most recent Dockerfile for Vault:
FROM alpine:3.2
MAINTAINER Martin Devlin <martin.devlin@pearson.com>
ENV VAULT_VERSION 0.5.2
ENV VAULT_HTTP_PORT ###SOME_HIGH_PORT_HTTP###
ENV VAULT_HTTPS_PORT ###SOME_HIGH_PORT_HTTPS###
COPY config.json /etc/vault/config.json
RUN apk --update add openssl zip\
&& mkdir -p /etc/vault/ssl \
&& wget http://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip \
&& unzip vault_${VAULT_VERSION}_linux_amd64.zip \
&& mv vault /usr/local/bin/ \
&& rm -f vault_${VAULT_VERSION}_linux_amd64.zip
EXPOSE ${VAULT_HTTP_PORT}
EXPOSE ${VAULT_HTTPS_PORT}
COPY /run.sh /usr/bin/run.sh
RUN chmod +x /usr/bin/run.sh
ENTRYPOINT ["/usr/bin/run.sh"]
CMD []
Same basic docker image build on Alpine. Not too much has changed here other than some ports, version of Vault and we have added a config.json so we can dynamically create the consul backend and set our listeners.
Lets have a look at config.json
### Vault config
backend "consul" {
address = "%%CONSUL_HOST%%:%%CONSUL_PORT%%"
path = "vault"
advertise_addr = "https://%%VAULT_IP%%:%%VAULT_HTTPS_PORT%%"
scheme = "%%CONSUL_SCHEME%%"
token = %%CONSUL_TOKEN%%
tls_skip_verify = 1
}
listener "tcp" {
address = "%%VAULT_IP%%:%%VAULT_HTTPS_PORT%%"
tls_key_file = "/###path_to_key##/some_vault.key"
tls_cert_file = "/###path_to_crt###/some_vault.crt"
}
listener "tcp" {
address = "%%VAULT_IP%%:%%VAULT_HTTP_PORT%%"
tls_disable = 1
}
disable_mlock = true
We dynamically configure config.json with
CONSUL_HOST = Kubernetes Consul Service IP
CONSUL_PORT = Kubernetes Consul Service Port
CONSUL_SCHEME = HTTPS OR HTTP for connection to Consul
CONSUL_TOKEN = ACL TOKEN to access Consul
VAULT_IP = VAULT_IP
VAULT_HTTPS_PORT = Vault HTTPS Port
VAULT_HTTP_PORT = Vault HTTP Port
run.sh has changed significantly however. We’ve added ssl support and cleaned things up a bit. We are working on another project to transport the keys external to the cluster but for now this is a manual process after everything is stood up. Our intent moving forward is to store this information in what we call ‘the brain’ and provide access to each key to different people. Maybe sometime in the next few months I can talk more about that.
#!/bin/sh
if [ -z ${VAULT_HTTP_PORT} ]; then
export VAULT_HTTP_PORT=###SOME_HIGH_PORT_HTTP###
fi
if [ -z ${VAULT_HTTPS_PORT} ]; then
export VAULT_HTTPS_PORT=###SOME_HIGH_PORT_HTTPS###
fi
if [ -z ${CONSUL_SERVICE_HOST} ]; then
export CONSUL_SERVICE_HOST="127.0.0.1"
fi
if [ -z ${CONSUL_SERVICE_PORT_HTTPS} ]; then
export CONSUL_HTTP_PORT=SOME_CONSUL_PORT
else
export CONSUL_HTTP_PORT=${CONSUL_SERVICE_PORT_HTTPS}
fi
if [ -z ${CONSUL_SCHEME} ]; then
export CONSUL_SCHEME="https"
fi
if [ -z ${CONSUL_TOKEN} ]; then
export CONSUL_TOKEN=""
else
CONSUL_TOKEN=`echo ${CONSUL_TOKEN} | base64 -d`
fi
if [ ! -z "${VAULT_SSL_KEY}" ] && [ ! -z "${VAULT_SSL_CRT}" ]; then
echo "${VAULT_SSL_KEY}" | sed -e 's/\"//g' | sed -e 's/^[ \t]*//g' | sed -e 's/[ \t]$//g' > /etc/vault/ssl/vault.key
echo "${VAULT_SSL_CRT}" | sed -e 's/\"//g' | sed -e 's/^[ \t]*//g' | sed -e 's/[ \t]$//g' > /etc/vault/ssl/vault.crt
else
openssl req -x509 -newkey rsa:2048 -nodes -keyout /etc/vault/ssl/vault.key -out /etc/vault/ssl/vault.crt -days 365 -subj "/CN=vault.kube-system.svc.cluster.local"
fi
export VAULT_IP=`hostname -i`
sed -i "s,%%CONSUL_HOST%%,$CONSUL_SERVICE_HOST," /etc/vault/config.json
sed -i "s,%%CONSUL_PORT%%,$CONSUL_HTTP_PORT," /etc/vault/config.json
sed -i "s,%%CONSUL_SCHEME%%,$CONSUL_SCHEME," /etc/vault/config.json
sed -i "s,%%CONSUL_TOKEN%%,$CONSUL_TOKEN," /etc/vault/config.json
sed -i "s,%%VAULT_IP%%,$VAULT_IP," /etc/vault/config.json
sed -i "s,%%VAULT_HTTP_PORT%%,$VAULT_HTTP_PORT," /etc/vault/config.json
sed -i "s,%%VAULT_HTTPS_PORT%%,$VAULT_HTTPS_PORT," /etc/vault/config.json
cmd="vault server -config=/etc/vault/config.json $@;"
if [ ! -z ${VAULT_DEBUG} ]; then
ls -lR /etc/vault
cat /###path_to_/vault.crt###
cat /etc/vault/config.json
echo "${cmd}"
sed -i "s,INFO,DEBUG," /etc/vault/config.json
fi
## Master stuff
master() {
vault server -config=/etc/vault/config.json $@ &
if [ ! -f ###/path_to/something.txt### ]; then
export VAULT_SKIP_VERIFY=true
export VAULT_ADDR="https://${VAULT_IP}:${VAULT_HTTPS_PORT}"
vault init -address=${VAULT_ADDR} > ###/path_to/something.txt####
export VAULT_TOKEN=`grep 'Initial Root Token:' ###/path_to/something.txt### | awk '{print $NF}'`
vault unseal `grep 'Key 1:' ###/path_to/something.txt### | awk '{print $NF}'`
vault unseal `grep 'Key 2:' ###/path_to/something.txt### | awk '{print $NF}'`
vault unseal `grep 'Key 3:' ###/path_to/something.txt### | awk '{print $NF}'`
fi
}
case "$1" in
master) master $@;;
*) exec vault server -config=/etc/vault/config.json $@;;
esac
Alright now that we have our image, lets have a look at how we deploy it. Now that we have SSL in place and we’ve got some good ACLs we expose Vault external to the Cluster but still internal to our environment. This allows us to automatically populate Vault with secrets, keys and certs from various sources while still providing a high level of security.
service.yaml
apiVersion: v1
kind: Service
metadata:
name: vault
namespace: kube-system
labels:
name: vault
spec:
ports:
- name: vaultport
port: ###SOME_VAULT_PORT_HERE###
protocol: TCP
targetPort: ###SOME_VAULT_PORT_HERE###
- name: vaultporthttp
port: 8200
protocol: TCP
targetPort: 8200
selector:
app: vault
Ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: vault
namespace: kube-system
labels:
ssl: "true"
spec:
rules:
- host: ###vault%%ENVIRONMENT%%.somedomain.com###
http:
paths:
- backend:
serviceName: vault
servicePort: ###SOME_HIGH_PORT_HTTPS###
path: /
replicationcontroller.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: vault
namespace: kube-system
spec:
replicas: 3
selector:
app: vault
template:
metadata:
labels:
pool: vaultpool
app: vault
spec:
containers:
- name: vault
image: '###BUILD_YOUR_IMAGE_AND_PUT_IT_HERE###'
imagePullPolicy: Always
env:
- name: CONSUL_TOKEN
valueFrom:
secretKeyRef:
name: vault-mgmt
key: vault-mgmt
- name: "VAULT_DEBUG"
value: "false"
- name: "VAULT_SSL_KEY"
valueFrom:
secretKeyRef:
name: ###MY_SSL_KEY###
key: ###key###
- name: "VAULT_SSL_CRT"
valueFrom:
secretKeyRef:
name: ###MY_SSL_CRT###
key: ###CRT###
readinessProbe:
httpGet:
path: /v1/sys/health
port: 8200
initialDelaySeconds: 10
timeoutSeconds: 1
ports:
- containerPort: ###SOME_VAULT_HTTPS_PORT###
name: vaultport
- containerPort: 8200
name: vaulthttpport
nodeSelector:
role: minion
WARNING: Add your volume mounts and such for the Kubernetes Secrets associated with the vault ssl crt and key.
As you can see, significant improvements made to how we build Vault in Kubernetes. I hope this helps in your own endeavors.
Feel free to reach out on Twitter or through the comments.
