• This email address is being protected from spambots. You need JavaScript enabled to view it.
    • +44 (0) 20374 57368
Wednesday, 26 September 2018 13:11

How we migrated our GKE cluster to another region

We desided for a number of business reasons to move one of our existing Kubernetes cluster new a geographic region.

The thing we were most worried about is that we had persistent volumes attached to MySQL instances for our test environments running in k8s.

There isn’t a straightforward way for this. One common way is to create a snapshot of etcd but we’re on GKE so that’s out of the question. Luckily we found Ark.

Ark is a disaster recovery tool for Kubernetes clusters. It can take backups of the whole cluster with the ability to restore it using a single command. We can even have it run on a schedule. Persistent volumes are also taken care of. It has good documentation so setting it up was almost a breeze if not because of a bug with RBAC in GKE.


A simple git clone This email address is being protected from spambots. You need JavaScript enabled to view it.:heptio/ark.git was all I did to download Ark. Its master branch is frequently updated and is not stable. The maintainers recommend checking out the latest tagged version. At this time, the latest release is v0.9.5.

Setting it up

Ark works by creating custom resources in k8s for its operations conveniently defined in a single yaml file.
I had to kubectl apply the yaml file to the American cluster and the shiny new European cluster where we’re moving into.

This is where the RBAC bug on GKE appears:
User "This email address is being protected from spambots. You need JavaScript enabled to view it." cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope: No policy matched.

To work around this I had to have my Google account granted with the cluster-admin role in both clusters:
kubectl create clusterrolebinding paul-cluster-admin-binding --clusterrole=cluster-admin --user=This email address is being protected from spambots. You need JavaScript enabled to view it.

Ironically, it spits out the same error unless you’re an account with the Owner IAM Role.

Apparently, this is a known issue on GKE:

Because of the way Container Engine checks permissions when you create a Role or ClusterRole, you must first create a RoleBinding that grants you all of the permissions included in the role you want to create. An example workaround is to create a RoleBinding that gives your Google identity a cluster-admin role before attempting to create additional Role or ClusterRole permissions. This is a known issue in the Beta release of Role-Based Access Control in Kubernetes and Container Engine version 1.6.

Cloud Storage Bucket

Apart from persistent volumes, Ark stores its backups in a cloud storage bucket. This bucket should be exclusive to Ark because each backup is stored in its own subdirectory in the bucket’s root. A service account will be needed to authorize Ark to upload files into the bucket.

Service account

I created a service account just for Ark to use. It will need read and write access to the bucket. In GKE, persistent volumes are just disks attached to the nodes so I had to give it permissions for those too. These are permissions given to the service account:


The Ark server config

At this point the bucket has been created and Ark has been allowed upload to it. Now it will need to know which bucket to use by setting the Ark Config (a custom resource defined by Ark):

# examples/gcp/00-ark-config.yaml
  name: gcp
bucket: neso-cluster-backup

The Ark server Deployment

To hand off the service account to Ark a k8s secret named cloud-credentials containing the service account key will have to be created.

# download service account key
gcloud iam service-accounts keys create ark-svc-account \
     --iam-account $SERVICE_ACCOUNT_EMAIL

# create secret
kubectl create secret generic cloud-credentials \
    --namespace heptio-ark \
    --from-file cloud=ark-svc-account

In the Ark Deployment yaml file, there wasn’t anything that needed to be changed. All that’s left to start the server is to kubectl apply the Config and the Deployment.

Generating a backup

After everything’s been set up on both clusters and the Ark client install locally. It’s time to put Ark to the test. Making sure kubectl's context was set to the US cluster, with fingers crossed we generated the backup:

$ ark backup create us-cluster --exclude-namespaces kube-system,kube-public,heptio-ark

Gave it a few minutes and then:

$ ark backup get
NAME                           STATUS      CREATED                         EXPIRES   SELECTOR
us-cluster                     Completed   2018-09-21 15:59:35 +0800 +08   30d       <none>

Restoring the backup

The backup includes all the resources from pods to ingresses. We wanted to keep the IP addresses we used in the old cluster. To free up the IP addresses, down go the ingresses in the old cluster.

Now setting the kubectl context to the new cluster in Europe. It took a while for the cluster to see the backup but it did appear eventually:

$ ark backup get
NAME                           STATUS      CREATED                         EXPIRES   SELECTOR
us-cluster                     Completed   2018-09-21 15:59:35 +0800 +08   30d       <none>

$ ark restore create --from-backup us-cluster

Ark was able to restore everything except for the persistent volumes. Our applications could not connect to the databases. Taking a closer look, it appears that Ark created the disks but they were in the region where the backups were created. The maintainers are aware of this issue and added the fix for this in V0.10.0.

We can’t wait for that release, though. We had no choice but to move the databases out of k8s. We ultimately decided to spin up a CloudSQL instance and stick our test environments’ databases there.


Ark is an awesome tool. Although the migration did not go as smooth as it should have, some good came out of it. It forced us to move our database outside of kubernetes which we shouldn’t be doing in the first place. Also, we now have regular backups of our new cluster.

Published in Innovation
Monday, 17 September 2018 10:50

The Kubernetes config

The kubernetes config is a file that kubernetes uses to store different configuration variables.

Information on things like clusters, contexts and users are all stored here. Having all of these stored in the config allows us to easily switch between any of them when working with multiple projects.

If you have already started using kubernetes, you can sneak a peek at what it looks like using the following command on your terminal:

kubectl config view

This should print out all of your kubernetes configs as a single file to the terminal.


One of the collections in the config file will be for the clusters. This contains block sequences for each of your clusters. Each block sequence will contain mapping for the name of the cluster, it will also contain mapping for the server and the certificate authority data of the cluster.


Each block sequence in the user collection contains information on the name of the user block, and then the user information. The user information contains the auth provider. Which then contains the name of the service provider, and other details like token information and cmd paths and args.


For the context collection, each block sequence contains the name of the context, the user it uses and the cluster it uses. The context collection basically pairs the users with the correct clusters.

The config file

When working with multiple clusturs, being able to quickly change between them is a time saver. This can be achieved by creating a context for each cluster in your kubernetes config.

Creating the config file

Start with creating a file named 'example-config'

apiVersion: v1
kind: Config
preferences: {}

- cluster: 
  name: client-A
- cluster:
  name: client-B

- name: developer
- name: maintainer

- context:
  name: dev-production
- context:
  name: dev-staging
- context:
  name: dev-testing
- context:
  name: dev-development
- context:
  name: maint-production

The configuration file now describes two clusters named client-A and client-B respectively. It also describes two users, developer and maintainer. Finally it describes five contexts, dev-production, dev-staging, dev-testing, dev-development, and maint-production.

For each of your cluster details, you will also need to set the server details and if your server is using SSL, the certificate. Don't worry if your server is not using SSL though, you can always set insecure-skip-tls-verify: true for your cluster.

You can do this using kubectl:

kubectl config --kubeconfig=example-config set-cluster client-A --server= --certificate-authority=/home/user/some-ca-file.crt

In the above command, we tell kubectl to edit the definition of the client-A cluser in the example-config file. We want to set the server to and we want to use the crt file at /home/user/some-ca-file.crt.

If you open up your example-config file again, you will find that your definition for cluster client-A now looks like this

- cluster:
    certificate-authority: /home/user/some-ca-file.crt
  name: client-A

To set cluster client-B, we use the same command while changing some of the variables

kubectl config --kubeconfig=example-config set-cluster client-B --server= --insecure-skip-tls-verify

The client-B cluster definition will now look like this,

    insecure-skip-tls-verify: true
  name: client-B

Now that we have successfully set the definition for our clusters, we will have to add user details to the configuration file.

The following two lines will add the details we require to our developer user and our maintainer user:

kubectl config --kubeconfig=example-config set-credentials developer --client-certificate=some-client.crt --client-key=some-client.key

kubectl config --kubeconfig=example-config set-credentials maintainer --username=user --password=some-password

The updated user collection in your example-config should now look like this:

- name: developer
    client-certificate: some-client.crt
    client-key: some-client.key
- name: maintainer
    password: some-password
    username: user

Now that we have properly defined our clusters and users, we can define the contexts. We initially created five contexts so we will have to run five commands to add all their details

kubectl config --kubeconfig=example-config set-context dev-development --cluster=client-A --namespace=development --user=developer

kubectl config --kubeconfig=example-config set-context dev-testing --cluster=client-A --namespace=testing --user=developer

kubectl config --kubeconfig=example-config set-context dev-staging --cluster=client-A --namespace=staging --user=developer

kubectl config --kubeconfig=example-config set-context dev-production --cluster=client-A --namespace=production --user=developer

kubectl config --kubeconfig=example-config set-context maint-production --cluster=client-B --namespace=production --user=maintainer

One last look into our example-config file andd we should see this:

- context:
    cluster: client-A
    namespace: development
    user: developer
  name: dev-development
- context:
    cluster: client-A
    namespace: testing
    user: developer
  name:: dev-testing
- context:
    cluster: client-A
    namespace: staging
    user: developer
  name:: dev-staging
- context:
    cluster: client-A
    namespace: production
    user: developer
  name:: dev-production
- context:
    cluster: client-B
    namespace: production
    user: maintainer
  name: maint-production

Using the config file to pick a context

Now that we have everything set up, we can set a current context to be able to switch between clusters, users and namespaces as we have previously defined. Use the following command to set the current context:

kubectl config --kubeconfig=example-config use-context dev-development

This command will set all the user details, cluster details and the namespace defined in our example-config file.

Viewing a config file in the terminal

It is also possible to view the config file in the terminal using the view command:

kubectl config --kubeconfig=example-config view

or if you only want to see the details of the current context you can use the --minify flag with the view command:

kubectl config --kubeconfig=example-config view --minify
Published in Kubernetes
Wednesday, 08 August 2018 11:08

Kubernetes Ingress Round Up

Kubernetes running on Google Cloud is our go-to deployment architecture, also known as the Google Container Engine (or GKE for short). We have helped many of our clients set up their application infrastructure using this solution with excellent results. If you’re already here then you probably know all the great features and benefits Kubernetes offers.

One question that we found ourselves asking was ‘What’s is the best ingress for Kubernetes?’. Now the answer isn’t that straightforward because of cause it entirely depends what your setup and what you're trying to achieve. So here we will take a look through four of the most popular ingresses solutions that are available.

For people that don’t know, there are basically two out of the box natively supported solutions:

We will also take a look at two other solutions on offer:

A Quick Load Balance Primer

Before we delve into the different ingress options we have on Kubernetes let’s quickly review the two types of LB we might want to use; Layer 4 (L4) and Layer 7(L7) these layers correspond to the OSI model.

Layer 4

At this layer, load balancing is relatively simple. Normally is just a service which is able to forward traffic over TCP/UDP ports to a number of consuming services. This is generally done is a round robin fashion, and employs some basic health check to determine if the service should be sent request. This layer tends to be routed by simple high-performance applications.

Layer 7

This is where the magic happens. At Layer 7 load balancing offers the ability to intelligently redirect traffic by inspecting its content. This then allows the service to optimize the way it handles traffic. Not only that but it can also manipulate the content, perform security inspection and implement access controls. With the requirement to inspect traffic content and apply rules, Layer 7 load balancing require much more resources than at Layer 4 and the applications use are highly tuned to carry out these operations.

GKE Ingress

GKEFirst, we will take a look at the GKE ingress. We found this service to work great for simple deployments and as it available straight out of the box and required very little configuration. It worked well up until the point we need to do something other than the simple host and path routing.

 Features  Limitations
  •  Supports layer 4 and 7 load balancing
  • Available straight out of the box. No additional config required.
  • Hosted outside your cluster in Google architecture.
  • Makes use of Google’s global IP system. If you need to move your cluster to a new zone you can keep your IP.
  • Only available if your deploying on Google Cloud
  • No cross namespace support – We found it would be useful to have a single LB that was able to access all our services regardless of their namespace.
  • No access control – Being able to restrict access to certain resources would be great
  • No ability to enforce SSL.
  • Limited SSL options – No ability to refine SSL config
  • Letsencrypt not natively supported by available by using Kube-Lego

NGINX Ingress

The other out of the box solution is a NGINX reverse proxy setup. This offers a lot more flexibility in configuration that the GCE ingress.


  • Deep configuration though configmap
  • SSL Enforcement via redirect
  • ModSecurity Web Application Firewall – This can stop penetration attempt fore they they even reach your application.


  • Letsencrypt not natively supported by available by using Kube-Lego


The voyager ingress is backed by the well respected HAProxy, which is known as one of the best open source load balancers available. It also come with built in support for Prometheus another great piece of soft that can provide powerful metrics on your traffic. This sounds like it could be very promising.


  • Supports layer 4 and 7 load balancing
  • Cross Namespace traffic routing
  • Has semi-native Letsencrypt which can offer
  • Managed SSL certificates resource
  • Allows for monitoring using Prometheus


Traefik is the new kid on the block. It bills itself as a modern HTTP reverse proxy and load balancer for made for deploying microservices. It’s designed to complete with the likes of NGINX and HAProxy but more lightweight and focused towards container deployments. This seems very promising and it gathering quite a community around it. Support for main backends is already on offer including Kubernetes.


  • Web UI based on AngularJS
  • Let’s Encrypt via Lego
  • Choice of monitoring options nativily supported: Prometheus, DataDog, StatsD and InfluxDB


  • While Traefik offers very rich set of feature this are not all easily accessible via the ingress config

Web Application Firewall   

 GCE IngressNGINX IngressVoyagerTræfik
Layer 7 Support
Layer 4 Support    
Native To K8s  
SSL Enforcement  
Lets Encrypt Support  
Basic Authentication  
Access Control    
Built-in Monitoring UI  

(HAProxy stats)
Prometheus support    


If your using Google Cloud and all you need is a reliable, super simple entry point then the GCE Ingress will give you everything you need with minimal config.

Traefik has a great selection of features and for a young project shows great promise. However it’s integration with kubernetes is not as tight as its competitors and this makes configuring and managing it a bit more cumbersome.

On the other hand it offers a lot of integration options with different deployment, management and monitoring platforms so If you are already using one of these then this might be the best choice for you. We will certainly be keeping a close eye on this project.

This then leaves us with NGINX and Voyager, both of these platforms are backed by industry tried and tested load balancers and offer a similar set of features. Choosing between these two is likely to come down to a mixture of the features offered being the best match for your environment and a personal preference between the two load balance technologies.

One feature that Voyager offered that appealed to us was the cross Namespace traffic routing which worked well with the way we were employing namespaces. The allowed us to limit the amount of total ingresses we had and in turn simplified our deployment.

If none of the features offered by Voyager as of any consequence to you then you most likely better of sticking with NGINX ingress as this it supported under the Kubernetes umbrella.

We have heard about some other ingresses out there if you want to explore the subject further:

Published in Kubernetes