Spinning-up your own auto-scalable Selenium Grid in Kubernetes: Part 1

Background

To maintain a home grown Selenium grid cluster is not at all an easy task due to the complexities involved in running and maintaining these hundreds of browser containers. Due to this same reason, most of the companies end up outsourcing this task to vendors like Saucelabs, Browserstack and AWS to save the time and energy of their engineers and to bring more stability to their Selenium Grid infrastructure. These vendors are really expensive and one would need to shell out hundreds of dollars to execute their UI test cases on these public cloud of browsers.

With the advent of containerisation technologies like Kubernetes(K8s) and docker it is really very easy to spin up your own auto-scalable selenium grid at a very less cost. I have specially highlighted the word “auto-scalable” which is really important attribute to have, to make it more cost effective and highly available.

This piece of blog will require some basic understanding of these technologies like K8s, Docker and Selenium Grid.

Pre-requisites

We would need a working Kubernetes cluster which could be a single node or a multi node cluster. For this blog I will be using a Minikube cluster which is the easiest and simplest way to spin up a K8s cluster on your local machine.

You can follow these steps to install the minikube. Once it is installed, you can fire this command to start the cluster.

minikube start --vm-driver=virtualbox --insecure-registry="gcr.io"

We can use “minikube status” command to know the status of the cluster but in case you are someone who need to see things visually to confirm if it is really up, then you can run this command to enable the kubernetes dashboard.

minikube dashboard

Spinning up the Selenium Hub

Here is the deployment yaml file which will actually create a deployment object to mange the Selenium hub pod and also it will create a service which will act like a load balancer for balancing the traffic coming from all the registered nodes.

Let’s apply this yaml file to our cluster.

kubectl create -f selenium-hub-deployment.yaml

To confirm if these two objects are successfully created, let’s try to query them using this command:

kubectl get all -l name=selenium-hub

Exposing the Hub to the outside world using Nodeport

Till now we have created our Selenium hub but we would need to expose it to the outside world so that our selenium tests can talk to this hub.

The easiest way to expose a service in K8s is using a node port. This will expose a port on one of the K8s cluster node to access your application.

kubectl expose deployment selenium-hub-deployment --type=NodePort --port=4444

To get the URL to access the selenium hub, use this command:

~ $ minikube service selenium-hub-deployment --url
http://199.166.00.000:31178

Open the browser and navigate to your hub console.

http://192.168.99.100:31178/grid/console

In case you want to give a private name to your hub rather than accessing it using the IP address. Kubernetes has something called Ingress controller which can be used for this purpose.

Exposing the hub to the outside world using Ingress Routes

An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.

Ingress is not enabled by default in Minikube so lets enable it first

minikube addons enable ingress

Now the ingress controller is enabled so let’s define our ingress route rule as shown in this yaml.

As seen above, the private URL to access my grid is defined as “my-selenium-grid.com”

Let’s apply this yaml to our cluster.

kubectl create -f selenium-hub-deployement-ingress.yaml

We would need to add this IP address to our /etc/hosts file.

Note: If you are running Minikube locally, use minikube ip to get the external IP. The IP address displayed within the ingress list will be the internal IP.

Now let’s try to access our hub console using our private URL: “http://my-selenium-grid.com/grid/console”

Hurray! Thats brings to the end of our Selenium hub set-up. Now, let’s move to the Selenium nodes configuration.

Deploying the Selenium Chrome Nodes

In this tutorial we will only be covering the Chrome based nodes but the same concepts can be applied to other browser nodes as well.

Here is the node deployment yaml which by default has one replica which is the minimum scale we would like to have.

One important thing needs to be noticed here, that the hub address which we have defined in one of the environment property is same as the hub service name which we have used in the hub deployment yaml. This will help us to register our nodes automatically on starting-up.

That’s the whole purpose of having the services so that we can point them by their name rather than remembering the IPs which might get change on different clusters.

Applying the node yaml to our minikube cluster.

kubectl create -f selenium-node-chrome-deployment.yaml

Let’s see the hub console now.

Awesome!! We can see one chrome node already available.

Let’s see all the Kubernetes objects we have created till now.

kubectl get all

• Two Selenium Pods: Each representing the Selenium hub and node containers respectively.
• Three Selenium Services: Two of them are used to expose the hub and node containers and one (“service/selenium-hub-deployment”) is used to expose the hub to the outside world.
• Two deployment Objects: Which make sure our deployments are always running.
• Two Replication Controllers: which make sure that the actual number of the containers are always running against the desired value which we have set for each deployment.

Scaling the Selenium Grid Manually

The best part of using the containerisation technologies like Docker and Kubernetes is that with a simple one liner command we can scale up and down our selenium grid. Let’s see how we can do this.

Take an example of scenario where you have one node running on your grid and there are two requests which arrived to your hub for the same type of browser. Then you will see that one request will be served and the other one will be queued.

To deal with this scenario we would need to spin up one more node container which can be easily done by this command.

kubectl scale deployment.apps/selenium-node-chrome-deployment --replicas=2

We simply asked kubernetes to scale the node deployment to 2 replicas, which will internally check the present deployment state and according act to resolve the difference between the present and the desired state of your deployment.

As soon we have fired this command, with in seconds we will see a new node coming up on the hub console and the queued request will be served.

Don’t you think Kubernetes have made your life much easier as a Selenium Grid administrator to monitor the load on your grid and according scaling up/down, rather than running your grid at the full capacity all the time.

Auto-Scaling the Selenium Grid

Wouldn’t it be nice if we can embed this intelligence in our selenium grid that it can automatically monitor the load on the grid and scale up/down when it is needed. This will make our grid highly available, scalable and completely independent of any manual monitoring and maintenance.

As they say “Good Things Doesn’t Come Easy” so we have to do lot of hard and complex work to build this feature in to our selenium grid. I will leave you all to digest the information given in this blog and by that time I will do that hard work for you guys and will come back with the solution in the “Part 2” of this blog series.

Part -2 is released now.

Please feel free to comment your thoughts/questions/suggestions on this blog.