Ansible Tower 3.1 High Availability maintenance

 August 18, 2017

Around one year ago, I did a post around Ansible Tower High Availability maintenance, but in the mean time many things changed and that post is not up to date anymore, so I decided to create a new one that covers the same topic but for Ansible Tower 3.1.

From Ansible Tower 3.1 we lost the distinction of Primary Ansible Tower and Secondary Ansible Tower. That concept was related to the fact that the Secondary Ansible Towers were in a hot-standby mode. Since Ansible Tower 3.1, we have active-active clustering and therefore all Ansible Towers in your cluster are always active and there is no distinction between them.

The situation of PostgreSQL replication has not changed since my previous post, but if you want to learn more around how to make it Highly Available, you can read my post about PostgreSQL streaming replication for Ansible Tower.

Let’s see how this new clustering system changes operations.

Add a new Ansible Tower to the cluster

Although the easiest way to add Ansible Towers is during the installation, it may be useful to add additional Ansible Towers to the cluster in a different moment. To do so, firstly re-run the installer adding the new machine in the tower section of the Ansible Inventory, after ensuring that the machine running the installer can actually access the newly created machine via SSH using SSH keys.

At this point, the machine should be part of the cluster and working properly.

As you can see, the process is much easier then in the past. This is mainly due to the fact that many configurations that historically were conserved in configuration files got moved to the database, making them immediately available to all cluster nodes.

Remove an Ansible Tower from the cluster

There are cases where an Ansible Tower needs to be removed from the cluster. To do so, execute the following command on an Ansible Tower:

tower-manage remove_instance --hostname HOSTHERE

Even if a machine died “naturally”, it is very important to remove it from the cluster so that the cluster configuration is kept clean and functional.

This procedure is the same as it was in the past, with the difference that we can remove any Ansible Tower now, while historically we could only remove Secondary Ansible Towers.

Other operations

Due to the fact that we lost the distinction of Primary Ansible Tower and Secondary Ansible Tower, we do not have any other kind of procedure to manage our Highly Available cluster.

I hope that you’ll enjoy the new clustering method. I find it way more efficient and easy to manage.

 Ansible

 Ansible Tower, Infrastructure as Code

Say something

Comments

Michelle Xie on 2021-05-04 14:20:38 UTC said:

Hello. Thanks for sharing your knowledge on Ansible Tower, very helpful for new comers that is trying to get Ansible Clusters running.

By running 3.8.x, there’s no difference of primary/secondary, jobs are distributed within cluster, e.g. 3 nodes (e.g. tower1, tower2, tower3) cluster. Admin guide shows it’s optional to have a load balance in front of cluster. We need to have a consistent service URL (tower-service.xxx.org) for user access UI, as well as API access. Would you share some best practice regarding having consistent service URL from a clustered Ansible Tower environment, whether LB is required? Thanks!

Fabio Alessandro Locati on 2021-05-04 14:40:26 UTC said:

Hi! To have a stable FQDN such as tower.example.org, you can either use a Load Balancer, or use DNS load-balancing.

If you go for the Load Balancer, this will need to be highly available and configured to properly balance between Ansible Tower nodes. Then is enough to configure the DNS resolution of the FQDN to the Load Balancer IP.

If you go for the DNS load-balancing, you just configure the DNS to resolve the FQDN to all Ansible Tower nodes’ IP address. In this way, the client will be aware of all Ansible Tower Nodes and will try lo connect to a random one. If the chosen node is down, che client will retry with a different one, and so forth until one node replies or the client tried all nodes.

The second implementation is way leaner and cleaner, but assumes that the client is properly written following the standards. If the client is a current browser or a big SDK, you can be pretty sure it’s ok. If it’s a home-grown tool, better checking that it works as expected before rolling out such an architecture.