AIX configuration management with ... Something

Some thoughts on adopting configuration management for AIX

I'll just say it. Managing AIX is always something of a chore. In most of the cases I have seen, or when I have spoken to people about managing a (in many cases, aging) fleet of AIX servers, the process is largely a manual one. Admins who have grown tired of repetitive tasks developed shell scripts to deal with the tedium of configuration and patch management. But these home grown solutions are often still run manually, or at best, via cron.

I would argue that this is manageable to an extent. But there are some problems that start to make it more difficult:

  • The development of these solutions is often one-off, causing other problems down the road. What if that admin leaves the company before properly distributing their tribal knowledge? What if the complicated setup requires modification and the admin who developed it is unavailable? Volunteers to take on that thing will be non-existent. Self preservation will tell everyone to keep their distance. You know why: if you touch it YOU own it!
  • If you put ten admins in a terminal and ask them to automate something, you will get at least twelve completely different approaches, which will inevitably create friction. Because, you know, unix people.
  • Each admin will use their favorite approach in the end. The lack of consistency will create problems when the fleet starts to grow too large.
  • When we create things, we call them children, and we do not want to see them replaced. Even when they become more of a hassle than they are really worth, we hold onto the precious little bundle of (what used to be) joy.

The decision is simple, we have to adopt modern configuration management to survive. The good news is that these problems are not specific to AIX, and therefore have some pretty well defined solutions. Over the last decade or so, several configuration management platforms have been brought to maturity in the market. They are born out of the need for something that solves the above problems, is extensible, and works in a cross platform fleet. Their names are very recognizable: Ansible, Salt, Puppet, Chef, etc.

My specific exposure is with the product Chef. What is best for you or your situation may be different. However, I believe some very basic issues have to be kept in mind when searching for a solution to the main problem that is configuration management of AIX in a modern world.

The issues

Issue 1:
There is, in my opinion, one main issue with adopting any kind of configuration management platform for use with AIX: There is a serious lack of people talking about the subject! For instance, at the moment I'm writing this, searching Google for information about configuration management on AIX vs the same search for linux breaks down like this:

  • "ibm aix configuration management" yields ~659,000 results
  • "linux configuration management" yields ~3,300,000 results

While those might not be alarming on their own, the problem becomes readily apparent when the first three pages (the only important ones!) are extremely unhelpful when it comes to AIX and the search for a real configuration management solution, most especially if you are interested in any kind of cross-platform tools.

Next time you are talking to someone with system administration experience, bring up AIX. You are likely to get a quizical look back. The raw truth is that AIX is not as popular as it once was, and is not growing in the community at large.

Luckily, most of the tools that spring to mind when we think of configuration management (Ansible, Puppet, Chef, etc) all work well with AIX. Even if they are missing some desired functionality in regards to AIX integration, they are built in such a way to allow some fairly simple code to expand them where needed.

Issue 2:
A close second to the above issue is it's kissing cousin: Even when you have chosen a platform, finding people using it with AIX can be quite a chore.

Be the change you want to see in the world. As with any community, if you want to have a discussion about something, you might have to bring it up yourself. This seems to hold true when discussing AIX within the ecosystems of each product. During my searches, the question "does your product work with AIX out of the box", was usually met with a quick yes. Any product that you should be considering will treat the AIX platform as a first class citizen. If you do run into a company that responds more like: "Yes, we have seen it being used in a few cases." Just walk away. However, even when AIX is a first class citizen to the company, and they have spent actual time considering and providing functionality for it, that does not mean that you will find a large population of users like yourself. So the solution? Just be prepared to have to drive the conversation!

Issue 3:
We get too hung up on what is provided out of the box for AIX. In todays world any product you choose will have to be cross platform. There is no way around it. The days of having different tools for AIX and linux is, or should be, over. In most cases the AIX footprint is shrinking, and the linux footprint is growing. Products on the market will naturally focus on the larger audience.

This one is easy. Any product worth choosing will be cross platform. If it isn't, you really don't want to bother with it. Focus more on if it treats AIX as a first class citizen along with the more popular operating systems, and that it provides a well defined mechanism for extending functionality where needed.

Issue 4:
Change is hard. Adopting any product of value will be difficult. Once a product is chosen, that's just the beginning. Developing a new culture around a collaborative approach to server configuration management will require the adoption of more than one technology. Things like git can be a hurdle within an established, experienced unix admin team.

Be strong. Embrace the changes. What you will have in the end will be worth the journey.

I am sure there are more issues than these. But these are enough to make your head spin when you start looking for a product to suit you needs.

Don't focus on the issues around finding a product. Rather, keep focus on the problem you are trying to solve.

The good news

The good news is that once you have taken the leap, you will have a configuration management solution which will provide some pretty compelling benefits. To name a few:

  • A unified set of procedures for configuring new servers across all of your platforms
  • A single pane of glass for configuration management changes across all of your platforms
  • Less admin time spent on redundant tasks
  • More time available for improving the things that drive innovation and automation
  • Easier on-boarding of new employees
  • Using collaborative tools like git, you get a good view into when and why changes were made
  • Most tools to centrally manage git repositories will provide good areas for issue tracking, so managing changes becomes a piece of cake

So, dive in and find a product that will make your life as an unix admin (or a manager of unix admins) easier!

This is my first blog post, and I hope to give more attention to topics like these. Hit me up on twitter to discuss, and stay tuned!