Challenges and battle scars from OpenStack deployments

Sometimes, helping a large enterprise client choose and implement OpenStack is a real conquest.

DXC’s Anupriya Ramraj and Rick Mathot were joined by American Airlines’ Farhad Sayeed at the OpenStack Summit in Sydney to talk about the toughest battles (and a success story) in building OpenStack deployments for enterprise clients.

The first challenge is helping a client choose either a distribution or upstream OpenStack. Ramraj recommends the distribution from multiple vendors. Even then, it’s important to make that infrastructure needs of the company match up with the right distribution. “You might end up with a load balance that’s not supported with a particular version of OpenStack that’s supported by the distribution,” she said. “Just getting that mix right, and delays because of that, is something that we see as a challenge.”

The next trick is dealing with getting OpenStack on older hardware. “It’s got its own unique challenges when you’ve got to go look up those drivers, get everything updated,” Ramraj said. “It can easily add up to months of delays there.”

Even once you’ve gotten OpenStack deployed, there’s bound to be a chaotic incident at some point. “Trouble-shooting OpenStack does require some unique knowledge of OpenStack services and understanding the dependencies between the services,” she said.

Getting help from the distribution provider can be tricky, too, as just reproducing the error or finding the right log files requires OpenStack expertise, which your client may or may not have in-house. “OpenStack skills are tough to hire and they’re tough to retain,” said Ramraj. “The audience out here, you should be proud that you’ve got OpenStack on your resume. It makes you highly valuable, because it is a tough skill to get out there in the market.”

Once all this ground has been covered, you still need to win over the apps group. “It’s always a tug-of-war between apps and infra(structure),” said Ramraj. Apps owners can be reluctant to move to OpenStack and the business case for OpenStack is getting those apps to successfully run. Sometimes, app teams want to use newer app deployment methods, and convincing them that Ansible and containers work on OpenStack is another challenge.

“Is it a Heat API to do the infrastructure automation, is it Kubernetes on OpenStack,” Ramraj asked. “Winning the challenge, winning the mind share of the apps guys, is a unique challenge by itself.”

Assuming all of the previous issues have been solved — infrastructure, support, apps, production workloads — there will come a time where you must need to OpenStack. “Upgrades in recent years have gotten much less intrusive,” said Ramraj, “much more dependable, but still there’s some unique challenges.”

Using managed services for reinforcement

All of these issues can be mitigated with a good managed services provider, said Ramraj. The client does not need to do this alone. A provider brings two things to the table, including the initial consult on the design (which involves choosing the right distribution and architecture, setting up networking and storage block objects, for example).

Secondly, a good managed services provider can offer ongoing services, which can look a lot like traditional IT services management. “OpenStack does not exist in an island,” said Ramraj. “OpenStack exists in tandem with other services that the client is consuming.” For example, a large enterprise client will likely have to bill individual departments for OpenStack consumption. Setting the billing up correctly is a solid task for a provider.

Another is incident management. A good managed service provider will offer 24/ 7 operations with highly skilled OpenStack people who can help with capacity management, storage optimization, or growth-based compute scaling.

Different clients will need different hand-offs, too, in terms of managing the OpenStack deployment. “If you look at this as a layered graph,” said Ramraj, “typically we say the managed services provider will handle up to the OpenStack layer and then the client owns the retro machines and the applications that they onboard onto the stack.”

Of course, every client is different. Some might only want to own the business outcome, which means the provider manages even the VMs and app onboarding. In addition, handling the VMs might prove another challenge. “One client wanted to back up all those VMs and the rate of the backups and the frequency they wanted to do it caused some issues in terms of configuring Neutron to be able to deal with that network bandwidth,” said Ramraj. The key, however, is having the right conversations between client and provider on how to manage the RACI, or responsibility assignment matrix.

People are key to the whole process, too. There are many roles that are important for a managed services provider to use when implementing OpenStack for a client: the cloud ops support team and the dev ops cloud engineer. The former needs to be a highly skilled OpenStack specialist, not a generalist, while the latter is essential for making the OpenStack implementation work for the client, with apps to support business outcomes. Other roles include account delivery executives and the cloud service delivery managers. “The delivery managers enable governance roles in there, making sure they have regular conversations with the client around, ‘Hey, what’s your forward-looking capacity planning, how are we doing on handling your tickets,’ and making sure the governance is in there.”

Ultimately, Ramraj believes that the battle for OpenStack private clouds can be won. DXC’s work with American Airlines is a prime example as the two worked together to get a critical baggage handling app set up on OpenStack.

Winning the OpenStack battle at American Airlines

American Airlines’ Farhad Sayeed told how his company has been virtualizing things for more than 11 years. Two years ago, they started with a vision to create a hybrid solution with an internal data center and a public cloud. “During our journey,” said Sayeed, “our focus was to raise open-source usage across the enterprise. We already do quite a few, but we want to focus on them. The future of hosting must support our business agility, some portability, and DevOps culture.”

The journey was admittedly bumpy. First, networking presented a challenge. “We had to work very closely with our network team to set up a repeatable process where we can duplicate that over and over again in various environments,” he said. “In a large enterprise like ours, OpenStack doesn’t stand by itself. It has to integrate with existing networking, backup restore, monitoring, SLAs and so on.” Sayeed’s team ended up testing four different OpenStack distribution types. He recommends finding providers who already have a strong partnership with your company, a solid roadmap and a future.

Originally, American Airlines wanted to create their own platform with networking. “We quickly realized that we were much better off and would get there quicker if we started with the reference architecture and we ended up in a converged platform,” said Sayeed.

American’s OpenStack target workloads standardize on Cloud Foundry for both public and internal clouds. That was the first workload that went into production. “At American Airlines our goal for cloud, both internal and external, is PaaS first, platform as a service,” Sayeed said.

One such app, Bag Tracker, went live recently thanks to the partnership between American and DXC as a managed services provider. Bag Tracker is a web-based app that allows end-to-end tracking, from check-in to delivery, of passenger bags, priority parcels,and aircraft equipment. Their standard Linux distribution today is Red Hat, although his team also sees a rising demand for Ubuntu, which they are testing as well.

Sayeed’s team is still deciding which container to go with. “Various teams are deploying Docker and Kubernetes in various configurations…” he said.

The journey to cloud is a cultural shift, said Sayeed. “Besides all the technical challenges that you will face, you must take care of the culture first,” he said. “All the different silos and disciplines, as we know, in an enterprise, have to be broken down.”

In addition, leadership must support what you’re doing. “My team and I were lucky enough to have full support from all the way from our CIO to my managing director during this journey,” said Sayeed. “That’s important. I cannot stress enough about that. Train and restructure your teams. Just because we work this way today, that doesn’t mean that’s how it needs to be. The cloud journey requires high collaboration. Work hand in hand with networking, security, middleware, application team. Everybody needs to come together.”

Sayeed and American Airlines have been testing four distributions of OpenStack. “We found out that less is more,” he said. “Folks who had done less customization and stayed close to the trunk found OpenStack easy to implement, easy to maintain, easy to upgrade and easy to roll out.”

Finally, bring your operations team in early. “You may have your own IT operations internally. You may have a managed service provider,” Sayeed said. “Don’t wait until you’re ready to go to production to bring them in. Bring them early. You’ll need them, they need you. Partner with them, and partner with them early.”

The toughest battle: Managing client expectations

Rick Mathot runs a global engineering and architecture team that has been deploying private clouds since 2009 (and OpenStack private clouds since 2015) at DXC. While he notes that there are many technical challenged involved in making OpenStack work, his presentation focus was on the human side.

In his keynote, OpenStack CEO Jonathon Bryce noted that OpenStack is no longer a science experiment. Mathot said that two-thirds of deployments are in production. “That means that now is the time that we’re going to be judged on our business value,” said Mathot. “The success of our outcomes will be judged relating to the business. That’s critical, because we need to understand what that value pertains to, (you need to) know the ‘why.’ Understanding your north star, as I call it, is going to get you through those periods where uncertainty prevails.”

It’s even tougher to gain ground when the troops aren’t open to new ideas, said Mathot. “We know how frustrating it is when a business comes to us with a solution and equally and likewise we shouldn’t be coming to them with preconceived solution artifacts (or) jargonized narrative relating to their ask,” he said.

CXOs are usually pretty nervous; business is in a time of significant change and providers need to help them allay their fears with language that they understand. “We’ve got to be able to articulate things simply in terms of what it is,” said Mathot. “For American Airlines, it was a scalable automated platform to host, to perform or provide PaaS to the organization.” In this case, it was the baggage tracking app – it was a simple use case, understood well by the business, and it worked within the OpenStack platform.

Mathot said that partnerships like this exist in a hybrid state — you need to be able to identify all the affected parties, all the stakeholders in a project. “Really, from a human side, bringing them into the tent with you is going to make them a part of the solution and stop some of the problems we’ve seen in the past of creating speed humps just for the sake of that,” he said.

Doing so will ensure that you’ll get support when you need it, which is generally when things are critical. “I think knowing some of the organizations I’ve been into, you may have to put it on a Post-It note and stick it on their screens.” said Mathot. “The reality is that you’ve got to do what you’ve got to do to get buy-in.” He said you need to lead from the front, don’t stop using mode-two thinking and design principles. “Just always make sure you keep an eye on the rear view mirror,” added Mathot, “to ensure that they’re still with you, the organization is still with you.”

Once you’ve finished the deployment and it’s up and running, you need to quantify what you’ve achieved, even beyond the technical value. “It’s a real simple trick,” said Mathot. “You’ve got to have a story for every level within your organization.” For example, while 20 percent IT efficiency is really great, explaining a 15 percent productivity gain for your application providers is probably a better story for them. Showing a faster speed to market for new useful client-facing apps is what will excite the CXOs, as well. You want to tune your story for each organizational level.

Finally, Mathot cautioned against a Facebook-style “better done than perfect” attitude. “Progress is important and any progress is better than none at all,” he said, “but what you shouldn’t look to do is have that (type of) mantra permeate every level of your deployment.”

Relevance is key, he said. “We all love the challenge and we all want to play with the newest technologies,” said Mathot. “It’s rewarding both personally and professionally, but the (American Airlines) story was a success because your bags arrived at the same airport at the same time as you did. It was useful and it was relevant.”

Catch the whole presentation below.