Powering one billion page views and 170 million unique visitors per month with OpenStack

When locals need to perform search-based queries, including news, healthcare and finance, they often turn to goo, a Japanese web portal and Internet search engine that has been around for 18 years. Goo is a web service brand of NTT Group accommodating the web search, blog, Q&A service Oshiete!goo, smartphone applications and more, which generates one billion page views per month and 170 million unique users.

Online activity is particularly high in Japan, ranking as country with the fourth highest percentage of its population surfing the web. But what are the locals searching for? The trends suggest that online shopping, last year racking up ¥8.5 trillion yen – or $79.3 billion USD – is a popular pastime.

Seasons also influence online behavior, the team at NTT Resonant notes. Popular search terms range from hay fever (花粉症) during peak cherry blossom season – February through April – to “zip code” or “post code” (郵便番号) in December, when locals celebrate the new year, biggest holiday in Japan, by sending postcards to family and friends.

Goo is based on the Q&A service Oshiete!goo and Google’s provision of basic search functions, including web databases. The NTT Resonant team set out to streamline operations for its popular web portal by adopting OpenStack for its private cloud infrastructure in October 2014.

Approximately 90 percent of Japan’s 126.8 million residents own more than one device, accounting for each unique goo visit. NTT Resonant provides more than 80 services including goo and web services for the enterprise.

The deployment, operated by NTT Resonant, consists of 400 hypervisors and 4,800 physical cores, accommodating more than 1,800 virtual servers in production. It provides about 60 types of goo-related services, including an Internet search business featuring a search engine service developed by goo that searches for mobiles, images and blogs and services offering a wide variety of information including weather, news, healthcare, finance and business.

Committed to open source and the OpenStack community

NTT Resonant’s team has embraced the culture of open source, citing the learning opportunities from the trial and error phases as a key benefit. Toshikazu Ichikawa, a senior research engineer at the NTT Software Innovation Center (SIC), says that the community and open-by-design concept were key factors when deciding to work with OpenStack.

“Choosing OpenStack helps us avoid unnecessary vendor lock-in and gives us choices of vendor plugins and integration when needed,” he says. “Applying OpenStack to our production triggered us to join the OpenStack community deeply.”

The team is rooted in the community, contributing to Summits, meetups and the OpenStack project itself. Koji Iida, a senior research engineer supervisor at the NTT SIC says that identifying bugs, many related to nova-cell, and contributing fixes were mandatory for the success of this deployment.

One particular bug was a show-stopper for NTT Resonant’s team, Iida said, until they were able to push a fix upstream, which also benefits the broader user community.

“The shelve function didn’t work in the Icehouse release with nova-cell deployment,” he said. “With hypervisor maintenance, our workflow relies for our users to shelve/unshelve their virtual servers to migrate servers from the hypervisor; therefore, it was mandatory for us to fix.”

Creating efficiencies for operators and application developers

With a nimble team of 15, six members working for server software including OpenStack and 9 working on the physical server, network and data center, NTT Resonant has leveraged OpenStack to increase productivity among its key team members.

a0dziown5vktgqhratc6
Members of the NTT Resonant team (from left to right): Tomoya Hashimoto, Munetaka Komazaki and Kazuhiro Tooriyama

The team deployed KVM virtualization in 2010 to shift away from each team procuring and managing an infrastructure per application. However, even with KVM, Tomoya Hashimoto, a senior manager at NTT Resonant, says that a lot of tasks, including configuring and assigning virtual servers, remained highly manual and coordinating between application teams and the infrastructure team consumed too much of the project’s time.

By moving to a cloud model with OpenStack and KVM in 2014, many of the steps to provide a virtual server were automated.

“We could automate most of the related tasks, such as monitoring system configuration beside the installation of OpenStack,” says Hashimoto. “This contributed to the reduction of tasks and burdens for the infrastructure operators and application developers.”

Once this was implemented, application developers created and deleted virtual servers more frequently than before, increasing the speed of business overall.

“The time to provide a virtual server for application developers was shortened from five business days to five minutes,” says Kazuhiro Tooriyama, an engineer at NTT Resonant. “Moreover, a migration of virtual servers between physical servers became much easier, making it easier for our operators to handle a server failure.”

Tooriyama also says that since virtual servers can now be easily scrapped and built, this has increased the opportunity for technical evaluation among the team’s application developers.

What’s next?

Currently, the deployment is running on Icehouse in a single datacenter, but at the August operator’s mid-cycle meetup in Palo Alto, Ichikawa lead a lightning talk saying that the team intends to upgrade from Icehouse soon, with the goal to continue upgrading to the newest release of OpenStack.

The team also has plans for expansion, including increasing the number of hypervisors and launching a second datacenter.

This article was originally published in the print Superuser publication that was distributed at the OpenStack Summit Tokyo, where the NTT Group was also given the Superuser Awards based on a community vote.

Learn more about the NTT Group’s different use cases, including NTT Resonant with this episode of Superuser TV.

Author
Recent Posts

Allison Price

Director of Marketing & Community at Open Infrastructure Foundation

Allison manages Marketing Content initiatives including case study development, advertising campaigns, the Superuser online publication and OpenStack User Survey. Her past roles have specialized in social strategy and execution for both B2B and B2C global companies in the technology and consumer industries. In her spare time, she enjoys finding new hiking trails and sharing her love with Celine Dion with anyone and everyone who will listen.