)
This post originally appeared on Ramon Acedo’s Blog Tricky Cloud. Ramon is a Cloud Architect who started in the Open Source world in the age of the 33.6K modems. He works at Red Hat helping businesses in their journey to an enterprise-class OpenStack experience. You should follow him on Twitter.
OpenStack Heat can deploy and configure multiple instances in one command using resources we have in OpenStack. That’s called a Heat Stack.
Heat will create instances from images using existing flavors and networks. It can configure LBaaS and provide VIPs for our load-balanced instances. It can also use the metadata service to inject files, scripts or variables after instance deployment. It can even use Ceilometer to create alarms based on instance CPU usage and associate actions like spinning up or terminating instances based on CPU load.
All the above is done by Heat to provide autoscaling capabilities to our applications. In this post I explain how to do this in RHEL 7 instances. If you want to reproduce this in another OS it’s as simple as replacing how the example webapp packages are installed.
Steps to have Heat autoscaling
(1) Create a WordPress repo in a RHEL 7 box. Make sure it’s a basic installation so that all the dependencies are downloaded along with WordPress:
# Install EPEL and Remi repos first, then create a repo
yum -y install http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm
yum -y install http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
yum -y --enablerepo=remi install wordpress --downloadonly --downloaddir=/var/www/html/repos/wordpress
createrepo /var/www/html/repos/wordpress(2) Create a repo for rhel-7-server-rpms with something like:
# First register to Red Hat's CDN with subscription-manager register
# Then subscribe to the channels to be synchronised
reposync -p /var/www/html/repos/ rhel-7-server-rpms
createrepo /var/www/html/repos/rhel-7-server-rpms(3) Download the Heat template which consists on two files: autoscaling.yaml and lb_server.yaml
Note: The template autoscaling.yaml uses lb_server.yaml (nested stacks?) and it can’t be deployed from Horizon right now due to a bug. It works fine from the command line as described below.
[Update] Note: I made it work in horizon by:
- Publishing the two templates on a web server.
- Modifying the autoscaling.yaml template published in the web server to call the nested template like this:
type: http://172.16.0.129:81/repos/heat-templates/lb_server.yaml(4) Modify the Heat template so the first thing it does when cloud-init executes the script passed via user_data by Heat is installing the WordPress repos.
a. Right before yum -y install httpd wordpress add the repos making it look like this:
[...]
user_data_format: RAW
user_data:
str_replace:
template: |
#!/bin/bash -v
#Add local repos for wordpress and rhel7
cat << EOF >> /etc/yum.repos.d/rhel.repo
[rhel-7-server-rpms]
name=rhel-7-server-rpms
baseurl=http://172.16.0.129:81/repos/rhel-7-server-rpms
gpgcheck=0
enabled=1
[wordpress]
name=wordpress
baseurl=http://172.16.0.129:81/repos/wordpress
gpgcheck=0
enabled=1
EOF
yum -y install httpd wordpress
[...]b. And right before yum -y install mariadb mariadb-server do exactly the same.
Note: I’m assuming that your two repos are accessible via http from the instances.
Note: All of these steps are optional. If your instances pull packages directly from the Internet and/or another repository you can skip or adapt this to your environment.
(5) Take note of:
- The glance image you will use: nova image-list
- Note: I’m using the RHEL 7 image available in the Red Hat Customer Portal rhel-guest-image-7.0-20140618.1.x86_64.qcow2
- The ssh key pair you want to use: nova keypair-list
- The flavor you want to use with them: nova flavor-list
- The subnet where the instances of the Heat stack will be launched on.
(6) Create the Heat stack:
heat stack-create AutoscalingWordpress -f autoscaling.yaml
-P image=rhel7
-P key=ramon
-P flavor=m1.small
-P database_flavor=m1.small
-P subnet_id=44908b41-ce16-4f8c-ba6c-9bb4303e6d3f
-P database_name=wordpress
-P database_user=wordpressNote: Here we use all the parameters from the template downloaded before. The are found in the parameters: section of the YAML file. We could add default: value within the template alternatively.
Now, what I do right after is a tail -f /var/log/heat/log* in the controller node, where I have Heat installed, just to make sure everything is fine with the creation of the heat stack.
(7) Verify Heat created a LBaaS pool and VIP:
[root@racedo-rhel7-1 heat(keystone_demo)]# neutron lb-pool-list
+--------------------------------------+----------------------------------------+----------+-------------+----------+----------------+--------+
| id | name | provider | lb_method | protocol | admin_state_up | status |
+--------------------------------------+----------------------------------------+----------+-------------+----------+----------------+--------+
| 78f02e89-aa07-40fd-917b-1481175b43e8 | AutoscalingWordpress-pool-46zb7elgzamo | haproxy | ROUND_ROBIN | HTTP | True | ACTIVE |
+--------------------------------------+----------------------------------------+----------+-------------+----------+----------------+--------+[root@racedo-rhel7-1 heat(keystone_demo)]# neutron lb-vip-list
+--------------------------------------+----------+-----------+----------+----------------+--------+
| id | name | address | protocol | admin_state_up | status |
+--------------------------------------+----------+-----------+----------+----------------+--------+
| 8da663cb-43d7-49af-9343-360431e02655 | pool.vip | 10.1.1.14 | HTTP | True | ACTIVE |
+--------------------------------------+----------+-----------+----------+----------------+--------+ (8) Associate a floating IP to the VIP: neutron floatingip-associate FLOATING_IP_ID VIP_NEUTRON_PORT_ID. In my case I need a floating IP:
[root@racedo-rhel7-1 heat(keystone_demo)]# neutron lb-vip-show pool.vip | grep port_id
| port_id | 13c01599-23f1-4e1e-96d9-72f2775e6183 |[root@racedo-rhel7-1 heat(keystone_demo)]# neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 0525f959-5213-4291-a1f0-a2ea2b40e11c | | 172.16.0.53 | |
| 09f1bdc9-228b-4057-a5d1-3327ccc0bfc8 | | 172.16.0.54 | |
| 5538961a-3423-46a3-9744-aba699e722c5 | | 172.16.0.52 | |
+--------------------------------------+------------------+---------------------+-------------------------------------root@racedo-rhel7-1 heat(keystone_demo)]# neutron floatingip-associate 0525f959-5213-4291-a1f0-a2ea2b40e11c 13c01599-23f1-4e1e-96d9-72f2775e6183
Note: This is optional if your instances are connected to a provider network where you can access directly instead of to a tenant network like in this example.
(9) Verify that Heat created the two Ceilometer alarms; one to scale out on high CPU usage and another one to scale down on low CPU:
root@racedo-rhel7-1 heat(keystone_demo)]# ceilometer alarm-list
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| Alarm ID | Name | State | Enabled | Continuous | Alarm condition | Time constraints |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| 1610f404-8df7-46ed-b131-6d3797fc9e4e | AutoscalingWordpress-cpu_alarm_low-vinrbn2rdjpx | alarm | True | False | cpu_util < 15.0 during 1 x 600s | None |
| 53c124bd-db57-4909-af55-009f5a635937 | AutoscalingWordpress-cpu_alarm_high-42dc5funjeds | ok | True | False | cpu_util > 50.0 during 1 x 60s | None |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+(10) Verify you can access the WordPress using the VIP:
(11) Now ssh into the WordPress web instance (not the DB one) and put some CPU load, just a couple of dd commands will suffice. Add a floating IP to the instance first if necessary.
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null &
[1] 908
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null &
[2] 909
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null &
[3] 910
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null &
[4] 911
[cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ dd if=/dev/zero of=/dev/null &
[5] 912 [cloud-user@au-g6hl-ye4uglqb5t7r-ylpghgnzyck3-server-nobvg6ftaoe7 ~]$ top
top - 11:01:05 up 10 min, 1 user, load average: 6.81, 1.12, 0.71
Tasks: 90 total, 8 running, 82 sleeping, 0 stopped, 0 zombie
%Cpu(s): 24.3 us, 75.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 1018312 total, 235068 used, 783244 free, 688 buffers
KiB Swap: 0 total, 0 used, 0 free. 95480 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
908 cloud-u+ 20 0 107920 620 528 R 15.8 0.1 0:09.80 dd
909 cloud-u+ 20 0 107920 616 528 R 15.5 0.1 0:08.49 dd
911 cloud-u+ 20 0 107920 620 528 R 15.5 0.1 0:07.88 dd
912 cloud-u+ 20 0 107920 616 528 R 15.5 0.1 0:07.71 dd
910 cloud-u+ 20 0 107920 620 528 R 15.2 0.1 0:08.11 dd (12) Observe how Ceilometer triggers an alarm (State goes to alarm) and how a new instance is launched:
[root@racedo-rhel7-1 heat(keystone_demo)]# ceilometer alarm-list
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| Alarm ID | Name | State | Enabled | Continuous | Alarm condition | Time constraints |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| 1610f404-8df7-46ed-b131-6d3797fc9e4e | AutoscalingWordpress-cpu_alarm_low-vinrbn2rdjpx | ok | True | False | cpu_util < 15.0 during 1 x 600s | None |
| 53c124bd-db57-4909-af55-009f5a635937 | AutoscalingWordpress-cpu_alarm_high-42dc5funjeds | alarm | True | False | cpu_util > 50.0 during 1 x 60s | None |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
(13) Kill the dd processes (directly from top press k and kill all of them)
(14) Wait about 10 minutes, which is the duration of the scale down alarm by default in our template. The state of the alarm in Ceilometer will go to alarm just like before but now due to lack of CPU load. You’ll see how one of the two instances is deleted:
[root@racedo-rhel7-1 heat(keystone_demo)]# ceilometer alarm-list
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| Alarm ID | Name | State | Enabled | Continuous | Alarm condition | Time constraints |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+
| 1610f404-8df7-46ed-b131-6d3797fc9e4e | AutoscalingWordpress-cpu_alarm_low-vinrbn2rdjpx | alarm | True | False | cpu_util < 15.0 during 1 x 600s | None |
| 53c124bd-db57-4909-af55-009f5a635937 | AutoscalingWordpress-cpu_alarm_high-42dc5funjeds | ok | True | False | cpu_util > 50.0 during 1 x 60s | None |
+--------------------------------------+--------------------------------------------------+-------+---------+------------+---------------------------------+------------------+ That’s all.
Kien Nguyen-Tuan
Dat Vu Tuan
Drew Smith
James Page
