Feeding Your Starved VMware VMs -- From Your Phone

  • Posted on: 11 January 2015
  • By: David La Motta

Imagine you wanted to get an email on a weekly basis showing you how your virtual machines were doing in terms of CPU and memory utilization. Specifically, you wanted to know if you had allocated more CPUs or more memory than you should have; or, to the contrary, that you had allocated too little to virtual machines that really needed it. Knowing which virtual machines you inadvertently starved was a great first step, but lets go one step further and imagine that you actually wanted to do something about it. Lets imagine you had the ability to relieve the pressure on those strained VMs and allocate more resources whenever it was necessary, from wherever duty called. 


Integra has the ability to reconfigure VMware virtual machines via its VMware Provider. This reconfiguration can be either a downgrade for those over-provisioned virtual machines; or it can be an upgrade for under-provisioned virtual machines--as in the example used in this post. For either reconfig, you are presented with CPU and memory thresholds that the VMs must meet, and also with the values that CPU count and memory will be affected by. Most importantly, however, is the ability to only report the offending VMs; in other words, even if offending VMs are found to exceed the thresholds you specified, no action is taken if you so choose. This is particularly useful in an automated environment: few people, if any, are going to want their virtual machines altered automatically. 

In the paragraphs that follow you are going to use Integra to create a weekly schedule which runs a workflow that:

  1. Finds all VMs that need to be upgraded
  2. Creates a chart with those VMs
  3. Sends an email
  4. Deletes the chart created
For this exercise we are going to need 4 Providers to configure the 4 actions mentioned above.

Providers




VMware - this is the provider at the crux of the matter.  The reconfigure action is here, and the VMs affected are all VMware virtual machines.  The provider is also responsible for getting all metrics for the VMs.
Plugin - allows us to execute any OS command where the provider is running.  Here we simply issue a remove command so we clean up after the chart that was created.
Smartcharts - we use this provider to create the chart that will be sent out in an email.
Notification - the provider that actually sends out the email.

After adding the providers, the next step is to configure all the actions that will be part of the workflow.  In addition to the 4 actions mentioned above, we are going to create a fifth action, which is the one you can invoke to actually modify the virtual machines from your mobile device of choice.

Actions


The first action we configure is that which sends an email:



In the action above you see that we do not specify any type of email attachment.  This is because the attachment will be created dynamically at runtime thanks to Integra's piping capability.  I'll stress this point again during our workflow creation so that it is clear.  Next up are the reconfig actions:



If we had to give an importance ranking to all the actions being used in this example, the identify-vms action shown in the graph above would rank #1.  You can see we are going to upgrade VMs by the type of reconfig in the dropdowns, and in order for both upgrades to take place CPU and memory utilization have to be greater than 70% (your numbers may be higher, but for our illustration purposes we'll stick to 70%).  If both thresholds are exceeded, we will add 1 CPU and also add 256 MB of memory to the virtual machine.  However, notice the Report checkbox--this tells the action to perform a "what if" (for those of you who know PowerShell).  In other words, that Report field tells the action to find VMs that meet the criteria and show what the CPU and memory would like if the values we specified were applied.  This is why you see the CPU count at 3 and memory at 4352 MB in the bottom portion of the screenshot.

The image above is also revealing something interesting.  We see 2 virtual machines that meet the criteria, but why is that information displayed?  The reason is because Integra gives you the ability to test actions as you craft them, to fine tune them so to speak.  In this particular case I had ran the action to see what results I got; in order to run an action, simply click the green Play button at the far right of the toolbar.



Integra has the flexibility of cloning existing actions to save data entry time.  That is exactly what we did to create the upgrade-vms action.  If you notice, it is an almost exact replica of identify-vms with the exception that the Report checkbox is not checked.  That means if the upgrade-vms action executes, the values for CPU and memory will be applied to any VMs that meet the criteria.  This is the one action that is not part of our main workflow; instead, we will add this single action to a new workflow which you can invoke using your mobile device.  Hang tight!  The next image shows you how we configure the chart action.



The chart-vms action is particularly useful because with Integra you have the ability of creating dynamic charts.  We will use the information from the identify-vms action above, and pipe it into this action.  If you look at the input fields in the highlighted rectangle, you can see the fields we are using from the previous action's output.  If you scroll back to the identify-vms action, you will see that it produces output objects call vmDetails, with fields such as vmName and memory.  It is precisely those fields that we use to create our chart, and just as in this case, this means we can use any fields from any output to create charts dynamically using Integra.  And if you look closely, you will see that this chart-vms action is producing output of its own: a file, which corresponds to the chart, stored on the local filesystem where the provider is running.  We use this output and pipe it into the email action, and the Notification provider knows to consume the file on disk and add it as an attachment to the outgoing email.

Last but not least, we don't want to leave .png files laying around in the /tmp directory.  For this we use the Plugin provider, which gives us the ability to execute any command on the OS where the provider is running.



Workflows

With all of our actions configured, it is now time to put them to good use.  We will create 2 workflows:  the first called report-vm-hogs will find offending VMs, create a chart, send an email and erase the chart.  We'll use this workflow to create a schedule further below.   The second workflow we create, called upgrade-vm-hogs, will be the workflow that actually triggers a reconfiguration of your virtual machines.



As you can see, we pipe the output of the first action (identifying offending VMs) to the chart action.  After the chart is created, we pipe that into the email action which takes the file and adds it as an attachment to the email.  We could have piped the output of step 1 directly into the email, too.  In that case what would happen is another type of attachment gets created:  a raw .json file with all the details that went into the chart would get attached to the email.  Getting raw .json allows you to do more post-processing, if that is something you're interested in doing.

The second workflow, upgrade-vm-hogs, contains a single action: upgrade-vms.  This is the workflow that can be executed to trigger the actual reconfiguration of your virtual machines.  It is very important to mention that in order for the reconfiguration to take place, the VMs in play have to be shut down (unless they support hot add of CPU and memory).  If your virtual machines are capable of modifying CPU and memory while running, Integra will take advantage of that.  Regardless, an upgrade of this nature requires careful planning so you will most definitely want to find a window where your users are comfortable with their VMs being shut down in case they can't be upgraded while powered on.

Schedules

With both our workflows created, we'll proceed to schedule the one that only sends us the email with all the details that we want.  The weekly-high-usage-report schedule contains a single workflow (report-vm-hogs) which you can see at the bottom of the screenshot, and a single task which executes that workflow at 1:00 AM every 7 days.



Once you have your schedule set, Integra gives you the ability to test out your end-to-end workflows.  Using the execute button on the far right of the toolbar, you can test your schedules and see if they produce the desired results.  Once you are satisfied with your workflows and their results, you can sit back and relax until the schedule kicks in.

Results

The screenshot below shows the results of a test run.  As you can see, everything passed with flying colors.



We see the results are successful, which mean we should have an email waiting in our inbox with the chart and the details we specified in the chart-vms action.



Mobile Portal

We started out with the promise that you could execute the upgrade from your phone, so here we show you how you would do that.  All of the workflows that you create are exposed in Integra's mobile self-service portal.  Since we didn't create a schedule for the upgrade-vm-hogs workflow, we know the only time it will run is when you decide to run it manually.  If we allow ourselves to imagine you've worked out a timeframe when you can perform the upgrade, and that timeframe happens to be when you are sipping on a margarita by the beach, you could invoke the upgrade-vm-hogs workflow from your mobile device by selecting said workflow and tapping on the Execute button on the upper right-hand corner of your device's screen.



There are, however, 2 steps missing from the upgrade-vm-hogs workflow.  Can you guess what they are?

Obviously, as automation architects we want to be as hands off as possible.  Earlier we talked about our VMs having to be powered off in order for us to be able to do the upgrade.  The 2 steps that could have been added are one for shutting down the VMs before the upgrade and the other one for powering them on after the upgrade.  Try it out yourself; adding these 2 steps are left as an exercise for you.


In closing, this and many other operations are made simple thanks to the flexibility, simplicity and power of Integra.  If you are giving this post a shot and would like some help, don't hesitate to drop by our Google+ community and ask us questions.

Until next time!

--