VMware Cross-Cloud vMotion is another major milestone on VMware Seemless Hybrid Cloud Vision

If you are like me or many of VMware customers who has been following the VMware Hybrid Cloud Story closely for a while, you have probably asked yourself or your sales rep when will you be able to move your Virtual Machines back and forth between On Premise (Private Cloud) to vCloud Air (VMware Public Cloud) without encountering downtime. Many of us are used to the concept of VMotion and being able to move our VMs across hosts within a site and in vSphere 6 we had the capability to move VMs across sites and vCenters.

The next logical step in that vMotion evolution is being able to vMotion VMs to the Cloud and back, and that is just what VMware has announced at VMworld today. In fact, a live demonstration of what is called Cross-Cloud vMotion was shown during today VMworld General session. This exciting news has been a part of  Technology Preview of Project SkyScraper, a new set of hybrid cloud capabilities for VMware vSphere that will enable customers to confidently extend their data center to the public cloud and vice-a-versa by seamlessly operating across boundaries while providing enterprise-level security and business continuity.

VMware Hybrid Network Cross Cloud VMotion

Below is the two features of Project SkyScraper that was shared today at VMworld:

  • Cross-Cloud vMotion is a new technology based on vSphere vMotion that allows customers to seamlessly migrate running virtual machines between their on-premises environments and vCloud Air.
Read More

vRealize Automation Modifying Approval E-mails

One of the questions I get often asked by customers is how to modify vRA e-mail templates. To be honest the most one customers usually care to modify is the Approval e-mail to include more information than what is provided in the out of the box template. In this article, I will show you how to simply modify the vRA Approval e-mail to include additional custom properties. I will not drill into more advanced e-mail modification topics to keep this as simple as possible and get you straight to the point if that all you needed to do, though I will have more advanced vRA e-mail template modifications links at the end of this post.

First, you will need to upload the vRA e-mail templates to vRA using the below steps:

  1. Download the 2088805_vrealize_automation.tar.gz file from KB208805
    • Windows: Use a program such as WinSCP to copy the 2088805_vrealize_automation.tar.gz to the VMware vRealize Automation
    • Linux: Run these commands:scp 2088805_vrealize_automation.tar.gz root@vCAC VA:/
      ssh root@vCACVA
      cd /
      tar -xvzf 2088805_vrealize_automation.tar.gz
      find /vcac -type d -exec chmod o+rx {} \;
      find /vcac -type f -exec chmod o+r {} \;
  2. Restart VMware vRealize Automation by running this command:service vcac-server restart.

The above steps should get you to a position where you are ready to modify the vRA e-mail templates as needed, and changes to these templates going forward will take effect without needing to restart the service, but might take up to 2 minutes to see the changes.… Read More

vRealize Automation Gugent stuck and machine deployment timeout

After installing the vRA 6.2.2 Gugent into our Windows 2012 R2 template, blueprint deployment started to time out. As I had followed the exact steps I have previously used to install the Gugent for quite few customers before which in turn I have already documented on my following blog post: vRA Guest Agent Installation, I was surprised with the issue and thought to document the issue and resolution of it in this blog post in case others encounter it. Before I start, just few words of the environment versions:

vRA: 6.2.2
Gugent: 6.2.2
vCenter: 6.0
ESXi: 6.0
VMware Tools: 6.0
OS: Windows 2012 R2

While above these are the exact versions I have seen the issue with, you might face it with previous or future versions, as I am not sure yet which versions this specifically apply to, I wanted to ensure to point out the exact versions where I have observed it. Below is the timeout error I saw in vRA while trying to deploy from a blueprint that uses the Gugent: “Request failed: Machine vttest001: Timeout Customizing Guest OS.”

Request failed machine timeout customizing guest os


The error logs within the vRA portal was not providing me with enough info to debug the error with, so I started digging into one of the machines deployed through the problematic blueprint and after digging around I have found the following error in Windows Applications event log:

“faulting application name: DynamicOps.agent.guest.exe libeay32.dll”

 vRA Gugent Faulting Application name: DynamicOps.Agent.Guest.exe

For some reason the issue end up being caused by the vRA Gugent not being able to locate the VMware Tool libeay32.dll package, although VMware tool 6.0 was installed.… Read More

vRealize Automation deployment to vCloud Air fails

After configuring vRealize Automation Center to be ready to deploy to vCloud Air (End Point, Fabric Group, Reservation, Reservation Policy, Blueprints, etc), my initial vAPP blueprint deployment to vCloud Air has started to fail with the below error message.

“Request failed: Machine VTTest001: an error occurred while customizing the vapp..”

vRA vAPP deployment to vCloud Air fails error occurred while customizing the vAPP


Checking the vRA logs under monitoring has given me a bit more descriptive error, which has definitely helped me resolve the issue much more than the original error: “vCloud Customize VM failed for vApp: VTTest001 Error occurred while customizing vAPP. Inner Exception: Auto-logon count must be within 1 to 100 range if enabled or 0 otherwise.”

vAPP deployment fails in vRA autologin error under monitoring log

This has reminded me that I has never setup the auto-logon setting under customization on the VMs building my vAPP template. You will need to check mark the “Automatically log on as Administrator” option as shown in the below screenshot. I would leave the number of automated log on to 1 as well.

vRA deployment to vcloud Air fails Automatically login as an administratorThis has done the trick for me and things worked afterward. I assume others might miss this step as well, and as the initial error might not be obvious indication of the issue, I thought I will try to save you the hassle of trying to figure out how to fix it.… Read More

vRealize Automation iaas-service does not register after installation

While doing the usual vRA 6.2.2 large deployment distributed install, and after everything in the installation went smoothly without any errors, the vRA infrastructure tab was not working. Checking the vRA Services through the vRA Appliance VAMI, and noticed all of my services are up and running as  expected with its status being registered, beside the iaas-service which its status was left blank. Look at the below screenshot and notice where the service column ” iaas-service        com.vmware.csp.iaas.blueprint.service” status being blank.

vrealize automation iaas-service not starting

Going through every test I can think of, I could find no meaningful error in any of the logs, till I opened my infrastructure tab again but in Chrome this time and rather than getting the usual 404 error it was giving me earlier the below message came up: Invalid Argument. Host name may not be null

vra Infrastructure tab Invalid Argument Host name may not be null


After a bit of digging/research, I have found out that the VIP names the customer is using included under scores which is not supported by vRA for both hostnames or VIP names. This has been documented in the vRA 6.2 Installation and configuration document here as well in KB2086553.

Resolution for this issue: You will need to avoid this scenario before hand, else you will have to redo the full installation.… Read More

vRealize Automation 6.2 moving to Internal replicated vPostgres

When delivering a vRealize Automation 6.1 and prior in a distributed install, the recommendation was always to have an external vPostgres instance. This has changed in vRealize Automation 6.2 and it’s briefly touched upon in the vRA 6.2 reference architecture found at: vRealize Automation 6.2 reference Architecture. I wanted to make sure to highlight this, as many have missed this change and others are still wondering how it can be implemented or why this change. It’s important to mention as well, you can use this replication architecture for any version of vCAC/vRA that’s higher than 6.0.1.

Let’s start by highlighting the change, we are moving from having a separate/external vPostgres  machine to host the vPostgres (being a vRA appliance that is re-purposed to be used as vPostgres or a full vPostgres install) to using the internal built-in vRA Appliance vPostgres and replicate across the two vRA Appliances. The below image demonstrate this change:

vRA internal vPostgres replication
So what do you get of moving to this new model? Actually there is few advantages of implementing your vRA 6.2 this way, below is couple of them.

1- You have less machines to manage by eliminating the extra one or two external vPostgres machines you needed in the 6.1 reference architecture setup.… Read More

Resource Actions does not show up in vRealize Automation 6.2.1

I had an instance of vRealize Automation 6.2 with few resource actions that were built under Advanced Services with things like take snapshot and move VMs to a different folder in vCenter. These were working like a charm in vRealize Automation 6.2, but as soon I upgraded to 6.2.1 they stopped showing in the items actions menu. I have re-checked my configuration and tried few additional resource actions with no luck.

After a bit of research I have found out that its a known problem caused by a bad DLL within the 6.2.1 and its easy to fix by following KB2111713. I have as well happen to be able to reproduce the same issue with a fresh install of vRA 6.2.1, so if your resource actions does not show up in the actions menu, although you have done all the right steps KB2111713 or the step below is your way to fix it.

1- Download the new DLL by following the download link in KB2111713, the download name is vRA_621.zip, where it includes a single DLL file which in turn called  “DynamicOps.Api.v61.dll".

2- Stop all of your vRA IaaS Services in the following order: 1- Proxy Agents 2- DEMs, 3- DEM Orchestrator 4- vCloud Automation Center Manager service

3- Backup your old DynamicOps.Api.v61.dll found at C:\Program Files (x86)\VMware\vCAC\Web API\bin by copying it to a different location.… Read More

vCAC 6.x using gugent to run scripts

Lately, I have been getting a lot of questions about how to use the vCAC Gugent to run scripts within the guest OS of your deployed VM. While if you are deploying simple VMs using vCAC this use case might not sound obvious, it has a lot of uses to it in the real world. Before I go through how to get your vCAC Gugent or now called vRA Gugent (Guest Agent) to run scripts, let’s take few examples of why you want to do this:

– Imagine the use case where you want to assign an IP Address in a Guest OS that is not supported by VMware Customization tool. You can actually assign an IP Addess to it by running after deployment  script using the Guagent to change the IP Address of that machine. Actually while I had this post in mind for a while, it end up getting to my top priority today as a colleague was asking me about this use case scenario.

– Imagine the use case where you want to run some security scanning scripts after the OS was fully deployed. This is will be any easy way of doing it.

– Imaging the use case where you want to initialize a client or agent for application deployment/management tool (Ex: Puppet, Chef, Altiris, TSM Client, …)

– It can be even be used as a cheaper dirty way of deploying applications or services.… Read More

VMware vRealize Automation 6.2 and the missing vCO

If you have tried to upgrade your current vCloud Automation Center distributed install to vRealize Automation 6.2 or if you have tried to plan for the upgrade lately, you will notice one piece of the puzzle seems to be missing. At the time vRealize Automation 6.2 was released, vRealize Orchestrator 6.0 was not released yet as it suppose to be a part of the vSphere 6.0 release(not yet released).

While the vRealize Automation 6.2 appliance has came up with a builtin vCO 6.2 with the 6.2 plugin installed which was sufficient for small deployment that did not require a distributed install, customers with distributed install are wondering what to do. In this article I wanted to highlight the three options available to you and when to approach each of these routes.

1- vRealize Orchestrator(vRO) 6.0 has been made available to vRealize Automation 6.2 customers before vSphere 6.0 go GA, where you will have to open a support ticket to obtain it. Here is the KB article documenting that: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2100951

2- Convert a vRA 6.2 appliance into vRO 6.2 appliance, again here is the KB documenting this: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2100951     (Almost there is no more use case for this with vRO 6.0 available to you through GSS.… Read More

vCloud Automation Center 6.x Time Bomb Issues

As I know quite few of my vCloud Automation Center 6.x (lately renamed to vRealize Automation) customers are following my blog, I wanted to share this important information of a two different time bombs issues that affect the 6.x releases of the product.  The purpose of this post is to highlight these two time bomb issues and the simple changes required to avoid them affecting the availability & manageability of your vCAC environment. If you have not done this or not sure if it was done during the installation time, please take the time to do it now. It is worth checking even if your environment was setup by a vCAC expert as it might have not been known during the the time your setup was installed:

1- vCAC 6.x VAMI Root account time bomb.

In vCAC 6.x There is a one year expiration time bomb in the VAMI Root account. This will prevent you from logging in to the VAMI consoles. To fix run this command…

# chage –M 99999 root

 This will change the timeout  and should prevent future issues. This occurs in both the vCAC Identity Appliance , and the vCAC Appliance.

2- vCAC 6.x SSO internal tenant admin password  time bomb.Read More

VMware vRealize Automation 6.2 is now available for download

vRealize Automation 6.2, which is the new name for vCloud Automation Center has been made available for download earlier today. This release will be adding a lot of new enhancement & features where you can find the full list in the vRealize Automation 6.2 Release notes. In here I want to cover few main enhancements, that I was patiently waiting for.

1- Ability to edit custom properties for published applications in the service catalog. While this enhancement is sneaked in very late down in the release note & many have not given it as much attention, it is one of the greatest enhancement in my opinion in this release. If you have worked with vCAC in the past and tried to integrate it with VMware Application Director (just renamed to vRealize Automation Application Services), you were always faced with the challenge that if you publish applications using application director into vCAC, you will not be able to pass custom properties to the underlying VMs blueprint, which was a big challenge. Actually in many cases, it eliminated the ability to use application director without a very heavy customization. Now this constrain being lifted, I can see the amount of great power vRealize Automation will enjoy when coupled with vRealize Automation Application Services.… Read More

vCAC Guest Agent Installation

While using VMware vCloud Automation Center Guest Agent or as many call it Gugent is not a mandatory requirements to use vCAC, it has its merits and use cases. In fact, many customers don’t end up deploying the vCAC Guest Agent within their Windows & Linux machines from day one, as not having the agent won’t stop vCAC from deploying a Windows or Linux VM and carry out the standard OS Customization. The interesting fact though, most of them end up installing it within their templates couple of months later, as they come up with a use case or two for it.

Let’s start by going over the vCAC Guest Agent use cases, and why should you install the Gugent within the templates you want to deploy with vCAC:

vCloud Automation Center Guest Agent Use Cases:

Launch in guest Scripts, this is has always been the biggest driver for installing the vCAC Guest Agent. The nice part about it, after installing the guest agent, you can customize which script you want to run and what parameters you want to pass to it using custom properties. Further being able to invoke these scripts from share drives and passing them your desired parameters using custom properties,  you can avoid having to edit your template, every time you update your script.… Read More

How to use vCAC new NetworkProfileName Custom Properties

Before the release of vCloud Automation Center 6.1.1, it was common to combine the use of the two below custom properties to assign a particular virtual machine to a particular portgroup/Network Path and a particular network profile:

VirtualMachine.NetworkN.Name: This custom property is used to put the virtual machine network adapter N, on the portgroup name supplied as a value for this custom property.

VirtualMachine.NetworkN.ProfileName: This custom property is used to tell the virtual machine network adapter N to obtain an IP from the network profile named in the value of this custom property.

I have seen the combined use of these two custom properties many time in the past and they seemed to work properly before vCAC 6.1.0 (It might have stopped working a bit earlier than that but I did not notice it). On the other hand using each of these custom properties on its own still work properly in vCloud Automation Center 6.1 and beyond, combining both custom properties on the same blueprint seems to produce some odd behaviors and unexpected results after 6.1. To avoid having such a problem, its highly recommended to use the newly introduced VirtualMachine.NetworkN.NetworkProfileName custom property.

vCAC NetworkProfileName custom property

VirtualMachine.NetworkN.NetworkProfileName kinda combine both custom properties in a single property.… Read More

Windows could not parse or process the unattend answer file for pass [specialize]

I was at a customer which I have deployed vCloud Automation Center 6.1 for, and they tried to create a new blueprint. They have copied the old blueprint and only changed the VM template used in that blueprint from Windows 2012 to Windows 2008 R2. They saved and published the new blueprint without any problem, then they tried to deploy the new blueprint. The blueprint deployment kept failing with the following error:

“Request failed: Machine VTTestVM001: Timeout customizing machine”

From the error, I was able to tell it was getting stuck at the guest customization stage, so I deployed one more VM and waited till it get to the guest customization stage, then opened a console to the VM to be greeted  with the following Sysprep error:

Windows could not parse or process the unattend answer file for pass [specialize]. The settings specified in the answer file cannot be applied. The error was detected while processing settings for component [Microsoft-Windows-Shell-Setup]”

Below is a screenshot showing the actual error on screen:

Windows could not parse or process the unattend answer file or pass [specialize]

After I hit OK, the machine restarted again and gave the following error and kept repetitively restarting and giving the same error afterword:

“The computer restarted unexpectedly or encountered an unexpected error.Read More

vCAC 6 Property Dictionary Relationship Builder

While you can create property Dictionary Relationship XML by hand as I have shown in my previous post: vCAC 6 Property Dictionary Relationship, it has been proven to be so tedious of task to do when having a large environment,  that I wanted a more compelling way of doing it. That’s why I came up with my vCAC 6.x Property Dictionary Relationship Builder (This work with all versions of vCAC 5.1 and higher). I am going to continue with the example of my previous two posts: vCAC 6 Custom Properties – Build Profiles – Property DictionaryvCAC 6 Property Dictionary Relationship and show you how to generate the same property dictionary relationship XML using my my Property Dictionary Relationship Builder. Please read my previous post for more background of the scenario used in this example.

Let’s start by quickly showing the same table used in my previous post to demonstrate the relationship we are trying to generate. In this post I will use my generator to produce the location and storage relationship demonstrated in the below table

Location Storage Network
Toronto Toronto_T1 Toronto_Prod_PG01
Toronto_T2 Toronto_Prod_PG02
Toronto_T3 Toronto_DMZ_PG01
Boston Boston_T1 Boston_Prod_PG01
Boston_T2 Boston_Prod_PG02
Boston_T3 Boston_DMZ_PG01
Paris Paris_T1 Paris_Prod_PG01
Paris_T2 Paris_Prod_PG02
Paris_T3 Paris_DMZ_PG01

The expected Property Dictionary relationship XML file between Location & Storage for the above table look like below:

<?xml version=”1.0″ encoding=”utf-8″ standalone=”yes”?><ArrayOfPropertyValue xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Toronto</FilterValue><Value>Toronto_T1</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Toronto</FilterValue><Value>Toronto_T2</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Toronto</FilterValue><Value>Toronto_T3</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Boston</FilterValue><Value>Boston_T1</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Boston</FilterValue><Value>Boston_T2</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Boston</FilterValue><Value>Boston_T3</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Paris</FilterValue><Value>Paris_T1</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Paris</FilterValue><Value>Paris_T2</Value></PropertyValue><PropertyValue><FilterName>Vrm.DataCenter.Location</FilterName><FilterValue>Paris</FilterValue><Value>Paris_T3</Value></PropertyValue></ArrayOfPropertyValue>

While the above XML file look not too bad and could be edited manually, imagine if you have to generate the same type of XML for a large Property Dictionary Relationship that include 10s of filter values and for each filter 10s of values.… Read More