How to configure vCAC not to delete VMs after deployment failure

I have a customer approach me asking of how we can stop vCAC from deleting VMs after deployment has failed for debugging purposes. The default behavior of vCAC is to delete the VM after deployment failure even if that failure was just caused by Guest customization failure or by the Guest Agent execution failure. The problem when a deployment fail due to a Guest Customization failure or by Guest Agent execution failure and the VM get delete as the deployment fail, you are left with no in guest logs to trace the cause of the problem which explain the need for stopping vCAC from deleting VMs after VMs deployment failure for debugging/troubleshooting.

Luckily vCAC Proxy Agent can be configured not to delete VMs after deployment failure and send it to VRMDeleted folder instead of deleting it and it can be deleted manually after you finish your debugging. If you think you need to stop vCAC from deleting VMs from vCenter after deployment failure for troubleshooting or any other reason, then below is how to do just that.

A system administrator can modify proxy agent configuration settings, such as provisioning machine credentials and deletion policy for virtualization platforms, after installation. The proxy agent utility can be used to modify the initial configurations that are encrypted in the agent configuration file.… Read More

vCenter Upgrade Error 28035. Setup failed to copy LDIFDE.EXE from System folder to ‘%windir%\ADAM’ folder

During the upgrade of vCenter I have faced the exact error that is documented in KB#2013675. The error was as follow:


Error 28035. Setup failed to copy LDIFDE.EXE from System folder to ‘%windir%\ADAM’ folder


Below is an actual screenshot of the error:

vCenter upgrade error 28035 setup failed to copy LDIFDE

Further, in the vminst.log file, we found the following error:


  • Unable to copy c:/Windows/ADAM/LDIFDE.EXE to C:/Windows/ADAM/LDIFDE.EXE


If you have read KB#2013675, then you will notice the above errors are an exact match of the errors documented in that KB. If so why, I am creating this post if a KB is out there that include the solution, because that solution alone which is shown below did not resolve my issue:

——- KB#2013675  Solution Start —-

To resolve this issue, manually install the Active Directory Lightweight Services Role for the server.
To manually install the Active Directory Lightweight Services Role for the server:
  1. Open the Server Manager for the server and click Add Role.
  2. Select the Active Directory Lightweight Directory Services option.
You should be able to install vCenter Server after the Role tasks complete

——- KB#2013675  Solution End —-

If the above solution does not work for you, just like was the case in our situation, then try the below solution after you apply the solution documented in the KB.… Read More

vCAC 6 Service Temporarily Unavailable

As I have deployed vCloud Automation Center 6.0.1 in my home lab on vSphere 5.5, the installation worked flawlessly as I have carried out the same installation several times in production for several customers before. vCAC seemed to work without any problem at all for few hours, then I started getting the following error every time I tried to access my vCAC Portal:


Service Temporarily Unavailable 

The Server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.


Below is a screenshot of the actual error:

vCAC Service temporary unavailable


I was surprised to see the above error, and when checked the status of my vCAC Services under the services tab in the appliance management console of vCAC the pages was not loading although all the other tab were working just fine. I logged in through ssh to the vCAC Appliance and restarted the vCAC Service using the following command:

# service vcac-server restart

After restarting the vCAC Service and waiting for about 5 minutes, I have noticed that all the services of vCAC is now showing up on the service tab and right after that the vCAC portal started to function again. Initially I thought the service might hanged for some reason or so on, but after few hours the same issues came up, and I have noticed that I am getting the same problem repeatedly every few hours where my services stop again.… Read More

Upgrading vCenter with vCenter Heartbeat

I have went through the upgrade process of vCenter 5.0 to vCenter 5.1 which was protected by vCenter heartbeat(Please note same process apply for upgrade to 5.5 as well). The process end up being more of uninstalling and re-installing vCenter heartbeat rather than upgrade to avoid the undesired behavior of SSO lockdown that can be caused by the upgrade process of vCenter Hearbeat when SSO is hosted on the same VM as vCenter which was the case in my scenario. Let start by giving a summary of the upgrade process before going into a details:

a.  Uninstall vCenter Heartbeat 6.5 if installed
b.  Setup Single Sign On.
c.  Upgrade vCenter Server from to 5.0 to 5.1
d.  Upgrade Update Manager from 5.0 to 5.1
f. Install vCenter Heartbeat 6.6 (UAT & Prod).

below is the process  changes involved when upgrading vCenter that is protected by vCenter Heartbeat in more details:

1-      Preparation:

  • Make sure you have adequate backup. Maybe Image backup of your vCenter VMs?
  • Make sure to have license keys on hand.
  • Make sure All Windows Updates is disabled.
  • Write down all the IPs configurations.
  • Prepare a service account to use for heartbeat with local admin and vcenter admin privillage
  • Make sure to have a local administrator account on the machine that you know the password off.
Read More

vCenter 5.5 Upgrade fails when installing Microsoft Visual C++ Redistributable Package prerequisite with the error: Error Code 3010

While at a customer site and trying to upgrade their vCenter 5.0 to 5.5, every time we tried to run the installer it ran for a bit then gave us the follow error:

vCenter SSO Error 1722

Then the following errors appeared in a log files that was opened automatically in a notepad.

—————–  Error start —————–

Action 12:39:49: VM_InstallVCREDIST_x64. Configuring Microsoft Visual C++ Redistributable Package (x64)…
Action start 12:39:49: VM_InstallVCREDIST_x64.
CustomAction VM_InstallVCREDIST_x64 returned actual error code 3010 (note this may not be 100% accurate if translation happened inside sandbox

MSI (c) (5C:E4) [12:40:32:470]: Note: 1: 1722 2: VM_InstallVCREDIST_x64 3: F:\ 4: D:\Single Sign-On\prerequisites\vcredist_x64.exe /s /v/qn
Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor. Action VM_InstallVCREDIST_x64, location: F:\, command: D:\Single Sign-On\prerequisites\vcredist_x64.exe /s /v/qn

MSI (c) (5C:E4) [12:42:05:103]: Product: vCenter Single Sign-On — Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor. Action VM_InstallVCREDIST_x64, location: F:\, command: D:\Single Sign-On\prerequisites\vcredist_x64.exe /s /v/qn

Action ended 12:42:05: VM_InstallVCREDIST_x64. Return value 3.
MSI (c) (5C:E4) [12:42:05:103]: Doing action: FatalError
Action 12:42:05: FatalError.Read More

vCenter Server Appliance 5.5 SSO Issues

While rebuilding my home lab to use the latest version of vCloud Automation Center, I have decided to rebuild it from scratch with the latest vCenter Server Appliance and ESXi. After deploying the vCenter Appliance 5.5c and going through the configuration, I wanted to join the appliance to my domain and allow users from my domain to authenticate.  While I was able to join the appliance to my domain and that seemed to work just fine as in the below screen shot, I was having a problem configuring SSO for native active directory. I kept getting the following error:  ” ‘alias’ value should not be empty ”

Alias Value should not be empty

The solution for this particular problem was easy, actually it was more of Ooops I have forget to restart the vCenter Appliance after joining the appliance to my domain. Actually it tell you to do so as in the below screenshot:

You actually need to restart the appliance after configuring AD

I wanted to document this, as I am positive there will be the case where others will miss this as well and try to fight this error and thought I will save them time. This is especially true that I have seen other posts on forums and blogs that claim rebooting after this step is not required although the documentation state so.… Read More

Unitrends Virtual Backup 8.0 Sneak Peek

I have participated in the Unitrends Virtual Backup 8.0 (formerly PHD Virtual Backup) beta program, and Unitrends has been kind enough to allow me to share my finding with you on my blog. Please note while the screenshots below still show PHDVirtual logo, I believe these will be replaced with the Unitrends logo in the official release.

I have downloaded the beta and the first thing that has impressed me is their new setup wizard. It is really slick to say the least, just for those in doubt, below is the screen shots of the setup wizard while trying to deploy it for the first time in my environment.

The below screenshot are an actual setup steps that I took from my lab environment and not from Unitrends website or documentation as I can see these setup wizard simulating many power point slides I have seen before for how descriptive they are. Well done Unitrends!

Unitrends Virtual Backup Wizard

This is the first screen I was welcomed with as soon I pointed my browser to the appliance. Yes, that was an instructional video that explain the PHD Virtual topology in a very quick manner and here is the video for those of you curious to see what that video was about.… Read More

VMware VCAP5-DCD fresh tips

Last Saturday, I have thought it is finally the time to attend the overdue VCAP5-DCD exam. After obtaining my VCDX4, I have decided to award my self some break of doing certification exams, which got me to push my VCAP5-DCD exam beyond what I originally anticipated. Last Monday, I looked at the availability of the next exam slot for VCAP5-DCD in a nearby test center and Saturday 21st of June was it. That meant on Saturday morning while people are cheering for their favorite team to score in the World Cup, I was scoring through my VCAP5-DCD. The good part, I have won my game and got the “You have Passed” message at the end of the exam.

vSphere VCAP Design

I thought I might share my experience on here and some tips of what to study.

Let me start with my VCAP5-DCD experience:

  1. This exam is a real marathon. I went to the exam with this being expected already from what I have read on other blogs. While 4 hours for 100 Questions sound like a lot of time, I had only 3 minutes left when I got to question 90. It is important to note, I was already running through questions as quickly as possible as I kinda expected it to be tight on time, but still were barely got to Q99 when the exam timed out.
Read More

vCloud Automation Center and vCenter Appliance

Yesterday, I was asked twice if using our vCenter Server Appliance (vCSA) with vCloud Automation Center (vCAC) is a viable option. As one of the two requests came from inside our VCDX family, It seems to be not a common knowledge, not when I get asked by another VCDX about it.  I thought this seems to be a good topic to address in a blog post, as the following questions seems to come often:

  • Does vCloud Automation Center (vCAC) 6 support vCenter Server Appliance (vCSA)?
  • Do your recommend using vCSA with vCAC?
  • What’s the added benefits of using vCSA  in my vCloud setup?
  • What extra limitations does using the vCenter Server Appliance with vCloud Automation Center impose to my environment?

I am sure you either had one of these questions on your mind or had been asked one of these at least once if you are working on a vCloud Automation Center setup. I will try to address these questions in this post.

Does vCloud Automation Center (vCAC) 6 support vCenter Server Appliance (vCSA)?

vCloud Automation Center and vCenter Appliance

I guess the image above gave the answer away, but for those of you who liked it spelled out, yes vCAC 6 support vCSA. Just make sure you check the vCenter interoperability matrix to ensure you are getting a supported version of the vCenter Server Appliance with the version of vCAC you are planning to install.… Read More

vCAC 6 Missing VM Actions

This problems seems to lately surface more than not when delivery a vCAC 6 Distributed installation. After you complete your vCloud Automation Center 6 installation without a hitch and you thought everything is working perfectly, you go to your Catalog Entitlements and try to assign Actions allowed and you notice that you have only few of what you really should have. The below screenshot provide a sample of which only actions I were able to entitle to my catalog items:

vCAC 6 missing VMs Actions


Before you jump to a conclusion that this the same problem you are having, please ensure that you have assigned the right permissions to your blueprint and entitlements to your Catalog item. For initial testing try to give all permissions to your blueprint as in the below screenshot:

 vCAC provide permissions on Blueprints

After providing the permissions on the vCAC Blueprint then you want to ensure your Catalog entitlements are set correctly. Go to Catalog action entitlements and check which Actions could you entitle to your Catalog item. If it look like the below screenshot then you are definitely having the same problem this article is trying to address as you should have much more actions that you can entitle to your Catalog Items.… Read More

Software Defined PBX

I remember the days when I used to live in an apartment building and when my analog home phone used to stop working due to a line problem, the first thing we used to suspect is one of my neighbor has missed up our building PBX wiring while fixing his phone line or installing a new phone line (more than likely, they did not mean to). I used to call the building maintenance and walk down with them to the PBX box and the fun begin when you try to identify which cable is my home phone line.

As that PBX grid grew, that wires spaghetti got out of hand. It got to a stage where it was almost impossible to add new lines or fix old ones without disturbing someone else phone line. The sad part, this PBX fun is not limited to old apartment buildings but many enterprises still depend on it.  If one of the below images remind you of how painful this sometime get, then keep reading as the solution might be Virtualization Again! And might be cheaper than you always thought.

PBX very hard to manageOld PBX System


As many companies has over grown their PBX system to unmanageable state, VoIP seems to be the natural choice to replace that old hard to manage PBX till companies look at the cost of switching to VoIP and complexity of traditional VoIP solutions they shy away and stick with what they have.… Read More

vCAC 6 Service Unreachable – Reference error REPO404

While delivering a distributed install of  vCloud Automation Center using exactly the same steps I have used for few previous engagements, & while the setup completed perfectly without any errors, accessing the infrastructure tab in vCAC has continuously reported the following error:


Service Unreachable

A required service cannot be reached at the expected address.

Please contact your system Administrator for Assistance.

Reference error REPO404.


vCloud Automation Center 6 repo404 error

I was quite certain I have done the certs right, as I followed my certs guide that I had followed in few other engagements previously and posted it on my blog before at: vCloud Automation Center 6 Certificates A to Z.  Just for your reference if you have not read that post all the certs was generated by Active Directory CA.

As the above error can be caused by few different causes, I have went into checking my different vCAC logs and the error that helped me identify the cause was in the vCAC IaaS Web Server Windows Event log and below is a copy of that error (I marked in red the part that gave it away)

——————————— vCAC 6 SSL/TLS secure channel Error Start ——————

Log Name:      Application

Source:        VMware GUI Administration

Date:          05/28/2014 5:36:52 PM

Event ID:      0

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A



Timestamp: 5/28/2014 9:36:52 PM


Message: Error occurred writing to the respository tracking log

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.Read More

vCAC 5.2 to 6.x Construct Mappings

vCAC 5.x admins/architects might get surprised by vCAC 6.x construct naming and think VMware has abundant the constructs vCloud Automation Center used in the past. After a closer look though, you will notice these construct functionality is still the same as they used to be in 5.x and they were just renamed to fit the wider audience vCAC is currently address and to be better aligned with those constructs functionality. The main different is that a new Tenant Construct was introduced in vCAC 6.x which did not exist in 5.2 as vCAC 5.2 did not support  multi-tenancy.

I get asked quite often about the construct mapping between vCAC 5.2 and 6.x. Further, the longer I have been delivering just vCAC 6.x engagements, I seem to start forgetting the contruct mapping between vCloud Automation Center 5.2 and 6.x, so I decided I will document it here for a reference to myself and others looking for such info. Below is the best diagram I was able to find that highlight the construct mapping between vCAC 5.2 and vCAC 6.x:

vCAC 5.2 to vCAC 6.x Construct Mapping

Hope this help those of you familiar with vCAC 5.2 jump on 6.x with confident :).… Read More

vCloud Automation Center 6 Load Balancer configuration

Every Distributed vCloud Automation Center 6 installation involves configuring load balancing at several levels. As load balancers have historically been the responsibility of the network team, many virtualization/server admins are not comfortable with them.  Further, if your load balancing request is not clear and detailed the network team will return the request back and ask for more clarifications and any extra required details. This seems to give many Cloud/Virtualization admins a hard time when trying to complete a vCloud Automation 6 Distributed install. In a try to help the rest of us get the info they need to pass to the network team to configure the load balancing required to complete a vCAC Distributed install, I will try to provide as much details required in this post.

Let’s start by trying to figuring out where do we need to plug in our load balancers, and which components we need to load balance. How about a diagram which present where load balancers fit in a vCAC Distributed install:

vCAC Distributed install Load balancer

 Below is the required configuration for load balancing vCloud Automation Center 6 at each level:

vCloud Automation Center Appliance (VIP):

The vCAC Appliances will be setup in an Active/Active configuration with the below configuration:

  • Transport Protocol: Https (443)
  • Load balancing Method: Least Response Time
  • Enable session affinity or sticky session  (Source IP or Cookies)
  • Health Check URL: https://<Your vCAC Appliance FQDN>      Ex:

IaaS Windows Virtual Machine (Model Manager and Web Service) – VIP:

  • Transport Protocol: Https (443)
  • Load balancing Method: Least Response Time
  • Enable session affinity or sticky session
  • Health Check URL: https://<Your IaaS Web Machine FQDN>      Ex:

IaaS Windows Virtual Machine (Manager Service) – VIP:

  • Transport Protocol: Https (443)
  • Load balancing Method: Failover
  • Session affinity or stick session will not be setup for Manager Service as it is active passive and the passive node should have no requests sent to it at all.
Read More

Application Director unable to login to cloud provider

I was delivering an engagement lately where I had to integrate VMware Application Director with vCloud Automation Center 6.0.  I have got to the stage where I needed to connect App Director to a Cloud provider.

Although I was certain, I have entered the correct information on the New Cloud Provider screen as shown below, I kept getting the following error: “Could not connect to the Cloud Provider at https://vtvcaciaas Unable to login to cloud provider. Please verify the user credentials as well as other parameters you entered.”

vCAC fail to connect to Application Director

This error was quite frustrating, as I was sure I have provided the correct information as well verified it over a 100 time. After digging for an answer, I was surprised to know that the User Name here is case sensitive, so basically is not equal to I had to go back to my Active Directory Domain Controller, and search for the user and copy it case sensitive as is to this User Name field which in my case was “”. To my surprise this worked every time.

As most of us are used to none case sensitive user name, where every one knows by now password is case sensitive, I can see this same issue catching many others, and hope this help save you a bit of time :).… Read More