How to remove orphaned VM from vCenter the easy way

I have lately had few orphaned VMs in my Home Lab vCenter, as I was recreating my setup. Some of the Virtual machines were deleted directly from ESXi host, but still had records in the vCenter inventory. Below how orphaned VMs looked in my vCenter.

orphaned VM in vCenter

I have looked online for a way to remove these orphaned VMs, and while one of the KB article suggested to add it to a folder then remove the folder, that did not work as that KB was only for older versions of vSphere.

One method to do this, is to use one of the below command lines methods where any of them will do the trick:

PowerCLI
Remove-VM vm_name -deletepermanently

vMA
vmware-cmd --server esxhost –s unregister path_to_vmx_file
vmware-cmd --server vcenter --vihost esxhost –s unregister path_to_vmx_file vifs --server esxhost --rm “[datastore] path_to_vmx_on_datastore

CLI
vim-cmd vmsvc/destroy vmid

For those lazy ones, that don’t want to fire up a command line utility and then construct a command line to do the trick, I have a good news for you. You can delete that orphan VM in the GUI by right clicking the VM, then choosing All Virtual Infrastructure Actions ==> More UnCategorized Actions ==> Remove from Inventory. … Read More

Unable to establish an SSL connection with vCenter Server

While trying to export one of my VMs into OVF from my virtual infrastructure using the vSphere Web Client, I kept getting the following error:  “Could not connect to the remote resource: SSL connect error”. I have as well tested to import OVF templates into my VMware vSphere environment, and I was getting the same error, but this time I was getting one more additional error which states: “Unable to validate that the OVF can be imported on the vCenter Server”. The below screenshot demonstrate what the error look like in my environment.

Could not connect to the remote resource ssl connect error

I have tried to connect to the vSphere Web Client from a jump box within my lab environment and that seemed to work perfectly without a glitch, while my laptop machine kept to suffer from the same error. This lead me to compare my jump box with my own laptop config and the two obvious difference that I were able to identify is my laptop is not in the same domain as my jump box(which is joined to my lab domain), as well I am using a different DNS on my own laptop.

Changing the dns configuration on my laptop to match the machine in my lap seemed to get things working on my laptop as well (I did not need to join the domain).… Read More

Myth busted: vCenter Site Recovery Manager using vSphere Replication for Datacenter migration causes data loss

It seems there is a lot of confusion out there on how vCenter Site Recovery manager work with vSphere Replication when used for a data center migration, thanks in big part to competing products FUDs. Many admins still believe that using vSphere Replication with vCenter Site Recovery Manager for data center migration you will still lose up to 15 minutes of Data. This confusion has evolved due to the following two limitations of vSphere Replication:

  • Lowest RPO possible using vSphere replication  is 15 minutes
  • You cannot replicate powered-off virtual machines. Replication begins when the virtual machine is powered on. You cannot use vSphere Replication to replicate virtual machines templates. <== This statement right of the vSphere documentation.

Here is how the confusion came to life. If you have experienced or read about vCenter Site Recover Manager with storage replication and looked at the sequence of events when doing a data center migration, you will notice it will do a final sync of the data between the two sites right before it cut the replication between the the two sites. If you try to compare the same method with what is happening in vSphere replication and knowing the above two limitations, you will think that when the Data Center Migration is initiated in SRM, it will shutdown the VM and at that time the VM replication was lagging with up to 15 minutes behind based on the provided RPO and as vSphere replication can not replicate after the VM is turned off, the VM will be losing up to 15 minutes of data when coming on the other site, but that is not true as its missing a very minor but important detail that many people seems to over look.… Read More

vCenter Server Appliance (vCSA) 6 limitations removed.

If you have been using the vCenter Appliance (vCSA) in vSphere 5.5 or been considering it, then you are more likely aware of its limitations. You might have in the past as well looked at my blog post that highlighted the vCSA 5.5 limitations found at: vCenter Server Appliance 5.5 limitations. The good news in vCSA 6.0 seems to over come the biggest one of these as follow:

– vCenter Linked mode is now supported with vCSA 6.0. This one by far was the biggest limitation of the vCSA 5.5 and the one that had been most mentioned by my customers who were considering vCSA 5.5 at that time. Now this has been resolved for you and you can use vCenter Linked mode with vCSA 6.0. Wohooooo!

– You were limited to 100 hosts and 3000 VMs when using the embeded DB with vCSA 5.5, where you had to use an external Oracle Database to expand beyond these limits, which did not make larger customers happy. The good news these limits were waived in vCSA 6.0 and you will be able to use up to 1000 hosts and 10,000 VMs using the embedded database. I am sure between this one and the availability of vCenter Linked mode, many customers who were considering to move to the vCSA appliance, they will be eager to put their hands on the vCSA 6.0… Read More

VMware vCenter 6 Installation Steps

I wanted to demonstrate how VMware vCenter 6 installation is a bit different than vCenter 5.x, and the below post show the installation steps of VMware vCenter 6 basic install.

1. Make sure you have minimum requirements and the OS you are using is on the compatibility list of vCenter 6.0. The below table demonstrate the minimum hardware requirements for vCenter 6.0.

2. Make sure the user to run vCenter Server service under has the log on as a service rights

3. Run the vCenter installer executable

4. Hit Install

5. Hit next

6. Accept license agreement and hit next

7. Choose your deployment type and hit Next.

8. Choose System Network Name and hit next

9. Provide the Single Sign-On Configuration & make sure your password meet the password complexity requirements ad demonstrated in the below screenshots

10. Choose the service account to run the vCenter Service under

11. Choose your vCenter DB Configuration

12. Confirm the network ports to use

13. Confirm the installation destination folder

 

14. Hit install

 

 

15. Hit launch the vsphere web client to start using it and hit finish the installation

 

16. Check out your new vCenter in the vSphere web client

 

You are now ready to enjoy your vCenter 6!… Read More

VMware vSphere 6.0 is here!

Today VMware has made a big announcement of the long waited VMware vSphere 6.0, while the bits are not available for download yet, it will follow shortly. The all new vSphere 6.0 is loaded with new features, that many of you were waiting for it for years. In this post, I will cover what’s new with VMware vSphere 6.0 and to get you excited it to try it as soon the bits come out. Please note the bits out there today is only the beta, and the GA bits will follow shortly. Let’s start going through what’s excite me about the new release.

  • VMotion Enhancements seams to break new ground in vSphere 6 with the availability of:
    • Cross vSwitch vMotion
    • Cross vCenter vMotion
    • vMotion L2 adjacency restrictions are lifted
    • vMotion of MSCS VMs using pRDMs
    • Long Distance vMotion – Enable vMotion to operate across distance of greater than 100ms RTTs.
    • Sound to me we are building the ground for vMotion to the Cloud!
  • vSMP Fault Tolerance is finally here where vSphere 6 will allow up to 4 vCPU VMs to be protected by VMware Fault Tolerance.
  • NFS v4.1 with Kerberos support. If you are an NFS shop, then I am sure you have been waiting this for a while!
Read More

Windows could not parse or process the unattend answer file for pass [specialize]

I was at a customer which I have deployed vCloud Automation Center 6.1 for, and they tried to create a new blueprint. They have copied the old blueprint and only changed the VM template used in that blueprint from Windows 2012 to Windows 2008 R2. They saved and published the new blueprint without any problem, then they tried to deploy the new blueprint. The blueprint deployment kept failing with the following error:

“Request failed: Machine VTTestVM001: Timeout customizing machine”

From the error, I was able to tell it was getting stuck at the guest customization stage, so I deployed one more VM and waited till it get to the guest customization stage, then opened a console to the VM to be greeted  with the following Sysprep error:

Windows could not parse or process the unattend answer file for pass [specialize]. The settings specified in the answer file cannot be applied. The error was detected while processing settings for component [Microsoft-Windows-Shell-Setup]”

Below is a screenshot showing the actual error on screen:

Windows could not parse or process the unattend answer file or pass [specialize]

After I hit OK, the machine restarted again and gave the following error and kept repetitively restarting and giving the same error afterword:

“The computer restarted unexpectedly or encountered an unexpected error.Read More

How to configure vCAC not to delete VMs after deployment failure

I have a customer approach me asking of how we can stop vCAC from deleting VMs after deployment has failed for debugging purposes. The default behavior of vCAC is to delete the VM after deployment failure even if that failure was just caused by Guest customization failure or by the Guest Agent execution failure. The problem when a deployment fail due to a Guest Customization failure or by Guest Agent execution failure and the VM get delete as the deployment fail, you are left with no in guest logs to trace the cause of the problem which explain the need for stopping vCAC from deleting VMs after VMs deployment failure for debugging/troubleshooting.

Luckily vCAC Proxy Agent can be configured not to delete VMs after deployment failure and send it to VRMDeleted folder instead of deleting it and it can be deleted manually after you finish your debugging. If you think you need to stop vCAC from deleting VMs from vCenter after deployment failure for troubleshooting or any other reason, then below is how to do just that.

A system administrator can modify proxy agent configuration settings, such as provisioning machine credentials and deletion policy for virtualization platforms, after installation. The proxy agent utility can be used to modify the initial configurations that are encrypted in the agent configuration file.… Read More

vCenter Upgrade Error 28035. Setup failed to copy LDIFDE.EXE from System folder to ‘%windir%\ADAM’ folder

During the upgrade of vCenter I have faced the exact error that is documented in KB#2013675. The error was as follow:

————

Error 28035. Setup failed to copy LDIFDE.EXE from System folder to ‘%windir%\ADAM’ folder

———–

Below is an actual screenshot of the error:

vCenter upgrade error 28035 setup failed to copy LDIFDE

Further, in the vminst.log file, we found the following error:

———-

  • Unable to copy c:/Windows/ADAM/LDIFDE.EXE to C:/Windows/ADAM/LDIFDE.EXE

———

If you have read KB#2013675, then you will notice the above errors are an exact match of the errors documented in that KB. If so why, I am creating this post if a KB is out there that include the solution, because that solution alone which is shown below did not resolve my issue:

——- KB#2013675  Solution Start —-

To resolve this issue, manually install the Active Directory Lightweight Services Role for the server.
To manually install the Active Directory Lightweight Services Role for the server:
  1. Open the Server Manager for the server and click Add Role.
  2. Select the Active Directory Lightweight Directory Services option.
You should be able to install vCenter Server after the Role tasks complete

——- KB#2013675  Solution End —-

If the above solution does not work for you, just like was the case in our situation, then try the below solution after you apply the solution documented in the KB.… Read More

Upgrading vCenter with vCenter Heartbeat

I have went through the upgrade process of vCenter 5.0 to vCenter 5.1 which was protected by vCenter heartbeat(Please note same process apply for upgrade to 5.5 as well). The process end up being more of uninstalling and re-installing vCenter heartbeat rather than upgrade to avoid the undesired behavior of SSO lockdown that can be caused by the upgrade process of vCenter Hearbeat when SSO is hosted on the same VM as vCenter which was the case in my scenario. Let start by giving a summary of the upgrade process before going into a details:

a.  Uninstall vCenter Heartbeat 6.5 if installed
b.  Setup Single Sign On.
c.  Upgrade vCenter Server from to 5.0 to 5.1
d.  Upgrade Update Manager from 5.0 to 5.1
f. Install vCenter Heartbeat 6.6 (UAT & Prod).

below is the process  changes involved when upgrading vCenter that is protected by vCenter Heartbeat in more details:

1-      Preparation:

  • Make sure you have adequate backup. Maybe Image backup of your vCenter VMs?
  • Make sure to have license keys on hand.
  • Make sure All Windows Updates is disabled.
  • Write down all the IPs configurations.
  • Prepare a service account to use for heartbeat with local admin and vcenter admin privillage
  • Make sure to have a local administrator account on the machine that you know the password off.
Read More

vCenter 5.5 Upgrade fails when installing Microsoft Visual C++ Redistributable Package prerequisite with the error: Error Code 3010

While at a customer site and trying to upgrade their vCenter 5.0 to 5.5, every time we tried to run the installer it ran for a bit then gave us the follow error:

vCenter SSO Error 1722

Then the following errors appeared in a log files that was opened automatically in a notepad.

—————–  Error start —————–

Action 12:39:49: VM_InstallVCREDIST_x64. Configuring Microsoft Visual C++ Redistributable Package (x64)…
Action start 12:39:49: VM_InstallVCREDIST_x64.
CustomAction VM_InstallVCREDIST_x64 returned actual error code 3010 (note this may not be 100% accurate if translation happened inside sandbox

MSI (c) (5C:E4) [12:40:32:470]: Note: 1: 1722 2: VM_InstallVCREDIST_x64 3: F:\ 4: D:\Single Sign-On\prerequisites\vcredist_x64.exe /s /v/qn
Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor. Action VM_InstallVCREDIST_x64, location: F:\, command: D:\Single Sign-On\prerequisites\vcredist_x64.exe /s /v/qn

MSI (c) (5C:E4) [12:42:05:103]: Product: vCenter Single Sign-On — Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor. Action VM_InstallVCREDIST_x64, location: F:\, command: D:\Single Sign-On\prerequisites\vcredist_x64.exe /s /v/qn

Action ended 12:42:05: VM_InstallVCREDIST_x64. Return value 3.
MSI (c) (5C:E4) [12:42:05:103]: Doing action: FatalError
Action 12:42:05: FatalError.Read More

vCenter Server Appliance 5.5 SSO Issues

While rebuilding my home lab to use the latest version of vCloud Automation Center, I have decided to rebuild it from scratch with the latest vCenter Server Appliance and ESXi. After deploying the vCenter Appliance 5.5c and going through the configuration, I wanted to join the appliance to my domain and allow users from my domain to authenticate.  While I was able to join the appliance to my domain and that seemed to work just fine as in the below screen shot, I was having a problem configuring SSO for native active directory. I kept getting the following error:  ” ‘alias’ value should not be empty ”

Alias Value should not be empty

The solution for this particular problem was easy, actually it was more of Ooops I have forget to restart the vCenter Appliance after joining the appliance to my domain. Actually it tell you to do so as in the below screenshot:

You actually need to restart the appliance after configuring AD

I wanted to document this, as I am positive there will be the case where others will miss this as well and try to fight this error and thought I will save them time. This is especially true that I have seen other posts on forums and blogs that claim rebooting after this step is not required although the documentation state so.… Read More

vCenter Appliance fsck failed

While I have been working with a customer on upgrading their infrastructure to vSphere 5.5, they had a power outage during the night which got their full virtual infrastructure down including storage (Not sure what the story with their UPS/Power generator). After they have brought their virtual infrastructure backup up both their vCenter Appliance and VDP appliance and few other Linux VMs has failed the fsck check while it was booting and it was stuck at the following error:

=========================================

fsck failed.  Please repair manually and reboot.  The root
file system is currently mounted read-only.  To remount it
read-write do:

bash# mount -n -o remount,rw /

Atention:  Only CONTROL-D will reboot the system in this
maintanance mode.  shutdow or reboot will not work.

Give root password for maintenance
(or type Control-D to continue):

==========================================

Below is an actual screenshot of the error for you to compare it with what you got:

VMware vCSA fsck failed

Below is the steps I have followed to fix this on both vCSA and VDP Advanced:

  1. Stay Calm & Don’t Panic!
  2. Take a backup/snapshot in case the below steps does not work for you and you need to revert it back to the current state when you call support or call for help.
Read More

vCenter Single Sign On 5.5 Whats New

vCenter Single Sign On has a considerable amount of changes in vSphere 5.5, with few major ones. Many of these changes have went undetected or unnoticed by the Virtual Infrastructure Admins. If you have deployed vSphere 5.5 and missed these changes or planning to install vCenter SSO 5.5 and want to learn what has changed from the vSphere 5.1 days, then this post is for you:

Below is the list of the major changes introduced in vCenter Single Sign On:

 vCenter SSO Architecture Improvements:

1- Multi master: Unlike 5.1, vCenter SSO 5.5 has A built-in automatic multi-master replication architecture that ensure that all SSO instances are always kept in sync. While this sound great, most admins are not sure what it means nor how it affect the way they design SSO. To understand the value of this change, you must understand how SSO worked in 5.1. and how that changed in 5.5.

In 5.1 if you wanted to enable SSO for multiple vCenters in your environment, you needed to point all of them to the same SSO instance which made all of those vCenters dependent on that single SSO instance. This has made that single SSO instance to be crucial for the operation of all of your vCenters, as if that SSO instance goes down you will not be able to access any of your vCenters.… Read More

vCenter Server Appliance Error: failed to connect to VMware Lookup Service

While working with VMware vCenter Server Appliance 5.5, I needed to change the IP Address of the appliance as my customer were changing their IP scheme. It was quite to go to the VMware vCSA then to the network tab then Address tab and change the IP Address. After rebooting the appliance to ensure the IP change has taken affect and is working properly and consistent, I was able to login to the vCenter Server Appliance Management portal, but I was not able to connect to using the Web Client and was kept getting the following error:

“Failed to connect to VMware Lookup Service https://<vCenter Appliance IP>/lookupservice/sdk – SSL certificate verification failed”

Or as shown in the below screenshot:

Web Client Error after changing vCenter Appliance IP

 

It seems changing the IP Address of the vCenter Server Appliance does not regenerate the certificate with the new IP and you will have to configure the appliance to generate a new one manually. The good news it is pretty easy to do, and below is the instruction:

  1. Log in to the vCenter Server Appliance Web interface at https://<vCSA IP>:5480/.
  2. Click the Admin tab.
  3. Regenerate certificates:
    • vCenter Server 5.1: Select Toggle certificate setting so that the Certificate regeneration enabled displays Yes.
Read More