vCenter 6.7 Appliance fails after the first reboot

I have been getting few requests lately of vCenter 6.7 Appliance installation fails after the first reboot. It sounds like a common theme mostly in lab environments. Below is one of the quite few similar errors that I have received.

“A problem occurred during setup. Services might not be working as expected – Starting VMware Component Manager ….”

vCenter Server Fails upon first reboot

Unfortunately, the error is not very perspective. Checking on the cause of this issue for couple of friends, and it end up being caused by a time difference between their ESXi hosts, and the being deployed vCenter Appliance. While most take care of NTP configuration religiously in production environment, that seems to often being skipped into lab environments. As vCenter 6.7 & higher time sync is becoming more crucial than ever, which in turn can cause your installation to fail. KB56124 is covering this particular issue, and provide more details on what error to expect in the log file in such situation.

In both cases that I have helped my friends with, they have inserted an NTP server to sync the vCenter appliance with but forgot to do the same for their ESXi host. The way to setup NTP on your ESXi host is as follow:

  1. Login to your host using the vSphere Web Client
  2. Go to Host => Manage => System => Time & Date
  3. Hit Edit Settings
  4. Configure the NTP Server as in the below screenshot & Save
  5. Start the NTP Service Client on the ESXi host, by going to Actions=>NTP Service => Start

Configure NTP client on ESXi 6.7 hostRead More

Host Profiles – Number of ipv4 routes did not match

As we have setup a new cluster of four ESXi 6.5 hosts, we have extracted a host profile from one of the hosts that we have setup to my customer corporate standards. After applying that host profiles to the other three hosts, we were able to quickly resolove all non-compliant issues except one where it keep complaining: “Number of ipv4 routes did not match”.

There is a good KB article on this which can be found at:

We had two challenges with KB2106112 though, which I wanted to document to help others stumbling with the same issue as follow:

1- The KB state it is only covering up to vSphere 6.0, where our environment was vSphere 6.5 and the fix still apply

2- The KB only stated to look at the “defaultTcpipStack” Netstack Instance, where in our case it was more to do with the “VMotion” Netstack Instance. Exactly same resolution, but you need to apply it to the “VMotion” Netstack Instance.

As I have mentioned above, this can affect any of your Netstack Instances, so while I am demonstrating how to fix it for the “VMotion” Netstack Instance, you can follow exactly the same steps to fix it for other Netstack Instances include the “defaultcpipStack”.… Read More

How to Downgrade VM Hardware Version

I have often got the question if you can downgrade your Virtual Machine Hardware version from the latest version to a previous version. There is many reasons why you might want to downgrade your VM Hardware Version, but below is few common ones I have heard:

  • VM hardware version you upgraded to is not supported in the vSphere thick client version you are using, this used to be the most common one in vSphere 5.5. Luckily this has dramatically improved in vSphere 6.0.
  • VM hardware version you upgraded to is not supported on older versions hosts that you still keeping or waiting to upgrade.
  • VM hardware version you upgraded to is not supported by your cloud vendor, so you can’t sync your internal VMs/templates to your cloud.

I guess if you had hit this post, you already have a reason of why you want to downgrade your VM Hardware version, so I will not spend too much time on that. Let’s look on how to downgrade VM Hardware Version.

I have to first admit that there is no straight out of the box button that allow you to downgrade your VM Hardware version, unlike when you upgrade it. This might be the reason why when you ask many people about it out there, they will tell you there is no supported way of doing it, but that’s not exactly true.… Read More

My Home Lab Converged Infrastructure Challenge

As I had some challenges with my Home Lab due to some hardware failure, I had to revert from using a dedicated storage box (White box + Windows + bunch of disk + StarWind Virtual SAN) as I no longer have the luxury of having that extra machine to use. I have ended up with two ESXi hosts (Specs: White box with i5 + 16GB RAM + 1x SSD + 2x Sata Drivers + 2 NICs). I have though to investigate if I could still come up with a way that would provide me with a reliable storage without having to acquire any extra hardware.

While I have considered to buy another dedicated box to use as storage, converged infrastructure seemed as a great viable alternative. I decided to investigate, which option I had within the popular converged infrastructure offering that I knew of would fit my needs best. I can see quite few small businesses that might fall in the same boat where they have been limited to two lower specs servers to run their small environment. Here are the different options I have considered, and why I could not use them.

Simplivity: I wanted to check them out as they have a very interesting offering especially when it comes to deduplication which could save me a good amount of disk space.… Read More

Unable to establish an SSL connection with vCenter Server

While trying to export one of my VMs into OVF from my virtual infrastructure using the vSphere Web Client, I kept getting the following error:  “Could not connect to the remote resource: SSL connect error”. I have as well tested to import OVF templates into my VMware vSphere environment, and I was getting the same error, but this time I was getting one more additional error which states: “Unable to validate that the OVF can be imported on the vCenter Server”. The below screenshot demonstrate what the error look like in my environment.

Could not connect to the remote resource ssl connect error

I have tried to connect to the vSphere Web Client from a jump box within my lab environment and that seemed to work perfectly without a glitch, while my laptop machine kept to suffer from the same error. This lead me to compare my jump box with my own laptop config and the two obvious difference that I were able to identify is my laptop is not in the same domain as my jump box(which is joined to my lab domain), as well I am using a different DNS on my own laptop.

Changing the dns configuration on my laptop to match the machine in my lap seemed to get things working on my laptop as well (I did not need to join the domain).… Read More

Learn Virtualization & VMware vSphere from Scratch

Few months back I got to speak to different friends of mine in the IT field, who were specialized in networking, storage, SAP, Database, Hardware and so on. I was surprised they had zero to little exposure to Virtualization although they are well versed into their field of expertise. They even held highest certifications in their fields like CCIE & the equivalent for the different specialty. This has kinda surprised me as working with Virtualization/Cloud Admins on daily basis, I had thought most IT professional by now are well versed with Virtualization. This has proved being far from the true, the more I speak to IT Professionals outside the Virtualization field.

Figuring out that many IT Professionals are still stuck in the past within the physical infrastructure, I decided to explore the cause and try to help improve the situation. Asking my friends about the cause of not catching up on Virtualization although they are aware its rapidly changing every IT field. I asked my networking friends if they are already aware that there is more virtual network ports than there is physical these days and they were surprised of that fact. I asked them what will they do when 90% of the network ports become virtual network ports and a part of a virtual switch.… Read More

VMware Health Check Analyzer stuck

Alright this post is dedicated to my colleagues at VMware as well VMware partners who got access to the VMware Health Check Analyzer and use it to help them collect data to include in their vSphere health check report. I have noticed when running the VMware Health Check analyzer with the default options against a larger environment it get stuck at the inventory data collection stage or some time even at the inventory discovery stage. Increasing the number of CPU and memory of the VHA or the machine where the virtualized heath check analyzer application run does not seems to resolve this issue.

After researching internally for sometime, I have found out that adding extra memory to the appliance itself does not automatically allow the vSphere Health Check Analyzer collector to use it. You will actually have to increase the collector process memory setting using the VMware Health Check Analyzer web interface. I have tested this with VHA 3.3.2, though it should work with most other versions out there.

The below procedure explain how you can increase the collector process memory setting for your vSphere health check analyzer tools.

  1.   In the Health Analyzer UI click the Admin menu and choose “Config Settings”.
Read More

Why VMware Load Based Teaming (LBT)

VMware load based Teaming (LBT) is one of the great feature that was introduced a while back with vSphere 4.1, though it seems many admins have over looked it. As part of my job in consulting, I get to see many customers environments through out the year. It is really common to see customers who never get their hands or heads around LBT. What surprise me the most that many of these are in a desperate need for a similar feature. Many of them already have the Enterprise Plus license which include LBT in it, & its just a matter of enabling to start benefiting of it. For these reasons, I have decided to share why the Load Based Teaming feature offered within our Distributed switches is a great feature and can be of a great use to many of you out there.

The easiest way to see how beneficial LBT can be is to look at the problem it solve. Imagine you are using a virtual switch with the default setting of “Route based on the originating Virtual Port ID” or “Route based on MAC addresses” for the teaming load balancing policy. The way  “Route based on the originating Virtual Port ID” & “Route based on MAC addresses” distribute VMs network load across your uplinks by distributing the number of VMs evenly across all the vNICs involved in the team.… Read More

Free Transition to ESXi essential training

As many organization today are in the process for moving from VMware ESX to to ESXi, it has been important to VMware to educate its partners & customers of the process and difference.  For that VMware is offering a free web training “Free Transition to ESXi essential training” to everyone who want to learn more about ESXi. The training can be found at:  Free Transition to ESXi essential training Link . I hope you wall enjoy this free training. Free as in free Beer :).

Ah below are the description of the training as officially was posted on the VMware site:

Title: Transition to ESXi Essentials


– Format: Self-Paced

– Length: 4 Hours

Overview: This self-paced training course covers the requirements and effects of transitioning your VMware vSphere™ environment to VMware® ESXi. It provides the knowledge necessary to make fundamental design decisions and successfully add ESXi to a deployed vSphere environment. This course is based on ESXi 4.1.

Special Note: For those who prefer to learn this content in a classroom environment, whether virtual or face-to-face, there are several two-day classes scheduled worldwide.  For a fee, you will receive the added value of hands-on lab exercises as well as interaction with fellow students and an instructor.… Read More

VMware ESX is End of Life, in the future only VMware ESXi will exist

Going through VMworld 2010 video, I was not too surprised of getting the news that VMware ESX will be discontinued & only VMware ESXi will exist forward. Actually it seemed a news that had been circulating for a while, but finally its official now. Watch the below VMworld video from VMworld TV which clearly state that:

Ok, so what does this mean to me. I thought I will share my opinion of this decision.

Although VMware customers will have to pay some efforts to upgrade to ESXi, the upgrade process more than likely will be streamed line & not much more efforts than doing the normal upgrade between ESX releases. Though the gained benefits will be much worth it.

VMware ESXi has a much smaller foot print than ESX, which ensure it has a higher reliability and security level. Further, as ESXi get rid of the service console (a big chuck of the ESX code) it reduce the amount of patching required for ESX tremendously. In regards of the service console command line access, RCLI & Support Tech mode shall do the trick of replacing it. VMware has made a great development on both that I believe they can replace the service console command line in the future.… Read More

When VMware ESX & When VMware ESXi

I have seen tons of articles talking about the differences between VMware ESX & VMware ESXi, but I have not seen many that discuss when to use them. I have noticed that many of my customers get even more confused when reading comparisons between the two, as not all of them have deep understanding of the Virtualization Technology. That means they got to know the differences, but still wondering which one is best fit for their environments. Below is few rules that can put you on the right path.

– Do you need a free but reliable Virtualization platform as you don’t have a budget? or maybe you wanna use it for non critical applications? maybe you don’t want to pay for fancy features or an official support. In this case VMware ESXi is for you. As ESXi offer a free version, where a license can be applied to it at a later stage when you need to go fancier or require official support.

– Is security & reducing the required patching time is one of your main concerns? then you should consider using VMware ESXi as it does not have a service console which requires most of the patching time.… Read More

VMWare ESXi editing the Hostfile

If you are as paranoid with redundancy as my team & I are, then entering the hostnames & IPs of all the server in a cluster into the VMware host file is a must. As your VMware HA is dependant on your servers being able to resolve the names of itself & other ESX server in the cluster, the DNS get to play a major role in HA. What if DNS fail? As many customers are using Microsoft DNS as their primary means of DNS, there is a chance it will fail a day or another. Would you like a way to ensure that your HA will work even if your DNS fail? then Adding these records to the host files of each of the ESXi server in the cluster will do the trick. What if you did not even have a DNS & still want to implement HA & other feature that depend on name resolution, then again adding the records to the host file is the solution.

The steps below show what to do after you login to the console of your VMware Server. If you are using ESX, then logging in to the service console is straight forward.… Read More

VMware ESXi – Console Access (Unsupported)

Did the title bring your attention? Have you heard that one of the main reasons for VMware ESXi was to get rid of the service console? Are you sure there is no console in VMware ESXi? The true is there is still a very tiny console in VMware ESXi, though it has way smaller foot print than the one found in the ESX version. In addition, the usage of the VMware ESXi Console Access is not supported unless if instructed by VMware to do so. Please use this tip wisely & don’t try to do wild changes using your VMware ESXi console as that is not supported. Yeah, again I would repeat its not supported :).

Hmm you still insist that you want to know how to do so, although its not supported. I can admit it, it can get to be handy if you know what you are doing. Below is the steps of how to access the ESXi Console:

To access the VMware ESXi console (Not Supported) do the following.

1. Open the console of your ESXi host, you should see a screen like the below:

VMWare ESXi default screen

2. Use the key-combo Alt+F1, and you’ll get a virutal terminal (vtty1) with some log messages in it like below:

VMWare ESXi console after alt+f1

3- After you hit Alt+F1 & end up with the screen above, you will notice that what ever you type does not get on the screen & does not get interrupted. … Read More

ESXi cmd addnode failed for secondary node: /opt/vmware/aam/bin/ft_startup failed to complete within three minutes

In the past few weeks, I had been hit by many calls of many of my colleagues & customers being hit by the following error when trying to configure VMware ESXi HA:

“cmd addnode failed for secondary node: /opt/vmware/aam/bin/ft_startup failed to complete within three minutes”

After investigating the error for sometime & searching on the internet I found many misleading trial and error solutions for this error around Google results. The resolution which seems to work for everyone of my colleagues & customers is below.

1- Disable HA for the affected cluster

2- Delete the User Worlds Swap file called uwswap from all the nodes in the problematic cluster.

3- Enable HA for the affected cluster again.

Note: If its possible to move the uwswap file to the local harddisk do it by all mean, as its much better than putting it on the shared storage & will save you a lot of headache.

Recommendation: Add all the host short names & fully qualified names to the host files of all the ESXi servers in that cluster. If you have a doubt on how to do so let me know & I will make a post on how to do this :). … Read More

Changing the VLAN ID of a ESX Service Console using Command Line

A good amount of ESX admins ask on dialy basis how to change the VMware ESX Service Console Vlan using command line. As I decided to save time by writing the steps for these kind of questions to send it to others when asked. I will document these commands in here:

First of all check which vSwitch the Service Console is on (and the name of the Service Console) with esxcfg-vswitch -l (Note: The default service Console switch is vSwitch0 & default name is “Service Console”.)

To set a vlan id on the service console use:

esxcfg-vswitch vSwitch0 -v X -p “Service Console” (Replace X with the desired VLAN number)

To remove the vlan id completely, you will need to set it to 0 using the following command:

esxcfg-vswitch vSwitch0 -v 0 -p “Service Console”

Please note both commands above had assumed the name of your service console & switch are “Service Console” & vSwitch0 as they more probably are as these are the default values. If you have changed them, please replace these with your values.

I hope this tip will help many of you out there. If I had gave you the link to fix your problem as a friend or a customer, then I hope it save you the cost of an engineer trip to your place :).… Read More