Host Profiles – Number of ipv4 routes did not match

As we have setup a new cluster of four ESXi 6.5 hosts, we have extracted a host profile from one of the hosts that we have setup to my customer corporate standards. After applying that host profiles to the other three hosts, we were able to quickly resolove all non-compliant issues except one where it keep complaining: “Number of ipv4 routes did not match”.

There is a good KB article on this which can be found at: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2106112

We had two challenges with KB2106112 though, which I wanted to document to help others stumbling with the same issue as follow:

1- The KB state it is only covering up to vSphere 6.0, where our environment was vSphere 6.5 and the fix still apply

2- The KB only stated to look at the “defaultTcpipStack” Netstack Instance, where in our case it was more to do with the “VMotion” Netstack Instance. Exactly same resolution, but you need to apply it to the “VMotion” Netstack Instance.

As I have mentioned above, this can affect any of your Netstack Instances, so while I am demonstrating how to fix it for the “VMotion” Netstack Instance, you can follow exactly the same steps to fix it for other Netstack Instances include the “defaultcpipStack”.… Read More

How to Downgrade VM Hardware Version

I have often got the question if you can downgrade your Virtual Machine Hardware version from the latest version to a previous version. There is many reasons why you might want to downgrade your VM Hardware Version, but below is few common ones I have heard:

  • VM hardware version you upgraded to is not supported in the vSphere thick client version you are using, this used to be the most common one in vSphere 5.5. Luckily this has dramatically improved in vSphere 6.0.
  • VM hardware version you upgraded to is not supported on older versions hosts that you still keeping or waiting to upgrade.
  • VM hardware version you upgraded to is not supported by your cloud vendor, so you can’t sync your internal VMs/templates to your cloud.

I guess if you had hit this post, you already have a reason of why you want to downgrade your VM Hardware version, so I will not spend too much time on that. Let’s look on how to downgrade VM Hardware Version.

I have to first admit that there is no straight out of the box button that allow you to downgrade your VM Hardware version, unlike when you upgrade it. This might be the reason why when you ask many people about it out there, they will tell you there is no supported way of doing it, but that’s not exactly true.… Read More

My Home Lab Converged Infrastructure Challenge

As I had some challenges with my Home Lab due to some hardware failure, I had to revert from using a dedicated storage box (White box + Windows + bunch of disk + StarWind Virtual SAN) as I no longer have the luxury of having that extra machine to use. I have ended up with two ESXi hosts (Specs: White box with i5 + 16GB RAM + 1x SSD + 2x Sata Drivers + 2 NICs). I have though to investigate if I could still come up with a way that would provide me with a reliable storage without having to acquire any extra hardware.

While I have considered to buy another dedicated box to use as storage, converged infrastructure seemed as a great viable alternative. I decided to investigate, which option I had within the popular converged infrastructure offering that I knew of would fit my needs best. I can see quite few small businesses that might fall in the same boat where they have been limited to two lower specs servers to run their small environment. Here are the different options I have considered, and why I could not use them.

Simplivity: I wanted to check them out as they have a very interesting offering especially when it comes to deduplication which could save me a good amount of disk space.… Read More

Unable to establish an SSL connection with vCenter Server

While trying to export one of my VMs into OVF from my virtual infrastructure using the vSphere Web Client, I kept getting the following error:  “Could not connect to the remote resource: SSL connect error”. I have as well tested to import OVF templates into my VMware vSphere environment, and I was getting the same error, but this time I was getting one more additional error which states: “Unable to validate that the OVF can be imported on the vCenter Server”. The below screenshot demonstrate what the error look like in my environment.

Could not connect to the remote resource ssl connect error

I have tried to connect to the vSphere Web Client from a jump box within my lab environment and that seemed to work perfectly without a glitch, while my laptop machine kept to suffer from the same error. This lead me to compare my jump box with my own laptop config and the two obvious difference that I were able to identify is my laptop is not in the same domain as my jump box(which is joined to my lab domain), as well I am using a different DNS on my own laptop.

Changing the dns configuration on my laptop to match the machine in my lap seemed to get things working on my laptop as well (I did not need to join the domain).… Read More

Myth busted: vCenter Site Recovery Manager using vSphere Replication for Datacenter migration causes data loss

It seems there is a lot of confusion out there on how vCenter Site Recovery manager work with vSphere Replication when used for a data center migration, thanks in big part to competing products FUDs. Many admins still believe that using vSphere Replication with vCenter Site Recovery Manager for data center migration you will still lose up to 15 minutes of Data. This confusion has evolved due to the following two limitations of vSphere Replication:

  • Lowest RPO possible using vSphere replication  is 15 minutes
  • You cannot replicate powered-off virtual machines. Replication begins when the virtual machine is powered on. You cannot use vSphere Replication to replicate virtual machines templates. <== This statement right of the vSphere documentation.

Here is how the confusion came to life. If you have experienced or read about vCenter Site Recover Manager with storage replication and looked at the sequence of events when doing a data center migration, you will notice it will do a final sync of the data between the two sites right before it cut the replication between the the two sites. If you try to compare the same method with what is happening in vSphere replication and knowing the above two limitations, you will think that when the Data Center Migration is initiated in SRM, it will shutdown the VM and at that time the VM replication was lagging with up to 15 minutes behind based on the provided RPO and as vSphere replication can not replicate after the VM is turned off, the VM will be losing up to 15 minutes of data when coming on the other site, but that is not true as its missing a very minor but important detail that many people seems to over look.… Read More

vCenter Server Appliance (vCSA) 6 limitations removed.

If you have been using the vCenter Appliance (vCSA) in vSphere 5.5 or been considering it, then you are more likely aware of its limitations. You might have in the past as well looked at my blog post that highlighted the vCSA 5.5 limitations found at: vCenter Server Appliance 5.5 limitations. The good news in vCSA 6.0 seems to over come the biggest one of these as follow:

– vCenter Linked mode is now supported with vCSA 6.0. This one by far was the biggest limitation of the vCSA 5.5 and the one that had been most mentioned by my customers who were considering vCSA 5.5 at that time. Now this has been resolved for you and you can use vCenter Linked mode with vCSA 6.0. Wohooooo!

– You were limited to 100 hosts and 3000 VMs when using the embeded DB with vCSA 5.5, where you had to use an external Oracle Database to expand beyond these limits, which did not make larger customers happy. The good news these limits were waived in vCSA 6.0 and you will be able to use up to 1000 hosts and 10,000 VMs using the embedded database. I am sure between this one and the availability of vCenter Linked mode, many customers who were considering to move to the vCSA appliance, they will be eager to put their hands on the vCSA 6.0… Read More

VMware vCenter 6 Installation Steps

I wanted to demonstrate how VMware vCenter 6 installation is a bit different than vCenter 5.x, and the below post show the installation steps of VMware vCenter 6 basic install.

1. Make sure you have minimum requirements and the OS you are using is on the compatibility list of vCenter 6.0. The below table demonstrate the minimum hardware requirements for vCenter 6.0.

2. Make sure the user to run vCenter Server service under has the log on as a service rights

3. Run the vCenter installer executable

4. Hit Install

5. Hit next

6. Accept license agreement and hit next

7. Choose your deployment type and hit Next.

8. Choose System Network Name and hit next

9. Provide the Single Sign-On Configuration & make sure your password meet the password complexity requirements ad demonstrated in the below screenshots

10. Choose the service account to run the vCenter Service under

11. Choose your vCenter DB Configuration

12. Confirm the network ports to use

13. Confirm the installation destination folder

 

14. Hit install

 

 

15. Hit launch the vsphere web client to start using it and hit finish the installation

 

16. Check out your new vCenter in the vSphere web client

 

You are now ready to enjoy your vCenter 6!… Read More

VMware vSphere 6.0 is here!

Today VMware has made a big announcement of the long waited VMware vSphere 6.0, while the bits are not available for download yet, it will follow shortly. The all new vSphere 6.0 is loaded with new features, that many of you were waiting for it for years. In this post, I will cover what’s new with VMware vSphere 6.0 and to get you excited it to try it as soon the bits come out. Please note the bits out there today is only the beta, and the GA bits will follow shortly. Let’s start going through what’s excite me about the new release.

  • VMotion Enhancements seams to break new ground in vSphere 6 with the availability of:
    • Cross vSwitch vMotion
    • Cross vCenter vMotion
    • vMotion L2 adjacency restrictions are lifted
    • vMotion of MSCS VMs using pRDMs
    • Long Distance vMotion – Enable vMotion to operate across distance of greater than 100ms RTTs.
    • Sound to me we are building the ground for vMotion to the Cloud!
  • vSMP Fault Tolerance is finally here where vSphere 6 will allow up to 4 vCPU VMs to be protected by VMware Fault Tolerance.
  • NFS v4.1 with Kerberos support. If you are an NFS shop, then I am sure you have been waiting this for a while!
Read More

Learn Virtualization & VMware vSphere from Scratch

Few months back I got to speak to different friends of mine in the IT field, who were specialized in networking, storage, SAP, Database, Hardware and so on. I was surprised they had zero to little exposure to Virtualization although they are well versed into their field of expertise. They even held highest certifications in their fields like CCIE & the equivalent for the different specialty. This has kinda surprised me as working with Virtualization/Cloud Admins on daily basis, I had thought most IT professional by now are well versed with Virtualization. This has proved being far from the true, the more I speak to IT Professionals outside the Virtualization field.

Figuring out that many IT Professionals are still stuck in the past within the physical infrastructure, I decided to explore the cause and try to help improve the situation. Asking my friends about the cause of not catching up on Virtualization although they are aware its rapidly changing every IT field. I asked my networking friends if they are already aware that there is more virtual network ports than there is physical these days and they were surprised of that fact. I asked them what will they do when 90% of the network ports become virtual network ports and a part of a virtual switch.… Read More

vCenter Server Appliance 5.5 SSO Issues

While rebuilding my home lab to use the latest version of vCloud Automation Center, I have decided to rebuild it from scratch with the latest vCenter Server Appliance and ESXi. After deploying the vCenter Appliance 5.5c and going through the configuration, I wanted to join the appliance to my domain and allow users from my domain to authenticate.  While I was able to join the appliance to my domain and that seemed to work just fine as in the below screen shot, I was having a problem configuring SSO for native active directory. I kept getting the following error:  ” ‘alias’ value should not be empty ”

Alias Value should not be empty

The solution for this particular problem was easy, actually it was more of Ooops I have forget to restart the vCenter Appliance after joining the appliance to my domain. Actually it tell you to do so as in the below screenshot:

You actually need to restart the appliance after configuring AD

I wanted to document this, as I am positive there will be the case where others will miss this as well and try to fight this error and thought I will save them time. This is especially true that I have seen other posts on forums and blogs that claim rebooting after this step is not required although the documentation state so.… Read More

Assign Domain Users vCenter Single Sign Administrator Privileges

If you have upgraded your Virtual Infrastructure to vSphere 5.1 or 5.5, you are already aware of the addition of Single Sign On. When installing Single Sign On, a default user Admin@System-Domain is created for you with a password that you have assigned to it during the vCenter Single Sign On installation. The Admin@System-Domain user is initially the only single user that have access to manage your the Single Sign On portion of your vSphere environment.

As a good practice & to be able to track who is responsible for a particular change in your SSO you might want to assign your vSphere Administration team domain accounts administrator privillages to your vCenter Single Sign On. While it is an easy task to do, the way it must be done seems to confuse many admins who is new to vCenter SSO. Below is the instruction of doing so:

  1. Browse your vSphere Web Client Portal (https://<Your vCenter Machine>:9443).  (Note: SSO only can be managed using the Web Client)
  2. Browse to Administration > Access > SSO Users and Groups in the vSphere Web Client.
  3. Click on the Groups Tab
  4. Click on the desired Group (_Administrators_)
  5. Click Add Principals (The icon of a person with a plus sign next to it highlighted in the above screenshot)
  6. Select the identity source that contains the principal to add to the group (Probably your Domain)
  7. Search for the desired user
  8. Select the desired user and click Add
  9. Repeat step 6-8 to add the rest of the desired users.
Read More

Backup & Archive to the Cloud with PHD Virtual Backup 6.5

PHD Virtual has approached me for feedback on their upcoming PHD Virtual Backup 6.5. I have been granted access to an early beta version of the product to try it out, while they have added many enhancement to the product, the one that have got my attention the most is being able to send your backup or Archive directly to the Cloud.  As I have not tried their backup to the Cloud feature which was released in 6.2, I have decided to test out combined with their new backup archiving to the Cloud feature to be introduced in PHD Virtual Backup 6.5.

With many Storage Cloud Services out there & many more expected to surface in upcoming few months, this backing up & in particular archiving could become of high demand. Further, as many of the Storage Cloud Services offers a price per GB that is very hard to beat with in-house  storage, the solution will become attractive from cost perspective. As most things in the market today, the bigger bulk you buy of a certain item, the less you have to pay per item. This explain why Costco & Walmart(Mega Stores in general) for example can offer cheaper prices than smaller super markets.… Read More

ATS-Only VMFS Volume ‘VMFS5’ not mounted. Host does not support ATS or ATS initialization has failed.

While being at a customer site last week, I was asked if I could help with a mysterious VMFS datastore behavior. That particular datastore out of a sudden became none accessible and they could not carry out any changes to it. They can not VMotion in or out of it, or even create a folder into it. After running Storage Adapters Rescan on some of the ESXi hosts, they could not view that datastore any more. Checking out the logs at /var/log/vmkernel.log, we have noticed the error “ATS-Only VMFS Volume ‘VMFS5′ not mounted. Host does not support ATS or ATS initialization has failed.” shown in the below screenshot (Double Click it for full size).

ATS-Only VMFS Volume ‘VMFS5′ not mounted. Host does not support ATS or ATS initialization has failed.

What is Atomic Test & Set (ATS)?

Before I go about explaining the cause of the above error & how to resolve it, I thought it will make  sense to share some background about ATS and where is the idea of ATS-Only VMFS Volumes has came from which directly relate to the cause of this problem.

Atomic Test & Set (ATS) was introduced as one of the fundamentals operations of vStorage API Array Integration(VAAI). ATS is used during creation and locking of files on the VMFS volume.… Read More

How to replace vCenter 5.1, SSO, Web Client, vCO Certificates

With the release of vSphere 5.1 certificates started to play a much more vital role, where having invalid certificates in your environment is not an option anymore as it could break the operation of your environment as well forbid you from logging in. This change has been done to increase the security of your Virtual Infrastructure Management Components (vCenter Service, Inventory Services, SSO, Web Client, vCO, Update Manager, & vCenter Log Browser) & to compact the possibilities of man in the middle attacks. This change has brought a lot of challenges to many VMware customers who had invalid and expired certificates in their environment without even noticing it. The tedious process of replacing any of these certificates have not been a pleasure work for many, the good news is that VMware has just released vCenter Certificate Automation Tool 1.0 to streamline the process & release much of that pain.

VMware has just announced the general availability of vCenter Certificate Automation Tool 1.0. This tool provides an automated mechanism to replace certificates in the following components of the vCenter management platform:

  • vCenter Server
  • vCenter Single Sign On
  • vCenter Inventory Service
  • vSphere Web Client
  • vCenter Log Browser
  • vCenter Orchestrator (VCO)
  • vSphere Update Manager (VUM)

The tool can be downloaded for free from: https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/5_1#drivers_toolsRead More

vCenter Single Sign On 5.1 best practices

Since vCenter Single Sign On was introduced in vSphere 5.1, many questions have been rising around it. There seems to be a very limited amount of resources out there that document best practices related to vCenter Single Sign On, which is the reason for me to develop this post where I will try to combine as many best practices and answers related to vCenter 5.1 Single Sign On as possible.

I have been one of the lucky consultants who has already got to design/implement vSphere 5.1 for quite few enterprise customers where I have got to debate and drive best practices that I used across those implementations. I am sharing them here where others can benefit from them as well to allow a room for others to debate them and contribute their feedback.

Where to install vCenter Single Sign On (Physical vs Virtual)?

Just as the recommendations have always been for vCenter using virtual machine(s) is the best practice to save on cost and benefit of the availability features built in vSphere, that is no difference in vSphere 5.1. You can host all vCenter 5.1 components including SSO on virtual or physical machine, where virtual machine is the recommended practice due to the same reason mentioned earlier.… Read More