Alright this post is dedicated to my colleagues at VMware as well VMware partners who got access to the VMware Health Check Analyzer and use it to help them collect data to include in their vSphere health check report. I have noticed when running the VMware Health Check analyzer with the default options against a larger environment it get stuck at the inventory data collection stage or some time even at the inventory discovery stage. Increasing the number of CPU and memory of the VHA or the machine where the virtualized heath check analyzer application run does not seems to resolve this issue.

After researching internally for sometime, I have found out that adding extra memory to the appliance itself does not automatically allow the vSphere Health Check Analyzer collector to use it. You will actually have to increase the collector process memory setting using the VMware Health Check Analyzer web interface. I have tested this with VHA 3.3.2, though it should work with most other versions out there.

The below procedure explain how you can increase the collector process memory setting for your vSphere health check analyzer tools.

1.   In the Health Analyzer UI click the Admin menu and choose “Config Settings”.
2. A dialog will come up and display the “Memory Settings” tab.
3. Change the “Collector Process Memory (MB)” value to the required allocation
4. 128 MB is default one, so bump up to 256 MB or 512 MB should be enough for large environment and if problem persists increase memory of appliance also. Before increasing this memory please make sure that the machine where you are running this VHA instance has enough memory otherwise your application will not start.

The below screen shot demonstrate the VHA Collector Process Memory (MB) field.

This should resolve most of the incidents where your VMware Health Check tool get stuck while collecting data for a larger environment. If this did not resolve your problem, then you will need to collect the log and open a support ticket. The below process demonstrate how you collect VHA logs.

1. Go to the Admin menu in the upper right and select “Log Files”
2.  At the bottom of the resulting dialog, click the “Download Log Files” button.
3. Save the resulting zip file and email it to the VHA support team.

After collecting your VHA logs make sure you open a support ticket with the VHA team using the proper channel. Please note that support is handled differently depending upon whether you are a partner or a VMware consultant. To find out the right mean to open your VHA related support ticket please check the VHA Release note included within the VHA package.

I hope this help some one complete his health check analyzer a bit faster!

VMware load based Teaming (LBT) is one of the great feature that was introduced a while back with vSphere 4.1, though it seems many admins have over looked it. As part of my job in consulting, I get to see many customers environments through out the year. It is really common to see customers who never get their hands or heads around LBT. What surprise me the most that many of these are in a desperate need for a similar feature. Many of them already have the Enterprise Plus license which include LBT in it, & its just a matter of enabling to start benefiting of it. For these reasons, I have decided to share why the Load Based Teaming feature offered within our Distributed switches is a great feature and can be of a great use to many of you out there.

The easiest way to see how beneficial LBT can be is to look at the problem it solve. Imagine you are using a virtual switch with the default setting of “Route based on the originating Virtual Port ID” or “Route based on MAC addresses” for the teaming load balancing policy. The way  “Route based on the originating Virtual Port ID” & “Route based on MAC addresses” distribute VMs network load across your uplinks by distributing the number of VMs evenly across all the vNICs involved in the team. While this cut it well in normal scenarios, there is scenarios where this can cause a problem. Let’s take a look at the image below.

Looking at the above image, you can understand that load balancing policy such “Route based on the originating Virtual Port ID ” & “Route based on MAC addresses” can end up with more heavy load on one of your vNICs rather than the other vNICs as the mapping between the VM & the vNIC is fixed and does not consider the load the NIC has it just consider how many vm each vNIC has and it make sure that they are distributed evenly across vNICs though without taking into consideration the amount of load each vm is generating. As you can tell few vms in your environment can make up the majority of network traffic load in your environment and if these all landed randomly on the same vNIC that vNIC can be over loaded and your vms will perform poorly although you have other vNICs that almost idle. VMware has introduced LBT to resolve this problem in particular. VMware Load Based Teaming (LBT) is aware of how much traffic is pushed toward each vNIC and will balance the VMs accordingly in case of contention in any vNIC which help reduce the chance of over loading a single vNIC while others are idle. As you can see LBT resolve a very valid case, & you should really consider using it if you own the Enterprise Plus edition, & maybe consider upgrading to the ENT+ license if you need such a feature. Its important to mention that Distribute Switch is required in order to have LBT, as LBT is not offered with the standard virtual switches.

Enabling VMware Load Based Teaming (LBT) is an easy task. Below are the steps you need to do:

1- In your distributed switch right click the port group that you want to enable LBT for. (LBT is enabled per port group)

2- Choose Edit settings

3- Choose Teaming and failover

4- for the load balancing setting choose “Route based on physical NIC Load”

5- hit OK accepting the change

You are now ready to enjoy your LBT!. For those of you who love to see instructions in video below is a great video by Erick Sloof that show how to enable LBT as well how it compares to the other policies in the way it perform.

Hope this get to be useful for many of you!

As many organization today are in the process for moving from VMware ESX to to ESXi, it has been important to VMware to educate its partners & customers of the process and difference.  For that VMware is offering a free web training “Free Transition to ESXi essential training” to everyone who want to learn more about ESXi. The training can be found at:   Free Transition to ESXi essential training Link . I hope you wall enjoy this free training. Free as in free Beer .

Ah below are the description of the training as officially was posted on the VMware site:

Title: Transition to ESXi Essentials

Summary

- Format: Self-Paced

- Length: 4 Hours

Overview: This self-paced training course covers the requirements and effects of transitioning your VMware vSphere™ environment to VMware® ESXi. It provides the knowledge necessary to make fundamental design decisions and successfully add ESXi to a deployed vSphere environment. This course is based on ESXi 4.1.

Special Note: For those who prefer to learn this content in a classroom environment, whether virtual or face-to-face, there are several two-day classes scheduled worldwide.  For a fee, you will receive the added value of hands-on lab exercises as well as interaction with fellow students and an instructor.
Review the upcoming schedule and register today.

Objectives At the end of the course, you should be able to:

•  Identify ESXi installation and configuration procedures
•  Leverage scripting interfaces to manage ESXi
•  Identify risks when migrating to ESXi

Who Should Attend? System administrators, systems engineers, technical support engineers, and consultants responsible for managing and supporting a vSphere installation

Prerequisites Experience with VMware vSphere and VMware ESX™.

Outline:

Module 1: Transition to ESXi Overview

• Inspect why ESXi and not ESX any more
• Questions that customers might ask – when moving from ESX to ESXi

Module 2: Compare and Contrast VMware ESXi and VMware ESX

• Identify differences between ESX and ESXi

Module 3: Install and Configure VMware ESXi

• Analyze VMware ESXi Installable and ESXi Embedded editions
• Install ESXi installable
• Identify the tasks and procedures required to migrate to ESXi

Module 4: Script Interfaces and Command Line Tools

• Discuss primary scripting interfaces
• Install vSphere Management Assistant (vMA)
• Identify the scripting interfaces and programming tools for ESXi
• Identify differences for the commands that were used with ESX and what to use with ESXi
• Recognize any custom commands implemented and how to substitute them with vCLI or power vCLI commands

Module 5: Management Tasks

• Identify hardware monitoring techniques
• Discuss system management
• Identify the VMware back up tools
• Distinguish the log files
• Setup centralized logging
• Recognize file management strategies
• Analyze lockdown mode
• Enable Tech Support Mode (TSM)

Happy transition to ESXi or Upgrade what ever you are doing, I am sure you will benefit from this course .

Going through VMworld 2010 video, I was not too surprised of getting the news that VMware ESX will be discontinued & only VMware ESXi will exist forward. Actually it seemed a news that had been circulating for a while, but finally its official now. Watch the below VMworld video from VMworld TV which clearly state that:

Ok, so what does this mean to me. I thought I will share my opinion of this decision.

Although VMware customers will have to pay some efforts to upgrade to ESXi, the upgrade process more than likely will be streamed line & not much more efforts than doing the normal upgrade between ESX releases. Though the gained benefits will be much worth it.

VMware ESXi has a much smaller foot print than ESX, which ensure it has a higher reliability and security level. Further, as ESXi get rid of the service console (a big chuck of the ESX code) it reduce the amount of patching required for ESX tremendously. In regards of the service console command line access, RCLI & Support Tech mode shall do the trick of replacing it. VMware has made a great development on both that I believe they can replace the service console command line in the future.

One more advantage that I see & not emphasized much by other is unifying the platform. It will finally kill the confusion many customers have between ESX & ESXi. Which one shall they use? when to use it? I mean one of the question I have been asked tons of time will I be able to use VMotion & HA if I install ESXi instead of ESX. I believe this move will wipe this confusion. It will even put VMware one more step a head of the competetion.

What does this shall mean for VMware Admins & consultants? If you have a new vSphere implementation make sure you set it up with ESXi from the start, as ESX will be  phasing out with the next release.  If you have ESX & you are still planning to carry out the upgrade, you might want to consider upgrading to VMware ESXi direclty.

I would like to hear other people opinions of the move to ESXi, Please leave your points & feedbacks in the comment area below.

I have seen tons of articles talking about the differences between VMware ESX & VMware ESXi, but I have not seen many that discuss when to use them. I have noticed that many of my customers get even more confused when reading comparisons between the two, as not all of them have deep understanding of the Virtualization Technology. That means they got to know the differences, but still wondering which one is best fit for their environments. Below is few rules that can put you on the right path.

- Do you need a free but reliable Virtualization platform as you don’t have a budget? or maybe you wanna use it for non critical applications? maybe you don’t want to pay for fancy features or an official support. In this case VMware ESXi is for you. As ESXi offer a free version, where a license can be applied to it at a later stage when you need to go fancier or require official support.

- Is security & reducing the required patching time is one of your main concerns? then you should consider using VMware ESXi as it does not have a service console which requires most of the patching time.

- Do you hate to waste resources used to run the service console, then ESXi for you?

Hmmm, you might think I am advertising the VMware ESXi over ESX. I am not, but as per VMware road map it seems ESXi will be the way to the future. Unfortunately though before going with VMware ESXi there are certain limitation that you will have to take into your consideration:

- VMware ESXi does not have a service console, which means if you are going to need the service console to run some odd application in there or some batching process, you would need to use VMware ESX. Though I don’t recommend running anything in the service console, as it will affect the speed you can apply upgrades & always having to ensure your apps running in the service console is compatible with the new update or VMware release. You don’t need to go into this headache, unless you really have to.

- Always check if your required backup, replication, monitoring or any other add-on software you are planning to obtain is supporting VMware ESXi. If it does not, then try to find a good alternative. If you must use a software which still does not ESXi, then you are again stuck with VMware ESX. A good note though, most software vendors today are supporting ESXi. Even software vendors who still not supporting ESXi, they will support it in the near future as they know it will be the way to go.

If you are not affected by the last two limitations of VMware ESXi, then ESXi should be the way to go. In the other hand, if you require one of these two then you must go with VMware ESX. A final note, I can see these two limitations vanishing in the very near future.

I hope this have been helpful, & enjoy the rest of the day. If I have missed something & you would love to add it. Please leave me a note in the comments area.

If you are as paranoid with redundancy as my team & I are, then entering the hostnames & IPs of all the server in a cluster into the VMware host file is a must. As your VMware HA is dependant on your servers being able to resolve the names of itself & other ESX server in the cluster, the DNS get to play a major role in HA. What if DNS fail? As many customers are using Microsoft DNS as their primary means of DNS, there is a chance it will fail a day or another. Would you like a way to ensure that your HA will work even if your DNS fail? then Adding these records to the host files of each of the ESXi server in the cluster will do the trick. What if you did not even have a DNS & still want to implement HA & other feature that depend on name resolution, then again adding the records to the host file is the solution.

The steps below show what to do after you login to the console of your VMware Server. If you are using ESX, then logging in to the service console is straight forward. If you are using ESXi, then please check out the other article in here on how to access the VMware ESXi Console: VMware ESXi – Console Access (Unsupported)

After logging into the VMware Console you will need to run the following command as shown in the image below to open the host file of you VMware:

vi /etc/hosts

Enter the host names & ip addresses in the host file as shown in the image below:

Save the file & exit from the service console. Repeat the above steps for the rest of the ESX/ESXi server in the same cluster.
I hope that help someone. If it helped you, please leave me a comment so I will know that I helped someone not just writing for myself. If you have something to add to this please comment it.

Did the title bring your attention? Have you heard that one of the main reasons for VMware ESXi was to get rid of the service console? Are you sure there is no console in VMware ESXi? The true is there is still a very tiny console in VMware ESXi, though it has way smaller foot print than the one found in the ESX version. In addition, the usage of the VMware ESXi Console Access is not supported unless if instructed by VMware to do so. Please use this tip wisely & don’t try to do wild changes using your VMware ESXi console as that is not supported. Yeah, again I would repeat its not supported .

Hmm you still insist that you want to know how to do so, although its not supported. I can admit it, it can get to be handy if you know what you are doing. Below is the steps of how to access the ESXi Console:

To access the VMware ESXi console (Not Supported) do the following.

1. Open the console of your ESXi host, you should see a screen like the below:

2. Use the key-combo Alt+F1, and you’ll get a virutal terminal (vtty1) with some log messages in it like below:

3- After you hit Alt+F1 & end up with the screen above, you will notice that what ever you type does not get on the screen & does not get interrupted.  You will have to type ‘unsupported’ and then hit enter before anything get interrupted or viewed on the screen. As soon you type ‘unsupported‘ and then hit enter you will get the following screen:

3. As you see in the snapshot above, right after you type unsupported & hit enter, VMWare ESXi remind you that you should not use this console without consultation with VMware Tech Support. It then ask you for your password. Enter the password & hit enter. You will get to the console as the image below. Ah yes, you will be amazed of how many of your old friends still work in this tiny console (Ex: esxtop and esxcfg-mpath).

4- After completing your work in the unsupported mode, you will want to clear the screen, Exit & return to the DCUI screen. (Actually Mark has asked how to do this in the comments, so I decided I will add it in the post for future readers). Thanks Mark for asking.

• a- Enter the command clear to clear the screen of any residual from previous steps. This may be required by your local security policies.
• b- Enter the command exit to exit Tech Support Mode.
• c- Press Alt+F2 to return the server to DCUI mode.

Again I have to mention, use this console at your own risk. Your usage of the above console without VMware consultation is not supported. I mainly use it to edit the local host files, which I will show at upcoming article. I hope this help.

In the past few weeks, I had been hit by many calls of many of my colleagues & customers being hit by the following error when trying to configure VMware ESXi HA:

“cmd addnode failed for secondary node: /opt/vmware/aam/bin/ft_startup failed to complete within three minutes”

After investigating the error for sometime & searching on the internet I found many misleading trial and error solutions for this error around Google results. The resolution which seems to work for everyone of my colleagues & customers is below.

1- Disable HA for the affected cluster

2- Delete the User Worlds Swap file called uwswap from all the nodes in the problematic cluster.

3- Enable HA for the affected cluster again.

Note: If its possible to move the uwswap file to the local harddisk do it by all mean, as its much better than putting it on the shared storage & will save you a lot of headache.

Recommendation: Add all the host short names & fully qualified names to the host files of all the ESXi servers in that cluster. If you have a doubt on how to do so let me know & I will make a post on how to do this .  <== Well as many requests had hit my mailbox the following link is another article I posted on here to show just that:

VMWare ESXi editing the Hostfile

I thought I will document this as I am sure many of you will hit it one day or another. I hope to save all you of you some time resolving this issue. I would appreciate if you leave me your feed back

A good amount of ESX admins ask on dialy basis how to change the VMware ESX Service Console Vlan using command line. As I decided to save time by writing the steps for these kind of questions to send it to others when asked. I will document these commands in here:

First of all check which vSwitch the Service Console is on (and the name of the Service Console) with esxcfg-vswitch -l (Note: The default service Console switch is vSwitch0 & default name is “Service Console”.)

To set a vlan id on the service console use:

esxcfg-vswitch vSwitch0 -v X -p “Service Console” (Replace X with the desired VLAN number)

To remove the vlan id completely, you will need to set it to 0 using the following command:

esxcfg-vswitch vSwitch0 -v 0 -p “Service Console”

Please note both commands above had assumed the name of your service console & switch are “Service Console” & vSwitch0 as they more probably are as these are the default values. If you have changed them, please replace these with your values.

I hope this tip will help many of you out there. If I had gave you the link to fix your problem as a friend or a customer, then I hope it save you the cost of an engineer trip to your place . Please leave your feedback & comments as these are highly appreciated .

I had always been using the VMware ESX version in my implementation. As I noticed that VMware seems to be pushing toward the move to ESXi, I had decided to try the latest version for a full scale installation. Though when I was trying to configure HA for my VMware ESXi Cluster, I had been hit with the below error:

HA agent has an error : Host in HA Cluster must have userworld swap enabled

Luckily the VMware support was quite helpfull as the usual and gave me the simple solution for the problem in less than a minute, I was back up and running with HA configured. So I decided to share the solution with you. Read below the steps for the solution:

Well, it end up that ESX Server 3i systems with swap not enabled cannot be added to HA clusters, so you have to enable the Swap in order for HA to work. Below is how you enable the Swap in ESXi.
To enable swap on your ESXi 3 host system:

1. On the VirtualCenter Server, select the ESXi 3 Server host.
2. Click the Configuration tab.
3. Click Advanced Settings.
4. Choose ScratchConfig.

5. Configure ScratchConfig.ConfiguredScratchLocation to a valid directory with sufficient space (1GB) to hold the userworld swap file. The userworld swap can be configured on local storage or shared storage.

   Note: Each swapfile needs a unique name across all ESX hosts.

   Using the following Syntax /vmfs/volumes/<DatastoreName>/<folder you want to use for the swap file>  (Ex:/vmfs/volumes/VMFS1/vmware1swap)

Important note: If you are putting your userworld swap file on the shared storage then make sure each server have a dedicated folder for its swap files else it won’t work.

Recommendation: This recommendation I have found stated by VMware nowhere, but of my experience and observation I would recommend to put the userworld swap file on the local storage of each server when possible for the following reasons:

• You don’t have to create a folder for each server on your shared storage & don’t have to keep up with folders naming
• You won’t waste a valuable disk space & I/Oson your more expensive SAN storage for swap files
• SWAP files will get better performance on the local drives
• If the server crash, you don’t need the swap file to recover. Furthermore, you would have to make sure you keep cleaning up to not to end up with a bulk of unused swap files on your shared storage when replacing servers if you decided to keep your swap file on the shared storage.

Again remember that was my own recommendation & best practice, but it will work just fine if you run it from your shared storage. Some time you got no option but to run it from the shared storage as well ex: running ESXi from flash.

6. Select the ScratchConfig.ConfiguredSwapState option.

7. Click OK.

8. Reboot the ESX Server 3i System

After you have enabled swap on the ESXi 3 host system, you can add the host to a VMware HA cluster. I hope this save all of you some trouble .