Don’t use VMware Raw Device Mapping (RDM) for performance, but ..

Till today everywhere I looked recommended using RDM over VMFS for performance (when a higher I/O is required), though today while studying for my VMware Design Exam I have got my hand on the “Performance Characterization of VMFS and RDM Using a SAN” document by VMware. The interesting executive summary of the document is copied up to the letter below:

=======

Executive Summary:

The main conclusions that can be drawn from the tests described in this study are:

  • For random reads and writes, VMFS and RDM yield a similar number of I/O operations per second.
  • For sequential reads and writes, performance of VMFS is very close to that of RDM (except on sequential reads with an I/O block size of 4K). Both RDM and VMFS yield a very high throughput in excess of 300 megabytes per second depending on the I/O block size.
  • For random reads and writes, VMFS requires 5 percent more CPU cycles per I/O operation compared to RDM.
  • For sequential reads and writes, VMFS requires about 8 percent more CPU cycles per I/O operation compared to RDM.

=====

Reading the above executive summary, I was kinda shocked at first glance. Further looking at the graphs showing the tests numbers they were close enough that I felt the difference might not be noticeable between RDM & VMFS.… Read More

VMware ESX & ESXi Connections & ports

While preparing for my VMware Design Exam, I have stumbled upon this great pdf that document VMware ESX & ESXi connections & ports in details. It even cover the connections & ports for VMware additional components like VMware Update Manager & vCenter Converter.  Although many documents do the same, what got my attention is all this are fitted into one detailed compact image. I though this document will be useful for many others planning to take the exam, or even Virtualization Experts who just want to design their network & adopt for the ports they need to open through firewall. Below is a small foot print of the image. To download the document just hit on the image below.

VMware ESX & ESXi Connections & portsRead More

ESX host disconnect from VirtualCenter or vCenter Server

VMware ESX host can disconnect from VirtualCenter & have a problem connecting to it again for many reasons. Below I will cover few ways to diagnose the problem, few common resolutions to this issue.

In these cases usually the first thing you want to do is to log on through ssh, vcli, or directly on the server console & check the following log files for errors:

Vmkernel – /var/log/vmkernel
Vmkernel Warnings – /var/log/vmkwarning
ESX Server host agent log – /var/log/vmware/hostd.log
Service Console – /var/log/messages
VirtualCenter agent – /var/log/vmware/vpx

Note: it would be great if you get to know what each of the above log files is for.  VMwareWolf has a great post explaining these log files that can be found at: http://www.vmwarewolf.com/which-esx-log-file/

Usually reading the log files above with enough knowledge you will be able to tell what is wrong, or at least get a clue of what to look for. Below are few further things to check for.

At the service console make sure the mgmt-vmware service is running by using the following command:

service mgmt-vmware status

If the mgmt-VMware service is stopped try to start it by running:

service mgmt-vmware start

Some other time, the service might be running but not doing what its suppose to do.… Read More

VMware ESX How to enable SNMP

As many VMware ESX monitoring tools are hitting the market place, and with the tend of these tools being dependent on SNMP (example: Starwind Free VM Monitor for VMware ESX), the more I am getting the question on how can I enable SNMP on my VMware ESX Server.. The instruction below will cover how to enable SNMP on VMware ESX (Note: VMware ESXi 3.5 does not support SNMP queries.).

To enable SNMP on VMware ESX Server 3.5:

1. Login to the console as root
2. Open snmpd.conf file with your favorite editor (Usually you can find snmpd.conf  in /etc/snmpd) and add “rocommunity xxxx” where xxxx is your read-only community string).

3. While you’re editing the snmpd.conf file, also add “dlmod SNMPESX /usr/lib/vmware/snmp/libSNMPESX.so”
4. Save & close snmpd.conf

5. Restart the SNMP daemon by executing:   /etc/init.d/snmpd restart

You will need to ensure that the firewall settings on the ESX server allow SNMP traffic using the below instructions:

1. Login as root and issue the following commands:
2. esxcfg-firewall -e snmpd            <== This will open the SNMP port in your ESX firewall.
3. chkconfig snmpd on                    <== This will ensure the SNMP Service will start automatically each time you start your ESX Server
4.… Read More

How to enable VMware EVC on VMware vSphere & VI3 without downtime

If you don’t know what is VMware EVC or you need to find out more about it, please look at my last post VMware EVC enhanced VMotion Compatibility enable VMware VMotion across CPU generations

Ok, if you had read that and now wondering how to get it setup without having to shutdown your virtual machines. Then this small tip is for you.

As you know & I had mentioned that in my previous post on this topic that VMware will request you to shutdown all the virtual machines you have running on the cluster you want to enable VMware EVC for. To avoid having to shutdown your virtual machines what you can do is to create a temp cluster with few temp servers on it & VMotion your VMs to these servers. Enable VMWare EVC on your evacuated production cluster. Then again VMotion the Virtual Machines back to your original production cluster.

Although this seems to work, If possible to take the downtime & follow the proper procedure then you might be better of doing just that. Good luck and hope this help.… Read More

VMware EVC (Enhanced VMotion Compatibility) enable VMware VMotion across CPU generations

Everyone who had been using VMware for a while, would appreciate VMware VMotion. Actually VMware VMotion have always been a distinguished feature of VMware ESX servers. Despite of VMotion has always been a great feature, people using it at earlier stages used to have a tough time when having to use it across different CPU generation. That meant companies had to actually put similar hardware server in the same cluster, to guarantee that VMotion work smoothly. This was not possible at all the time, and was a tough call for implementers as they had to cheat masking manually in order to avoid VMotion not accepting to run between different CPU genrations.

Although VMware EVC (Enhanced VMotion Compatibility) has been around for a while, many VMware administrators still does not know about it or just over looked it. VMware EVC is actually the defacto solution for VMotion across hosts with different CPU generations. It will automatically masks the non compatible features of different generation cpus from the hosts, which give you the ability to VMotion across hosts with different CPU generations. Please note VMware EVC has been supported since VMware ESX 3.5U2, but was highly Enhanced in VMware vSphere.

Ok, now I got your attention & you want to know how to actually enable VMware EVC & what is the pre-requisite for it.… Read More

VMWare ESX: A general system error occurred: Ufad server is not running.

It seems to me many people are facing the same ugly error when they try to convert a physical server to a virtual server using VMware Consolidation Planning tool. The error usually get after the tool has finished annalyzing the physical machine & it was deemed ‘Ready for consolidation’ with Confidence ‘high’.

When the user select ‘Plan Consolidation’, select the server, and click on ‘Consolidate’, they get the famous error message, “A general system error occurred: Ufad server is not running.”

The image below show the error:

vmware esx consolidation Tool ufad error msg

As a lot of people seems to stumble by this every day, I have decided to discuss the way I usually resolve this problem.

First step: I usually check if all the VMware Services specially the VMware Convertor & Consolidation tools has been started normally. If any is down I will start it & then check again.

Second Step: If all the VMware Services are started & seems operating normally I would restart the VMware Convertor & Consolidation tools & then check again if the consolidation process work.

Please note usually step one & two are efficient to fix the problem, though in many times I had noticed although the services are started when you restart them they don’t start again.… Read More

Changing the VLAN ID of a ESX Service Console using Command Line

A good amount of ESX admins ask on dialy basis how to change the VMware ESX Service Console Vlan using command line. As I decided to save time by writing the steps for these kind of questions to send it to others when asked. I will document these commands in here:

First of all check which vSwitch the Service Console is on (and the name of the Service Console) with esxcfg-vswitch -l (Note: The default service Console switch is vSwitch0 & default name is “Service Console”.)

To set a vlan id on the service console use:

esxcfg-vswitch vSwitch0 -v X -p “Service Console” (Replace X with the desired VLAN number)

To remove the vlan id completely, you will need to set it to 0 using the following command:

esxcfg-vswitch vSwitch0 -v 0 -p “Service Console”

Please note both commands above had assumed the name of your service console & switch are “Service Console” & vSwitch0 as they more probably are as these are the default values. If you have changed them, please replace these with your values.

I hope this tip will help many of you out there. If I had gave you the link to fix your problem as a friend or a customer, then I hope it save you the cost of an engineer trip to your place :).… Read More

VMware ESX & USB & Parralel dongles

When ever I attend a VMware presentation, one of the most common question “How would we handle our USB or parralel licenses dongles in VMware ESX.” Are they supported? Can I get them to work some how? What make the question more critical that most presenters are from a sale or pre-sale background and does not have the clear answer. The real answer to the question, yes its workable but you might need some third party utility to get the required setup to run. Some good resources to get you started are below:

VMware ESX Server Using AnywhereUSB to Connect USB Devices

VMware esx21admin_modvm_parallel_running

USB Device Support within VMware ESX 3.x and ESXi

VMware ESXi – Serial & Parallel Port (use physical serial port on the host)

I hope this information will be a helpful leads for many people to come and face problems with this matter :).

Read More

VMware ESX & Virtual Center 3.5 upgrade guide

It might be the time for you to upgrade your VMware ESX servers to VMware 3.5U3 and Virtual Center 3.5U4.

You might even just want to upgrade to any version of VMware ESX & vCenter 3.5. Then this guide linked to below is all you need. It has a great step by step instruction of the process & better its provided by the solution Vendor VMWare :).

VMware ESX & Virtual Center Upgrade guideRead More

VMWare ESX: How to recover your VMFS partition table

You might wake up for a bad day after a power outage or a storage failure. You thought it was over, after you had succeeded to bring all your machines back up and running. In few minutes you found out that some of your virtual machines are missing. After a small investigation you found out that your VMware Datastores (VMFS) are empty when you try to browse them from VMware vCenter or VI client. If you face this problem there is a big chance that your VMFS Partition table for these LUNs or disks are missing.

To Check Your VMware ESX server VMFS Partition table follow the following procedure:

1- Connect to the VMware ESX server where is the missing datastore (VMFS) was connected using SSH. Make sure you have a root access.

2- Run the following command to find out your SAN devices: esxcfg-vmhbadevs

The output will look something like below:
vmhba0:0:0 /dev/cciss/c0d0
vmhba1:0:1 /dev/sda
vmhba1:0:2 /dev/sdb
vmhba1:4:2 /dev/sdc

3- If you know the SAN device that is holding the missing datastore (VMFS) then run the following command on that device to check its partition table, else run it on all the devices and check them one by one. (Hint: The command to show the partition table for all the devices is ‘fdisk -lu’)

fdisk -lu /dev/sda    <== run this if you know that sda is the device holding the missing datastore (VMFS)

Output should look something like below for a LUN with the VMFS Partition table is missing:

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk /dev/sda doesn’t contain a valid partition table

or it could look something like below on some versions

Disk /dev/sde: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id

System Where a normal working entry with a valid partition table will look something like below:
Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 128 31455269 15727571 fb Unknown <== Its your partition table entry Notice

in the last example the line of the partition table:

‘/dev/sdb1 128 31455269 15727571 fb Unknown’ <== This means the partition table exist.… Read More

Top 10 VMware VMworld 2009 Videos

Finally back home :). Don’t get me wrong Cannes was great, though nothing like being back home.

OK for people who did not get to join us in Cannes, I had for them a free ticket for the replay :).

I wish I could afford to give our visitors a VMworld pass for free, but I have not been making that much money yet. Here I will make up for the tickets, by sharing with you the links for the top 10 VMware VMworld 2009 Vids including the CEO Keynotes & the closing :). So you will feel like you were there with us. Below are the long waited links :).

VMware VMworld 2009 General Keynotes Videos

VMware VMworld 2009 Keynotes & General Labs Vids

VMware VMworld Europe 2009 Community Videos

I hope that give you a nice close look at what happened at VMworld as a video is much better than a 1,000,000 word. LOL, yeah I know the original version, but this is my new one.… Read More

VMworld 2009 announce vSphere, vCenter Server Heartbeat, VMware vShield Zones

VMworld Europe 2009. VMware has used the VMworld Europe 2009 to announce two new products that will be available this year:

1- vCenter Server Heartbeat:

Comes to solve the problem arising from a possible collapse of the server that manages the VMware virtualized platforum, specifically vCenter Server earlier known by Virtual Center. The vCenter Server Heartbeat comes to provide High availability for the vCenter. Prior to vCenter Server Heartbeat you were only able to have one vCenter to manage your infrastructure if that vCenter fail VMware’s Live migration with VMotion, load balancing with Distributed Resource Scheduler (DRS), license management, and performance monitoring all cease to function.

Architecturally, Server Heartbeat is implemented as an active/passive vCenter servers, running on physical or virtual machines. In addition to server and network hardware, Server Heartbeat monitors the actual vCenter instance, its back-end database, and the underlying operating system. In the case of failure, the passive node takes over, and Heartbeat restarts the vCenter service. Failover can occur over both LANs and WANs.

I can sense some people already arguing that you could achieved the same result of vCenter heartbeat Server by running vCenter inside a virtual machine & VMware HA. If you had implemented that then you will notice three big limitations with that:

a- As HA is only supported in a single site then that redundancy for vCenter was only valid for a single site & you were not able to use it for DR.… Read More

VMware ESX – VMware VMotion

What’s VMware VMotion?

VMware VMotion enables the active migration of running virtual machines from one physical server to another without interruption of service, availability of service and complete integrity of transactions.

VMware VMotion moves running virtual machines from one physical server to another while maintaining service availability.

VMware VMotion allows IT organizations to:

  • Perform live migrations with zero downtime, undetectable to the user.
  • Continuously and automatically optimize virtual machines within resource pools.
  • Perform hardware maintenance without scheduling downtime and disrupting business operations.
  • Proactively move virtual machines away from failing or underperforming servers.


VMotion is a key technology which makes the creation of dynamic data center
, automation and optimization possible.

VMware VMotion Diagram

When to use VMware VMotion?

  • To Perform hardware maintenance without having to schedule an interruption of service (Downtime)
  • Migrate virtual machines from servers with flaws or poor performance proactively .
  • Any time you have a schedule downtime for the host (Ex: updating the VMware ESX)

How does VMotion work?Read More

VMware ESX Healtcheck script

Before asking VMware or your vendor for a health check on your VMware enviornemnt you can use the free Healthcheck script as an early start. Actually its a great idea to automate it to send you and update of your enviornment status weekly. Below is a bit more of the VMware Healthcheck script as posted by Ibeerens on the VMware Community forum:

What is VMware HealthCheck Script?

VMware Healthcheck is a Powershell script that reports information like snapshots, VMware tools version, datastore space, CDROM and/or floppy drives connected, VM details etc. to HTML and e-mail the output. This script can be scheduled so that for example you get every week a rapport in your mailbox with the status of the VMware environment.

Reason for creating this script:

As VMware Consultant I see a lot of common problems in VMware environments like:

  • Snapshots are enabled and forgot the commit to the VM.
  • Datastores are almost full (for example if snapshots are enabled)
  • VMware tools versions are different
  • CDROM and floppy drives are still mounted to the VM
  • Virtual Machines have CPU and Memory limits or reservations (VMs are swapping)
  • In the VM, the VMware Tools timesync option is not enabled

In the Virtual Infrastructure Client (VIC) it is difficult to see this sort of information.… Read More

Sponsors