Notes

How to make DRACT run in a 64-bit system

2011-11-02T13:52:00.000-04:00

Dell™ Remote Access Configuration Tool (DRACT) is a tool that provides a central console to discover and configure Dell Remote Access Controller (DRAC) for all the servers in your network. It automates some of the common repetitive tasks, such as firmware upgrade, and AD authentication configuration.

As of Nov 2011, the latest version of DRACT is 1.0, which can be run in 32-bit Windows systems only. Nowadays, 64-bit systems are commonly used everywhere. Setting up a system just DRACT is kind of wasting resource. After some research, I found out that we CAN make it run in 64-bit environment.

You can have DRACT installed into a 64-bit system without problems. The symptom showing it is not working is that it can not discover any DRAC in your network. Here is what you need to do to make it work.
1) You need a file: corflags.exe in Microsoft SDK. You can install Microsoft SDK, or you can just copy corflags.exe from a system that has Microsoft SDK installed. Corflags.exe is a Conversion tool allows you to configure the CorFlags section of the header of a portable executable image.
2) Run the following command of the ract.exe executable file:

corflags.exe /32bit+ “D:\Program Files (x86)\Dell\RACT\RACT.exe”

The /32bit+ sets the 32-bit flag, so that the executable file always runs under WOW64 in a 64-bit system. (WOW64 is a compatibility environment that enables a 32-bit application to run on a 64-bit system. WOW64 is included in the system.)

That is it. Now DRACT will be able to discover the DRAC with the IPs that you provide in your network.

How to Cancel a Stuck VMware Tools Install

2010-11-17T13:48:00.000-05:00

Some co-worker ran into a problem with not able to VMotion a VM off a vSphere host. The error is “The virtual machine is installing VMware Tools and cannot initiate a migration operation.” Right click to “End VMware tools install” won’t help. After some digging, we found it can be resolved in two ways.

1) Click on the VM, go to the toolbar on the top in vSphere client, click on the CD/DVD icon -> CD/DVD Drive 1-> disconnect. This should resolve the issue and you should be able to VMotion this VM after this. This mostly happens to a Linux VM after it upgrades its VMware tools.

2) Found this in Bob Plankers’ The Lone Sysadmin blog. It works great too.
First, you need the ID of the VM (all on one line if it wraps):
/usr/bin/vmware-cmd /vmfs/volumes/datastore-name/vm-folder/vmx-file.vmx getid
Then you can do a:
/usr/bin/vmware-vim-cmd vmsvc/tools.cancelinstall idnumber

P2V fails at 94% after finished cloning disks

2010-09-08T16:45:00.000-04:00

Issue:
Physical server rebooted itself when VMware Converter reached 94% or 98%. By the time of its crash, it had already finished cloning its disks, and failed while it was trying to connect to the VM or reconfigure the VM. And the VM always goes to BSOD when it boots up.

Cause:
The cause the reboot of the physical server is unknown. It involves the physical hardware, the OS, and the application, and none of them can pinpoint where the failure comes from.
The cause of the BSOD in the VM is the virtual disk controller driver isn’t in place and the VM isn’t configured to be ready to run in a virtual environment.

Solution:
In the VMware Converter, run “Configure Machine” and use the wizard to configure the VM. After it is finished, we should be able to power on the VM into Windows.

VMware Update Manager 4.0

2009-11-19T12:33:00.001-05:00

Can't download patchs via VMware Update Manager 4.0

My environment is known for having complex firewalls and proxy. Lots of time it takes lots of effort and time to make application works. This time is no exception for VMware Update Manager 4.0.

The installation process for VUM 4 was very straight forward. Then it comes to the hard part. It can't download any patches for ESX hosts nor for VMs. After lots of testing, and some help from VMware's communities forum, finally I made it work. Here is what I have done.

1) Have proxy server opened up HTTP/HTTPS to the following 3 domains: *.vmware.com, *.shavlik.com, and *.microsoft.com. The last domain is not for downloading patches, but for Test Connection since that is one of the websites that it use to test connectivity.

2) Set up proxy setting. Do not put http in front of your proxy server name in the box. Make sure you have the right authentication information in it if it is required.

If you are seeing some errors like this, it could be caused by your proxy setting.

httpDownload, 730 Error 12007 from WinHttpSendRequest for url https://hostupdate.vmware.com/software/VUM/PRODUCTION/index.xml

'httpDownload' 4284 ERROR [httpDownload, 726] Error 12029 from WinHttpSendRequest for url https://hostupdate.vmware.com/software/VUM/PRODUCTION/index.xml

Please check out the following KB for more information.
http://kb.vmware.com/kb/1012926

How to replace SSL cert in VirtualCenter

2009-06-05T15:10:00.005-04:00

When replacing a SSL cert, there might be chance to cause encryption problems between ESX and VC. Doing it improperly will lose permissions setting, and it will take a long time for it to get stabilized.

To generate a new SSL cert, we can follow the instruction here at KB 1009092, or in Leo's blog. The way they listed to install a new SSL cert could be working for someone, but it didn't work on many other folks like this one, and this one. Here I list some basic steps that needed to perform in order to have a smooth change.

1) Disable HA

2) Disconnect ESX from VC

3) Stop VC service

4) Replace SSL cert

5) Start VC service

6) Reconnect ESX to VC with root/pw

If it still have issues, like having an error of vim.faultlogin, we can remove VC agent, and delete vpxuser from SC. Reconnecting ESX to VC will recreate the user and reinstall the VC agent.

Here are a couple KBs and articles might be helpful with changing SSL cert:

Enabling Server-Certificate Verification for Virtual Infrastructure Clients

Configuring Custom SSL Certificates in VirtualCenter 2.5

ESX WRITE10 error

2009-04-29T12:44:00.004-04:00

Recently a WRITE10 error in one of my ESX host caught my attention, and it occurs more than 10 times every second.

Apr 29 12:01:10 cla1011 vmkernel: 11:22:45:34.946 cpu4:1077)WARNING: VSCSI: 5291: WRITE10 past end of virtual device with 29365, length 128

After search on Google and VMware's communities, and still could not find detail information and solution about it, I turned to VMware technical support. The technical support engineer sent me their internal KB.

Symptoms

Repeatedly logging messages similar to either of the following in /var/log/vmkernel (or /var/log/messages on ESXi):

Feb 5 15:44:46 USPLVS02 vmkernel: 63:05:31:58.181 cpu3:1129)WARNING: VSCSI: 5292: WRITE10 past end of virtual device with 33554432 numBlocks, offset 33554351, length 128

Feb 12 17:03:04 pa-tse-h02 vmkernel: 156:05:50:47.889 cpu0:1174)WARNING: VSCSI: 3430: READ10 past end of virtual device with 20971520 numBlocks, offset 20980737, length 16

These messages indicate that I/O is being attempted that is outside the boundaries of the virtual device (virtual disk). In layman's terms, the VM has a list of ten items, and the guest OS is asking for the 12th item on the list.

Resolution

To find out which VM is responsible for these, the World ID (WID) must be determined from the log messages. The WID is after the cpu specifier, and before the WARNING in the above messages. In the case of the WRITE10 message above, the WID is 1129; for the READ10 message the WID is 1174.

If you look in /proc/vmware/sched/cpu, then the vcpu column (first one) will list the number identified in the logs.

To determine the VM responsible if it is not running:

cat `ls -rt vmkern*` | less

Find the first instance of the log message (using "/WRITE10" or "/READ10" will likely find it for you very well) Then search backwards in the logs for the WID value (in less this can be done with "?". ex: ?1129 Note: It searches beginning just before the top line on screen. Press 'n' to find the next match.) and keep searching earlier in logs until you find something similar to:

Feb 12 16:51:55 pa-tse-h02 vmkernel: 156:05:39:38.873 cpu2:1173)Sched: vm 1174: 4836: adding 'vmm0:ProblemVMName': group 'host/user/pool0': cpu: shares=2911 min=0 max=-1

The text will show you the name of the problematic VM after the vmm entry. In this case, "adding 'vmm0:ProblemVMName'" shows that the VM causing the issue is named ProblemVMName.

If you look at the contents of the descriptor file for the offending Virtual Machine's disks, you will find an entry listing the number of cylinders for the virtual disk. As an example:

ddb.geometry.cylinders = "2088"

In this case, the virtual disk has 2088 cylinders. Running "fdisk -l" against the flat file of the virtual disk will return information similar to:

You must set cylinders.

You can do this from the extra functions menu.

Disk ANSGOOD-flat.vmdk: 0 MB, 0 bytes

255 heads, 63 sectors/track, 0 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

ANSGOOD-flat.vmdk1 * 1 2089 16779861 7 HPFS/NTFS

Partition 1 has different physical/logical endings:

phys=(1023, 254, 63) logical=(2088, 254, 63)

Note that in this case, the end value for the partition/disk set in the partition table is 2089, exceeding the number of cylinders set in the descriptor file. If this were proper, it would show 2088 as the end value, instead of 2089. The operating system, as a result of this incorrect partition table

Extending the VMDK just enough to allow it to contain the size of the partition table might fix this, but because it is an invalid conglomeration of settings, it can not be safely assumed that this can be fixed. Other possible means of fixing the behavior is to cause the partition table to fit within the disk, by correcting it's ending value. Moving the data to a new, properly configured disk/partition table/file system is the best bet as the state of the file system by trying to modify the VMDK or partition table isn't known, and may be damaged by the changes, or already is damaged. Give the customer the options, and let them choose how to handle the changes to their system, as they can best judge how they want to protect their data.

The solution to it is either extend the VMDK file or shrink the partition table. Extending seems safer than shrink. The solution that I chose was using VMware Converter. By the way, VMware Converter 4 is offering some cool features over the previous version 3.0.3.

How to upgrade HBA firmware

2009-04-17T12:14:00.005-04:00

In my case, I was to upgrade firmware on Emulex HBA for my ESX 3.5 Update 1 host. Here is my hardware:

Server: Dell PowerEdge 6950

HBA: Emulex LPe1150-E

Download HBAnyware Library and Utilities Kit at Emulex Download section for Dell.

Download Emulex LPe1150-E latest firmware (as of this writing, it is 2.80A4)

Upload both the HBAnywhere and firmware to the ESX host

Verify the firmware version before the upgrade

cat /proc/scsi/lpfc/*

Install HBAnywhere by running

rpm -ivh elxvmwarecorekit-2.1a40-1.i386.rpm

The binary executable files will be store at /usr/sbin/hbanyware/

List the HBA that are manageable by HBAnywhere

/usr/sbin/hbanyware/hbacmd ListHBAs

Upgrade firmware in HBA

./hbacmd Download 10:00:00:00:c9:63:f4:19 /location_of_firmware(etc, /root/wf280a4.all)

Now we can verify the upgraded firmware version

cat /proc/scsi/lpfc/*

VMware VIMA

2009-04-08T14:03:00.002-04:00

While I was searching for a syslog appliance, ran across Duncan Epping's blog about VMotioning Service Console. Following his information and went over to VMware's VIMA webpage, downloaded it and gave it a try. I got to say it is like a bomb. Got to love it. It give ESX admins another central point of management of ESX, besides vCenter, via command lines. William Lam's VMware ESX and ESXi Scripts & Resources even gives some great examples on how to make use of this great appliance. With RCLI & VI Perl Toolkit integrated into the applicance, ESX admin now can make use of it from one central place. VIMA still got lots of places needed to improve, but it is really a greate starting point.

ESX 3.5 Update 4

2009-03-31T14:04:00.003-04:00

Just surprisingly found out VMware just released ESX 3.5 Update 4 yesterday on 30 Mar 2009. A couple months back, a consultant told me that VMware ESX 3.5 Update 3 was the last VI3 updates, and the next release is going to be ESX 4, or vSphere. Well, we will have to wait for a while before ESX 4 come out. It is not like I can't wait for ESX 4 coming out (well, maybe I do), I just wish VMware does a better job in QA. There is no need to have a buggy major release. I think I can wait.

After glancing at the release note, I found a couple things that I don't want to use ESX 3.5U4 in my production environment. Using these will japordize my currently stable environment. Normally when VMware has new update release for ESX, the first thing I would check out is What's New, which will contain the major change. The second thing is Known Issues, which will determine how soon I will be putting it into our production environment.

In this realse ESX 3.5U4, What's New mainly brings out more hardware support, and Network drivers, HBA drivers updates, along with a couple newer version of OS support. What really stop me from putting ESX 3.5U4 into production is the Known Issues.

And here are the major Known Issues that I don't like.

x64-based versions of Windows Vista and Windows Server 2008 guest operating systems require Microsoft hotfix x64-based versions of Windows Vista and Windows Server 2008 guest operating systems without Microsoft hotfix http://support.microsoft.com/kb/950772 might encounter a situation where the guest operating system stops responding and returns the following error:

MONITOR PANIC: vcpu-3:ASSERT vmcore/vmm/cpu/segment.c:430

Linux guest operating systems lose network connectivity after automatic tools upgrade

If the version of VMware tools in a Linux guest operating system becomes out of date and an automatic tools upgrade is performed, the guest operating system loses network connectivity. After an automatic tools upgrade, the guest operating system stops the network service and does not restart the service automatically after tools upgrade.

Workaround: Restart the network service in the guest operating system manually or reboot the guest operating system after an automatic tools upgrade.

The cursor disappears after VMware Tools has been configured

This issue has been observed on ESX Server 3.5 Update 4 with SUSE Linux Enterprise Server 8 (with or without SP4) guests. Immediately after you configure VMware Tools, the mouse cursor is not visible.

Workaround: Reboot the virtual machine.

Virtual Machines Might Not Power on After a Failover When the Host Is Isolated

Virtual machines might not start after a failover when a host is isolated and the isolation response is set to Guest Shutdown, which is the default configuration for a cluster. This might occur on clusters with fewer than five nodes and can happen on virtual machines
that take more time to complete the guest shutdown.

Workaround: Set the Isolation Response to either Leave powered on or Power off for clusters which have fewer than five nodes. To set the Isolation Response for a virtual machine, select the cluster, click the Edit Settings link, and select Virtual Machine Options under VMware HA. From the Isolation Response pop-up menu, select either Leave powered on or Power off options for the specific virtual machine.

There might be more dislikes in the Known Issues sections, since here is just what I have at the first glance. Another thing that make me think twice before an upgrade is the VMware tools upgrade. It is painful for Windows and Linux. Well, back to reality. All of these known issues wont' prevent me from installing it in my lab.

Could not power on VM II

2009-03-30T14:22:00.007-04:00

Here is a revisit the problem with not being able to power on a VM. In my previous post, Could not power on VM, it use a Linux command to kill the process so that the locked swap file can be release. Today I found an even easier way to fix the problem.

In my situation, I have an ESX server has some physical problem, which I haven't figured out what caused it yet. Basically two hostd and one vpxa processes are taking up 100% usage of CPU and 95% of 800MB memory and 50% of 1.6GB swap in service console. Contacted VMware technical support, and after he tried a couple things that I did to kill the hostd processes and vpxa without success, he gave me a solution, which is to reboot the ESX server. That is right, the solution is to reboot the ESX. In VC, the ESX has a "not responding" status. I have to export a list of the VM running in this server, record what datastore these VMs reside in. After that, I disconnected the ESX, and removed it from VC. Now I need to use RDP or SSH to connect to VM, and to shutdown the VM so that I can register the VMs in another ESX and power it on. Most of the VMs are working fine. But there always one or two VMs doesn't behave the way they are supposed to. One VM gave me an error of "Could not power on VM: No swap file. Failed to power on VM.". When I tried to remove the file from service console, it gave me an error of "Device or resource busy". The service console in this ESX isn't 100% fully functioning. "ps -ef | grep VM_name" doesn't returned anything. It means, killing the process won't work in this case. The solution I found is to rename the VM's vswp file.

Error: Could not power on VM: No swap file. Failed to power on VM.
Solution: Rename to vm_name-xxxxxx.vswp file name to something else, and VM can be powered on.

Here are two good articles about how to kill a VM: Gabe's virtual world and Technoblog. The reason I mentioned above is basically the last resort.

NetApp ESX Host Utilities Kit 6.0

2009-03-13T13:45:00.004-04:00

NetApp is going to have an ESX Host Utilities (6.0 replease) vCenter Plugin, with release date dependent upon VMware official release of vSphere. It was NetApp's showcase in VMworld Europe 2009.

This Host Utilities needs only a single installation to vCenter as a plugin instance, eliminates the hassle of the needs to install it into every ESX host.

Check out the video here

Improving ESX performance by unloading USB

2009-03-11T15:13:00.002-04:00

Many systems have shared interrupts between the USB controller and other components. In order to reduce any performance impact due to the shared interrupts, we can configure the ESX to avoid loading the USB drivers.

Configure the USB controller setting in BIOS to enable USB, so that the system supports USB devices both during and after ESX boot process even if USB drivers are not loaded by the ESX software.

Remove the following USB modules aliases from /etc/modules.conf

Save /etc/modules.conf and restart the system

This VMware KB 1290 has more detail information
on how you can tell if your system has shared interrupts.

Disabling ESX WebAccess

2009-01-27T12:51:00.004-05:00

Haven't thought about it, but it is interesting to know how to do that. Here is a discussion about it in VMware communities. Since vmware-hostd is using port 80 and 443, blocking firewall ports don't seem to be a right way to do it. Replacing the index page looks like a right way to go.

Another ESX bug

2009-01-21T13:14:00.004-05:00

I was planning to upgrade my VirtualCenter from version 2.5 Update 1 to version 2.5 Update 3, and then I received an email from VMware. This email states there is an issue with ESX 3.5 U3 could adversely effects my environment with I/O failure on SAN LUNs. I guess ESX 3.5 U3 can’t go in my environment, and I won’t be getting any benefits from VC 2.5 U3. I told my boss to hold off on this upgrade to VC 2.5 U3.

Here is the detail of this email:

ISSUE DETAILS:

VMware ESX and ESXi 3.5 U3 I/O failure on SAN LUN(s) and LUN queue is blocked indefinitely. This occurs when VMFS3 metadata updates are being done at the same time failover to an alternate path occurs for the LUN on which the VMFS3 volume resides. The effected releases are ESX 3.5 Update 3 and ESXi 3.5 U3 Embedded and Installable with both Active/Active or Active/Passive SAN arrays (Fibre Channel and iSCSI).

PROBLEM STATEMENT AND SYMPTONS:

ESX or ESXi Host may get disconnected from Virtual Center

All paths to the LUNs are in standby state

Esxcfg-rescan might take a long tome to complete or never complete (hung)

VMKernel logs show entries similar to the following:

Queue for device vml.02001600006006016086741d00c6a0bc934902dd115241 49442035 has been blocked for 6399 seconds.

Please refer to KB 1008130.

SOLUTION:

A reboot is required to clear this condition.

VMware is working on a patch to address this issue. The knowledge base article for this issue will be updated after the patch is available.

NEXT STEPS:

If you encounter this condition, please collect the following information and open an SR with VMware Support:

1. Collect a vsi dump before reboot using /usr/lib/vmware/bin/vsi_traverse.

2. Reboot the server and collect the vm-support dump.

3. Note the activities around the time where a first “blocked for xxxx seconds” message is shown in
the VMkernel.

How to check VMware tools version in a Linux VM not running X win

2008-12-11T13:11:00.002-05:00

Since in a Linux VM not running X windows, we can't use vmware-toolbox or .vmx to check the version of the VMware tools. We can check the version of VMware tools listed in vmware-config-tools.pl, although it could be the case that VMware tools got installed, but haven't been configured yet. Run the following command to grab the version of VMware tools have been installed.

#grep buildNr /usr/bin/vmware-config-tools.pl
my $buildNr;
$buildNr = '3.5.0 build-82663'
return remove_whitespaces($buildNr);

3.5.0 build-82663 is the VMware tools version number.

ESX intermittently disconnects from VirtualCenter

2008-11-18T09:09:00.004-05:00

Error:

Sometimes an ESX host could intermittently disconnects from VirtualCenter. There are many things could cause it, but here are two common, but haven't paying much attention places that we need to look for.

Solution:

1) Use "df -h" Check to see if / directory in ESX Service Console is filling up. After / directory is clean up and wait for a while before reconnecting ESX to VirtualCenter.

2) When seeing a message like below in /var/log/vmware/hostd.log, it is time to upgrade the memory in service console, and change its default soft and hard limit. If the memory checker reach its hard limit, bad things could happen to ESX.

/var/log/vmware/hostd.log

"[2008-11-14 22:05:13.319 'Memory checker' 20937648 warning] Current value 138020 exceeds soft limit 122880."

"Unable to fork: Cannot allocate memory"

SC memory: With VirtualCenter 2.5 and above, it is easy to upgrade the ESX service console memory to 800MB, but it requires a reboot. See KB1003313 and KB1003501 in VMware's website.

Soft/Hard limit: Edit the config.xml under /etc/vmware/hostd and add the following into the <config> section:

Do a restart on hostd process by:

service mgmt-vmware restart

Creating a floppy image

2008-11-12T14:52:00.002-05:00

Occasionally need to create floppy image, and an easy way to do it is to use Linux to create it. And you can mount it right there to add any files to it to.

The following command is to create a blank floppy, filled with zero.

/bin/dd if=/dev/zero of=/root/BlankFloppy.flp bs=512 count=2880

And this blank floppy can be formatted in Linux too.

/sbin/mkfs.vfat -v -c /root/BlankFloppy.flp

Mount the floppy image so that you can put files into it.

mount -o loop /root/BlankFloppy.flp /media/floppy

In Windows, you can use some Winimage, Floppyimage, undisker to create image of floppy.

Replacing O2 sensor in 2005 Honda CR-V LX

2008-07-15T23:26:00.000-04:00

After the Just Brakes scam, I decided to DIY on my car if it has problems. It didn’t take long before I had a chance.

My 2005 Honda CR-V LX 2WD had an engine light came on one day. I went over to the dealership close to my house the next morning. I was told that I would be charged $99 for diagnostic to find out why the light came on, and I would have to come back again since that day’s schedule was full. I was glad that he told me to come back; otherwise I would have to come up with my own reason to give him that I need to come back later. I don’t like they charge their customers like this. I remembered my co-worker told me that AutoZone offers free diagnostic on cars. After work I bought my car over to AutoZone, and the clerk gave me a printout of code P0135, which said it was O2 sensor’s error. I came back home and get online for some research. It took me a while. I spent quite a bit to time to see if my car is still under emission warranties. Finally, I downloaded my car’s Emission Warranty from Honda Owner Link website, and it confirmed with the information I got from two dealers’ service departments and America Honda – my car’s O2 sensor (A/F sensor) has 3 year/36K miles warranty only.

In the beginning I ordered the O2 sensor for $100 from The Parts Authority in Amazon. But it just wasted my time since the one they shipped me wasn’t a right one. Calling them back and found out that they don’t know which one is for my car. Finally they told me that they don’t have it. I ended up with paying both ways shipping for nothing. Finally I placed my order with Majestic Honda at www.hondaautomotiveparts.com. They have diagrams in their website for each part they are selling. It took about a week to get to me. I went to AutoZone to rent the O2 sensor socket, anti-seize and prepared to have a hard time to take off the original O2 sensor. I ended up taking less than one minute to take it off. Putting the new one on was very easy too. The engine light kept on in that day, but it went off in the next early morning. I felt pretty good about it. This is my first time doing maintenance on my car, and I am saving about $190 replacing it myself. Honda dealer was going to charge me $400 if I let them do it, plus $99 diagnostic.

Free tool secures VMware ESX

2008-06-23T00:18:00.001-04:00

ConfigCheck is a free utility that can compares VMware ESX configurations against best practices guidelines development by VMware Inc. The tool was jointly released Wednesday by VMware and configuration management software maker Tripwire Inc. ConsigCheck only works on ESX 3.5.

http://www.tripwire.com/configcheck/configcheckdownload.cfm

Just Brakes Scam

2008-06-23T00:17:00.000-04:00

Just Brakes is advertising $99 for 4 brake pads brake job to show their cares for their customers. What a scam it is. The one I went to completely lied to me and ripped me off.

After I drove into Just Brakes, at 11235 Alpharetta Rd, #100, Roswell, GA 30076, Dean was the technician did the quote to me. After he had someone take everything off, including tires, calipers, brake pads. He said my disk pads need to be replaced, which was $99.98. Since my 2005 Honda CR-V’s rear brake pads has clips on it, which are different from 2004’s Honda CR-V. They don’t have rear brake pads in stock, and I need to pay $75 so that they can have the rear brake pads. And he said disk hardware also needs to be replaced. When he brought me into the shop to show me what it was, he made it very vague. He pointed at the calipers and said the things on it bent, and needed to be replaced. Sorry for my naïve, but I didn’t know what caliper was, and what disk hardware was. I felt pretty confused. Disk hardware cost about $85. And bleeding is another $39.99. The total came to $304.83, which includes a $10 discount coupon. At that time, I was very confused, and they had taken everything apart. I wondered what would happened if I refused the deal. And I have taken half a day off already. What the heck. I agreed on the quote written on the back of a piece of paper. The whole thing was pretty thick though. They finished it in less than an hour. However, no one delivered any parts during that hour, and Dean was with me all the time. When a guy brought in my car key, Dean tried to explain that the delivery guy must come in from the back.

After I came back, and did my research online and found out this is a scam that Just Brakes has been doing to lots of its customers. The so-called Disk Hardware is just a clip on a caliper, costs about $10 online. I know part of this scam was my fault, for being too naïve on cars. I didn’t grow up in the US, and cars aren’t any hobbies to me. I guess it is about time for me to learn about cars. Secondly, I trusted Just Brakes too much. Car salesmen are infamous, now I guess cars shops are too. Certain business industries aren’t honors during their business practice.

2008-05-26T00:17:00.001-04:00

ESX3: ramcheck

I have been using Memtest86+ for all the ESX servers I deployed. It has been all good, but it requires booting from Memtest86+. During the memory check, ESX server isn’t running. Ran across a tool posted in xtravirt.com, which is actually an ESX built-in utility to do memory check. The command is ramcheck. The good thing about this is it is a built-in utility, and it runs in the background of ESX, which means no down time will be involved. Just run

service ramcheck start

It runs in a background and writes out a log file to /var/log/vmware/ramcheck.log and ramcheck-err.log. Since it not only checks the memory in Service Console, but also the memory in VMkernel, it has to run as a world in VMkernel space. It consumes only a few percent of CPU resource in the background, but the trade of is the time will take a couple weeks to complete. Run esxtop will show it as ramcheck.<id>. It will stop automatically once it finished all the memory check.

It might not be a complete replace of Memtest86+, but it does offer an alternative, non intrusive maintenance memory check during operation of an ESX server.

vmkping error: syscall version mismatch (compiled: 0x7616c4e3, kernel: 0x50f63116)

2008-05-07T14:45:00.000-04:00

Today after I configured an ESX 3.0.2 Update 1 server, I wanted to use vmkping to test vmkernel IP stack. However, it returned an error stating "syscall version mismatch (compiled: 0x7616c4e3, kernel: 0x50f63116)". Odd, since it doesn't look like an network issue, which was what I was always getting. After a search online, found a VMware KB 1002880 posted on Feb 2008 at http://kb.vmware.com/selfservice/dynamickc.do?externalId=1002880&sliceId=2&command=show&forward=nonthreadedKC&kcId=1002880. The KB states that only commands vmkping and ramcheck are affected, and all other system utilities, ESX Server stability and reliability are not jeopardized.Ok, I was planning to patch it anyway.

How to assigned Customization role

2008-05-01T14:52:00.002-04:00

After spending a whole day testing and researching, finally found a solution to this "tiny" problem - assigning rights to use Customization Specifications in VirtualCenter 2.0.2.
1) Create a role for customization.
Customization User:
Virtual Machine ->Provisioning ->Read Customization Specifications
2) At the root of the VirtualCenter tree, assign the AD group to this new role - Customization User, uncheck Propagate.

How to check BIOS version in Linux without rebooting the server?

2008-04-23T21:52:00.002-04:00

Occassionally when we are doing troubleshooting, we are asked for the BIOS version. Here is a tool can help to find out the version of BIOS without rebooting the server - dmidecode, DMI table decoder. It is a tool for dumping DMI (Desktop Management Interface), or SMBIOS(System Management BIOS) table contents into a human readable format.

VMware Storage VMotion GUI

2008-02-28T13:25:00.003-05:00

VMware's Enthusiasts can't wait VMware come out with their own GUI for Storage VMotion, and they have been creating their own GUI for this wonderful feature.

VI plugins from Schley Andrew Kutz
http://www.lostcreations.com/code/wiki/vmware/viplugins/svmotion

SVMotion GUI from Koen Warson
http://www.svmotion.com/

Changing ESX Service Console's IP, Gateway, DNS

2008-02-28T13:23:00.001-05:00

Found this KB in VMware's website, and thought it should be useful for anyone who is new to VMware ESX from a Windows background.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=4309499

VMware Storage VMotion

2008-02-04T21:53:00.000-05:00

With ESX 3.5 and VirtualCenter 2.5, VMware introduces a new feature called Storage VMotion. As a matter of fact, it isn’t a completely new feature. It has been used during Virtual Machine upgrade from ESX 2.x to ESX 3.x, which was called DMotion. It moves the Virtual Machine’s memory from a ESX 2.x host to a ESX 3.x host, and moves its virtual disks from VMFS 2 to VMFS 3 datastore. It greatly reduces the Virtual Machine’s downtime during upgrade. In ESX 3.5/VC2.5, the Virtual Machine stays in the same host, but moves its virtual disks to a new location. I think it is a great extend of VMotion feature. It helps a lot in SAN LUN maintenance, and load balancing.

You will need RCLI package installed. It has Windows and Linux version. Or better yet, just use VirtualCenter to download a RCLI Virtual Appliance directly from vmware.com. The command to run Storage VMotion is svmotion. It can be run in interactive mode or non-interactive mode.

Interactive Mode: use the --interactive flag to enter this mode, and it will ignore the rest of flags if it has any.

atl1396:~# svmotion --interactive

Entering interactive mode. All other options and environment
variables will be ignored. Enter the VirtualCenter service url you wish to connect to (e.g. https://myvc.mycorp.com/sdk, or just myvc.mycorp.com): VC.domain.com

Enter your username: VCadmin

Enter your password:

Attempting to connect to https://VC.domain.com/sdk.

Connected to server.

Enter the name of the datacenter: DataCenter

Enter the datastore path of the virtual machine

(e.g. [datastore1] myvm/myvm.vmx): [sanlun] VM1/VM1.vmx

Enter the name of the destination datastore: sanlun2

You can also move disks independently of the virtual machine. If you want the disks to stay with the virtual machine, then skip this step..

Would you like to individually place the disks (yes/no)? no

Performing Storage VMotion.

0% |-----------------------------------------------------| 100%

Non-interactive Mode:

svmotion --url=https://virtualcenter.domain.com/sdk

--username=user_name to login into VirtualCenter

--datacenter=DataCenter

--vm='[testlun1] TestVM1/TestVM1.vmx:testlun3'

--disks='[testlun3] TestVM1/TestVM1.vmdk:testlun3'

The --disks flags is optional. If the disks need to be stored at different datastores, or to separate datastore for .vmx file and .vmdk files. Sometimes it is good to use it to separate the datastore. Snapshots store in the same directory as the .vmx file by default.

VC 2.5 upgrade preparation list

2008-01-20T02:27:00.002-05:00

Preparation:

Database

1) Backup SQL database: VirtualCenter, msdb, master

2) Check to make sure SQL user, vcenter, has the db_owner role in both VCenter database, and msdb database, but don’t have the sa role.

3) Copy VCenter, msdb, master backup to ATL1502

4) Create a new database for VMware Update Manager

5) Setup OCBD for Update Manager

ESX servers

1) Make sure /tmp/vmware-root exists in every ESX servers.

· A tool ITQ VLan and Portgroup Manager from Paul Geerlings will make this task very easy. It can execute shell commands via SSH connection to multiple ESX servers from a central location. Use with care though. It is powerful enough to help with your tasks, and it is powerful enough to do some damage too.

·VirtualCenter

We have about 400 VMs sessions running in 17 physical hosts in our VMware environment. During upgrade, if a VM needs to do a hard reboot, it will be nightmare to search that VM by going to the hosts one by one. One of the prep jobs is to generate a list of all VMs along with all the host names.
HA error has been a known problem post VC upgrade. The error that upgrade could cause doesn’t limit to HA error, but also Service Console’s network connectivity. Although I have never seen this kind of recommendation in VMware’s documentation, I have experienced it myself, and I have seen many other people have the same problem.

The installation of VirtualCenter is pretty simple and straight forward. It is just like a normal Windows application.

NIC Teaming in SuSE Linux

2007-12-27T13:30:00.000-05:00

NIC Teaming in SuSE Linux

This change has to be done through console. Running ssh to the box wont' get the job done.

Create a bonding file, ifcfg-bond0 in /etc/sysconfig/network.

    linux880:/etc/sysconfig/network # vi ifcfg-bond0
    DEVICE=bond0
    IPADDR=172.18.9.97
    NETMASK=255.255.255.0
    NETWORK=172.18.9.0
    BROADCAST=172.18.9.255
    STARTMODE=onboot
    BOOTPROTO=static
    BONDING_MASTER=yes
    BONDING_MODULE_OPTS='mode=1 miimon=100 
    / use_carrier=0'
    BONDING_SLAVE0=eth0
    BONDING_SLAVE1=eth1

Create an ifcfg-ethx file for each interface in the bod. All interfaces should have MASTER and SLAVE definitions.

    linux880:/etc/sysconfig/network # vi ifcfg-eth0
    DEVICE=eth0
    STARTMODE=none
    BOOTPROTO=static
    MASTER=bond0
    SLAVE=yes
    USERCTL=no
   
    atl880:/etc/sysconfig/network # vi ifcfg-eth1
    DEVICE=eth1
    STARTMODE=none
    BOOTPROTO=static
    MASTER=bond0
    SLAVE=yes
    USERCTL=no

Delete the original NIC configuration file ifcfg-eth-id-MACADDRESS, or edit the configuration file to disable it.

Add the bonding alias to /etc/modprobe.conf.local and /etc/modules.conf so that system knows which one is the bonding file to loaded.

    linux880:/# more /etc/modprobe.conf.local
    #
    # please add local extensions to this file
    #
    alias bond0 bonding
   
    linux880: # more /etc/modules.conf
    alias bond0 bonding
    options bond0 miimon=100 mode=1

Now restart the network devices: /etc/init.d/network restart

    linux880:~ # /etc/init.d/network restart
    Shutting down network interfaces:
       bond0                                  done
       eth0      device: Broadcom BCM5708 (rev 12)
       eth1      device: Broadcom BCM5708 (rev 12)
    Shutting down service network  .  . .  .  done
    Setting up network interfaces:
       lo
       lo        IP address: 127.0.0.1/8      done
       eth0      device: Broadcom BCM5708 (rev 12)
       eth0      Startmode is 'none'       skipped
       eth1      device: Broadcom BCM5708 (rev 12)
       eth1      Startmode is 'none'       skipped
       bond0
       bond0     enslaving interfaces:  eth0 eth1
       bond0     IP address: 172.18.9.97/24   done
    Setting up service network  .  .  .  .  . done

Run ifconfig to view the configuration change in network setting. Add it should shows bond0, eth0, eth1 and lo in the configuration.

Changing ESX Service Console memory

2007-12-27T11:05:00.000-05:00

During default ESX installation, Service Console will be allocated 272MB of memory. Most of time, it is sufficient. However, if you have third party software installed in Service Console, or you have a powerful server (like Sun Fire x4600, with 8 Dual core CPU, 64GB memory) that can support more than 50 vCPUs, I strongly suggest to change the Service Console memory to 512MB. In ESX 2.x, VMware suggested 512MB should be used to support more than 32 vCPUs. Since ESX 3.x has a much more effect method to manage its memory, it runs 50 to 60 vCPUs without any problems with 512MB. If you have other third party software in the Service Console, you might want to increase to more than 512MB.

Here is how to change the Server Console’s memory, and it needs a reboot to take in effect.

First of all, back up the files that we are going to modify.

cp /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

cp /boot/grub/grub.conf /boot/grub/grub.conf.bak

Second, modify esx.conf and grub.conf:

1. Edit /etc/vmware/esx.conf to change the number of 272 to 512

Original: /boot/memSize = "272"

After: /boot/memSize = "512"

2. Edit /boot/grub/grub.conf to change:

Original: 272M

After: 512M

Original: uppermem 277504

After: uppermem 523264

Original: 272M

After: 800MB

Original: uppermem 277504

After: uppermem 818176

Update:

Just found sed is a stream editor in Linux. Using sed in a script would make this job much easier and less errors.

#!/bin/sh

#back up config files

/bin/cp /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

/bin/cp /boot/grub/grub.conf /boot/grub/grub.conf.bak

#editing esx.conf and grub.conf

/bin/sed -i -e 's/272/512/' /etc/vmware/esx.conf

/bin/sed -i -e 's/272M/512M/' /boot/grub/grub.conf

/bin/sed -i -e 's/277504/523264/' /boot/grub/grub.conf

Back to my blog

2007-12-20T06:32:00.000-05:00

Have been working with VMware's product for almost two year. The more I work with VMware ESX, the more I love it. It has been a great experience. From VMware ESX 2.5.x to the latest one ESX 3.5, I have set up different solutions and designs to create a stable, redundant virtualization environment. From basic setup, to upgrade, to backup, the flexibility that virtualization has provided really amazed me, and make me love my job more and more from one day to another. Of course, I have been getting help from VMware's virtualization communities. Some of them are from VMware's forum, some are from VMware's enthusiasists. They also helped me to pass VMware's VCP (VMware Certified Professional) exam. I really appreciate their help. Here I am trying to list some of my favirite help resources. They are very good VMware resources.

http://www.xtravirt.com/ - At that time I was search from a backup solution for ESX 3.0.1's VM. VMware's VCB 1.0.1 was failing on me. VCB was able to create a snapshot of the VM, let backup software back it up, but lots of times, it failed to delete the snapshots. In VMware's forum, Alex Mittell posted his VISBU backup script. After trying out his script, found it was awesome. After a couple other releases, it is getting better and better. Laster on, he and other guys found this company in UK. There are a lot of White Papers, tips, and utilities for VMware in this web site.

Could not power on VM

2007-02-28T10:28:00.001-05:00

Problem: No swap file. Failed to Power on VM

Solution: The reason that the VM not be able to start up is that the VM process is running on another ESX host in the same cluster group. Run "ps –ef grep VM_server_name" on each ESX host to find out which one owns it and then kill that process. VM should be able to powered on from VirtualCenter at this point.

Object Not Exist

2007-02-28T09:44:00.001-05:00

Problem: The request refers to an object that no longer exists or has never existed.

Solution: This would happen in different scenarios. Restarting VirtualCenter Agent service in the host would most likely fix the problem (command is "service mgmt-vmware restart). If not, restart the VirtualCenter service in the VirtualCenter Server.

P.S. The platform here is ESX 3.0.1 and VC 2.0.1.

how to check daylight saving time in VMware ESX

2007-02-16T16:35:00.001-05:00

It took me a while to find out if my ESX hosts had the patch for dayling saving time change. Here are a couple defferent commands to check it.

zdump -v /etc/localtime grep 2007

zdump -v US/Eastern grep 2007

zdump -v EST5EDT grep 2007

Failed to install VirtualCenter Agent service

2007-02-13T17:20:00.001-05:00

Problem:

Upon upgrading VMware VirtualCenter 2.0.1 patch 33643, I was trying to connect to the VI3 host back to VirtualCenter, but one host keep giving an error "Failed to install VirtualCenter Agent service".

Solution:

VMware VirtualCenter is having problem to stop and uninstall the VirtuaCenter Agent service in the VI3 host. All we need to do is manually stop the service, uninstall the agent, and reconnect the host. A new VirtualCenter Agent would be able to install in the host upon reconnection.

___________________________________

To double check the version of the VirtualCenter Agent:

[root@ESX3 root]# rpm -qa grep vpxa

VMware-vpxa-2.0.1-32042

To stop mgmt-vmware:

[root@ESX3 root]# service mgmt-vmware stop

Stopping VMware ESX Server Management services:

VMware ESX Server Host Agent Services [ OK ]

VMware ESX Server Host Agent Watchdog [ OK ]

VMware ESX Server Host Agent [ OK ]

To stop vpxa:

[root@ESX3 root]# /etc/init.d/vmware-vpxa stop

Stopping vmware-vpxa: [ OK ]

Uninstall vpxa:

[root@ESX3 root]# rpm -e VMware-vpxa-2.0.1-32042

Stopping vmware-vpxa: [ OK ]

warning: /etc/vmware/vpxa.cfg saved as /etc/vmware/vpxa.cfg.rpmsave

To make sure vpxa is no longer there:

[root@ESX3 root]# rpm -qa grep vpxa

[root@ESX3 root]# rpm -qa grep vpx

Restart up mgmt-vmware service:

[root@ESX3 root]# service mgmt-vmware start

Starting VMware ESX Server Management services:

VMware ESX Server Host Agent (background) [ OK ]

Availability report startup (background) [ OK ]

After reconnecting to the VI3 host from VirtualCenter, double check to make sure the version of the VirtualCenter Agent is right:

[root@ESX3 root]# rpm -qa grep vpxa

VMware-vpxa-2.0.1-33643

Can’t VMotion VM error

2007-02-12T22:29:00.001-05:00

Problem:

When trying to VMotion a VM in ESX 2.5.3, a error message comes up in the validation check "Unable to migrate from host: Virtual machine as CPU and/or memory affinities configured, preventing VMotion."

Solution:

There is a config file for VirtualCenter 2.1 called vpxd.cfg located in the VirtualCenter server's "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter. Search for </vmcore>, and add the following lines between </vmacore> and </config>.

<test>

<AffinityNotConfigured>false</AffinityNotConfigured>

</test>

</migrate>

Troubleshooting 1

2006-11-15T12:30:00.001-05:00

Problem: Recovers a VMFS when a VMFS is locked and no ESX server is controlling it

Solution:

vmkfstools

-R --recover

This command enables you to recover a VMFS (accessible by multiple ESX servers) when other vmkfstools commands indicate that the file system is locked by another ESX Server machine, but, in fact, no other server is currently accessing this file system. This situation may occur if the VMFS was being accessed by a server (for example, running a virtual machine) and that server crashed.

Note: You should only use this command if you are certain that no other ESX Server is still accessing the file system.

Problem: Scans the specified vmhba adapter for devices and LUNs

Solution: vmkfstools

-s --scan <FC_SCSI_adapter>

This option is particularly useful for adapters connected to storage area networks, particularly if you are reconfiguring your SAN. If a new device or LUN becomes accessible through the adapter, then ESX Server registers this new virtual device for use by virtual machines. If an existing device or LUN is no longer used and appears to be gone, then it is removed from use by virtual machines.

Note: Only use this -s option for Fibre Channel adapters.

You can see the results of the scan by using ls /vmfs or looking at the contents of /proc/vmware/scsi.

VMware Converter 3.0 Beta

2006-11-15T10:34:00.000-05:00

Just try out VMware Converter 3.0 Beta, and I got to say it is damn cool. It can hot clone a server (a running server) into a virtual machine in ESX. And I can modify the hard drive size on C and D drive. For a total of 10GB hard drive space, it takes about 20 minutes to move it, plus another 10 minutes for it to connect to VirtuaCenter 2.1. With P2V 2.1.1 or P2V 2.1.2, I need to use a image software to create a image of the hard drive first and then restore it in to a VM, then use the P2V assistant to reconfigure it. But with VMware Converter, it is just one step. The time spending on transforming a physical server into a virtual server has dramatically reduced. And the downtime reduces to less than one minute. In our test, it dropped 2 packets while it’s being pinged. But overall, it is a very good tool.

However, if they could shutdown the physical server after Converter finish cloning it, before it starts up the new VM, it would be much better.

Just found a cool web site about virtualization.

http://www.easyvmx.com/ is the simple and failsafe way to create complete virtual machines for VMware Player on the web. It will generate the files for a virtual machine that can run in VMware Player.

Upgrade IBM BladeServer HS20's firmware

2006-09-29T15:30:00.000-04:00

IBM BladeCenter HS20 Type 8843

Blade Server Firmware Update
In IBM BladeCenter, blade servers’ firmware updates are very critical. It greatly impact on its stability, and performance.

Here are the steps that I normally do to update the firmware.
1) Check the firmware version on the particular blade server.
a. Login into the BladeCenter management module.
b. Go to Monitor->System Status->Firmware VPD to check the firmware version of the blade server’s BIOS, Diagnostics, and Blade System Management Processor. The Firmware VPD displays the BIOS and firmware levels of the chassis, switches, blades and management modules within the BladeCenter. There are always recommended minimum levels of code for all flashable devices. Always check the IBM support Web site for the recommended levels of code. It may not be possible to work with a problem if firmware is down-level (or the problem may be caused by down-level firmware).

In the chart below, IBMBlade3 has the latest firmware, IBMBlade4’s firmware has been outdated, and it needs to be updated.

Firmware update figure 1

2) Locate the firmware for the right blade server
a. Go to Blade Tasks -> Firmware update, choose which blade server as the target to do the firmware update, and then browse to find the firmware file, which should have an extension “.PKT”. After clicking on Update, it reads the file, and the “Confirm Firmware Update” page will appear. We can click on Continue to perform the task.

During the update process, DO NOT, power off, restart, or remove the module from the chassis. All of these are being done in the background, and no restart of the blade server needs to be done. And it shows you how the process of the update.

Firmware update figure 2

Firmware update figure 3

3) It will let you know if it is finished. The whole update process takes about 10 minutes for the firmware update.

Firmware update figure 4

4) Now we can go back to Firmware VPD (Vital Product Data) to check the Blade System Management Processor version. (In this case, the Build ID should be BWBT23A)

Enabling VMotion

2006-09-08T09:46:00.000-04:00

Select the host that you want to enable VMotion
Click on Configuration, and on the left hand side there is a panel Hardware, click on Networking
In the Virtual Switch that you want to enable VMotion, click on Properties
In the vSwitch properties’ tab Ports, add a VMkernel configuration
After you add it, click on Edit
Here are the fields that are in the VMkernel Properties

Port Properties
Network Label: Use a name that is meaningful to you
VLAN ID (Optional): Like what it said, you might not need it. But if you do, you should know what ID you need to use.
VMotion: check to box to enable VMotion
IP Settings
IP Address: 10.10.10.100
Subnet Mask: 255.255.255.0
VMkernel Default Gateway: 10.10.10.1
Here should be a subnet just for VMotion only, since it will generate lots of traffic when VMotion is in process. And the default gateway doesn’t have to be a real one. blockquote>

Example of VMkernel Properties

This is what it looks like after the VMkernel has been setup in vSwitch1.

Do the same thing to the other host that is in same cluster (the resource pool).
Now VMotion has been created in the cluster. We can go ahead to test VMotion, HA, and DRS. In fact, VMotion is the foundation of HA and DRS. If VMotion is working fine, all you need for HA and DRS is the licenses. Pretty simple and straight forward.

Installation of VMware VirtualCenter 2.0

2006-08-01T10:17:00.000-04:00

VMware VirtualCenter is the tool from VMware to monitor and manage its virtualization. VirtualCenter 2.0 is the latest version, and it provides excellent scalability, high availability, networking, and event management. Here is the VMware VirtualCenter 2.0 installation practice that I have done. Now someone can benefit from this.

First, we need to prepare a SQL server database to work with VirtualCenter. Use SQL Server Enterprise Manager to create a SQL Server database and a database user with database operator DBO rights. That’s it for the SQL Server database.

Now we can go to the VirtualCenter server to install VirtualCenter. Before we do the installation, we need to set up the ODBC Data Source.

Go to Settings -> Control Panel -> Administrative Tools -> Data Source (ODBC)

Select System DSN tab, and create a new SQL Server ODBC connection by clicking on Add, and select SQL Server, and click Finish.

Type an ODBC DSN name in the Name field. For example, Type in Virtual Center. Optionally, type an ODBC DSN description in the Description field. Select or type in the SQL Server host name

After clicking on Next, select SQL Server authentication (Windows NT authentication is supported only if the SQL Server is running on the same system as VirtualCenter Server), enter the VirtualCenter database’s login information.

After choosing the correct database, we can test the Test Data Source. If it passes the test, we are done.

VirtualCenter installation
During VMware VirtualCenter Server 2.0’s installation wizard, the first thing it would ask is the database storage. For testing purpose, you could use MSDE. But for production, it is recommended to use Microsoft SQL Server or Oracle. VMware won’t provide technical support in MSDE.

After clicking on Next, it would ask for the database information. Enter the Data Source Name (DSN) for the VMware VirtualCenter database that was created previously (if it is not created yet, click on the button ODBC DSN Setup), along with the ODBC Source Administrator and password.

Proper licensing is required to run the VMware VirtualCenter Server, and here we are configuring it to connect to a VMware License Server, which has been installed in the same server as the VirtualCenter Server.

Next step is to configure VMware VirtualCenter Web Service. And the following port numbers are default.

The next step is to set up how to start up VirtualCenter Web Service.

After this, you just need to confirm that you want to continue to install VMware VirtualCenter, and it will do the rest. Now you are done.

VI 3 Notes

2006-07-25T16:31:00.000-04:00

Service Console
To change hostname: /etc/sysconfig/network

******************

Installation

Installation Logs:
/root/install.log is a complete log of installation
/root/anaconda-ks.cfg is a kickstart file containing the selected installation

Using ILO, DRAC might encounter corruption problems if using the virtual CD feature.
RSAII is not recommended to use virtual CD feature to install or upgrade ESX Server.

VMFS is not supported on IDE or SATA.

An ESX Server must have SCSI storage, NAS or a SAN non which to store VM.

ESX Server supports up to 256 LUN for Operation, but the installer supports a maximum of 128 LUN only.

VMware recommends that you do not rearrange driver controllers among PCI slots after you have installed the ESX Server software.

DO NOT install ESX Server on a SHARED LUN.

Before the installation, ZONE and MASK all the SAN LUN away from the server.

******************

Hardware

• VMware provides a driver disk for
• New hardware for VM: after installation, use VirtualCenter to assign the hardware to the virtual machines.
• New hardware for the Service Console: A simple reboot won’t make the service console aware of the newly installed hardware; Run kudzu to detects any new hardware and adds appropriate entries to /etc/modules.conf; or edit /etc/modules.conf manually:for example, adding a new SCSI adapter that use a driver named adapXXXX, add this line:
alias scsi_hostadapter adapXXXX
• Device Driver Mappings during upgrade to VI3
ESX Server installer renames the file
/etc/vmware/vmware-device.map.local
to /etc/vmware/vmware-device.map.local.orig

******************

Backup

Creating a Rescue Floppy Disk
Use dd, rawwritewin, rewrite to create a floppy image called bootnet.img. This file is located on ESX Server CD in the /images directory.

******************

Storage
How to add storage to VI3 in VirtualCenter 2

In VC2, go to the server's configuration, click on Storage (SCSI, SAN, and NFS) in the left inside the hadware panel, and click on "Add Strorage..." in the upper right corner.
In the Add Storage dialog box, we can choose either a)Disk/LUN for Fibre Channel, iSCSI SAN storage, or local SCSI disk; b)Network File System for a shared folder over a network for VMware datastore.
In the next step, it will display all the scanned volume device. Choose the correct one, and click on next.

P2V

For a image containing a OS, after restoring the image to attached virtual hard drive, we need to make it an active drive in the attaching virtual server. Otherwise, it won't be able to boot.
It is the same thing for P2V Assistant helper virtual server. We need to make the attached virtual hard drive active before P2V Assistant can recongnize it by size, OS and reconfigure it.

Server Virtialization

2006-07-25T13:30:00.000-04:00

Last week I went to VMware's Virtual Infrastruture 3 seminar. It was very informative, and got a chance to hear 2 VMware customers real experience with VI3. At the end, I won a VMware T-shirt. VMware has improved itself over the past a couple years, and now it is still growing by partnering with Intel, Microsoft and etc. Anyway, here are some information I learned from this seminar.

********************************************

As the interface between server hardware and multiple operating systems, ESX Server performs a complex set of tasks. It controls all physical resources, such as processors, memory, storage, graphics cards and network adapters, and arbitrates between the guess OS to support IT service level and avoid conflicts.

OS traditionally run in a privileged space in the software stack, which gives them prioritized access to physical resources. In today’s virtualization solutions, ESX Server runs in this privileged space, and guest OS run in the space traditionally used for applications.
Since ESX Server runs directly on server hardware, it impacts the performance and reliability of all the other applications running on the same system. Now Intel has introduce Intel Virtualization Technology to optimize the virtualization on Intel architecture.
Intel Virtualization Technology has helped to create a new, privileged space for ESX Server, so guest OS and applications can run in the environment they were designed for. It greatly increases virtualization-friendly hardware environment. This reduces the need for intervention, and allows guest OS to run directly on the hardware. Hardware-based transitions-handoffs between ESX Server and guest OS are supported in hardware, reducing the need for complex, compute-intensive software transitions. Hardware-based memory protection-Processor state information for ESX Server and each guest OS is retained in dedicated address spaces to accelerate transitions.

Hyper-Threading Technology
This technology enables each processor to handle two software threads or tasks simultaneously, and benchmark tests have shown it can increase application performance for some server applications by as much as 30 percent. In an ESX server running multiple virtual machines, this can be a significant advantage to increase performance and flexibility.

Multi-core processors
Intel is on a track to deliver dual-core and then multi-core processors, which will continue to increase total server power and flexibility for virtualized environments. In conjunction with Hyper-Threading, a 2-way box with dual-core processors will support up to 8 simultaneous tasks; a 4-way box up to 16 tasks; an 8-way box up to 32 tasks; and a 16-way box up to 64 tasks.

64-bit Virtual Machine will be support in ESX, which will enable IT organizations to support a much broader range of business applications in a virtual environment. They will be able to fun both 64-bit and 32-bit OS on the same server, and each 64-bit OS will be able to host both 64-bit and 32-bit applications. This flexibility will enable IT organizations to plan and implement selective and incremental 64-bit software migrations, to optimize ROI based on specific business and application requirements.

Virtual memory in Windows

2006-07-20T10:47:00.000-04:00

Memory paging occurs when memory resources required by the processes running on the server exceed the physical amount of memory installed. Windows, like most other operating systems, employs virtual memory techniques that allow applications to address greater amounts of memory than what is physically available. This is achieved by setting aside a portion of disk for paging. This area, known as the paging file, is used by the operating system to page portions of physical memory contents to disk, freeing up physical memory to be used by applications that require it at a given time. The combination of the paging file and the physical memory is known as virtual memory. Virtual memory is managed in Windows by the virtual memory manager (VMM). Physical memory can be accessed at exponentially faster rates than disk. Every time a server has to move data between physical memory and disk will introduce a significant system delay. While some degree of paging is normal on servers, excessive, consistent memory paging activity is referred to as thrashing and can have a very debilitating effect on overall system performance. Thus, it is always desirable to minimize paging activity. Ideally servers should be designed with sufficient physical memory to keep paging to an absolute minimum. The paging file, or pagefile, in Windows, is PAGEFILE.SYS. Virtual memory settings are configured via the System control panel. To configure the page file size: 1. Open the System Control Panel. 2. Select the Advanced tab. 3. Within the Performance frame, click the Settings button. 4. Select the Advanced tab. 5. Click the Change button. The window shown in Figure 1-1 will appear. Window Server 2003 has several options for configuring the page file that previous versions of Windows did not. Windows Server 2003 has introduced new settings for virtual memory configuration, including letting the system manage the size of the page file, or to have no page file at all. If you let Windows manage the size, it will create a pagefile of a size equal to physical memory + 1MB. This is the minimum amount of space required to create a memory dump in the event the server encounters a STOP event (blue screen).

Figure 1-1 Virtual memory settings

A pagefile can be created for each individual volume on a server, up to a maximum of sixteen page files and a maximum 4 GB limit per pagefile. This allows for a maximum total pagefile size of 64 GB. The total of all pagefiles on all volumes is managed and used by the operating system as one large pagefile.
When a pagefile is split between smaller pagefiles on separate volumes, the virtual memory manager optimizes the workload by selecting the least busy disk based on internal algorithms when it needs to work write to the pagefile. This ensures best possible performance for a multiple-volume pagefile.
While not best practice, it is possible to create multiple page files on the same volume. This is different to what most documentation suggests. This is achieved by placing the page files in different folders on the same volume. This change is carried out via editing the system registry rather than via the standard GUI interface. The process to achieve this is outlined in Microsoft KB article 237740: http://support.microsoft.com/?kbid=237740

Installing HS20 HBA driver

2006-06-30T17:15:00.000-04:00

In order to connect ESX3.0 to the SAN, the IBM blade server HS20’s HBA bios, and driver needs to be EMC approved, and have the PowerPath installed.
Although in the BIOS it says it is a QLogic 2310 HBA, however, in QLogic’s website, there is no driver for 2310. It is weird. And further shows that it is a QLogic 2340 made for IBM.

[root@bdc1esxv1 byron]# rpm -ivh qla2x00-v7.03.00-2.i686.rpm
Add these two lines:
alias scsi_hostadapter2 qla2300_conf to /etc/modules.conf
alias scsi_hostadapter3 qla2300 to /etc/modules.conf

Manually edit /etc/modules.conf to add this line:
options qla2300 ConfigRequired=0 ql2xfailover=0

Want to check if the driver for QLogic HBA has been installed? Open and take a look at the file /proc/modules:
qla2300_conf 301624 0 (autoclean)
sg 36612 0 (autoclean)
loop 12432 0 (autoclean)
…
…

Windows Vista features

2006-06-28T14:15:00.000-04:00

Windows TAP team introduces a new service as the solution to delegating the install of per-machine ActiveX controls to a standard user in the enterprise. This is the subject of today’s Live Meeting with Microsoft TAP team. It consists of a Windows service, a Group Policy administrative template, and a few changes in Internet Explorer. However, it is not in the current Beta 2 yet. Still need to wait. However, I found a very nice and neat feature in Vista. Under Control Panel -> Programs -> Currently Running Programs, the Windows Defender shows all the currently running processes, and their details information. For example, Microsoft Office Live Meeting application is being shown like this:

File Name: PWConsole.exe
Display Name: Microsoft Office Live Meeting
Description: Microsoft Office Live Meeting application
Publisher: PlaceWare, Inc., a wholly owned subsidiary of Microsoft Corporation.
Digitally Signed By: Microsoft Code Signing PCA
File Type: Application
Auto Start: No
File Path: E:\Program Files\Microsoft Office\Live Meeting 7\Console\7.9.1709.0\PWConsole.exe
File Size: 2702608 Bytes
File Version: 7.9.1709.0
Date Installed: 5/9/2006 9:03:46 AM
Process ID: 7540
User Name: CO\byron.zhao
Classification: Allowed
Ships with Operating System: No

Now we can find out any non-legitimate processes easily.

Dell BladeCenter 1855

2006-06-22T10:38:00.000-04:00

Just got informed that we got a Dell BladeCenter in to test. Wolf… Sweat. Can’t wait to get downstairs to pick it up. After the inventory, here is what we got.

7U PowerEdge 1855 Chassis that can hold 10 blade servers
1 PowerEdge 1855 blade server

2 Hard drive, 146 GB UltraSCSI 320 hard drives
1 QLogic HBA
2 Xerox CPU
6 GB Memory
2 Front USB/Video cable
1 Rear KVM cable
2 Ethernet Pass-Through blades
2 Fibre Channel Pass-Through blades

Here is Dell PowerEdge 1855 BladeCenter's detail. The look and design are very similiar to Fujitsu BX 600 BladeCenter. I haven't tried the 1855 yet, but I like the Fujitsu BX600 BladeCenter more over the IBM HS20. The deployment, and management software is way better than IBM's. However, Fujitsu BX600 only has Brocade switch module, but not have McData fibre channel module to support SAN. I think we will go for Pass-Through module than the Brocade Silkworm module.

Anyway, after knowing that we only have one Dell blade server to test, kind of lose insteresting in it. Going back to work on VMware.

Whats is going on

2006-06-21T16:51:00.000-04:00

In the past a couple days, I have been working on VMware Virtual Infrastructure 3.0. This time VMware provides very powerful features in its enterprise product. I joined their webinar yesterday, and watched the demo they did. It has very similiar features as Fujitsu-siemens' Adaptive Services Control Center (ASCC), which could dynamically distribute the resource to different applications based on the management policies. And VMware make it software base to share the resources.

Can't wait. I have installed the VI3.0 on a HS20 in the IBM BladeCenter. The installation process is even simplier than the ESX 2.53. And I also installed the VirtualCenter 2.0 and the Virtual Infrastructure Client 2.0 on a BX620 S2 blade (Blade7) in the Fujitsu BladeCenter. This time I have no problem to have the VirtualCenter connect to the SQL server on Blade8.

Now I am reading the documentations on the VI3.0, and will have it connect to the SAN. It is fun to learn all these. Not sure how much will be trained in the Jumpstart training, but I am looking forward to it.

Today I got 4 GB from Keith to put into a HP ML 370. I am going to use it to test VI 3.0 too. :-) Got to love all these toys.

Upgrading the ISO in the CIGESM

2006-06-13T11:33:00.000-04:00

Last Friday the IOS in CIGESM in the IBM BladeCenter needs to be upgrade. CIGESM is a Cisco Intelligent Gigabit Ethernet Switch Module for IBM eServer BladeCenter (Part number: 13N2281). In Cisco web site I found a couple of the latest ISO for CIGESM. This time I wanted to try something new. I downloaded an ISO crypto tar file. A crypto ISO supports SSH connection. For a bin file, I can just copy the bin file from the tftp server to the switch. But now with a tar file, I need to do some research to see how to untar it in Cisco switch. After the research, I found that it is quite easy. Here is the fun.

SDC1BLAD2#archive download-sw /overwrite
tftp://172.16.130.43/cigesm-i6k212q4-tar.121-22.EA6a.tar

“archive download” command’s usage can be found here.

It overwrote the old IOS file, and the HTML folder. The switch needs to be reloaded in order to reflect the new IOS since the old IOS has been loaded into the memory. After the reload, we need to put the domain name in it in order to generate the key for security connection.

SDC1BLAD2(config)#ip domain-name test.com
The name for the keys will be: SDC1BLAD2.test.com
SDC1BLAD2(config)#crypto key generate rsa
Choose the size of the key modulus in the range of 360 to 2048 for your General Purpose Keys. Choosing a key modulus greater than 512 may takea few minutes.
How many bits in the modulus [512]:Generating RSA keys ...[OK]
SDC1BLAD2(config)#

CPU Upgrade - IBM Blade Server HS20 (type 8843)

2006-06-12T10:33:00.000-04:00

Ok, it is time to upgrade the blade server HS20 to dual processors. On Sunday’s maintenance window, a second CPU was successfully installed into 5 IBM blade servers HS20.

It was much easier than I did it to my own PC long time ago.
Here are a couple steps that I went through to install the second CPU, alone with a couple snap shots using my Sony W600i cell phone.

Installation:
Here is the CPU and the heat sink.

After opening up the cover, I carefully removed bezel assembly (need to watch out the control panel cable), disconnected the control panel cable from the control panel connector. At this moment, the CPU heat sink filler can be easily lift up and removed.

Peeled off the sticker use to cover the CPU’s pin holes. Pull out and lift up the CPU locking lever to unlock the CPU socket. Touch the static-protective package containing the CPU to any unpainted metal surface on the blade server. Align up the triangle on the corner of the CPU with the triangle on the corner of the socket and carefully let the CPU sit into the socket. Make sure that the CPU is oriented and aligned correctly in the socket, and do not use excessive force to press it. It should be able to sit into the socket itself without pressing it. After that, the lever can be carefully close to secure the CPU in the socket.

After removing the heat sink from the plastic protective cover, align and place the heat sink on top of the CPU in the retention bracket, with grease side down. Press firmly on the heat sink. A screwdriver is used to secure the heat sink to the retention bracket on the system board alternating between two captive mounting screws.
After all these, replace the bezel assembly and cover, reinstall the blade server in the BladeCenter, and turn on the blade server.

Configuration:
The BIOS detected the second CPU automatically. And the Windows 2003 server detects the CPU upgrade automatically too, so basically there is no OS configuration needs to be done.

Performance:
The CPU upgrade was done in 5 production blade servers, so I didn’t use and benchmark to test it. But there is a noticeable speed increase in the performance.

And here are a couple more snap shots of the device manager and task manager before and after the CPU upgrade.
Before:

After:

Mounting a ISO file in ESX VMware

2006-06-08T09:35:00.000-04:00

Since I have the ESX ISO image file already, why I still need to use the CD? After some research, I found the command used to mount an ISO image on a Linux system.

mount -o loop -t iso9660 /usr/lib/vmware/isoimages/esx-2.5.2-16390.iso /mnt/cdrom/

The flag –o loop make the mount try to find some unused loop (none explicit loop) device and use it.
I was able to read the files from the ISO image without any problem.
However, when I run
/usr/bin/scriptedinstall-setup.pl
it failed with the following error messages:

mount: you must specify the file system type
umount: /mnt/loop0: not mounted
Unable to open
/usr/lib/vmware-mui/lib/perl5/site_perl/5.005/VMware/Management/SS/
ScriptedInstall/syslinux.cfg
for reading No such file or directory

After I umount the ISO image, and mount a real ESX CD by running:
mount /dev/scd0 -t iso9660 -o ro /mnt/cdrom/
and run the scripted install setup script, it finished successfully with the following messages:

Setup requires Apache to be restarted. This will invalidate all current VMware Management Interface sessions. Restart Apache now? (y/n)

To umount the ISO or the CD use the same command:
umount /mnt/cdrom

Dell PowerEdge 6600 RAID controller error

2006-06-07T09:11:00.000-04:00

Error:
Last Saturday, someone called me and said the Dell OpenManage management server had an amber light in the front and it kept beeping. Today when I came to work to take a look at it, I found that it has one hard drive failed, and it was beeping loud.

Solution:
Open up the OpenManage Server Administrator of the server, go to System->Storage->PERC 4/DC, click on Information/Configuration in its Properties. Here a couple Controller Tasks can be chosen to execute. Click on Available Tasks, and choose Quiet Alarm, click on Execute to run this task. It will stop the RAID controller’s beeping.

Mount & unmount a medium in Linux

2006-05-12T17:08:00.000-04:00

This was done in ESX VMware.

1) Make sure /mnt/cdrom exists. If not, type
mkdir /mnt/cdrom

2) To mount the CD, type
mount /dev/scd0 –t iso9660 –o ro /mnt/cdrom

In Linux system, mount is a command to mount a file system.
–t flag is a argument used to indicate the file system type.
iso9660 is a CD file system type that it supports.
-o ro means it mounts the file system read only.
/dev/scd0 is the source.
/mnt/cdrom is the destination.
3) Before a CD can’t be reject, the CD ROM has to be unmounted.
umount /mnt/cdrom

To mount/unmount a USB floppy drive:

1) Make sure /mnt/floppy exists. If not, type
mkdir /mnt/floppy
2) To mount the diskette, type
mount /dev/sda –o auto /mnt/floppy
3) To unmount a diskette,
umount /mnt/cdrom

The First Post

2006-05-12T00:18:00.000-04:00

Life is funny. Remember there was a commercial about a job hunting website. One human was working for a bunch of monkeys. Well, sometimes in real life, it is like that. However, it is still life. It keeps going on. Better enjoy it even if it is not that funny. If you can't change what is around you, change yourself to adapt it. It is raining now, why not dancing in the rain?