DL380/385 Gen8 Install ESXi4.1

Case: Missing NIC and storage drivers in our ESXi4.1 installation

First thing I noticed when the I tried to install ESXi4.1 on the Gen8 series is that our standard (VMware) image wasn’t able to install. Not by our scripted UDA and also manual failed.
With the scripted install I noticed the NIC drivers where not recognized and the hardware was unidentified which resulted in the NIC not connecting to our deployment server to grab the KS.CFG files.
The manual install failed by a missing Storage Adapter driver.

When I used the ESXi4.1 recommended image by HP with all the drivers inserted and loaded, the installation went fine. After the installation I used

esxcfg-scsidevs –l
esxcfg-nics –l

To see what hardware was exactly in the machine.

NIC: Broadcom NetXtreme BCM5719
Storage Array: HP System Array P420I
With this information I found the drivers on the HP site.
Driver: vmware-esx-drivers-scsi-hpsa-400.4.1.0-26OEM.x86_64
Driver : vmware-esx-drivers-net-tg3-400.3.123b.v40.1-1vmw.2.17.00000.x86_64

So ? How do we add this to our PXE deployment?
ESXi4.1 has a sort of local hardware database file with all components and vendors which are supported, and a list of which hardware uses what drivers.
Located in /etc/vmware the files are PCI.IDS and SIMPLE.MAP

In this file the hardware database is located, it’s a flat file. If the device ID is recognized during boot, ESX will give the corresponding name to it and it’s later used to load the correct drivers. Luckily I had 2 systems. One system I installed with the HP OEM VMware image, the other was my texting machine. I simply pulled the PCI.IDS file from the HP OEM server and saved it locally for later use.
In our case we are missing the HP Smart Array P420i

Example of PCI.IDS

103c Hewlett-Packard Company
323b Smart Array P222
103c 3350 Smart Array P222
103c 3351 Smart Array P420
103c 3352 Smart Array P421
103c 3353 Smart Array P822
103c 3354 Smart Array P420i
103c 3355 Smart Array P220i
103c 3356 Smart Array P721m

This file represents a list of device ID’s and which driver modules they need to load. I also needed to add the correct drivers and modules here. Also in this case I just copied the simple.map file.
103c:323b 103c:3354 storage hpsa
The first explains the PCI device ID, the second category (storage) the last module is the driver that is loaded (hpsa).

So wrap up. Now the hardware devices can be recognized and the right drivers can be loaded. Too bad we still have older drivers in our image and need to update them first.

Logging in to the HP CD installed servers I figured out which driver versions are needed.

vmkload_mod -l|egrep "tg3|hpsa"
tg3 4 176
hpsa 2 76
~ # vmkload_mod -s hpsa|grep Version
Version: Version 4.1.0-26OEM, Build: 00000, Interface: ddi_9_1 Built on: Jan 19 2012
~ # vmkload_mod -s tg3|grep Version
Version: Version 3.123b.v40.1, Build: 00000, Interface: 9.0, Built on: Apr 3 2012

So now we now which drivers and versions are used to control the hardware.
Creating Custom Image

To inject drivers we can use the VMWare provided vibddi tool on a Linux/CentOS machine.
– Install vibddi

rpm -ivh vmware-esx-vibddi-.i386.rpm

– Installation instructions

Copy image file and extract
– Copy the original ISO using WINSCP to /opt/vmware/vibddi/iso

– Mount the original image:

Mount /var/public/smbmount/esx41iso/VMware-VMvisor-Installer-4.1.0.Update3-custom.iso /var/public/www/esx41i/ESXi41U3Cust –t= iso9660 (rw,loop=/dev/loop1)

– Copy the original imagedd.bz2 to a temporary folder /imagebuild/

cp /var/public/www/esx41i/ESXi41U3Cust/imagedd.bz2 /imagebuild[text]
total 405929
-r-xr-xr-x 1 root root 11225 Nov 6 22:36 a.z
-r-xr-xr-x 1 root root 2048 Nov 6 22:40 boot000.cat
-r-xr-xr-x 1 root root 2048 Nov 7 15:24 boot.cat
-r-xr-xr-x 1 root root 1162824 Nov 6 22:36 cimstg.tgz
-r-xr-xr-x 1 root root 15283377 Nov 6 22:36 cim.vgz
-r-xr-xr-x 1 root root 7306598 Nov 6 22:36 ienviron.vgz
-r-xr-xr-x 1 root root 312820483 Nov 7 15:14 imagedd.bz2
-r-xr-xr-x 1 root root 67 Nov 7 15:16 imagedd.md5
-r-xr-xr-x 1 root root 971501 Nov 6 22:40 install.vgz
-r-xr-xr-x 1 root root 12241 Nov 6 22:36 isolinux.bin
-r-xr-xr-x 1 root root 298 Nov 6 22:36 isolinux.cfg
-r-xr-xr-x 1 root root 47404 Nov 6 22:36 mboot.c32
-r-xr-xr-x 1 root root 51020 Nov 6 22:36 menu.c32
-r-xr-xr-x 1 root root 2593205 Nov 6 22:36 open_sou.txt
-r-xr-xr-x 1 root root 2156 Nov 6 22:36 readme.txt
-r-xr-xr-x 1 root root 72786024 Nov 6 22:36 sys.vgz
-r-xr-xr-x 1 root root 41155 Nov 6 22:36 tboot.gz
-r-xr-xr-x 1 root root 48339 Nov 6 22:36 vmkboot.gz
-r-xr-xr-x 1 root root 2489796 Nov 6 22:36 vmkernel.gz
-r-xr-xr-x 1 root root 34816 Nov 7 15:23 vmware_v.iso

– Now unzip the imagefile you copied

cd /imagebuild
bunzip2 imagedd.bz2

Adding VIB’s / offline bundles to the image
– Use VIBDDI to see which items are already in the image.

Vibddi –i imagedd –q

– Example:

< oem-vmware-esx-drivers-net-vxge (400. >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400. >
< vmware-esx-firmware (4.1.0-3.26.800380) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-3-26 >
< vmkepsecapi_1.0.0.0 >
< vmkvsepapi_2.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmkuservsepapi_2.0.0.0 >
< vmware-esx-tools-light (4.1.0-3.26.800380) >

– Now we are going to inject the correct drivers , I downloaded and copied them to /bundles

vibddi -i /imagebuild/imagedd -v cross_oem-vmware-esx-drivers-scsi-hpvsa_400.4.1.0-22OEM.vib
Setting up configuration for esxupdate
This may take a few seconds…

– Repeat this for the other drivers. When you finished use the -q switch again to see if the correct drivers are injected. Now you see that they’re added.

vibddi -i /imagebuild/imagedd –q
< oem-vmware-esx-drivers-net-vxge (400. >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400. >
< vmware-esx-drivers-net-tg3 (400.3.123b.v40.1-1vmw.2.17.00000) >
< vmware-esx-drivers-scsi-qla2xxx (400.841.k1.42.1-1vmw.2.17.00000) >
< vmware-esx-firmware (4.1.0-3.26.800380) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-3-26 >
< vmkepsecapi_1.0.0.0 >
< vmkvsepapi_2.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmkuservsepapi_2.0.0.0 >
< vmware-esx-tools-light (4.1.0-3.26.800380) >

Custom driver modules
– Download or extract the .o files from the offline bundles/vib files. Luckily I could capture them from the HP VMware OEM CD-ROM.
– The files we need are tg3.o and hpsa.o.
– Go back to your temp directory where your CD image is mounted and mount imagedd

mount imagedd /mnt -o loop,rw,offset=4210688
cd /mnt

total 91368
-rwxr-xr-x 1 root root 11225 Nov 6 22:21 a.z
-rwxr-xr-x 1 root root 128 Nov 6 22:21 boot.cfg
-rwxr-xr-x 1 root root 48339 Nov 6 22:21 b.z
-rwxr-xr-x 1 root root 1162824 Nov 6 22:21 cimstg.tgz
-rwxr-xr-x 1 root root 15283377 Nov 6 22:21 c.z
-rwxr-xr-x 1 root root 2489796 Nov 6 22:21 k.z
-rwxr-xr-x 1 root root 137 Nov 6 22:21 license.tgz
-rwxr-xr-x 1 root root 754246 Nov 6 22:21 m.z
-rwxr-xr-x 1 root root 950084 Nov 6 22:21 oem.tgz
-rwxr-xr-x 1 root root 3862 Nov 6 22:21 pkgdb.tgz
-rwxr-xr-x 1 root root 72786024 Nov 6 22:21 s.z
-rwxr-xr-x 1 root root 41155 Nov 6 22:21 tboot.gz
-rwxr-xr-x 1 root root 2030 Nov 6 22:21 vibddi

– Extract the oem.tgz file and copy the simple.map, pci.ids and .o files in the right directories.
tar xvzf oem.tgz

1) /usr/lib/vmware/vmkmod
Kopieer in deze map de .o files
2) /etc/vmware/
Kopieer naar deze map de sample.map en pci.ids bestanden

– Recreate oem.tgz

tar cvzf oem.tgz ./*

– Copy the oem.tgz back to imagedd

cp oem.tgz /mnt

– Unmount the imagedd

umount /mnt/imaged

– Recompress imagedd file

bzip2 /tmp/imagedd/imagedd

– Check the MD5 checksum and replace it with the checksum in the file : imagedd.md5

md5sum imaged.bz2

Create new ISO

cd /tmp/iso

mkisofs -o VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64_customdriver.iso -b isolinux.bin -c BOOT.CAT-no-emul-boot -boot-load-size 4 -boot-info-table ./

You now have an image that is ready to install with, when you use a manual install.

PXE Boot
As we use PXE here, I added the image, rebooted and….FAIL!

WTH! Oh right…I forgot that the boot image needs the correct drives too, else the NIC can’t connect to the share..

To simply add the drivers to the PXE image I created another OEM.tgz file with the drivers needed for the PXE boot.

– I created a new folder /oem
– I then created the following folders in /oem
1) /usr/lib/vmware/vmkmod
2) /etc/vmware

In the first folder we put the .o files (drivers) and in the second folder the simple.map and pci.ids
Create a a new oem.tgz file to send with the PXE installation.

cd /tmp/oem/
tar –zcf /tmp/oem/oem.tgz etc lib sbin usr var

Copy the oem.tgz file to the location where the PXE files are loaded. In our case this is


Now add the –oem.tgz to your kernel options command line and you’re ready to fire up !

Good luck and have fun!