ESXi5.1 KS.Conf for HP DL380p Gen8

Because I noticed some searches on ESXi5.1, Gen8 and KS.conf I have an example of what I use. I will try to do a bit more explanation later

I didn’t comment all the actions, but some are pretty clear.


# +-----------------------------------+
# | Begin default ESXi 5.1 install    |
# +-----------------------------------+
# VMWare License options accepting EULA
vmaccepteula
# Partitioning
install --firstdisk=usb-storage,hpsa,local --overwritevmfs --novmfsondisk
# root Password
rootpw --iscrypted <password>

# Network install type
network --device=vmnic0 --bootproto=static --ip=[HOSTIP] --netmask=<subnet> --gateway=<gateway> --nameserver=<nameserver> --hostname=[HOSTNAME] --addvmportgroup=0
%post --interpreter=busybox
Echo Installing ESXi5.1
#Reboot after copying image to disk
reboot
%firstboot --interpreter=busybox
# +---------------------------------------------------------------------------+
# | Creating management network                                     |
# +---------------------------------------------------------------------------+
# Remove vSwitch0
sleep 30
esxcli network ip interface remove -i vmk0
esxcli network vswitch standard portgroup remove -p 'Management Network' -v vSwitch0
esxcli network vswitch standard remove -v vSwitch0
# Create management switch
esxcli network vswitch standard add -v vsw-management
# Add nics to management switch
esxcli network vswitch standard uplink add -u vmnic0 -v vsw-management
esxcli network vswitch standard uplink add -u vmnic7 -v vsw-management
# Add portgroups to management switch
esxcli network vswitch standard portgroup add -p 'Management Network' -v vsw-management
# Configure vmkNIC
esxcli network ip interface add -i vmk0 -p 'Management Network'
# Set IP Settings [HOSTIP] is dynamic resolved from the template
esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=[HOSTIP] -N <subnet> -t static
# Set default gateway
esxcfg-route -a default <gateway>
# Put management nics to active
esxcli network vswitch standard policy failover set --active-uplinks vmnic0,vmnic7 --vswitch-name vsw-management
# Setting the Network NIC Speed
esxcli network nic set -n vmnic0 -S 1000 -D full
esxcli network nic set -n vmnic1 -S 1000 -D full
esxcli network nic set -n vmnic2 -S 1000 -D full
esxcli network nic set -n vmnic3 -S 1000 -D full
esxcli network nic set -n vmnic4 -S 1000 -D full
esxcli network nic set -n vmnic5 -S 1000 -D full
esxcli network nic set -n vmnic6 -S 1000 -D full
esxcli network nic set -n vmnic7 -S 1000 -D full
esxcli network nic set -n vmnic8 -S 1000 -D full
esxcli network nic set -n vmnic9 -S 1000 -D full
echo Create VMotion netwerk
# +---------------------------------------------------------------------+
# | Creating vMotion Netwerk                                        |
# +---------------------------------------------------------------------+
# Create vMotion vSwitch
esxcli network vswitch standard add -v vsw-vmotion
# Add nics to vsw-vmotion
esxcli network vswitch standard uplink add -u vmnic2 -v vsw-vmotion
esxcli network vswitch standard uplink add -u vmnic5 -v vsw-vmotion
# Add portgroups to vsw-vmotion
esxcli network vswitch standard portgroup add -p 'vmotion' -v vsw-vmotion
# Configure vmkNIC
esxcli network ip interface add -i vmk1 -p 'vmotion'
esxcli network ip interface ipv4 set --interface-name=vmk1 --ipv4=[VMOTIONIP] --netmask=<subnet> --type=static
# Put management nics to active
esxcli network vswitch standard policy failover set --active-uplinks vmnic2,vmnic5 --vswitch-name vsw-vmotion

echo Create Virtual Machine netwerk
# +-------------------------------------------------------------------+
# | Create alle Virtual Machine VLANs                               |
# +-------------------------------------------------------------------+
# Create 256 port switch instead of default 64
esxcfg-vswitch -a vsw-vms01:256
# Add nics to vsw-vms01
esxcfg-vswitch -L vmnic1 vsw-vms01
esxcfg-vswitch -L vmnic3 vsw-vms01
esxcfg-vswitch -L vmnic4 vsw-vms01
esxcfg-vswitch -L vmnic6 vsw-vms01
# Add portgroups.
# Download vsw-vms01-<esxcl>.sh script for the portgroup settings
wget http://<httpserver>/esx51rep/scripts/vsw-vms01-[CLUSTER].sh -O /tmp/vsw-vms01-[CLUSTER].sh
chmod a+x /tmp/vsw-vms01-[CLUSTER].sh
/tmp/vsw-vms01-[CLUSTER].sh > /tmp/vsw-vms01-[CLUSTER].log 2>&1
# Set DNS and hostname
esxcli system hostname set --fqdn=[HOSTNAME]
esxcli network ip dns server add --server=<nameserver>
esxcli network ip dns server add --server=145.70.12.203
#echo add DNS configuration
echo search domain.corp.net  > /etc/resolv.conf
echo nameserver <nameserver>  >> /etc/resolv.conf
echo nameserver <nameserver> >> /etc/resolv.conf

echo Configure NTP
# +--------------------------------------------------------------------+
# | Add NTP Settings                                                   |
# +--------------------------------------------------------------------+
# Backup
mv /etc/ntp.conf /etc/ntp.conf.bak
# ntp.conf creation
cat > /etc/ntp.conf << __NTP_CONFIG__
restrict default kod nomodify notrap noquerynopeer
restrict 127.0.0.1
server <NTP Server>
__NTP_CONFIG__
/sbin/chkconfig --level 345 ntpd on
echo "driftfile /etc/ntp.drift" >> /etc/ntp.conf

echo Configure Syslog
# +--------------------------------------------------------------------+
# | Add syslog confiuration to ESX host                                   |
# +--------------------------------------------------------------------+
vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string <Syslog Server>
#Disable MOB
vim-cmd proxysvc/remove_service "/mob" "httpsWithRedirect"
# +--------------------------------------------------------------------+
# | SNMP Trap                                                            |
# +--------------------------------------------------------------------+
#echo "<config><snmpSettings><enable>true</enable><communities>patrol</communities><port>161</port><targets>snmp1.fqdn@162 patrol;snmp2.fqdn@162 patrol</targets></snmpSettings></config>" > /etc/vmware/snmp.xml

echo Rename local datastore
# +---------------------------------------------------------------------------+
# | Rename local datastore if --novmfsondisk is not used                      |
# +---------------------------------------------------------------------------+
vim-cmd hostsvc/datastore/rename datastore1 "[SUBTEMPLATE]-local"

# +---------------------------------------------------------------------------+
# |Create SSH Banner                                                          |
# +---------------------------------------------------------------------------+
/bin/cat > /etc/banner.new <<SSHEOF
${INDENTATION:-}<Company>
${INDENTATION:-}ESXi 5.1
${INDENTATION:-}=========================================
${INDENTATION:-}WARNING: UNAUTHORIZED USE IS PROHIBITED
${INDENTATION:-}-----------------------------------------
${INDENTATION:-}Property of <Company> Groep, and should only
${INDENTATION:-}be accessed by authorized <Company> employees.
${INDENTATION:-}Do not attempt to login unless you are an
${INDENTATION:-}authorized user.

${INDENTATION:-}Any authorized or unauthorized access and use,
${INDENTATION:-}will be monitored and anyone using this system
${INDENTATION:-}expressly consents to such monitoring. If such
${INDENTATION:-}monitoring reveals possible envidence of criminal
${INDENTATION:-}activity, such evidence will be provided to law
${INDENTATION:-}enforcement personnel and can result in criminal
${INDENTATION:-}or civil prosecution under applicable law of
${INDENTATION:-}<country>.

${INDENTATION:-}This system is restricted to authorized users.
${INDENTATION:-}Individuals attempting unauthorized access
${INDENTATION:-}will be prosecuted. If unauthorized,
${INDENTATION:-}terminate access now!
${INDENTATION:-}By logging in you indicate your acceptance of
${INDENTATION:-}the information above.
${INDENTATION:-}-----------------------------------------
SSHEOF
# copy new banner file to overwrite /etc/issue (esxi 5 store it's banner file here)
cp /etc/banner.new /etc/issue

# +---------------------------------------------------------------------------+
# | enable VMotion                                          |
# +---------------------------------------------------------------------------+
vim-cmd hostsvc/vmotion/vnic_set vmk1
vim-cmd internalsvc/refresh_network

# Set Multipatch Policies
for i in `esxcli storage nmp device list | grep -E naa.\{33\}$` ; do
esxcli storage nmp device set --device $i --psp "VMW_PSP_RR";
esxcli storage nmp psp roundrobin deviceconfig set --device $i --type "iops" --iops=1;
done
esxcli storage nmp satp set --default-psp "VMW_PSP_RR" --satp "VMW_SATP_SVC"

# backup ESXi configuration to persist changes
/sbin/auto-backup.sh

#enter maintenance mode
esxcli system maintenanceMode set -e true

#Removing iSCSI driver and CD-ROM driver
esxcli software vib remove -n scsi-bnx2i
esxcli software vib remove -n ata-pata-amd

#Removing iSCSI initiator
esxcli iscsi software set -e false

# Needed for configuration changes that could not be performed in esxcli
esxcli system shutdown reboot -d 60 -r "Rebooting after host configurations"

DL380/385 Gen8 Install ESXi4.1

Case: Missing NIC and storage drivers in our ESXi4.1 installation

First thing I noticed when the I tried to install ESXi4.1 on the Gen8 series is that our standard (VMware) image wasn’t able to install. Not by our scripted UDA and also manual failed.
With the scripted install I noticed the NIC drivers where not recognized and the hardware was unidentified which resulted in the NIC not connecting to our deployment server to grab the KS.CFG files.
The manual install failed by a missing Storage Adapter driver.

When I used the ESXi4.1 recommended image by HP with all the drivers inserted and loaded, the installation went fine. After the installation I used

esxcfg-scsidevs –l
esxcfg-nics –l

To see what hardware was exactly in the machine.

NIC: Broadcom NetXtreme BCM5719
Storage Array: HP System Array P420I
With this information I found the drivers on the HP site.
Driver: vmware-esx-drivers-scsi-hpsa-400.4.1.0-26OEM.x86_64
Driver : vmware-esx-drivers-net-tg3-400.3.123b.v40.1-1vmw.2.17.00000.x86_64

So ? How do we add this to our PXE deployment?
ESXi4.1 has a sort of local hardware database file with all components and vendors which are supported, and a list of which hardware uses what drivers.
Located in /etc/vmware the files are PCI.IDS and SIMPLE.MAP

PCI.IDS
In this file the hardware database is located, it’s a flat file. If the device ID is recognized during boot, ESX will give the corresponding name to it and it’s later used to load the correct drivers. Luckily I had 2 systems. One system I installed with the HP OEM VMware image, the other was my texting machine. I simply pulled the PCI.IDS file from the HP OEM server and saved it locally for later use.
In our case we are missing the HP Smart Array P420i

Example of PCI.IDS

103c Hewlett-Packard Company
323b Smart Array P222
103c 3350 Smart Array P222
103c 3351 Smart Array P420
103c 3352 Smart Array P421
103c 3353 Smart Array P822
103c 3354 Smart Array P420i
103c 3355 Smart Array P220i
103c 3356 Smart Array P721m

SIMPLE.MAP
This file represents a list of device ID’s and which driver modules they need to load. I also needed to add the correct drivers and modules here. Also in this case I just copied the simple.map file.
Example:
103c:323b 103c:3354 storage hpsa
The first explains the PCI device ID, the second category (storage) the last module is the driver that is loaded (hpsa).

So wrap up. Now the hardware devices can be recognized and the right drivers can be loaded. Too bad we still have older drivers in our image and need to update them first.

Logging in to the HP CD installed servers I figured out which driver versions are needed.

vmkload_mod -l|egrep "tg3|hpsa"
tg3 4 176
hpsa 2 76
~ # vmkload_mod -s hpsa|grep Version
Version: Version 4.1.0-26OEM, Build: 00000, Interface: ddi_9_1 Built on: Jan 19 2012
~ # vmkload_mod -s tg3|grep Version
Version: Version 3.123b.v40.1, Build: 00000, Interface: 9.0, Built on: Apr 3 2012

So now we now which drivers and versions are used to control the hardware.
Creating Custom Image

To inject drivers we can use the VMWare provided vibddi tool on a Linux/CentOS machine.
– Install vibddi

rpm -ivh vmware-esx-vibddi-.i386.rpm

– Installation instructions

Copy image file and extract
– Copy the original ISO using WINSCP to /opt/vmware/vibddi/iso

– Mount the original image:

Mount /var/public/smbmount/esx41iso/VMware-VMvisor-Installer-4.1.0.Update3-custom.iso /var/public/www/esx41i/ESXi41U3Cust –t= iso9660 (rw,loop=/dev/loop1)

– Copy the original imagedd.bz2 to a temporary folder /imagebuild/

cp /var/public/www/esx41i/ESXi41U3Cust/imagedd.bz2 /imagebuild[text]
[text]
total 405929
-r-xr-xr-x 1 root root 11225 Nov 6 22:36 a.z
-r-xr-xr-x 1 root root 2048 Nov 6 22:40 boot000.cat
-r-xr-xr-x 1 root root 2048 Nov 7 15:24 boot.cat
-r-xr-xr-x 1 root root 1162824 Nov 6 22:36 cimstg.tgz
-r-xr-xr-x 1 root root 15283377 Nov 6 22:36 cim.vgz
-r-xr-xr-x 1 root root 7306598 Nov 6 22:36 ienviron.vgz
-r-xr-xr-x 1 root root 312820483 Nov 7 15:14 imagedd.bz2
-r-xr-xr-x 1 root root 67 Nov 7 15:16 imagedd.md5
-r-xr-xr-x 1 root root 971501 Nov 6 22:40 install.vgz
-r-xr-xr-x 1 root root 12241 Nov 6 22:36 isolinux.bin
-r-xr-xr-x 1 root root 298 Nov 6 22:36 isolinux.cfg
-r-xr-xr-x 1 root root 47404 Nov 6 22:36 mboot.c32
-r-xr-xr-x 1 root root 51020 Nov 6 22:36 menu.c32
-r-xr-xr-x 1 root root 2593205 Nov 6 22:36 open_sou.txt
-r-xr-xr-x 1 root root 2156 Nov 6 22:36 readme.txt
-r-xr-xr-x 1 root root 72786024 Nov 6 22:36 sys.vgz
-r-xr-xr-x 1 root root 41155 Nov 6 22:36 tboot.gz
-r-xr-xr-x 1 root root 48339 Nov 6 22:36 vmkboot.gz
-r-xr-xr-x 1 root root 2489796 Nov 6 22:36 vmkernel.gz
-r-xr-xr-x 1 root root 34816 Nov 7 15:23 vmware_v.iso

– Now unzip the imagefile you copied

cd /imagebuild
bunzip2 imagedd.bz2

Adding VIB’s / offline bundles to the image
– Use VIBDDI to see which items are already in the image.

Vibddi –i imagedd –q

– Example:

< oem-vmware-esx-drivers-net-vxge (400.2.0.28.21239-1OEM) >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400.2.26.08.036vm40-1OEM) >
< vmware-esx-firmware (4.1.0-3.26.800380) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-3-26 >
< vmkepsecapi_1.0.0.0 >
< vmkvsepapi_2.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmkuservsepapi_2.0.0.0 >
< vmware-esx-tools-light (4.1.0-3.26.800380) >

– Now we are going to inject the correct drivers , I downloaded and copied them to /bundles

vibddi -i /imagebuild/imagedd -v cross_oem-vmware-esx-drivers-scsi-hpvsa_400.4.1.0-22OEM.vib
Setting up configuration for esxupdate
This may take a few seconds…

– Repeat this for the other drivers. When you finished use the -q switch again to see if the correct drivers are injected. Now you see that they’re added.


vibddi -i /imagebuild/imagedd –q
< oem-vmware-esx-drivers-net-vxge (400.2.0.28.21239-1OEM) >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400.2.26.08.036vm40-1OEM) >
< vmware-esx-drivers-net-tg3 (400.3.123b.v40.1-1vmw.2.17.00000) >
< vmware-esx-drivers-scsi-qla2xxx (400.841.k1.42.1-1vmw.2.17.00000) >
< vmware-esx-firmware (4.1.0-3.26.800380) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-3-26 >
< vmkepsecapi_1.0.0.0 >
< vmkvsepapi_2.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmkuservsepapi_2.0.0.0 >
< vmware-esx-tools-light (4.1.0-3.26.800380) >

Custom driver modules
– Download or extract the .o files from the offline bundles/vib files. Luckily I could capture them from the HP VMware OEM CD-ROM.
– The files we need are tg3.o and hpsa.o.
– Go back to your temp directory where your CD image is mounted and mount imagedd

mount imagedd /mnt -o loop,rw,offset=4210688
cd /mnt

total 91368
-rwxr-xr-x 1 root root 11225 Nov 6 22:21 a.z
-rwxr-xr-x 1 root root 128 Nov 6 22:21 boot.cfg
-rwxr-xr-x 1 root root 48339 Nov 6 22:21 b.z
-rwxr-xr-x 1 root root 1162824 Nov 6 22:21 cimstg.tgz
-rwxr-xr-x 1 root root 15283377 Nov 6 22:21 c.z
-rwxr-xr-x 1 root root 2489796 Nov 6 22:21 k.z
-rwxr-xr-x 1 root root 137 Nov 6 22:21 license.tgz
-rwxr-xr-x 1 root root 754246 Nov 6 22:21 m.z
-rwxr-xr-x 1 root root 950084 Nov 6 22:21 oem.tgz
-rwxr-xr-x 1 root root 3862 Nov 6 22:21 pkgdb.tgz
-rwxr-xr-x 1 root root 72786024 Nov 6 22:21 s.z
-rwxr-xr-x 1 root root 41155 Nov 6 22:21 tboot.gz
-rwxr-xr-x 1 root root 2030 Nov 6 22:21 vibddi

– Extract the oem.tgz file and copy the simple.map, pci.ids and .o files in the right directories.
tar xvzf oem.tgz

1) /usr/lib/vmware/vmkmod
Kopieer in deze map de .o files
2) /etc/vmware/
Kopieer naar deze map de sample.map en pci.ids bestanden

– Recreate oem.tgz

tar cvzf oem.tgz ./*

– Copy the oem.tgz back to imagedd

cp oem.tgz /mnt

– Unmount the imagedd

umount /mnt/imaged

– Recompress imagedd file

bzip2 /tmp/imagedd/imagedd

– Check the MD5 checksum and replace it with the checksum in the file : imagedd.md5

md5sum imaged.bz2

Create new ISO

cd /tmp/iso

mkisofs -o VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64_customdriver.iso -b isolinux.bin -c BOOT.CAT-no-emul-boot -boot-load-size 4 -boot-info-table ./

You now have an image that is ready to install with, when you use a manual install.

PXE Boot
As we use PXE here, I added the image, rebooted and….FAIL!

WTH! Oh right…I forgot that the boot image needs the correct drives too, else the NIC can’t connect to the share..

To simply add the drivers to the PXE image I created another OEM.tgz file with the drivers needed for the PXE boot.

– I created a new folder /oem
– I then created the following folders in /oem
1) /usr/lib/vmware/vmkmod
2) /etc/vmware

In the first folder we put the .o files (drivers) and in the second folder the simple.map and pci.ids
Create a a new oem.tgz file to send with the PXE installation.

cd /tmp/oem/
tar –zcf /tmp/oem/oem.tgz etc lib sbin usr var

Copy the oem.tgz file to the location where the PXE files are loaded. In our case this is

/var/public/tftproot

Now add the –oem.tgz to your kernel options command line and you’re ready to fire up !

Good luck and have fun!