ESXi5.1 KS.Conf for HP DL380p Gen8

Because I noticed some searches on ESXi5.1, Gen8 and KS.conf I have an example of what I use. I will try to do a bit more explanation later

I didn’t comment all the actions, but some are pretty clear.


# +-----------------------------------+
# | Begin default ESXi 5.1 install    |
# +-----------------------------------+
# VMWare License options accepting EULA
vmaccepteula
# Partitioning
install --firstdisk=usb-storage,hpsa,local --overwritevmfs --novmfsondisk
# root Password
rootpw --iscrypted <password>

# Network install type
network --device=vmnic0 --bootproto=static --ip=[HOSTIP] --netmask=<subnet> --gateway=<gateway> --nameserver=<nameserver> --hostname=[HOSTNAME] --addvmportgroup=0
%post --interpreter=busybox
Echo Installing ESXi5.1
#Reboot after copying image to disk
reboot
%firstboot --interpreter=busybox
# +---------------------------------------------------------------------------+
# | Creating management network                                     |
# +---------------------------------------------------------------------------+
# Remove vSwitch0
sleep 30
esxcli network ip interface remove -i vmk0
esxcli network vswitch standard portgroup remove -p 'Management Network' -v vSwitch0
esxcli network vswitch standard remove -v vSwitch0
# Create management switch
esxcli network vswitch standard add -v vsw-management
# Add nics to management switch
esxcli network vswitch standard uplink add -u vmnic0 -v vsw-management
esxcli network vswitch standard uplink add -u vmnic7 -v vsw-management
# Add portgroups to management switch
esxcli network vswitch standard portgroup add -p 'Management Network' -v vsw-management
# Configure vmkNIC
esxcli network ip interface add -i vmk0 -p 'Management Network'
# Set IP Settings [HOSTIP] is dynamic resolved from the template
esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=[HOSTIP] -N <subnet> -t static
# Set default gateway
esxcfg-route -a default <gateway>
# Put management nics to active
esxcli network vswitch standard policy failover set --active-uplinks vmnic0,vmnic7 --vswitch-name vsw-management
# Setting the Network NIC Speed
esxcli network nic set -n vmnic0 -S 1000 -D full
esxcli network nic set -n vmnic1 -S 1000 -D full
esxcli network nic set -n vmnic2 -S 1000 -D full
esxcli network nic set -n vmnic3 -S 1000 -D full
esxcli network nic set -n vmnic4 -S 1000 -D full
esxcli network nic set -n vmnic5 -S 1000 -D full
esxcli network nic set -n vmnic6 -S 1000 -D full
esxcli network nic set -n vmnic7 -S 1000 -D full
esxcli network nic set -n vmnic8 -S 1000 -D full
esxcli network nic set -n vmnic9 -S 1000 -D full
echo Create VMotion netwerk
# +---------------------------------------------------------------------+
# | Creating vMotion Netwerk                                        |
# +---------------------------------------------------------------------+
# Create vMotion vSwitch
esxcli network vswitch standard add -v vsw-vmotion
# Add nics to vsw-vmotion
esxcli network vswitch standard uplink add -u vmnic2 -v vsw-vmotion
esxcli network vswitch standard uplink add -u vmnic5 -v vsw-vmotion
# Add portgroups to vsw-vmotion
esxcli network vswitch standard portgroup add -p 'vmotion' -v vsw-vmotion
# Configure vmkNIC
esxcli network ip interface add -i vmk1 -p 'vmotion'
esxcli network ip interface ipv4 set --interface-name=vmk1 --ipv4=[VMOTIONIP] --netmask=<subnet> --type=static
# Put management nics to active
esxcli network vswitch standard policy failover set --active-uplinks vmnic2,vmnic5 --vswitch-name vsw-vmotion

echo Create Virtual Machine netwerk
# +-------------------------------------------------------------------+
# | Create alle Virtual Machine VLANs                               |
# +-------------------------------------------------------------------+
# Create 256 port switch instead of default 64
esxcfg-vswitch -a vsw-vms01:256
# Add nics to vsw-vms01
esxcfg-vswitch -L vmnic1 vsw-vms01
esxcfg-vswitch -L vmnic3 vsw-vms01
esxcfg-vswitch -L vmnic4 vsw-vms01
esxcfg-vswitch -L vmnic6 vsw-vms01
# Add portgroups.
# Download vsw-vms01-<esxcl>.sh script for the portgroup settings
wget http://<httpserver>/esx51rep/scripts/vsw-vms01-[CLUSTER].sh -O /tmp/vsw-vms01-[CLUSTER].sh
chmod a+x /tmp/vsw-vms01-[CLUSTER].sh
/tmp/vsw-vms01-[CLUSTER].sh > /tmp/vsw-vms01-[CLUSTER].log 2>&1
# Set DNS and hostname
esxcli system hostname set --fqdn=[HOSTNAME]
esxcli network ip dns server add --server=<nameserver>
esxcli network ip dns server add --server=145.70.12.203
#echo add DNS configuration
echo search domain.corp.net  > /etc/resolv.conf
echo nameserver <nameserver>  >> /etc/resolv.conf
echo nameserver <nameserver> >> /etc/resolv.conf

echo Configure NTP
# +--------------------------------------------------------------------+
# | Add NTP Settings                                                   |
# +--------------------------------------------------------------------+
# Backup
mv /etc/ntp.conf /etc/ntp.conf.bak
# ntp.conf creation
cat > /etc/ntp.conf << __NTP_CONFIG__
restrict default kod nomodify notrap noquerynopeer
restrict 127.0.0.1
server <NTP Server>
__NTP_CONFIG__
/sbin/chkconfig --level 345 ntpd on
echo "driftfile /etc/ntp.drift" >> /etc/ntp.conf

echo Configure Syslog
# +--------------------------------------------------------------------+
# | Add syslog confiuration to ESX host                                   |
# +--------------------------------------------------------------------+
vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string <Syslog Server>
#Disable MOB
vim-cmd proxysvc/remove_service "/mob" "httpsWithRedirect"
# +--------------------------------------------------------------------+
# | SNMP Trap                                                            |
# +--------------------------------------------------------------------+
#echo "<config><snmpSettings><enable>true</enable><communities>patrol</communities><port>161</port><targets>snmp1.fqdn@162 patrol;snmp2.fqdn@162 patrol</targets></snmpSettings></config>" > /etc/vmware/snmp.xml

echo Rename local datastore
# +---------------------------------------------------------------------------+
# | Rename local datastore if --novmfsondisk is not used                      |
# +---------------------------------------------------------------------------+
vim-cmd hostsvc/datastore/rename datastore1 "[SUBTEMPLATE]-local"

# +---------------------------------------------------------------------------+
# |Create SSH Banner                                                          |
# +---------------------------------------------------------------------------+
/bin/cat > /etc/banner.new <<SSHEOF
${INDENTATION:-}<Company>
${INDENTATION:-}ESXi 5.1
${INDENTATION:-}=========================================
${INDENTATION:-}WARNING: UNAUTHORIZED USE IS PROHIBITED
${INDENTATION:-}-----------------------------------------
${INDENTATION:-}Property of <Company> Groep, and should only
${INDENTATION:-}be accessed by authorized <Company> employees.
${INDENTATION:-}Do not attempt to login unless you are an
${INDENTATION:-}authorized user.

${INDENTATION:-}Any authorized or unauthorized access and use,
${INDENTATION:-}will be monitored and anyone using this system
${INDENTATION:-}expressly consents to such monitoring. If such
${INDENTATION:-}monitoring reveals possible envidence of criminal
${INDENTATION:-}activity, such evidence will be provided to law
${INDENTATION:-}enforcement personnel and can result in criminal
${INDENTATION:-}or civil prosecution under applicable law of
${INDENTATION:-}<country>.

${INDENTATION:-}This system is restricted to authorized users.
${INDENTATION:-}Individuals attempting unauthorized access
${INDENTATION:-}will be prosecuted. If unauthorized,
${INDENTATION:-}terminate access now!
${INDENTATION:-}By logging in you indicate your acceptance of
${INDENTATION:-}the information above.
${INDENTATION:-}-----------------------------------------
SSHEOF
# copy new banner file to overwrite /etc/issue (esxi 5 store it's banner file here)
cp /etc/banner.new /etc/issue

# +---------------------------------------------------------------------------+
# | enable VMotion                                          |
# +---------------------------------------------------------------------------+
vim-cmd hostsvc/vmotion/vnic_set vmk1
vim-cmd internalsvc/refresh_network

# Set Multipatch Policies
for i in `esxcli storage nmp device list | grep -E naa.\{33\}$` ; do
esxcli storage nmp device set --device $i --psp "VMW_PSP_RR";
esxcli storage nmp psp roundrobin deviceconfig set --device $i --type "iops" --iops=1;
done
esxcli storage nmp satp set --default-psp "VMW_PSP_RR" --satp "VMW_SATP_SVC"

# backup ESXi configuration to persist changes
/sbin/auto-backup.sh

#enter maintenance mode
esxcli system maintenanceMode set -e true

#Removing iSCSI driver and CD-ROM driver
esxcli software vib remove -n scsi-bnx2i
esxcli software vib remove -n ata-pata-amd

#Removing iSCSI initiator
esxcli iscsi software set -e false

# Needed for configuration changes that could not be performed in esxcli
esxcli system shutdown reboot -d 60 -r "Rebooting after host configurations"

VM import Failed to open disk scsi0:0

VM import Failed to open disk scsi0:0: Unsupported and/or invalid disk type 7. Did you forget to import the disk first?Unable to create virtual SCSI device for scsi0:0, Module DevicePowerOn power on failed.

I noticed this issue after we tried to import a VM created in VMware View in our ESXi 5.1 environment. When we tried to power on the machine the error message below appeared. This has something to do with the disk format.

Error_Disk

In this example a disk of 100GB was created for the VM in VMware View. After enabling SSH on the ESXi host and looking on the datastore you’ll notice a vmdk of 27 GB.

/vmfs/volumes/4dc3d515-66b8d17d-49db-00237d54cab0/Windows 7 x64_1 # ls –l
<b>-rw-------    1 root     root     28533391360 May  8 12:58 Windows 7 x64_1-0-s001.vmdk</b>
-rw-------    1 root     root           537 May  8 13:56 Windows 7 x64_1-0.vmdk
-rw-------    1 root     root          8684 May  8 12:58 Windows 7 x64_1.nvram
-rw-------    1 root     root             0 May  8 12:58 Windows 7 x64_1.vmsd
-rw-------    1 root     root          2842 May 13 07:19 Windows 7 x64_1.vmx
-rw-------    1 root     root          3049 May 13 07:18 Windows 7 x64_1.vmxf
-rw-r--r--    1 root     root         42582 May 13 07:19 vmware.log
-rw-------    1 root     root         25730 May  8 12:00 vprintproxy.log

To resolve this, we need to re-import the disk with the right format (Zeroed Thick).
To convert the disk to zerothicked, use vmkfstools. See the example below.

/Windows 7 x64_I-work1 # <i>vmkfstools -i  "Windows 7 x64_I-work1-0.vmdk" -d zeroedthick "Windows 7 x64_I-work1.vmdk"

 

With this command we make a clone of the existing disk (Windows 7 x64_1-0.vmdk) to a new disk (Windows 7 x64_1.vmdk) with the proper diskformat (-d –diskformat [zeroedthick|thin|eagerzeroedthick]).

After the cloning is completed, you could re-import the newly created disk in your VM and start it. (Don’t forget to remove the old one :))

 

 

Kiwi Syslog Errors

My Kiwi Syslog server kept crashing a lot of times during a weekly period, after applying the changes below, the problem looks resolved.

Issue:

When attempting to start the Kiwi Syslog Server (KSS) service, one of the following error messages displays:

  • Error 1053: The service did not respond to the start or control request in a timely fashion.
  • Error 7000: The Kiwi Syslog Server failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.
  • Error 7009: Timeout (30000 milliseconds) waiting for the Kiwi Syslog Server service to connect.
  • Component ‘ipdaems81.ocx’ or one of its dependencies not correctly registered: a file is missing or invalid

http://knowledgebase.solarwinds.com/kb/questions/4386/Kiwi+Syslog+Server+Service+Startup+Failure+in+Versions+9.3.3+and+9.3.4

 

CPU affinity or not!

For our Citrix environment there is an existing design to use 24 vCPU on a 16 core Hyperthreaded ESX host which presents 32 logical CPU’s. This should work properly in my opinion.
But as our project team comes across some performance issues, they searched around a bit and came with the following proposal.
6 VM’s – 4 vCPU’s each and set affinity to 0-3,4-7 etc.

I already asked them to do some proper research and testing about this but they lacked a bit on this, so I tested this myself. As I found on different sites, they almost all mention : DON’T USE IT! Unless you really know what to do. As far as I can see, they found some best practice guides and decided to follow them blindly!

With a little test I could prove them they need to put at least 1 extra CPU in the affinity rule for some hypervisor stuff. Or just don’t use it.

See the below results when I turned it on and off, you immediately see an increase on  Ready times for the machine I turned it on for, and a decrease on the machine I added another affinity CPU.

VM1 used no affinity before and was default, around 09:30 I put the CPU affinity to 0-3; as you can see an immediately increase in ready times. Removed the CPU affinity at 14:00 the same day, as you can see the graph drops instant.

VM2 was set to affinity 16-19, I added another CPU to the affinity as seen in some best practice resources. After submitting the changes around 09:25 you can see the ready times drop to a lower level.

There is a little peak at 10:05 because there were a lot of VM’s started on that particular ESXi host, but you can see the clear difference after that time, as soon as I had enough logging I turned everything back to the original settings. You can see this at 13:55-14:00. Also noticeable is the little higher average after the settings are returned as before, but as I mentioned a little earlier this could be explained because there was more load on the ESXi host where the VM’s were residing.

Graph 1 (Long day)

Day_CPU_Ready
Graph 2 (Today)

Day_CPU_Ready2

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
http://frankdenneman.nl/2011/01/11/beating-a-dead-horse-using-cpu-affinity/
http://www.yellow-bricks.com/2009/04/28/cpu-affinity/

ESXi5.1 U1 resolves CIM issues

A while ago when we rolled out ESXi5.1 on our environment I noticed a lot of strange CIM errors coming trough in our Kiwi Syslog server. I tried to investigate where this was coming from, but couldn’t find a proper solution.

CIM

Finally when I was reading the release notes of ESXi5.1U1 I noticed there were some things concerning CIM and API issues.
http://www.vmware.com/support/vsphere5/doc/vsphere-esxi-51u1-release-notes.html#resolvedissuescimapi

We already came accross the issue with the full ram disk trough the SNMP trap settings. After installing U1 on some test ESXi hosts I noticed the strange CIM errors dissappear. After a week, I didn’t see any coming back while they normally occured in a 4 hours time frame.

So if you might have similar problems, try U1 as it solves some of these issues.

Resetting the inventory service database

When a colleague cleaned some log files which filled up the C: drive; we noticed that this caused the Inventory Service to die. Some of the vCenter user where also complaining that the “Search” function didn’t work anymore.

The location of the log files was: “C:\Program Files\VMware\Infrastructure\Inventory Service\data”

This caused the “inventory service” to get killed and remain unstartable. Because we had no backup of the database we needed to reset it. See the link below for the proper VMware article. In short here this is my summary.

Caution

This procedure can cause data loss. Perform this procedure with VMware Technical Support.

I took the guess and did it without VMware support because it wasn’t working anyway 🙂

  1. Stop the Inventory Service
  2. Remove folder: “C:\Program Files\VMware\Infrastructure\Inventory Service\data”
  3. Go to “c:\Program Files\VMware\Infrastructure\Inventory Service\scripts”
  4. “Createdb.bat” with no arguments
  5. Start “Inventory Service”
  6. Reregister the “inventory service”
  7. “cd C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool”
    “register-is.bat <vcenterFQDN>:443/sdk <vcenterFQDN>:10443 <vcenterFQDN>:7444/lookupservice/sdk”
  8. Restart the “Inventory Service”
com.vmware.vim.vcauthorization.impl.provider.InMemoryAuthCache] Processing unresolved entities 06F31E7F-0667-4628-A5E2-FAB1AE6B9D61
com.vmware.vim.query.server.provider.AbstractAtomPullProviderBase] Provider 06F31E7F-0667-4628-A5E2-FAB1AE6B9D61 new generation: 2102329
com.vmware.vim.vcauthorization.impl.provider.InMemoryAuthCache] Done processing feed update for 06F31E7F-0667-4628-A5E2-FAB1AE6B9D61
com.vmware.vim.vcauthorization.impl.provider.InMemoryAuthCache] Processing unresolved entities 06F31E7F-0667-4628-A5E2-FAB1AE6B9D61
com.vmware.vim.query.server.provider.AbstractAtomPullProviderBase] Provider 06F31E7F-0667-4628-A5E2-FAB1AE6B9D61 new generation: 2102330
com.vmware.vim.dataservices.federation.FederationReconfigurator] Checking/updating federation configuration
com.vmware.vim.dataservices.federation.FederationReconfigurator] No peers reachable - skipping reconfiguration

The search function will now work again 🙂

net stop vimQueryService
del C:\Program Files\VMware\Infrastructure\Inventory Service\data
C:\Program Files\VMware\Infrastructure\Inventory Service\scripts\createDB.bat
net start vimQueryService
cd C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool
register-is.bat vcenter.fqdn:443/sdk https://vcenter.fqdn:10443 <a href="https://wsv10372.office01.internalcorp.net:7444/lookupservice/sdk">https://vcenter.fqdn:7444/lookupservice/sdk</a>
net stop vimQueryService
net start vimQueryService

http:\\kb.vmware.com\selfservice\microsites\search.do?language=en_US&cmd=displayKC&externalId=2017370