Get-View to speed up things using regex

I was playing around with the get-view cmdlet and found something I didn’t know yet.

On the site there is an example

Get-View -ViewType VirtualMachine -Filter @{"Name" = "VM"}

Cool, let’s test that. Hey , why do I get 2 items returned?
I searched for a VM called “Test” and there returned 2 VM’s which are started with “Test”.

I didn’t know that, in the description of the filter object you see:

Specifies a hash of <name>-<value> pairs, where <name> represents the property value to test, and <value> represents a regex pattern the property must match. If more than one pair is present, all the patterns must match.

Ah…

So let’s throw some regex in then:

One way to tighten our patterns is to define a pattern that describes both the start and the end of the line using the special ^ (hat) and $ (dollar sign) metacharacters. Maybe not the best option but it works fine.

To do an exact match on “Test””we use : “^Test$

Now let’s start exact match on Test and do a reset of the VM:

(get-view -ViewType VirtualMachine -Filter @{"name"="^Test$"}).ResetVM()

Cool that works, I’m no regex expert, but the way mentioned above works, because the start..end of the line only exists of VM names.

It would be nicer to use a word boundary so you capture the word only and not the line.

So we can also use \bServername\b:


get-view -ViewType VirtualMachine -Filter @{"name"="\bTest\b"}

Both options work, but may come in handy for the future.

So what I learned, the filter object can be ‘regexed’ !

AD authentication and Windows Passthrough VCSA appliance

While we are deploying a new vCenter 6.0 environment we planned to make a setup like this:

2 physical sites, on both sites we create a vCenter appliance and a vCenter Platform Service controller. Both the PSC are joined to the same (replication) domain.

Deployment worked perfect, logged in as SSO user, and configured the LDAP settings so we could login with our domain accounts.

Wow easy as hell, but hey a colleague mentions he cannot login while using the checkbox “use windows credentials”. We got an error

Problem:

You see a popup with the error:
Window session credentials cannot be used to log into this server. Enter a user name and password

Troubleshooting:

Well I tried a lot of VMware KB’s like :
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2050701

And also a few others but still with no real result.

Then I found out on a few different sites you need to join the appliances to your Active Directory Domain. That makes sense, searched for some guides and  you can do it by GUI but then only for the PSC’s this didn’t solve the problem.

For testing I started to deployed an embedded VCSA and configured that the way I did with external version. Joined the machine to the domain, tested this and wow that worked.

So somewhere in the communication flow between VC – PSC – AD something will go wrong. The AD connection should be good as it worked flawless with the embedded version (assumption) so the problem should be something between VC and PSC.

I remembered a note on a site to join ALL nodes from your vSphere environment to AD to make it work. But damn, why does the GUI only show the PSC’s ? Makes sense that only the PSC are connected to a domain and do the authentication.

A colleague then tied the knots and a found a command to join AD from command line. Hey I already saw it, but what if we also look on the vCenter server to see if we can join them too.

Wow the command is there.

/opt/likewise/bin/domainjoin-cli

So let’s try again :

  • Re-deploy 2 VC’s and 2 PSC
  • Login with SSH
  • Join PSC’s to domain, join VC’s to domain
  • /opt/likewise/bin/domainjoin-cli join <domain><domain admin user> 
  • Restart all servers
  • Login with SSH
  • Query DC to see if the join was succesfull
    /opt/likewise/bin/domainjoin-cli query
  • Configure SSO
  • Test …….BAM works!

Solution:

Join all nodes (vCenter servers & Platform Service Controllers) to the Active Directory Domain.

/opt/likewise/bin/domainjoin-cli join <domain><domain admin user>

 

Bonus : For troubleshooting I checked a lot of log files, here is a good list of log file locations:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2110014

KDC has no support for encryption type (14)

This morning was like every morning, everything up and running and had good cup of coffee. All VMware components worked fine.

After that I went to a meeting, had some talk with colleagues and came back to my workplace. Mmm a colleague tells me that he can’t login to vCenter anymore, we have 2 seperate vCenters in 2 diffent datacenters. Somehow he couldn’t login in both vCenters. Strange, as they have very few components shared.

First thing I tought of was authentication, logged in with admin@vsphere.local user in webclient and checked SSO settings. When testing the connection I got an error.

The settings in SSO for LDAP are:

  • Reuse session (this was our setting and didn’t work anymore)
  • Password (this works with my own user account so AD should be ok?)
  • Anonymous (Prohibited error)

So somehow, when I use the option password the authentication to AD is fine, but with reuse session I get an error.

Let’s check the SSO logging :

C:\Program Files\VMware\Infrastructure\SSOServer\logs\imsTrace.log

The errors I found where like:

Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: KDC has no support for encryption type (14))]
Caused by: GSSException: No valid credentials provided (Mechanism level: KDC has no support for encryption type (14))

So let’s ask the security team if happened something in the time I had a meeting.

“Uh yes we upgraded the functional domain level from Windows2003 to Windows2008”

*To be sure I rebooted the vCenters in the meanwhile

Ah so we might have a trigger which caused this issue. When searching the internet I find a similar error messages:

http://visualplanet.org/blog/?p=20

http://www.winsysadminblog.com/2013/02/fixing-kdc-authentication-problems-when-upgrading-your-domain-and-forest-functional-level-from-2003-to-2008-r2/

I adviced the security team to restart the Domain Controllers or restart the service. Because the impact of restarting the service is unknown we decided to restart the domain controllers later that evening.

As a workaround I used the SSO “Password” option with a service account to let people login and don’t disturb the backup process.

The day after domain controllers where rebooted, I changed the authentication type back to “Reuse session” and tested the connection….BINGO! Worked again.

The problem was probably like described in the posts, there isn’t much public information on this I could find. Our security team even verified with Microsoft for impact and stuff, but never had this mentioned.

Well took me a few minutes of my day :S

 

HP DL380Gen9 installing on SD Card using UEFI

Intro

Recently we purchased a few HP DL380 GEN9 servers to test our installations for ESXi, Windows and Linux.

Normally we install our ESXi environment on a SD Card, which worked fine on the G7 and Gen8 series.

Now the new Gen9 series start default with UEFI boot, tried the normal install, but huh that failed.

Switched to “Legacy Boot” which worked fine, I still wanted to see if we can get it to work with UEFI so started browsing the internet and I found the following posts which already refer to this issue.

The problem seems to be the PXE boot environment which isn’t UEFI Compatible.

Bootloader / PXE

The problem seems to be the bootloader, originally the “Legacy bootloader” is configured default on many PXE boot environments, so we need to prepare them for the future and support UEFI also.

PXE Bootloader

PXE Bootloader

See the article below which explains how it works and what needs to be done on Windows/RedHat PXE boot environments.

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04565930

VMWare says you cannot use it now according to this article. (It will work when you install and use local disks etc. The only thing that isn’t supported is Networkprovisioning/PXE boot)

https://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc%2FGUID-DEB8086A-306B-4239-BF76-E354679202FC.html

You cannot provision EFI hosts with Auto Deploy unless you switch the EFI system to BIOS compatibility mode.

Network boot of VMware ESXi or provisioning with VMware Auto Deploy requires the legacy BIOS firmware and is not available with UEFI.

This probably will probably be fixed in newer versions of the Auto Deploy environment.

SD Card problem?

There also can be problems when using the SD-CARD on a HP DL380 Gen9 which can be solved by 2 solutions

1) ILO Bug, according to this post there has been a ILO bug which causes the server not detecting the SD-CARD

http://v-strange.de/index.php/hardware/190-installing-vmware-esxi-on-sd-card-on-hp-proliant-gen9

The scope of servers which have this problem could be any HP ProLiant Gen8 or ProLiant Gen9 server with HP Integrated Lights-Out 4 (iLO 4) Firmware Version 1.51 through 2.02 and a Secure Digital (SD) card installed.

This can be solved by an ILO firmware update :

http://h20566.www2.hp.com/hpsc/doc/public/display?docId=c04555714&jumpid=em_alerts_us-us_Feb15_xbu_all_all_2311152_208935_proliantserversstorageoptionsandaccessories_recommended_006_0

If your firmware is up to date you should get to the next step

2) According to this post you should disable USB3.0, after a little search I found the article below which describes how to turn it off.

Use this option is set the USB 3.0 Mode.

  1. From the System Utilities screen, select System ConfigurationBIOS/Platform Configuration (RBSU)System OptionsUSB OptionsUSB 3.0 Mode and press Enter.

  2. Select a setting and press Enter:

    • Auto (default)—USB 3.0-capable devices operate at USB 2.0 speeds in the pre-boot environment and during boot. When a USB 3.0 capable OS USB driver loads, USB 3.0 devices transition to USB 3.0 speeds. This mode is compatible with operating systems that do not support USB 3.0 while still allowing USB 3.0 devices to operate at USB 3.0 speeds with modern operating systems.

    • Enabled—USB 3.0-capable devices operate at USB 3.0 speeds at all times (including the pre-boot environment) when in UEFI Boot Mode. Do not use this mode with operating systems that do not support USB 3.0. When operating in Legacy BIOS Boot Mode, the USB 3.0 ports do not function in the pre-boot environment and are not bootable.

    • Disabled—USB 3.0-capable devices function at USB 2.0 speeds at all times.

  3. Press F10 to save your selection.

What I found remarkable is the fact that you shouldn’t enable it unless you know for sure the OS supports USB3.0

Reference : http://h17007.www1.hp.com/docs/iss/proliant_uefi/s_Accessing_USB_Options_201310290123.html#s_USB_30_mode

I keep this post up to date if I have more information.

 

Everything  I tried so far didn’t let me boot from Network in UEFI mode or doesn’t see the SD-Card even with a manual install.

VCSA API and plugin commands

When I first logged in to the appliance I noticed there is a complete list of API’s and plugins.

The complete list and description can be found here:

Plug-Ins in the vCenter Server Appliance Shell
API Commands in the vCenter Server Appliance Shell

So let’s play a bit :

Command> help pi com.vmware.vimtop
top
Display vSphere processes information.

Wonder if that will display a top view, let’s hit it !

Command> pi com.vmware.vimtop

vimtop

Nice, but it also possible just to hit “vimtop” from command.

What more do we have :

Looks that besides the root account there will be a user

Command> com.vmware.appliance.version1.localaccounts.user.list
Config:
Configuration:
Username: root
Status: enabled
Role: superAdmin
Passwordstatus: valid
Fullname: root
Email:
Configuration:
Username: postgres
Status: disabled
Role:
Passwordstatus: notset
Fullname:
Email:

Mm strange, this doesn’t look like the complete list, when I look to the users from a shell environment I see the users below:

localhost:~ # cut -d: -f1 /etc/passwd
bin
daemon
dhcpd
haldaemon
ldap
mail
man
messagebus
nobody
ntp
polkituser
postfix
root
sshd
stunnel
uuidd
wwwrun
nginx
tcserver
cm
netdumper
vapiEndpoint
postgres
mbcs
eam
deploy
vdcs
vpx-workflow
vsm
vsphere-client
perfcharts
vpostgres

 

Like you probably noticed in the normal mode you only can do some API and plugin calls. But you can switch to shell.

Switching to shell

Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Enable BASH access: "shell.set --enabled True"
* Launch BASH: "shell"

Standard shell access is disabled, this can be seen by using shell.get

Command> shell.get
Config:
Enabled: False
Timeout: 0

Now let’s enable the access :

Command> shell.set --enabled True
Command> shell.get
Config:
Enabled: True
Timeout: 3597

Once we entered shell, we can use some basic linux commands


localhost:/ # df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/sda3                              11G  3.7G  6.6G  36% /
udev                                  4.0G  164K  4.0G   1% /dev
tmpfs                                 4.0G   32K  4.0G   1% /dev/shm
/dev/sda1                             128M   38M   84M  31% /boot
/dev/mapper/core_vg-core               25G  173M   24G   1% /storage/core
/dev/mapper/log_vg-log                9.9G  1.1G  8.3G  12% /storage/log
/dev/mapper/db_vg-db                  9.9G  199M  9.2G   3% /storage/db
/dev/mapper/dblog_vg-dblog            5.0G  171M  4.5G   4% /storage/dblog
/dev/mapper/seat_vg-seat              9.9G  188M  9.2G   2% /storage/seat
/dev/mapper/netdump_vg-netdump       1001M   18M  932M   2% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy  9.9G  151M  9.2G   2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc          5.0G  157M  4.6G   4% /storage/invsvc
localhost:/ # mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/sda1 on /boot type ext3 (rw,noexec,nosuid,nodev,noacl)
/dev/mapper/core_vg-core on /storage/core type ext3 (rw)
/dev/mapper/log_vg-log on /storage/log type ext3 (rw)
/dev/mapper/db_vg-db on /storage/db type ext3 (rw,noatime,nodiratime)
/dev/mapper/dblog_vg-dblog on /storage/dblog type ext3 (rw,noatime,nodiratime)
/dev/mapper/seat_vg-seat on /storage/seat type ext3 (rw,noatime,nodiratime)
/dev/mapper/netdump_vg-netdump on /storage/netdump type ext3 (rw)
/dev/mapper/autodeploy_vg-autodeploy on /storage/autodeploy type ext3 (rw)
/dev/mapper/invsvc_vg-invsvc on /storage/invsvc type ext3 (rw,noatime,nodiratime)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)

localhost:/var/log # tail -f messages.log
2015-04-28T12:51:30.980333+00:00 localhost kernel: [1652125.714038] IPfilter Dropped: IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:22:bd:37:fc:00:08:00 SRC=145.70.12.252 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=61046 PROTO=2
2015-04-28T12:51:42.931041+00:00 localhost su: (to vpostgres) root on none
2015-04-28T12:51:50.860391+00:00 localhost kernel: [1652145.593672] IPfilter Dropped: IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:22:bd:37:fc:00:08:00 SRC=145.70.12.252 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=5260 PROTO=2
2015-04-28T12:51:51.044384+00:00 localhost kernel: [1652145.777641] IPfilter Dropped: IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:22:bd:37:fc:00:08:00 SRC=145.70.12.252 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=5416 PROTO=2
2015-04-28T12:52:15.344457+00:00 localhost su: (to vpostgres) roo

Something else I noticed when enabling shell access the timeout starts counting down. As soon as I keep hitting

Command> shell.get
Config:
Enabled: True
Timeout: 3597
Command> shell.get
Config:
Enabled: True
Timeout: 3596
Command> shell.get
Config:
Enabled: True
Timeout: 3596
Command> shell.get
Config:
Enabled: True
Timeout: 3596
Command> shell.get
Config:
Enabled: True
Timeout: 3595

You see the timeout decrease I guess it’s in seconds so 3600 seconds = 60 minutes = 1 hour.

It’s possible to change the timeout using the shell.set command


Command> shell.set -help
shell.set: error: unrecognized arguments: -help
Command> shell.set --help

Usage:
shell.set [--help/-h] --enabled BOOL --timeout INT
Description:
Set enabled state of BASH, that is, access to BASH from
within the controlled CLI.
Input Arguments:
--enabled BOOL
Enabled can be set to true or false
--timeout INT
The timeout (in seconds) specifies how long you enable the
Shell access.

VMware vCenter Server + PSC Appliance 6.x deployment

Choices ?

Due the expansions of the limitations in the vSphere appliances, it now could be a worth replacing the Windows vCenter servers by appliances. Because I’m reasearching the upgrade to 6.x I need to make a decision to go for Windows or the Appliance. Too bad VMware’s Update manager is still a Windows-only product, but ok, I could live with that.

Because we have two sites I normally go for 2 single vCenter installations with their own databases, SSO, VUM etc. So they will be separate entities, no matter what happens on one of the sites, there is no relation, so the surviving one will keep working.

When digging through the new documentation and install guides there are basically a few major decisions to make. Because it’s not able to change this after deployment you have to think beforehand and
make the right decisions.

Below are a few “Major” choices for your design which I will explain later from my point of view. There are several good blog posts on the internet already.
Choice 1 : Embedded or external Platform Service Controller
Choice 2 : Internal or external Database
Choice 3 : One or more vCenters
Choice 4 : Windows, appliance or both ?
Choice 5 : New install or upgrade

Which are beautifully explained in the install guide:
http://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc%2FGUID-ACCD2814-0F0A-4786-96C0-8C9BB57A4616.html

At the moment I’m not going to explain or discuss the decisions and facts I used to choose for this setup.

Design6As you can see I chose to setup 1 vCenter per site (like I used to do before) and made a local external PSC for each site and connected them in the same “Site”. So basically the PSC’s will replicate with each other across the sites, while the vCenters are local.

What does that PSC do actually? Well it handles vCenter Single Sign On, vSphere License Service, VMware Certifcate Authority. Those components where previously part of the vCenter Installation, by seperating them, it now is also possible to link both PSCs.

Because I join the second PSC to the same site it’s possible to use Enhanced Linked Mode out of the box. The site below will explain what it means. It’s awesome that roles/permissions etc. now can be managed from a single point.
http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-91EF7282-C45A-4E48-ADB0-5A4230A91FF2.html

Migrate Windows vCenter Server to vCenter Appliance?

Yes it’s  possible, haven’t tried it yet, but came accross this fling which let’s you migrate a Windows vCenter server with an external SQL Database to a vCenter Appliance with Embedded vPostgres Database

https://labs.vmware.com/flings/vcs-to-vcva-converter

Design  & Installation

For my testing purposes I decided to deploy 4 appliances, 2x vCenter server and 2x Platform Service Controller.

Instead of using the CD-ROM and follow the instructions (which I did already a few times), I decided to create a simple deployment script. It is possible to deploy the appliances with a pretty simple script and configuration file. Let’s dive in here:

You”ll need the installation CD and mount that to a Windows machine where you start the deployment using the deploy command which is located on the CD-ROM.

Secondly you need a few .json files which hold the configuration for the VCSA deployment.

To follow the example as in the picture I have 4 .json files, 2 for the vCenters and 2 for the PSC controllers.

Let’s start setting up the PSC Controllers and SSO Site, choose 1 to deploy first, I take PSC_Site1

The first part is the deployment information, what is the hostname where the PSC needs to be deployed too, fill in the username,password,datastore,network etc.

Next is the VCSA information, what is the root password of the appliance, does SSH need to be enabled and

After that there are some seperate configuration options for setting up the SSO domain. Because it’s the first one, the option “first-instance” is set to true, the rest looks self explanatory so fill it in like you designed it.

PSC Site 1 JSON

{
"__comments":
[
"Template to deploy PSC Controller Site1."
],

"deployment":
{
"esx.hostname":"ESX_Site1",
"esx.datastore":"Datastore_Site1",
"esx.username":"root",
"esx.password":"VMWare!",
"deployment.network":"VLAN1234",
"deployment.option":"infrastructure",
"appliance.name":"PSC_Site1",
"appliance.thin.disk.mode":false
},

"vcsa":
{
"system":
{
"root.password":"VCSA_VMWare!",
"ssh.enable":true
},

"sso":
{
"password":"SSO_VMWare!",
"domain-name":"vsphere.local",
"site-name":"vSphere_Test",
"first-instance":true
},

"networking":
{
"ip.family":"ipv4",
"mode":"static",
"ip":"192.168.1.214",
"prefix":"24",
"gateway":"192.168.1.254",
"dns.servers":"192.168.1.203",
"system.name":"SSO_Site1"
}
}
}

Nice, now we have the configuration file for the first External SSO server in the domain. In the JSON directory on the CD are some examples for the different installations. We now need to build the second external PSC controller on a different host in a different physical site, but still need to connect it to the primary site which is created above.

PSC Site 2 JSON


{
"__comments":
[
"Template to deploy PSC Controller Site2."
],

"deployment":
{
"esx.hostname":"ESX_Site2",
"esx.datastore":"Datastore_Site2",
"esx.username":"root",
"esx.password":"VMWare!",
"deployment.network":"VLAN1234",
"deployment.option":"infrastructure",
"appliance.name":"PSC_Site2",
"appliance.thin.disk.mode":true
},

"vcsa":
{
"system":
{
"root.password":"VCSA_VMWare!",
"ssh.enable":true
},

"sso":
{
"password":"SSO_VMWare!",
"domain-name":"vsphere.local",
"site-name":"vSphere_Test",
"replication-partner-hostname":"192.168.1.214"
},

"networking":
{
"ip.family":"ipv4",
"mode":"static",
"ip":"192.168.1.215",
"prefix":"24",
"gateway":"192.168.1.254",
"dns.servers":"192.168.1.203",
"system.name":"192.168.1.215"
}
}
}

As you can see, you configure the replication partner and site name where the second PSC needs to connect/register too.

Good, this will later on deploy the upper part of the picture and configure and SSO Domain which exists of 2 PSC Servers.

vCenter JSON

Let’s configure 2 vCenters, because the configuration is just the same, I’ll post only 1 example. I configure vCenter Site 1 here:


{
"__comments":
[
"Template to deploy vCenter Server Site1."
],

"deployment":
{
"esx.hostname":"ESX_Site1",
"esx.datastore":"Datastore_Site1",
"esx.username":"root",
"esx.password":"VMware!",
"deployment.network":"VLAN1234",
"deployment.option":"management-tiny",
"appliance.name":"vCenter_Site1",
"appliance.thin.disk.mode":true
},

"vcsa":
{
"system":
{
"root.password":"VCSA_VMware!",
"ssh.enable":true,
"platform.service.controller":"192.168.1.214"
},

"sso":
{
"password":"SSO_VMware!",
"domain-name":"vsphere.local",
"site-name":"vSphere_Test"
},

"networking":
{
"ip.family":"ipv4",
"mode":"static",
"ip":"192.168.1.216",
"prefix":"24",
"gateway":"192.168.1.254",
"dns.servers":"192.168.1.203",
"system.name":"192.168.1.216"
}
}
}

The second vCenter configuration file is the except the relation to the PSC which connects to the “local PSC”.
“platform.service.controller”:”192.168.1.215″

So let’s wrap it up, I have 4 .json files which hold the configuration of 1 external PSC controller, which is the first of the site. Then a second PSC is deployed which is connected to the first PSC “Site”.

After that we deploy two vCenters which connect to the PSC Site within their own site.

Good that’s a start. On the Installation CD there is  a file called  : vcsa-deploy.exe which can be found here Z:\vcsa-cli-installer\win32\vcsa-deploy.exe

All command line options an configuration can be found here:

http://www.vmware.com/files/pdf/products/vsphere/VMware-vsphere-60-vcenter-server-appliance-cmdline-install.pdf

Now let’s create a simple deployment batch file which contains the installation.

As you can see, you start the vcsa-deploy.exe, with the json file we created, save the log file to a directory and turn off the SSL verification.


Z:\vcsa-cli-installer\win32\vcsa-deploy.exe D:\JSON\PSC_Site1.json --log D:\JSON\PSC_Heerlen_install.log --no-esx-ssl-verify --verbose
Z:\vcsa-cli-installer\win32\vcsa-deploy.exe D:\JSON\PSC_Site2.json --log D:\JSON\PSC_Beek_install.log --no-esx-ssl-verify
Z:\vcsa-cli-installer\win32\vcsa-deploy.exe D:\JSON\VC_Site1.json --log D:\JSON\VC_Heerlen_install.log --no-esx-ssl-verify
Z:\vcsa-cli-installer\win32\vcsa-deploy.exe D:\JSON\VC_Site2.json --log D:\JSON\VC_Beek_install.log --no-esx-ssl-verify

Let’s fire it up  :

deplot

Cool let’s keep it running until it ‘s finished. I experienced that after it’s finished there seem to be some background tasks, so let it run for a few more minutes to let it sync and setup correctly.

Done?

Now let’s go to the webclient: https://192.168.1.216/vsphere-client

And login with “Administrator@vsphere.local” and the password you provided in the configuration. You’ll now see your 2 vCenters connected in the same interface.

Ah pretty, now let’s see what we can do.

For example go to the “Adminstration” tab, here we can see “Global permissions, licensing etc.”What about cross  vMotion to another datacenter, I read about that..let’s see:

Right click a virtual machine and select the “Migrate” option, choose the option to “Change both compute resource and storage” (because you don’t have the same shared storage on the other side, at least I don’t). Cool, now I can select other vCenters and underlying resources. Select the resource you want to move to, select the datastore, select another network if needed and click “Finish”.

Awesome that worked flawless.

I’m still doing some more testing and will update the text and pictures asap. This was a quick reference & braindump

 

HP Power settings to maximize perfomance

Below are some BIOS settings we set on our ESXi hosts, but also on other HP DL series servers. Everything is used for maximum performance, we don’t look for powersaving features. Or processor C-states which scale up/down the frequencies, this mostly has a negative impact on hard working machines.

HP Power settings to maximize perfomance

Setting Value Description
Intel Turbo Boost Technology Enabled Allows processors to make a transition to a frequency that is higher than its rated speed
Thermal Configuration Optimal/Increased Cooling Use the one that provides the desired performance for the lowest power consumption
HP Power Profile Maximum Performance Disables all power management options that may negatively affect performance
HP Power Regulator HP Static High Performance Keeps processors in their maximum power/performance state
Intel QPI Link Power Management Disabled Precludes placing unutilized QPI links into low power state

Minimum Processor Idle Power Core State

No C-States

Precludes processor transitions into low power core C-States
Minimum Processor Idle Power Package State No Package State

Precludes processor transitions into low power package C-States

Energy/Performance Bias

Maximum Performance

Configures processor subsystems for high performance/low latency
Collaborative Power Control Disabled Precludes the OS from changing clock frequency
DIMM Voltage Preference Optimized for Performance Runs DIMMs at a higher voltage if it increases performance

Dynamic Power Capping Functionality

Disabled This option allows for disabling System ROM Power Calibration during the boot process
Doing so accelerates boot times but precludes enabling of a Dynamic Power Cap

Memory Power Savings Mode

Maximum Performance

This option configures several memory parameters to optimize the memory subsystems performance and is configured to Balanced by default

These are quite the main features to disable.

References:

http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf

http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf

 

 

ESXi CPU Status Demand/Usage/Ready?

ESXi CPU Status Demand/Usage/Ready?

Demand: Amount of CPU the virtual machine is demanding / trying to use
Usage: Amount of CPU the virtual machine is actually currently being allowed to use
Ready: Amount of time the virtual machine is ready to run but unable to because vSphere could not find physical resources to run the virtual machine on.

Virtual machines can be in any one of four high-level CPU States:
Wait: This can occur when the virtual machine’s guest OS is idle (Waiting for Work), or the virtual machine could be waiting on vSphere tasks. Some examples of vSphere tasks that a vCPU may be waiting on are either waiting for I/O to complete (Blocked) or waiting for ESX level swapping to complete (SWPWT). These non-idle vSphere system waits are called VMWAIT.
Ready (RDY): A vCPU is in the Ready state when the virtual machine is ready to run but unable to run because the vSphere scheduler is unable to find physical host CPU resources to run the virtual machine on. One potential reason for elevated Ready time is that the virtual machine is constrained by a user-set CPU limit or resource pool limit, reported as max limited (MLMTD).
CoStop(CSTP): Time the vCPUs of a multi-way virtual machine spent waiting to be co-started. This gives an indication of the co-scheduling overhead incurred by the virtual machine.
Run: Time the virtual machine was running on a physical processor

ESXi Setting syslog and firewall settings with PowerCLI

Syslog and firewall configuration with PowerCLI

Setting PowerCLI

Due the arrival of some SIEM solution I needed to reconfigure ESXi hosts to not only point to our Kiwi Syslog server, but also to the appliance. So a good job for some PowerCLI

I had some trouble using the set-VMHostSysLogServer as it didn’t seem to work as expected. It worked on 2 hosts which hadn’t any syslog configured, but somehow I couldn’t set all to $Null or to the new value, very strange. But I don’t give up and found the Set-VMHostAdvancedConfiguration cmdlet to set the syslog values on another way.

get-vmhost| Set-VMHostAdvancedConfiguration -NameValue @{'Syslog.global.logHost'='syslog'} -confirm:$false

While testing I noted the message:

This cmdlet is deprecated. Use New-AdvancedSetting, Set-AdvancedSetting, or Remove-AdvancedSetting instead.

Mmm let’s have a look here:

get-vmhost|select -first 1|get-advancedsetting -Name syslog* |select name,value|Ft -a

Name                                        Value
—-                                           —–
Syslog.Remote.Port                   514
Syslog.Remote.Hostname          syslog
Syslog.Local.DatastorePath        [] /vmfs/volumes/4dd2476c-etc.

Let’s try to set it

Get-AdvancedSetting -Entity (get-vmhost|select -first 1) -Name Syslog.Remote.Hostname|Set-AdvancedSetting -Value syslog -confirm:$false

You also can set multiple values like:

Get-AdvancedSetting -Entity (get-vmhost|select -first 1) -Name Syslog.Remote.Hostname|Set-AdvancedSetting -Value syslog1,syslog2 -confirm:$false

After setting the proper syslog setting it was necessary to open the syslog firewall ports on ESXi. To do this on all hosts, it can easily be done with the onelinerbelow using the Get-VMHostFirewallException cmdlet

Get-VMHostFirewallException -VMHost (get-vmhost) -Name syslog|Set-VMHostFirewallException -Enabled:$True -Confirm:$false

Locked files ESXi5.1

Locked files 101

Locked files ?

Since  a few months we see more and more “consolidation needed messages” in our vCenter environment, we’re still searching for the reason. Of course we used some VMware articles to get in a bit deeper. But for a successful we need to remove the file lock. I made a little procedure which worked best for us. The information also can be found in the VMware KB’s but sometimes it’s a bit confusing because you have different approaches in different versions.

Investigating virtual machine file locks on ESXi/ESX (10051)Unable to delete the virtual machine snapshot due to locked files (2017072)
Unable to perform operations on a virtual machine with a locked disk (1003397)
Investigating hosted virtual machine lock files (1003857)

Find locked files

It’s easiest to power off the VM. So you’re sure there shouldn’t be any lock anymore. If you’re not sure or have a lot of disks it can be a b*tch to check for locks using the vmkfstools -D. It’s doable, but I found a easier way using the touch command.

Start a SSH session to one of the ESXi hosts in the cluster where the VM resides you want to investigate.

Move to the location of the VM on the datastore: # cd /vmfs/volumes/<datastore>/<VMname>
Now use the touch * command to change the datestamp of all file in this directory. If you have  a lock you will get output like :

touch: VM1-flat.vmdk: Device or resource busy
touch: VM1_3-flat.vmdk: Device or resource busy

Who owns me?

We know now that the 2 files above are in some kind of lock and cannot be written to. Now we can move on using the vmkfstools command to see which ESXi host owns the file.

# vmkfstools -D VM1-flat.vmdk
Lock [type 10c00001 offset 13957120 v 448, hb offset 3510272
gen 41, mode 2, owner 00000000-00000000-0000-000000000000 mtime 845230
num 1 gblnum 0 gblgen 0 gblbrk 0]

We now see 2 things Mode 2 which indicates the kind of lock. See the list below with the kind of modes there can be:

  • mode 0 = no lock
  • mode 1 = is an exclusive lock (vmx file of a powered on VM, the currently used disk (flat or delta), *vswp, etc.)
  • mode 2 = is a read-only lock (e.g. on the ..-flat.vmdk of a running VM with snapshots)
  • mode 3 = is a multi-writer lock (e.g. used for MSCS clusters disks or FT VMs)

We see mode 2 which indicates a file lock. Normally there also is an owner, but this shows only zeroes so we can’t use this information.

In the few lines below we see this information about the Read Only Owner:


RO Owner[0] HB Offset 3510272 53baaace-0e95cf36-46a4-441ea15dea84

Addr <4, 5, 119>, gen 269, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 34359738368, nb 32768 tbz 7056, cow 0, newSinceEpoch 32768, zla 3, bs 1048576

The read only owner of this lock is MAC address 441ea15dea84 which converts to 44:1e:a1:5d:ea:84.

Find the MAC

A fast and easy way to find the corresponding MAC address is with powerCLI. Because we know the cluster where it happens I came up with the command line below:

get-cluster *001*|Get-VMHost|Get-VMHostNetworkAdapter|where {$_.Mac -eq "<strong>44:1e:a1:5d:ea:84</strong>"}|select vmhost,,mac

VMHost                                                                                  Mac
——                                                                                          —
ESX.host.net                              44:1e:a1:5d:ea:84
Now we put this host in maintanance and reboot. Normally the file lock should be gone now and we can consolidate the VM.

1 2 3 4