HP-AMS keeps restarting

Problem

A few weeks ago we started to deploy HP Custom Image for ESXi 5.1.0 Update 2 on all our ESXi hosts. Everything seemed to work without problems until a colleague recently discovered in the logfiles that the HP-AMS provider keeps restarting every 5 minutes and gives an error message that it can’t start because it only works on.

We also noticed the problem only occured on ESXi hosts which are not HP DL380Gen8. So DL585 G5,G6,G7 gave these errors. Makes sense, the error also notices that it runs on Gen8 and older!

Solution

Luckily I found a VMWare article KB2085618 which described our problem:

Too bad the only solution is to remove the Agentless Management agent…by hand on the command line on 50+ ESXi hosts.

Damn! I was too lame to do this by hand so build a little powerCLI script. It’s not completed yet or error free. It was just a quick and dirty solution for fast results. So it’s not yet completed, but would like to share it already as it is faster then enabling SSH everywhere, connecting to ESXi hosts, insert commands, reboot etc.

Script

Pre-requirements

– Connect to your vCenter

– Put host in maintenance mode

– Load the module of function

– Plink installed and edited the script to use the right Plink directory

Download Plink here:

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

Function Enable-TSM {
Param (
[parameter(valuefrompipeline = $true, mandatory = $true,
HelpMessage = "Enter an ESX(i) entity")]
[PSObject]$VMHost,
[switch]$Local)

process {
switch ($VMHost.gettype().name) {
"String" {
if ($Local) {$VMHost = Get-VMHost -Name $VMHost | Enable-TSM -Local}
else {$VMHost = Get-VMHost -Name $VMHost | Enable-TSM}
}
"VMHostImpl" {
if ($Local) {
$VMHost | Get-VMHostService | Where {$_.Key -eq "TSM"} | %{
if ($_.running -eq $false) {
$_ | Start-VMHostService -Confirm:$false | Out-Null
Write-Host "$($_.Label) on $VMHost started"
}
else {Write-Warning "$($_.Label) on $VMHost already started"}
}
}
else {
$VMHost | Get-VMHostService | Where {$_.Key -eq "TSM-SSH"} | %{
if ($_.running -eq $false) {
$_ | Start-VMHostService -Confirm:$false | Out-Null
Write-Host "$($_.Label) on $VMHost started"
}
else {Write-Warning "$($_.Label) on $VMHost already started"}
}
}
}
default {throw "No valid type for parameter -VMHost specified"}
}
}
}

Function Disable-TSM {
Param (
[parameter(valuefrompipeline = $true, mandatory = $true,
HelpMessage = "Enter an ESX(i) entity")]
[PSObject]$VMHost,
[switch]$Local)

process {
switch ($VMHost.gettype().name) {
"String" {
if ($Local) {$VMHost = Get-VMHost -Name $VMHost | Disable-TSM -Local}
else {$VMHost = Get-VMHost -Name $VMHost | Disable-TSM}
}
"VMHostImpl" {
if ($Local) {
$VMHost | Get-VMHostService | Where {$_.Key -eq "TSM"} | %{
if ($_.running -eq $true) {
$_ | Stop-VMHostService -Confirm:$false | Out-Null
Write-Host "$($_.Label) on $VMHost stopped"
}
else {Write-Warning "$($_.Label) on $VMHost already stopped"}
}
}
else {
$VMHost | Get-VMHostService | Where {$_.Key -eq "TSM-SSH"} | %{
if ($_.running -eq $true) {
$_ | Stop-VMHostService -Confirm:$false | Out-Null
Write-Host "$($_.Label) on $VMHost stopped"
}
else {Write-Warning "$($_.Label) on $VMHost already stopped"}
}
}
}
default {throw "No valid type for parameter -VMHost specified"}
}
}
}

These functions were still in my profile so I put them on the site but were not created by me, these are only needed to enable/disable SSH.

function Get-HP{
<#
#Help file
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[System.String]
$VMhostName,

[switch]$Status,

[switch]$Remove
)
try {
$Hosts = Get-VMHost $VMhostName
if ($Status -eq $true){
#######Check for HP-AMS Provider Status #######
foreach ($VMHost in $Hosts){
$ESXCLI = Get-EsxCli -VMHost $VMHost
$HP = $ESXCLI.software.vib.list() | Where { $_.Name -like "hp-ams"} | Select @{N="VMHost";E={$ESXCLI.VMHost}}, Name, Version
if ($HP.name -eq "hp-ams"){
if($Hosts.Model -match "Gen8"){
Write-Host -fore Green "HP-AMS Provider found on" $HP.VMhost $hosts.model "Version:" $HP.version
}
else {
Write-Host -fore Red "Please remove HP-AMS Provider found on" $HP.VMhost $hosts.model "Version:" $HP.version
}
}
else{
Write-Host -ForegroundColor Red "No HP-AMS Provider found on $HP.VMhost $hosts.model"
}
}
}
elseif ($Remove -eq $true){
#######Remove option#######
# Maintenance mode check

Write-Host "Checking Maintenance mode"
if ((Get-VMHost $hosts | select ConnectionState).Connectionstate -ne "Maintenance")
{throw "Put host in maintenance mode please"}
else
{
Write-Host -ForegroundColor Green "Maintenance mode OK"
#2 Enable SSH
Enable-TSM $Hosts
if ((Get-VMHostService -VMHost $Hosts|?{$_.key -eq "TSM-SSH"}).running -eq "True")
{Write-Host -ForegroundColor Green "SSH running succesfull"}
else
{Write-Host -ForegroundColor Red "SSH failed starting"}

#3     HP Service stoppen middels Plink actie
# Creating alias for plink and test path
if (-not (test-path "D:\Putty\plink.EXE")) {throw "D:\Putty\plink.EXE needed"}
set-alias plink "D:\Putty\plink.EXE"
$Str1 = 'echo Y | plink -pw Password -l root '
$Stop = ' /etc/init.d/hp-ams.sh stop'
$Server = $hosts.name
$command= $str1+$server+$Stop
$output = Invoke-Expression -Command $command
$output

#4Verwijderen HP service
Write-Host "Starting removal"
$Str2 = 'plink -pw Password -l root '
$Remover = ' esxcli software vib remove -n hp-ams'
$command= $str2+$Server+$Remover
$output1 = Invoke-Expression -Command $command
$output1
if ($output1 -like "*successfully*"){
Write-Host -ForegroundColor green "Removal completed succesfully"
if ($output1 -like "*reboot*")
{
Write-Host -ForegroundColor Yellow "Reboot required and starting now"
Restart-VMhost -VMHost $Hosts -Confirm:$false|Out-Null
Write-Host -ForegroundColor Yellow "Restart started"
}
Else{
write-host "Possible dry-run?"
}
}
Else {
if ($output1 -like "*NoMatchError*"){
Write-Host "Nothing to do already removed probably restart required";Disable-TSM $Hosts
}
else{}
}

}
}
else {
Write-Host "No switch parameter found, use -remove or -status";Disable-TSM $Hosts
}
}
catch {throw}
}

Switches

-status : Checks the status of the host, is the agent installed and which model is the host.

-remove : Checks if host is in maintenance mode, stops the HP-AMS service, uninstalls the HP-AMS service and restarts the VMhost

Execution

Example for a DL585G5

get-hp -VMhostName esx1.net –Status
Please remove HP-AMS Provider found on esx1.net ProLiant DL585 G5 Version: 500.10.0.0-18.434156

Example for a DL380 Gen8

get-hp -VMhostName esx2.net -Status
HP-AMS Provider found on esx2.net ProLiant DL380p Gen8 Version: 500.10.0.0-18.434156

#Remove exection (need to paste)