System Administration

Validating VM advanced settings

Now that we have covered how to make advanced settings changes we need to validate that the settings have been applied.  Assuming that a script or command worked is never as good as checking it.  Below is a script to do just that.

The script lists all statuses so if you only want to see issues you can pipe the output into the Where-Object cmdlet like so:

<vmObject(s)> | .\Validate-VmAdvancedSettings.ps1 | Where-Object {$_.Status -ne “Ok”}

The script:

# Usage: <vmobject> | .\Validate-VmAdvancedSettings.ps1
# Examples: Get-VM myTestVM | .\Validate-VmAdvancedSettings.ps1
#           Get-Cluster myCluster | Get-VM | .\Validate-VmAdvancedSettings.ps1
 
BEGIN
{
    # The settings as an array of arrays.  ("key", "value)
    $advancedSettings = @( ("isolation.tools.copy.disable", "true"),
                           ("isolation.tools.paste.disable", "true"),
                           ("isolation.tools.setGUIOptions.enable", "false"),
                           ("log.rotateSize", "100000"),
                           ("log.keepOld", "10"),
                           ("isolation.tools.connectable.disable", "true"),
                           ("isolation.device.connectable.disable", "true"),
                           ("isolation.tools.diskWiper.disable", "true"),
                           ("isolation.tools.diskShrink.disable", "true")
                         )
    $keyIndex = 0
    $valueIndex = 1
}
 
PROCESS
{                      
    $vmView = Get-View $_.Id
 
    # Get the existing settings in an easy lookup format.
    $vmSettings = @{}
    foreach ($item in $vmView.Config.ExtraConfig)
    {
        $vmSettings[$item.Key] = $item.Value
    }
 
    # Validate each of the advanced settings.
    foreach ($setting in $advancedSettings)
    {
        $status = $null
        if (!$vmSettings.ContainsKey($setting[$keyIndex]))
        {
            $status = "Missing"
        }
        elseif ($vmSettings[$setting[$keyIndex]] -ne $setting[$valueIndex])
        {
            $status = "Misconfigured"
        }
        else
        {
            $status = "Ok"
        }
 
        $vmView | Select-Object Name, 
                                @{Name="Setting"; Expression={$setting[$keyIndex]}},
                                @{Name="Status"; Expression={$status}}
    }
}

Making advanced settings changes to a running VM

If you use a method like that in the Powershell script I posted the changes can be made to a running VM. This is great because that dialog box is disabled in Virtual Center when the vm is up.

The settings do not take effect until the VM is power cycled (not just restarted in the OS).  This is not that big of a deal on clusters with VMotion licensed because a VMotion starts a new instance of the VM on the destination host.  I have validated that security settings like isolation.tools.connectable.disable work immediately after the VMotion, but I have not checked all settings.

As always, be careful with the advanced settings and be prepared with a backout plan before doing this on a large scale (i.e. before ESX host patching).

Let me know if you run into any settings that do not apply after a VMotion.

Get-AllRolePrivileges

This is a good script to easily report on privileges and can easily be extended to audit against a known good list of expected roles/privileges.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#Must already be connected to a viserver. This works on the default vi server.
$si = Get-View ServiceInstance
$am = Get-View ($si.Content.AuthorizationManager)
 
$roles = $am.RoleList
 
foreach ($role in $roles)
{
    foreach ($privilege in $role.Privilege)
    {
        $role| Select-Object RoleId, System, Name,
                             @{Name="Privilege"; Expression={$privilege}}
    }
}

Script to update tools in templates.

Here is a script that takes template objects and updates their VMware tools.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# Usage: <template object(s)> | .\Upgrade-ToolsTemplate.ps1
# Example: Get-Template template1 | .\Upgrade-ToolsTemplate.ps1
#          Get-Datacenter corporate | Get-Template | .\Upgrade-ToolsTemplate.ps1
# Note: When getting template make sure to get the correct datacenter in case
#       there are duplicate names.
 
BEGIN
{
    $sleeptime = 60
}
 
PROCESS
{
    "Converting {0} to VM" -f $_.Name
    $vm = $_ | Set-Template -ToVM
 
    "Powering on the VM: {0}" -f $vm.Name
    $vm | Start-VM | Out-Null
 
    #TODO: Find a better way of doing this than just sleeping.  Perhaps poll 
    #      on tools status.
    "Sleeping for $sleeptime seconds"
    Start-Sleep -Seconds $sleeptime
 
    $vmview = Get-View $vm.ID
    "Existing Tools Version {0}" -f $vmview.config.tools.toolsVersion
 
    "Upgrading Tools"
    $vm | Update-Tools
 
    "Sleeping for $sleeptime seconds"
    Start-Sleep -Seconds $sleeptime
 
    $vmview = Get-View $vm.ID
    "New Tools Version {0}" -f $vmview.config.tools.toolsVersion
 
    "Powering off the VM: {0}" -f $vm.Name
    $vm | Stop-VM -confirm:$false | Out-Null
 
    "Converting to template"
    $vmview.MarkAsTemplate()
}

Validating Ubuntu Installed Files

I had an issue where I ran a vendor provided install script and my kernel sound drivers were deleted.  I recovered them just fine, but it had me wondering about how I could be sure that no other files were missing or corrupted.

After some searching online I found a reference to someone checking a package using dpkg’s md5sums.  This and some poking around gave me what I needed.


cd /
sudo /usr/bin/md5sum -c /var/lib/dpkg/info/*.md5sums | tee /tmp/md5sum_results.txt

After that run is complete you can use grep to find things that are not OK and investigate them. If this is to be used for security validation the md5sums and related tools need to be stored on read only media.

Basic install of bcfg2 on ESX 3.5

I am working on the configuration management problem and have been toying with getting bcfg2 working on ESX 3.5. My goal is to have an isolated install that does not require modifying the existing system files or upgrading any packages. My approach to doing this is to create a local Python 2.5 install and install the bcfg2 packages there.

This is just a start, but I thought I would put it out there for anyone who wants to work on something similar.

Installing bcfg2 on ESX 3.5 v 0.00001

ESX Server Configuration Management

I am not very happy with the landscape of configuration management software out there for VMware ESX server.  I am investigating using projects such as Bcfg2 to administer my servers.  I am working with Bcfg2 because I do not have a lot of experience with these types of software and I am interested in learning more about Python.

Of course these types of packages fail when you look at ESX 3i, but I am not convinced that I will be an early adopter of that platform.  I have tried to automate a few things with the VI SDK, but I am having trouble finding all of the information that I want to audit.

If anyone knows about a good CM solution for ESX please let me know.

IT Infrastructure Performance – Research

I have been working on methods to benchmark the Windows infrastructure team at work and I have not been finding much luck finding existing metrics. I was not too happy with the last Gartner review of our IT department, but perhaps that is all that is out there. I have developed some thoughts on the matter and once I am fairly certain they make sense I will post them here.

I have come across a pretty interesting book on measuring business processes. It is How to Measure Anything: Finding the Value of “Intangibles” in Business. The book walks you through quantitative decision making and how to do this with things that you might have previously thought were immeasurable.

Infrastructure Value != Utilization

A common problem in infrastructure groups is determining the value that we provide to the organization. One common way of thinking about this is in terms of asset utilization. Messages such as “take your servers from 5% utilization to 60%” are very common in virtualization value propositions, but I do not think that capacity utilization is the correct measurement of infrastructure value. That value is derived from being able to run an application at adequate performance levels. It does not matter what the utilization is. Take the following hypothetical scenario measuring performance in CPU or memory:

utilization_comparison_small

Both of these systems are running the same application and provide the same benefit to the organization, but the higher utilization solution actually costs more. When thinking about it this way I think that it becomes clear that higher utilization is not a good indicator of infrastructure efficiency in providing value to the organization.