VMware ESXi VirtualBoot Advanced Troubleshooting

Article Number: 
356

Summary:

This article provides additional advanced troubleshooting details and technical information regarding the configuration of the SPX VirtualBoot VMware vSphere Plugin. For initial basic troubleshooting steps, please see the SPX User Guide.

Background

The VirtualBoot vSphere plugin provides VirtualBoot capability in the VMware virtualization environment. Using the vSphere APIs for I/O Filtering it transparently provides an ESXi host with data from backup chains hosted on an external SMB store. As far as the host is concerned, it is working with a standard .VMDK file. The volume data is persisted permanently in the .VMDK which means it is possible to eventually migrate completely into the virtualization infrastructure and stop using the backup data.

 

Advanced Troubleshooting

 

Managed Object Browser

The Managed Object Browser (MOB) provides a number of links that serve as functions that can be used as-needed for troubleshooting and fixing tougher issues. It is a power-user interface provided by the vCenter server. To access it, login at:

https://<your-vcenter-server>/mob

You'll want to know a few key values to make good use of the MOB. They are:

1. Your cluster's managed object ID (MOID) - Your Cluster MOID is the unique ID assigned to each cluster in the ESX Environment. The format of a Cluster MOID is usually domain-c####. Your MOID is located at

https://vCenter_IP/mob > content > rootFolder > childEntity > hostFolder > in the childEntity VALUE column.

2. Your Host MOID(s) - Your Host MOID(s) are the uniqu IDs assigned to each host in a given cluster. The format of a Host MOID is usually host-### Your Host MOID(s) are located at

https://vCenter_IP/mob > content > rootFolder > childEntity > hostFolder > childEntity > in the host VALUE column

3. The URL for the filter's installation bundle - Currently, the most recent package will be available at: https://downloads.storagecraft.com/ESXFilterCurrent

4. The ioFilter Driver MOID - Currently (Sep-24-2016) that is: STA_bootbank_stastcesxi_1.0.2-1OEM.600.2.34.3657938.  (This can be found in the download package \vib20\stastcesxi folder.) You can locate your ioFilter MOID by going to

https://vCenter_IP/mob > content > ioFilterManager > QueryIoFilterInfo

To use the QueryIoFilterInfo task, replace MOID in the VALUE column with your Cluster's MOID and select "Invoke Method":

Method to query the IO filter driver information for ESX VirtualBoot

There are many things you can investigate with the MOB related to ESX VirtualBoot. Under IoFilterManager section (https://vCenter_IP/mob > content > ioFilterManager) you'll see the following links. They each perform a fairly self-explanatory operation and use the above values to drive the tasks.

          • InstallIoFilter_Task - Installs the filter from a given URL to the given cluster.
          • QueryDisksUsingFilter - Displays the disks currently using the given filter ID and a given cluster MOID.
          • QueryIoFilterIssues - Shows a brief summary of any issues that happened during a filter install or uninstall with a given filter ID and a given Cluster MOID.
          • ResolveInstallationErrorsOnCluster_Task - Tries to resolve installation errors on a cluster from a given filter ID and Cluster MOID.
          • ResolveInstallationErrorsOnHost_Task - Tries to resolve installation errors on a given host from a given filter ID and Host MOID.
          • UninstallIoFilter_Task - Uninstalls the filter from the chosen cluster with a given filter ID and cluster MOID.
          • UpgradeIoFilter_Task - Uninstalls the current filter and then installs a new one from a given filter ID, cluster MOID, and filter URL.

To use the functions, simply fill in the appropriate values on the dialog and hit submit. In the case of values like the MOID, you will be replacing the text "MOID" with the appropriate value. For example:

<compRes type="ComputeResource">MOID</compRes>

is changed to:

<compRes type="ComputeResource">domain-c3483</compRes>

Be careful to not add additional whitespace when you do that, as it will be interepreted (incorrectly) as part of the value.

 

ESX Command Line
 

Manually Modifying a Disk's Filter

To manually attach, detach, or change the properties of an attached filter, you can use the ESXi CLI tool vmkfstools.

The syntax is: vmkfstools -v 5 --iofilters stastcesxi:path=<SMB Path>:domain=<SMB Domain>:user=<SMB username>:password=<SMB Password> /path/to/your/vmdk

If a given key doesn't have a value in your case, e.g. no SMB domain, you can use an asterisk to match the default that vSphere uses, or leave it key out entirely. Remember to:

1. Escape any backslashes or asteriks, as they will otherwise be interpreted early by the command line itself.
2. Put quotes around any value that contains spaces.

For example:

vmkfstools -v 5 --iofilters stastcesxi:path=\\\\255.255.255.255\\MySmbShare:domain=\*:user=smbuser:password="This is my SMB password." /vmfs/volumes/datastore1/myVm/myvm.vmdk
 

Reenabling a Disabled Filter

In the case of a full loss of connection to SMB, the filter will disable itself on the affected host(s). All new I/O will stop, and no new disks can be attached. To resolve this:

1. Power down the affected machines. There isn't a risk of data loss, because it's all stored with the .VMDK itself.
2. Resolve the connectivity issues with your SMB server.
3. From the ESX command line, reenable the filter on the affected hosts and restart the ESX host services. Use the following commands:

esxcli storage iofilter enable -f stastcesxi
services.sh restart

Uninstalling the ioFilter Driver using the MOB

There are times when uninstalling a filter driver and reinstalling it is necessary for troubleshooting. To manually uninstall the ioFilter using the MOB:

1. Locate your cluster MOID
2. Locate your IOFilter ID
2. Navigate in the MOB to https://vCenter_IP/mob > content > ioFilterManager > UninstallIoFilter_Task
3. Input the filterID and compRES (cluster MOID) fields
4. Invoke the method to run the IOFilter remove task

Installing the ioFilter Driver using the MOB 

The preferred method of installing the ioFilter Driver is by running a VirtualBoot process in SPX. If that doesn't seem to install the driver, we recommend checking for errors that may have happened during the install. To attempt a manual install of the ioFilter Driver using the MOB, take the following steps:

1. Locate your cluster MOID
2. Navigate in the MOB to https://vCenter_IP/mob > content > ioFilterManager > InstallIoFilter_Task
3. Input the vibUrl and compRES (cluster MOID) fields. Currently, the ESX filter is hosted at https://downloads.storagecraft.com/ESXFilterCurrent
4. Invoke the method to run the IOFilter installation task

Hand-editing

As a last resort, it's sometimes necessary to clean up an attached filter by hand. This can happen, for instance, if the SMB server is no longer reachable, as the detach process requires going through the open process first. Hand-removal of the filter requires access to the datastores where the VMDK and metadata are stored. To fully remove the filter:

1. Delete the filter's "sidecar" files
These are files used to persist filter-specific information. They are located next to the .VMDK and will be named based on the .VMDK's name, with an additional random suffix and the .VMFD file extension. For instance, if you have the .VMDK "my-vmdk.vmdk", the sidecar files might be called "my-vmdk-d83a1fcb04d09e83.vmfd" and "my-vmdk-49699f39bd97ab17.vmfd". You'll need to delete both.
2. Edit the .VMDK's configuration file
The .VMDK file on the VMFS datastore is actually a plaintext .INI file. It has two lines in it that must be removed to disassociate the disk from the filter. You'll need to SSH to the ESXi machine and use the vi text editor to modify them. Delete the two following lines:

ddb.iofilters = "stastcesxi"
ddb.sidecars = "..."  (Between the quotes will be the name of the filter and the name of the sidecar files in them.)

After those steps, the filter will no longer be associated with the disk. If the vSphere client still believes it is, you can now safely set the .VMDK storage policy back to "Datastore Default"

AttachmentSize
queryiofilterinfo.png38.23 KB

Terms and Conditions of Use - Privacy Policy - Cookies