Full VMDK Backups

The first thing I’d like to point out is that the vcbMounter does not “mount” the datastore of the VM as one might imagine. When using the vcbMounter program to access the VMDK files, vcbMounter only copies the files to your VCB server. For smaller VMs, 30GB or so, this really isn’t much of an issue; however, for the larger VMs, say 300GB, this could pose a problem.

At first, you may be frustrated as the vcbMounter program is a quirky program. Unless you are lucky or brilliant, give yourself some time to play with the command options.

For my purposes, I use the following vcbMounter statement:

vcbmounter -h [VC or HOST] -u [USERNAME] -p [PASSWORD] -M 1 -a name:[VM NAME] -r [LOCATION]\%1_vmdk

vmdkbkp.bat

As expected, I am backing up many VMs; who would have an ESX(i) host with only one VM? (that is an entirely different subject I may address another day). Instead of scheduling the full vcbMounter command with each backup script or scheduled task, I created a batch file I use to extract the VMDK files. If you are interested in my script, download here.

@ECHO OFF
rem ----------------------------------------------------------------
rem Batch file to create a "mount" dump of a VM, as specified in %1.
rem Created, 11/23/2009, Aaron Gilbert, www.devtrends.com
rem
rem Usage: vmdkbkp.bat [server name]
rem  e.g.: vmdkbkp MYSERVERNAMEINESX
rem
rem ----------------------------------------------------------------

rem check to make sure the user supplied an argument
if "%1"=="" goto error

rem format date in yyyy-mm-dd to apply to a directory, use is %today%
For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set today=%%c-%%a-%%b)

rem update the path statement so we can use vcbmounter
PATH=%PATH%;"C:\Program Files\VMware\VMware Consolidated Backup Framework\"

rem begin vcbmounter to copy the VMDK files down in file size (no 2GB split)
vcbmounter -h [VC or HOST] -u [USERNAME] -p [PASSWORD] -M 1 -a name:%1 -r [LOCATION]\%1_vmdk_%today%

rem lets finish up without error!
goto done

rem fail...
:error
echo.
echo Please provide one argument in the form of a VM server name, such as MYSERVERNAMEINESX
echo.

rem clean complete
:done

The batch file requires only one argument, the name of the VM server you wish to backup. I use the batch file in conjunction with a non-traditional vcbMounter “mounting” approach. I “mount” with the intent of keeping that “mount” for an extended period of time. In addition to the vmdkbkp.bat file, I use another batch file to clean out the backup directory of all “mounted” directories older than a specific date. This allows me to keep the VMDK files on my backup-to-disk server for multiple weeks, while rotating out the older files.

@ECHO OFF
rem -------------------------------------------------------------
rem Batch file to remove all directories that are older than %1.
rem Created, 11/24/2009, Aaron Gilbert, www.devtrends.com
rem
rem Usage: clndir.bat [negative days]
rem  e.g.: clndir -3
rem
rem -------------------------------------------------------------

rem check to make sure the user supplied an argument
if "%1"=="" goto error

rem clean up the directory from all directories older than 3 days
forfiles.exe /p c:\backup\ /m *.* /d %1 /c "cmd /c if @isdir==TRUE rmdir /s /q @path"

rem lets finish up without error!
goto done

rem fail...
:error
echo.
echo Please provide one argument in the form of a negative number, such as -3
echo.

rem clean complete
:done

Aaron Gilbert

Getting Your DroboPro Online!

Obviously you should follow the guides provided to you by Data Robotics for initially setting up your DroboPro. If you seem to be blinded by consumer user guides, the initial seup is easy: (a) insert drives; (b) power on; (c) install Drobo Dashboard; and (d) connect via USB to a workstation. Yes – we start with configuring the device through USB…

Configuring Volumes on DroboPro

Now that your DroboPro is accessible through the Drobo Dashboard, we will start with configuring the volumes. For those that are not familar with iSCSI, these volumes will become iSCSI LUNs from the VMware host perspective. Do not worry about the fact that you have to create the volumes in NTFS as that filesystem will be destroyed when we begin to utilize the volumes for VMware ESX(i). Also worthy of mentioning, the drive letter that DroboPro is assigning to each volume has no value once VMware ESX(i) takes ownership of the volume.

As you may be aware, VMware does not understand LUNs larger than 2 TB, so if your DroboPro has a large disk capacity, you will need to create multiple 2TB volumes. My DroboPro has 8x 1TB hard drives installed. This is accomplished in the Drobo Dashboard software application as shown in the screenshot below:

Configuring iSCSI on DroboPro

The next step is to configure the DroboPro with an IP address for iSCSI connectivity. Not that anyone would want to route iSCSI traffic, however, you will notice that the iSCSI IP address designation does not allow you to configure a gateway IP address. You must use the DroboPro in the same subnet/VLAN as the ESX(i) hosts. This configuration change will require a reboot.

Connecting ESX(i) and DroboPro

If you have an existing SAN, you will want to follow the same configuration, connecting your DroboPro into the Storage Area Network (SAN). Separate networks requires additional VMware configuration, such as another Virtual Switch and VMKernel. If you are a small shop and have the DroboPro on the same network as your production environment, then the only modification will be adding the iSCSI IP address to the VMware iSCSI Software Initiator and performing a Rescan on the adapter.

Note: The DroboPro seems to have a difficult time being an iSCSI target while also connected via the USB port. Now would be a good time to disconnect the DroboPro from the USB cable and visa versa.

On the Configuration tab of your ESX(i) host, choose Storage Adapters and click on the Properties… link within the iSCSI Software Adapter (vmhba32). First, within the iSCSI Initator (vmhba32) window, General tab, ensure that the Software Initiator Properties states Status Enabled. Second, on the Dynamic Discovery tab, add a new iSCSI server and enter the same IP address as you configured on the DroboPro.

In addition, ENSURE that you use port 3261.

The standard iSCSI port 3260 may work; I ran into connection issues when I tried to add another ESX host in the mix. According to DroboPro VMware best practices, the DroboPro can handles up to 4 ESX(i) hosts and they recommend port 3261. Perform a rescan of the HBAs and you should see the LUNs appear, matching the volumes you created earlier.

Adding Storage on ESX(i)

If you have ever worked with iSCSI and VMware, and hopefully you have if you are using VMware in a production environment, then you understand the next step. You will need to create the VMFS volumes from the iSCSI LUNs. If you are confused with iSCSI LUNs, think of an iSCSI LUN as a SCSI LUN, for even though it is software based in our environment, it is utilized as though it is a hardware device or array.

At this point, VMware will destroy the current file system (NTFS) that we created when we create the volumes earlier.

Now you can use the storage to offload test/production VMs, create new VMs, et cetera.

Performance

I did some simple performance testing on the DroboPro using two methods, one with Windows and VMware file copies and the other using iometer.

Before we get too much into the performance I gathered, I want to state two things: (a) my lab environment has fairly old Dell servers and workstations, however, even iometer can produce 57MB/sec on the IDE workstation hard drive; and (b) the VMware results are too low to believe, especially considering that the DroboPro is now VMware certified. To be honest, I am not even certain I set up the test appropriately in iometer. My goal with the configuration was to simulate a file copy with 100% sequential write or read. Click here to download the iometer configuration I used for this test.

Overall, I am fairly pleased with the performance using the Microsoft iSCSI software. As with VMware, the performance is lacking. I will be doing a more thorough performance test with VMware in the near future.

As a final note, be sure to review the Data Robotics document on DroboPro VMware best practices.

-Aaron Gilbert

This article discusses issues I have come across while building or testing opensource iSCSI target software with various iSCSI initiators. I began testing the IET product on Ubuntu Server 8.04 with VMware ESXi 3.5, which worked well in a segregated lab. Then I moved forward to testing a “production” IET target with Microsoft’s iSCSI Software Initiator, which also worked exceptionally well. Finally, I moved into the VMware ESX 3.5 realm, which did not go well at all, initially.

Test 1: VMware ESXi 3.5 and Ubuntu Server 8.04, iSCSI Enterprise Target (IET)

The test was a combination of feasibility and reliability, with an emphasis on the feasibility aspect. For the Ubuntu Server that served as an iSCSI target, I used an older 1U Dell PowerEdge 850 server. The network fabric was a HP ProCurve 48 port switch, which was used for both the standard network environment in the lab and the iSCSI network.

The test went well with a new VM build to the iSCSI target of Windows Server 2003. Heck yes. The feasibility was there as it took an afternoon to make it happen. The reliability was also there, especially since everyone touts Linux as the most solid operating system in the world.

Test 2: Microsoft iSCSI Initiator and IET

http://iscsitarget.sourceforge.net/
http://www.microsoft.com/windowsserver2003/technologies/storage/iscsi/default.mspx

Although I am a Microsoft guy by trade, when it comes to new technology I am generally weary of Microsoft’s implementation as a solid product. However, during my testing of the iSCSI Enterprise Target (IET) opensource target software (0.4.17), I have had absolutely no issues or concerns. After testing in a production lab environment, we moved to testing with our backup to disk project. In this instance, the iSCSI target is used for backups to disk and is hammered almost 24/7 by EMC Retrospect through the Microsoft iSCSI Software Initiator. The product has been running strong for 3 months as of 8/1/2009.

Test 3: VMware ESX 3.5 to 4.0 and IET

If you notice the title of this third test, you may gather that the initial testing did not go as well as was anticipated. In fact, I had a heck of a time getting the ESX 3.5 software iSCSI to even see the IET target from the start. At first I though the configuration was invalid, however, the Microsoft iSCSI Software Initiator worked exceptionally well with the same target with no reconfigurations.

To FINALLY get to a semi-solid state, I began with simplification…

Simplify

As this was the third test, following two successful tests, I decided that I would enter the test with more features enabled, such as multipathing or bonding. This ended up over-complicating the project as there are some known issues with multipathing IET with VMware. To help resolve the issues with connectivity and failures, I went back to one gigabit link, no bonding or multipathing.

Upgrade ESX host from 3.5 to 4.0

Following a technical support call with VMware, I decided to upgrade ESX to vSphere 4. The technical support person wanted me to update ESX 3.5 to the latest build, so I figured, let’s jump to the next level (should I have revisited “simplify”??).

Immediately following the ESX 4.0 installation, I was able to see the iSCSI LUNS. However, I was not able to use them as I kept receiving an error message about not being able to get disk information. If I remember correctly, this was resolved by using “blockio” type on the IET LUNs instead of “fileio”.

Modify IETd.conf to match ESX iSCSI settings

This change was to ensure that the VMware iSCSI software initiator shared the same “mindset” as the IET target. If I remember correctly, only ESX 4 hosts can modify the iSCSI initiator settings.

Conclusion

After running into a list of complexities getting IET to work well with VMware, I finally got it to play nicely for almost 2 weeks. And then…I received the an “unknown command” error in /var/log/messages and the IET target software jammed up causing the VMware virtual machines to lock up.

From my perspective, enough time has been given to IET and I do not feel as though it is ready for VMware – or that VMware is ready for it. I am, therefore, moving forward with OpenSolaris for two reasons, built in iSCSI target and ZFS. More to come…