Full VMDK Backups

The first thing I’d like to point out is that the vcbMounter does not “mount” the datastore of the VM as one might imagine. When using the vcbMounter program to access the VMDK files, vcbMounter only copies the files to your VCB server. For smaller VMs, 30GB or so, this really isn’t much of an issue; however, for the larger VMs, say 300GB, this could pose a problem.

At first, you may be frustrated as the vcbMounter program is a quirky program. Unless you are lucky or brilliant, give yourself some time to play with the command options.

For my purposes, I use the following vcbMounter statement:

vcbmounter -h [VC or HOST] -u [USERNAME] -p [PASSWORD] -M 1 -a name:[VM NAME] -r [LOCATION]\%1_vmdk

vmdkbkp.bat

As expected, I am backing up many VMs; who would have an ESX(i) host with only one VM? (that is an entirely different subject I may address another day). Instead of scheduling the full vcbMounter command with each backup script or scheduled task, I created a batch file I use to extract the VMDK files. If you are interested in my script, download here.

@ECHO OFF
rem ----------------------------------------------------------------
rem Batch file to create a "mount" dump of a VM, as specified in %1.
rem Created, 11/23/2009, Aaron Gilbert, www.devtrends.com
rem
rem Usage: vmdkbkp.bat [server name]
rem  e.g.: vmdkbkp MYSERVERNAMEINESX
rem
rem ----------------------------------------------------------------

rem check to make sure the user supplied an argument
if "%1"=="" goto error

rem format date in yyyy-mm-dd to apply to a directory, use is %today%
For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set today=%%c-%%a-%%b)

rem update the path statement so we can use vcbmounter
PATH=%PATH%;"C:\Program Files\VMware\VMware Consolidated Backup Framework\"

rem begin vcbmounter to copy the VMDK files down in file size (no 2GB split)
vcbmounter -h [VC or HOST] -u [USERNAME] -p [PASSWORD] -M 1 -a name:%1 -r [LOCATION]\%1_vmdk_%today%

rem lets finish up without error!
goto done

rem fail...
:error
echo.
echo Please provide one argument in the form of a VM server name, such as MYSERVERNAMEINESX
echo.

rem clean complete
:done

The batch file requires only one argument, the name of the VM server you wish to backup. I use the batch file in conjunction with a non-traditional vcbMounter “mounting” approach. I “mount” with the intent of keeping that “mount” for an extended period of time. In addition to the vmdkbkp.bat file, I use another batch file to clean out the backup directory of all “mounted” directories older than a specific date. This allows me to keep the VMDK files on my backup-to-disk server for multiple weeks, while rotating out the older files.

@ECHO OFF
rem -------------------------------------------------------------
rem Batch file to remove all directories that are older than %1.
rem Created, 11/24/2009, Aaron Gilbert, www.devtrends.com
rem
rem Usage: clndir.bat [negative days]
rem  e.g.: clndir -3
rem
rem -------------------------------------------------------------

rem check to make sure the user supplied an argument
if "%1"=="" goto error

rem clean up the directory from all directories older than 3 days
forfiles.exe /p c:\backup\ /m *.* /d %1 /c "cmd /c if @isdir==TRUE rmdir /s /q @path"

rem lets finish up without error!
goto done

rem fail...
:error
echo.
echo Please provide one argument in the form of a negative number, such as -3
echo.

rem clean complete
:done

Aaron Gilbert

From your ESX host console window, hit ALT-F1 to reach the prompt and sign in as root. At the terminal prompt, open the sshd_config file using the following command:

# nano /etc/ssh/sshd_config

Find the line that states “PermitRootLogin” and change the value to yes. Save your file and exit nano (CTRL-O, CTRL-X). Now you will need to restart the SSH daemon.

# service sshd restart

Now attempt to sign in as root from your SSH client.

-Aaron

On to the true purpose of this artcle, let’s get virtualizing with OpenSolaris’ rendition of Xen, SUN xVM.

OpenSolaris

I am going to assume that you already have OpenSolaris installed. At the time of this article, I was using 2009-06. The second assumption is that all the following commands are ran as root. The third assumption is that your installation of OpenSolaris has access to the Internet.

Installing xVM

In the first step we will create a new boot environment using the beadm tool. Option “-a” states to activate the new boot environment and the “-d xvm” defines the description for the boot environment. The final xvm is the name of the boot environment.

# pfexec beadm create -a -d xvm xvm

The second step will install the xVM software packages into the newly created boot environment. The first command mounts the boot environment. The second command installs the xvm-gui package which has all of the xVM dependencies associated, forcing a full install of xVM.

# pfexec beadm mount xvm /tmp/xvm-be
# pfexec pkg -R /tmp/xvm-be install xvm-gui
# pfexec beadm umount xvm

Configuring your Grub Boot Menu

The next step requires that you modify your boot menu, grub/menu.lst. In the new installation of OpenSolaris, this file will reside as /rpool/boot/grub/menu.lst. Use your favorite editor to open menu.lst:

# nano /rpool/boot/grub/menu.lst

At the bottom of the file, add the following lines:

title xvm
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/xvm
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=text
module$ /platform/i86pc/$ISADIR/boot_archive

Note: if you already have a “title xvm” section, and your installation was similar to mine, you will likely find that the boot configuration I specified above will work better. However, if you would like to try the boot configuration that was automatically entered, move along with the article and test a connection to the xVM services using the Virtual Machine Manager. If it fails, replace the “title xvm” with my version above and reboot.

Save the menu.lst file and reboot your OpenSolaris server:

# reboot

Enabling xVM Services

The final step to installing the xVM hypervisor in OpenSolaris is to enable the services for xVM, this is accomplished with the following two commands:

# svcadm enable -r xvm/virtd
# svcadm enable -r xvm/domains

Jump In!

Being that I am a Windows administrator by trade, I enjoy the GUI interface. Such would be my choice in any Linux environment too. Using the Applications menu choose System Tools > Virtual Machine Manager and connect to the localhost.

As a final note, be sure to check out the documentation on OpenSolaris.org.

-Aaron Gilbert

This article discusses issues I have come across while building or testing opensource iSCSI target software with various iSCSI initiators. I began testing the IET product on Ubuntu Server 8.04 with VMware ESXi 3.5, which worked well in a segregated lab. Then I moved forward to testing a “production” IET target with Microsoft’s iSCSI Software Initiator, which also worked exceptionally well. Finally, I moved into the VMware ESX 3.5 realm, which did not go well at all, initially.

Test 1: VMware ESXi 3.5 and Ubuntu Server 8.04, iSCSI Enterprise Target (IET)

The test was a combination of feasibility and reliability, with an emphasis on the feasibility aspect. For the Ubuntu Server that served as an iSCSI target, I used an older 1U Dell PowerEdge 850 server. The network fabric was a HP ProCurve 48 port switch, which was used for both the standard network environment in the lab and the iSCSI network.

The test went well with a new VM build to the iSCSI target of Windows Server 2003. Heck yes. The feasibility was there as it took an afternoon to make it happen. The reliability was also there, especially since everyone touts Linux as the most solid operating system in the world.

Test 2: Microsoft iSCSI Initiator and IET

http://iscsitarget.sourceforge.net/
http://www.microsoft.com/windowsserver2003/technologies/storage/iscsi/default.mspx

Although I am a Microsoft guy by trade, when it comes to new technology I am generally weary of Microsoft’s implementation as a solid product. However, during my testing of the iSCSI Enterprise Target (IET) opensource target software (0.4.17), I have had absolutely no issues or concerns. After testing in a production lab environment, we moved to testing with our backup to disk project. In this instance, the iSCSI target is used for backups to disk and is hammered almost 24/7 by EMC Retrospect through the Microsoft iSCSI Software Initiator. The product has been running strong for 3 months as of 8/1/2009.

Test 3: VMware ESX 3.5 to 4.0 and IET

If you notice the title of this third test, you may gather that the initial testing did not go as well as was anticipated. In fact, I had a heck of a time getting the ESX 3.5 software iSCSI to even see the IET target from the start. At first I though the configuration was invalid, however, the Microsoft iSCSI Software Initiator worked exceptionally well with the same target with no reconfigurations.

To FINALLY get to a semi-solid state, I began with simplification…

Simplify

As this was the third test, following two successful tests, I decided that I would enter the test with more features enabled, such as multipathing or bonding. This ended up over-complicating the project as there are some known issues with multipathing IET with VMware. To help resolve the issues with connectivity and failures, I went back to one gigabit link, no bonding or multipathing.

Upgrade ESX host from 3.5 to 4.0

Following a technical support call with VMware, I decided to upgrade ESX to vSphere 4. The technical support person wanted me to update ESX 3.5 to the latest build, so I figured, let’s jump to the next level (should I have revisited “simplify”??).

Immediately following the ESX 4.0 installation, I was able to see the iSCSI LUNS. However, I was not able to use them as I kept receiving an error message about not being able to get disk information. If I remember correctly, this was resolved by using “blockio” type on the IET LUNs instead of “fileio”.

Modify IETd.conf to match ESX iSCSI settings

This change was to ensure that the VMware iSCSI software initiator shared the same “mindset” as the IET target. If I remember correctly, only ESX 4 hosts can modify the iSCSI initiator settings.

Conclusion

After running into a list of complexities getting IET to work well with VMware, I finally got it to play nicely for almost 2 weeks. And then…I received the an “unknown command” error in /var/log/messages and the IET target software jammed up causing the VMware virtual machines to lock up.

From my perspective, enough time has been given to IET and I do not feel as though it is ready for VMware – or that VMware is ready for it. I am, therefore, moving forward with OpenSolaris for two reasons, built in iSCSI target and ZFS. More to come…