Getting Your DroboPro Online!

Obviously you should follow the guides provided to you by Data Robotics for initially setting up your DroboPro. If you seem to be blinded by consumer user guides, the initial seup is easy: (a) insert drives; (b) power on; (c) install Drobo Dashboard; and (d) connect via USB to a workstation. Yes – we start with configuring the device through USB…

Configuring Volumes on DroboPro

Now that your DroboPro is accessible through the Drobo Dashboard, we will start with configuring the volumes. For those that are not familar with iSCSI, these volumes will become iSCSI LUNs from the VMware host perspective. Do not worry about the fact that you have to create the volumes in NTFS as that filesystem will be destroyed when we begin to utilize the volumes for VMware ESX(i). Also worthy of mentioning, the drive letter that DroboPro is assigning to each volume has no value once VMware ESX(i) takes ownership of the volume.

As you may be aware, VMware does not understand LUNs larger than 2 TB, so if your DroboPro has a large disk capacity, you will need to create multiple 2TB volumes. My DroboPro has 8x 1TB hard drives installed. This is accomplished in the Drobo Dashboard software application as shown in the screenshot below:

Configuring iSCSI on DroboPro

The next step is to configure the DroboPro with an IP address for iSCSI connectivity. Not that anyone would want to route iSCSI traffic, however, you will notice that the iSCSI IP address designation does not allow you to configure a gateway IP address. You must use the DroboPro in the same subnet/VLAN as the ESX(i) hosts. This configuration change will require a reboot.

Connecting ESX(i) and DroboPro

If you have an existing SAN, you will want to follow the same configuration, connecting your DroboPro into the Storage Area Network (SAN). Separate networks requires additional VMware configuration, such as another Virtual Switch and VMKernel. If you are a small shop and have the DroboPro on the same network as your production environment, then the only modification will be adding the iSCSI IP address to the VMware iSCSI Software Initiator and performing a Rescan on the adapter.

Note: The DroboPro seems to have a difficult time being an iSCSI target while also connected via the USB port. Now would be a good time to disconnect the DroboPro from the USB cable and visa versa.

On the Configuration tab of your ESX(i) host, choose Storage Adapters and click on the Properties… link within the iSCSI Software Adapter (vmhba32). First, within the iSCSI Initator (vmhba32) window, General tab, ensure that the Software Initiator Properties states Status Enabled. Second, on the Dynamic Discovery tab, add a new iSCSI server and enter the same IP address as you configured on the DroboPro.

In addition, ENSURE that you use port 3261.

The standard iSCSI port 3260 may work; I ran into connection issues when I tried to add another ESX host in the mix. According to DroboPro VMware best practices, the DroboPro can handles up to 4 ESX(i) hosts and they recommend port 3261. Perform a rescan of the HBAs and you should see the LUNs appear, matching the volumes you created earlier.

Adding Storage on ESX(i)

If you have ever worked with iSCSI and VMware, and hopefully you have if you are using VMware in a production environment, then you understand the next step. You will need to create the VMFS volumes from the iSCSI LUNs. If you are confused with iSCSI LUNs, think of an iSCSI LUN as a SCSI LUN, for even though it is software based in our environment, it is utilized as though it is a hardware device or array.

At this point, VMware will destroy the current file system (NTFS) that we created when we create the volumes earlier.

Now you can use the storage to offload test/production VMs, create new VMs, et cetera.

Performance

I did some simple performance testing on the DroboPro using two methods, one with Windows and VMware file copies and the other using iometer.

Before we get too much into the performance I gathered, I want to state two things: (a) my lab environment has fairly old Dell servers and workstations, however, even iometer can produce 57MB/sec on the IDE workstation hard drive; and (b) the VMware results are too low to believe, especially considering that the DroboPro is now VMware certified. To be honest, I am not even certain I set up the test appropriately in iometer. My goal with the configuration was to simulate a file copy with 100% sequential write or read. Click here to download the iometer configuration I used for this test.

Overall, I am fairly pleased with the performance using the Microsoft iSCSI software. As with VMware, the performance is lacking. I will be doing a more thorough performance test with VMware in the near future.

As a final note, be sure to review the Data Robotics document on DroboPro VMware best practices.

-Aaron Gilbert

SUN iSCSI Initiator

Obviously you need to have your network configured properly prior to working with the iSCSI initiator or targets. In this example, I was working with the iSCSI initiator and target in a lab environment, with the host (target) being 192.168.0.18 and the client (initiator) being 192.168.0.19. Secondly, you will need the iSCSI initiator software to be installed from Package Manager – easiest way is to search for iscsi and ensure all four of those packages are installed.

For this example we will use Send Targets for discovering iSCSI targets/paths:

iscsiadm add discovery-address 192.168.0.18:3260
iscsiadm modify discovery --sendtargets enable

Now that you have added the discovery address of the iSCSI target and enabled the discovery “Send Targets”, check to see if your target was discovered:

iscsiadm list target

If you saw the targets you were expecting, the next step is to create the device links between iSCSI and your system – so you can see the iSCSI LUN as a disk on your system.

devfsadm -i iscsi

Check the available disks on your system using the format command. After running the format command, hit CTRL-C to bounce back to the terminal prompt.

Using / Mounting the iSCSI LUN

Now that you have the iSCSI initiator working properly and the iSCSI targets discovered, lets actually use the new iSCSI disk with ZFS.

First we need to create the ZFS pool using the new disk. If you forgot the name of the new disk, you can learn it using the format command. If it is anything like the one on my OpenSolaris box, the name is rather large, (c0t600144F0F9A00C0000004AA9510A0001d0):

zpool create diskpool c0t600144F0F9A00C0000004AA9510A0001d0

To verify the pool was created, use:

zpool list

Now we can create a ZFS file system on top of that pool:

zfs create diskpool/test

Check your new ZFS filesystem was created:

zfs list

And now you can create files within the mount point:

root@opensolaris:/diskpool/test/#

Everything you write to that mount point will be written to the iSCSI target, as one would hope.

Good luck with your performance!

Credits

http://wikis.sun.com/display/OpenSolarisInfo/How+to+Configure+an+iSCSI+Initiator

This article discusses issues I have come across while building or testing opensource iSCSI target software with various iSCSI initiators. I began testing the IET product on Ubuntu Server 8.04 with VMware ESXi 3.5, which worked well in a segregated lab. Then I moved forward to testing a “production” IET target with Microsoft’s iSCSI Software Initiator, which also worked exceptionally well. Finally, I moved into the VMware ESX 3.5 realm, which did not go well at all, initially.

Test 1: VMware ESXi 3.5 and Ubuntu Server 8.04, iSCSI Enterprise Target (IET)

The test was a combination of feasibility and reliability, with an emphasis on the feasibility aspect. For the Ubuntu Server that served as an iSCSI target, I used an older 1U Dell PowerEdge 850 server. The network fabric was a HP ProCurve 48 port switch, which was used for both the standard network environment in the lab and the iSCSI network.

The test went well with a new VM build to the iSCSI target of Windows Server 2003. Heck yes. The feasibility was there as it took an afternoon to make it happen. The reliability was also there, especially since everyone touts Linux as the most solid operating system in the world.

Test 2: Microsoft iSCSI Initiator and IET

http://iscsitarget.sourceforge.net/
http://www.microsoft.com/windowsserver2003/technologies/storage/iscsi/default.mspx

Although I am a Microsoft guy by trade, when it comes to new technology I am generally weary of Microsoft’s implementation as a solid product. However, during my testing of the iSCSI Enterprise Target (IET) opensource target software (0.4.17), I have had absolutely no issues or concerns. After testing in a production lab environment, we moved to testing with our backup to disk project. In this instance, the iSCSI target is used for backups to disk and is hammered almost 24/7 by EMC Retrospect through the Microsoft iSCSI Software Initiator. The product has been running strong for 3 months as of 8/1/2009.

Test 3: VMware ESX 3.5 to 4.0 and IET

If you notice the title of this third test, you may gather that the initial testing did not go as well as was anticipated. In fact, I had a heck of a time getting the ESX 3.5 software iSCSI to even see the IET target from the start. At first I though the configuration was invalid, however, the Microsoft iSCSI Software Initiator worked exceptionally well with the same target with no reconfigurations.

To FINALLY get to a semi-solid state, I began with simplification…

Simplify

As this was the third test, following two successful tests, I decided that I would enter the test with more features enabled, such as multipathing or bonding. This ended up over-complicating the project as there are some known issues with multipathing IET with VMware. To help resolve the issues with connectivity and failures, I went back to one gigabit link, no bonding or multipathing.

Upgrade ESX host from 3.5 to 4.0

Following a technical support call with VMware, I decided to upgrade ESX to vSphere 4. The technical support person wanted me to update ESX 3.5 to the latest build, so I figured, let’s jump to the next level (should I have revisited “simplify”??).

Immediately following the ESX 4.0 installation, I was able to see the iSCSI LUNS. However, I was not able to use them as I kept receiving an error message about not being able to get disk information. If I remember correctly, this was resolved by using “blockio” type on the IET LUNs instead of “fileio”.

Modify IETd.conf to match ESX iSCSI settings

This change was to ensure that the VMware iSCSI software initiator shared the same “mindset” as the IET target. If I remember correctly, only ESX 4 hosts can modify the iSCSI initiator settings.

Conclusion

After running into a list of complexities getting IET to work well with VMware, I finally got it to play nicely for almost 2 weeks. And then…I received the an “unknown command” error in /var/log/messages and the IET target software jammed up causing the VMware virtual machines to lock up.

From my perspective, enough time has been given to IET and I do not feel as though it is ready for VMware – or that VMware is ready for it. I am, therefore, moving forward with OpenSolaris for two reasons, built in iSCSI target and ZFS. More to come…

System Specifications

Case. SuperMicro 3U rack mounted case, model SC836A-R1200B ($1100). My first prototype was with the SuperMicro SC932T-760B case ($800) that had a triple redundant 760watt PS and 15 SATA bays and 15 SATA connectors requiring miniSAS to SATA fan cables. The power supply could not handle the motherboard, raid controller, and system fans – bad design. The new case is also a 3U, but holds 16 SATA/SAS drives and the backplane has 4 miniSAS connectors.

SC836A-R1200B: http://www.supermicro.com/products/chassis/3U/836/SC836A-R1200.cfm
SC932T-760B: http://www.supermicro.com/products/chassis/3U/932/SC932T-R760.cfm

RAID Controller. The first prototype was using the LSI MegaRAID 84016E ($600), which in my opinion is very good controller – however, it does not work well with the IET iSCSI target software. During my initial stages of the iSCSI SAN project, I moved forward to the more expensive 31605 ($850?) because it was on sale ($500) and the LSI produced borderlining pathetic performance with a RAID5 configuration with the IET iSCSI application interface (15MBytes/sec). The Adaptec 31605 card holds up strong and can produce nearly 5x (3GB file copy in around 50 seconds…) the throughput with the same drives and RAID set type, 5. Both controllers have 4 miniSAS MFF8087 connectors, which allow up to 16 SATA devices through a fan-out cable or backplane. I used miniSAS on the controller to miniSAS on the backplane in the current configuration.

Update: I recommend using OpenSolaris, COMSTAR and ZFS for all custom iSCSI SANs – in my opinion, there is no better solution available at the moment. In addition, the LSI MegaRAID 84016E with the battery kit is a solid controller with OpenSolaris and COMSTAR.

Storage SATA hard drives. I have been using two sizes of drives, 640GB in the SAN that will be used for a VMware environment, creating 2 RAID60 RAID sets (not completely tested yet) and 1.5TB in the backup to disk creating a large 22TB RAID5. As a side note, NewEgg seems to have the BEST pricing when it comes to drives.

Boot controller / drives. In the new SC836A-1200B case, I had the option for two 2.5” drives. I am using the SATA ports on the motherboard and mirroring between the two drives using the ZFS file system.

Network Interfaces. In my first prototype I utilized an Intel dual port PCI-express x2 card and within RedHat I bonded the ports together using LACP (mode=4). In the VMware environment, I am using a PCI-express x4 Intel quad port card. For those that are wondering about jumbo framing: although I am currently using the Dell PowerConnect 5448, which supports jumbo framing, I am not using jumbo frames at this time.

Motherboard/RAM/CPU. My first prototype used an ASUS board that had one PCI express and around 3 PCI slots, my second was a GigaByte board, which performed better, however it is still missing a key CORE component, ECC ram.  With the first motherboard (ASUS), the idea was to utilize the PCI slots for the network interfaces – however, after I purchased the motherboard, I realized/remembered that the throughput of a standard PCI bus cannot even support one gigabit Ethernet, yet alone two or three bonded. I am currently using the Intel Server Board S3200SH with 8GB ECC DDR2 800 RAM and an Intel Core2 Duo…

After Thoughts

While building both the prototype and other SANs, I ran into some issues, which were for the most part overcome with some engineering or parts purchasing.

Make sure the motherboard you will be using has enough expansion slots and bus speed to support the required throughput of the RAID controller and the network interfaces. PCI Express comes in many sizes, from x1 to x16. IBM wrote a create article about the differences of each speed: http://www.redbooks.ibm.com/abstracts/tips0456.html

Each motherboard configuration is different, when purchasing the motherboard, make sure the cables from the power supply are long enough to reach the ports on the motherboard. This is typically not a problem with a standard case, however, using a SuperMicro server case, in both instances some power connectors were not reachable. With the SC932T-760B case, the power supply only had a quad 12v plug for the CPU, which means it would only support a dual core processor. The SC836A-1200B has a dual quad 12v plug which would support quad core.

The SuperMicro case uses a proprietary power/reset/harddrive cable, if you do not use a SuperMicro server motherboard, you may need to purchase the cable fan-out adapter. Additionally, in the SC932T-760B case, I had to manually extend the power switch cable.

For the 2.5” optional bays, you will need to purchase drive sliders for the SC836A-1200B case to hold those drives. SuperMicro has parts available for this purpose: MCP-220-00024-01 and MCP-220-00007-01.

As a final note, the SC836A-1200B case was not assembled properly in my instance. I had to disable the alarm for a disconnected case fan; the backplane handles 4 fans but the new case design (released in June 09) only uses 3 fans.

SC836A-1200B front bezel.

SC836A-1200B top of case.

SC836A-1200B motherboard, Adaptec, quad port NIC

SC836A-1200B front view

SC932T-760B front bezel

SC932T-760B top view of case

SC932T-760B motherboard, RAID, NIC

SC932T-760B SATA cables and 4 fans

The project behind this article was to create a cheap iSCSI SAN for a backup to disk solution. The requirements of the project were foremost price, second were space, and third were performance. Amazingly, or should I say surprisingly, all three requirements were met, and in a short amount of time! In another article I will explain the custom server and the issues I had building a custom server system that could support large (10TB+) disk arrays. On to iSCSI…

Installing RHEL

For those that have installed RHEL, it is a fairly simple process, easy enough for any end user really, with the exception of partitioning if RHEL can’t figure out the correct disk order. Do what you will, however, for this project I installed RHEL with no firewall and disabled SELinux. If you are building a test machine, make sure you have enough space available for an additional partition to be served by IET.

Note: if you have no Internet connection while installing RHEL or you do not want to enter your installation key (subscription code), you may activate the RedHat Network (RHN) connection after you have completed your installation. To do so, from a terminal window, run rhn_register.

Configuring and Preparing RHEL for IET

In order to compile IET, you must configure RHEL with additional applications, another download repository, and kernel development files. First, I would suggest running a “yum update” from a terminal window to update your RHEL. As a side note, all of these commands assume you are root. Be sure to configure network connectivity, especially to the Internet.

Extra Packages for Enterprise Linux (EPEL)

Next, let’s configure RHEL to use an additional download repository, EPEL. From the web browser in RHEL, browse to the following link:
http://fedoraproject.org/wiki/EPEL

Click on 1.1 Getting Started with EPEL, then click on How to use EPEL. Then under the Using EPEL heading, click on the RHEL 5 link which allow you to download an RPM to configure the EPEL on your RHEL.

For those that are unaware, EPEL is a repository that contains considerably more applications ready for installation through “yum” or the GUI package manager. The EPEL project is maintained by Fedora, which is essentially RedHat’s open source Linux.

Installing Development Files

In order to compile IET on your RHEL, you will need the following development files installed. This task is easy as “yum” will do all the work for you. From a terminal window, run the following commands:

“yum install openssl-devel”
“yum install kernel-devel”
“yum install kernel-PAE-devel” (32bit RHEL only)
“yum install gcc”

Although not required, since I am a Windows administrator by trade, I have been rebooting my RHEL server after I completed these updates.

Compile IET

Using your web browser, visit http://iscsitarget.sourceforge.net/ and download the latest version of IET, which at the time of this article is 0.4.17. Once downloaded, extract the files to your Desktop and open a terminal window to that location. Such as /~/Desktop/iscsitarget/.

From the iscsitarget folder in the terminal window, run “make”, verify for success and then run “make install”. Assuming all went well, you should now have the IET installed on your RHEL.

Configure IET

Let’s configure the IET application to serve a partition. Although in this example I will use a SCSI disk device, you can use the LVM to create your own unique disks and serve those through IET.

The configuration is fairly simple, replaced with your favorite editor tool from the terminal window, type “nano /etc/ietd.conf”. Until you feel like diving deeper into the IET world, you will only modify two lines in this file, the Target and Lun lines.

Modify the file to look similar to the following:

Target iqn.2009-07.test:iscsi.disk
     Lun 0 Path=/dev/sdb,Type=fileio

You may modify the iSCSI target name, just make sure it follows the iSCSI standard. Additionally, you will need to replace the path statement with the path to your disk. To add more than one Target to the same server, just duplicate those lines, such as:

Target iqn.2009-07.test:iscsi.diska
     Lun 0 Path=/dev/sdb,Type=fileio

Target iqn.2009-07.test:iscsi.diskb
     Lun 0 Path=/dev/sdc,Type=fileio

Starting IET

As you probably know, in RHEL “services” reside within the /etc/init.d/ directory. Using the following command from a terminal window we will start the iscsitarget service:

/etc/init.d/iscsi-target start

To stop the service, run:

/etc/init.d/iscsi-target stop

Testing IET

For simplistic testing of the iSCSI target connectivity, I would suggest using the Microsoft iSCSI software initiator found at: http://www.microsoft.com/iscsi/. Within the Microsoft iSCSI Initiator, add the IP address of the RHEL server to the discovery list and then refresh the available targets. Once you connect to a target/LUN, you will need to open Disk Management to initialize and/or format the disk.

Depending on the hardware set up, you should see fairly quick speeds. For a simple speed test, copy a large file, such as a 3GB file to the iSCSI disk from the Windows machine. In my tests, using an Adaptec 31605 with 1.5TB Seagate drives, a 3GB file would take around 50 seconds to complete – roughly 1GB / minute.

Auto Start IET

If you want the IET to automatically start at system boot up, use the following commands from a terminal window:

“chkconfig –add iscsi-target”
“chkconfig iscsi-target on”

That is it for now…if you have questions, let me know.

As time progresses, I will update an article with troubleshooting tips I have learned while configuring around 5 of these systems. Additionally, if you are really interested in IET, I would suggest joining the Nabble forum group at: http://www.nabble.com/iSCSI-Enterprise-Target-f4401.html