SUN iSCSI Initiator

Obviously you need to have your network configured properly prior to working with the iSCSI initiator or targets. In this example, I was working with the iSCSI initiator and target in a lab environment, with the host (target) being 192.168.0.18 and the client (initiator) being 192.168.0.19. Secondly, you will need the iSCSI initiator software to be installed from Package Manager – easiest way is to search for iscsi and ensure all four of those packages are installed.

For this example we will use Send Targets for discovering iSCSI targets/paths:

iscsiadm add discovery-address 192.168.0.18:3260
iscsiadm modify discovery --sendtargets enable

Now that you have added the discovery address of the iSCSI target and enabled the discovery “Send Targets”, check to see if your target was discovered:

iscsiadm list target

If you saw the targets you were expecting, the next step is to create the device links between iSCSI and your system – so you can see the iSCSI LUN as a disk on your system.

devfsadm -i iscsi

Check the available disks on your system using the format command. After running the format command, hit CTRL-C to bounce back to the terminal prompt.

Using / Mounting the iSCSI LUN

Now that you have the iSCSI initiator working properly and the iSCSI targets discovered, lets actually use the new iSCSI disk with ZFS.

First we need to create the ZFS pool using the new disk. If you forgot the name of the new disk, you can learn it using the format command. If it is anything like the one on my OpenSolaris box, the name is rather large, (c0t600144F0F9A00C0000004AA9510A0001d0):

zpool create diskpool c0t600144F0F9A00C0000004AA9510A0001d0

To verify the pool was created, use:

zpool list

Now we can create a ZFS file system on top of that pool:

zfs create diskpool/test

Check your new ZFS filesystem was created:

zfs list

And now you can create files within the mount point:

root@opensolaris:/diskpool/test/#

Everything you write to that mount point will be written to the iSCSI target, as one would hope.

Good luck with your performance!

Credits

http://wikis.sun.com/display/OpenSolarisInfo/How+to+Configure+an+iSCSI+Initiator

I owe some of my new-found knowledge to my friend Chuck Hechler and the remaining to Mike La Spina (http://blog.laspina.ca/). I don’t pretend to be anywhere as knowledgeable about OpenSolaris, ZFS, iSCSI, SANs, or storage in general as Mike or Chuck, however, I can create SharePoint workflow actions in VisualStudio, can they? =)

The Equipment

If you read my other posts on the IET and RedHat implementation, you would know that I created a custom server housed in a SuperMicro case. During the testing phase of that project, I learned that the IET product does not work well with the LSI MegaRAID 84016 controller, specifically when using RAID5. Unsure why, however, in OpenSolaris, I had an issue with our Adaptec 31605 controller and ended up using the LSI MegaRAID 84016E controller. Works like a charm. I mention this as you will need to verify that the equipment you are going to use with OpenSolaris should be supported by OpenSolaris (http://www.sun.com/bigadmin/hcl/).

If you want more information on the hardware specifications for the system I used, please review my Custom Server Case article.

OpenSolaris Configuration

Using OpenSolaris to serve as an iSCSI target is surprisingly easy to configure. I say surprisingly because the same implementation in RedHat was thoroughly difficult, both due to a learning curve and due to many complications with functionality. On to OpenSolaris!

As I designed this system, I was focused on using ZFS as the sole solution for disk management and the file system. In other words, for my boot disks, I created a ZFS mirror and for the iSCSI LUN disks I created a ZFS raidz2.

Installation

For the most part, the default installation settings were used. Too be honest, the options during installation of OpenSolaris are limited. If you do not want the graphical interface, see my short article, Disable GUI, for disabling it after installation.

ZFS

Boot pool – mirror

Mirroring the boot partition is about as difficult as it gets with this article. I did not figure out how to mirror the boot disk on my own. Using the steps in the following blog post, I managed to mirror my ZFS boot partition, rpool.

http://darkstar-solaris.blogspot.com/2008/09/zfs-root-mirror.html
If you do not know the devices names of your drives, you can use “# format” to list the drives/devices on your system.

After the mirroring has been completed, display the status of the pool using the command and output below:

# zpool status
  pool: rpool
 state: ONLINE
 scrub: none requested
config:

 NAME        STATE     READ WRITE CKSUM
 rpool       ONLINE       0     0     0
   mirror    ONLINE       0     0     0
     c8d0s0  ONLINE       0     0     0
     c9d0s0  ONLINE       0     0     0

errors: No known data errors

iSCSI pool – raidz2

After reading quite a few posts on best practices for ZFS pools, I propose that you use a raidz set per controller. In my instance, I had a 16 port controller and all 16 drives for the iSCSI pool were connected to this controller. Using the following commands I created a raidz2 pool with 2 spares:

# zpool create diskpool raidz2 c7t0d0 c7t1d0 c7t2d0 c7t3d0 c7t4d0 c7t5d0 c7t6d0 c7t7d0 c7t8d0 c7t9d0 c7t10d0 c7t11d0 c7t12d0 c7t13d0
# zpool add diskpool spare c7t14d0 c7t15d0

That is it; we just created a large disk pool of 16 disks, 14 in a dual-parity set for volumes and 2 for spares. Easy? Yes.

Volumes

Best practice for creating ZFS volumes depends on your perspective. In my case, the iSCSI target will be used with VMware and I will create a new ZFS volume and iSCSI view for each VMware machine (not host). Follow the command below to create the base mountpoint in the ZFS pool:

# zfs create diskpool/iscsi

Next we will create a volume for sharing as an iSCSI target. As I stated above, I create one volume per VM. Also, notice the “-b 64K”, which helps with partition alignment in VMware. The command “-V 40G” creates a volume with a maximum size of 40 gigabyes – keep in mind that ZFS will thin provision the file system.

# zfs create -s -b 64K -V 40G diskpool/iscsi/lun0_vm

Display the pool and the ZFS volumes as shown below:

# zpool status
  pool: diskpool
 state: ONLINE
 scrub: resilver completed after 0h32m with 0 errors on Wed Sep  2 10:52:54 2009
config:

 NAME           STATE     READ WRITE CKSUM
 diskpool       ONLINE       0     0     0
   raidz2       ONLINE       0     0     0
     c7t0d0     ONLINE       0     0     0
     c7t1d0     ONLINE       0     0     0
     c7t2d0     ONLINE       0     0     0
     c7t3d0     ONLINE       0     0     0
     c7t4d0     ONLINE       0     0     0
     c7t5d0     ONLINE       0     0     0�
     c7t6d0     ONLINE       0     0     0
     c7t7d0     ONLINE       0     0     0
     c7t8d0     ONLINE       0     0     0
     c7t9d0     ONLINE       0     0     0�
     c7t10d0    ONLINE       0     0     0
     c7t11d0    ONLINE       0     0     0�
     c7t12d0    ONLINE       0     0     0
     c7t13d0    ONLINE       0     0     0�
 spares
   c7t14d0      AVAIL
   c7t15d0      AVAIL

errors: No known data errors

  pool: rpool
 state: ONLINE
 scrub: none requested
config:

 NAME        STATE     READ WRITE CKSUM
 rpool       ONLINE       0     0     0
   mirror    ONLINE       0     0     0
     c8d0s0  ONLINE       0     0     0
     c9d0s0  ONLINE       0     0     0

errors: No known data errors

# zfs list
NAME                             USED  AVAIL  REFER  MOUNTPOINT
diskpool                         9.9G  6.76T  51.1K  /diskpool
diskpool/iscsi                   9.9G  6.76T  48.5K  /diskpool/iscsi
diskpool/iscsi/lun0_vm     9.9G  6.76T   9.9G  -
rpool                           10.8G  62.1G  77.5K  /rpool
rpool/ROOT                      3.18G  62.1G    19K  legacy
rpool/ROOT/opensolaris          3.18G  62.1G  3.04G  /
rpool/dump                      3.75G  62.1G  3.75G  -
rpool/export                     114M  62.1G    21K  /export
rpool/export/home                114M  62.1G    21K  /export/home
rpool/export/home/vcssan         114M  62.1G   114M  /export/home/vcssan
rpool/swap                      3.75G  65.7G   101M  -

Network

There are a few approaches to spreading the load across multiple links, my favorite is LACP, which is what I used – see my article on LACPBONDING. The other option that may arise in the planning phase is IPMP. However, I was unable to overcome the issue of having more than one inbound interface. If you are knowledgeable with source addressing, you will have better luck than I did.

I aggregated 4 gigabit links together using LACP and a Dell PowerConnect 5448 switch.

iSCSI

During the design phase, I tested both of SUN’s iSCSI targets, the standard iscsitadm and COMSTAR iSCSI. At this point, you can choose one based on word of mouth, or you can try both and use the one that works the best for you. I ended up using COMSTAR.

iscsitadm

The first configuration change we will make defines which interfaces the iSCSI target software will use. We accomplish this by creating a target group:

# iscsitadm create tpgt 1
# iscsitadm modify tpgt –i 192.168.0.101 1

Because I am using link aggregation there is only one IP address to add to the group. If you decide to use multi-pathing as provided by VMware, you will need to add all of those IP addresses to the group.

Are you ready for the next statement, it is very complex…well, maybe not.

# zfs set shareiscsi=on diskpool/iscsi/lun0_vm

That’s it; the ZFS volume is now served through the iSCSI target.

COMSTAR

On a default installation of OpenSolaris, you will need to install the COMSTAR iSCSI software package. Open the Package Manager and search for iSCSI, choose to install all of them, specifically the SUNWiscsit package.

After you have installed the software package, you will need to enable the COMSTAR iSCSI service:

# svcadm enable -r svc:/network/iscsi/target:default

I am having a vague recollection that you may need to restart OpenSolaris before continuing…

The next configuration we will make defines which interfaces the iSCSI target software will use. We accomplish this by creating a target group and target:

# itadm create-tpg iscsi0 192.168.0.101
# itadm create-target –t iscsi0

Now, we will share the ZFS volume through the COMSTAR iSCSI target – this is a two step process:

# sbdadm create-lu /dev/zvol/rdsk/diskpool/iscsi/lun0_vm

Now let’s check for the new logical unit and remember the GUID… as we will use it in the next command:

# sbdadm list-lu

Found 1 LU(s)

       GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f08e6c0f0000004a8331d70002      42949607424      /dev/zvol/rdsk/diskpool/iscsi/lun0_vm

Now we will add the view to share the ZFS volume through iSCSI:

# stmfadm add-view 600144f08e6c0f0000004a8331d70002

That’s it!

I’m out….

Creating the Link Aggregate

If your configuration was similar to mine, you will already have your interfaces plumbed – which means they cannot be added to the link aggregate until they are unplumb’ed.

Checking for Plumbing

If you try to create an aggregate with plumbed interfaces, you will receive a “dladm: create operation failed: link busy” error. You can check if they have been plumbed as they will show in the list for “ifconfig -a”:

e1000g0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 8
 inet 0.0.0.0 netmask 0
 ether 0:15:17:b8:47:a8
e1000g1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 9
 inet 0.0.0.0 netmask 0
 ether 0:15:17:b8:47:a9
e1000g2: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 10
 inet 0.0.0.0 netmask 0
 ether 0:15:17:b8:47:aa
e1000g3: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 11
 inet 0.0.0.0 netmask 0
 ether 0:15:17:b8:47:ab

If they appear, as shown above, unplumb them using the following commands:

ifconfig e1000g0 unplumb
ifconfig e1000g1 unplumb
ifconfig e1000g2 unplumb
ifconfig e1000g3 unplumb

Create the Aggregate and Plumb It!

Using the dladm tool, you can easily create the aggregate usign the following command:

dladm create-aggr -d e1000g0 -d e1000g1 -d e1000g2 -d e1000g3 1

You’lll notice a -d with each interface and a number 1 at the end. The number 1 references the aggregate number and it must start at 1, not 0.

Next we will plumb the newly created aggregate with an IP address:

ifconfig aggr1 plumb 192.168.0.101 up

Show me the Aggregate

If you are like me, then you will want to see the aggregate configured:

dladm show-aggr

You should see an output similar to the following:

LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L4       auto                 off           short       -----

Modify Aggregate for LACP

To ensure that LACP is functioning as I would hope, I modified the aggregate, changing LACPACTIVITY to active instead of off. Use the following command to modify LACPACTIVITY:

dladm modify-aggr -L active -T short 1

Now run the following command to verify the settings took:

dladm show-aggr

You should now see an output similar to the following:

LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L4       auto                 active        short       -----

Setting Aggregate IP Configuration as Persistent

To ensure that your IP address configuration for the aggregate is persistent across reboots, modify the /etc/hostname.aggr1 file and add the desired IP address. The /etc/hostname.aggr1 should contain one line consisting of the IP address:

192.168.0.101

Switch Configuration

Obviously, you must configure the switch for LACP otherwise your aggregate will fail. Each switch is different, refer to the manual for your switch.

I’m out…

Disable GUI

To disable the GUI, type the following command in a terminal window:

pfexec svcadm disable gdm

Enable GUI

To enable the GUI, type the following command at the terminal prompt:

pfexec svcadm enable gdm

I’m out…