-*- mode: text; tab-width:8; fill-column: 72 -*-
vi: set tabstop=8 textwidth=72:
------------------------------------------------------------------------
	      Building and Using UNH-ISCSI in User-Space
			    Ethan A. Burns
			      2008-02-03



The UNH-iSCSI project comes with two Makefiles: Makefile-kernel and
Makefile-user for kernel-space and user-space respectively.


		  ==================================
		  UNH-iSCSI for the Linux user-space
		  ==================================

	 In order to aid in development and debugging, the UNH-iSCSI
project can be build to run as two applications (initiator and target)
for the Linux user-space*.  To build UNH-iSCSI for user-space you need
to use the 'Makefile-user' makefile:

   # cp Makefile-user Makefile


===============
Makefile layout
===============
The user-space Makefile contains a few lists of object files that it
will use to build the imitator and target applications.  These lists
are all named XXX_obj, where XXX is a descriptive word, for example:
target_obj contains object files that are only linked with the target,
initiator_obj contains object files that are only linked with the
initiator, security_obj contains security object files that are used
by both the target and initiator, etc.

The object file lists are populated by sourcing the 'Make.inc' files
located in various sub directories (the SUBDIRS variable in the
makefile specifies which sub directories to use).  Each Make.inc file
should add, to the appropriate list, the object files that are build
from the source files contained in that directory.  For example: the
target/Make.inc file will add 'target/iscsi_cmnd.o' to the target_objs
list:

target_obj+=target/iscsi_cmnd.o

The procedure for adding a new source file into the user-space build
system is simply to add the object file (that will be built from the
new source file) to the appropriate list in the Make.inc file of the
sub directory that contains the new source file.

* In fact, there is no reason that this shouldn't be able to run in any
  POSIX environment.


======================
Building in user-space
======================
To build UNH-iSCSI in user-space, make sure that your Makefile is a
copy (or a link to) 'Makefile-user'.  UNH-iSCSI is able to run in one
of two different modes:

Traditional iSCSI mode - traditional iSCSI (RFC3720) over TCP/IP.
iSER-Assisted mode - uses iSER (RFC5046).  (Due to current hardware
	             constraints this mode does not conform directly
		     to RFC5046)

Unfortunately, the current implementation does not allow for selecting
between modes after compile-time.  In order to switch between
iSER-Assisted and Traditional iSCSI mode you must re-compile.

-----------------
Traditional iSCSI
-----------------
To build with traditional iSCSI support (with out iSER) you must make
sure that the 'datamover/' directory is not included in the SUBDIRS
list in the Makefile.  If you open the makefile in a text editor and
search for 'datamover', you should see a line near the top of the file
that looks like this:

#
# Uncomment the following line only if compiling in iSER/RDMA Support
#
SUBDIRS+=datamover

If the line is not commented out, then you need to comment it out now.

Once you have made sure that the datamover directory will not be
included in the compilation, you can simply type 'make'.

# make

-------------------
iSER-Assisted iSCSI
-------------------
To build with iSER mode, you must make sure that the 'datamover/'
directory is in the SUBDIRS list in the Makefile.  This can be done by
opening the Makefile in a text editor and searching for the string
'datamover'.  Around the top of the file you should see:

#
# Uncomment the following line only if compiling in iSER/RDMA Support
#
SUBDIRS+=datamover

Make sure that the 'SUBDIRS+=datamover' is not commented out.
Alternatively you can specify 'SUBDIRS=datamover' on the command-line
before the 'make' command (see below)

If you want to use real RDMA hardware with the OFA user-space
libraries, are done and can type 'make'

# make


If you do not want to use real RDMA hardware, instead you can use the
OSC software RNIC in order to have a fully user-space fully software
iSER-Assisted iSCSI.  To do this, you must select the value 'osc' for
the VERBS variable in the Makefile.  In a text editor, search the
Makefile for 'VERBS':

#
# Select either OSC or OFA for the verbs abstraction to use.
# This can also be set on the command line.
#
ifndef VERBS
#VERBS=osc
VERBS=ofa
endif

First, you must install the OSC software RNIC library on the computer
that you are building UNH-iSCSI on.

------------------------------------
Currently UNH-iSCSI only runs on a special modified version of the OSC
software RNIC that is hidden on an svn server at the
UNH-InterOperability lab:

svn+ssh://gladiator/var/svn/iol_iwarp/trunk

which I believe needs root access.
------------------------------------

To enable the OSC software RNIC, comment out the line that reads
'VERBS=ofa' and un-comment the line that reads 'VERBS=osc'.
Alternatively you can prepend the string 'VERBS=osc' to the command
line when you type 'make' (see below).


The following is a list of command lines that will build the two
different iSER-Assisted modes with-out the hassle of editing makefiles.

-- build with iSER mode using the OFA verbs
# SUBDIRS=datamover VERBS=ofa make

-- build with iSER mode using the OSC software RNIC
# SUBDIRS-datamover VERBS=osc make

=======================
The User-Space Programs
=======================

After building the user-space version of UNH-iSCSI there will be two
binaries:

target/unh_main_scsi_target -

 This binary runs the user-space target.  When executed it accepts
 management commands (see doc/ISCSI_MANAGE) on its standard input.
 The commands automatically have the 'target' keyword prefixed onto
 them.  For example to enable trace flags enter the command: 'force
 V=<hex string>' to the standard input of the target program.  (For a
 further description of commands, see doc/ISCSI_MANAGE).  Additionally
 an include command has been added to allow the target to source a
 command file.  The 'user-target' file has been provided as an example
 command file that can be included by the target.  Finally, text
 following a '#' character until the end of the line is interpreted as
 a commend and is ignored.

initiator/unh_main_scsi_initiator -

 This binary runs the user-space initiator.  When executed it accepts
 management commands (see doc/ISCSI_MANAGE) on its standard input.
 The commands automatically have the 'init' keyword prefixed onto them.
 In addition to management commands the initiator accepts a small set
 of extra commands that enable it to emulate a SCSI session (see below).
 Typically an input script is created and is piped into the initiator
 program.  For an example input script, refer to user-ini-local.


------------------------
Initiator Extra Commands
------------------------
Since the user-space initiator has no SCSI mid-level to drive it,
extra commands have been added to emulate a SCSI session.  There are
four different commands: do commands, the repeat command, the echo
command and the include command.

Additionally, the initiator interprets text following a '#' character
until the end of line as a comment and ignores it.

-------
do
-------
do commands are in the form "do <SCSI CDB TYPE> [blocks]" where <SCSI
CDB TYPE> is the name of a SCSI command to send to the target and
[blocks] is an optional number of blocks (which can only be specified
for read or write commands). The valid "do" commands are:

 do testunitready
 do inquiry
 do readcapacity
 do readcapacityshort
 do readcapacitylong
 do reportluns
 do modesense
 do requestsense
 do read <blocks>
 do read6 <blocks>
 do read10 <blocks>
 do read12 <blocks>
 do read16 <blocks>
 do write <blocks>
 do write6 <blocks>
 do write10 <blocks>
 do write12 <blocks>
 do write16 <blocks>

-------
repeat
-------
The repeat command allows you to repeat a "do" command or another,
nested, repeat command and log some performance measures to a file.  The
form of the repeat command is: "repeat <num_times> [output_file]" where
<num_times> is a required argument that specifies the number of times to
repeat the following "do" command, and [output_file] is an optional file
name argument that specifies which file to write the performance
measures to.  If [output_file] is not specified, then the output is
written to standard output.  The next command immediately following a
repeat command *must* be a "do" command or another, nested, repeat
command.

Performance results, in the case of nested repeat commands are only
generated for the outter-most repeat.  Also, nested repeat commands must
not specify an output file, since they do not generate any performance
values.

The performance results logged by the repeat command consist of three
columns:

 <repeat_count> <total_bytes_transferred> <time_in_seconds>

From these three measures various performance metrics can be computed,
for example:

 throughput = <total_bytes_transferred> / <time_in_seconds>
 response time for SCSI a cmnd = <repeat_count> / <time_in_seconds>


Example:
 # do a write of 1000 blocks 100 times and log to the file 'data.out'
 repeat 100 data.out
 do write 1000

-------
echo
-------
The echo command allows the script to echo textual data to an output
file.  This command allows you to output comments about the meaning of
the performance measures given from repeat commands.  The format for
the echo command is "echo <file_name> <string delimited by new line>"

Example:
 # do 100 writes of 1000 blocks but also print that to the data.out
 # file preceding the performance results so that later on we know what
 # they mean.
 echo data.out 100 wries of 1000 blocks
 repeat 100 data.out
 do write 1000

-------
include
-------
The include command allows you to source another file that contains
more initiator commands.  The form for the include command is:
"include <file_name>" where <file_name> is the name of the file to include.

Example:
 include ini-do-some-commands
