TARGET_UNSOLICITED	26-May-2006	Bob Russell

How the UNH target deals with unsolicited DataOut PDUs
as part of a SCSI WRITE Command from the initiator.


When the rx_thread of the iscsi target receives a SCSI command PDU
(opcode 0x01) from the initiator, it calls the function handle_cmnd()
which sets up and initializes a new command structure of type
"struct iscsi_cmnd" (done in the common function get_free_command()).

Handle_cmnd() sets the "unsolicited_data_present" flag in this command to 1
if the F bit of the SCSI command PDU is 1, because that means that the command
must be a WRITE command of some kind, and also means that the initiator will
send unsolicited DataOut PDUs immediately after this SCSI command PDU in the
input stream.  This flag will remain 1 until all unsolicited DataOut PDUs for
this command have been read by the target's rx_thread.  If the F bit is 0
(either a READ command of some kind, or a WRITE but with no unsolicited
DataOut PDUs following it), then "unsolicited_data_present" is set to 0.
The case we are discussing here is a WRITE command with unsolicited DataOut
PDUs following it, so the "unsolicited_data_present" flag in "struc iscsi_cmnd"
is set to 1.

If the command is a WRITE command with immediate data attached to it,
this data is dealt with by calling the function save_unsolicited_data().
This function is also called to deal with the data in unsolicited DataOut PDUs,
and will be discussed with them later.

Handle_cmnd() then calls the function check_cmd_sn() to check the command for
ordering, and if it is in-order, the "state" field is set to ISCSI_AWAIT_BUFFER
(if it is out of order, the "state" is set to ISCSI_AWAIT_BUFFER_QUEUE).
Handle_cmnd() then calls the function ack_sent_cmnds() to add this
command structure to the end of the session's "active_command_list".

As its last step, the handle_cmnd() calls the function deliver_to_midlevel(),
which in turn calls the function rx_cmnd() in the SCSI midlevel in order
to hand the SCSI command off to the SCSI midlevel.  This midlevel is simulated
in our target, in the file scsi_target.c, and operates asynchronously to the
rx_thread and tx_thread which are in the iSCSI front-end.  Note that rx_cmnd()
simply puts builds a midlevel command structure of type "Target_Scsi_Cmnd"
and adds it to the end of the midlevel's "cmd_queue", then wakes up the SCSI
midlevel's scsi_target_process_thread -- it does not do any processing of the
command at this time.

The rx_thread is finished with the SCSI command at this point, and it loops
back to wait for the next input PDU from the initiator.  So we have two
asynchronous events which can happen next:

1) either an unsolicited DataOut PDU can be received by the rx_thread

2) or the SCSI midlevel's scsi_target_proces_thread gets control and sets up
   the buffers to receive all the WRITE data that the initiator wants to send
   as part of this SCSI command.

If the rx_thread receives an unsolicited DataOut PDU (opcode 0x05), it calls
the function handle_data() which calls the common function search_tags()
to find the corresponding SCSI command structure "struct iscsi_cmnd" set up
earlier in the session's "active_command_list" (to find the matching command,
search_tags() uses information in this DataOutPDU (i.e., the Initiator Task
Tag (ITT) and the Target Transfer Tag (TTT))).

Handle_data() then checks the "unsolicited_data_present" flag in this command
and if it is set, it calls the function save_unsolicited_data() (which was
also called if the SCSI command PDU itself had immediate data attached).
Save_unsolicited_data() sets up a buffer structure of type "struct udata_list"
and reads the data payload from the unsolicited DataOut PDU (or from the SCSI
command PDU) into a data buffer pointed to by this structure; it then adds
this buffer to the end of the command's "unsolicited_data_list".  In the case
of iSER/RDMA, the data payload has already been read into a data buffer by the
RDMA layer, so the unsolicited data buffer structure is just set to point to
that data buffer directly.  Notice that at this time, the immediate and/or
unsolicited data is read into a buffer that has NOT been allocated by the SCSI
midlevel.

The last thing that save_unsolicited_data() does is call update_after_read(),
which checks the F bit in the header of the unsolicited DataOut PDU (or the
SCSI Command PDU).  If this F bit is set, it means no more unsolicited
DataOut PDUs follow this PDU, and the "unsolicited_data_present" flag in this
command is reset to 0 at this time; otherwise this flag remains set to 1.

Completion of save_unsolicited_data() also completes handle_data()'s
processing of the unsolicited DataOut PDU.

The SCSI midlevel's scsi_target_proces_thread is responsible for processing
the SCSI command.  If the command is a READ, this thread must actually read
the data from the device and put it into memory buffers, then give those
buffers back to the iSCSI front-end which will send them to the initiator in
DataIn PDUs.  If the command is a WRITE, the midlevel thread must set up empty
buffers and then give those buffers back to the iSCSI front-end which will
fill them up with data received from the initiator in DataOut PDUs.
In both cases the midlevel thread notifies the iSCSI front-end when it has set
up the buffers by calling back the function iscsi_rdy_to_xfer() in the
file iscsi_target.c (this is a call-back made from the function
hand_to_front_end() in scsi_target.c).

The function iscsi_rdy_to_xfer() changes the command's state to ISCSI_DONE if
no data needs to be transfered, to ISCSI_QUEUE_CMND_RDY if it has data to
transfer but is still not in the proper order, and to ISCSI_BUFFER_RDY if it
has data to transfer and is in the proper order so it can be processed now
(which is the case we are considering here).  This function then wakes up the
tx_thread to carry on further processing.

Notice that the midlevel buffers could become ready before the first
unsolicited DataOut PDU arrives, after the first one arrives but before
the last one arrives, or after the last one arrives.  Due to the asynchronous
nature of the threads, the code has to work in all cases.  Therefore, the
payload from the unsolicited DataOut PDUs (and the immediate data payload from
the SCSI Command PDU) is always stored in buffers that are NOT allocated by
the midlevel (since those midlevel buffers might not be set up when the
DataOut PDU arrives, and the rx_thread cannot wait since that could block the
network input indefinitely).

The completion of the actions of the midlevel (to set up the empty midlevel
buffers) and the rx_thread (to read all the unsolicited DataOut payloads into
unsolicited buffers) is synchronized by the value of the command's
"unsolicited_data_present" flag and the command's "state" field.

"unsolicited_data_present" is set to 1 until the rx_thread has finished reading
all the unsolicited DataOut PDUs, at which point it is reset to 0.

"state" is set to ISCSI_AWAIT_BUFFER until the midlevel thread has finished
setting up the midlevel buffers, at which point it is set to ISCSI_BUFFER_RDY.

Only when BOTH of these actions have been taken can the next step be taken
(by the tx_thread).

When the tx_thread finds a command in the ISCSI_BUFFER_RDY state (which can
only happen AFTER the midlevel thread has set up it's midlevel buffers), it
calls the function iscsi_tx_r2t().  The first thing iscsi_tx_r2t() does is see
if the command's "unsolicited_data_present" flag has been reset to 0.  If it
is now 0, and if the command's "unsolicited_data_list" list is not empty
(which can only happen AFTER the rx_thread has read ALL unsolicited DataOut
PDUs from the initiator), then the function send_unsolicited_data() is called.
It is this function which copies the data from the command's
"unsolicited_data_list" (which holds the buffers that were allocated and
filled when the unsolicited DataOut PDUs were read by the rx_thread) into the
midlevel buffers allocated for this command by the midlevel thread.  Once this
copying has been done, the buffers in the "unsolicited_data_list" are freed
and the "unsolicited_data_list" becomes empty, so this step is never taken
again for this command.

In the SCSI layer, "struct scsi_request" (which is defined in the file
/usr/src/linux/include/scsi/scsi_request.h, and which is made equivalent
to Scsi_Request due to the typedef in the file
/usr/src/linux/include/scsi/sg_request.h) contains three fields:

unsigned sr_bufflen		/* size of data buffer */
void *sr_buffer			/* Data buffer */
unsigned short sr_use_sg;	/* Number of pieces of scatter-gather */

When the SCSI midlevel sets up buffers, it can set them up in two different
ways:

1) it can allocate one big buffer to hold the entire request;
2) it can allocate a set of scatter-gather buffers to hold pieces of the
   request.

If it allocates only one big buffer, then it sets "sr_use_sg" to 0,
points "sr_buffer" at the base address of the single buffer, and sets
"sr_bufflen" to the total number of bytes allocated in the single buffer.

If it allocates a set of scatter-gather buffers, then it sets "sr_use_sg"
to the number of scatter-gather pieces, points "sr_buffer" at an array
containing one element of type "struct scatterlist" for each piece,
and sets "sr_bufflen" to the total number of bytes allocated in all
the pieces together.

In the UNH SCSI midlevel code, we have a single routine, called get_space(),
which allocates the midlevel buffers for all SCSI commands processed
by the UNH target.  This function always allocates scatter-gather lists,
never a single big buffer.  This involves allocating two things:
the space to hold the array of "struct scatterlist" elements,
and for each element in that array, a buffer.
When the number of elements in the array is less than or equal to
MAX_ST_LIST_SLOTS, we avoid allocating the array itself and instead
use a preallocated array that is part of the Target_Scsi_Cmnd structure
pointed to by the cmnd variable.  This preallocated array is the field
"st_list" in that structure.  The purpose of doing this is to avoid the
overhead of allocating and freeing this array for the common case of
small arrays (not more than MAX_ST_LIST_SLOTS, which is defined as 8).

Therefore, the ONLY use of the field "st_list" in the Target_Scsi_Cmnd
structure is for the purpose of avoiding allocation of small scatter-gather
arrays.  All subsequent uses of the scatter-gather lists must be done by
using the req->sr_buffer field in the Target_Scsi_Cmnd structure, because
that is the array that is actually filled in with the appropriate data (in
particular, with the addresses of the buffers), and it is also always big
enough to hold the entire scatter-gather array, not just a small array.

