Installation notes for Linux iSCSI driver

 Copyright (C) 2001, 2002, Cisco Systems, Inc.
 maintained by linux-iscsi@cisco.com

 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or (at
 your option) any later version.

 This program is distributed in the hope that it will be useful, but
 WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 General Public License for more details.

 See the file COPYING included with this distribution for more
 details.

===============================================================================
CONTENTS
===============================================================================

  Before Installing the iSCSI Driver
  Installing And Configuring The iSCSI Driver

===============================================================================
BEFORE INSTALLING THE iSCSI DRIVER
===============================================================================

  The iSCSI drivers, README files, and example configuration files are
  available on the Linux-iSCSI homepage at:

    http://linux-iscsi.sourceforge.com

  In addition, at the website you can get information about the availability
  of new drivers, updated drivers, driver compatibility, and other relevant
  information.


===============================================================================
INSTALLING AND CONFIGURING THE iSCSI DRIVER
===============================================================================

  This section is the starting point for installing and configuring the
  iSCSI Driver for Linux and contains the following topics:

   o  Product Overview
   o  System Requirements
   o  Installation Procedures
   o  Device Names
   o  Mounting Filesystems
   o  Unmounting Filesystems
   o  Making Configuration Changes
   o  Removing the Driver
   o  Kernel upgrades
   o  Driver upgrades

----------------
PRODUCT OVERVIEW
----------------

  The iSCSI Driver for Linux provides an IP host with the ability to
  access storage through an IP network. The iSCSI driver acts as an iSCSI
  protocol initiator to transport SCSI requests and responses over an IP
  network between the IP host and an iSCSI target device such as a
  Cisco SN 5400 Series system. (The iSCSI protocol is an IETF-defined
  protocol for IP storage. For more information about the iSCSI protocol,
  refer to the IETF standards for IP storage at http://www.ietf.org.)

  Architecturally, the iSCSI driver combines with the IP host TCP/IP
  stack, network drivers, and NICs to provide the same functions as a SCSI
  or Fibre Channel adapter driver with an HBA. (See Figure 1.)


  FIGURE  1  iSCSI Driver Architecture

  ====================================================================
                                   |
             IP host OS            |         Storage host OS
           and applications        |         and applications
                                   |
  ====================================================================
                                   |
              +------------+       |           +-----------+
              |   iSCSI    |       |           |           |
              |   driver   |       |           |   SCSI    |
              |            |       |           |    or     |
    +---------+------------+       |           |    FC     |
    |        TCP/IP        |       |           |  adapter  |
    +----------------------+       |           |  driver   |
    |   Network drivers    |       |           |           |
    +----------------------+       |           +-----------+
               |                   |                 |
  =============|=====================================|================
               |                   |                 |
         +-----------+             |           +-----------+
         |    NIC    |             |           |    HBA    |
         +-----------+             |           +-----------+
               |                   |                 |
  =============|=====================================|================
               |                   |                 |
              _|_                  |                 |
            _(   )_                |           +-----------+
          _(  IP   )_              |           |  SCSI or  |
         (_ network _)             |           | FC media  |
           (_     _)               |           +-----------+
             (_ _)                 |                 |
               |                   |                 |
      +------------------+         |        +------------------+
      |      iSCSI       |         |        |      Storage     |
      |      target      |         |        +------------------+
      |     interface    |         |        |  ___  ___  ___   |
      +------------------+         |        | (___)(___)(___)  |
               |                   |        | (___)(___)(___)  |
               |                   |        | (___)(___)(___)  |
      +------------------+         |        | (___)(___)(___)  |
      |     Storage      |         |        +------------------+
      +------------------+         |
      |  ___  ___  ___   |         |
      | (___)(___)(___)  |         |
      | (___)(___)(___)  |         |
      | (___)(___)(___)  |         |
      | (___)(___)(___)  |         |
      +------------------+         |
                                   |
  ====================================================================

  The iSCSI driver provides a transport for SCSI requests and responses for
  storage devices; however, instead of providing a transport for directly
  attached devices, the driver transports the SCSI requests and responses
  between the IP host and an iSCSI target system via an IP network.
  The iSCSI target system, in turn, transports SCSI requests and responses
  between it and the storage devices attached to it.

  Once the iSCSI driver is installed and started, the IP host will proceed
  with a discovery process for iSCSI storage devices as follows:

  1.  The iSCSI driver requests available iSCSI targets from the
      iSCSI target system.

  2.  The iSCSI target system sends available iSCSI target names to
      the IP host.

  3.  The IP host logs in to the iSCSI targets.

  4.  The iSCSI target system accepts the IP host login.

  5.  The IP host queries targets for device information.

  6.  Targets respond with device information.

  7.  The IP host creates a table of internal devices.

  The iSCSI Driver for Linux provides IP access to a maximum of sixteen
  remote SCSI targets.  Each target will be probed for up to 256 LUNs,
  until the Linux kernel's limit of SCSI devices has been reached.

-------------------
SYSTEM REQUIREMENTS
-------------------

  This driver requires either a 2.2.x Linux kernel version of 2.2.16 or
  later, or a 2.4.x Linux kernel version of 2.4.16 or later. SMP is only
  supported on 2.4.x kernels. Compilation will require the kernel header
  files matching the kernel version you wish to use the driver with. 
  Kernels released after June 7, 2002 may or may not work with this driver,
  depending on what if any changes have been made to the kernel's SCSI
  midlayer code.

  As of June 6, 2002 there are several issues with the Linux kernel
  code that can cause problems when using SCSI devices (including iSCSI
  devices).  Linux kernels released after this date may or may not have
  fixed these problems.

  o  Linux kernels 2.2.16 through 2.2.20 and 2.4.0 through 2.4.18 are known
     to have a problem in the SCSI error recovery process. In some cases, a
     successful device reset may be ignored and the SCSI layer will continue
     on to the later stages of the error recovery process.

     The problem occurs when multiple SCSI commands for a particular device
     are queued in the low-level SCSI driver when a device reset occurs. Even
     if the low-level driver correctly reports that all the commands for
     the device have been completed by the reset, Linux will assume only
     one command has been completed and continue the error recovery
     process. (If only one command has timed out or failed, Linux will
     correctly terminate the error recovery process following
     the device reset.)

     This action is undesirable because the later stages of error recovery
     may send other types of resets, which can affect other SCSI
     initiators using the same target or other targets on the same
     bus. It is also undesirable because there are more serious bugs
     in the later stages of the Linux SCSI error recovery process.

     The Linux iSCSI driver now attempts to avoid this problem by
     replacing the usual error recovery handler for SCSI commands
     that timeout or fail.


  o  Linux kernels 2.2.16 through 2.2.20 and 2.4.0 through 2.4.2 may take
     SCSI devices offline after Linux issues a reset as part of the
     error recovery process.  Taking a device offline causes all I/O to the
     device to fail until the HBA driver is reloaded.

     After the error recovery process does a reset, it sends a SCSI
     Test Unit Ready command to check if the the SCSI target is operational
     again. If this command returns SCSI sense data, instead of correctly
     retrying the command, Linux will treat it as a fatal error, and
     immediately take the SCSI device offline.

     The Test Unit Ready will almost always be returned with sense data
     because most targets return a deferred error in the sense data of the
     first command received after a reset. This is a way of telling
     the initiator that a reset has occurred. Therefore, the affected 
     Linux kernel versions almost always take a SCSI device offline
     after a reset occurs.

     This bug is fixed in Linux kernels 2.4.3 and later.

     The Linux iSCSI driver now attempts to avoid this problem by
     replacing the usual error recovery handler for SCSI commands
     that timeout or fail.


  o  Linux kernels 2.2.16 through 2.2.21 and 2.4.0 through 2.4.18 appear to
     have problems when SCSI commands are completed with a check condition
     containing sense data. This can result in applications receiving I/O
     errors, short reads or short writes.

     After receiving a SCSI command with sense data, the Linux SCSI
     midlayer checks if some sectors of an I/O request have been
     transferred and informs the application layer if an I/O request has
     partially completed. The SCSI midlayer then retries the request for
     any sectors that have not yet been transferred. Because of the partial
     I/O request completion, applications may receive short reads or
     writes.

     All UNIX applications should handle these conditions, but there may be
     some applications which do not.

     There are also some cases where the application receives an I/O error
     rather than a short read or write.  The exact cause of the I/O errors
     is still being investigated, but it appears to be a bug in the Linux
     kernel's SCSI layer or block device layer.


  o  Linux kernels 2.2.16 through 2.2.21 and 2.4.0 through 2.4.18 may crash
     on a NULL pointer if a SCSI device is taken offline while one of the
     Linux kernel's I/O daemons (e.g. kpiod, kflushd, etc.) is trying to do
     I/O to the SCSI device. The exact cause of this problem is still being
     investigated.

     Note that some of the other bugs in the Linux kernel's error recovery
     handling may result in a SCSI device being taken offline, thus
     triggering this bug and resulting in a Linux kernel crash.


  o  Linux kernels 2.2.16 through 2.2.21 running on uniprocessors may
     hang if a SCSI disk device node is opened while the Linux SCSI
     device structure for that node is still being initialized.

     This occurs because the sd driver which controls SCSI disks
     will loop forever waiting for a device busy flag to be cleared
     at a certain point in the open routine for the disk device.
     Since this particular loop will never yield control of the 
     processor, the process initializing the SCSI disk device is
     not allowed to run, and the initialization process can never 
     clear the device busy flag which the sd driver is constantly 
     checking.

     A similar problem exists in the SCSI generic driver in some 
     2.4 kernel versions.  The sg driver may crash on a bad 
     pointer if a /dev/sg* device is opened while it is being 
     initialized.

----------
INSTALLING
----------

  1. Use tar(1) to uncompress the source archive into a directory of
     your choice. The archive will contain a subdirectory
     corresponding to the archive name.

       cd /usr/src
       tar xvzf /path/to/linux-iscsi-<version>.tgz
       cd linux-iscsi-<version>

  2. Compile the iSCSI driver. If your kernel sources are not in the
     usual place, add 'TOPDIR=/path/to/kernel' or edit the definition
     of TOPDIR in the Makefile.

       make

  3. As root, install the driver. If you are currently using the iSCSI
     driver, first unmount all iSCSI devices and unload the old iSCSI
     driver. If your Linux distribution includes an iSCSI driver, it
     may be necessary to uninstall that package first.

       make install

  4. Update /etc/iscsi.conf to include the IP addresses for your
     iSCSI targets. A sample configuration file might include entries
     like this:

       DiscoveryAddress=192.168.10.94
       Target,Lun=0,0

     The iscsi.conf man page has a more detailed description of the
     configuration file format.  To read the man page, type:

     man iscsi.conf

  5. Manually start iSCSI services to test your configuration. On
     Redhat systems, run:

       /etc/rc.d/init.d/iscsi start

     If there are problems loading the iSCSI kernel module, diagnostic
     information will be placed in /var/log/iscsi.log.

     The iSCSI initialization will report information on each detected
     device to the console or in dmesg(8) output. For example:

       Vendor: SEAGATE   Model: ST39103FC         Rev: 0002
       Type:   Direct-Access                      ANSI SCSI revision: 02
       Detected scsi disk sda at scsi0, channel 0, id 0, lun 0
       SCSI device sda: hdwr sector= 512 bytes.
                                       Sectors= 17783240 [8683 MB] [8.7 GB]
       sda: sda1
       scsi singledevice 0 0 0 1

     Normal disk commands like fdisk, mkfs, and fsck will work on the
     iSCSI devices like a local drive.

     /proc/scsi/iscsi will contain a file (the controller number) that
     contains information about the iSCSI devices.

     To manually stop the iSCSI driver enter:

       /etc/rc.d/init.d/iscsi stop

  6. Modify your init scripts to manage iSCSI. If you are using a
     non-RedHat Linux distribution you may need to edit your boot
     scripts to properly run the iSCSI setup script. You may also need
     to change the order of the boot script to ensure that iSCSI
     services are started after the network has been initialized.

  7. List your iSCSI partitions in /etc/fstab.iscsi. It has the same
     format as /etc/fstab. The init scripts will mount and unmount
     these partitions automatically.  See the next few sections
     of this document for more details on how to do this correctly.

------------
DEVICE NAMES
------------

  Because Linux assigns SCSI device nodes dynamically whenever
  a SCSI logical unit is detected, the mapping from device nodes
  (e.g /dev/sda, /dev/sdb) to iSCSI targets and logical units may 
  vary.  

  Variations in process scheduling and network delay may result in 
  iSCSI targets being mapped to different SCSI device nodes every time
  the driver is started.  Because of this variability, configuring
  applications or operating system utilities to use the standard 
  SCSI device nodes to access iSCSI devices may result in SCSI
  commands being sent to the wrong target or logical unit.

  To provide a more reliable namespace, the iSCSI driver will scan the
  system to determine the mapping from SCSI device nodes to iSCSI
  targets, and then create a tree of directories and symbolic links
  under /dev/iscsi to make it easier to use a particular iSCSI
  target's logical units.

  Under /dev/iscsi, there will be a directory tree containing 
  subdirectories for each iSCSI bus number, each target id number 
  on the bus, and each logical unit number for each target. For
  example, the whole disk device for bus 0, target id 0, LUN 0 would
  be /dev/iscsi/bus0/target0/lun0/disk 

  In each logical unit directory there will be a symbolic link for 
  each SCSI device node that may be connected to that particular 
  logical unit.  These symbolic links are modeled after the Linux 
  devfs naming convention.

  The symbolic link 'disk' will map to the whole-disk SCSI device node
  (e.g. /dev/sda, /dev/sdb, etc). 

  The symbolic links 'part1' through 'part15' will map to each
  partition of that SCSI disk (e.g. /dev/sda1, dev/sda15, etc).
  Note that these links will exists regardless of the number
  of disk partitions.  Opening the partition devices will result 
  in an error if the partition does not actually exist on the disk.

  The symbolic link 'mt' will map to the auto-rewind SCSI tape device 
  node for this LUN (e.g. /dev/st0), if any.  Additional links for 
  'mtl', 'mtm', and 'mta' will map to the other auto-rewind devices
  (e.g. /dev/st0l, /dev/st0m, /dev/st0a), regardless of whether these
  device nodes actually exist or could be opened.

  The symbolic link 'mtn' will map to the no-rewind SCSI tape device 
  node for this LUN (e.g. /dev/nst0), if any.  Additional links for 
  'mtln', 'mtmn', and 'mtan' will map to the other no-rewind devices
  (e.g. /dev/nst0l, /dev/nst0m, /dev/nst0a), regardless of whether 
  those device nodes actually exist or could be opened.

  The symbolic link 'cd' will map to the SCSI cdrom device node
  for this LUN (e.g. /dev/scd0), if any.

  The symbolic link 'generic' will map to the SCSI generic device 
  node for this LUN (e.g. /dev/sg0), if any.

  Because the symlink creation process must open all of the SCSI
  device nodes in /dev in order to determine which nodes map to
  iSCSI devices, you may see many modprobe messages logged to syslog
  indicating that modprobe could not find a driver for a particular
  combination of major and minor numbers.  This is harmless, and can
  be ignored.  The messages occur when Linux is unable to find a
  driver to associate with a SCSI device node that the iSCSI daemon
  is opening as part of it's symlink creation process.
	
--------------------
MOUNTING FILESYSTEMS
--------------------

  Because the Linux boot process normally mounts filesystems listed in
  /etc/fstab before the network is configured, adding mount entries in
  iSCSI devices to /etc/fstab will not work. The script iscsi-mountall
  will manage the checking and mounting of devices listed in the file
  /etc/fstab.iscsi which has the same format as /etc/fstab. This
  script is automatically invoked by the iSCSI startup script.

  Because of the variability of the mapping between SCSI device nodes
  and iSCSI targets, instead of directly mounting SCSI device nodes, 
  it is recommended to either mount the /dev/iscsi tree symlinks,
  mount filesystem UUIDs or labels (see man pages for mke2fs, mount, 
  and fstab), or use logical volume management (see Linux LVM) to
  avoid mounting the wrong device due to device name changes resulting
  from iSCSI target configuration changes or network delays.

----------------------
UNMOUNTING FILESYSTEMS
----------------------
	
  It is very important to unmount all filesystems on iSCSI devices
  before the iSCSI driver stops.  If the iSCSI driver stops while
  iSCSI devices are mounted, buffered writes may not be committed to
  disk and filesystem corruption may occur.
	
  Since Linux will not unmount filesystems that are being used by a
  running process, before iSCSI devices can be unmounted, any
  processes using those devices must be stopped (see fuser(1)).

  To avoid filesystem corruption, the iSCSI shutdown script will
  automatically kill all processes using devices in /etc/fstab.iscsi,
  first by sending them SIGTERM, and then by sending any remaining
  processes SIGKILL. It will then unmount all iSCSI filesystems and
  kill the iSCSI daemon, terminating all connections to iSCSI devices.

  *** WARNING: filesystems not listed in /etc/fstab.iscsi may not be
               automatically unmounted!

-----------
MAINTENANCE
-----------

  Making changes to your storage configuration, including adding or
  removing targets or LUNs, remapping targets, or modifying target
  access, may change how the devices are presented to the operating
  system. This may require corresponding changes in your iSCSI driver
  configuration and /etc/fstab.iscsi.

  For example, adding a new storage volume may change the order of
  device discovery, thus changing the operating system's numbering
  of existing devices. Applications running on the host may
  require modification to appropriately access the current drives.

  Note: if a host's iSCSI configuration file contains an IP address
  that does not exist or is unreachable, the iSCSI driver will not be
  able to login and discover targets at that address, and the driver
  will not discover targets at IP addresses that appear later in the
  configuration file until a login to the failing address completes
  successfully.

  It is recommended to use filesystem UUIDs or labels (see man pages
  for mke2fs, mount, and fstab) or logical volume management (see
  Linux LVM) to avoid the configuration problems associated with
  changing device names.

  In general, the following steps are normally required when
  reconfiguring iSCSI storage:

  1. Unmount any file systems and stop any applications using iSCSI
     devices.

  2. Stop the iSCSI driver.

       /etc/rc.d/init.d/iscsi stop

  3. Make the appropriate changes to the iSCSI driver configuration
     file. Remove any references to iSCSI discovery addresses that
     have been removed, or that no longer have valid targets for this
     host.

  4. Modify /etc/fstab.iscsi and application configurations as
     appropriate.

  5. Restart the iSCSI driver or, optionally, reboot the host.

       /etc/rc.d/init.d/iscsi start

  Failure to appropriately update the iSCSI configuration using the
  above procedure may result in a situation that prevents the host
  from accessing iSCSI storage resources.


-------
REMOVAL
-------

  1. Change to the driver source directory from installation step 1.

  2. While logged in as root, run:

       make remove

     This deletes the appropriate files from /lib/modules and
     /usr/local/sbin.  The configuration files in /etc are not
     deleted, since they will be needed if another driver version
     is installed later.

  3. Back up one directory and delete the source code:

       cd ..
       rm -fr linux-iscsi-<ver>


---------------
KERNEL UPGRADES
---------------

  Because the driver contains a Linux kernel module, the driver must
  be rebuilt and reinstalled if you make any changes to your Linux
  kernel.  To remove, rebuild, and reinstall the driver, do the
  following while logged in as root:

       make remove
       make clean
       make
       make install


---------------
DRIVER UPGRADES
---------------

   To upgrade the iSCSI driver to the latest version, first deinstall
   the existing package following the steps given in the REMOVAL
   section of the README for the driver you currently have installed.

   Upon successful deinstallation, install the latest driver as
   described in the INSTALLING section of the README.
	
