		     What is going on with iSER's
			 Memory Registration?


			    Ethan A. Burns

			      2008-02-29


Memory is registered from an iser_mem_t data structure.  This is
just *another* type of buffer vector (addr/len pair structure).
Addresses are translated to physical addresses in iser_ib_reg_mr() (in
iser_ofa_kernel.c) by calling ib_dma_map_single().


There are two types of memory that we deal with:
------------------------------------------------

      - Header/immediate data buffers which are allocated from
        kmem_caches or from kmalloc.  These are used for RDMA sends
        and recvs.  These are trivial, we just use put them in a
        single iser_mem_t and register them.

      - Scatter/gather lists.  These are given to us via the SCSI
        mid-level and are used for the RDMA READ and WRITEs.  This is
        a bit more complex.  The mid-level gives us this, it basically
        is an opaque wrapper for a 'struct page'.  These need to be
        accessed via macros because they *always* seem to be
        changing.  The functions in iser_utils.c take a scatterlist
        and convert it into an iser_mem_t structure (filled with the
        sg_virt() virtual addresses of the scatterlist).  The
        iser_mem_t is then given to iser_ib_reg_mr() to be mapped
        (each buffer is mapped individually) and registered.

When doing ib_dma_map_single, we need to build an ib_phys_buf (which
is yet *another* buffer vector) which is what ib_reg_phys_mr() wants
us to pass to it.


Caching of header/immediate data buffers:
-----------------------------------------

Memory that is registered for iSER/iSCSI headers or immediate data is
cached in a 'struct iser_regd_buf_cache'.  Each connection has two
caches:

	1. For headers where each entry is exactly
	   ISER_TOTAL_HEADERS_LEN in size.
	2. For immediate data (used mostly in login messages) where
	   each entry is exactly SND_RECV_DATA_SIZE.

These two caches (which are per-connection) start out empty.  When a
buffer allocation is made, if there is not a buffer ready in one of
these caches (or if the allocation size is not consistent with one of
these caches) kmalloc allocates the buffer and it is registered with
RDMA.  If a buffer allocation is made and there is a buffer ready of
the proper size, then the buffer from the cache is used (it is already
registered).

When a buffer is freed, if it fits in one of the caches and the cache
is not full (each cache has a separate max fill value specified
per-connection) then the buffer is put on the cache, and possibly
zeroed.  If there is not an appropriate cache for the buffer size, or
the proper cache is full, then the memory is unregistered and
kfreed().  Caches are completely flushed (unregistered and kfreed)
when a connection is freed.



What should be done in the future:
----------------------------------

     There is no reason to have iser_mem_t.  This is the *exact* same
     thing as ib_phys_buf.  Eventually we should remove all these
     iser_mem_t's and just make them use ib_phys_bud's.  The only
     thing about this is that iser_mem_t's do not hold physical
     addresses.

     We may want to look into supporting two types of memory
     registrations: one for single buffers and one for scatterlists.
     This is because ib_verbs provides a nice ib_dma_map_sg function
     for us to map a scatterlist all in one go... just a thought.
