===============================================================================
				Changes in 0.97
===============================================================================
- MPI-2 one-sided communication has been implemented in the CH3 device.

- mpigdb works as a simple parallel debugger for MPI programs started
  with mpd.  New since MPICH1 is the ability to attach to running
  parallel programs.  See the README in mpich2/src/pm/mpd for details.

- MPI_Type_create_darray() and MPI_Type_create_subarray() implemented including
  the right contents and envelope data.

- ROMIO flattening code now supports subarray and darray combiners.

- Improve scalability and performance of some ROMIO PVFS and PVFS2 routines 

- An error message string parameter was added to MPID_Abort().  If the parameter
  is non-NULL this string will be used as the message with the abort output.
  Otherwise, the output message will be base on the error message associated
  with the mpi_errno parameter.

- MPID_Segment_init() now takes an additional boolean parameter that specifies
  if the segment processing code is to produce/consume homogenous (FALSE) or
  heterogeneous (TRUE) data.

- The definitions of MPID_VCR and MPID_VCRT are now defined by the device.

- The semantics of MPID_Progress_{Start,Wait,End}() have changed.  A typical
  blocking progress loop now looks like the following.

  if (req->cc != 0)
  {
      MPID_Progress_state progress_state;

      MPID_Progress_start(&progress_state);
      while (req->cc != 0)
      {
          mpi_errno = MPID_Progress_wait(&progress_state);
          if (mpi_errno != MPI_SUCCESS)
          {
              /* --BEGIN ERROR HANDLING-- */
              MPID_Progress_end(&progress_state);
              goto fn_fail;
              /* --END ERROR HANDLING-- */
          }
      }
      MPID_Progress_end(&progress_state);
  }

  NOTE: each of these routines now takes a single parameter, a pointer to a
  thread local state variable.

- The CH3 device and interface have been modified to better support
  MPI_COMM_{SPAWN,SPAWN_MULTIPLE,CONNECT,ACCEPT,DISCONNECT}.  Channels
  writers will notice the following.  (This is still a work in progress.  See
  the note below.)

  * The introduction of a process group object (MPIDI_PG_t) and a new
    set of routines to manipulate that object.

  * The renaming of the MPIDI_VC object to MPIDI_VC_t to make it more
    consistent with the naming of other objects in the device.

  * The process group information in the MPIDI_VC_t moved from the channel
    specific portion to the device layer.

  * MPIDI_CH3_Connection_terminate() was added to the CH3 interface to allow
    the channel to properly shutdown a connection before the device deletes all
    associated data structures.

  * A new upcall routine, MPIDI_CH3_Handle_connection(), was added to allow the
    device to notify the device when a connection related event has completed.
    A present the only event is MPIDI_CH3_VC_EVENT_TERMINATED, which notify the
    device that the underlying connection associated with a VC has been
    properly shutdown.  For every call to MPIDI_CH3_Connection_terminate() that
    the device makes, the channel must make a corresponding upcall to
    MPIDI_CH3_Handle_connection().  MPID_Finalize() will likely hang if this
    rule is not followed.

  * MPIDI_CH3_Get_parent_port() was added to provide MPID_Init() with the port
    name of the the parent (spawner).  This port name is used by MPID_Init()
    and MPID_Comm_connect() to create an intercommunicator between the parent
    (spawner) and child (spawnee).  Eventually, MPID_Comm_spawn_multiple() will
    be update to perform the reverse logic; however, the logic is presently
    still in the sock channel.

  Note: the changes noted are relatively fresh and are the beginning to a set
  of future changes.  The goal is to minimize the amount of code required by a
  channel to support MPI dynamic process functionality.  As such, portions of
  the device will change dramaticly in a future release.  A few more changes to
  the CH3 interface are also quite likely.

- MPIDI_CH3_{iRead,iWrite}() have been removed from the CH3 interface.
  MPIDI_CH3U_Handle_recv_pkt() now returns a receive request with a populated
  iovec to receive data associated with the request.
  MPIDU_CH3U_Handle_{recv,send}_req() reload the iovec in the request and
  return and set the complete argument to TRUE if more data is to read or
  written.  If data transfer for the request is complete, the coomplete
  argument must be set to FALSE.


=============================================================================== 
                               Changes in 0.96p2
===============================================================================
The shm and ssm channels have been added back into the distribution.
Officially, these channels are supported only on x86 platforms using the gcc
compiler.  The necessary assembly instructions to guarantee proper ordering of
memory operations are lacking for other platforms and compilers.  That said, we
have seen a high success rate when testing these channels on unsupported
systems.

This patch release also includes a new unsupported channel.  The scalable shared
memory, or sshm, channel is similar to the shm channel except that it allocates
shared memory communication queues only when necessary instead of preallocating
N-squared queues.


===============================================================================
                               Changes in 0.96p1
===============================================================================
This patch release fixes a problem with building MPICH2 on Microsoft Windows
platforms.  It also corrects a serious bug in the poll implementation of the
Sock interface.


===============================================================================
                                Changes in 0.96
===============================================================================
The 0.96 distribution is largely a bug fix release.  In addition to the many
bug fixes, major improvements have been made to the code that supports the
dynamic process management routines (MPI_Comm_{connect,accept,spawn,...}()).
Additional changes are still required to support MPI_Comm_disconnect().

We also added an experimental (and thus completely unsupported) rdma device.
The internal interface is similar to the CH3 interface except that it contains
a couple of extra routines to inform the device about data transfers using the
rendezvous protocol.  The channel can use this extra information to pin memory
and perform a zero-copy transfer.  If all goes well, the results will be rolled
back into the CH3 device.

Due to last minute difficulties, this release does not contain the shm or
ssm channels.  These channels will be included in a subsequent patch release.


===============================================================================
				Changes in 0.94
===============================================================================
Active target one-sided communication is now available for the ch3:sock channel.
This new functionality has undergone some correctness testing but has not been
optimized in terms of performance.  Future release will include performance
enhancements, passive target communication, and availability in channels other
than just ch3:sock.

The shared memory channel (ch3:shm), which performs communication using shared
memory on a single machine, is now complete and has been extensively tested.
At present, this channel only supports IA32 based machines (excluding the
Pentium Pro which has a memory ordering bug).  In addition, this channel must
be compiled with gcc.  Future releases with support additional architectures
and compilers.

A new channel has been added that performs inter-node communication using
sockets (TCP/IP) and intra-node communication using shared memory.  This
channel, ch3:ssm, is ideal for clusters of SMPs.  Like the shared memory
channel (ch3:shm), this channel only supports IA32 based machines and must be
compiled with gcc.  In future releases, the ch3:ssm channel will support
additional architectures and compilers.

The two channels that perform communcation using shared memory, ch3:shm and
ch3:ssm, now support the allocation of shared memory using both the POSIX and
System V interfaces.  The POSIX interface will be used if available; otherwise,
the System V interface is used.

In the interest of increasing portability, many enhancements have been made to
both the code and the configure scripts.

And, as always, many bugs have been fixed :-).


***** INTEFACE CHANGES ****

The parameters to MPID_Abort() have changed.  MPID_Abort() now takes a pointer
to communicator object, an MPI error code, and an exit code.

MPIDI_CH3_Progress() has been split into two functions: MPIDI_CH3_Progress_wait()
and MPIDI_CH3_Progress_test().


===============================================================================
				Changes in 0.93
===============================================================================
Version 0.93 has undergone extensive changes to provide better error reporting.
Part of these changes involved modifications to the ADI3 and CH3 interfaces.
The following routines now return MPI error codes:

MPID_Cancel_send()
MPID_Cancel_recv()
MPID_Progress_poke()
MPID_Progress_test()
MPID_Progress_wait()
MPIDI_CH3_Cancel_send()
MPIDI_CH3_Progress()
MPIDI_CH3_Progress_poke()
MPIDI_CH3_iRead()
MPIDI_CH3_iSend()
MPIDI_CH3_iSendv()
MPIDI_CH3_iStartmsg()
MPIDI_CH3_iStartmsgv()
MPIDI_CH3_iWrite()
MPIDI_CH3U_Handle_recv_pkt()
MPIDI_CH3U_Handle_recv_req()
MPIDI_CH3U_Handle_send_req()

*******************************************************************************
Of special note are MPID_Progress_test(), MPID_Progress_wait() and
MPIDI_CH3_Progress() which previously returned an integer value indicating if
one or more requests had completed.  They no longer return this value and
instead return an MPI error code (also an integer).  The implication being that
while the semantics changed, the type signatures did not.
*******************************************************************************

The function used to create error codes, MPIR_Err_create_code(), has also
changed.  It now takes additional parameters, allowing it create a stack of
errors and making it possible for the reporting function to indicate in which
function and on which line the error occurred.  It also allows an error
to be designated as fatal or recoverable.  Fatal errors always result in
program termination regardless of the error handler installed by the
application.

A RDMA channel has been added and includes communication methods for shared
memory and shmem.  This is recent development and the RDMA interface is still
in flux.
