SUMMARY
-------
This package implements a VFS interface to PVFS for the Linux 2.4
kernels.

See the INSTALL file for more details on how to proceed with
installation.

There are three components of note here.  The first is the kernel
module itself, which will provide the VFS functions and PVFS device
necessary to access the system.  Second is the user space PVFS client
daemon (pvfsd), which handles communication with the PVFS file system.
Last is the mount.pvfs command, which handles mounting PVFS file systems.

USAGE
-----
There are three steps for mounting PVFS file systems (these
examples assume you are currently in the pvfs-kernel build
directory and are logged in as root):

1) Load the PVFS module.  See the notes below on the module options.

insmod ./pvfs.o

If the insmod fails, please check the INSTALL file for common
errors and suggestions for correcting them.

2) Start the PVFS daemon.

./pvfsd

3) Mount the PVFS file system (after creating the mountpoint).

./mount.pvfs <manager>:<metadata_dir> <mountpoint>

For example, I'm running a PVFS file system manager on a machine "foo"
here.  The manager is storing metadata in /pvfs-meta on machine "foo".
I want to mount the PVFS file system in the directory /mnt/pvfs on my
local machine:

./mount.pvfs hell:/pvfs-meta /mnt/pvfs

That's it.  You should be able to access files on the PVFS file system
as usual.  When you want to unmount the file system, use "umount" as
usual.

NOTE:
If you have a version of util-linux that is 2.10f or later, "mount" will
automatically look for a fs-specific mount program in /sbin.  So, if you
install mount.pvfs in /sbin, you can put pvfs entries in /etc/fstab and use
"mount" to mount them.

*NEW NOTE*: 
'make install' will install mount.pvfs into ${PREFIX}/sbin/mount.pvfs, with
$PREFIX defaulting to /usr/local.  This is a change from earlier versions.
Install mount.pvfs by hand to /sbin if you want the fstab + "mount" trick
mentioned above to work.

MODULE OPTIONS
--------------
The module will accept an optional debug parameter, which will turn on
logging of various types of debugging information.  This is specified on
insmod (eg. "insmod pvfs debug=0x077").  The values of the various
message types are described in pvfs_kernel_config.h.  Using 0x3ff will
get pretty much all the messages.  By default you only get error
messages.

The maximum request size, in bytes, is settable via the "maxsz" parameter
(eg.  "insmod pvfs maxsz=33554432").  By default this is 16MB.  This can
be used to restrict the total amount of memory the PVFS module is
willing to use.  All requests that are larger than this size will be
split into one or more requests of the maximum size and one to request
the remainder.

The buffering technique used to move data between applications and the
pvfsd is also selectable via the "buffer" parameter.  There are three
valid options: static, dynamic, and mapped.  With the static technique,
a static buffer is allocated when the module is loaded.  This buffer is
the same size as the maximum request size.  With the dynamic approach, a
new buffer is allocated each time a request is made and freed when the
request finished.  With the mapped approach, the kernel rawio
functinality is used to map user pages and eliminate an extra
buffer copy.

The mapped option is only available with kernels that do not have
the "highmem" kernel configuration option enabled.

Note that all the options may be used at the same time, so that:

insmod pvfs debug=0x3ff maxsz=33554432 buffer=mapped

would enable more debugging output, set the maximum request size to
32MB, and turn on the mapped buffering technique.

Finally, a single-line message is printed to the kernel logs when the
module is loaded, indicating the settings applied.


LOGGING AND DEBUGGING
---------------------
There are two places where this system logs.  First, the kernel module
will be spitting out logging information that will end up in your system
logs.  Second, the pvfsd will be logging information into
/tmp/pvfsdlog.XXXXXX.  Both of these sources are potentially useful for
finding bugs.


EXPERIMENTAL FEATURES: 
---------------------

/proc statistics:
----

The PVFS kernel module provides several statistics and controls
which can be found in the /proc/sys/pvfs directory.  The following
list summarizes how to interact with each entry: 

debug: This indicates the current debugging level for the
  PVFS module.  The value may be read (for example, "cat debug"),
  or written (for example, "echo "0" > debug") in order to change
  the debugging level at runtime.  It uses the same values as the
  debug= module parameter documented above.
io_size: This indicates the current buffer size being used for
  I/O through the kernel module.  It may be read (for example,
  "cat io_size") or written (for example, "echo "16777216" >
  io_size) to change the buffer size at runtime.
collect_stats: This indicates if statistics gathering is enabled
  or not for the PVFS module.  "echo "1" > collect_stats" enables
  statistics gathering, while "echo "0" > collect_stats" turns it
  off.
vfs_stats: This displays statistics about VFS level operations
  within the PVFS kernel module.  The left column shows the
  name of the function while the right column shows the number of 
  times that it has been called.  "echo "1" > vfs_stats" resets
  all of the statistics shown in this file to zero.
upcall_stats: This displays statistics about each upcall used in
  the PVFS kernel.  Upcalls are low level requests between the PVFS VFS
  implementation and the pvfsd or kpvfsd.  Upcall statistics are not
  enabled by default (even with collect_stats turned on).  "echo "1" >
  upcall_stats" to enable and/or reset them.  The first column
  shows the upcall name, the second column shows the number of times
  it has been issued, the third column shows the average time that each
  upcall has taken to complete, and the fourth column shows the standard
  deviation of the timings.

kpvfsd:
----

The PVFS kernel module now supports kpvfsd, which is an alternative to
the pvfsd daemon.  Kpvfsd operates within the kernel rather than at user
level.  This provides several advantages:

- reduced context switching
- reduced buffer copying
- easier administration (no need to start any user level processes after
  loading the pvfs.o module)

In order to enable kpvfsd, add the "--with-kpvfsd" option to your normal
configure options.  The general usage of the kernel module with this feature
enabled will be the same as described elsewhere in the documentation,
except that it will no longer be necessary to launch the pvfsd program
before mounting a PVFS volume.

An additional configure option, "--with-newstyle", will cause the module
to use a new 2.4 kernel API for launching the kernel thread that runs
kpvfsd.


CAVEATS
-------

- Multiple pvfsds should not be used at this time.

- The pvfsd can potentially run out of open file descriptors if too
  many files are opened simultaneously.  If it does application tasks
  will have I/O operations fail.  I put in a workaround for this, but it
  doesn't work yet <smile>.  There is a timeout mechanism built into the
  pvfsd that will, after about a minute, get things back to sane.  I know
  that isn't really a solution, but it's all you get for the moment.

- If a mgr or iod dies, the pvfsd will hold on to the open FD until the
  first instance of a failed attempt to use the FD.  This results in the
  TCP connection lying around in FIN_WAIT2, which makes it impossible to
  reasonably restart the daemons.  The appropriate action here is to try
  an operation that uses the pvfsd; it will fail, the connection will be
  closed, and the daemon can be restarted.


FILES
-----
For your information, here's a summary of the files:

README - this file
INSTALL - notes on installation and potential warnings/errors
Makefile.in - used by configure to create makefile for the PVFS module
dir.c - VFS directory operations
file.c - VFS file operations
inode.c - VFS inode operations
kmods - kernel modifications necessary for PVFS module
ll_pvfs.c - implementation of low-level VFS operations
ll_pvfs.h - header for low-level PVFS interface
mount.pvfs.c - PVFS mount command
pvfs_linux.h - header for linux-specific defines
pvfs_mod.c - wrapper calls to define exported symbols for the module
pvfs_mount.h - generic superblock info passed in by mount
pvfs_v1_xfer.c - calls to implement VFS-like interface to PVFS v1.xx
pvfs_v1_xfer.h - header for VFS-like interface calls
pvfsd.c - pvfsd implementation
pvfsd.h - pvfsd header
pvfsdev.c - pvfsd device implementation
pvfsdev.h - prototype for pvfsdev
pvfs_kernel_config.h - configuration parameters for the PVFS kernel module
config.h.in - used by configure to create config.h
configure.in - used by autoconf to create configure script
test/ - test code, probably useless to you


FIXES/WORKAROUNDS
-----------------
- 10/23/2003- more verbose logging from pvfsdev.c

- 10/22/2003- applied patch from Murali Vilayannur to speed up
  return of iod info to client

- 10/16/2003- applied patch from Murali Vilayannur to improve 
  signal/interruption handling

- 10/16/2003- applied Murali Vilayannur's sticky bit patches

- 10/16/2003- applied patch from Alan Rainey <Alan.Rainey@acxiom.com> to
  help with revalidation problems

- 10/16/2003- applied patch from Murali Vilayannur to fix some IA64
  issues

- 9/16/2003 - patch submitted by Jim Schutt <jaschut@sandia.gov> for
  trusted port and restricted IP ranges.

- 8/26/2003 - patch submitted by Murali Vilayannur to provide symlink 
  support (matches symlink patches to main pvfs tree)

- 6/19/2003 - patch submitted by Murali Vilayannur to fix kpvfsd
  memory leak

- 6/11/2003 - patch submitted by Richard Jones
  <Richard.T.Jones@uconn.edu> that fixes bug that could cause
  duplicate device sequence numbers in some cases

- 6/4/2003 - implimented operation retry in pvfs_v1_xfer; allows
  mgr to be killed and restarted without affecting pvfs-kernel in
  many cases

- 5/15/2003 - updated mount.pvfs to honor the "ro" option

- 5/15/2003 - fixed argument parsing of mount.pvfs to better match
  traditional mount command

- 5/8/2003 - removal of variables named st_[amc]time to avoid
  conflict with glibc 2.3.2

- 4/28/2003 - added configure time check for pvfs header files

- 4/28/2003 - bug fix to dcache serialization; rename works now

- 4/18/2003 - removed bsize and soff fields from metadata in kpvfsd
  headers to match what's going on in pvfs core

- 3/11/2003 - fixed bug in the way that invalidated sequence numbers
  were handled (in particular, the way that 2 step writes were cleaned
  out once invalidated).  Also boosted range of sequence numbers.

- 2/4/2003 - removed --with-single option, made its behavior the
  default (use only one socket per iod, regardless of number of
  files open)

- 1/23/2003 - renamed LOCK_SERIALIZER to PVFS_SERIALIZE_DCACHE and
  turned it on by default (to prevent dcache corruption that users are
  still able to trigger in some cases)

- 1/3/2003 - made it so that attempting to open a file larger than
  2G on a non-LFS system results in an open error

- 12/29/2002 - fixed a bug in do_invalidate_pages in which page
  cache was not being indexed correctly- showed up as a system
  crash when mapping files larger than 2G, among other problems

- 12/17/2002 - fixed bug in read/write for --with-single mode-
  code was not correctly counting which iods had been contacted in
  order to gather return values and file sizes

- 12/13/2002 - patch from Pete Wyckoff to fix logic for detecting
  size on read operations (was previously getting triggered
  unecessarily for writes as well)

- 12/13/2002 - patch from Pete Wyckoff to set CFLAGS correctly
  from configure to makefile

- 12/5/2002 - fixed error in pvfs_file_read in which the logic
  that breaks up large reads in to <= max buffer size chunks would
  fail if EOF was reached early.

- 11/13/2002 - added support for specifying sbindir for installation,
  various other config/code cleanups submitted by Pete Wyckoff

- 11/13/2002 - patch from Murali Vilayannur to currect a bug in
  reading proc statistics

- 11/13/2002 - patch to fix mmap page invalidation so that it will
  work on any 2.4 kernel, not just 2.4.9 and later

- 11/01/2002 - added kpvfsd support (implemented by Murali
  Vilayannur).  This allows the user level pvfsd daemon to be replaced
  by a kernel process.

- 10/30/2002 - added rw semaphore component which can be used to
  serialize dcache access if -DLOCK_SERIALIZER is defined.
  Implemented by Murali Vilayannur.

- 10/28/2002 - adjusted how pages are invalidated when mmap is used.
  Should lead to better executable behavior.  Implemented by Murali
  Vilayannur.

- 10/28/2002 - modified getdents to retrieve more than one directory
  entry at a time to make directory listing more efficient.
  Implemented by Murali Vilayannur.

- 10/24/2002 - added support for iod connection management; limits
  connections so that there is never more than one per iod.
  Implemented by Murali Vilayannur.

- 10/24/2002 - added support for reading statistics from /proc;
  implemented by Murali Vilayannur.

- 10/15/2002 - rewrote README and INSTALL files.

- 10/2002 - rewrote much of the autoconf script to test for
  features in the kernel headers, added warnings if they do not
  match the running system.

- 10/03/2002 - added support for release numbers in the pvfs request
  protocol.

- 10/3/2002 - started removing autconf tests and #ifdef portions
  of code that were only needed for 2.2 kernel

- 10/2/2002 - 2.2 kernel version branched off; autoconf modified
  in this branch to only run if it detects 2.4 kernel

- 10/2/2002 - fixed bug in inode.c in which code may attempt to
  fill in a null inode if pvfs_getmeta fails (pointed out by
  Murali)

- 7/15/2002 - fixed misc. bugs in pvfs_v1_xfer, mount.pvfs, and
  pvfsdev pointed out by Murali Vilayannur.

- 7/1/2002 - fixed setmeta operation so that cp -p now works for files.
  Still does not work for directories, however

- 6/?/2002 - fixed oops that could be triggered by interrupting
  concurrent dentry revalidations

- 6/?/2002 - fixed "missing .. entry" bug as reported by Doug Hoffman

- 6/?/2002 - fixed many dcache bugs in pvfs kernel code

- 6/?/2002 - fixed possible inode data memory leak

- 5/1/2002 - fixed bug setmeta bug in pvfs_v1_xfer; it was possible for
  a chmod to set a file as read only, which then caused the utime part
  of the setmeta to fail.  utime and truncate are now performed as root
  if a chmod or chown was successful in the same setmeta operation.

- 4/25/2002 - fixed bug in pvfs_v1_xfer; case in which read was
  attempted beyond EOF was not handled correctly.  Discovered by Frank
  Shorter's test harness.

- 4/4/2002 - implemented pvfs_file_llseek() so that week can make sure
  that we have an up to date file size before seeking to the end of a
  file

- 4/8/2002 - fixed semaphore handling bug that was causing problems on
  some SMP systems (particularly when kill processes that were doing
  PVFS I/O)

- 3/28/2002 - fixed bug found by Brian Behlendorf in which write 
  operations were sometimes not being cleaned up properly in the pvfsdev
  when interrupted. 

- 3/20/2002 - added Brian Behlendorf's patch (with some minor
  modifications) to make the module acquire a dynamic major number
  rather than take over 60 by default

- 3/13/2002 - bugfixes to inode revalidation

- 11/02/2001 - added new SMP detection string to configure

- 09/24/2001 - Dan Nurmi's permission setting patch to mount.pvfs

- 09/20/2001 - bugfix for inode revalidation problem

- 09/19/2001 - bugfix of permissions checking on setmeta

- 09/19/2001 - added test for highmem symbols in configure- if found,
  mapped transfers are disabled

- 09/17/2001 - added /lib/modules/'uname -r'/build/include to include
  paths on 2.4 kernel systems using configure

- 08/13/2001 - added support for mgr_lookup operation

- 07/25/2001 - fixed pvfsd's handling of aborted operations

- 07/25/2001 - removed some more big static declarations from code

- 07/25/2001 - pvfsdev code restructuring, including lots of bug fixes

- 07/25/2001 - integrated patch from Terje Eggestad to add distclean
  target to makefiles.

- 07/14/2001 - fixed stack overflow problem that was occuring in
  pvfs_super_iget because we put a large string on the stack.  mounts
  should be more reliable now.

- 07/14/2001 - big cleanup of pvfsdev code, including some bug fixes for
  when it recovers from errors and interruptions

- 07/14/2001 - fixed semantics of pvfsdev_read so that it blocks if no
  upcalls are available

- 07/14/2001 - started using set_current_state macro in kernel (when
  available) to change process states

- 07/14/2001 - removed run_task_queue calls

- 06/11/2001 - added setting of s_maxbytes field in the super_block
  structure.  this should fix problems with lseek() on some 2.4.x
  kernels, in particular RH7.1 machines.

- 06/08/2001 - added checks in pvfs_file_read() and pvfs_file_write()
  for 0-byte operations; drop out after checking for valid buffer
  address.  the code below this does not like 0-byte accesses,
  especially the dynamic allocation code, which will die.

  This fixes a problem seen with using ROMIO noncontig test on a PVFS
  mounted file system when specifying that ROMIO should use the VFS
  interface (ufs:).

- 05/31/2001 - made pvfs_dentry_revalidate() call
  pvfs_revalidate_inode() in all cases where valid inode is passed in;
  this seems to fix problems we were seeing with multiple processes
  opening the same file on a single machine (related to dcache).

- 04/16/2001 - made hint operations "one way" to help with client hang
  problems 

- 01/14/2001 - added checks on vmalloc() return value in pvfsdev.c,
  renamed variables to make more sense.

- 11/29/2000 - changed --with-pvfslib-dir to --with-libpvfs-dir; it was
  driving me mad.

- 11/20/2000 - added workaround in pvfs_v1_xfer to not talk to I/O
  daemons when closing a file if we haven't previously communicated with
  any of them.

- 11/15/2000 - added Dan Nurmi's patch to allow options to mount.pvfs,
  including port.

- 11/14/2000 - added Dan Nurmi's patch to mount.pvfs to update
  /etc/mtab.

- 11/14/2000 - added option to configure to turn on support of kernel
  patch (--with-pvfs-kpatch).  This is used instead of the __PVFS__XXX
  stuff in the makefile before.

- 11/13/2000 - modified file.c to use pgoff2loff(page->index) instead
  of page->offset (which didn't exist on the kernel I was compiling for)

  Not exactly sure which kernels this is right for.

- 10/23/2000 - implemented statfs so that we can get file system
  information back

- 07/25/2000 - fixed pvfs_dentry_revalidate to return correct values
  when the inode information is missing

- 07/25/2000 - added some checks for smp and modversions configuration
  in pvfs_kernel_config.h

- -7/25/2000 - fixed maxsz module parameter so that the transfer buffer
  size can be specified when the module is loaded

- 03/14/2000 - ll_pvfs_create() now silently ignores the case where a
  file exists when it is called.  This condition occurs when a parallel
  application creates a new file.  One node will create the file between
  the time another detects it is not there and the attempts to create
  it.  This may break O_EXCL in some cases, but I think it is a
  necessary evil.

- 03/13/2000 - options available for maximum size and buffering
  technique now.  Removed more unnecessary mandatory debugging info.
  It is no longer necessary to patch your kernel to use the module.

- 03/08/2000 - pvfsd now handles SIGPIPE correctly, catching and
  resuming operation.  Would exit before.

- 03/08/2000 - changed the default debugging mask to 0 to reduce the
  logging I/O during extended operations.  See debugging option for
  module (mentioned above) to turn on more debugging.

- 03/08/2000 - fixed bug in which restarting an iod would leave bad
  dentries in place.

- 03/08/2000 - fixed bug in which empty superblock was locked on
  failed mount attempts.

- long ago - stat calls return "blocks" as if block size were 512
  bytes.  Required for "du" to work.


THANKS
------
Thanks to Scyld Computing for funding this work!  Thanks also to Brian
Haymore of the University of Utah Center for High Performance Computing
for preliminary testing of the code.

Thanks go to the developers of the Coda linux VFS module; it was
particularly helpful as a working example.

Dan Nurmi contributed code to get /etc/mtab support working.  Very
helpful.
