Planned enhancements to ZSyncer should be logged here.

BUGS:

 * XMLRPC is currently (0.6.0-pre6) failing on comparisons with
   "OverflowError: long int exceeds XML-RPC limits"
   Should we fix it or just kill it instead?

 * sourceforge #988027:  do not sync webdav locks.

 * Big syncs:  Transferring a lot of stuff fails because 
   ZSyncer does everything in memory.  E.g. if I sync a folder containing 
   ~70 MB of stuff, the destination server's ram useage shoots up by
   300 MB!!!  

   This may be somewhat mitigated by using ZPublisher.Client, test it
   again.

   Note that import of large .zexp's seems to succeed in cases when 
   zsyncer fails.  

   Solution: In manage_addRemote, when importing, if the size of the 
   pickle is larger than a certain amount, dump it to a tempfile first, 
   then delete the pickle from memory, then import the tempfile.  
   To implement this:
   1. check if the name of the .zexp file matters to zope - if so tmpfile 
      can't help!  
   2. dump unconditionally and see if it helps with large stuff.  
   3. use a "finally" clause to delete the tmpfile.
   4. make it conditional on size.
   5. add max. size knob to the UI.

 * If syncing from a member of a ZEO cluster, your $INSTANCE_HOME must
   be the same path as that of the ZEO server or else you won't find
   the ZSyncer.log.  This is silly, it should use the local $INSTANCE_HOME.

 * A report of failure on Windows that I have not reproduced: "illegal
   data at start of file", from Kamal Gill to zope list, from 29 Oct
   2003.

 * UI should be written in zpt, not dtml :-P


UI enhancements:

 * finish integration work with CMF.
   In the simplest form, this would mean finishing the sync tool
   and providing some UI for syncer actions.  More interesting would
   be integrating it with workflow somehow - that could make life much
   easier for content managers. I suspect that workflow integration 
   should be left up to the developer; just make some scripts that
   use the syncer tool and have some transitions call them.

 * Switching syncers while working is clunky. You either have to edit
   the syncer, which is error-prone, or go out of the current syncer
   and into another one.  Instead of having a bunch of syncers
   configured for different destination servers, maybe have one syncer
   configured with multiple destinations, and in Folders.dtml allow
   the user to toggle which of the configured destinations are used
   for comparisons and syncing.

 * Currently with multiple destination servers, only the first one is
   used for comparisons. This is not very informative.

   Quick solution: If multiple destination servers, allow user to
   toggle which is used for comparison (without changing the syncer
   config)

   More extensive solution: provide a combined comparison
   mode in which each object's status is shown on all destination
   servers.  The status could then be any of:
       ok
       missing
       out-of-date
       ok/missing
       ok/out-of-date
       missing/out-of-date
       ok/missing/out-of-date

   alternatively, have a status column for each destination.
   Or maybe rows within each object's row?
   This a lot of information - especially with some of the other
   recent enhancements like showing the mod. time for the destination -
   now we have to decide WHICH destination to show, or how to show all
   of them, or whether to show the latest only, etc.
   Hard to keep the UI from getting cluttered with all this info.
   The next item might help wit this.

 * status column could contain links to more detailed information,
   e.g. in the multi-destination case, show the usual comparison
   for each server in turn. This could also be used to do more
   expensive comparisons as described below.

 * Write some Help for the manage_sync page.


Functional enhancements:


 * Log dir should be configurable in the config file.

 * COMPARISON ENHANCEMENTS

   Motivation:

   - Timestamps give wrong results if the servers do not have closely synced 
   clocks (destination = source +/- network latency), and even then can 
   be misleading.
   
   - Prevent false out-of-date results: wastes time for the user since
   they might come back later and find tons of stuff out of sync.
   
   - Prevent false OK results: if something is changed directly on the 
   destination server, it will compare "OK", which makes
   it very likely that emergency patches to a production server will
   get clobbered unless great care is taken to back-sync them.

   Goals:

   - copies of the same object, with the same id, should initially 
     compare equal.

   - any difference in DC Metadata -> not equal.

   - any difference in Properties -> not equal.

   - any difference in security settings (permission/role mapping, local
     roles) -> not equal.

   - for ObjectManagers: comparison should continue to be atomic rather than
     recursive by default.
     Recursive means that if foo/bar/bat/baz is considered different,
     then foo, bar, and bat would be considered different even if
     there are no other differences.  Atomic means that we ignore
     foo._objects.  Atomic is both easier and cheaper.
     Also, recursive would be confusing since you don't know how much
     of the tree you need to sync to fix the problem.
     A better solution to recursive comparison is already implemented in the UI.

   Ideas for comparisons:
   There are many potential comparisons but many are likely to be
   too expensive for routine recursive use. this should be investigated.
   One option would be to give the user control over which comparisons
   are performed.

   - improve timestamp comparison:
     I don't think this is possible for non-CMF objects, 
     for the reason given below.
 
     this was my strategy:

     dest_time = remote call to DateTime()
     local_time = DateTime()
     fudge_factor = dest_time - local_time

     when comparing times,
     if dest_modtime <= source_modtime + fudge_factor:
        status is OK

     PROBLEM: if the clocks on the two systems have changed
     since dest_modtime or source_modtime, this doesn't work
     at all.

     Somebody on zope@zope.org suggested that zsyncer should
     forcefully set the mod time of the source & destination
     objects so that they match. This is not possible
     since comitting the object in the zodb will force the
     mod time to be updated again. :-P

   - CMF content is easier since it has real metadata: we can
     ignore bobobase_modification_time and use the DublinCore
     modification date as our timestamp. When comparing, 
     we're OK only if they are identical. This should work very
     reliably. DONE!

   - add a new status ("Changed"? "Newer"?) if the timestamps look ok
     but other comparisons indicate a difference. 
     (should this be red, orange, or other color?)

   - compare size - easy and cheap, will catch many differences,
     but ObjectManagers and many other objects don't support it or 
     might have a different interface for getting it. (get_size vs 
     getSize, maybe others)

   - compare metatype: this always works.  unfortunately it's
     unlikely that we have 2 different types, but it does happen. 
     so this is probably most useful as a short-circuit for
     occasionally avoiding more expensive comparisons.

   - compare all DublinCore metadata if available. Again,
     mostly useful as a short-circuit; even so, comparing DC
     modification time should prevent the need for this.
 
   - simple comparison of the object's "content", whatever that is - 
     e.g. body of a DTML Method. Not useful for all types.
     Potentially too expensive on the network and not useable for all 
     types - would need to be toggleable. Should only be done
     after a meta_type check.

   - Checksum: compare some kind of hash or checksum.  Maybe using the 
     md5 module (hash() would be ok but i doubt hashes are guaranteed 
     to be portable).
     These can take a while to compute for large objects (~ 12 seconds to 
     checksum a 200 MB string on a PIII-866; plus we have to figure the ZODB
     load time to get the data). 
     But once computed, these are very small and very likely 
     to give good results. 
     For proof-of-concept, implement this with md5 and don't worry about
     speed.

     (premature optimization note:
     i suspect that to compute checksums fast enough we'll need
     a small C extension... but only if it's faster than md5.
     md5 is probably severe overkill anyway.
     Note that gnu sum is quite fast; the algorithm is pretty simple,
     and i found c code and a python translation for it in a 
     comp.lang.python search for python fast checksum.)

     We have to be careful not to checksum data that will cause
     otherwise-identical objects to compare inequal: e.g. 
     bobobase_modification_time, and unhashable stuff like references 
     to other objects.

     We also have to be careful to check all interesting aspects
     that could differ: properties & security settings,
     DC metadata, CMF workflow state. Some of these may be implemented 
     with dictionaries; how do you checksum a dictionary?? 
     one answer:  mylist = mydict.items(); mylist.sort(); str(mylist)
     ... which fails if any of the values are dictionaries or instances...

     Maybe we need some way to configure how to checksum a given
     meta_type.  e.g. register an adapter object that
     cooks up some subset of the real object to checksum.

     (premature optimization note 2:
     if the checksum will take a lot of CPU time to compute due to 
     enormous size of an object, we could do a 2-stage comparison: first 
     do all available metadata; if those come back equal we move on to 
     more expensive stuff.
     This should be decided by the source zsyncer based on characteristics
     of the local object, because we want to minimize rpc calls to avoid 
     network delays. The remote comparison should return as
     soon as it finds a difference.)

     (premature optimization note 3:
     Might be worthwhile to cache the checksums. The ZSyncer itself
     could store these in a OOBTree. The keys would be paths
     (relative to the syncer?). Each leaf would contain something like
     (bobobase_modification_time, security_checksum, properties_checksum,
     dc_metadata_checksum, content_checksum). If any of those cannot
     be computed, it's None.
     The destination server, when asked to compare an object, would do
     something like:

     if (not cache.has_key(obj_path)) or \ 
        cache(obj_path)[0] < obj.bobobase_modification_time:
         compute the tuple and cache it
         return the tuple 

     The source ZSyncer could then just compare each checksum
     optionally falling back to other comparisons.
     )

 * recursive sync: a new (toggleable) sync mode, in which 
   zsyncer tries to walk down the tree and automagically do the right thing 
   depending on status: sync only missing or out-of-date things 
   (don't try to descend into missing folders, just sync them). 
   This could leave some out-of-sync objects if the comparisons are wrong,
   but it could also be a big win when syncing a few scattered changes
   in a huge folder tree.
   Result should report only the items that were actually synced.

   There should always be a confirmation page in this mode, that shows you 
   what would be synced and allows deselecting items.

   Hmm, this is not really necessary if I just fix the filtering in 
   recursive comparison mode.

 * Aspect-based syncing:
   Currently, sync is all-or-nothing.  What if you have a 200 MB file
   and change its permissions, or its properties, or its CMF metadata?
   You have to sync the whole thing.  Worse, what if you have a huge
   portal_skins folder, with some stuff you don't want to sync yet,
   but there are changes to the skin paths (properties) that you need
   to sync ASAP?  Another use case: Access Rule settings on a folder.
   
   This is not a YAGNI - I have needed it for months :-) 
 
   one problem is making this extensible to handle more features, e.g.
   custom products. 
   maybe just register the attributes that are considered "content".

   The new _callRemote() method is intended to be useful here.

 * There are no tests whatsoever. Ugh.
   The new relative mode may help with some functional tests since you
   no longer need two zopes. Or maybe we'll still have to fake
   everything with a suitable test harness.
   Also check out ZopeTestCase.

 * "Unsyncable": it should be possible to flag an object (not just a
   meta_type) as unsyncable. On the source, this would disallow syncing it;
   on the destination, this would disallow manage_deleteRemote.
   (OK, that's really 2 flags.) The UI should tell who marked
   it, when, & why.  The rationale is similar to the "immutable"
   bit in unix. The syncer itself could store these as
   an OOBTree whose items are path: flag, which could get stale
   if objects move; or it could set properties on objects,
   like zsyncer_unsyncable (boolean); but that might annoy people.
   TANSTAAFL.   



