Reorganized and cleaned up TODO list.

1 files changed, 617 insertions, 383 deletions

diff --git a/TODO b/TODO
index e876ee57..106c9a5f 100644
--- a/TODO
+++ b/TODO
@@ -1,8 +1,94 @@
 -*- indented-text -*-
 
 BUGS ---------------------------------------------------------------
+Fix hardlink reporting						2002/03/25
+Fix progress indicator to not corrupt log
+lchmod question	
+Do not rely on having a group called "nobody"
+Incorrect timestamps (Debian #100295)
+Win32
+
+FEATURES ------------------------------------------------------------
+server-imposed bandwidth limits
+rsyncd over ssh
+Use chroot only if supported
+Allow supplementary groups in rsyncd.conf			2002/04/09
+Handling IPv6 on old machines
+Other IPv6 stuff:
+Add ACL support							2001/12/02
+Lazy directory creation
+Conditional -z for old protocols
+proxy authentication						2002/01/23
+SOCKS								2002/01/23
+FAT support
+Allow forcing arbitrary permissions				2002/03/12
+--diff						david.e.sewell	2002/03/15
+Add daemon --no-detach and --no-fork options
+
+DOCUMENTATION --------------------------------------------------------
+Update README
+Keep list of open issues and todos on the web site
+Update web site from CVS
+Perhaps redo manual as SGML
+
+LOGGING --------------------------------------------------------------
+Make dry run list all updates					2002/04/03
+Memory accounting
+Improve error messages
+Better statistics:					Rasmus	2002/03/08
+Perhaps flush stdout like syslog
+Log deamon sessions that just list modules
+Log child death on signal
+Keep stderr and stdout properly separated (Debian #23626)
+Log errors with function that reports process of origin
+verbose output					David Stein	2001/12/20
+Add reason for transfer to file logging
+debugging of daemon						2002/04/08
+internationalization
+
+DEVELOPMENT --------------------------------------------------------
+Handling duplicate names
+Use generic zlib						2002/02/25
+TDB:								2002/03/12
+Splint								2002/03/12
+Memory debugger
+Create release script
+Add machines to build farm
+
+PERFORMANCE ----------------------------------------------------------
+File list structure in memory
+Traverse just one directory at a time
+Hard-link handling
+Allow skipping MD4 file_sum					2002/04/08
+Accelerate MD4
+String area code
+
+TESTING --------------------------------------------------------------
+Torture test
+Cross-test versions						2001/08/22
+Test on kernel source
+Test large files
+Create mutator program for testing
+Create configure option to enable dangerous tests
+If tests are skipped, say why.
+Test daemon feature to disallow particular options.
+Create pipe program for testing
+Create test makefile target for some tests
+Test "refuse options" works
+
+RELATED PROJECTS -----------------------------------------------------
+rsyncsh
+http://rsync.samba.org/rsync-and-debian/
+rsyncable gzip patch
+rsyncsplit as alternative to real integration with gzip?
+reverse rsync over HTTP Range
+
+
+
+BUGS ---------------------------------------------------------------
 
-There seems to be a bug with hardlinks
+Fix hardlink reporting						2002/03/25
+  (was: There seems to be a bug with hardlinks)
 
   mbp/2 build$ ls -l /tmp/a /tmp/b -i
   /tmp/a:
@@ -72,8 +158,12 @@ There seems to be a bug with hardlinks
   -rw-rw-r--    5 mbp      mbp            29 Mar 25 17:30 b2
   -rw-rw-r--    5 mbp      mbp            29 Mar 25 17:30 b3
 
+                      --          --
 
-Progress indicator can produce corrupt output when transferring directories:
+
+Fix progress indicator to not corrupt log
+
+  Progress indicator can produce corrupt output when transferring directories:
 
   main/binary-arm/
   main/binary-arm/admin/
@@ -92,21 +182,16 @@ Progress indicator can produce corrupt output when transferring directories:
   main/binary-arm/math/
   main/binary-arm/misc/
 
+                      --          --
+
+
+lchmod question	
 
-lchmod
   I don't think we handle this properly on systems that don't have the
   call.  Are there any such?
 
+                      --          --
 
-Cross-test versions
-  Part of the regression suite should be making sure that we don't
-  break backwards compatibility: old clients vs new servers and so
-  on.  Ideally we would test the cross product of versions.  
-
-  It might be sufficient to test downloads from well-known public
-  rsync servers running different versions of rsync.  This will give
-  some testing and also be the most common case for having different
-  versions and not being able to upgrade.
 
 Do not rely on having a group called "nobody"
 
@@ -114,30 +199,44 @@ Do not rely on having a group called "nobody"
 
   On Debian it's "nogroup"
 
-DAEMON --------------------------------------------------------------
+                      --          --
 
-server-imposed bandwidth limits
 
-rsyncd over ssh
+Incorrect timestamps (Debian #100295)
 
-  There are already some patches to do this.
+  A bit hard to believe, but apparently it happens.
 
-  BitKeeper uses a server whose login shell is set to bkd.  That's
-  probably a reasonable approach.
+                      --          --
+
+
+Win32
+
+  Don't detach, because this messes up --srvany.
+
+  http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html
 
 
+
+                      --          --
+
 FEATURES ------------------------------------------------------------
 
+server-imposed bandwidth limits
 
---dry-run is too dry
+                      --          --
 
-  Mark Santcroos points out that -n fails to list files which have
-  only metadata changes, though it probably should.
 
-  There may be a Debian bug about this as well.
+rsyncd over ssh
+
+  There are already some patches to do this.
+
+  BitKeeper uses a server whose login shell is set to bkd.  That's
+  probably a reasonable approach.
 
+                      --          --
 
-use chroot
+
+Use chroot only if supported
 
   If the platform doesn't support it, then don't even try.
 
@@ -147,208 +246,243 @@ use chroot
     http://lists.samba.org/pipermail/rsync/2001-August/thread.html
     http://lists.samba.org/pipermail/rsync/2001-September/thread.html
 
+                      --          --
+
 
-supplementary groups
+Allow supplementary groups in rsyncd.conf			2002/04/09
 
   Perhaps allow supplementary groups to be specified in rsyncd.conf;
   then make the first one the primary gid and all the rest be
   supplementary gids.
 
+                      --          --
 
-File list structure in memory
 
-  Rather than one big array, perhaps have a tree in memory mirroring
-  the directory tree.
+Handling IPv6 on old machines
 
-  This might make sorting much faster!  (I'm not sure it's a big CPU
-  problem, mind you.)
+  The KAME IPv6 patch is nice in theory but has proved a bit of a
+  nightmare in practice.  The basic idea of their patch is that rsync
+  is rewritten to use the new getaddrinfo()/getnameinfo() interface,
+  rather than gethostbyname()/gethostbyaddr() as in rsync 2.4.6.
+  Systems that don't have the new interface are handled by providing
+  our own implementation in lib/, which is selectively linked in.
 
-  It might also reduce memory use in storing repeated directory names
--- again I'm not sure this is a problem.
+  The problem with this is that it is really hard to get right on
+  platforms that have a half-working implementation, so redefining
+  these functions clashes with system headers, and leaving them out
+  breaks.  This affects at least OSF/1, RedHat 5, and Cobalt, which
+  are moderately improtant.
 
-Performance
+  Perhaps the simplest solution would be to have two different files
+  implementing the same interface, and choose either the new or the
+  old API.  This is probably necessary for systems that e.g. have
+  IPv6, but gethostbyaddr() can't handle it.  The Linux manpage claims
+  this is currently the case.
 
-  Traverse just one directory at a time.  Tridge says it's possible.
+  In fact, our internal sockets interface (things like
+  open_socket_out(), etc) is much narrower than the getaddrinfo()
+  interface, and so probably simpler to get right.  In addition, the
+  old code is known to work well on old machines.
 
-  At the moment rsync reads the whole file list into memory at the
-  start, which makes us use a lot of memory and also not pipeline
-  network access as much as we could.
+  We could drop the rather large lib/getaddrinfo files.
 
+                      --          --
 
-Handling duplicate names
 
-  We need to be careful of duplicate names getting into the file list.
-  See clean_flist().  This could happen if multiple arguments include
-  the same file.  Bad.
+Other IPv6 stuff:
+  
+  Implement suggestions from http://www.kame.net/newsletter/19980604/
+  and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
 
-  I think duplicates are only a problem if they're both flowing
-  through the pipeline at the same time.  For example we might have
-  updated the first occurrence after reading the checksums for the
-  second.  So possibly we just need to make sure that we don't have
-  both in the pipeline at the same time.
+  If a host has multiple addresses, then listen try to connect to all
+  in order until we get through.  (getaddrinfo may return multiple
+  addresses.)  This is kind of implemented already.
 
-  Possibly if we did one directory at a time that would be sufficient.
+  Possibly also when starting as a server we may need to listen on
+  multiple passive addresses.  This might be a bit harder, because we
+  may need to select on all of them.  Hm.
 
-  Alternatively we could pre-process the arguments to make sure no
-  duplicates will ever be inserted.  There could be some bad cases
-  when we're collapsing symlinks.
+  Define a syntax for IPv6 literal addresses.  Since they include
+  colons, they tend to break most naming systems, including ours.
+  Based on the HTTP IPv6 syntax, I think we should use
+ 
+     rsync://[::1]/foo/bar [::1]::bar
 
-  We could have a hash table.
+  which should just take a small change to the parser code.
 
-  The root of the problem is that we do not want more than one file
-  list entry referring to the same file.  At first glance there are
-  several ways this could happen: symlinks, hardlinks, and repeated
-  names on the command line.
+                      --          --
 
-  If names are repeated on the command line, they may be present in
-  different forms, perhaps by traversing directory paths in different
-  ways, traversing paths including symlinks.  Also we need to allow
-  for expansion of globs by rsync.
 
-  At the moment, clean_flist() requires having the entire file list in
-  memory.  Duplicate names are detected just by a string comparison.
+Add ACL support							2001/12/02
 
-  We don't need to worry about hard links causing duplicates because
-  files are never updated in place.  Similarly for symlinks.
+  Transfer ACLs.  Need to think of a standard representation.
+  Probably better not to even try to convert between NT and POSIX.
+  Possibly can share some code with Samba.
 
-  I think even if we're using a different symlink mode we don't need
-  to worry.
+                      --          --
 
-  Unless we're really clever this will introduce a protocol
-  incompatibility, so we need to be able to accept the old format as
-  well.
 
+Lazy directory creation
 
-Memory accounting
+  With the current common --include '*/' --exclude '*' pattern, people
+  can end up with many empty directories.  We might avoid this by
+  lazily creating such directories.
 
-  At exit, show how much memory was used for the file list, etc.
+                      --          --
 
-  Also we do a wierd exponential-growth allocation in flist.c.  I'm
-  not sure this makes sense with modern mallocs.  At any rate it will
-  make us allocate a huge amount of memory for large file lists.
 
+Conditional -z for old protocols
 
-Hard-link handling
+  After we get the @RSYNCD greeting from the server, we know it's
+  version but we have not yet sent the command line, so we could just
+  remove the -z option if the server is too old.
 
-  At the moment hardlink handling is very expensive, so it's off by
-  default.  It does not need to be so.
+  For ssh invocation it's not so simple, because we actually use the
+  command line to start the remote process.  However, we only actually
+  do compression in token.c, and we could therefore once we discover
+  the remote version emit an error if it's too old.  I'm not sure if
+  that's a good tradeoff or not.
 
-  Since most of the solutions are rather intertwined with the file
-  list it is probably better to fix that first, although fixing
-  hardlinks is possibly simpler.
+                      --          --
 
-  We can rule out hardlinked directories since they will probably
-  screw us up in all kinds of ways.  They simply should not be used.
 
-  At the moment rsync only cares about hardlinks to regular files.  I
-  guess you could also use them for sockets, devices and other beasts,
-  but I have not seen them.
+proxy authentication						2002/01/23
 
-  When trying to reproduce hard links, we only need to worry about
-  files that have more than one name (nlinks>1 && !S_ISDIR).
+  Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
+  HTTP Basic Proxy-Authentication.
 
-  The basic point of this is to discover alternate names that refer to
-  the same file.  All operations, including creating the file and
-  writing modifications to it need only to be done for the first name.
-  For all later names, we just create the link and then leave it
-  alone.
+  Multiple schemes are possible, up to and including the insanity that
+  is NTLM, but Basic probably covers most cases.
 
-  If hard links are to be preserved:
+                      --          --
 
-    Before the generator/receiver fork, the list of files is received
-    from the sender (recv_file_list), and a table for detecting hard
-    links is built.
 
-    The generator looks for hard links within the file list and does
-    not send checksums for them, though it does send other metadata.
+SOCKS								2002/01/23
 
-    The sender sends the device number and inode with file entries, so
-    that files are uniquely identified.
+  Add --with-socks, and then perhaps a command-line option to put them
+  on or off.  This might be more reliable than LD_PRELOAD hacks.
 
-    The receiver goes through and creates hard links (do_hard_links)
-    after all data has been written, but before directory permissions
-    are set.
+                      --          --
 
-  At the moment device and inum are sent as 4-byte integers, which
-  will probably cause problems on large filesystems.  On Linux the
-  kernel uses 64-bit ino_t's internally, and people will soon have
-  filesystems big enough to use them.  We ought to follow NFS4 in
-  using 64-bit device and inode identification, perhaps with a
-  protocol version bump.
 
-  Once we've seen all the names for a particular file, we no longer
-  need to think about it and we can deallocate the memory.
+FAT support
 
-  We can also have the case where there are links to a file that are
-  not in the tree being transferred.  There's nothing we can do about
-  that.  Because we rename the destination into place after writing,
-  any hardlinks to the old file are always going to be orphaned.  In
-  fact that is almost necessary because otherwise we'd get really
-  confused if we were generating checksums for one name of a file and
-  modifying another.
+  rsync to a FAT partition on a Unix machine doesn't work very well at
+  the moment.  I think we get errors about invalid filenames and
+  perhaps also trying to do atomic renames.
 
-  At the moment the code seems to make a whole second copy of the file
-  list, which seems unnecessary.
+  I guess the code to do this is currently #ifdef'd on Windows;
+  perhaps we ought to intelligently fall back to it on Unix too.
 
-  We should have a test case that exercises hard links.  Since it
-  might be hard to compare ./tls output where the inodes change we
-  might need a little program to check whether several names refer to
-  the same file.
+                      --          --
 
 
+Allow forcing arbitrary permissions				2002/03/12
 
-Handling IPv6 on old machines
+  On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote:
+  > If we would add an option to do that functionality, I
+  > would vote for one that was more general which could mask
+  > off any set of permission bits and possibly add any set of
+  > bits.  Perhaps a chmod-like syntax if it could be
+  > implemented simply.
 
-  The KAME IPv6 patch is nice in theory but has proved a bit of a
-  nightmare in practice.  The basic idea of their patch is that rsync
-  is rewritten to use the new getaddrinfo()/getnameinfo() interface,
-  rather than gethostbyname()/gethostbyaddr() as in rsync 2.4.6.
-  Systems that don't have the new interface are handled by providing
-  our own implementation in lib/, which is selectively linked in.
+  I think that would be good too.  For example, people uploading files
+  to a web server might like to say
 
-  The problem with this is that it is really hard to get right on
-  platforms that have a half-working implementation, so redefining
-  these functions clashes with system headers, and leaving them out
-  breaks.  This affects at least OSF/1, RedHat 5, and Cobalt, which
-  are moderately improtant.
+  rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
 
-  Perhaps the simplest solution would be to have two different files
-  implementing the same interface, and choose either the new or the
-  old API.  This is probably necessary for systems that e.g. have
-  IPv6, but gethostbyaddr() can't handle it.  The Linux manpage claims
-  this is currently the case.
+  Ideally the patch would implement as many of the gnu chmod semantics
+  as possible.  I think the mode parser should be a separate function
+  that passes back something like (mask,set) description to the rest
+  of the program.  For bonus points there would be a test case for the
+  parser.
 
-  In fact, our internal sockets interface (things like
-  open_socket_out(), etc) is much narrower than the getaddrinfo()
-  interface, and so probably simpler to get right.  In addition, the
-  old code is known to work well on old machines.
+  Possibly also --chown
 
-  We could drop the rather large lib/getaddrinfo files.
+  (Debian #23628)
 
+                      --          --
 
-Other IPv6 stuff:
-  
-  Implement suggestions from http://www.kame.net/newsletter/19980604/
-  and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
 
-  If a host has multiple addresses, then listen try to connect to all
-  in order until we get through.  (getaddrinfo may return multiple
-  addresses.)  This is kind of implemented already.
+--diff						david.e.sewell	2002/03/15
 
-  Possibly also when starting as a server we may need to listen on
-  multiple passive addresses.  This might be a bit harder, because we
-  may need to select on all of them.  Hm.
+  Allow people to specify the diff command.  (Might want to use wdiff,
+  gnudiff, etc.)
 
-  Define a syntax for IPv6 literal addresses.  Since they include
-  colons, they tend to break most naming systems, including ours.
-  Based on the HTTP IPv6 syntax, I think we should use
- 
-     rsync://[::1]/foo/bar [::1]::bar
+  Just diff the temporary file with the destination file, and delete
+  the tmp file rather than moving it into place.
 
-  which should just take a small change to the parser code.
+  Interaction with --partial.
+
+  Security interactions with daemon mode?
+
+                      --          --
+
+
+Add daemon --no-detach and --no-fork options
+
+  Very useful for debugging.  Also good when running under a
+  daemon-monitoring process that tries to restart the service when the
+  parent exits.
+
+                      --          --
+
+DOCUMENTATION --------------------------------------------------------
+
+Update README
+
+                      --          --
+
+
+Keep list of open issues and todos on the web site
+
+                      --          --
+
+
+Update web site from CVS
+
+                      --          --
+
+
+Perhaps redo manual as SGML
+
+  The man page is getting rather large, and there is more information
+  that ought to be added.
+
+  TexInfo source is probably a dying format.
+
+  Linuxdoc looks like the most likely contender.  I know DocBook is
+  favoured by some people, but it's so bloody verbose, even with emacs
+  support.
+
+                      --          --
+
+LOGGING --------------------------------------------------------------
+
+Make dry run list all updates					2002/04/03
+
+  --dry-run is too dry
+
+  Mark Santcroos points out that -n fails to list files which have
+  only metadata changes, though it probably should.
+
+  There may be a Debian bug about this as well.
+
+                      --          --
+
+
+Memory accounting
+
+  At exit, show how much memory was used for the file list, etc.
+
+  Also we do a wierd exponential-growth allocation in flist.c.  I'm
+  not sure this makes sense with modern mallocs.  At any rate it will
+  make us allocate a huge amount of memory for large file lists.
+
+                      --          --
 
 
-Errors
+Improve error messages
 
   If we hang or get SIGINT, then explain where we were up to.  Perhaps
   have a static buffer that contains the current function name, or
@@ -369,207 +503,194 @@ Errors
   be good.
 
 
-File attributes
 
-  Transfer ACLs.  Need to think of a standard representation.
-  Probably better not to even try to convert between NT and POSIX.
-  Possibly can share some code with Samba.
+                      --          --
 
-Empty directories
 
-  With the current common --include '*/' --exclude '*' pattern, people
-  can end up with many empty directories.  We might avoid this by
-  lazily creating such directories.
+Better statistics:					Rasmus	2002/03/08
 
+  <Rasmus>
+      hey, how about an rsync option that just gives you the
+      summary without the list of files?  And perhaps gives
+      more information like the number of new files, number
+      of changed, deleted, etc. ?
 
-zlib
+  <mbp>
+      nice idea there is --stats but at the moment it's very
+      tridge-oriented rather than user-friendly it would be
+      nice to improve it that would also work well with
+      --dryrun
 
-  Perhaps don't use our own zlib.
+                      --          --
 
-  Advantages:
-   
-    - will automatically be up to date with bugfixes in zlib
 
-    - can leave it out for small rsync on e.g. recovery disks
+Perhaps flush stdout like syslog
 
-    - can use a shared library
+  Perhaps flush stdout after each filename, so that people trying to
+  monitor progress in a log file can do so more easily.  See
+  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
 
-    - avoids people breaking rsync by trying to do this themselves and
-      messing up
+                      --          --
 
-  Should we ship zlib for systems that don't have it, or require
-  people to install it separately?
 
-  Apparently this will make us incompatible with versions of rsync
-  that use the patched version of rsync.  Probably the simplest way to
-  do this is to just disable gzip (with a warning) when talking to old
-  versions.
+Log deamon sessions that just list modules
 
+  At the connections that just get a list of modules are not logged,
+  but they should be.
 
-logging
+                      --          --
 
-  Perhaps flush stdout after each filename, so that people trying to
-  monitor progress in a log file can do so more easily.  See
-  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
 
-  At the connections that just get a list of modules are not logged,
-  but they should be.
+Log child death on signal
 
   If a child of the rsync daemon dies with a signal, we should notice
   that when we reap it and log a message.
 
-  Keep stderr and stdout properly separated (Debian #23626)
+                      --          --
 
-  After we get the @RSYNCD greeting from the server, we know it's
-  version but we have not yet sent the command line, so we could just
-  remove the -z option if the server is too old.
 
-  For ssh invocation it's not so simple, because we actually use the
-  command line to start the remote process.  However, we only actually
-  do compression in token.c, and we could therefore once we discover
-  the remote version emit an error if it's too old.  I'm not sure if
-  that's a good tradeoff or not.
+Keep stderr and stdout properly separated (Debian #23626)
 
+                      --          --
 
-proxy authentication
 
-  Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
-  HTTP Basic Proxy-Authentication.
+Log errors with function that reports process of origin
 
-  Multiple schemes are possible, up to and including the insanity that
-  is NTLM, but Basic probably covers most cases.
-
-SOCKS
+  Use a separate function for reporting errors; prefix it with
+  "rsync:" or "rsync(remote)", or perhaps even "rsync(local
+  generator): ".
 
-  Add --with-socks, and then perhaps a command-line option to put them
-  on or off.  This might be more reliable than LD_PRELOAD hacks.
+                      --          --
 
-FAT support
 
-  rsync to a FAT partition on a Unix machine doesn't work very well at
-  the moment.  I think we get errors about invalid filenames and
-  perhaps also trying to do atomic renames.
+verbose output					David Stein	2001/12/20
+  
+  Indicate whether files are new, updated, or deleted
 
-  I guess the code to do this is currently #ifdef'd on Windows;
-  perhaps we ought to intelligently fall back to it on Unix too.
+  At end of transfer, show how many files were or were not transferred
+  correctly.
 
+                      --          --
 
-Better statistics:
 
-  <Rasmus> mbp: hey, how about an rsync option that just gives you the
-  summary without the list of files?  And perhaps gives more
-  information like the number of new files, number of changed,
-  deleted, etc. ?  <mbp> Rasmus: nice idea <mbp> there is --stats
-  <mbp> but at the moment it's very tridge-oriented <mbp> rather than
-  user-friendly <mbp> it would be nice to improve it <mbp> that would
-  also work well with --dryrun
+Add reason for transfer to file logging
 
-TDB:
+  Explain *why* every file is transferred or not (e.g. "local mtime
+  123123 newer than 1283198")
 
-  Rather than storing the file list in memory, store it in a TDB.
+                      --          --
 
-  This *might* make memory usage lower while building the file list.
 
-  Hashtable lookup will mean files are not transmitted in order,
-  though... hm.
+debugging of daemon						2002/04/08
 
-  This would neatly eliminate one of the major post-fork shared data
-  structures.
+  Add an rsyncd.conf parameter to turn on debugging on the server.
 
+                      --          --
 
-chmod:
 
-  On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote: > If we
-would add an option to do that functionality, I would vote for one >
-that was more general which could mask off any set of permission bits
-and > possibly add any set of bits.  Perhaps a chmod-like syntax if it
-could be > implemented simply.
+internationalization
 
-  I think that would be good too.  For example, people uploading files
-  to a web server might like to say
+  Change to using gettext().  Probably need to ship this for platforms
+  that don't have it.
 
-  rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
+  Solicit translations.
 
-  Ideally the patch would implement as many of the gnu chmod semantics
-  as possible.  I think the mode parser should be a separate function
-  that passes back something like (mask,set) description to the rest
-  of the program.  For bonus points there would be a test case for the
-  parser.
+  Does anyone care?  Before we bother modifying the code, we ought to
+  get the manual translated first, because that's possibly more useful
+  and at any rate demonstrates desire.
 
-  Possibly also --chown
+                      --          --
 
-  (Debian #23628)
+DEVELOPMENT --------------------------------------------------------
 
+Handling duplicate names
 
---diff
+  We need to be careful of duplicate names getting into the file list.
+  See clean_flist().  This could happen if multiple arguments include
+  the same file.  Bad.
 
-  Allow people to specify the diff command.  (Might want to use wdiff,
-  gnudiff, etc.)
+  I think duplicates are only a problem if they're both flowing
+  through the pipeline at the same time.  For example we might have
+  updated the first occurrence after reading the checksums for the
+  second.  So possibly we just need to make sure that we don't have
+  both in the pipeline at the same time.
 
-  Just diff the temporary file with the destination file, and delete
-  the tmp file rather than moving it into place.
+  Possibly if we did one directory at a time that would be sufficient.
 
-  Interaction with --partial.
+  Alternatively we could pre-process the arguments to make sure no
+  duplicates will ever be inserted.  There could be some bad cases
+  when we're collapsing symlinks.
 
-  Security interactions with daemon mode?
+  We could have a hash table.
 
-  (Suggestion from david.e.sewell)
+  The root of the problem is that we do not want more than one file
+  list entry referring to the same file.  At first glance there are
+  several ways this could happen: symlinks, hardlinks, and repeated
+  names on the command line.
 
+  If names are repeated on the command line, they may be present in
+  different forms, perhaps by traversing directory paths in different
+  ways, traversing paths including symlinks.  Also we need to allow
+  for expansion of globs by rsync.
 
-Incorrect timestamps (Debian #100295)
+  At the moment, clean_flist() requires having the entire file list in
+  memory.  Duplicate names are detected just by a string comparison.
 
-  A bit hard to believe, but apparently it happens.
+  We don't need to worry about hard links causing duplicates because
+  files are never updated in place.  Similarly for symlinks.
 
+  I think even if we're using a different symlink mode we don't need
+  to worry.
 
-Check "refuse options works"
+  Unless we're really clever this will introduce a protocol
+  incompatibility, so we need to be able to accept the old format as
+  well.
 
-  We need a test case for this...
+                      --          --
 
-  Was this broken when we changed to popt?
 
+Use generic zlib						2002/02/25
 
-PERFORMANCE ----------------------------------------------------------
+  Perhaps don't use our own zlib.
 
-MD4 file_sum
+  Advantages:
+   
+    - will automatically be up to date with bugfixes in zlib
 
-  If we're doing a local transfer, or using -W, then perhaps don't
-  send the file checksum.  If we're doing a local transfer, then
-  calculating MD4 checksums uses 90% of CPU and is unlikely to be
-  useful.
+    - can leave it out for small rsync on e.g. recovery disks
 
-  Indeed for transfers over zlib or ssh we can also rely on the
-  transport to have quite strong protection against corruption.
+    - can use a shared library
 
-  Perhaps we should have an option to disable this, analogous to
---whole-file, although it would default to disabled.  The file
-checksum takes up a definite space in the protocol -- we can either
-set it to 0, or perhaps just leave it out.
+    - avoids people breaking rsync by trying to do this themselves and
+      messing up
 
-MD4
+  Should we ship zlib for systems that don't have it, or require
+  people to install it separately?
 
-  Perhaps borrow an assembler MD4 from someone?
+  Apparently this will make us incompatible with versions of rsync
+  that use the patched version of rsync.  Probably the simplest way to
+  do this is to just disable gzip (with a warning) when talking to old
+  versions.
 
-  Make sure we call MD4 with properly-sized blocks whenever possible
-  to avoid copying into the residue region?
+                      --          --
 
-String area code
 
-  Test whether this is actually faster than just using malloc().  If
-  it's not (anymore), throw it out.
-	  
+TDB:								2002/03/12
 
-PLATFORMS ------------------------------------------------------------
+  Rather than storing the file list in memory, store it in a TDB.
 
-Win32
+  This *might* make memory usage lower while building the file list.
 
-  Don't detach, because this messes up --srvany.
+  Hashtable lookup will mean files are not transmitted in order,
+  though... hm.
 
-  http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html
+  This would neatly eliminate one of the major post-fork shared data
+  structures.
 
+                      --          --
 
-DEVELOPMENT ----------------------------------------------------------
 
-Splint
+Splint								2002/03/12
 
   Build rsync with SPLINT to try to find security holes.  Add
   annotations as necessary.  Keep track of the number of warnings
@@ -577,14 +698,8 @@ Splint
   security bugs.  Knowing the percentage of likely hits would be
   really interesting for other projects.
 
-Torture test
-
-  Something that just keeps running rsync continuously over a data set
-  likely to generate problems.
-
-Cross-testing
+                      --          --
 
-  Run current rsync versions against significant past releases.
 
 Memory debugger
 
@@ -592,168 +707,276 @@ Memory debugger
 
     http://devel-home.kde.org/~sewardj/
 
-Release script
+                      --          --
+
+
+Create release script
   
-  Update spec files
+  Script would:
+
+     Update spec files
 
-  Build tar file; upload
+     Build tar file; upload
 
-  Send announcement to mailing list and c.o.l.a.
+     Send announcement to mailing list and c.o.l.a.
   
-  Make freshmeat announcement
+     Make freshmeat announcement
 
-  Update web site
+     Update web site
 
+                      --          --
 
 
-TESTING --------------------------------------------------------------
+Add machines to build farm
 
-Cross-test versions
+  Cygwin (on different versions of Win32?)
 
-  Part of the regression suite should be making sure that we don't
-  break backwards compatibility: old clients vs new servers and so on.
-  Ideally we would test both up and down from the current release to
-  all old versions.
+  HP-UX variants (via HP?)
 
-  We might need to omit broken old versions, or versions in which
-  particular functionality is broken
+  SCO
 
-  It might be sufficient to test downloads from well-known public
-  rsync servers running different versions of rsync.  This will give
-  some testing and also be the most common case for having different
-  versions and not being able to upgrade.
 
 
-Test on kernel source
+                      --          --
 
-  Download all versions of kernel; unpack, sync between them.  Also
-  sync between uncompressed tarballs.  Compare directories after
-  transfer.
+PERFORMANCE ----------------------------------------------------------
 
-  Use local mode; ssh; daemon; --whole-file and --no-whole-file.
+File list structure in memory
 
-  Use awk to pull out the 'speedup' number for each transfer.  Make
-  sure it is >= x.
+  Rather than one big array, perhaps have a tree in memory mirroring
+  the directory tree.
 
+  This might make sorting much faster!  (I'm not sure it's a big CPU
+  problem, mind you.)
 
-Test large files
+  It might also reduce memory use in storing repeated directory names
+  -- again I'm not sure this is a problem.
 
-  Sparse and non-sparse
+                      --          --
 
-Mutator program
 
-  Insert bytes, delete bytes, swap blocks, ...
+Traverse just one directory at a time
 
-configure option to enable dangerous tests
+  Traverse just one directory at a time.  Tridge says it's possible.
 
-If tests are skipped, say why.
+  At the moment rsync reads the whole file list into memory at the
+  start, which makes us use a lot of memory and also not pipeline
+  network access as much as we could.
 
-Test daemon feature to disallow particular options.
+                      --          --
 
-Pipe program that makes slow/jerky connections.
 
-Versions of read() and write() that corrupt the stream, or abruptly
-fail
+Hard-link handling
 
-Separate makefile target to run rough tests -- or perhaps just run
-them every time?
+  At the moment hardlink handling is very expensive, so it's off by
+  default.  It does not need to be so.
 
-Test "refuse options" works
+  Since most of the solutions are rather intertwined with the file
+  list it is probably better to fix that first, although fixing
+  hardlinks is possibly simpler.
 
-  What about for --recursive?
+  We can rule out hardlinked directories since they will probably
+  screw us up in all kinds of ways.  They simply should not be used.
 
-  If you specify an unrecognized option here, you should get an error.
+  At the moment rsync only cares about hardlinks to regular files.  I
+  guess you could also use them for sockets, devices and other beasts,
+  but I have not seen them.
 
+  When trying to reproduce hard links, we only need to worry about
+  files that have more than one name (nlinks>1 && !S_ISDIR).
 
-DOCUMENTATION --------------------------------------------------------
+  The basic point of this is to discover alternate names that refer to
+  the same file.  All operations, including creating the file and
+  writing modifications to it need only to be done for the first name.
+  For all later names, we just create the link and then leave it
+  alone.
 
-Update README
+  If hard links are to be preserved:
 
-Keep list of open issues and todos on the web site
+    Before the generator/receiver fork, the list of files is received
+    from the sender (recv_file_list), and a table for detecting hard
+    links is built.
 
-Update web site from CVS
+    The generator looks for hard links within the file list and does
+    not send checksums for them, though it does send other metadata.
 
+    The sender sends the device number and inode with file entries, so
+    that files are uniquely identified.
 
-Perhaps redo manual as SGML
+    The receiver goes through and creates hard links (do_hard_links)
+    after all data has been written, but before directory permissions
+    are set.
 
-  The man page is getting rather large, and there is more information
-  that ought to be added.
+  At the moment device and inum are sent as 4-byte integers, which
+  will probably cause problems on large filesystems.  On Linux the
+  kernel uses 64-bit ino_t's internally, and people will soon have
+  filesystems big enough to use them.  We ought to follow NFS4 in
+  using 64-bit device and inode identification, perhaps with a
+  protocol version bump.
 
-  TexInfo source is probably a dying format.
+  Once we've seen all the names for a particular file, we no longer
+  need to think about it and we can deallocate the memory.
 
-  Linuxdoc looks like the most likely contender.  I know DocBook is
-  favoured by some people, but it's so bloody verbose, even with emacs
-  support.
+  We can also have the case where there are links to a file that are
+  not in the tree being transferred.  There's nothing we can do about
+  that.  Because we rename the destination into place after writing,
+  any hardlinks to the old file are always going to be orphaned.  In
+  fact that is almost necessary because otherwise we'd get really
+  confused if we were generating checksums for one name of a file and
+  modifying another.
 
+  At the moment the code seems to make a whole second copy of the file
+  list, which seems unnecessary.
 
-BUILD FARM -----------------------------------------------------------
+  We should have a test case that exercises hard links.  Since it
+  might be hard to compare ./tls output where the inodes change we
+  might need a little program to check whether several names refer to
+  the same file.
 
-Add machines
+                      --          --
 
-  Cygwin (on different versions of Win32?)
 
-  HP-UX variants (via HP?)
+Allow skipping MD4 file_sum					2002/04/08
 
-  SCO
+  If we're doing a local transfer, or using -W, then perhaps don't
+  send the file checksum.  If we're doing a local transfer, then
+  calculating MD4 checksums uses 90% of CPU and is unlikely to be
+  useful.
 
+  Indeed for transfers over zlib or ssh we can also rely on the
+  transport to have quite strong protection against corruption.
 
-LOGGING --------------------------------------------------------------
+  Perhaps we should have an option to disable this,
+  analogous to --whole-file, although it would default to
+  disabled.  The file checksum takes up a definite space in
+  the protocol -- we can either set it to 0, or perhaps just
+  leave it out.
 
-  Perhaps flush stdout after each filename, so that people trying to
-  monitor progress in a log file can do so more easily.  See
-  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
+                      --          --
 
-  At the connections that just get a list of modules are not logged,
-  but they should be.
 
-  If a child of the rsync daemon dies with a signal, we should notice
-  that when we reap it and log a message.
+Accelerate MD4
 
-  Keep stderr and stdout properly separated (Debian #23626)
+  Perhaps borrow an assembler MD4 from someone?
 
-  Use a separate function for reporting errors; prefix it with
-  "rsync:" or "rsync(remote)", or perhaps even "rsync(local
-  generator): ".
+  Make sure we call MD4 with properly-sized blocks whenever possible
+  to avoid copying into the residue region?
 
-verbose output
-  
-  Indicate whether files are new, updated, or deleted
+                      --          --
 
-  At end of transfer, show how many files were or were not transferred
-  correctly.
 
--vv
+String area code
 
-  Explain *why* every file is transferred or not (e.g. "local mtime
-  123123 newer than 1283198")
+  Test whether this is actually faster than just using malloc().  If
+  it's not (anymore), throw it out.
 
+                      --          --
 
-debugging of daemon
+TESTING --------------------------------------------------------------
 
-  Add an rsyncd.conf parameter to turn on debugging on the server.
+Torture test
 
+  Something that just keeps running rsync continuously over a data set
+  likely to generate problems.
 
+                      --          --
 
-NICE -----------------------------------------------------------------
 
---no-detach and --no-fork options
+Cross-test versions						2001/08/22
 
-  Very useful for debugging.  Also good when running under a
-  daemon-monitoring process that tries to restart the service when the
-  parent exits.
+  Part of the regression suite should be making sure that we
+  don't break backwards compatibility: old clients vs new
+  servers and so on.  Ideally we would test both up and down
+  from the current release to all old versions.
 
-hang/timeout friendliness
+  Run current rsync versions against significant past releases.
 
-internationalization
+  We might need to omit broken old versions, or versions in which
+  particular functionality is broken
 
-  Change to using gettext().  Probably need to ship this for platforms
-  that don't have it.
+  It might be sufficient to test downloads from well-known public
+  rsync servers running different versions of rsync.  This will give
+  some testing and also be the most common case for having different
+  versions and not being able to upgrade.
 
-  Solicit translations.
+  The new --protocol option may help in this.
 
-  Does anyone care?  Before we bother modifying the code, we ought to
-  get the manual translated first, because that's possibly more useful
-  and at any rate demonstrates desire.
+                      --          --
+
+
+Test on kernel source
+
+  Download all versions of kernel; unpack, sync between them.  Also
+  sync between uncompressed tarballs.  Compare directories after
+  transfer.
+
+  Use local mode; ssh; daemon; --whole-file and --no-whole-file.
+
+  Use awk to pull out the 'speedup' number for each transfer.  Make
+  sure it is >= x.
+
+                      --          --
+
+
+Test large files
+
+  Sparse and non-sparse
+
+                      --          --
+
+
+Create mutator program for testing
+
+  Insert bytes, delete bytes, swap blocks, ...
+
+                      --          --
+
+
+Create configure option to enable dangerous tests
+
+                      --          --
+
+
+If tests are skipped, say why.
+
+                      --          --
+
+
+Test daemon feature to disallow particular options.
+
+                      --          --
+
+
+Create pipe program for testing
+
+  Create pipe program that makes slow/jerky connections for
+  testing Versions of read() and write() that corrupt the
+  stream, or abruptly fail
+
+                      --          --
+
+
+Create test makefile target for some tests
+
+  Separate makefile target to run rough tests -- or perhaps
+  just run them every time?
+
+                      --          --
+
+
+Test "refuse options" works
+
+  What about for --recursive?
+
+  If you specify an unrecognized option here, you should get an error.
+
+  We need a test case for this...
+
+  Was this broken when we changed to popt?
+
+                      --          --
+
+RELATED PROJECTS -----------------------------------------------------
 
 rsyncsh
 
@@ -763,22 +986,33 @@ rsyncsh
    current host, directory and so on.  We can probably even do
    completion of remote filenames.
 
+                      --          --
 
-RELATED PROJECTS -----------------------------------------------------
 
 http://rsync.samba.org/rsync-and-debian/
 
+
+                      --          --
+
+
 rsyncable gzip patch
 
   Exhaustive, tortuous testing
 
   Cleanups?
 
+                      --          --
+
+
 rsyncsplit as alternative to real integration with gzip?
 
+                      --          --
+
+
 reverse rsync over HTTP Range
 
   Goswin Brederlow suggested this on Debian; I think tridge and I
   talked about it previous in relation to rproxy.
 
+                      --          --