<?XML version="1.0" encoding="utf-8"?>



FREEDUP
              
allows you to reclaim space on your drive.  It's simple.

Introduction
Syntax
Download
Examples
How_it_Works
Feedback
Hints
===============================================================================

Freedup walks through the file trees (directories) you specify.

When it finds two identical files on the same device, it hard links them
together.

In this case two or more files still exist in their respective directories,

but only one copy of the data is stored on disk; both directory entries
point to the same data blocks.

[FREEDUP IN ACTION]

If both files reside on different devices, then they are symlinked together

except there are relative paths given (and the -s option is unused).


*****  SYNTAX: 	freedup [options] [<tree> ...] *****


-a         provide compatibility to freedups by William Stearns.[=-gup]
-c         count file space savings per linked file.
-d         requires the modification time stamps to be equal.
-f         requires the path-stripped file names to be equal.
-g         requires groups to be equal.
-h         shows this help. [other option are ignored]
-i         decide in interactve mode what to do with a pair of identical files.
-l         only allow hardlinks. No symlinks are established.
-m <bytes> only touch larger files. (deprecated: use -o -size +#c)
-n         do not really perform links [no action].
-o <opts>  pass an option string to the initially called find command.
-p         requires file permissions to be equal.
-q         produces no output during the run (also toggles -c and -v to off)
-s         generate symlinks although some given paths are relative.
-u         requires users to be equal.
-v         display shell commands to perform linking [verbose].
-w         only weak symbolic links allthough hardlinks might be possible
-#         disable usage of hash functions, but compare all files byte by byte.
-0         disable linking of empty files i.e. files of size 0.
<tree>     any directory tree to scan for duplicate files recursively.

	Options are toggle switches. Their final state applies.

	Later <tree> entries are linked to the earlier ones.

	Providing no <tree> means to take filenames(!) from stdin.

	When standard input is used the option -o has no effect.

	Version 0.9-1 by Andreas Neuper 2007.



***** Download *****


    *
    * ChangeLog .txt
    * Bugs_and_ToDos .txt
    * freedup_linked(libc.so.6)_and_stripped
    * freedup.exe_(Version_0.7)_linked_for_cygwin
    * Packman_RPMs_for_SuSE_(by_Toni_Graffy)
    * RPM_File_Version_0.9
    * BZ2_Archive_Version_0.9
    * TGZ_Archive_Version_0.9

===============================================================================

    *
    * RPM_File_Version_0.8
    * BZ2_Archive_Version_0.8
    * TGZ_Archive_Version_0.8

===============================================================================

    *
    * RPM_File_Version_0.7
    * BZ2_Archive_Version_0.7
    * TGZ_Archive_Version_0.7

===============================================================================

    *
    * RPM_File_Version_0.6
    * BZ2_Archive_Version_0.6
    * TGZ_Archive_Version_0.6

===============================================================================

    *
    * RPM_File_Version_0.5
    * BZ2_Archive_Version_0.5
    * TGZ_Archive_Version_0.5

===============================================================================

    *
    * RPM_File_Version_0.4
    * BZ2_Archive_Version_0.4
    * TGZ_Archive_Version_0.4

===============================================================================

    *
    * RPM_File_Version_0.3
    * BZ2_Archive_Version_0.3
    * TGZ_Archive_Version_0.3

===============================================================================

    *
    * RPM_File_Version_0.2
    * BZ2_Archive_Version_0.2
    * TGZ_Archive_Version_0.2

===============================================================================

    *
    * RPM_File_Version_0.1
    * BZ2_Archive_Version_0.1
    * TGZ_Archive_Version_0.1

===============================================================================


Versions 0.2 and 0.3 have some quality problems when taking more than one input
tree.

Other similar or related implementations that might be interesting for
comparison may be found at
Wikipedia.org/wiki/fdupes.


*****  How freedup works  *****


   1.
   2.  scan all directory trees recursively for all regular files
   3.  build a list of those files and keep their name, lstat() and arg
      position
   4.  sort the files by comparing their sizes using qsort()
   5.  in case the comparison has to report equal file size
      	 additional properties are compared
   6.  most property checks have to be added using command line options
   7.  if all demands are fullfilled, the files are compared block by block
      (4k)
   8.  if both files are identical and on the same file system they will
      	 be renamed, hard linked, renamed file removed.
   9.  if hardlinking is not possible soft links are tried,
      	 except one of the paths is not starting at root (but can be forced)
  10.  sorting is repeated, the reason why it is needed was not checked yet
  11.  finally a short report is delivered



*****  Examples  *****


****  An Example using FREEDUP in native mode:  ****


SYSTEM1:root# freedup -cf /home/freedup/holidays/2006/family /home/freedup/
holidays/2006/friends
Run through both trees, compare the files (selections of my holiday snapshots)
and link those files, that have identical names and contents. When linking say
how much space each link saves.
**** An Example using LOCATE: ****
The intention is to see, what freedup would do on all registered JPEGs on
SYSTEM1. We do run the command as root, just to see all allowed links.
SYSTEM1:root# locate '*.jpg' | freedup -nv
lstat() failed while reading file statistics: No such file or directory
lstat() failed while reading file statistics: No such file or directory
...
lstat() failed while reading file statistics: No such file or directory
lstat() failed while reading file statistics: No such file or directory
1085 files to investigate

ln "/opt/kde3/share/apps/pixie/doc/en/pixielogo.jpg" "/opt/kde3/share/apps/
pixie/pixielogo.jpg"
ln "/opt/kde3/share/apps/quanta/templates/binaries/images/jpg/demo.jpg" "/opt/
kde3/share/apps/quanta/templates/images/jpg/demo.jpg"
ln "/usr/lib/webmin/mscstyle3/images/cats/net.jpg" "/usr/lib/webmin/mscstyle3/
images/cats_over/net.jpg"
ln "/usr/lib/webmin/mscstyle3/images/cats/webmin.jpg" "/usr/lib/webmin/
mscstyle3/images/cats_over/webmin.jpg"
ln "/usr/share/games/freedroid/graphics/transfer.jpg" "/usr/src/packages/BUILD/
freedroid-0.8.4/graphics/transfer.jpg"
ln "/usr/share/doc/packages/id3lib-devel/attilas_id3logo.jpg" "/usr/src/
packages/BUILD/audacity-src-1.0.0/id3lib/doc/attilas_id3logo.jpg"
ln "/usr/share/doc/packages/mgp/sample/mgp3.jpg" "/usr/X11R6/lib/X11/mgp/
mgp3.jpg"
ln "/usr/share/doc/packages/mgp/sample/mgp2.jpg" "/usr/X11R6/lib/X11/mgp/
mgp2.jpg"
ln "/usr/share/doc/packages/mgp/sample/mgp1.jpg" "/usr/X11R6/lib/X11/mgp/
mgp1.jpg"
ln "/opt/kde3/share/apps/kworldclock/maps/caida_bw/1280.jpg" "/usr/X11R6/lib/
X11/xglobe/caida_bw_1280.jpg"
ln "/opt/kde3/share/apps/kworldclock/maps/caida/1280.jpg" "/usr/X11R6/lib/X11/
xglobe/caida_1280.jpg"
ln "/opt/kde3/share/apps/kworldclock/maps/alt/1200.jpg" "/usr/X11R6/lib/X11/
xglobe/alt_1200.jpg"
ln "/opt/kde3/share/apps/kworldclock/maps/bio/1600.jpg" "/usr/X11R6/lib/X11/
xglobe/bio_1600.jpg"
ln "/opt/kde3/share/apps/kworldclock/maps/depths/1440.jpg" "/usr/X11R6/lib/X11/
xglobe/depths_1440.jpg"
ln "/opt/kde3/share/apps/kworldclock/maps/mggd/1440.jpg" "/usr/X11R6/lib/X11/
xglobe/mggd_1440.jpg"
15 files of 1085 will be replaced by links.
The total size of replacable files is 1506387 bytes.
md5 hash algorithm had to read 86 files to avoid 31 file comparisons.
Initially failed lstat() executions show, that the locate database was not
updated since the last JPEG removals. The stats of 1085 files have been read
and compared after that. 15 files turned out to match 15 others. Most of them
seem having been transfered by install commands. The amount of saved space is
about 1.5MB. Using the md5 hashing was not a good idea in this case. Instead of
reading and evaluating a hash sum it would have been easier to read 62 files
for direct comparison.
Please be aware, that the displayed commands cannot be piped into a file and
executed later. You need to remove the target first, before you link it.
Otherwise you will receive "file exists".
**** An Example using FIND: ****
SYSTEM1:root#  find /usr/src/linux -type f -xdev -atime +12 | freedup -nv
SYSTEM1:/home/freedup # find /usr/src/linux -type f -xdev -atime +12 | freedup
-c
Taking file names from stdin

0 files to investigate

0 files of 0 replaced by links.
The total size of replaced files was 0 bytes.
md5 hash algorithm had to read 0 files to avoid 0 file comparisons.
The starting tree is not a tree but a symbolic link. You need to append a slash
to descend into the referenced directory. This trick only works for the
starting tree.
SYSTEM1:/home/freedup # find /usr/src/linux/ -type f -xdev -atime +12 | time
freedup -c
Taking file names from stdin

1045 files to investigate

0 files of 1045 replaced by links.
The total size of replaced files was 0 bytes.
md5 hash algorithm had to read 0 files to avoid 0 file comparisons.
0.00user 0.01system 0:00.29elapsed 6%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1310minor)pagefaults 0swaps
You see, that I tried freedup already before. No file that was not touched for
12 days did match another. -xdev was used to confine the find command to the
local directory. The prefix command time was used to show some not very
interesting performance statistics. Another way to write it down is
SYSTEM1:/home/freedup # freedup -c /usr/src/linux/ -o "-xdev -atime +12"
The option passing really pays if you need to scan a number of trees. Just
compare yourself:
SYSTEM1:/home/freedup # freedup -c /usr/src/linux-2.6.12.1 /usr/src/linux-
2.6.21.1 -o "-xdev -atime +12"
versus
SYSTEM1:/home/freedup # ( find /usr/src/linux-2.6.12.1 -type f -xdev -atime +12
; find /usr/src/linux-2.6.21.1 -type f -xdev -atime +12 ) | time freedup -c
Please note, that I omitted (incorrectly) the find default action -print.
**** Directories to use freedup for the first time: ****
    * Several versions of software contain identical files, e.g. linux kernel.
    * You have multiple copies of the file COPYING in /usr/doc or /usr/share/
      DOC
    * Depending on your system, the following might be good places
      to try linking (size in parentheses are my SuSE 9.2 savings):
      freedup /lib/kbd                                            (463kB)
      freedup /usr/doc /usr/share/doc
      freedup /usr/src/linux-2.6.10 /usr/src/ 20/68329            (9k)
      linux-2.6.11
      freedup /usr/src/linux-2.6.1*           930/207000          (1.52MB)
      freedup /usr/share                      37/163335           (2.6MB)
      freedup /usr/lib                        22/41368            (97kB)
      freedup /usr/src/packages/BUILD         3030/108427         (17.5MB)
      freedup /usr/man /usr/share/man         14/10772            (19kB)
      freedup /usr/share/locale /etc/locale   36/1436 files       (29kB)
      Warwick_Pooles_Personal_Files           26% space reduction
    * Directories holding multimedia files are good candidates.
***** Questions *****
The Frequently Asked Ones are not on this page, since there is an excellent FAQ
section
on William Stearns freedups page http://www.stearns.org/freedups/README
Please note, that freedup is a completely independant implementation, with
other means and other capabilities.
The Main difference is the ability of freedup to provide symlinks from
different file systems.
And here are my questions.
How_do_you_like_the_performance_of_freedup?
Are_the_provided_packages_what_you_want?
What_about_the_documentation._Is_it_sufficient?
Please provide me some feedback here or per mail.
***** Contacts and Credits *****
Please send comments, suggestions, bug reports, patches, and/or additions to
Andreas_Neuper .
A_lot_of_different_web_sites_were_helpful. I used William Stearns freedups for
many years quite successfully until I lacked some features, which freedup
fixes.
Thanks to Tony_Graffy_for_compiling_and_publishing_SuSE_RPMs_at_packmans.
Thanks to Michael_Riepe_and_iX_(11/2007)_on_writing_and_publishing
"Entdoppeltes_Lottchen", which is a German article about "Dateien mit gleichem
Inhalt finden und beseitigen". He spent one third of the paraphrases on
freedups.
