ln - command line hardlinks
Last Updated June 28, 2014 Version 2.8.5.2
spacer
Quick Start Download
Documentation
Frequently asked questions (FAQ)
History
Donations

Introduction The NTFS filesystem supports a lot of features like Hardlinks, Junctions Symbolic Links and Mount Points, which are sparsley supported by standard tools. This tool is kind of a NTFS Swiss Army Knife and can do lots of things, like create SmartCopies, which preserve the inner and outer Hardlink/Junction/SymbolicLink structure, does hardlink based incremental Backups and a lot of other things which can not be found anywhere.

Installation copy ln.exe to some directory referenced by your PATH environment variable. %systemroot% is a good place.

Using ln ln.exe is a typical command line utility, like those found within the resource kit. Its' highlights are as follows:
 
Normal linking

Just for completeness: ln.exe can do the normal ln stuff too.


ln sourcefile.txt destination.txt

In the above example sourcefile.txt gets hard-linked to destination.txt



--absolute
Absolute
Symbolic Links
The target of a symbolic link can either be
  • a fully qualified path starting at the root of a drive, e.g e:\data\cpp\myfile.txt
  • also fully qualified path but pointing to a volume, e.g. \\?\Volume{f6c8b244-1ce4-11e0-a607-005056c00008}\
  • or can be be specified relatively, e.g ..\..\data\cpp\myfile.txt

ln.exe by default tries to create relative target path names for symbolic links as long as this is possible, e.g the file and its target are on the same logical drive. Having relative symbolic link targets is much smarter especially when the target of links is in the same directory. If a symbolic link and its target are on different drives, ln.exe uses absolute pathnames.

If the creation of absolute target pathnames should be forced use the --absolute option.


ln --absolute --symbolic sourcefile.txt destination.txt

In the above example sourcefile.txt gets symbolically linked to destination.txt, but the target path in the symbolic link is absolute.

ln --absolute --symbolic SourceDirectory DestinationDirectory

In the above example a symbolic link directory SourceDirectory gets symbolically linked to DestinationDirectory, but the target path in the symbolic link is absolute.



--backup
Backup Mode

Using the Backup Mode ACLs aka Access Control Lists, which contain the security for Files, Folders, Junctions or SymbolicLinks, and Encrypted Files are also copied.

ln.exe thus copies
  • Alternative Streams on files and folders
  • EA Records on files and folders (rarely used)
  • Reparse Info
  • File Attributes
  • Timestamps: Creation Time, Last Access Time, Last Write Time
  • Sparse Files and Alternative Sparse Streams
  • Encrypted files
  • ACLs

The --backup switch is a modifier to all other operations like e.g --copy, --mirror or --delorean.

ln --backup --copy x:\SourcePath x:\DestinationPath

All files with all the above mentioned attributes are copied from x:\SourcePath to x:\DestinationPath

The --backup mode automatically enables the --keepsymlinkrelation modifier.


--copy
Smart Copy

Smart Copy basically creates a copy of the directory structure from the source location to the destination, but it preserves the inner hardlink structure and inner junction/symbolic link relations of the source, and recreates this inner hardlink structure and inner junction/symbolic link relation at the destination location:

With hardlinks it behaves as follows:

Smart Copy How To

By closely looking at the above picture one can find three different types of files:

Normal Files

The file B is a normal file. It gets copied as any other copy tool would do.


Saturated Hardlinks

The files E and F are hardlinked together. In LSE terms they are called Saturated Hardlinks, because the reference count, which is here 2, matches the number of occurrences below 'Folder 1', which is here 2.

In General: A hardlink is called Saturated with respect to a folder F, if the number of occurrences below the folder F matches the reference count.

Saturated Hardlinks can be copied completely via Smart Copy.


Unsaturated Hardlinks The Files A, C, D are hardlinked together. In LSE terms they are called Unsaturated Hardlinks, because the reference count, which is here 3, does not match the number of occurrences below 'Folder 1', which is here 2. Only C and D are below Folder 1.

In General: A hardlink is called Unsaturated with respect to a folder F, if the number of occurrences below the folder F is smaller than the reference count.

Unsaturated Hardlinks can only be partially copied by Smart Copy. In the above example C and D are hardlinked together in the destination location, but the hardlink to A is broken. This means that the reference count of C and D is 2 in the destination location.


With junctions or symbolic link directories the default behaviour during smartcopy is as follows:

Behaviour for inner junctions

By closely looking at the above picture one can find three different types of folders/junctions:

Normal Folders

The folder 'Folder 3' is a normal folder. It gets copied with its content as any other copy tool would do.


Inner Junctions
Symlinks

The folder 'Inner Junction/Symlink' is targeted at 'Inner Junction/Symlink Target'. In LSE terms this kind of folder is called Inner Junction/Symlink, because its target points to a folder, which is below the common anchor 'Folder 1'.

Inner Junctions/Symlinks are restored properly via Smart Copy in the destination location.


Outer Junctions
Symlinks

The folder 'Outer Junction/Symlink' is targeted at the folder 'Outer Junction/Symlink Target'. In LSE terms this kind of folder is called Outer Junction/Symlink, because its target points to a folder, which is in parallel and thus outside the anchor 'Folder 1'.

Outer Junctions/Symlinks can be handled in three different ways. Please see the section on Outer Junction/Symlink Handling.


Windows7/8 and Windows Vista support Symbolic Links, which behave as follows during Smart Copy:

Behaviour for inner symbolic links

By closely looking at the above picture one can find three different types of files/symbolic links:

Normal Files

The file A is a normal file. It gets copied as any other copy tool would do.


Inner Symbolic
Links

The symbolic link 'Inner Symlink' is targeted at 'Inner Symlink Target'. In LSE terms this kind of symbolic link is called Inner Symlink, because its target points to a file, which is below the common anchor 'Folder 1'.

Inner Symlinks are restored properly via Smart Copy at the destination location.


Outer Symbolic
Links

The symlink 'Outer Symlink' is targeted at the folder 'Outer Symlink Target'. In LSE terms this kind of symbolic link is called Outer Symlink, because its target points to a file, which is in parallel and thus outside the anchor 'Folder 1'.

Outer Symlinks are handled by Smart Copy depending on the Outer Junction/Symbolic Link handling.


e.g.:

ln --copy x:\dir\dir2 y:\dir\newdir

In the above example the directories and files below x:\dir\dir2 are copied to y:\dir\newdir, and all hardlinks/junctions/symbolic links within x:\dir\dir2 are copied/tied to their new locations under y:\dir\newdir.

The --copy operation only copies if the files in the destination are either not there, or if the files in the source are newer. Otherwise the item is skipped. See also the note on quotes.

If you use ln.exe as a backup tool with the --copy option, please make sure you test that everything is copied. I tested ln --copy many times, and had no problems, but when it comes to backing up data, one has to be careful.

Smart Copy is a must if e.g. the whole content of a hard disk, which has lots of hardlinks/junctions/symbolic links, should be copied to another hard disk. During the Smart Copy operation empty folders get copied too and the time stamps of folders/junctions/symbolic links are also restored at the corresponding destination locations.

Because Smart Copy creates inner hardlinks/junctions/symbolic links, this feature is only available on NTFS volumes.

Smart Copy also processes all available alternative NTFS streams, EA Records, Sparse Files of a file. To also copy the ACLs, see the Backup Mode

Output

See Output among Delorean Copy for a detailed description of Smart Copies output.


Shell Integration

The Copy or Smart Copy functionality is also available via Link Shell Extension

--crop
Crop/Unroll/Splice
Outer Junctions/
Symbolic Links

During SmartCopy, Smart Mirror, Delorean Copy and Clone so called Outer Junctions/Symlink directories may need processing. There are 3 different ways to deal with those Outer Junctions/Symlink directories:

Crop

Crop breaks links to Outer Junctions/Symlink directories in the destination.

Crop also applies to Outer Symlink Files.

Cropping outer Junctions

In the above example Folder1 is copied to Destination/Folder1, but Outer Junction/Symlink is not available in the destination, because Folder1/Outer Junction/Symlink pointed to Folder0/Outer Junction/Symlink Target, which is not below Folder1.

The objective behind cropping Outer Junctions/Symlink Directories is to get a pure copy during Smart Copy, Smart Mirror, Delorean Copy and Clone without connections to the source.

Enabling Crop for Outer Junction/Symbolic Links
Crop can be selected via the configuration tool.



Unroll

Unroll follows Outer Junctions/Symlink Directories and rebuilds the content of Outer Junctions/Symlink Directories inside the hierarchy at the destination location.

Unroll also applies to Outer Symlink Files, which means, that unroll causes the target of Outer Symlink Files to be copied to the destination location.


Unroll Outer Junctions

In the above example Folder1 is copied to Destination/Folder1, and Outer Junction/Symlink and all the files/directories below Outer Junction/Symlink Target are copied to the folder Outer Junction/Symlink in the destination.

The objective behind unrolling Outer Junctions/Symlink Directories is to get everything with which the source is connected and rebuild it as separate copy in the destination. It resembles the 'hair of the elephant' pattern: Pull on a hair of an elephant, and get the whole elephant.


The command line invocation below unrolls all junctions/symbolic link directories below "c:\Folder 1"

ln --unroll --copy "c:\Folder 1" "c:\Destination\Folder 1"

In certain situations only certain junctions/symbolic link directories should be unrolled during SmartCopy, DeloreanCopy, SmartMirror or HardlinkClone. To accomplish this ln.exe can be run with wildcard expressions to specify certain directories for unrolling.

ln --unroll "MyJunc*" --copy "c:\Folder 1" "c:\Destination\Folder 1"

To specify junctions/symbolic link directories for unrolling more fine grained regular expressions can be given:

ln --unrollregexp "[a-z]*" --copy "c:\Folder 1" "c:\Destination\Folder 1"

The regular expressions used must conform to POSIX 1003.2 extended RE (ERE) syntax. The regular expression specified under --unrollregexp is applied to the whole source path:

ln --unrollregexp "Folder 1\\MyJunc*" --copy "c:\Folder 1" "c:\Destination\Folder 1"

In this example only Outer Junctions/Symbolic Link Directories, matching "Folder 1\\MyJunc*" are unrolled. Please make sure that the back slash \ is escaped by a double \\ as shown in the above example.


Advanced thoughts on Unrolling

The picture above was just the simplest case, because Unroll does much more when it encounters complex situations. Think of a outer junctions/symbolic links, which itself contains junctions/symbolic links, which are inner with respect to the first outer junction symbolic link:

Unroll Inner Outer Junctions

In the above example Folder1 is copied to Destination/Folder1, and Outer Junction/Symlink and is unrolled as expected, but since Junction/Symlink is a inner junction with respect to Outer Junction/Symlink Target, the junction/symlink relation is restored in the destination.



This kind of nesting can be much more complex:

Unroll Inner Outer Junctions

In the above example Folder1 is copied to Destination/Folder1, and Outer Junction/Symlink and is unrolled as expected, but then it starts to get fascinating, because we have two levels of outer junctions/symlinks which all have respective inner junctions/symlinks, and which are restored properly. Once you have followed through the above picture, you got it. It is not simple I know, but it is necessary to properly unroll.



And complexity increases if symbolic link files are within unrolled outer junctions/symbolic links:

Unroll Inner Outer Junctions

In the above example Folder1 is copied to Destination/Folder1, and Outer Junction/Symlink and is unrolled as expected, but it contains Inner Outer Symlink which points to Inner Outer Symlink Target and this is a inner junction/symbolic link with respect to Outer Junction/Symlink Target

But worth mentioning is the Symbolic Link Outer Symlink, which would be a definitive outer symbolic link, but since its targets parent-directory Outer Junction/Symlink Target is unrolled, Outer Symlink becomes an inner symbolic link with respect to Folder1.

File1 Symlink is also a outer symbolic link, but its target parent-directory Folder2 is not that lucky to get unrolled, so in the destination File1 Symlink is not a symbolic link any more, but a copy of the symbolic links' target.



Nested Reparse Points are also an interesting use case, which the algorithm has to tackle with:

Nested Reparse Points

In the above example Folder1 is copied to Destination, and Junction/Symlink F0 and is unrolled as expected, but it contains inner nested reparse points. Nested means Reparse point pointing to Reparse Points



The Unroll functionality also opens up the possibility to have circular Junction/Symbolic Link relations among a set of copied items:

Unroll Junctions Circularities

In the above example Folder1 is copied via the --unroll option to Destination/Folder1. Smart Copy/Smart Mirror and Delorean Copy operations can deal with the above shown circularities and break circularities by not following the affected Junction/Symbolic Link.



Junctions/Symbolic Links can also point to FAT drives or other NTFS drives requiring as a prerequisite unique Disk-IDs on all disks, which are chained together via Junctions/Symbolic links:

Unroll Junctions Unique DiskID

Hardlink siblings are found by matching the per NTFS volume unique file-id, but if more volumes are chained together it might happen that the same file-ids can be found on two different NTFS volumes. To address this all operations use the disk-id and the file-id to match hardlink siblings.

Furthermore it is not allowed to have the disk-id 0xFFFF-FFFF, because the algorithms use this as internal indicator of a FAT drive.



The Unroll option also allows to point multiple junctions to point to the same target location, which causes the algorithms to traverse the same items many times:

Unroll Junctions Multi-Traverse

At the first glance multiple traversing of items looks simple, but for files this means that multiple traversed files are the same in the destination and are hardlinked together. So don't be confused when you see hardlinks, which have never ever been there before.


Copying Junction/Symbolic Links from mapped network drives also needs extra care, because junctions on the remote machine relate with path references only valid on the remote machine to each other:

Mapped Network Drive Reparsepoint Resolution
The example in the above picture shows such a situation
  • X:\Src\Foo\Junction/Symlink is a junction on a Remote Machine, which points to X:\Src\Foo\Folder. It is a valid Junction.
  • X:\Src\Foo is sharded via \\RemoteMachine\MyShare, which still is no problem.
  • \\RemoteMachine\MyShare mapped to T:\ on a local machine is fine, but the Junction T:\Junction/Symlink, if asked for its target, will still return X:\Src\Foo\Folder even on the Local Machine. One could expect that the target is T:\Folder, but it is not for Junctions.
  • When SmartCopy/Mirror/Delorean comes across such a situation, it translates the Junction correctly when it is copied onto the Local Machines drive Y:\, so that the final Junction target points correctly to Y:\Dest\Bar\Folder

Splice

Splice reconnects Outer Junctions/Symlink directories in the destination to their original targets.

Splice outer Junctions

In the above example Folder1 is copied to Destination/Folder1, and Outer Junction/Symlink is available in the destination as junction, which points to the original location Outer Junction/Symlink Target.

The objective behind splicing Outer Junctions/Symlink Directories to its original location is to get a copy during smartcopy, but to reuse Outer Junctions/Symlink Directories source locations.

The command line invocation below splices all junctions/symbolic link directories below "c:\Folder 1"

ln --splice --copy "c:\Folder 1" "c:\Destination\Folder 1"

In certain situations only certain junctions/symbolic link directories should be spliced during SmartCopy, DeloreanCopy, SmartMirror or HardlinkClone. To accomplish this ln.exe can be run with wildcard expressions to specify certain directories for splicing.

ln --splice "MyJunc*" --copy "c:\Folder 1" "c:\Destination\Folder 1"

To specify junctions/symbolic link directories for splicing more fine grained regular expressions can be given:

ln --spliceregexp "[a-z]*" --copy "c:\Folder 1" "c:\Destination\Folder 1"

The regular expressions used must conform to POSIX 1003.2 extended RE (ERE) syntax.


The Splice functionality is also useful, when mountpoints should be copied. Mountpoints are very similar to Junctions, but point to a path always starting with \\?\VolumeGuid.

Copy Mountpoint with Splice

To specify a certain mountpoint to be copied specify it as argument to --splice:

ln --splice "Mountpoint" --copy "x:\Folder 1" "x:\Destination\Folder 1"

In the above example Folder1 is copied to Destination/Folder1 and the Mountpoint is created in the destination as Mountpoint again. It points to the original location under \\?\Volume{b4..bc}.

Or drive complexity up one level and just use splice for mountpoints with wildcards, but unroll the rest.

ln --unroll --splice "Mountpo*" --copy "x:\Folder 1" "x:\Destination\Folder 1"



Multiple Source

Multpiple source locations can be specified for SmartCopy, Clone and Delorean Copy. If there are junctions/symlinks between these source locations, they are handled as inner junctions/symlinks, because all source locations are dealt with as a common root.


Multiple Sources

In the above example Location 1 and Location 2 are copied to Destination. Location2/Junction_Symlink is treated as inner junctions to Location1/Folder 10 in the source, and that's why Destination/Junction_Symlink points to Destination/Folder 10 in the Destination.

The objective behind this is to treat all junctions/symlinks as inner junctions/symlinks as long as they are in the set of source folders.


The command line invocation below copies from "c:\Location 1" and "c:\Location 2" to c:\Destination :

ln --source "c:\Location2" --copy "c:\Location1" "c:\Destination"


There can be more than one additional --source command-line switches:

ln --source "c:\Location 2" --source "d:\Location 5" --copy "c:\Location 1" c:\Destination


Specifying additional source paths can be combined with --splice or --unroll:

ln --unroll --source "c:\Location 2" --copy "c:\Location 1" c:\Destination


The above command copies all content from "c:\Location 2" and "c:\Location 1" to c:\Destination, but does not create a "c:\Destination\Location 2" and a "c:\Destination\Location 1". To even acchive this the --destination option must be used:

ln --unroll --source "c:\Location 2" --destination "c:\Destination\Location 2" --copy "c:\Location 1" "c:\Destination\Location 1"


The --destination option must alsways be used with a corresponding --source option building pairs. The order of --source --destination is also important. For --copy the normal destination builds the pair:

ln --source SrcA --destination DstA --source SrcB --destination DstB --source SrcC --destination DstC --copy SrcC DstC



--deeppathdelete
delete very
long path

Since ln.exe can handle Deep Paths which are longer than 256 characters, there is also the need to delete such pathes.


ln --deeppathdelete x:\DeepPath

Recursively delete all files and directories under x:\DeepPath


--delorean
DeLorean Copy

DeLorean Copy is a way of creating incremental backups by using a combination of Hardlink Clone and SmartMirror.

The following picture gives an overview of what DeLorean Copy is about

DeLorean Copy Behaviour

In general a DeLorean Copy has 3 principals: Source(t), InitialBackup and Backup(n), where t represents time and n the number of backups.

Phase 1: Initial SmartCopy

The folder Source(t) is initially copied to InitialBackup. This is shown by the blue arrow.


Changes
happen

During this phase the files under source change, and Source(t) becomes Source(t+1).


Phase 2: Hardlink Clone

The folder InitialBackup is Hardlink Cloned to Backup1, which ties InitialBackup and Backup1. This is shown by the red arrow.


Phase 3: Mirror
Mirror the folder Source to Backup1. This is shown by the green arrow:
  • Keep unchanged files as hardllinks to InitialBackup.
  • Delete files longer in Source(t+1) from Backup1.
  • Copy different files from Source(t+1) to Backup1.

With completion of this first round Backup1 contains the first lean and mean copy of Source only consisting of either hardlinks to InitialBackup, or of copied files if there was the need to copy them over from Source(t+1) because they were newer under Source(t+1).

The point is that all files in Backup1 are transparently accessible, but little space is used, because not all files in the Source(t+1) changed, so that there was only the need to effectively copy over a few files from Source(t+1) to Backup1.

The time stamps for LastWriteTime, CreationTime and LastAccessTime of files/folders/junctions/symbolic links are also restored at the corresponding destination locations during DeloreanCopy.

Different items means: Either the file size has changed, or the LastWriteTime has changed.


This can be repeated on and on. The second round would be using Source, Backup1 and Backup2 for DeLorean Copy:

Changes
happen

During this phase the files under source change, and Source(t+1) becomes Source(t+2).


Phase 2: Hardlink Clone

The folder Backup1 is Hardlink Cloned to Backup2, which ties Backup1 and Backup2. This is shown by the red arrow.


Phase 3: Mirror
Mirror the folder Source(t+2) to Backup2. This is shown by the green arrow:
  • Keep unchanged files as hardllinks to Backup1.
  • Delete files longer in Source(t+2) from Backup2.
  • Copy different files from Source(t+2) to Backup2.

A real word example looks like this:

ln --copy x:\dir\Source x:\dir\InitialBackup

This creates the initial Backup as shown above in Phase 1


ln --delorean x:\dir\Source x:\dir\InitialBackup x:\dir\Backup1

This command does the first round of DeLorean copy as shown above in Phase 2 and Phase 3 creating a backup.

ln --delorean x:\dir\Source x:\dir\Backup1 x:\dir\Backup2

This command creates another set of incremental DeLorean copy as shown above in Phase 2 and Phase 3 creating another backup.

See also the note on quotes.

Output

The output generated by DeLorean Copy/Smart Copy/Smart Clone/Smart Mirror looks like e.g.:


-f c:\backup\test\deleteme.dat
=d c:\backup\test\directorygone
+f c:\data\test\1.dat
+f c:\data\foo\3.dat
*h c:\data\foo\Hardlink of 3.dat
*h c:\data\foo\Another Hardlink of 3.dat
.h c:\data\already\x.dat
*h c:\data\already\Hardlink of x.dat
~f c:\data\failed\DidntMatch--splice.txt
!?f (0x00000002) c:\data\failed\AccessDenied.txt


Delorean Copy logs each action it did, and prefixes two characters to each item which it processed for each line of output. The first column of the output contains the Operation which was performed, and the second column specifies the Type of item which was processed.

Operation Description
= Item is already present in the target. Used during Smart Mirror
+ Copy/Create an item. Used during Smart Copy
* Hardlink a file
. Item is already present in the target and is used as the source of a newly created hardlink.
- Remove an item from the target that is not present in the source. Used during Smart Mirror
? Enumerate an item.
~ Item has been excluded by command line arguments.
# Item has been cropped due to being an outer junction/symlink.
& Junction or Symbolic Link has been re-created but turned out to be dangling in the destination.
\ Opening a file.
/ Map file into address space.
! An error happened.
' Informational Message.


Item Description
f A File is processed.
h A Hardlink is processed.
s A Symbolic link file or Symbolic Link Directory.
j A Junction is processed.
m A Mountpoint is processed.
d A Directory is processed.
t An Alternative Stream is processed.
e An E/A Record is processed.
p A Sparse file is processed.


Sample Description
+f c:\data\myfile.txt

A normal file is copied.

*h c:\data\hardlnk.txt

c:\data\hardlnk.txt is hardlinked to c:\data\myfile.txt.

+s d:\dest\symlink.txt

A Symbolic link has been created.

+j d:\dest\junc01

A Junction has been created.

+d d:\dest\mydir

A Directory has been created.

-f d:\dest\delfile

The file d:\dest\delfile has been removed during e.g. Smart Mirror.

-j d:\dest\junc02

The Junction d:\dest\junc02 has been removed.

+m d:\dest\mountpoint

The mountpoint d:\dest\mountpoint has been created during e.g. Smart Copy.

=d d:\dest\mydir

The directory d:\dest\mydir is the same as in the source.

=s d:\dest\symlink

The symlink d:\dest\symlink is the same as in the source.

.h d:\dest\file.txt

means that the name of this file is only printed, because some hardlinks have been newly tied to it. The file itself is already in the backup available, but the files following it are prefixed with * and are the newly tied hardlinks.

*h d:\dest\new_hlink

If the line above is prefixed with . it means that d:\dest\new_hlink is tied to e.g. the already existing file d:\dest\file.txt.

~d d:\source\mydir

The directory d:\source\mydir has been excluded intentionally by e.g. --exclude.

~f d:\source\aFile

The file d:\source\aFile has been excluded intentionally by e.g. --exclude. Items are also listed with ~ if the --unroll option is not used and outer Junctions or Symbolic Links are cropped.

#d d:\source\outerreparse

The junction/symlink d:\source\outerreparse has been excluded because it was an outer junction/symlink.

&d d:\dir\danglingreparse

The Junction or Symbolic Link d:\dir\danglingreparse has been restored, but it turned out that it is dangling in the destination.

!-d (0x00000002) d:\source\aFile

The ! operator indicates that there was an error. It is followed by the operation, the item which was processed and an Win32 error code.

!?d (0x00000005) d:\src\deny

The ? operator means the current operation was unable to enumerate all available files. It is always prefixed with ! to signal that during enumeration of items something went wrong.

!?j (0x00000423) d:\src\deny

Circularities are detected during Smart Copy/Smart Mirror/Delorean Copy and are flagged as an error with the code 0x0423, which means ERROR_CIRCULAR_DEPENDENCY.

!?s (0x00000780) d:\src\deny

The error ERROR_CANT_ACCESS_FILE aka 0x0780 is thrown only under Windows XP if a symbolic link is among the files to operate on, because Windows XP can not process Symbolic Links unless the symbolic link driver for Windows XP is installed.

!&j (0x000000a1) d:\src\dangling_junction

The error ERROR_BAD_PATHNAME aka 0x00a1 is thrown only under Windows XP if a dangling junction can not be restored.

!+f (0x000003e7) d:\v\v.tc

The ERROR_SWAPERROR means, that errors occurred during reading of a file. e.g. caused by unplugging the network cable.

!*h (0x00000585) d:\

The error INVALID_INDEX aka 0x0585 means that the NTFS implementation of this drive is broken. It returns the same file-index for files with different file size.
This is only a warning and ln.exe continues but ignores the retrieved file-indices.
See also --dupemerge

!+e (0x000000005) d:\test\A

The error ERROR_ACCESS_DENIED with !+e means EA Records could not be copied. This happens if a filesystem does not support EA Records, as it is with some NAS implementations.

!+t (0x00000007b) d:\test\A:ADS1

The error ERROR_INVALID_NAME aka 0x007b with !+t means an Alternative Data Stream of a file could not be copied. This happens if a filesystem does not support Alternative Data Streams, as it is with some NAS implementations.

!+s (0x00000013d) t:\f3\f3_j0

The error ERROR_MR_MID_NOT_FOUND aka 0x007b with !+s means that the driver for symbolic links under Windows XP was not able to resolve a symbolic link on a mapped network drive.
This is a technical limiation, which can't be overcome as the author of the driver Masatoshi Kimura explained, because to resolve a symbolic link via network one has to send the symbolic link target to the server, which hosts the network connection. Since SMBv1, which is available on WindowsXP is not able to transport this string, it is impossible to resolve a symbolic link on network drives.
a problem. It stops in resolving with the error code STATUS_STOPPED_ON_SYMLINK. I stumbled accross that message when ln.exe tried to follow symbolic links on mapped network drives. Since the symbolic link driver is not maintained anymore by his author... Sorry!

'Entering on the fly dupemerge ln.exe was forced into on the fly dupemerge mode.


Statistics

After DeLorean Copy is finished it displays a statistics like this:

  Total Copied Linked Skipped Removed Excluded Failed
Folder: 10 1 - 6 1 3 0
File: 1050 1026 4 0 3 0 20
Junction: 7 - 2 0 2 3 0
Symlink: 8 - 2 5 1 0 1
Byte: 458200 272600 185600 0 1392 0 0
 
  Overall Clone Mirror
Times: 00:00:02.621 00:00:02.559 00:00:00.062

The statistics of DeLorean Copy only counts the operations which are performed during the Mirror phase between Source and Backup(n+1). The operations which are performed during cloning between Backup(n) and Backup(n+1) are not counted in the final DeLorean Copy statistics.

Total: The total number of items as they were found under the Source. Total is the sum of all other columns but the Removed column.

Copied:

  • Items prefixed with + in the verbose output and which were were copied from Source to Backup
  • Items prefixed with & in the verbose output and which have turned out to be dangling Junctions or Symbolic Links in the Backup

Linked:

  • Items prefixed with * in the verbose output which were either hardlinks or
  • Items prefixed with + in the verbose output which were symbolic links

Skipped:

  • Items prefixed with = in the verbose output, which were skipped during copy, because they were already there

Removed:

  • Items prefixed with - in the verbose output, and which were removed from Backup because they are not in the Source any more.
Excluded:
  • Items prefixed with ~ in the verbose output and which were excluded from Backup, because they were excluded from the Source via --exclude.
  • Items prefixed with # in the verbose output, because --splice/--unroll was specified with a regular expression, but the regexp didn't match.
  • Items prefixed with # in the verbose output, because no --unroll or --splice was given at all, thus it was cropping.

Failed:

  • The number of items which failed during creation.
  • The number of items which failed during cleaning while doing SmartMirror and while doing Delorean Copy.

Overall: Time used up for the whole DeLorean Copy including the Clone from Backup(n) to Backup(n+1) and the Mirror from Source to Backup(n+1)

Clone: Time used up for the Clone from Backup(n) to Backup(n+1)

Mirror: Time used up for the Mirror from Source to Backup(n+1)


Limitations

It is little known, but NTFS has a limit to create a maximum of 1023 hardlinks to one file. For DeLorean Copy this means that it will display an error message if this limit is exceeded, because exceeding this limit means loss of data among the most recent backup sets:


Delorean Copy FAILED! The NTFS link limit of 1023 has been exceeded for:

The reason for exceeding this limit could either be, that there have been more than 1023 backup sets but no hardlinks within the Source, or there are hardlinks within the Source and less than 1023 backup sets.

The filters of --includeXXX or --excludeXXX are applied in the source and the destination location


Shell Integration

DeLorean Copy functionality is also available via Link Shell Extension


--dupemerge
On the fly Dupemerge

Smart Copy, Smart Mirror and SmartClone rely on finding the hardlink relations by enumerating the unique file-index attribute of each file. But there might be more equal files below a given path, some of them probably not hardlinked. When issuing the --dupemerge option ln.exe does not only rely on the file-indices, but also creates incremental hash sums on all files, so that it finds additional not yet hardlinked dupes and hardlinks them in the destination:


Smartcopy with On the fly Dupemerge

e.g.:
ln --dupemerge --copy Folder1 Destination\Folder1

In the above example the directory Folder1 is searched for dupes and copied/hardlinked to destination. The files A, C, D are already hardlinked in the source, but B is not, even if it is the same as A,C,D. Similar situation with E,F, which are already hardlinked, but G is the same as E,F but not hardlinked. In the destination all equal files are hardlinked: There is a dupel B,C,D and a dupel E,F,G. A was omitted as in normal Smartcopy, because it is out of the source scope.


Broken NTFS implementations

The --dupemerge option is also of help, when data from a drive should be copied which has a broken NTFS implementation.
NTFS drives which are accessed via SMB or CIFS sometimes show the same file-index attribute for files of different size. Well that's impossible, because the file-index attribute can only be the same for hardlinks, and hardlinks, since they are pointing to the same data, are the same files, with for sure the same size.

So the SMB or CIFS implementation of such drives is broken, and this means one can not detect already existing hardlinks by reading all file-indices from all files and searching for equal indices. ln.exe has a built in sanity check on file-indices, and once it finds files with same file-index but different size, it assumes the view on this NTFS drive is broken, and invalidates all file-indices, thus assuming no hardlinks are on this drive.

In such a situation --dupemerge is of great help because it scans the given path and calculates incremental MD5 hash sums for all files. This enables ln.exe to detect dupes, and hardlink equal files in the destination.

On the fly Dupemerge can take a while and ln.exe prints out a warning, that it entered the On the Fly Dupemerge mode.
The Dupemerge functionality can also be downloaded as a standalone tool for in-place dupemerge.

See also the note on quotes.


--exclude
Exclude files via wildcards

In certain situations not all files below a path should be used in Smart Copy, Smart Mirror, Delorean Copy or Clone. To accomplish this ln.exe can be run with wildcards specified to exclude certain files.


ln --exclude *.ocx --exclude *.dll --delorean c:\source d:\b\dest d:\b\bk1

Basically any arbitrary wildcard expressions can be used, because the wildcard expressions are translated into a regular expression. This means that e.g *file*.ext*.* is also a valid wildcard expression for --exclude.


Sometimes it is necessary to just exclude certain files along a given path:

ln --exclude "*\\Res\\*\\Logfile.t*" --copy c:\source d:\b\dest

The above example excludes all files Logfile.* from all directories which have \Res\ in it.



--excludedir
Exclude directories via wildcards

In certain situations not all directories should be used in Smart Copy, Smart Mirror, Delorean Copy or Clone. To accomplish this ln.exe can be run with wildcards to exclude certain directories.


ln --excludedir *test* --delorean c:\source d:\b\dest d:\b\bk1

Basically any arbitrary wildcard expressions can be used, because the wildcard expressions are translated into a regular expression. This means that e.g *file*.ext*.* is also a valid wildcard expression for --excludedir.

The wildcard expression specified under --excludedir is applied to the whole path, which means that e.g


ln --excludedir "fotos\\temp" --copy c:\source d:\b\dest

will exclude all directories containing 'fotos\temp' and their subdirectories. The above example will e.g. exclude 'fotos\tempur\myfotos', 'fotos\temp\myfotos' or 'fotos\tempomat\myfotos'. Please note sure that '\' has to be escaped via '\\'



--excluderegexp
Exclude files via regular expressions

In certain situations not all files below a path should be used in Smart Copy, Smart Mirror, Delorean Copy or Clone. To accomplish this ln.exe can be run with regular expressions specified to exclude certain files.


ln --excluderegexp "[a-z]*" --copy c:\source d:\b\dest

The regular expressions used must conform to POSIX 1003.2 extended RE (ERE) syntax.

The regular expression specified under --excluderegexp is applied to the whole path with the filename.



--excluderegexpdir
Exclude directories via regular expressions

In certain situations not all directories should be used with Smart Copy, Smart Mirror, Delorean Copy or Clone. To accomplish this ln.exe can be run with regular expressions to exclude certain directories.


ln --excluderegexpdir "[a-z]*" --delorean c:\source d:\b\dest d:\b\bk1

The regular expressions used must conform to POSIX 1003.2 extended RE (ERE) syntax.

The regular expression specified under --excluderegexpdir is applied to the whole path with the filename.



--enum
Enum hardlinks

Sometimes it is convenient to find out which files are linked together, because hardlinked files might be spread around a NTFS volume. This is somehow the inverse hardlink functionality and it can be invoked as follows.


ln --enum x:\path

In the above example files below path are searched, and the files which are hardlinked are printed out.

The --enum option first prints so called saturated and afterwards unsaturated hardlinks. For saturated hardlinks the number of files found below path matches the hardlink reference count of these files, but unsaturated hardlinks have a reference count greater than the number found below path.



--include
Include files via wildcards

To selectively run Smart Copy, Smart Mirror, Delorean Copy or Clone on a certain set of files, the --include option can be used with wildcards.


ln --include *.ocx --include *.dll --delorean c:\source d:\b\dest d:\b\bk1

Basically any arbitrary wildcard expressions can be used, because the wildcard expressions are translated into a regular expression. This means that e.g *file*.ext*.* is also a valid wildcard expression for --include.

Sometimes it is necessary to just include certain files along a given path:

ln --include "*\\Res\\*\\Logfile.t*" --copy c:\source d:\b\dest

The above example includes all files Logfile.* from all directories which have \Res\ in it.



--includedir
Include directories via wildcards

To selectivley run Smart Copy, Smart Mirror, Delorean Copy or Clone on a certain set of directories, the --includedir option can be used with wildcards.


ln --includedir *test --delorean c:\source d:\b\dest d:\b\bk1

Basically any arbitrary wildcard expressions can be used, because the wildcard expressions are translated into a regular expression. This means that e.g *src\\sub?older* is also a valid wildcard expression for --includedir.

The wildcard expression specified under --includedir is applied to the whole path, which means that e.g


ln --includedir "*fotos\\temp*" --copy c:\source d:\b\dest

will include all directories containing 'fotos\temp' and their subdirectories. The above example will e.g. include 'fotos\tempur\myfotos', 'fotos\temp\myfotos' or 'fotos\tempomat\myfotos'. Please note that '\' has to be escaped via '\\'



--includeregexp
Include files
via regular expressions

To selectively run Smart Copy, Smart Mirror, Delorean Copy or Clone on a certain set of files, the --includeregexp option can be used with regular expressions specified to include certain files.


ln --includeregexp "[a-z]*" --copy c:\source d:\b\dest

The regular expressions used must conform to POSIX 1003.2 extended RE (ERE) syntax.

The regular expression specified under --includeregexp is applied to the whole path with the filename.



--includeregexpdir
Include directories
via regular expressions

To selectively run Smart Copy, Smart Mirror, Delorean Copy or Clone on a certain set of directories, the --includeregexpdir option can be used with regular expressions to include certain directories.


ln --includeregexpdir "[a-z]*" --delorean c:\source d:\b\dest d:\b\bk1

The regular expressions used must conform to POSIX 1003.2 extended RE (ERE) syntax.

The regular expression specified under --includeregexpdir is applied to the whole path with the filename.



--junction
Create Junctions

ln.exe can also create junctions, and print the target of junctions


ln --junction JunctionName TargetDirectory

In the above example a junction JunctionName is created, which points to TargetDirectory


ln --junction JunctionName

In the above example the junction JunctionName is searched for its Target and the target is printed out.


rd JunctionName

In the above example the junction JunctionName is removed. This is accomplished with an simple built in command prompt command.

ln --junction JunctionName \\?\Volume{f6c8b244-1ce4-11e0-a607-005056c00008}\tmp

In the above example the junction JunctionName is created to a volume-guid target. This is useful if you are either out of drive letters, or just want to have a link to a directory on a mounted drive regardless of which drive-letter it is mounted on. To find out the available volume-guids use the mountvol.exe command.

Pushing this idea further you can now mount whole volumes onto arbitrary locations in your filesystem, even if the volume is not mounted to a drive letter at all:
  • Format the disk in 'Disk Manager', but do not mount at all in 'Disk Manager'
  • Use mountvol.exe to see which volumes are available.
  • Mount the volume:
    ln --junction Mountpoint \\?\Volume{f6c8b244-1ce4-11e0-a607-005056c00008}\
  • Use 'rmdir Mountpoint' to unmount.

In the early days of ln.exe the arguments of --junction were unfortunately defined in the opposite order as for --symbolic. So --junction has the target as first argument and --symbolic has it as second. So far so bad. To fix this and avoid breaking compatibility (and keep my regression test simple;-) for --junction an additional option --noitcnuj was introduced, which creates junctions and has the target as its second argument:

ln --noitcnuj \\?\Volume{f6c8b244-1ce4-11e0-a607-005056c00008}\tmp JunctionName

So everything which can be done with --junction can be done with --noitcnuj, but reversed argument order. Why 'noitcnuj'? Read noitcnuj backwards ... ;-)



--keepsymlink
relation

Save absolute/
relative status

Symbolic links can be created either absolute or relative. Setting this modifier creates symbolic links in the destination in the same way as the were in the source.


ln --keepsymlinkrelation --copy x:\source x:\destination

This modifier can be applied to all operation like --copy, --mirror, or --delorean. It is enabled by default if ln.exe is used with the --backup option.



--list
List Hardlink
siblings

With Windows7 and above finding out the siblings of a hardlink is very simple and needs no time-consuming operations


ln --list filename

In the above example the hardlink siblings for filename are printed out.

With XP, W2K or W2K3 finding the hardlink siblings now also works, but it takes more time, because in the worst case the whole logical volume must be searched.



--mirror
Smart Mirror
Smart Mirror is very similar to Smart Copy and not only copies but synchronises the folder Source to Destination:
  • Smart Copy only different items from Source to Destination.
  • Delete items no longer in Source from Destination.

Smart Mirror takes care of Hardlink Relations, restores Inner Junctions or Inner Symbolic links or when issued unrolls or splices Outer Junctions or Outer Symbolic Links. The time stamps for LastWriteTime, CreationTime and LastAccessTime of files/folders/junctions/symbolic links are also restored at the corresponding destination locations during SmartMirror.

Different Items means: Either the file size changed, or the LastWriteTime changed.

e.g.:

ln --mirror x:\source x:\destination

In the above example the directory x:\source is mirrored to x:\destination, and all symbolic links and junctions within x:\destination are updated with respect to their new locations under x:\destination.

The filters of --includeXXX or --excludeXXX are applied in the source and the destination location

See also the note on quotes.


Shell Integration
Smart Mirror functionality is also available via Link Shell Extension


--mirror
--symbolic

Symbolic Clone
Mirror
The Clone Mirror function is a combination of Smart Mirror and SmartClone. Basically it creates a Symbolic Link Clone from Source to Destination directory, but also synchronises the folder Source to Destination:
  • Create a symbolic link Clone from Source to Destination.
  • Delete symbolic links in Destination which are no longer in Source

Clone Mirror takes care of Inner Junctions, Inner Symbolic links or when issued unrolls or splices Outer Junctions or Outer Symbolic Links.
Different Items means: The LastWriteTime has changed.

e.g.:
ln --symbolic --mirror x:\source x:\destination

In the above example the directory x:\source is cloned to x:\destination, and all symbolic links and junctions within x:\destination are updated with respect to their new locations under x:\destination. Furthermore symbolic links, which are in the Destination, but have no longer a target in the Source, are deleted from the Destination.

See also the note on quotes.


--mirror
--recursive

Hardlink Clone
Mirror
The Hardlink Clone Mirror function is a combination of Smart Mirror and SmartClone. Basically it creates a Hardlink Clone from Source to Destination directory, but also synchronises the folder Source to Destination:
  • Create a Hardlink Clone from Source to Destination.
  • Delete Hardlinks in Destination which are no longer in Source

Hardlink Clone Mirror takes care of Inner Junctions, Inner Symbolic links or when issued unrolls or splices Outer Junctions or Outer Symbolic Links.
Different Items means: The LastWriteTime has changed.

e.g.:
ln --recursive --mirror x:\source x:\destination

In the above example the directory x:\source is cloned to x:\destination, and all symbolic links and junctions within x:\destination are updated with respect to their new locations under x:\destination. Furthermore files, which are in the Destination, but are no longer in the Source, are deleted from the Destination.

See also the note on quotes.


--move
Smart Move

Smart Move enables folders with junctions and symbolic links beneath to be renamed, and the junctions and symbolic links' targets are updated below that folder. Without Smart Move renaming of such folders would end in dangling junctions and symbolic links.

With junctions or symbolic link directories it behaves as follows:

Smart Move behaviour for junctions and symbolic links

By closely looking at the above picture one can find three different types of folders/junctions:

Normal Folders

The folder 'Folder 3' is a normal folder. It gets moved with its content straight forward.


Inner Junctions
Symlinks

The folder 'Inner Junction/Symlink' is targeted at 'Inner Junction/Symlink Target'. In LSE terms this kind of folder is called Inner Junction/Symlink, because its target points to a folder, which is below the common anchor 'Folder 1'.

Inner Junctions/Symlinks are updated properly via Smart Move in the destination location.


Outer Junctions
Symlinks

The folder 'Outer Junction/Symlink' is targeted at the folder 'Outer Junction/Symlink Target'. In LSE terms this kind of folder is called Outer Junction/Symlink, because its target points to a folder, which is in parallel and thus outside the anchor 'Folder 1'.

Outer Junction/Symlink are not touched by Smart Move and thus stay connected to their respective target. Please note that this is different to Smart Copy, which has 3 different ways to deal with Outer Junctions/Symlinks.


e.g.:

ln --move x:\dir\dir2 x:\dir\newdir

In the above example the directory x:\dir\dir2 is moved to x:\dir\newdir, and all symbolic links and junctions within x:\dir\dir2 are updated with respect to their new locations under x:\dir\newdir.

See also the note on quotes.

Shell Integration
Smart Move functionality is also available via Link Shell Extension


--noads
Skip ADS

The --noads option tells ln.exe not to copy Alternative Data Streams (ADS) of files and directories. This option can be useful if the destination supports NTFS, but can not deal with ADS, which happens on certain NAS drives.


ln --noads --copy src dest

In the above example the files are copied normally from src to dest except for ADS.



--noea
Skip EA Records

The --noea option tells ln.exe not to copy EA Records of files and directories. This option can be useful if the destination supports NTFS, but can not deal with EA Records, which happens on certain NAS drives.


ln --noea --copy src dest

In the above example the files are copied normally from src to dest except for EA Records.



--output
Redirect
Output

The output of ln.exe can be redirected to files via this commandline option


ln --output filename --copy src dest

In the above example all output is redirected to filename



--probefs

This option of ln.exe returns the type of the filesystem. e.g NTFS, FAT, ReFs ...

ln --probefs x:\
ln --probefs \\server\share\path


--quiet
specify
log level

The amount of output during various operations can be controlled via the --quiet switch:


ln --quiet ...

The --quiet option without arguments totally disables any output during ln.exe' operations.
To specify the amount of output an optional argument can be passed to the --quiet switch:


0 Completely suppress output. This is the if same as no argument would have been passed to --quiet.
1 Only show severe errors.
2 Show the changes, e.g if a file is copied additionally during delorean copy. Furthermore show severe errors.
3 Show all output available. This is same as --quiet has been ommitted.

ln --quiet 2 ...

In the above example only changes are shown.



--recursive
Smart Clone

ln.exe can recursively hardlink all files found below a given directory location to a newly created directory structure under a different location.

In other words this means, the given directory structure is rebuilt under a different location, and all files from the given directory structure are hardlinked to the newly generated structure. Something like a xcopy, but the files are hardlinked and not copied:


Hardlink Clones how to

e.g.:

ln --recursive x:\dir\dir2 x:\dir\newdir

In the above example the directories below x:\dir\dir2 are rebuilt below x:\dir\newdir, and all files from x:\dir\dir2 are hardlinked to their corresponding locations under x:\dir\newdir.

A folder tree might also contain Junctions or Symbolic Links. The clone process recreates inner junction/symbolic links at the destination location very similar to the way SmartCopy does.


Hardlink Clones the smart way

Outer Junctions/Symbolic links are recreated with respect to the specified Outer Junction/Symbolic Link handling. e.g.

ln --unroll --recursive x:\dir\dir2 x:\dir\newdir

In the above example the directories below x:\dir\dir2 are rebuilt below x:\dir\newdir, all files from x:\dir\dir2 are hardlinked to their corresponding locations under x:\dir\newdir and all junctions/symbolic links are unrolled.


With Windows Vista and Windows7/8 this cloning process is also available with Symbolic Links instead of Hardlinks.

Symbolic Link Clones the smart way

Output See Output among Delorean Copy for a detailed description of HardlinkClone's Output

Shell Integration
The recursive support or HardlinkClone functionality is also available via Link Shell Extension



--symbolic
Symbolic Links

When used under Vista or Windows7/8 ln.exe can create symbolic links

ln --symbolic Sourcefile.txt SymbolicLinkToSourceFile.txt

In the above example a symbolic link SymbolicLinkToSourceFile.txt is created which points to sourcefile.txt.

ln --symbolic SourceDirectory SymbolicLinkToSourceDirectory

In the above example a symbolic link SymbolicLinkToSourceDirectory is created which points to SourceDirectory.

ln.exe can also only show the target of a symbolic link

ln --symbolic SymbolicLinkToSourceFile.txt

ln.exe can also symbolically link to volume-guid targets:

ln --symbolic \\?\Volume{f6c8b244-1ce4-11e0-a607-005056c00008}\tmp SymbolicLinkName

In the above example the symbolic link SymbolicLinkName is created to a volume-guid target. This is useful if you are either out of drive letters, or just want to have a link to a directory on a mounted drive regardless with of which drive-letter it is mounted on. To find out the available volume-guids use the mountvol.exe command.


Please note that symbolic links can only be successfully created from an administrative command prompt. ln.exe will fail on symbolic links from a normal command prompt!

--skipfiles

This option is a modifier to Smart Copy, Smart Mirror, Delorean Copy, or Clone, so that only directories symlinks and junctions are processed. The files are ommitted with this modifier used.

ln --skipfiles --copy source destination


--smartrename
Smart Rename

Smart Rename is very similar to Smart Move, but only renames junctions/symbolic links references below a given folder.


ln --smartrename x:\dir\dir2 x:\dir\newdir

In the above example the symbolic links/junction references below x:\dir\dir2 are renamed to x:\dir\newdir. This leaves x:\dir\dir2 with dangling symbolic links/junction, but this feature is useful if someone wants to migrate a whole disk to another physical media.


ln --copy x:\ y:\
unmount disk from x:\
mount disk under y:\
ln --smartrename y:\ x:\

unmount disk from y:\
mount disk under x:\

See also the note on quotes


--timetolerance

During --mirror and --delorean it is sometimes useful to not have an exact timestamp comparison between source and destination, but kind of a fuzzy comparison, because the systemtime of NAS drives is not exactly synced with the host.

To overcome this ln.exe has a --timetolerance switch to specify a value in milliseconds

ln --timetolerance 2000 --mirror source destination


--traditional
Traditional
mode

Some NAS boxes only support a very outdated version of the SMB protocol. SMB is used when network drives are connected. This old version of SMB in certain situations does not support the fast enumeration methods of ln.exe, which causes ln.exe to simply do nothing in SmartXXX functions.

To overcome this ln.exe has a --traditional switch, which forces ln.exe to enumerate files the old, but a little slower way.

ln --traditional --copy source destination ...


--trueSize
Size
calculation

Prints the true space used taking into account hardlinks. This means that a file with a size of e.g 12345kb and a reference count 5, will be listed with 61725kb by explorer, but indeed it only uses 12345kb, because it has a refcount of 5.

ln --truesize path

returns the following output

  Total Bytes
File: 14.012 6.871.558.902
Hardlink: 5 133.991
Total: 14.007 6.871.424.911
Folder: 2.294 -
Junction: 0 -
Symlink: 1 -
 
  Overall
Times: 00:00:06.171

The --truesize option does not return info on alternate data streams right now.



UNC Name
support

A local path can be a UNC name, which maybe reached from outside the computer via a share. Sometimes it is necessary to use UNC names to refer to local pathes.
ln resolves local UNC names to local path names and does its job on those resolved path names. e.g.:


ln \\local_computer\local_share\sourcefile.txt x:\dir_on_local_computer\destination.txt

In the above example there is a share called local_share on your local_computer which contains a file sourcefile.txt. ln resolves the UNC name to a local path if possible, and creates a link between the resolved location and destination.txt in x:\dir_on_local_computer.


ln --copy \\server\share\source \\server\share\destination

In the above example there is a share called share on your server which contains a directory source. ln.exe copies over the content from the source UNC path to the destination UNC path and preserves hardlink structures.


ln --delorean \\server\share\source \\server\share\destination \\server\share\backup

UNC Path are also supported for --delorean and preserves hardlink structures.


ln --recursive \\server\share\source \\server\share\destination

UNC Path are also supported for --recursive so that hardlink clones are created.



Quotes
around path

If there are blanks in the argument path to ln.exe, the pathname has to be put inside quotes, e.g.

ln --copy "x:\dir with blanks\dir2" "y:\destination with blanks\newdir"

Please make sure not to end a quoted path with \ because ln.exe treats this as escaping of the quote (\") and can not parse the command line.

ln --copy "x:\" y:\dir\newdir


Subst Handling

With the subst.exe command one can create driveletters, which point to a certain path on a NTFS volume. This means that two different driveletters in the end might resolve to locations on the same NTFS volume.

ln.exe checks these situations when it comes to allow the creation of hardlinks, and as a consequence allows the creation of hardlinks among different logical drives if they resolve to the same NTFS volume.


Snapshot Service

Shadow Copy (Volume Snapshot Service or Volume Shadow Copy Service or VSS), is a technology included in Microsoft Windows that allows taking manual or automatic backup copies or snapshots of data, even if it has a lock, on a specific volume at a specific point in time over regular intervals.

There are two ways to achieve this:

The way with a drive-letter

ln.exe can be used to create consistent copies from a snapshot: 4 files are needed: vshadow.exe, ln.exe, dosdev.exe, vss-exec.cmd. Place the files where you want, but vshadow.exe and dosdev.exe need to be in the path.

A backup of an entire e.g. E:\ or parts from it can be accomplished with this command:

vshadow.exe -script=vss-setvar.cmd -exec=vss-exec.cmd e:

The -script=vss-setvar.cmd is a command line argument to vshadow.exe that gives the shadow copy an identifier string so that a drive letter can be assigned to it - which allows ln.exe to access the files.

The -script=vss-exec.cmd is a command line argument to vshadow.exe that contains the name of a file, which is executed within vshadow.exe and does the very backup job:


REM
REM called from vshadow via commandline e.g
REM
REM vshadow.exe -script=vss-setvar.cmd -exec=vss-exec.cmd e:
REM

REM vss-setvar.cmd is generated by vshadow.exe
REM
call vss-setvar.cmd

REM assign a DOS device to the created snapshot
REM
dosdev x: %shadow_device_1%

REM run the intended copy job
REM
set LN=c:\tools\ln.exe
%LN% --copy x:\path e:\backup

REM finally remove the drive letter
REM
dosdev -r -d x:

The event process is this:

The vshadow.exe program creates a shadow copy, and then calls the on the fly generated vss-setvar.cmd that assigns a variable to the shadow copy. Then the given batch file vss-exec.cmd is executed within vshadow.exe. Dosdev.exe first assigns the drive letter x: to the shadow copy. ln.exe can be called now with whatever parameters are necessary at this time. By the end dosdev.exe removes the drive letter and vshadow.exe is allowed to exit.


The vshadow.exe generates an e.g vss-setvar.cmd:

@echo.
@echo [This script is generated by VSHADOW.EXE for the shadow set {95a6cd9f-bc45-43b8-8722-6c25ec06fe94}]
@echo.

SET SHADOW_SET_ID={95a6cd9f-bc45-43b8-8722-6c25ec06fe94}
SET SHADOW_ID_1={e38169c1-6aa7-4843-ac50-4c7b652397b3}
SET SHADOW_DEVICE_1=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1


There are different versions of vshadow.exe (32bit, 64bit, workstation, and server). You can download them from here in the same way as for dosdev


The way with a UNC share

If for some reason no drive letter is available, a similar strategy as above can be applied via an UNC share:


REM
REM General settings, please adjust to your needs!
REM
set SRC_DRIVE=e:
set SRC_PATH=tmp\src
set DST_PATH=x:\bla\foo

set VSHADOW=vshadow.exe
set LN=ln.exe

REM Create a temporary name for the share, e.g. take the timestamp
REM
for /f "delims=" %%a in ('%LN% --datetime') do set VSS_SHARE=%%a

REM call vshadow to create a persistent shadow
REM
%VSHADOW% -p -script=vss-setvar.cmd %SRC_DRIVE%

REM vss-setvar.cmd has been generated by vshadow.exe
REM
call vss-setvar.cmd

REM Create the temporary share, and specify a path relative to %SRC_DRIVE
REM via %SRC_PATH%. %VSS_SHARE% will then point to %SRC_DRIVE\%SRC_PATH%
REM
%VSHADOW% -er=%shadow_id_1%,"%VSS_SHARE%","%SRC_PATH%"

REM Run the intended copy job
REM
%LN% --copy \\localhost\%VSS_SHARE% %DST_PATH% > sortout

REM Remove the share
REM
%VSHADOW% -ds=%shadow_id_1%

The vss_unc.cmd script can be found here.

The vss_unc.cmd must be adapted with the proper SRC_DRIVE, SRC_PATH, DST_PATH in line 3 to 6, and issuing


  vss_unc.cmd

will do the job.


Deep Path Support

ln.exe can handle Deep Paths which are longer than 256 characters. This is usefully especially in the --copy, -delorean, --move, --mirror, --recursive mode. Deep Path Support is tested for the whole ln.exe functionality.


Support for
Volume{GUID} Path

All --copy, -delorean, --move, --mirror, --recursive commands can handle Volume{GUID} as path for operations:


ln --copy SrcDir \\?\Volume{97410ad7-54ec-11e3-97ab-005056c00008}\

The above example copies files onto a formatted disk which is not mounted. To find out which volumes are available in the system use mountvol.exe from a command prompt. Please make sure that the root of a volume has a trailing slash, e.g. \\?\Volume{GUID}\, but a normal path specified has not, e.g.: \\?\Volume{GUID}\MyPath\dir


Symbolic links for Windows XP

Technically it seems that NTFS5 even as shipped with WindowsXP has always supported symbolic links, but the functionality was not available for user mode applications, and even not for the upper layers of ntfs.sys

BUT: There are filter drivers available from Masatoshi Kimura for WindowsXP, which enable symbolic links under XP. These drivers can be downloaded from his home page for 64 bit and for 32 bit. The sources are also available.

Installation of these drivers is also quite simple.
  • Unpack the archive for your plattform,
  • Start the driver via executing 'senable.exe install' from the command prompt.

If you want to stop the driver, execute from a command prompt 'senable.exe delete'

ln.exe enables its symbolic link functions even when running WindowsXP, if it finds this driver installed.

But unfortunatley this driver has a few limitations, which in 99% of the use cases are not a problem, but for completeness:
  • Symbolic links, which have been created wiht the symbolic link driver installed, can not be copied/used wihtout the symbolic link driver.
  • Symbolic links can be created on mapped network drives, but such a symbolic link can't be resolved aka followed.

Useful .bat files

ln.exe can be used also from .bat files to accomplish various tasks:

Delete all hardlink siblings. This .bat file uses the --list option to enumerate all hardlinks and successively deletes them.
  Usage: DeleteAllHardlinks.bat Filename

Create timestamped DeLorean Copies as Link Shellextension does: 'data - yyyy-mm-dd hh-mm-ss', e.g 'MyData 2010-09-11 17-04-03'. Furthermore the number of backup-sets to be kept can be specified via the optional KeepEntries argument.
  Usage: DeloreanCopy.bat SourceDir BackupDir [KeepEntries]

Determines whether a path is a junction, symlink or directory and prints this on the command prompt. Courtesy of archimede0 at bluewin.ch .
  Usage: QueryPath.cmd Path

Tower of Hanoi backup rotation scheme using Delorean Copies.
Creates a number of backups using the Tower of Hanoi backup rotation scheme
  Usage: DeloreanHanoi.bat SourceDir BackupDir [BackupSets]


Return Values

As a typical command line utility ln.exe exits with return values, so that it can be conveniently used from .bat files:

 
0 Success. Everything is fine.

-1 The source directory given via command line parameters does not exist.
This happens in recursive mode if the first command line argument is not there

-2 The first command line argument in recursive mode specifies a directory, and it is not a directory, but some other kind of file.

-3 The second command line argument specifies a file, or in recursive mode a directory, which already exists.

-4 The destination directory could not be created in recursive mode.

-5 The source file aka first command line parameter does not exist in normal mode.

-6 The filesystem is not NTFS and does not support the creation of hardlinks.

-7 For some unknown reason the creation of a hardlink failed.

-8 Less arguments were given via command line.

-9 Ln was not able to enumerate the files in a directory. This might happen in recursive or in hardlink enumeration mode if the file permissions for a directory are insufficient.

-12 Ln.exe was not able to create a SmartCopy.

-14 Ln.exe was not able to create a clone.

-15 Ln.exe failed in --mirror.

-16 Ln.exe failed in --delorean.

Backgrounders

Hardlinks are a feature which are common to every Unix system, but are not directly available with NT4/W2K/WXP. It is a feature, which must be basically supported by the file system of the operating system.

So what are hardlinks in detail. It is common knowledge, that a file has a name and data associated to it. By opening the explorer, the NTFS filesystem of NT4/W2K/WXP can be browsed comfortably. There is a 1:1 relationship between the filename and the filedata, but this assumption does not hold for every filesystem.

Some filesystems, such as UFS, XFS, or NTFS have a N:1 relationship between filename and the filedata. So this means there is more than one directory entry for a file.

But how does one create those multiple entries? There is a command under Unix called ln, which creates many file entries for a file, so that there are many filenames or also called hardlinks for one filedata.

For each hardlink created, the filesystem increments a reference count stored with the filedata, so that the filedata stores how many filenames point to it. If a filename pointing to a hardlinked filedata gets deleted, the reference count stored with the filedata gets decremented by one. The filedata gets deleted when the reference count of the filedata decreases to zero.


Limitations

Ln.exe can only be used under the supported platforms
HardLinks can only be created on NTFS volumes, under the supported platforms.
HardLinks can only be created within one NTFS volumes, and can not span across NTFS volumes.


Frequently Asked Questions
  • Q: I'm trying to use ln for a selective backup of the "Users" folder, and I'm having trouble with the --includedir option, because nothing gets copied:

    ln --includedir "*\\My Documents" --copy C:\Users X:\pc01-office\Users

    A: The point is that explorer.exe fools you into believing, that directories like 'My Documents' exist, but they do not exist. Open a command prompt, change to c:\users and type 'dir'.
    The files and directories you see now are very much different, than you can see in explorer.

    So far so bad. ln.exe unfortunately only sees what the filesystem really has, and this is not e.g 'My Documents'. If you put a --includedir filter for '*My Documents*' it will not work, because in the NTFS namespace 'My Documents' is not there.

  • Q: I only changed the attributes of a file, even the timestamp and content are the same, but --delorean copies the file instead of linking to the old backupsets and just changing the attributes in the current backup set.

    A: The files in the backup sets of --delorean are hardlinked if they are the same. NTFS provides one set of timestamps and attributes for all hardlink-siblings of a file, so if a file needs different attributes in a backup, it has to be copied.

History
June 28, 2014 Version 2.8.5.2
  • With --backup Encrypted Files are copied too.
  • Junctions on mapped network drives are handled properly during all operations.
  • Added the --noads option.
  • Added the --noea option.
  • Errors during copying ADS are marked with the t charachter.
  • Errors during copying EA Records are marked with the e character.
  • Fixed various issues under XP when symlinks were used.
  • Resource and speed improvments in --backup mode.
  • Listing of hardlink siblings with --traditional didn't show all siblings, when symbolic links were involved.
  • The .html docu has no .html format errors.
  • Added the QueryPath .bat file to determine the type of an file-object.
  • Tackled with the some more symlink and Windows XP issues.
  • ln.exe now can handle mountpoints during SmartXXX operations.
  • Documented the --destination option.
  • WindowsXP: Symbolic links across drives didn't work.
  • WindowsXP: Symbolic links to Volume GUIDs didn't work.
  • WindowsXP: Symbolic links between very long path didn't work.
  • WindowsXP: Lots of tweaks here and there.
  • Tweaked DeloreanHanoi.bat, a way to use ln.exe for the Tower of Hanoi backup rotation scheme.
  • Files with a size beeing a multiple of 16777216 were copied properly, but generated an error message even if there was none.
  • Added Hardlink Clone Mirror option: --recursive combined with --mirror now mirrors hardlink link clones.

February 11, 2014 Version 2.8.1.0
  • Introduced the --backup switch which also copies ACLs.
  • Introduced the --keepsymlinkrelation switch.
  • The --output option produced logfiles with LF but not CR/LF.
  • The --copy, --mirror and --delorean also copies sparse files and alternative sparse streams.
  • Fixed a bug during copy, when a file was already there and the new file was smaller, it didn't copy over completely.

December 29, 2013 Version 2.7.5.9
  • The --list function didn't return error messages when used on UNC pathes.
  • Just --symbolic without argument caused a crash.
  • ln --copy x:\ x caused the folder x to be completely invisible, because GetFileAttributes() runs havoc on root dirs, e.g h:\.
  • The UNC Support has been improved and supports hardlink preservation for --copy, --delorean, --mirror.
  • Introduced the --probefs option.
  • The shadow copy .cmd file for vss shares via UNC name has been adapted.
  • Junctions can have now volume names like \\?\Volume{GUID}\tmp as target.
  • Symbolic Links can have now volume names like \\?\Volume{GUID}\tmp as target.
  • Updated the documentation, and structured it based on alphabetical order of command line switches.
  • Fixed a bug in --exclude, which prevented excluding files along a path.
  • Fixed a bug in --include, which prevented including files along a path.
  • A changed file attribute didn't cause the file to be treated as changed during all SmartXXX/Delorean functions.
  • Arguments to common operations support \\?\Volume{GUID} as arguments.
  • The LastWriteTime, CreationTime and LastAccessTime for files/folders/junctions/symlinks is restored during --mirror or --delorean.
  • Is able to delete very long path with the --deeppathdelete option.
  • Symbolic Link Clones always created absolute symlinks regardless of the command line args.

August 30, 2013 Version 2.7.5.3
  • Dead Junctions to a different drive could lead to not detecting hardlinks during all operations. Very Nasty, but no dataloss caused.
  • Introduced a sanity check for broken NTFS implementations.
  • Introduced On the Fly DupeMerge.

May 9th, 2013 Version 2.7.4.9
  • Memory Deallocation at the very end of ln.exe took ages when operating on millions of items.
  • During --mirror files in the destination with ReadOnly Attribute on were not mirrored.
  • --mirror didn't report errors at the end.
  • --timetolerance was introduced so that during --mirror and --delorean timestamp comparison is kind of fuzzy.
  • Fixed a problem with --mirror when a directory changed into a file or vice versa from one mirror to the next and had exactly the same name.
  • Compressed files are handled during SmartCopy/SmartMirror and DeloreanCopy.
  • Directory matches with --excludedir did not work properly with every expression.
  • Introduced the --include* options.
  • Added the --skipfiles option.
  • Sorted the options in the doku alphabetically.
  • Added kind of 'clonemirror' option: --symbolic combined with --mirror now mirrors symbolic link clones.
  • Extended the DeloreanCopy.bat with a 'number of keep-entry' argument. Thx to MathiasB for providing it.
  • The filters for --includeXXX or --excludeXXX are applied on source and destination during --mirror and --delorean.
  • Added the DeloreanHanoi.bat to create backup sets using the Tower of Hanoi backup rotation scheme.
  • Fixed a problem in UNC name resolution, when the UNC share was created by the Volume Shadow Service.
  • Added the vss_unc.cmd to create Volume Shadow Service backups via a UNC share.
  • The handling of the compression bit during copying/mirroring/deloreaning for files and directories was broken.

June 24th, 2012 Version 2.7.2.0
  • Fixed a problem with --truesize when used under Windows XP.

April 11th, 2012 Version 2.7.1.6
  • Introduced a traditional mode for SmartCopy, DeloreanCopy and SmartClone when used on network shared implementing SMBV10 (NAS boxes).
  • During SmartCopy, DeloreanCopy and SmartClone nested junctions/symbolic link directories were not processed correctly if alphabetical order was bad.
  • During SmartCopy, DeloreanCopy and SmartClone dangling junctions or symlinks will be at least in the same way dangling restored.
  • Multiple source locations can be specified with SmartCopy, DeloreanCopy and SmartClone.
  • SmartRename was introduced.
  • SmartCopy, SmartMirror and DeloreanCopy accept FAT drives as source directory.
  • Support for DeepPath operations in SmartCopy, DeloreanCopy and SmartClone, SmartMove.
  • Fixed a problem in SmartMove which created dangling symbolic links, when using relative symbolic links in certain situations.
  • Fixed the problem of dangling junctions/symbolic links caused in certain situations with the multiple source option.
  • Create DeloreanCopy.bat is now independent of region specific date time settings.
  • Support for symbolic links under WXP.
  • Can handle subst.exe created driveletters.
  • Added a --truesize option.
  • More detailed reporting when the restore of a dangling junction under windows XP fails.
  • Create Symbolic Link sometimes did not create absolute links when needed in certain situations.

May 15th 2011 Version 2.6.5.0
  • Speed improvements during SmartCopy/SmartMirror/HardlinkClone and Delorean Copy.
  • The log level can be specified more granularly.
  • Introduced new Heap Manager Rockall for x64 and x86 builds to gain performance.

April 17th 2011 Version 2.6.0.4
December 20th 2010 Version 2.5.5.5
  • Added more powerful options to exclude files/directories via wildcards/regular expressions.
  • Changed the regular expression machine to tre-0.8.0.
  • Delorean Copy speed improvements by about 35% since 2.500 (without virus scanner).
  • Introduced DeLorean Copy, which is a way of creating incremental copies using hardlinks.
  • Added a .bat file section, which contains useful .bat wrappers around ln.exe
  • Fixed a minor memory leak/crash.
  • When specifying a rootpath e.g.: x:\ as source with --copy and --delorean, junctions were to rebuild properly in the destination.

October 3rd 2010 Version 2.4.0.0
May 22nd 2010 Version 2.3.5.4
  • Introduced the --move option for Smart Move
  • The --copy option now also supports copying Symbolic Links under Vista and Windows7/8
  • Directories can now be symbolically linked
  • With W2K & NT4 readonly/hidden/system files could not be hardlinked
  • Hardlink Clone now restores the attributes of cloned folders.

March 9th 2009 Version 2.2.0.6 Enumerate hardlink siblings for Vista & Wxp, Symbolic link support vista
Under W2K it turned out, that CreateHardlink() from kernel32.dll with long pathnames (e.g. \\?\) was broken.

September 12th 2008 Version 1.997 Introduced the --copy option
Enabled the --symbolic option for Vista
Introduced the --list option under Vista
Added long (32k) path support

February 17th 2006 Version 1.810 Various small fixes related to commandline parsing.

February 18th 2005 Version 1.803 Fixed NT4 compatibility issues.

February 7th 2005 Version 1.800 Fixed a few minor internal flaw related to hardlink enumeration. Improved html documentation and added description for return values.

November 26th 2004 Version 1.711 decides between saturated and unsaturated hardlinks when used with the --enum option.

November 18th 2004 Version 1.600 supports the enumerate functionality via --enum switch

June 24th 2004 Version 1.502 released

Status

The 2.8.5.x version is a stable version.


Acknowledgements I wish to thank those who have contributed significantly to the development of ln.exe. Those include:

Endre Both for being a great tester with endless patience and for excellent constructive feedback on features during development.


Open Issues

The --backup mode does not yet support encrypted directories.


Links

License
  • This program is provided as is. See license.txt from this distribution for legal issues.
  • ln.exe uses tre as the regular expression machine. See the tre license.
  • ln.exe uses ultragetopt for command line parsing. See the ultragetopt license.
  • ln.exe uses the Rockall v4.0 heapmanager for fast heap operations. See the EULA.

Contact / Donations

Bug reports, or feature requests send to Hermann Schinagl..

ln.exe is and will be freeware, but if ln.exe was really helpful for you and saved lots of your time please think of donations either via PayPal


or by flattring me


or by sending me a gift certificate from

amazon.de .




or by donating bitcoins:

bitcoinlogo
19irqhz5cMDkp2hf9YWqDY26PgYguJQdc7 bitcoinlseqr


spacer
Download
Windows 2000
Windows XP
Windows Server 2003
Windows Server 2008
Windows Vista
Windows 7
Windows 8.x

Please make sure that the necessary runtime .dlls are installed on your system. This prerequisites package can be downloaded from Microsoft:



vcredist_x86.exe for VS2005 SP1, version 6195/June 2011 (2.6 Mb)

Afterwards install the
ln.zip (1.93Mb)

 
Windows XP64
Windows Vista64
Windows 7 64bit
Windows 8.x 64bit
Windows Server 2012 64bit
Please make sure that the necessary runtime .dlls are installed on your system. This prerequisites package can be downloaded from Microsoft:

vcredist_x64.exe for VS2005 SP1, version 6195/June 2011 (3.0 Mb)

Afterwards install the
ln64.zip (1.99Mb)



Windows Itanium
Windows 7 Itanium
Windows 8 Itanium
Please make sure that the necessary runtime .dlls are installed on your system. This prerequisites package can be downloaded from Microsoft:

vcredist_IA64.exe for VS2005 SP1, version 6195/June 2011 (6.3 Mb)

Afterwards install the
lnItanium.zip (2.13Mb)

Symbolic Link Driver
for Windows XP
The driver to enable even WindowsXP with symbolic link functionality is provided courtesy of Masatoshi Kimura. Yo ucan download the driver from his homepage or from my site acting as a mirror.

Symbolic Drivers for WindowsXP 64 (86kb) [Original Location]
Symbolic Drivers for WindowsXP 64 (86kb) [Mirror schinagl.priv.at]

Symbolic Drivers for WindowsXP (86kb) [Original Location]
Symbolic Drivers for WindowsXP (86kb) [Mirror schinagl.priv.at]

Sources for Drivers (23kb) [Original Location]
Sources for Drivers (23) [Mirror schinagl.priv.at]