Mimic

"Mimic is a set of tools to create mirrors of file collections. It uses a combination of techniques to optimize the synchronization of the collections. Mimic can use either the HTTP, SSH or FTP protocol to copy files. It supports both pushing files to remote locations or pulling files from remote locations. This means that the Mimic tools only needs to be installed on one of the systems in the set of mirror hosts.

Mimic was written to address the performance issues that exist in tools like bbcp, rsync and rcp. While these tools are widely used they are not optimized for synchronizing large data collections. For example, bbcp achieves high speed copying of files, but the process is blind to the state of the destination file. While bbcp does support the use of check sums using this feature slows down the copy process as the check sum is calculated on demand. With rsync only those files (blocks) which are different are copied, but the process of building the block maps occurs with each invocation. For a collection with thousands or millions of files the building of the block list is very time consuming. With rcp copies are completely blind with every file copied everytime. Mimic takes the best of what bbcp, rsync and rcp offer and combines the features into a single tool designed specifically to deliver and synchronize mirrors of file collections at multiple distributed locations."

http://release.igpp.ucla.edu/igpp/mimic/

https://pds.nasa.gov/tools/tool-registry/