GridFTP

"GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.

 The aim of GridFTP is to provide a more reliable and high performance file transfer, for example to enable the transmission of very large files. GridFTP is used extensively within large science projects such as the Large Hadron Collider and by many supercomputer centers and other scientific facilities.

GridFTP also addresses the problem of incompatibility between storage and access systems. Previously, each data provider would make their data available in their own specific way, providing a library of access functions. This made it difficult to obtain data from multiple sources, requiring a different access method for each, and thus dividing the total available data into partitions. GridFTP provides a uniform way of accessing the data, encompassing functions from all the different modes of access, building on and extending the universally accepted FTP standard. FTP was chosen as a basis for it because of its widespread use, and because it has a well defined architecture for extensions to the protocol (which may be dynamically discovered)."

http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/

https://en.wikipedia.org/wiki/GridFTP

http://www.mcs.anl.gov/~kettimut/tutorials/SC07GridFTPTutorialSlides.pdf

https://hub.docker.com/r/wraithy/gridftp-server/~/dockerfile/

http://globustoolkit.com/alliance/publications/papers/Pipelining.pdf

https://www.mcs.anl.gov/software/data-intensive-software

BDM

Bulk Data Mover (BDM) is a scalable data transfer management tool for GridFTP transfer protocol.

https://sdm.lbl.gov/twiki/bin/view/Software/BDM/WebHome