Segmented SFTP downloading using LFTP

If you've ever tried saturating a fast connection using FTP/SFTP you may have run into problems where you can only achieve limited download speeds using a single thread. Segmented downloading can often be a solution. Bare in mind that segmented FTP/SFTP will open many sessions to the server you are connecting to. Depending on the situation this might not be ideal, however if you're sure you have sufficient resources to do it (without generating too much load if your server is within a shared environment), then it can work very well.

You'll need to install lftp - I run it on a Raspberry Pi.

sudo apt-get install lftp

Login to your server using lftp

lftp sftp://[email protected]

Change into the directory with files you want to download

cd /hdd01/downloads

Start a segmented download

A pget command using segmentation is used for single files.

pget -n 15 somefile.iso #where 15 is the number of segments

A mirror command using segmentation is for downloading whole directories.

mirror --use-pget-n=15 SomeDirectory #where 15 is the number of segments

You'll need to experiment with the amount of segments - It's best to use as few as you can, while still getting as much speed as you need. I tend to use 8 - 15 at absolute maximum.

lftp has queue support which can also be pretty useful. Essentially you can queue up a bunch of different transfers and pull up the status later on. You simply need to add queue to the start of your command. To check the queue you can use jobs -v