If you’ve every tried saturating a fast connection using FTP/SFTP you may have run into problems where you can only achieve limited download speeds using a single thread. Segmented downloading can often be a solution. Bare in mind that segmented FTP/SFTP will open many sessions to the server you are connecting to. Depending on the situation this might not be ideal, however if you’re sure you have sufficient resources to do it (without pissing anyone off if the server is in shared environment), then it can work very well. For example – From my home connection I can usually only pull ~800KB/s on a single thread SFTP download from a dedicated server based in the Netherlands. Using segmented downloading I can easily max out my connection (~7MB/s). I’ve found that other software such as Bitkinex and CuteFTP on Windows are not able to match the speeds I get when using lftp.
You’ll need to install lftp – I run it on my Raspberry Pi.
sudo apt-get install lftp
Login to your server using lftp
lftp sftp://[email protected]
Change into the directory with files you want to download
Start a segmented download
A pget command using segmentation is used for single files.
pget -n 15 somefile.iso #where 15 is the number of segments
A mirror command using segmentation is for downloading whole directories.
mirror --use-pget-n=15 SomeDirectory #where 15 is the number of segments
You’ll need to experiment with the amount of segments – It’s best to use as few as you can, while still getting as much speed as you need. I tend to use 8 – 15 at absolute maximum.
lftp has queue support which can also be pretty useful. Essentially you can queue up a bunch of different transfers and pull up the status later on. You simply need to add
queue to the start of your command. To check the queue you can use