Schedule commands in LFTP using “at”

You can schedule downloads and uploads in LFTP in a simple manner by using the “at” command. Anything following the syntax described in this image below should work.

at

For example –

Mirror/download a remote folder two hours from the current time, using 10 connections + segmentation
at now + 2 hours -- queue mirror --use-pget-n=10 linuxisos/

Mirror/download a remote folder at 1AM tomorrow, using 10 connections + segmentation
at 1:00 tomorrow -- queue mirror --use-pget-n=10 linuxisos/

TPG FTTN NBN Speeds

Roughly three months have passed since connecting to NBN FTTN via TPG. I’m on the “up to” 100/40mbit plan. At a guess I’m ~500meters from the node. At this point I haven’t had any issues with the horrible speeds/congestion that some users report. FTTN sucks compared to FTTH for a ton of reasons that I won’t go into. The bottom line so far is that I’m more or less able to saturate the speed that I’m syncing up at regardless of peak/offpeak.

Modem Stats (TPG supplied Huawei HG-658)
Line standard VDSL2
Channel type Interleaved
Downstream line rate (kbit/s) 62945
Upstream line rate (kbit/s) 29311
Downstream SNR (dB) 6.7
Upstream SNR (dB) 6.5
Downstream line attenuation (dB) 15
Upstream line attenuation (dB) 5.1
Downstream output power (dBmV) 14.3
Upstream output power (dBmV) 8.4

Speedtest.net

Speedtest.net result - Unusually high latency from the server on this result.
Speedtest.net result – Unusually high latency from the server on this result. I’ve seen much better results matching right to the sync speed.

Speedtest.net – (Updated 08/09/2016)

Ahhh that's more like it!
Ahhh that’s more like it!

LFTP segmented download from NL based server

6.63MB/s
6.63MB/s

Segmented SFTP downloading using LFTP

If you’ve every tried saturating a fast connection using FTP/SFTP you may have run into problems where you can only achieve limited download speeds using a single thread. Segmented downloading can often be a solution. Bare in mind that segmented FTP/SFTP will open many sessions to the server you are connecting to. Depending on the situation this might not be ideal, however if you’re sure you have sufficient resources to do it (without pissing anyone off if the server is in shared environment), then it can work very well. For example – From my home connection I can usually only pull ~800KB/s on a single thread SFTP download from a dedicated server based in the Netherlands. Using segmented downloading I can easily max out my connection (~7MB/s). I’ve found that other software such as Bitkinex and CuteFTP on Windows are not able to match the speeds I get when using lftp.

You’ll need to install lftp – I run it on my Raspberry Pi.
sudo apt-get install lftp

Login to your server using lftp
lftp sftp://[email protected]

Change into the directory with files you want to download
cd /hdd01/downloads

Start a segmented download

A pget command using segmentation is used for single files.
pget -n 15 somefile.iso #where 15 is the number of segments

A mirror command using segmentation is for downloading whole directories.
mirror --use-pget-n=15 SomeDirectory #where 15 is the number of segments

You’ll need to experiment with the amount of segments – It’s best to use as few as you can, while still getting as much speed as you need. I tend to use 8 – 15 at absolute maximum.

lftp has queue support which can also be pretty useful. Essentially you can queue up a bunch of different transfers and pull up the status later on. You simply need to add queue to the start of your command. To check the queue you can use jobs -v