Warp-CLI - Full Alpha Release V1.7

Warp-CLI is tool designed to make interacting with Facebook’s Warp Speed Data Transfer (WDT) pain-free.

https://github.com/JustinTimperio/warp-cli This tool is still under development. Please feel free to submit any bugs or feature requests.

Abstract


WDT is designed to provide the highest possible speed when transferring files(to be only hardware and network limited). WDT provides many advantages over most file transfer protocols including: native concurrency, end-to-end encryption, IPV6 support, and the ability to easily achieve +40Gbit speeds when supported. Unlike most file transfer solutions (Except NORM) WDT provides a native parallel solution for transferring files by separating files into chunks then queuing them across an arbitrary number of threads and TCP connections. In most cases, file transfer times are dramatically reduced compared to traditional methods like FTP or HTTP.

While WDT provides several benefits, it requires a lengthy build process making it unsuitable for one time transfers. Additionally, if you are already using a modified version of SSH such as HPN-SSH, you are likely to see smaller performance gains comparatively. Since WDT is designed to fully saturate even the highest-end hardware, it is likely to overwhelm even enterprise networking hardware. Please consider this when transferring more than a 1TB of files.

Performance Gains


Below are timed downloads(in seconds) over my personal network which is 1 Gigabit. Each progressive transfer increases the total size of the transfer in GB, while reducing the total number of files being transferredf. WDT easily maintains near full 1 Gigabit saturation across all 3 transfers while HPN-SFTP and SSH struggle to transfer multiple small files(single-thread limited). With encryption disabled HPN-SSH reaches full saturation when transferring large files, while stock SSH continues to struggle under heavy load. If you have access to +10 Gigabit networking hardware you can expect WDT to scale to ~40 Gigabit and HPN-SSH to scale to ~10 Gigabit.

Performance Graphs

Design


Warp-CLI is mainly a wrapper for the limited existing CLI app provided by WDT. While the tool works extremely well, building performant commands for daily use is often unwieldy.

For example:

  wdt -num_ports=8 -avg_mbytes_per_sec=100 -progress_report_interval_millis=5000 -overwrite=false -directory /dir/to/recv | ssh ssh.alias wdt -num_ports=8 -avg_mbytes_per_sec=100 -progress_report_interval_millis=5000 -overwrite=false -directory /dir/to/fetch/ -

Warp-CLI shortens this command to:

  warp -f ssh.alias /dir/to/fetch/ /dir/to/recv

Demo_Gif

Macros

Warp-CLI also includes a macro system for repeating custom transfers with a single command. Macros are stored transfer commands (kept in ~/warp-cli/macros) that are invoked with warp -m macro_name.

To generate a macro:

  warp -gm daily_backup -f source_ssh /dir/to/backup /dir/to/store/backup -tr 16 -ri 10000 -ow true

This macro can now be called with:

  warp -m daily_backup

Usage


Warp-CLI features a number of shortcuts that attempt to make sending files as trivial and intuitive as possible. Additionally, warp.py can be imported into any python3 script, which provides more granular control over transfers. To see a list of all available options use wdt --help | less.

Warp-CLI provides three core transfer modes:

  • -s, –ship: Send a remote directory to another remote directory.

    warp -s source_ssh /dir/to/send dest_ssh /dir/to/receive
    
  • -f, –fetch: Pull a remote directory to a local directory.

    warp -f source_ssh /dir/to/fetch /dir/to/receive
    
  • -p, –push: Send a local directory to a remote directory.

    warp -p /dir/to/push dest_ssh /dir/to/receive
    

Flags

  • -tr, --threads - default=8: In most cases, 8 threads is sufficient to saturate the connection. You may want to raise or lower this depending on your hardware.
  • -ri, --report_interval- default=5000: This limits the heartbeat report to 5000 milliseconds(5 seconds).
  • -ts, --throttle_speed - default=105: This setting throttles the transfer to an average mbytes per second.
  • -ow, --overwrite - default=false: Allow the receiver to overwrite existing files.

Default Speed Throttle

To provide optimal performance, Warp-CLI throttles transfers to ~90% the capacity of a standard 1 Gigabit NIC by default. This is done because WDT WILL fully saturate a 1 Gigabit NIC to the point that other sessions open on the machine will timeout and crash. You can set this throttle to unlimited by using --throttle_speed=-1.

Utilities

Warp-CLI provides a number of utilities to streamline the daily use of WTD when sending files in high frequency.

  • -m, –macro: Execute a custom macro from ~/warp-cli/config/ by name.

    warp -m macro_name
    
  • -gm, –gen_macro: Enter your transfer command as normal and include the gen_macro with a name for your new macro.

    warp -gm macro_name -f source_ssh /dir/to/fetch /dir/to/receive -tr 16 -ri 10000 -ow true
    
  • -cp, –custom_parms: Inject any additional parameters available from wdt --help.

    warp -f /dir/to/receive source_ssh /dir/to/send -c '-skip_writes=true -start_port=12345'
    
  • -d, –daemon: Start a permanent receiver daemon on a local directory and export a file containing the connection URL and meta-data.

    warp -d /dir/to/receive
    
  • -i, –install: Attempt to install WDT and dependencies.

    warp -i /var/app/warp-cli
    
  • -rm, –uninstall: Uninstall Warp-CLI and config files.

    warp --uninstall
    

Setup - STILL UNDER DEVELOPMENT


Since WDT requires multiple dependencies, Warp-CLI attempts to provide a fully automated installation process for as many linux flavors as possible. If your flavor is not supported, please refer to the manual install documentation, Once you install WDT and its dependencies Warp-CLI will function normally.

Automatic Installation

To install WDT and Warp-CLI automaticly on your machine:

  1. sudo mkdir /var/app && sudo chmod 777 /var/app
  2. cd /var/app && git clone https://github.com/JustinTimperio/warp-cli.git
  3. python3 /var/app/warp-cli/core/warp.py --install /var/app/warp-cli

So far, automatic installation is available on:

  • Arch Linux
  • Ubuntu 19.xx and 18.xx Workstation and Server
  • Debian 10.x and 9.x
  • Fedora 30, 29 and 28 Workstation and Server

WDT requires CMAKE version > 3.2 or greater, making it incompatible on:

  • CentOS 7
  • Debian 8

Uninstall

Warp-CLI will remove itself from ~/warp-cli but WDT will remain installed.

`warp --uninstall /var/app/warp-cli`

OpenSSH

Warp uses ssh to securely share connection URLs via a standard Linux pipe. It expects the use of an RSA key, which does not require a user password. While it is possible to use PAM authentication or key passwords, I have not yet added this as a feature.

URL Sharing While Using --ship

Warp-CLI uses the recommended method for transferring URLs except for the command to send directories between two remote machines. URLs are returned to the host rather than being tunneled through the remote machine to the next remote machine. This allows two remote servers to transfer files between one another without having the permission needed to connect directly via ssh.

SSH Aliases

Since Warp-CLI is designed mainly for daily use, it is highly recommended(if not assumed) that you already have an ssh alias for the server you are connecting to. If you don’t have an existing SSH alias for the server you are transferring files to, please consider creating one.

Avatar
Justin Timperio
Researcher and Freelance Dev

Justin is a freelance developer and private researcher.