Transforms are actions HyperTransfer™ can take on data as it is being transferred without loss of transmission speed. This ability is unique to TransferSoft™ and part of our intellectual property portfolio (patents pending). Transforms provide OEMs the ability to accelerate the applications and solutions they deliver to their end customers.
HyperTransfer implements the following data transforms, in virtually any combination, during data movement. All of these features operate at full bandwidth, independent of latency introduced by the distance between the source and destination services.
Checksums provide verification that the data transmitted is the data received. Supported checksum algorithms include MD5, SHA2, and SHA3.
Two data compression algorithms are provided: Compression A for higher compression at slightly higher CPU utilization, designed primarily for Links with bandwidth under 300Mbits per second; and Compression B, designed primarily for Links with bandwidth between 300Mbits per second up to 10GbE, with extremely high performance and slightly lower compression efficiency. All compression operations are auto-sensing; the algorithms determine whether the current data passing through the transport is compressible, and if not, turns compression off to save CPU cycles; when compressible data is encountered, compression operations are automatically re-enabled. Acceptable compression levels (for example 1.5:1) may be set, in which case compression will not be enabled unless the data passing through the transport is compressible to the selected level.
Two encryption algorithms and five alternative selections are provided. Available are AES CTR-mode or AES GCM-mode, both supporting either 128- or 256-bit key lengths. Also available is the TransferSoft proprietary RVX algorithm, with higher security by means of greater entropy, and with the MXC function of the RVX algorithm which destroys the keys instantly upon use. Enough RVX keys are created in the first few milliseconds of RVX operation to encrypt a data set of approximately 15 petabytes.
Statistical Delta Differencing
These statistically-based deduplication functions may be employed in four scenarios:
- Static Mode – a statistical “map” of a file(s) is maintained at the source, compared with a map created at the destination while files are being transferred, and only data that has changed is transferred. The destination files are updated in place, and the source Static “map” is updated to reflect the current state of the file(s).
- On-the-Fly Mode – as files are transferred, differences between the source and destination data is identified and only the changed data is transferred. HyperTransfer is orders of magnitude faster at high latency than the popular utility rsync.
- Cumulative Mode – as directories of files are transferred, a cumulative map is maintained so that any data chunk that has previously been encountered will result in the transfer of only the reference to the data at the destination. Thus, data transfer volume is reduced by taking advantage of previously encountered data, regardless of the file in which it originally resided.
- Stream Mode (Statistical Compression) – a unique method to deduplicate data that has not previously been transferred. In a manner similar to Cumulative Mode (above), a statistically-derived map of data sent during the process is used to identify data chunks that have previously passed through the stream, and only a reference to the destination location and length of the data is transferred, saving bandwidth by significantly reducing data volume on the network.