Understanding SFTP Performance Issues

As we navigate the ever-expanding digital landscape, we regularly confront the challenge of transferring enormous volumes of data securely and effectively. Secure File Transfer Protocol (SFTP) has emerged as a trusted solution, offering secure file transfers between servers. However, when dealing with large files or high-latency networks, traditional SFTP transfers may lack speed.

While SFTP is a reliable and secure method for file transfers, it has drawbacks. Let’s delve into why SFTP can sometimes lead to performance problems.

The primary reason SFTP can cause performance issues lies in its design. SFTP uses a request-response protocol where the server acknowledges every block of data the client sends, meaning it processes one block at a time in sequence. This approach can result in significant idle time if the client waits for each acknowledgement before sending the next block. Many SFTP clients send blocks asynchronously to overcome this limitation.

Another factor that impacts SFTP’s performance is network latency. SFTP, like other TCP-based protocols, is sensitive to network delay. In high-latency networks, SFTP must wait for acknowledgements from the receiving end before sending more data. This round-trip delay can significantly decrease the transfer rate, especially when dealing with large or multiple files.

TCP window size can also have a substantial impact on SFTP performance. The TCP window size determines the amount of unacknowledged data in transit at any given time. If the window size is small, the sender will frequently have to pause and wait for acknowledgements, leading to inefficient use of the network bandwidth. While modern systems use TCP window scaling to overcome this issue, it can still be a bottleneck in some environments.

SFTP uses SSH for secure file transfers, which involves encryption and decryption of data. While this ensures the security of your data, it also introduces computational overhead. Encrypting data before sending and decrypting upon receiving can slow transfer speeds, especially on systems with limited CPU resources.

While SFTP provides robust security features for file transfers, its single-threaded nature, sensitivity to network latency, dependence on TCP window size, and encryption overhead can lead to performance problems.

To overcome this, we can leverage the power of the open-source SFTP client, Push SFTP. Designed for faster SFTP transfers, in testing, Push SFTP has boosted file transfer speeds, regularly achieving rates in laboratory tests that are approximately two and a half times greater than standard SFTP put performance, and we have observed higher rates in other environments.

Let’s explore leveraging Push SFTP to accelerate your SFTP transfers. In these examples, we use the most widely used SSH server, OpenSSH. In most cases, the standard OpenSSH configuration will support push transfers out of the box. Some servers, for example, those backed by cloud providers like S3 or Google Storage, may not be compatible.

Push SFTP is a cross-platform, open-source SFTP client devised to increase the performance of SFTP transfers. Although it uses the same underlying technology as SFTP (SSH File Transfer Protocol), Push SFTP incorporates additional enhancements to boost the speed and performance of file transfers.

Ensure your OpenSSH server is correctly configured and updated to the latest version. SFTP operates over the SSH protocol; thus, the SSH server must be running and adequately set up to handle SFTP transfers.

To enjoy the speed benefits of Push SFTP, you need to install the Push SFTP client on your machine. As an open-source client, it is free to download from the developers website.

If you’re a developer you can checkout the source code at GitHub too.

Before we attempt an accelerated transfer, we want to establish the base line so we can compare performance. We do this by uploading the same file we are going to use later with the standard SFTP client. I’m uploading a 500MB file which is large enough to get a feel for the difference in performance we will experience.

The test is performed from our office network in the UK that has a 1Gbit leased line with transfers uploaded to an EC2 instance running on AWS in US-EAST-1.

The base line test resulted in a throughput rate of 11.9MB/s.

After installing the Push SFTP client, the next step is to connect to your SFTP server. Open up your command console and start push-sftp command. If you have used the standard SFTP command line client before, then this should be familiar to you as it aims to replicate that so that it can serve as a more powerful replacement utility.

To start the transfer we will be using the push command instead of the put command.

push tmp.dat

With this default configuration, you should notice a significant increase in speed — as mentioned earlier, often about two and a half times faster than traditional SFTP put performance is our experience. In this test we acheived 27.7MB/s thats 2.3 times the original upload.

You can change the number of connections Push SFTP uses with the -c switch. The default behaviour of Push SFTP is to use three connections. You can change this when you issue the push command; for example, to use five connections, you would use:

push -c 5 tmp.dat

Here we can see a further improvement in performance to 37.3MB/s which is 3.2 times the original upload speed.

Finally we compare our results in a graph to visualise the difference performance levels.

While Push SFTP enhances speed, certain conditions might still result in slower transfers. These could be due to:

The open-source Push SFTP client offers a reliable way to accelerate file transfers over SFTP to an OpenSSH server. It is handy for those dealing with large files or working in high-latency network environments.

Nevertheless, it’s important to remember that other factors can impact transfer speed. Therefore, regularly monitor and optimize your network and server conditions for the best performance.

Related posts:

Penthouses are luxurious, spacious and high-end properties that are located on the top floors of a building. The penthouse in Delhi offer breathtaking views of the city, and are often equipped with…

Bitcoin Cash Community. We are living moments that undoubtedly reflect everything for which we do so much work. The positive valuation of Bitcoin Cash in its price is the sum of the collective…

Mindbody is a convenient and resourceful app designed to book exercise classes, wellness services, fitness memberships, and beauty for users worldwide. Users can easily find all kinds of fitness…