Java, Software Development

SFTP in Batch Mode Hangs Using Runtime.exec() in Java

java Sort Enum Properties Monitor Directory File Lambda Expression parameter return types Validate XML String XSD ExecutorService AutoCloseable Try-With-Resources Convert LocalDateTime OffsetDateTime Run Codes Before Shutdown XML Tag Lambda PostgreSQL JDBC invoke JavaScript Applet SFTP Batch hang block main thread MySQL JDBC try-with-resources delete Unencrypted SoapFault Apache CXF

When we have applications that call native commands via java.lang.Runtime.exec(), be wary of the invocations. It could potentially output an overwhelmingly large number of texts to the standard output—for example, the sftp command in Linux.  Therefore, causing the codes to hang. For example, to transfer many files to a remote server,  one typical way is in batch mode with put <local-file> commands in a batch file. Then we feed the file to the sftp command. This post shows how codes for running SFTP commands in batch mode could hang in Java.

Requirements

We can replicate the issue using the following items.

  • Windows 7 Professional SP1
  • Eclipse – Kepler Release
  • Java 1.7 (1.7.0_67 – Windows x86)

SFTP in Batch Mode Command In Java

Consider the following command.

The all_put_cmd.txt could contain the following commands:

If the number of commands in the batch file is overwhelmingly large and we execute it via Runtime.exec(), the subprocess may become blocked after a few hundred lines of texts. Worse, the operation will never be complete. The JDK documentation, in fact, mentions and explains this issue:

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

It basically means that the buffers that temporarily hold the outputs emitting from the subprocess are limited in size, and when free spaces run out, the subprocess may become blocked. To avoid this from happening, the standard input and output streams are read from and outputs redirected to a file or the main process’ standard output, e.g., System.out.println().

SFTP Won’t Hang Using StreamGlobber

We could use the following codes to help us read data off the streams to reduce the chance of hitting the buffer size limit. Unless we extract data from the output stream, we use the StreamGobbler class to redirect data.

StreamGlobber Usage In Java For SFTP in Batch Mode

Using the StreamGlobber class is simple. Consider the following codes. When we use these Java codes with SFTP operation in batch mode, the application will not hang.

Got comments, or suggestions? Please visit our Facebook page!

You Might Also Like