SFTP in Batch Mode Hangs Using Runtime.exec() in Java

When we have applications that call native commands via java.lang.Runtime.exec(), be wary of the invocations. It could potentially output an overwhelmingly large number of texts to the standard output—for example, the sftp command in Linux. Therefore, causing the codes to hang. For example, to transfer many files to a remote server, one typical way is in batch mode with put <local-file> commands in a batch file. Then we feed the file to the sftp command. This post shows how codes for running SFTP commands in batch mode could hang in Java.

Requirements

We can replicate the issue using the following items.

Windows 7 Professional SP1
Eclipse – Kepler Release
Java 1.7 (1.7.0_67 – Windows x86)

SFTP in Batch Mode Command In Java

Consider the following command.

stfp -b all_put_cmd.txt user123@remote-host:/home/user123/

1	stfp -b all_put_cmd.txt user123@remote-host:/home/user123/

The all_put_cmd.txt could contain the following commands:

put /home/user123/2000-files/file0001
put /home/user123/2000-files/file0002
...
put /home/user123/2000-files/file2000

put /home/user123/2000-files/file0001

put /home/user123/2000-files/file0002

...

put /home/user123/2000-files/file2000

If the number of commands in the batch file is overwhelmingly large and we execute it via Runtime.exec(), the subprocess may become blocked after a few hundred lines of texts. Worse, the operation will never be complete. The JDK documentation, in fact, mentions and explains this issue:

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

It basically means that the buffers that temporarily hold the outputs emitting from the subprocess are limited in size, and when free spaces run out, the subprocess may become blocked. To avoid this from happening, the standard input and output streams are read from and outputs redirected to a file or the main process’ standard output, e.g., System.out.println().

SFTP Won’t Hang Using StreamGlobber

We could use the following codes to help us read data off the streams to reduce the chance of hitting the buffer size limit. Unless we extract data from the output stream, we use the StreamGobbler class to redirect data.

public class FileUtil {
 
...
public static class StreamGobbler extends Thread {
  private InputStream is;
  private String type;
  private OutputStream os;
 
  public StreamGobbler(InputStream is, String type) {
    this(is, type, null);
  }
 
  public StreamGobbler(InputStream is, String type, OutputStream redirect) {
    this.is = is;
    this.type = type;
    this.os = redirect;
  }
 
  public void run() {
    try {
      PrintWriter pw = null;
      if (os != null)
        pw = new PrintWriter(os);
 
      InputStreamReader isr = new InputStreamReader(is);
      BufferedReader br = new BufferedReader(isr);
      String line=null;
      while ( (line = br.readLine()) != null) {
        if (pw != null)
          pw.println(line);
        System.out.println(type + ">" + line);
      }
      if (pw != null) {
          pw.flush();
          pw.close();
      }
    } catch (IOException ioe) {
      ioe.printStackTrace();
    }
  }
}
 
...
 
}

public class FileUtil {

...

public static class StreamGobbler extends Thread {

private InputStream is;

private String type;

private OutputStream os;

public StreamGobbler(InputStream is, String type) {

this(is, type, null);

}

public StreamGobbler(InputStream is, String type, OutputStream redirect) {

this.is = is;

this.type = type;

this.os = redirect;

}

public void run() {

try {

PrintWriter pw = null;

if (os != null)

pw = new PrintWriter(os);

InputStreamReader isr = new InputStreamReader(is);

BufferedReader br = new BufferedReader(isr);

String line=null;

while ( (line = br.readLine()) != null) {

if (pw != null)

pw.println(line);

System.out.println(type + ">" + line);

}

if (pw != null) {

pw.flush();

pw.close();

}

} catch (IOException ioe) {

ioe.printStackTrace();

}

...

}

StreamGlobber Usage In Java For SFTP in Batch Mode

Using the StreamGlobber class is simple. Consider the following codes. When we use these Java codes with SFTP operation in batch mode, the application will not hang.

...
 
p = Runtime.getRuntime().exec(command);
 
/* Write sftp's error to sftp_error_01_ */
FileOutputStream fosErr = new FileOutputStream(
ftpDetailsDTO.getLocalFilePath().trim() + FileUtil.FILE_SEPARATOR + "sftp_error_01_" + idOne.toString());
StreamGobbler errorGobbler = new StreamGobbler(p.getErrorStream(), "ERROR", fosErr);
 
/* Write sftp's output to sftp_out_01_ */
FileOutputStream fosOut = new FileOutputStream(
ftpDetailsDTO.getLocalFilePath().trim() + FileUtil.FILE_SEPARATOR + "sftp_output_01_" + idOne.toString());
StreamGobbler outputGobbler = new StreamGobbler(p.getInputStream(), "OUTPUT", fosOut);
 
errorGobbler.start();
outputGobbler.start();
int ret = p.waitFor();
 
Thread.sleep(5000);
if (ret != 0) {
  // Some problems were encountered
} else {
  // Successful operation
}
 
...