Applications can communicate with each other in various means. One is by uploading and downloading files to and from an FTP server. But for development, we often test our applications on our local machines. Therefore, it makes sense to have an FTP server running locally.
This post demonstrates how to upload and download (and read) files to and from an SFTP server using Spring Boot and Apache Camel.
First, Set Up Local Directory And An FTP Server
The fastest way to set up an SFTP server is by using an SFTP Docker image. Make sure the Docker container is running, and the FTP server is accessible before moving on.
Next, we set up a directory structure in our local machine for files to upload to the SFTP server. Then, we place files in the “in” directory, and our application moves to either of the other subdirectories accordingly.
1 2 3 4 5 | C:\USERS\KARLDEV\DESKTOP\NEW FOLDER\LOCAL-WINDOWS-DIR ├───done ├───error ├───in └───in-progress |
We do not need to set up the directory structure in the FTP server. Apache Camel will do that for us as our application downloads the file.
Spring Boot With Apache Camel
The fastest way to create the initial Maven project with Spring Boot and Apache Camel dependencies is through Spring Initilzr. Then, we update or add codes to the project.
pom.xml And Apache Camel Dependency
The project is using Spring Boot 2.2.5.RELEASE and the Apache Camel version that comes with it is 3.0.1.
1 2 3 4 5 6 7 | ... <dependency> <groupId>org.apache.camel.springboot</groupId> <artifactId>camel-spring-boot-starter</artifactId> <version>3.0.1</version> </dependency> ... |
Therefore, we need to add a dependency to camel-ftp with the same version for consistency.
1 2 3 4 5 6 7 | ... <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-ftp</artifactId> <version>3.0.1</version> </dependency> ... |
With camel-ftp, we can upload and download from an SFTP server using Apache Camel.
We also need httpclient for building the FTP URL to our SFTP server.
1 2 3 4 5 6 7 | ... <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.11</version> </dependency> ... |
The complete pom.xml would be as follow.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.2.5.RELEASE</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.turreta.camelspringbootdemo</groupId> <artifactId>com-turreta-camelspringbootdemo</artifactId> <version>0.0.1-SNAPSHOT</version> <name>com-turreta-camelspringbootdemo</name> <description>Demo for SFTP, Apache Camel, and Spring Boot</description> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.apache.camel.springboot</groupId> <artifactId>camel-spring-boot-starter</artifactId> <version>3.0.1</version> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-ftp</artifactId> <version>3.0.1</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.11</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project> |
Run Application As A Service
Uploading and downloading files to and from an SFTP server requires the application to be running all the time. The application allows the Apache Camel to regularly poll the SFTP Server and the local directory for new files to process.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | package com.turreta.camelspringbootdemo.comturretacamelspringbootdemo; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import java.util.concurrent.TimeUnit; @SpringBootApplication public class ComTurretaCamelspringbootdemoApplication { public static void main(String[] args) { SpringApplication.run(ComTurretaCamelspringbootdemoApplication.class, args); System.out.println("Ok"); runAsService(); } private static synchronized void runAsService() { while (true) { try { TimeUnit.MINUTES.sleep(1); } catch (InterruptedException e) { e.printStackTrace(); } } } } |
Codes To Download Files
The following codes download files from the SFTP server. The application polls the “/foo-home/in” directory for files with “_TURRETA_” in their file names.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | package com.turreta.camelspringbootdemo.comturretacamelspringbootdemo; import org.apache.camel.builder.RouteBuilder; import org.apache.http.client.utils.URIBuilder; import org.springframework.stereotype.Component; import java.net.URI; @Component public class SFTPDownloadFileRoute extends RouteBuilder { @Override public void configure() throws Exception { URI fromFtpUrl = new URIBuilder() .setScheme("sftp") .setHost("127.0.0.1") .setPort(9922) .setPath("/foo-home/in") .addParameter("username", "foo") .addParameter("password", "pass") .addParameter("passiveMode", "false") .addParameter("antInclude", "*_TURRETA_*") .addParameter("initialDelay", "10s") .addParameter("delay", "50") .addParameter("moveFailed", "/foo-home/error") .addParameter("move", "/foo-home/done") .addParameter("preMove", "/foo-home/in-progress") .addParameter("readLock", "changed") .addParameter("readLockMinAge", "1m") .addParameter("readLockTimeout", "70000") .addParameter("readLockCheckInterval", "5000") .addParameter("stepwise", "false") .addParameter("useUserKnownHostsFile", "false") .build(); from(fromFtpUrl.toString()) .setHeader("useId").simple("system123") .setHeader("fileName").simple("${file:name}") .bean(SFTPService.class, "downloadAndProcessFile(${body}, ${headers})") .log("Downloaded file ${file:name} complete."); } } |
The SFTPService class allows us to display the content of every file the application reads.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | package com.turreta.camelspringbootdemo.comturretacamelspringbootdemo; import org.springframework.stereotype.Service; import java.util.Map; @Service public class SFTPService { public void downloadAndProcessFile(String fileContent, Map<String, Object> headers) { System.out.println("File: " + headers.get("fileName") + " | Content: " + fileContent + "|"); } } |
Codes To Upload Files
The following codes poll the local directory “C:/Users/karldev/Desktop/New folder/local-windows-dir/in” for files that have “_TURRETA_” in their file names. Then, they upload them to the SFTP server in the “/foo-home/in” directory.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | package com.turreta.camelspringbootdemo.comturretacamelspringbootdemo; import org.apache.camel.builder.RouteBuilder; import org.apache.http.client.utils.URIBuilder; import org.springframework.stereotype.Component; import java.net.URI; @Component public class SFTPUploadFileRoute extends RouteBuilder { @Override public void configure() throws Exception { URI toFtpUrl = new URIBuilder() .setScheme("sftp") .setHost("127.0.0.1") .setPort(9922) .setPath("/foo-home/in") .addParameter("initialDelay", "10s") .addParameter("delay", "50") .addParameter("username", "foo") .addParameter("password", "pass") .addParameter("passiveMode", "false") .addParameter("useUserKnownHostsFile", "false") .build(); StringBuilder fromURL = new StringBuilder(); fromURL.append("file:/C:/Users/karldev/Desktop/New folder/local-windows-dir/in?"); fromURL.append("antInclude=*_TURRETA_*&"); fromURL.append("moveFailed=../error&"); fromURL.append("move=../done&"); fromURL.append("preMove=../in-progress&"); fromURL.append("readLock=changed&"); fromURL.append("readLockMinAge=1m&"); fromURL.append("readLockTimeout=70000&"); fromURL.append("readLockCheckInterval=5000"); from(fromURL.toString()).to(toFtpUrl.toString()); } } |
Sample Console Output
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | "C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.1\jbr\bin\java.exe" -XX:TieredStopAtLevel=1 -noverify -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.1\lib\idea_rt.jar=54464:C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.1\bin" -Dfile.encoding=UTF-8 -classpath C:\Users\karldev\Desktop\dev\blogs\java\com-turreta-camelspringbootdemo\target\classes;C:\Users\karldev\.m2\repository\org\apache\camel\springboot\camel-spring-boot-starter\3.0.1\camel-spring-boot-starter-3.0.1.jar;C:\Users\karldev\.m2\repository\org\springframework\boot\spring-boot-starter\2.2.5.RELEASE\spring-boot-starter-2.2.5.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\boot\spring-boot\2.2.5.RELEASE\spring-boot-2.2.5.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-context\5.2.4.RELEASE\spring-context-5.2.4.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\boot\spring-boot-autoconfigure\2.2.5.RELEASE\spring-boot-autoconfigure-2.2.5.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\boot\spring-boot-starter-logging\2.2.5.RELEASE\spring-boot-starter-logging-2.2.5.RELEASE.jar;C:\Users\karldev\.m2\repository\ch\qos\logback\logback-classic\1.2.3\logback-classic-1.2.3.jar;C:\Users\karldev\.m2\repository\ch\qos\logback\logback-core\1.2.3\logback-core-1.2.3.jar;C:\Users\karldev\.m2\repository\org\apache\logging\log4j\log4j-to-slf4j\2.12.1\log4j-to-slf4j-2.12.1.jar;C:\Users\karldev\.m2\repository\org\apache\logging\log4j\log4j-api\2.12.1\log4j-api-2.12.1.jar;C:\Users\karldev\.m2\repository\org\slf4j\jul-to-slf4j\1.7.30\jul-to-slf4j-1.7.30.jar;C:\Users\karldev\.m2\repository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;C:\Users\karldev\.m2\repository\org\yaml\snakeyaml\1.25\snakeyaml-1.25.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-spring-boot\3.0.1\camel-spring-boot-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-spring\3.0.1\camel-spring-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-core-engine\3.0.1\camel-core-engine-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\spi-annotations\3.0.1\spi-annotations-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-base\3.0.1\camel-base-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-jaxp\3.0.1\camel-jaxp-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-bean\3.0.1\camel-bean-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-management-api\3.0.1\camel-management-api-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-core-xml\3.0.1\camel-core-xml-3.0.1.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-aop\5.2.4.RELEASE\spring-aop-5.2.4.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-beans\5.2.4.RELEASE\spring-beans-5.2.4.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-expression\5.2.4.RELEASE\spring-expression-5.2.4.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-tx\5.2.4.RELEASE\spring-tx-5.2.4.RELEASE.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-cloud\3.0.1\camel-cloud-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-caffeine-lrucache\3.0.1\camel-caffeine-lrucache-3.0.1.jar;C:\Users\karldev\.m2\repository\com\github\ben-manes\caffeine\caffeine\2.8.1\caffeine-2.8.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-main\3.0.1\camel-main-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-management\3.0.1\camel-management-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-util\3.0.1\camel-util-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-util-json\3.0.1\camel-util-json-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-log\3.0.1\camel-log-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\springboot\camel-core-starter\3.0.1\camel-core-starter-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-core\3.0.1\camel-core-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-browse\3.0.1\camel-browse-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-controlbus\3.0.1\camel-controlbus-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-dataformat\3.0.1\camel-dataformat-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-dataset\3.0.1\camel-dataset-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-direct\3.0.1\camel-direct-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-directvm\3.0.1\camel-directvm-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-language\3.0.1\camel-language-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-mock\3.0.1\camel-mock-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-ref\3.0.1\camel-ref-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-rest\3.0.1\camel-rest-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-saga\3.0.1\camel-saga-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-scheduler\3.0.1\camel-scheduler-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-seda\3.0.1\camel-seda-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-stub\3.0.1\camel-stub-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-timer\3.0.1\camel-timer-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-validator\3.0.1\camel-validator-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-vm\3.0.1\camel-vm-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-xpath\3.0.1\camel-xpath-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-xslt\3.0.1\camel-xslt-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-ftp\3.0.1\camel-ftp-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-file\3.0.1\camel-file-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-support\3.0.1\camel-support-3.0.1.jar;C:\Users\karldev\.m2\repository\org\apache\camel\camel-api\3.0.1\camel-api-3.0.1.jar;C:\Users\karldev\.m2\repository\com\jcraft\jsch\0.1.55\jsch-0.1.55.jar;C:\Users\karldev\.m2\repository\commons-net\commons-net\3.6\commons-net-3.6.jar;C:\Users\karldev\.m2\repository\org\apache\httpcomponents\httpclient\4.5.11\httpclient-4.5.11.jar;C:\Users\karldev\.m2\repository\org\apache\httpcomponents\httpcore\4.4.13\httpcore-4.4.13.jar;C:\Users\karldev\.m2\repository\commons-codec\commons-codec\1.13\commons-codec-1.13.jar;C:\Users\karldev\.m2\repository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-core\5.2.4.RELEASE\spring-core-5.2.4.RELEASE.jar;C:\Users\karldev\.m2\repository\org\springframework\spring-jcl\5.2.4.RELEASE\spring-jcl-5.2.4.RELEASE.jar com.turreta.camelspringbootdemo.comturretacamelspringbootdemo.ComTurretaCamelspringbootdemoApplication . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.2.5.RELEASE) 2020-02-29 23:11:12.290 INFO 7376 --- [ main] ComTurretaCamelspringbootdemoApplication : Starting ComTurretaCamelspringbootdemoApplication on LAPTOP-U5EN313J with PID 7376 (C:\Users\karldev\Desktop\dev\blogs\java\com-turreta-camelspringbootdemo\target\classes started by karldev in C:\Users\karldev\Desktop\dev\blogs\java\com-turreta-camelspringbootdemo) 2020-02-29 23:11:12.294 INFO 7376 --- [ main] ComTurretaCamelspringbootdemoApplication : No active profile set, falling back to default profiles: default 2020-02-29 23:11:13.331 INFO 7376 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.apache.camel.spring.boot.CamelAutoConfiguration' of type [org.apache.camel.spring.boot.CamelAutoConfiguration$$EnhancerBySpringCGLIB$$7af2485e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2020-02-29 23:11:14.055 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootRoutesCollector : Loading additional Camel XML routes from: classpath:camel/*.xml 2020-02-29 23:11:14.059 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootRoutesCollector : Loading additional Camel XML rests from: classpath:camel-rest/*.xml 2020-02-29 23:11:14.065 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootCamelContext : Apache Camel 3.0.1 (CamelContext: camel-1) is starting 2020-02-29 23:11:14.066 INFO 7376 --- [ main] o.a.c.management.JmxManagementStrategy : JMX is enabled 2020-02-29 23:11:14.470 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootCamelContext : StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html 2020-02-29 23:11:14.796 WARN 7376 --- [ main] o.a.c.c.file.remote.SftpOperations : JSCH -> Permanently added '127.0.0.1' (RSA) to the list of known hosts. 2020-02-29 23:11:15.274 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootCamelContext : Route: route1 started and consuming from: sftp://127.0.0.1:9922/foo-home/in?antInclude=*_TURRETA_*&delay=50&initialDelay=10s&move=%2Ffoo-home%2Fdone&moveFailed=%2Ffoo-home%2Ferror&passiveMode=false&password=xxxxxx&preMove=%2Ffoo-home%2Fin-progress&readLock=changed&readLockCheckInterval=5000&readLockMinAge=1m&readLockTimeout=70000&stepwise=false&useUserKnownHostsFile=false&username=foo 2020-02-29 23:11:15.287 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootCamelContext : Route: route2 started and consuming from: file:///C:/Users/karldev/Desktop/New%20folder/local-windows-dir/in?antInclude=*_TURRETA_*&move=..%2Fdone&moveFailed=..%2Ferror&preMove=..%2Fin-progress&readLock=changed&readLockCheckInterval=5000&readLockMinAge=1m&readLockTimeout=70000 2020-02-29 23:11:15.292 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootCamelContext : Total 2 routes, of which 2 are started 2020-02-29 23:11:15.294 INFO 7376 --- [ main] o.a.c.s.boot.SpringBootCamelContext : Apache Camel 3.0.1 (CamelContext: camel-1) started in 1.227 seconds 2020-02-29 23:11:15.302 INFO 7376 --- [ main] ComTurretaCamelspringbootdemoApplication : Started ComTurretaCamelspringbootdemoApplication in 3.583 seconds (JVM running for 5.265) Ok 2020-02-29 23:11:21.406 WARN 7376 --- [-windows-dir/in] o.a.c.c.file.remote.SftpOperations : JSCH -> Permanently added '127.0.0.1' (RSA) to the list of known hosts. File: TEST_TURRETA_KARL_01.txt | Content: This is a test file.| 2020-02-29 23:11:25.327 INFO 7376 --- [922/foo-home/in] route1 : Downloaded file TEST_TURRETA_KARL_01.txt complete. File: TEST_TURRETA_KARL_02.txt | Content: This is a test file.| 2020-02-29 23:13:22.784 INFO 7376 --- [922/foo-home/in] route1 : Downloaded file TEST_TURRETA_KARL_02.txt complete. |
In summary, the application copy files from a local directory to a remote directory in the SFTP server. Then, it reads them and displays their contents.