[ https://issues.apache.org/jira/browse/UNOMI-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706508#comment-17706508 ]
Kevan Jahanshahi edited comment on UNOMI-758 at 3/30/23 1:05 PM: ----------------------------------------------------------------- h1. Truncated CSV export to FTP investigations: useful set camel FTP operations to debug: {code:java} log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code} Or modify directly the configuration file: {noformat} /JCUSTOMER_ROOT/etc/org.ops4j.pax.logging.cfg{noformat} By adding those lines: {code:java} log4j2.logger.org_apache_camel_component_file_remotefileendpoint.level = TRACE log4j2.logger.org_apache_camel_component_file_remotefileendpoint.name = org.apache.camel.component.file.RemoteFileProducer log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.level = TRACE log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.name = org.apache.camel.component.file.remote.FtpOperations log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.level = TRACE log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.name = org.apache.camel.component.file.remote.FtpsOperations {code} We manage to set log level in the env that reproduce the issue, and get some logs of the export: {code:java} 2023-03-30T08:57:49,072 | INFO | qtp446305861-92 | ProfileExportCollectRouteBuilder | 230 - org.apache.unomi.router-core - 1.6.1 | Configure Recurrent Route 'Export :: Collect Data' 2023-03-30T08:57:49,099 | INFO | qtp446305861-92 | OsgiDefaultCamelContext | 126 - org.apache.camel.camel-core - 2.20.2 | Route: 0de80ae4-6f3e-4ef5-a990-5bd4640846f8 started and consuming from: timer://collectProfile?fixedRate=true&period=24h 2023-03-30T08:58:08,192 | INFO | Camel (camel-1) thread #22 - timer://collectProfile | ExportRouteCompletionProcessor | 230 - org.apache.unomi.router-core - 1.6.1 | Processing route 0de80ae4-6f3e-4ef5-a990-5bd4640846f8 completed. 2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Connecting using FTPClient: org.apache.commons.net.ftp.FTPSClient@795a34bd 2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Connecting to ftps://************************:21 using connection timeout: 10000 2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Using passive mode connections 2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Using SoTimeout=300000 2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Attempting to login user: ******** using password: ******** 2023-03-30T08:58:08,482 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | User ******** logged in: true 2023-03-30T08:58:08,511 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execPbsz=0 2023-03-30T08:58:08,541 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execProt=P 2023-03-30T08:58:08,570 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | buildDirectory(root/FOLDER-1064) 2023-03-30T08:58:08,628 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/) 2023-03-30T08:58:08,629 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: / 2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | storeFile(root/FOLDER-1064/profiles-export-202303300858.csv) 2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | getCurrentDirectory() 2023-03-30T08:58:08,687 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Current dir: / 2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(root/FOLDER-1064) 2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: root 2023-03-30T08:58:08,718 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: FOLDER-1064 2023-03-30T08:58:08,747 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | doStoreFile(profiles-export-202303300858.csv) 2023-03-30T08:58:08,748 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | existsFile(profiles-export-202303300858.csv) 2023-03-30T08:58:09,067 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: ., target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: .., target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300835.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300826.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300823.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300842.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300847.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300851.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300856.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300843.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | About to store file: profiles-export-202303300858.csv using stream: java.io.ByteArrayInputStream@373d2781 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Client storeFile: profiles-export-202303300858.csv 2023-03-30T08:58:09,340 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Took 0.271 seconds (271 millis) to store file: profiles-export-202303300858.csv and FTP client returned: false 2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/) 2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: / 2023-03-30T08:58:09,370 | WARN | Camel (camel-1) thread #22 - timer://collectProfile | RemoteFileProducer | 126 - org.apache.camel.camel-core - 2.20.2 | Writing file failed with: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv] 2023-03-30T08:58:09,370 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Client logout 2023-03-30T08:58:09,399 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Client disconnect 2023-03-30T08:58:09,400 | ERROR | Camel (camel-1) thread #22 - timer://collectProfile | DefaultErrorHandler | 126 - org.apache.camel.camel-core - 2.20.2 | Failed delivery for (MessageId: ID-node10419-jc-prod-1680163907988-0-17328 on ExchangeId: ID-node10419-jc-prod-1680163907988-0-17327). Exhausted after delivery attempt: 1 caught: org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv]\n\nMessage History\n------------------------------------------------------------------------------------------------ org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv] at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307) ~[!/:2.20.2] at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[!/:2.20.2] at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58) ~[!/:2.20.2] at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132) ~[!/:2.20.2] at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127) ~[!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765) [!/:2.20.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62) [!/:2.20.2] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) [?:?] at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62) [!/:2.20.2] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [!/:2.20.2] at org.apache.camel.processor.Splitter.process(Splitter.java:114) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [!/:2.20.2] at java.util.TimerThread.mainLoop(Timer.java:556) [?:?] at java.util.TimerThread.run(Timer.java:506) [?:?] 2023-03-30T08:58:09,401 | WARN | Camel (camel-1) thread #22 - timer://collectProfile | AggregateProcessor | 126 - org.apache.camel.camel-core - 2.20.2 | Error processing aggregated exchange. Exchange[ID-node10419-jc-prod-1680163907988-0-17327]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv]] org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv] at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307) ~[!/:2.20.2] at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[!/:2.20.2] at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58) ~[!/:2.20.2] at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132) ~[!/:2.20.2] at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127) ~[!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) ~[!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) ~[!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) ~[!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[!/:2.20.2] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) ~[!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765) [!/:2.20.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62) [!/:2.20.2] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) [?:?] at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62) [!/:2.20.2] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [!/:2.20.2] at org.apache.camel.processor.Splitter.process(Splitter.java:114) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [!/:2.20.2] at java.util.TimerThread.mainLoop(Timer.java:556) [?:?] at java.util.TimerThread.run(Timer.java:506) [?:?]{code} On the FTP server side we have those logs: {noformat} Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 230 OK. Current directory is / Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [type] [A] Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 200 TYPE is now ASCII Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pbsz] [0] Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 200 PBSZ=0 Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [prot] [P] Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 200 Data protection level set to "private" Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pwd] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 257 "/" is your current location Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [root/FOLDER-1064] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is /root/FOLDER-1064 Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [/] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is / Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pwd] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 257 "/" is your current location Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [root] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is /root Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [FOLDER-1064] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is /root/FOLDER-1064 Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pasv] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 227 Entering Passive Mode (91,134,164,148,117,82) Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [nlst] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 150 Accepted data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 226 11 matches total Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pasv] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 227 Entering Passive Mode (91,134,164,148,117,101) Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [stor] [profiles-export-202303300901.csv] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 150 Accepted data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [NOTICE] /ftp-space/users//root/FOLDER-1064/profiles-export-202303300901.csv uploaded (32484 bytes, 551.02KB/sec) Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Error during read from data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Transfer aborted Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451 0.058 seconds (measured here), 0.54 Mbytes per second Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [/] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is / Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [quit] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [INFO] Logout. Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 221-Goodbye. You uploaded 32 and downloaded 0 kbytes. Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 221 Logout.{noformat} Interesting lines are: {code:java} Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Error during read from data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Transfer aborted{code} Indicate something goes wrong on the FTP server side, so theres is no issue/fix inside unomi product itself regarding the FTP file truncated and the issue should be investigated on the FTP server side. Also we tried a lot of option on the FTP destination used for the export configuration: >From destination: {noformat} ftps://*********/FOLDER-1064?password=******&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true {noformat} We tried options: * fileExist=Append (same issue, file is truncated) * fileExist=Fail (same issue, file is truncated) But we didn't tried all the various possible options documented here: [https://camel.apache.org/components/3.20.x/ftps-component.html] h2. Test from a local Unomi to the same ftp: Destination used: {code:java} ftp://*********@*********?password=*********&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true {code} Worked correctly (12000 elements/1.3MB) of profiles export successfully without issues: * target FTP server was the same * *ftp* protocol and *ftps* protocol tested and worked properly. h2. Conclusion about CSV truncated: Issues is not related to Unomi product and should be investigate on: * Networks communications between Unomi and the FTP server * FTP server logs * Check firewalls was (Author: jkevan): h1. Truncated CSV export to FTP investigations: useful set camel FTP operations to debug: {code:java} log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code} Or modify directly the configuration file: {noformat} /JCUSTOMER_ROOT/etc/org.ops4j.pax.logging.cfg{noformat} By adding those lines: {code:java} log4j2.logger.org_apache_camel_component_file_remotefileendpoint.level = TRACE log4j2.logger.org_apache_camel_component_file_remotefileendpoint.name = org.apache.camel.component.file.RemoteFileProducer log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.level = TRACE log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.name = org.apache.camel.component.file.remote.FtpOperations log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.level = TRACE log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.name = org.apache.camel.component.file.remote.FtpsOperations {code} We manage to set log level in the env that reproduce the issue, and get some logs of the export: {code:java} 2023-03-30T08:57:49,072 | INFO | qtp446305861-92 | ProfileExportCollectRouteBuilder | 230 - org.apache.unomi.router-core - 1.6.1 | Configure Recurrent Route 'Export :: Collect Data' 2023-03-30T08:57:49,099 | INFO | qtp446305861-92 | OsgiDefaultCamelContext | 126 - org.apache.camel.camel-core - 2.20.2 | Route: 0de80ae4-6f3e-4ef5-a990-5bd4640846f8 started and consuming from: timer://collectProfile?fixedRate=true&period=24h 2023-03-30T08:58:08,192 | INFO | Camel (camel-1) thread #22 - timer://collectProfile | ExportRouteCompletionProcessor | 230 - org.apache.unomi.router-core - 1.6.1 | Processing route 0de80ae4-6f3e-4ef5-a990-5bd4640846f8 completed. 2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Connecting using FTPClient: org.apache.commons.net.ftp.FTPSClient@795a34bd 2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Connecting to ftps://************************:21 using connection timeout: 10000 2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Using passive mode connections 2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Using SoTimeout=300000 2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Attempting to login user: ******** using password: ******** 2023-03-30T08:58:08,482 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | User ******** logged in: true 2023-03-30T08:58:08,511 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execPbsz=0 2023-03-30T08:58:08,541 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execProt=P 2023-03-30T08:58:08,570 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | buildDirectory(root/FOLDER-1064) 2023-03-30T08:58:08,628 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/) 2023-03-30T08:58:08,629 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: / 2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | storeFile(root/FOLDER-1064/profiles-export-202303300858.csv) 2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | getCurrentDirectory() 2023-03-30T08:58:08,687 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Current dir: / 2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(root/FOLDER-1064) 2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: root 2023-03-30T08:58:08,718 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: FOLDER-1064 2023-03-30T08:58:08,747 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | doStoreFile(profiles-export-202303300858.csv) 2023-03-30T08:58:08,748 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | existsFile(profiles-export-202303300858.csv) 2023-03-30T08:58:09,067 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: ., target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: .., target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300835.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300826.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300823.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300842.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300847.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300851.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300856.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Existing file: profiles-export-202303300843.csv, target file: profiles-export-202303300858.csv 2023-03-30T08:58:09,069 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | About to store file: profiles-export-202303300858.csv using stream: java.io.ByteArrayInputStream@373d2781 2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Client storeFile: profiles-export-202303300858.csv 2023-03-30T08:58:09,340 | DEBUG | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Took 0.271 seconds (271 millis) to store file: profiles-export-202303300858.csv and FTP client returned: false 2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/) 2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Changing directory: / 2023-03-30T08:58:09,370 | WARN | Camel (camel-1) thread #22 - timer://collectProfile | RemoteFileProducer | 126 - org.apache.camel.camel-core - 2.20.2 | Writing file failed with: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv] 2023-03-30T08:58:09,370 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Client logout 2023-03-30T08:58:09,399 | TRACE | Camel (camel-1) thread #22 - timer://collectProfile | FtpsOperations | 128 - org.apache.camel.camel-ftp - 2.20.2 | Client disconnect 2023-03-30T08:58:09,400 | ERROR | Camel (camel-1) thread #22 - timer://collectProfile | DefaultErrorHandler | 126 - org.apache.camel.camel-core - 2.20.2 | Failed delivery for (MessageId: ID-node10419-jc-prod-1680163907988-0-17328 on ExchangeId: ID-node10419-jc-prod-1680163907988-0-17327). Exhausted after delivery attempt: 1 caught: org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv]\n\nMessage History\n------------------------------------------------------------------------------------------------ org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv] at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307) ~[!/:2.20.2] at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[!/:2.20.2] at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58) ~[!/:2.20.2] at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132) ~[!/:2.20.2] at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127) ~[!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765) [!/:2.20.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62) [!/:2.20.2] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) [?:?] at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62) [!/:2.20.2] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [!/:2.20.2] at org.apache.camel.processor.Splitter.process(Splitter.java:114) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [!/:2.20.2] at java.util.TimerThread.mainLoop(Timer.java:556) [?:?] at java.util.TimerThread.run(Timer.java:506) [?:?] 2023-03-30T08:58:09,401 | WARN | Camel (camel-1) thread #22 - timer://collectProfile | AggregateProcessor | 126 - org.apache.camel.camel-core - 2.20.2 | Error processing aggregated exchange. Exchange[ID-node10419-jc-prod-1680163907988-0-17327]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv]] org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [root/FOLDER-1064/profiles-export-202303300858.csv] at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307) ~[!/:2.20.2] at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[!/:2.20.2] at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58) ~[!/:2.20.2] at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132) ~[!/:2.20.2] at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445) ~[!/:2.20.2] at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127) ~[!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) ~[!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) ~[!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) ~[!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[!/:2.20.2] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) ~[!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765) [!/:2.20.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62) [!/:2.20.2] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) [?:?] at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357) [!/:2.20.2] at org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62) [!/:2.20.2] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634) [!/:2.20.2] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [!/:2.20.2] at org.apache.camel.processor.Splitter.process(Splitter.java:114) [!/:2.20.2] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [!/:2.20.2] at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [!/:2.20.2] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [!/:2.20.2] at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [!/:2.20.2] at java.util.TimerThread.mainLoop(Timer.java:556) [?:?] at java.util.TimerThread.run(Timer.java:506) [?:?]{code} On the FTP server side we have those logs: {noformat} Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 230 OK. Current directory is / Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [type] [A] Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 200 TYPE is now ASCII Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pbsz] [0] Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 200 PBSZ=0 Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [prot] [P] Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 200 Data protection level set to "private" Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pwd] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 257 "/" is your current location Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [root/FOLDER-1064] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is /root/FOLDER-1064 Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [/] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is / Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pwd] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 257 "/" is your current location Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [root] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is /root Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [FOLDER-1064] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is /root/FOLDER-1064 Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pasv] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 227 Entering Passive Mode (91,134,164,148,117,82) Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [nlst] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 150 Accepted data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 226 11 matches total Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [pasv] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 227 Entering Passive Mode (91,134,164,148,117,101) Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [stor] [profiles-export-202303300901.csv] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 150 Accepted data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [NOTICE] /ftp-space/users//root/FOLDER-1064/profiles-export-202303300901.csv uploaded (32484 bytes, 551.02KB/sec) Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Error during read from data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Transfer aborted Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451 0.058 seconds (measured here), 0.54 Mbytes per second Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [cwd] [/] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 250 OK. Current directory is / Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] Command [quit] [] Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [INFO] Logout. Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 221-Goodbye. You uploaded 32 and downloaded 0 kbytes. Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 221 Logout.{noformat} Interesting lines are: {code:java} Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Error during read from data connection Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Transfer aborted{code} Indicate something goes wrong on the FTP server side, so theres is no issue/fix inside unomi product itself regarding the FTP file truncated and the issue should be investigated on the FTP server side. Also we tried a lot of option on the FTP destination used for the export configuration: >From destination: {noformat} ftps://*********/FOLDER-1064?password=******&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true {noformat} We tried options: * fileExist=Append (same issue, file is truncated) * fileExist=Fail (same issue, file is truncated) But we didn't tried all the various possible options documented here: [https://camel.apache.org/components/3.20.x/ftps-component.html] h2. Test from a local Unomi to the same ftp: Destination used: {code:java} ftp://*********@*********?password=*********&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true {code} Worked correctly (12000 elements/1.3MB) of profiles export successfully without issues: * target FTP server was the same * *ftp* protocol and *ftps* protocol tested and worked properly. > Bunch of fixes and improvements for Profile Export system > --------------------------------------------------------- > > Key: UNOMI-758 > URL: https://issues.apache.org/jira/browse/UNOMI-758 > Project: Apache Unomi > Issue Type: Bug > Reporter: David Griffon > Assignee: David Griffon > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > h2. Issue 1: make the camel route update asynchron to avoid blocking the > requests until camel route is killed. > When a route is updated, we stop all execution of the current route. > This stop can take time, and can lead to a request timeout. > The goal of this task is to stop the execution asynchronously. > h2. Issue 2: provide clear error messages in case something fail during camel > route creation/update > currently we are hiding the Exceptions: > inside the class > {*}org.apache.unomi.router.services.ExportConfigurationServiceImpl.java{*}, > there is no stack trace: > {noformat} > 2022-10-26T16:15:56,667 | ERROR | qtp824972552-99936 | > ExportConfigurationServiceImpl | 183 - org.apache.unomi.router-service - > 1.4.4.jahia | Error when trying to save/update running Apache Camel Route: > d9fa4556-a967-4410-aa1f-869ad66dbda5 {noformat} > h2. Issue 3: investigate on truncated CSV export to FTP > Sometimes we have CSV file that is truncated when exporting to FTP. > Using endpoint like: > {code:java} > ftps://USERNAME@HOST/FOLDER_PATH?password=XXXXXX&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)