[
https://issues.apache.org/jira/browse/UNOMI-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706508#comment-17706508
]
Kevan Jahanshahi edited comment on UNOMI-758 at 3/30/23 9:42 AM:
-----------------------------------------------------------------
h1. Truncated CSV export to FTP investigations:
useful set camel FTP operations to debug:
{code:java}
log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code}
Or modify directly the configuration file:
{noformat}
/JCUSTOMER_ROOT/etc/org.ops4j.pax.logging.cfg{noformat}
By adding those lines:
{code:java}
log4j2.logger.org_apache_camel_component_file_remotefileendpoint.level = TRACE
log4j2.logger.org_apache_camel_component_file_remotefileendpoint.name =
org.apache.camel.component.file.RemoteFileProducer
log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.level = TRACE
log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.name =
org.apache.camel.component.file.remote.FtpOperations
log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.level =
TRACE
log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.name =
org.apache.camel.component.file.remote.FtpsOperations {code}
We manage to set log level in the env that reproduce the issue, and get some
logs of the export:
{code:java}
2023-03-30T08:57:49,072 | INFO | qtp446305861-92 |
ProfileExportCollectRouteBuilder | 230 - org.apache.unomi.router-core - 1.6.1 |
Configure Recurrent Route 'Export :: Collect Data'
2023-03-30T08:57:49,099 | INFO | qtp446305861-92 | OsgiDefaultCamelContext
| 126 - org.apache.camel.camel-core - 2.20.2 | Route:
0de80ae4-6f3e-4ef5-a990-5bd4640846f8 started and consuming from:
timer://collectProfile?fixedRate=true&period=24h
2023-03-30T08:58:08,192 | INFO | Camel (camel-1) thread #22 -
timer://collectProfile | ExportRouteCompletionProcessor | 230 -
org.apache.unomi.router-core - 1.6.1 | Processing route
0de80ae4-6f3e-4ef5-a990-5bd4640846f8 completed.
2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Connecting using FTPClient:
org.apache.commons.net.ftp.FTPSClient@795a34bd
2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Connecting to
ftps://************************:21 using connection timeout: 10000
2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Using passive mode connections
2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Using SoTimeout=300000
2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Attempting to login user: ******** using
password: ********
2023-03-30T08:58:08,482 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | User ******** logged in: true
2023-03-30T08:58:08,511 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execPbsz=0
2023-03-30T08:58:08,541 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execProt=P
2023-03-30T08:58:08,570 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | buildDirectory(root/FOLDER-1064)
2023-03-30T08:58:08,628 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/)
2023-03-30T08:58:08,629 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: /
2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 |
storeFile(root/FOLDER-1064/profiles-export-202303300858.csv)
2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | getCurrentDirectory()
2023-03-30T08:58:08,687 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Current dir: /
2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(root/FOLDER-1064)
2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: root
2023-03-30T08:58:08,718 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: FOLDER-1064
2023-03-30T08:58:08,747 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 |
doStoreFile(profiles-export-202303300858.csv)
2023-03-30T08:58:08,748 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 |
existsFile(profiles-export-202303300858.csv)
2023-03-30T08:58:09,067 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file: ., target file:
profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file: .., target file:
profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300835.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300826.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300823.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300842.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300847.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300851.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300856.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300843.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | About to store file:
profiles-export-202303300858.csv using stream:
java.io.ByteArrayInputStream@373d2781
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Client storeFile:
profiles-export-202303300858.csv
2023-03-30T08:58:09,340 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Took 0.271 seconds (271 millis) to store
file: profiles-export-202303300858.csv and FTP client returned: false
2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/)
2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: /
2023-03-30T08:58:09,370 | WARN | Camel (camel-1) thread #22 -
timer://collectProfile | RemoteFileProducer | 126 -
org.apache.camel.camel-core - 2.20.2 | Writing file failed with: Error writing
file [root/FOLDER-1064/profiles-export-202303300858.csv]
2023-03-30T08:58:09,370 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Client logout
2023-03-30T08:58:09,399 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Client disconnect
2023-03-30T08:58:09,400 | ERROR | Camel (camel-1) thread #22 -
timer://collectProfile | DefaultErrorHandler | 126 -
org.apache.camel.camel-core - 2.20.2 | Failed delivery for (MessageId:
ID-node10419-jc-prod-1680163907988-0-17328 on ExchangeId:
ID-node10419-jc-prod-1680163907988-0-17327). Exhausted after delivery attempt:
1 caught: org.apache.camel.component.file.GenericFileOperationFailedException:
Error writing file
[root/FOLDER-1064/profiles-export-202303300858.csv]\n\nMessage
History\n------------------------------------------------------------------------------------------------
org.apache.camel.component.file.GenericFileOperationFailedException: Error
writing file [root/FOLDER-1064/profiles-export-202303300858.csv]
at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307)
~[!/:2.20.2]
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
~[!/:2.20.2]
at
org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
~[!/:2.20.2]
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)
~[!/:2.20.2]
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)
~[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765)
[!/:2.20.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62)
[!/:2.20.2]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
[?:?]
at
org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
[!/:2.20.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
[!/:2.20.2]
at org.apache.camel.processor.Splitter.process(Splitter.java:114)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79)
[!/:2.20.2]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
2023-03-30T08:58:09,401 | WARN | Camel (camel-1) thread #22 -
timer://collectProfile | AggregateProcessor | 126 -
org.apache.camel.camel-core - 2.20.2 | Error processing aggregated exchange.
Exchange[ID-node10419-jc-prod-1680163907988-0-17327]. Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException - Error
writing file [root/FOLDER-1064/profiles-export-202303300858.csv]]
org.apache.camel.component.file.GenericFileOperationFailedException: Error
writing file [root/FOLDER-1064/profiles-export-202303300858.csv]
at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307)
~[!/:2.20.2]
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
~[!/:2.20.2]
at
org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
~[!/:2.20.2]
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)
~[!/:2.20.2]
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)
~[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
~[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
~[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
~[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
~[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
~[!/:2.20.2]
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
~[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765)
[!/:2.20.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62)
[!/:2.20.2]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
[?:?]
at
org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
[!/:2.20.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
[!/:2.20.2]
at org.apache.camel.processor.Splitter.process(Splitter.java:114)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79)
[!/:2.20.2]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]{code}
On the FTP server side we have those logs:
{noformat}
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 230 OK. Current directory is /
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [type] [A]
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 200 TYPE is now ASCII
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pbsz] [0]
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 200 PBSZ=0
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [prot] [P]
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 200 Data protection level set to "private"
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pwd] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 257 "/" is your current location
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [root/FOLDER-1064]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /root/FOLDER-1064
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [/]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pwd] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 257 "/" is your current location
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [root]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /root
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [FOLDER-1064]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /root/FOLDER-1064
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pasv] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 227 Entering Passive Mode (91,134,164,148,117,82)
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [nlst] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 150 Accepted data connection
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 226 11 matches total
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pasv] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 227 Entering Passive Mode (91,134,164,148,117,101)
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [stor] [profiles-export-202303300901.csv]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 150 Accepted data connection
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[NOTICE] /ftp-space/users//root/FOLDER-1064/profiles-export-202303300901.csv
uploaded (32484 bytes, 551.02KB/sec)
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451-Error during read from data connection
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451-Transfer aborted
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451 0.058 seconds (measured here), 0.54 Mbytes per second
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [/]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [quit] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[INFO] Logout.
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 221-Goodbye. You uploaded 32 and downloaded 0 kbytes.
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 221 Logout.{noformat}
Interesting lines are:
{code:java}
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451-Error during read from data connection Mar 30 09:01:39 1420705f75a4
pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Transfer
aborted{code}
Indicate something goes wrong on the FTP server side, so theres is no issue/fix
inside unomi product itself regarding the FTP file truncated and the issue
should be investigated on the FTP server side.
Also we tried a lot of option on the FTP destination used for the export
configuration:
>From destination:
{noformat}
ftps://*********/FOLDER-1064?password=******&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true
{noformat}
We tried options:
* fileExist=Append (same issue, file is truncated)
* fileExist=Fail (same issue, file is truncated)
But we didn't tried all the various possible options documented here:
[https://camel.apache.org/components/3.20.x/ftps-component.html]
was (Author: jkevan):
h1. Truncated CSV export to FTP investigations:
useful set camel FTP operations to debug:
{code:java}
log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code}
Or modify directly the configuration file:
{noformat}
/JCUSTOMER_ROOT/etc/org.ops4j.pax.logging.cfg{noformat}
By adding those lines:
{code:java}
log4j2.logger.org_apache_camel_component_file_remotefileendpoint.level = TRACE
log4j2.logger.org_apache_camel_component_file_remotefileendpoint.name =
org.apache.camel.component.file.RemoteFileProducer
log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.level = TRACE
log4j2.logger.org_apache_camel_component_file_remote_ftpoperations.name =
org.apache.camel.component.file.remote.FtpOperations
log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.level =
TRACE
log4j2.logger.org_apache_camel_component_file_remote_ftpsoperations.name =
org.apache.camel.component.file.remote.FtpsOperations {code}
We manage to set log level in the env that reproduce the issue, and get some
logs of the export:
{code:java}
2023-03-30T08:57:49,072 | INFO | qtp446305861-92 |
ProfileExportCollectRouteBuilder | 230 - org.apache.unomi.router-core - 1.6.1 |
Configure Recurrent Route 'Export :: Collect Data'
2023-03-30T08:57:49,099 | INFO | qtp446305861-92 | OsgiDefaultCamelContext
| 126 - org.apache.camel.camel-core - 2.20.2 | Route:
0de80ae4-6f3e-4ef5-a990-5bd4640846f8 started and consuming from:
timer://collectProfile?fixedRate=true&period=24h
2023-03-30T08:58:08,192 | INFO | Camel (camel-1) thread #22 -
timer://collectProfile | ExportRouteCompletionProcessor | 230 -
org.apache.unomi.router-core - 1.6.1 | Processing route
0de80ae4-6f3e-4ef5-a990-5bd4640846f8 completed.
2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Connecting using FTPClient:
org.apache.commons.net.ftp.FTPSClient@795a34bd
2023-03-30T08:58:08,193 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Connecting to
ftps://************************:21 using connection timeout: 10000
2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Using passive mode connections
2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Using SoTimeout=300000
2023-03-30T08:58:08,384 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Attempting to login user: ******** using
password: ********
2023-03-30T08:58:08,482 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | User ******** logged in: true
2023-03-30T08:58:08,511 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execPbsz=0
2023-03-30T08:58:08,541 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | FTPClient initializing with execProt=P
2023-03-30T08:58:08,570 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | buildDirectory(root/FOLDER-1064)
2023-03-30T08:58:08,628 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/)
2023-03-30T08:58:08,629 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: /
2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 |
storeFile(root/FOLDER-1064/profiles-export-202303300858.csv)
2023-03-30T08:58:08,658 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | getCurrentDirectory()
2023-03-30T08:58:08,687 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Current dir: /
2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(root/FOLDER-1064)
2023-03-30T08:58:08,688 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: root
2023-03-30T08:58:08,718 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: FOLDER-1064
2023-03-30T08:58:08,747 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 |
doStoreFile(profiles-export-202303300858.csv)
2023-03-30T08:58:08,748 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 |
existsFile(profiles-export-202303300858.csv)
2023-03-30T08:58:09,067 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file: ., target file:
profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file: .., target file:
profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300835.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300826.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300823.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300842.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,068 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300847.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300851.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300856.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Existing file:
profiles-export-202303300843.csv, target file: profiles-export-202303300858.csv
2023-03-30T08:58:09,069 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | About to store file:
profiles-export-202303300858.csv using stream:
java.io.ByteArrayInputStream@373d2781
2023-03-30T08:58:09,069 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Client storeFile:
profiles-export-202303300858.csv
2023-03-30T08:58:09,340 | DEBUG | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Took 0.271 seconds (271 millis) to store
file: profiles-export-202303300858.csv and FTP client returned: false
2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | changeCurrentDirectory(/)
2023-03-30T08:58:09,340 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Changing directory: /
2023-03-30T08:58:09,370 | WARN | Camel (camel-1) thread #22 -
timer://collectProfile | RemoteFileProducer | 126 -
org.apache.camel.camel-core - 2.20.2 | Writing file failed with: Error writing
file [root/FOLDER-1064/profiles-export-202303300858.csv]
2023-03-30T08:58:09,370 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Client logout
2023-03-30T08:58:09,399 | TRACE | Camel (camel-1) thread #22 -
timer://collectProfile | FtpsOperations | 128 -
org.apache.camel.camel-ftp - 2.20.2 | Client disconnect
2023-03-30T08:58:09,400 | ERROR | Camel (camel-1) thread #22 -
timer://collectProfile | DefaultErrorHandler | 126 -
org.apache.camel.camel-core - 2.20.2 | Failed delivery for (MessageId:
ID-node10419-jc-prod-1680163907988-0-17328 on ExchangeId:
ID-node10419-jc-prod-1680163907988-0-17327). Exhausted after delivery attempt:
1 caught: org.apache.camel.component.file.GenericFileOperationFailedException:
Error writing file
[root/FOLDER-1064/profiles-export-202303300858.csv]\n\nMessage
History\n------------------------------------------------------------------------------------------------
org.apache.camel.component.file.GenericFileOperationFailedException: Error
writing file [root/FOLDER-1064/profiles-export-202303300858.csv]
at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307)
~[!/:2.20.2]
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
~[!/:2.20.2]
at
org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
~[!/:2.20.2]
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)
~[!/:2.20.2]
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)
~[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765)
[!/:2.20.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62)
[!/:2.20.2]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
[?:?]
at
org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
[!/:2.20.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
[!/:2.20.2]
at org.apache.camel.processor.Splitter.process(Splitter.java:114)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79)
[!/:2.20.2]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
2023-03-30T08:58:09,401 | WARN | Camel (camel-1) thread #22 -
timer://collectProfile | AggregateProcessor | 126 -
org.apache.camel.camel-core - 2.20.2 | Error processing aggregated exchange.
Exchange[ID-node10419-jc-prod-1680163907988-0-17327]. Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException - Error
writing file [root/FOLDER-1064/profiles-export-202303300858.csv]]
org.apache.camel.component.file.GenericFileOperationFailedException: Error
writing file [root/FOLDER-1064/profiles-export-202303300858.csv]
at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:307)
~[!/:2.20.2]
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
~[!/:2.20.2]
at
org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
~[!/:2.20.2]
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)
~[!/:2.20.2]
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)
~[!/:2.20.2]
at
org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)
~[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
~[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
~[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
~[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
~[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
~[!/:2.20.2]
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
~[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:765)
[!/:2.20.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62)
[!/:2.20.2]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
[?:?]
at
org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:757)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.doProcess(AggregateProcessor.java:357)
[!/:2.20.2]
at
org.apache.camel.processor.aggregate.AggregateProcessor.process(AggregateProcessor.java:270)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
[!/:2.20.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:711)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:634)
[!/:2.20.2]
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
[!/:2.20.2]
at org.apache.camel.processor.Splitter.process(Splitter.java:114)
[!/:2.20.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
[!/:2.20.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
[!/:2.20.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197)
[!/:2.20.2]
at
org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79)
[!/:2.20.2]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]{code}
On the FTP server side we have those logs:
{noformat}
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 230 OK. Current directory is /
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [type] [A]
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 200 TYPE is now ASCII
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pbsz] [0]
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 200 PBSZ=0
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [prot] [P]
Mar 30 09:01:38 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 200 Data protection level set to "private"
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pwd] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 257 "/" is your current location
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [root/FOLDER-1064]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /root/FOLDER-1064
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [/]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pwd] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 257 "/" is your current location
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [root]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /root
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [FOLDER-1064]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /root/FOLDER-1064
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pasv] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 227 Entering Passive Mode (91,134,164,148,117,82)
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [nlst] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 150 Accepted data connection
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 226 11 matches total
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [pasv] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 227 Entering Passive Mode (91,134,164,148,117,101)
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [stor] [profiles-export-202303300901.csv]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 150 Accepted data connection
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[INFO] TLS: Enabled TLSv1.3 with TLS_AES_256_GCM_SHA384, 256 secret bits cipher
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[NOTICE] /ftp-space/users//root/FOLDER-1064/profiles-export-202303300901.csv
uploaded (32484 bytes, 551.02KB/sec)
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451-Error during read from data connection
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451-Transfer aborted
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451 0.058 seconds (measured here), 0.54 Mbytes per second
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [cwd] [/]
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 250 OK. Current directory is /
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] Command [quit] []
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[INFO] Logout.
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 221-Goodbye. You uploaded 32 and downloaded 0 kbytes.
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 221 Logout.{noformat}
Interesting lines are:
{code:java}
Mar 30 09:01:39 1420705f75a4 pure-ftpd: (*******@**************.amazonaws.com)
[DEBUG] 451-Error during read from data connection Mar 30 09:01:39 1420705f75a4
pure-ftpd: (*******@**************.amazonaws.com) [DEBUG] 451-Transfer
aborted{code}
Indicate something goes wrong on the FTP server side, so theres is no issue/fix
inside unomi product itself regarding the FTP file truncated and the issue
should be investigated on the FTP server side.
Also we tried a lot of option on the FTP destination used for the export
configuration:
>From destination:
{noformat}
ftps://*********/FOLDER-1064?password=******&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true
{noformat}
We tried options:
* fileExist=Append (same issue, file is truncated)
* fileExist=Fail (same issue, file is truncated)
But we didn't tried all the various possible options documented here:
[https://camel.apache.org/components/3.20.x/ftps-component.html]
> Bunch of fixes and improvements for Profile Export system
> ---------------------------------------------------------
>
> Key: UNOMI-758
> URL: https://issues.apache.org/jira/browse/UNOMI-758
> Project: Apache Unomi
> Issue Type: Bug
> Reporter: David Griffon
> Assignee: David Griffon
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Issue 1: make the camel route update asynchron to avoid blocking the
> requests until camel route is killed.
> When a route is updated, we stop all execution of the current route.
> This stop can take time, and can lead to a request timeout.
> The goal of this task is to stop the execution asynchronously.
> h2. Issue 2: provide clear error messages in case something fail during camel
> route creation/update
> currently we are hiding the Exceptions:
> inside the class
> {*}org.apache.unomi.router.services.ExportConfigurationServiceImpl.java{*},
> there is no stack trace:
> {noformat}
> 2022-10-26T16:15:56,667 | ERROR | qtp824972552-99936 |
> ExportConfigurationServiceImpl | 183 - org.apache.unomi.router-service -
> 1.4.4.jahia | Error when trying to save/update running Apache Camel Route:
> d9fa4556-a967-4410-aa1f-869ad66dbda5 {noformat}
> h2. Issue 3: investigate on truncated CSV export to FTP
> Sometimes we have CSV file that is truncated when exporting to FTP.
> Using endpoint like:
> {code:java}
> ftps://USERNAME@HOST/FOLDER_PATH?password=XXXXXX&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
