[ https://issues.apache.org/jira/browse/UNOMI-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706508#comment-17706508 ]
Kevan Jahanshahi edited comment on UNOMI-758 at 3/29/23 4:40 PM: ----------------------------------------------------------------- h1. Truncated CSV export to FTP investigations: useful set camel FTP operations to debug: {code:java} log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code} This give logs like: {noformat} 18:36:37.655 INFO [pool-21-thread-1] Configure Recurrent Route 'Export :: Collect Data' 18:36:37.683 INFO [pool-21-thread-1] Route: 82f5e3b0-67d2-4a8a-afd6-eda36f478485 started and consuming from: timer://collectProfile?fixedRate=true&period=1h 18:36:40.590 INFO [Camel (camel-1) thread #4 - timer://collectProfile] Processing route 82f5e3b0-67d2-4a8a-afd6-eda36f478485 completed. 18:36:40.612 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting using FTPClient: org.apache.commons.net.ftp.FTPClient@6ddfb396 18:36:40.613 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting to ftp://kevan@192.168.1.16:21 using connection timeout: 10000 18:36:40.656 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using passive mode connections 18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using SoTimeout=300000 18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Attempting to login user: kevan using password: ******** 18:36:41.294 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] User kevan logged in: true 18:36:41.298 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] buildDirectory(data3TO/trash) 18:36:41.304 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/) 18:36:41.305 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: / 18:36:53.156 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] storeFile(data3TO/trash/profiles-export-2023032918.csv) 18:36:53.157 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] getCurrentDirectory() 18:36:53.170 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Current dir: / 18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(data3TO/trash) 18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: data3TO 18:36:53.176 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: trash 18:36:55.827 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] doStoreFile(profiles-export-2023032918.csv) 18:36:55.828 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] About to store file: profiles-export-2023032918.csv using stream: java.io.ByteArrayInputStream@3cc24cfa 18:36:55.829 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Client appendFile: profiles-export-2023032918.csv 18:36:55.858 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] Took 0.030 seconds (30 millis) to store file: profiles-export-2023032918.csv and FTP client returned: true 18:36:55.859 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/) 18:36:55.860 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: /{noformat} was (Author: jkevan): h1. Truncated CSV export to FTP investigations: useful set camel FTP operations to debug: {code:java} log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code} This give logs like: {noformat} 18:36:37.655 INFO [pool-21-thread-1] Configure Recurrent Route 'Export :: Collect Data' 18:36:37.683 INFO [pool-21-thread-1] Route: 82f5e3b0-67d2-4a8a-afd6-eda36f478485 started and consuming from: timer://collectProfile?fixedRate=true&period=1h 18:36:40.590 INFO [Camel (camel-1) thread #4 - timer://collectProfile] Processing route 82f5e3b0-67d2-4a8a-afd6-eda36f478485 completed. 18:36:40.612 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting using FTPClient: org.apache.commons.net.ftp.FTPClient@6ddfb396 18:36:40.613 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting to ftp://kevan@192.168.1.16:21 using connection timeout: 10000 18:36:40.656 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using passive mode connections 18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using SoTimeout=300000 18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Attempting to login user: kevan using password: ******** 18:36:41.294 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] User kevan logged in: true 18:36:41.298 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] buildDirectory(data3TO/trash) 18:36:41.304 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/) 18:36:41.305 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: / 18:36:53.156 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] storeFile(data3TO/trash/profiles-export-2023032918.csv) 18:36:53.157 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] getCurrentDirectory() 18:36:53.170 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Current dir: / 18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(data3TO/trash) 18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: data3TO 18:36:53.176 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: trash 18:36:55.827 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] doStoreFile(profiles-export-2023032918.csv) 18:36:55.828 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] About to store file: profiles-export-2023032918.csv using stream: java.io.ByteArrayInputStream@3cc24cfa 18:36:55.829 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Client appendFile: profiles-export-2023032918.csv 18:36:55.858 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] Took 0.030 seconds (30 millis) to store file: profiles-export-2023032918.csv and FTP client returned: true 18:36:55.859 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/) 18:36:55.860 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: /{noformat} > Bunch of fixes and improvements for Profile Export system > --------------------------------------------------------- > > Key: UNOMI-758 > URL: https://issues.apache.org/jira/browse/UNOMI-758 > Project: Apache Unomi > Issue Type: Bug > Reporter: David Griffon > Assignee: David Griffon > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > h2. Issue 1: make the camel route update asynchron to avoid blocking the > requests until camel route is killed. > When a route is updated, we stop all execution of the current route. > This stop can take time, and can lead to a request timeout. > The goal of this task is to stop the execution asynchronously. > h2. Issue 2: provide clear error messages in case something fail during camel > route creation/update > currently we are hiding the Exceptions: > inside the class > {*}org.apache.unomi.router.services.ExportConfigurationServiceImpl.java{*}, > there is no stack trace: > {noformat} > 2022-10-26T16:15:56,667 | ERROR | qtp824972552-99936 | > ExportConfigurationServiceImpl | 183 - org.apache.unomi.router-service - > 1.4.4.jahia | Error when trying to save/update running Apache Camel Route: > d9fa4556-a967-4410-aa1f-869ad66dbda5 {noformat} > h2. Issue 3: investigate on truncated CSV export to FTP > Sometimes we have CSV file that is truncated when exporting to FTP. -- This message was sent by Atlassian Jira (v8.20.10#820010)