Re: camel-ftp 2.15.1 return ftp error code 421
Imagine that your FTP directory shows the "C:\Program Files" on Windows or "/usr/lib" on Linux. There is a very large number of files. Files can not be deleted. Process periodically scans all files in all folders, filters, and sorts. Uses a database table to store the names of files previously treated (IdempotentReposiotry). It is understood that when preprocessing is large. Here then is manifested error with ftp server. He believes that his overloaded work and an error 421. -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765276.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
There are only few changes between Camel 2.15.1 and Camel 2.15.0. I’m not sure if CAMEL-7565[1] is related to issue that you faced. [1]https://issues.apache.org/jira/browse/CAMEL-7565 -- Willem Jiang Red Hat, Inc. Web: http://www.redhat.com Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On April 3, 2015 at 9:21:02 PM, alexey-s (alex...@mail.ru) wrote: > Imagine that your FTP directory shows the "C:\Program Files" on Windows or > "/usr/lib" on Linux. There is a very large number of files. Files can not be > deleted. > Process periodically scans all files in all folders, filters, and sorts. > Uses a database table to store the names of files previously treated > (IdempotentReposiotry). > > It is understood that when preprocessing is large. > Here then is manifested error with ftp server. He believes that his > overloaded work and an error 421. > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765276.html > > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: camel-ftp 2.15.1 return ftp error code 421
Honestly. 2.15.0 On this effect is much less than 2.15.1. I used to CronScheduledRoutePolicy ftpDocReadRoutePolicy = new CronScheduledRoutePolicy(); ftpDocReadRoutePolicy.setRouteStartTime(routePolicyStartTime); ftpDocReadRoutePolicy.setRouteStopTime(routePolicyStopTime); ftpDocReadRoutePolicy.setRouteStopGracePeriod(routePolicyStopGracePeriod); I used to have broke the connection when receiving the contents of files from ftp server. Now the volume of data to be processed has increased by 50 times. To combat system start time and stop very large. No problem. To test the interval between the start and stop up to 10 minutes. The strangest thing that 2.15.1 error occurs frequently. For a couple of hours of tests in 2.15.0 fix one mistake. 2.15.1 At the same time, for the problem persists for more than 10 times. And here's remarks. -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765367.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
A short excerpt from karaf.log.xxx file -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765368.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
OK, here is another change CAMEL-8513[1] in Camel 2.15.1 which improves the download speed. I guess it may cause the ftp server complain about the server load. So the workaround could be set the receiveBufferSize option to 1024 to slow down the processing speed. [1]https://issues.apache.org/jira/browse/CAMEL-8513 -- Willem Jiang Red Hat, Inc. Web: http://www.redhat.com Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On April 6, 2015 at 10:35:40 PM, alexey-s (alex...@mail.ru) wrote: > Honestly. 2.15.0 On this effect is much less than 2.15.1. > I used to > CronScheduledRoutePolicy ftpDocReadRoutePolicy = new > CronScheduledRoutePolicy(); > ftpDocReadRoutePolicy.setRouteStartTime(routePolicyStartTime); > ftpDocReadRoutePolicy.setRouteStopTime(routePolicyStopTime); > ftpDocReadRoutePolicy.setRouteStopGracePeriod(routePolicyStopGracePeriod); > > I used to have broke the connection when receiving the contents of files > from ftp server. > Now the volume of data to be processed has increased by 50 times. > To combat system start time and stop very large. No problem. > To test the interval between the start and stop up to 10 minutes. > The strangest thing that 2.15.1 error occurs frequently. > For a couple of hours of tests in 2.15.0 fix one mistake. > 2.15.1 At the same time, for the problem persists for more than 10 times. > > > And here's remarks. > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765367.html > > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: camel-ftp 2.15.1 return ftp error code 421
It looks like quite a different matter. On the server, there is a restriction session. After 10 minutes, the server connection tearing. No problem in download speed of large files. Disconnection is the fault of the strategy routePolicyStopTime. But it's not scary. The main problem is to read a large number of file names. It is necessary to go into the database idempotentRepository. There check the name of a previously processed file. The total scan time a large number of file names transferred abroad in 10 minutes. FtpConsumer not time to start downloading the file contents. Parameter maxMessagesPerPoll I do not override. By default, it is 1000. Current indicators idempotentRepository repository size: 194159 files. -- Aleksey -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765482.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
I just try to guess what’s the cause of issue of camel-ftp 2.15.1 by looking up the change log. I’m not sure if changing the receiving buffer could make the difference. It looks like you just find the key of the issue, am I right? -- Willem Jiang Red Hat, Inc. Web: http://www.redhat.com Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On April 8, 2015 at 10:29:12 PM, alexey-s (alex...@mail.ru) wrote: > It looks like quite a different matter. > On the server, there is a restriction session. After 10 minutes, the server > connection tearing. > No problem in download speed of large files. Disconnection is the fault of > the strategy routePolicyStopTime. But it's not scary. > > The main problem is to read a large number of file names. > It is necessary to go into the database idempotentRepository. There check > the name of a previously processed file. > The total scan time a large number of file names transferred abroad in 10 > minutes. > FtpConsumer not time to start downloading the file contents. > > Parameter maxMessagesPerPoll I do not override. By default, it is 1000. > > Current indicators idempotentRepository repository size: 194159 files. > > -- > Aleksey > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765482.html > > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: camel-ftp 2.15.1 return ftp error code 421
Yes, I tried to find an answer to the question why the process ceases to take files from ftp server. The most simple - revert to the old assembly Camel. Lo and behold, the files were taken. But just a couple of days. Server file number increased by several hundred. The system stops to pick up the files. Began to appear regularly ftp error 421. Well, I have a filter on the list of directories. In the filter have multiple directories. A miracle occurred. The system is working properly. But I have to handle no small part, and all the directories. And then the session expires. I do not see the solution, how to raise the connection and to a certain point of the interrupted process, not the beginning. Aleksey -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765509.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
An exemplary work plan Ftp consumer. As we can see, at the beginning of a connection to the server. FTPClient establishes a new connection for a list of files in each directory. At the end of the first connection is switched off. Why hold the first connection? Another question. Some directories may not be available. Directory visible, and get a list of files is impossible. How can I go on to the next directory? --- Aleksey -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765763.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
The first connection is used to check if the ftp server is still alive before starting to pull the directory. If the RemoteFileConsumer cannot connect to the server, it just return the pulling process and wait for another pulling. -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765842.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel-ftp 2.15.1 return ftp error code 421
I think you may need to find another way to pull the sub directory instead of pulling root directory directly. -- View this message in context: http://camel.465427.n5.nabble.com/camel-ftp-2-15-1-return-ftp-error-code-421-tp5765272p5765843.html Sent from the Camel - Users mailing list archive at Nabble.com.