Concurrent FTP endpoints close each others session
Hi, We have a Camel application which is running multiple FTP consumers to the same server and it seems as if one transaction failing affects others, like Client A sending GOODBYE kills Client B still running STOR (please see the attached log). [2018-04-05 09:53:28,718][2188512939-11851-19450835297][TRACE][org.apache.camel.component.file.remote.FtpOperations][[]][2188512939-11851-19450835297] doStoreFile(file1.xml) [2018-04-05 09:53:28,721][2188512939-11851-19450835297][DEBUG][org.apache.camel.component.file.remote.FtpOperations][[]][2188512939-11851-19450835297] About to store file: file1.xml using stream: java.io.ByteArrayInputStream@10adca16 [2018-04-05 09:53:28,721][2188512939-11851-19450835297][TRACE][org.apache.camel.component.file.remote.FtpOperations][[]][2188512939-11851-19450835297] Client storeFile: file1.xml [2018-04-05 09:53:28,786][2188512931-11851-19450835141][TRACE][org.apache.camel.component.file.remote.FtpOperations][[]][2188512931-11851-19450835141] Client disconnect [2018-04-05 09:53:28,786][2188512931-11851-19450835141][DEBUG][org.apache.camel.component.file.remote.RemoteFileProducer][[]][2188512931-11851-19450835141] Disconnecting from: ftp://secrethost:21/?binary=false&disconnect=true&fastExistsCheck=true&fileExist=Override&ftpClient.connectTimeout=3&ftpClient.dataTimeout=3&ftpClient.defaultTimeout=3&ftpClient.receiveDataSocketBufferSize=30720&ftpClient.sendDataSocketBufferSize=30720&ftpClient.strictReplyParsing=false&passiveMode=true&password=xx&reconnectDelay=1000&soTimeout=3&stepwise=false&username=user [2018-04-05 09:53:28,786][2188512931-11851-19450835141][TRACE][org.apache.camel.component.file.remote.FtpOperations][[]][2188512931-11851-19450835141] Client logout [2018-04-05 09:53:29,135][2188512931-11851-19450835141][TRACE][org.apache.camel.component.file.remote.FtpOperations][[]][2188512931-11851-19450835141] Client disconnect [2018-04-05 09:53:29,135][2188512939-11851-19450835297][DEBUG][org.apache.camel.component.file.remote.RemoteFileProducer][[]][2188512939-11851-19450835297] Exception occurred during stopping: File operation failed: null Socket closed. Code: 221 Doing the same thing with Http4 works because it does connection pooling. Kind regards, Florian CONFIDENTIALITY: This email and any files transmitted with it are confidential, may be legally privileged and are intended solely for the use of the individual or entity to whom they are addressed. If this has come to you in error, you must not copy, distribute, disclose or use any of the information it contains. Please notify the sender immediately and delete them from your system. PRIVACY: Perform Media Services Ltd (a member of the Perform Group, with Companies House registration number 3426471) may monitor email traffic data and also the content of email for the purposes of security and staff training. For information on how the Perform Group processes your information, please refer to http://www.performgroup.com/privacy-notice/. SECURITY: Please be aware that emails are not secure and may contain viruses. AUTHORITY: Any views or opinions expressed in this email are solely those of the sender and do not necessarily represent those of the Perform Group. COPYRIGHT: Copyright of this email and any attachments belongs to Perform Group Limited, Companies House registration number 6324278. Further information about Perform Group is available at www.performgroup.com
RE: Camel FTP Producer catches errors silently
I believe we are bit smarter now. We are consuming from JMS with
concurrentConsumers and so it is possible the the same FTP endpoint is served
twice.
With more verbose logging, it seems as if the connection is somehow shared and
the GOODBYE initiated by the operation finishing first kills the second
connection as well.
Is this a race condition or do we need to make sure that concurrency isn't
possible at this level?
-Ursprüngliche Nachricht-
Von: Florian Posch [mailto:florian.po...@performgroup.com]
Gesendet: Mittwoch, 4. April 2018 16:36
An: users@camel.apache.org
Betreff: Camel FTP Producer catches errors silently
Hi.
We are using Camel 2.20.2 to transfer files to remote servers.
Our route is built like that - sorry that I can't provide a fully featured
application and all beans.
from("direct:chunk.process")
.routeId(DEFAULT_ROUTE_ID_STRING)
.process("feedNotificationProcessor")
.onException(Exception.class).handled(false)
.end()
.recipientList(exchangeProperty(RECIPIENTLIST_HEADER).tokenize(RECIPIENTLIST_SEPARATOR))
.parallelProcessing().timeout(4)
.executorService(customThreadPoolExecutor)
.streaming().aggregationStrategyRef("recipientAggregationStrategy")
.to("direct:chunk.evaluate");
We are dispatching to (multiple) dynamic FTP producers with routes like that:
ftp://myHost/?fileName=myfile.xml&password=secret&binary=false&passiveMode=false&username=user&disconnect=true&fastExistsCheck=true&ftpClient.dataTimeout=3&ftpClient.defaultTimeout=3&ftpClient.connectTimeout=3&soTimeout=3&stepwise=false
When something fails, we would expect that we recognize that in the 'aggregate'
callback of our aggregation strategy call with 'CamelFailureEndpoint' and
'CamelExceptionCaught' properties set, so that we can retry or report failure
to our system.
However, in certain cases it appears that we only get a WARN logging but no
error set in the Exchange
[2018-04-03 14:38:21,426][[WARN
][org.apache.camel.component.file.remote.RemoteFileProducer][[]][2188110951-169-19445712825]
Exception occurred during disconnecting from: ftp://myhost/?
binary=false&disconnect=true&fastExistsCheck=true&ftpClient.connectTimeout=3&ftpClient.dataTimeout=3&ftpClient.defaultTimeout=3&ftpClient.receiveDataSocketBufferSize=30720&ftpClient.sendDataSocketBufferSize=307
20&ftpClient.strictReplyParsing=false&passiveMode=false&password=xx&soTimeout=3&stepwise=false&username=user
File operation failed: Connection closed without indication.. Code: 221
It looks as if this just happens on cleanup of connections after upload but our
recipient confirmed that the corresponding FTP session dropped just after
creating the data connection
2018-04-03 14:38:21,372 [99773] : decoded 'myfile.xml' into
'myfile.xml'
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_exec.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_rewrite.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_tls.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_core.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_core.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_ratio.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_quotatab.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_xfer.c
2018-04-03 14:38:21,372 [99773] : decoded 'myfile.xml' into
'myfile.xml'
2018-04-03 14:38:21,374 [99773] : dispatching CMD command 'STOR
myfile.xml' to mod_xfer.c
2018-04-03 14:38:21,397 [99773] : 150 Opening ASCII mode data
connection for myfile.xml
2018-04-03 14:38:21,430 [99773] : decoded '/myfile.xml' into
'/myfile.xml'
2018-04-03 14:38:21,430 [99773] : dispatching POST_CMD_ERR command
'QUIT /myfile.xml' to mod_exec.c
2018-04-03 14:38:21,430 [99773] : dispatching LOG_CMD_ERR command
'QUIT /myfile.xml' to mod_log.c
2
Camel FTP Producer catches errors silently
Hi.
We are using Camel 2.20.2 to transfer files to remote servers.
Our route is built like that - sorry that I can't provide a fully featured
application and all beans.
from("direct:chunk.process")
.routeId(DEFAULT_ROUTE_ID_STRING)
.process("feedNotificationProcessor")
.onException(Exception.class).handled(false)
.end()
.recipientList(exchangeProperty(RECIPIENTLIST_HEADER).tokenize(RECIPIENTLIST_SEPARATOR))
.parallelProcessing().timeout(4)
.executorService(customThreadPoolExecutor)
.streaming().aggregationStrategyRef("recipientAggregationStrategy")
.to("direct:chunk.evaluate");
We are dispatching to (multiple) dynamic FTP producers with routes like that:
ftp://myHost/?fileName=myfile.xml&password=secret&binary=false&passiveMode=false&username=user&disconnect=true&fastExistsCheck=true&ftpClient.dataTimeout=3&ftpClient.defaultTimeout=3&ftpClient.connectTimeout=3&soTimeout=3&stepwise=false
When something fails, we would expect that we recognize that in the 'aggregate'
callback of our aggregation strategy call with 'CamelFailureEndpoint' and
'CamelExceptionCaught' properties set, so that we can retry or report failure
to our system.
However, in certain cases it appears that we only get a WARN logging but no
error set in the Exchange
[2018-04-03 14:38:21,426][[WARN
][org.apache.camel.component.file.remote.RemoteFileProducer][[]][2188110951-169-19445712825]
Exception occurred during disconnecting from: ftp://myhost/?
binary=false&disconnect=true&fastExistsCheck=true&ftpClient.connectTimeout=3&ftpClient.dataTimeout=3&ftpClient.defaultTimeout=3&ftpClient.receiveDataSocketBufferSize=30720&ftpClient.sendDataSocketBufferSize=307
20&ftpClient.strictReplyParsing=false&passiveMode=false&password=xx&soTimeout=3&stepwise=false&username=user
File operation failed: Connection closed without indication.. Code: 221
It looks as if this just happens on cleanup of connections after upload but our
recipient confirmed that the corresponding FTP session dropped just after
creating the data connection
2018-04-03 14:38:21,372 [99773] : decoded 'myfile.xml' into
'myfile.xml'
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_exec.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_rewrite.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_tls.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_core.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_core.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_ratio.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_quotatab.c
2018-04-03 14:38:21,372 [99773] : dispatching PRE_CMD command 'STOR
myfile.xml' to mod_xfer.c
2018-04-03 14:38:21,372 [99773] : decoded 'myfile.xml' into
'myfile.xml'
2018-04-03 14:38:21,374 [99773] : dispatching CMD command 'STOR
myfile.xml' to mod_xfer.c
2018-04-03 14:38:21,397 [99773] : 150 Opening ASCII mode data
connection for myfile.xml
2018-04-03 14:38:21,430 [99773] : decoded '/myfile.xml' into
'/myfile.xml'
2018-04-03 14:38:21,430 [99773] : dispatching POST_CMD_ERR command
'QUIT /myfile.xml' to mod_exec.c
2018-04-03 14:38:21,430 [99773] : dispatching LOG_CMD_ERR command
'QUIT /myfile.xml' to mod_log.c
2018-04-03 14:38:21,430 [99773] : dispatching LOG_CMD_ERR command
'QUIT /myfile.xml' to mod_core.c
Is this intended behavior of Camel? it is quite inconvenient to miss out failed
uploads due to that (not being aware of any errors except for logging)
Kind regards,
Florian
CONFIDENTIALITY: This email and any files transmitted with it are confidential,
may be legally privileged and are intended solely for the use of the individual
or entity to whom they are addressed. If this has come to you in error, you
must not copy, distribute, disclose or use any of the information it contains.
Please notify the sender immediately and delete them from your system. PRIVACY:
Perform Media Services Ltd (a member of the Perform Group, with Companies House
registration number 3426471) may monitor email traffic data and also the
content of email for the purposes of security and staff training. For
information on how the Perform Group processes your information, please refer
to htt
AW: Camel Spring Boot: Bean initialization skipped in auto-configuration
OK, thanks for clarification. I'll go ahead and get rid of the camel-http-starter, using http4 is on my refactoring list anyway. Cheers, Florian -Ursprüngliche Nachricht- Von: Claus Ibsen [mailto:claus.ib...@gmail.com] Gesendet: Donnerstag, 12. Oktober 2017 14:52 An: users@camel.apache.org Betreff: Re: Camel Spring Boot: Bean initialization skipped in auto-configuration Mind that camel-http / camel-http-starter is deprecated, you should use camel-http4 or something else. On Thu, Oct 12, 2017 at 2:40 PM, Mark Nuttall wrote: > Sorry. I guess my reply was not clear. I wasn't saying it need > changes. I was just saying (and i tried it about a year ago) is that > Spring Boot needs a little config to expose things "correctly" when using > that component. > That article was the first one i came too when i googled. :( Maybe it > doesn't anymore? :) > > I was only trying to figure out what might possible cause the issue he > was seeing. And also to give him a work around. > > On Thu, Oct 12, 2017 at 8:27 AM, Claus Ibsen wrote: > >> On Thu, Oct 12, 2017 at 2:20 PM, Mark Nuttall wrote: >> > well that does sound like an issue with the camel-http-starter >> > component and would be something for the maintainers. >> > If i remember rightly, using that component with spring boot >> > requires >> some >> > minor coding additions to get things to work (i.e. like this - >> > http://www.frommknecht.net/spring-boot-meets-apache-camel/). It >> > might be related. >> > >> > So, what I suggest is just use Spring Controllers instead. It is >> > VERY >> easy >> > to do and just use the producerTemplate to send thing to camel routes. >> This >> > is what I do. >> > >> >> No camel-http-starter does not need any changes. >> That link your provided is to some old article from 2014. >> >> Sure spring-mvc is a good web library and you can easily integrate >> with with Camel with producer templates or other means. >> >> > On Thu, Oct 12, 2017 at 4:12 AM, Florian Posch < >> > florian.po...@performgroup.com> wrote: >> > >> >> Hi, >> >> >> >> Thanks for your swift reply. >> >> >> >> As mentioned in the Stackoverflow question, it is not a matter of >> generic >> >> objects like Map<> or Object because injection of a >> >> domain-specific >> class >> >> doesn't work either. >> >> >> >> Finally, I've found out what could potentially cause the problem: >> >> For my case, it depends on whether camel-http-starter is on the >> >> classpath or >> not. >> >> If I remove that dependency MyInflightRepositorys' data field gets >> >> injected, otherwise not. Not sure if this makes any sense in the >> >> given sample but that changes our application's behaviour... >> >> >> >> See https://github.com/DoNuT1985/camel-dependency-sample >> >> >> >> I hope this helps. >> >> >> >> Kind regards, >> >> Florian >> >> >> >> >> >> >> >> >> >> Florian Posch // Software Engineer >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> T - >> >> >> >> RunningBall Sports Information GmbH M - >> >> >> >> Plabutscher Straße 63 >> >> F - >> >> >> >> 8051 Graz >> >> E florian.po...@performgroup.com >> >> >> >> Austria >> >> Skype rb.fposch >> >> >> >> >> >> >> >> >> >> W www.performgroup.com >> >> >> >> >> >> -Ursprüngliche Nachricht- >> >> Von: Mark Nuttall [mailto:mknutt...@gmail.com] >> >> Gesendet: Mittwoch, 11. Oktober 2017 13:45 >> >> An: users@camel.apache.org >> >> Betreff: Re: Camel Spring Boot: Bean initialization skipped in >> >> auto-configuration >> >> >> >> I looked at stacktrace and your code here. I agree with the >> >> comment >> about >> >> the Map. The same thing can happen with strings. I am not sure >> >> what the >> map >> >> does but you might be able to that in Spring properties. If not, >> >> either create a class to
AW: Camel Spring Boot: Bean initialization skipped in auto-configuration
Hi,
Thanks for your swift reply.
As mentioned in the Stackoverflow question, it is not a matter of generic
objects like Map<> or Object because injection of a domain-specific class
doesn't work either.
Finally, I've found out what could potentially cause the problem: For my case,
it depends on whether camel-http-starter is on the classpath or not. If I
remove that dependency MyInflightRepositorys' data field gets injected,
otherwise not. Not sure if this makes any sense in the given sample but that
changes our application's behaviour...
See https://github.com/DoNuT1985/camel-dependency-sample
I hope this helps.
Kind regards,
Florian
Florian Posch // Software Engineer
T -
RunningBall Sports Information GmbH
M -
Plabutscher Straße 63
F -
8051 Graz
E florian.po...@performgroup.com
Austria
Skype rb.fposch
W www.performgroup.com
-Ursprüngliche Nachricht-
Von: Mark Nuttall [mailto:mknutt...@gmail.com]
Gesendet: Mittwoch, 11. Oktober 2017 13:45
An: users@camel.apache.org
Betreff: Re: Camel Spring Boot: Bean initialization skipped in
auto-configuration
I looked at stacktrace and your code here. I agree with the comment about the
Map. The same thing can happen with strings. I am not sure what the map does
but you might be able to that in Spring properties. If not, either create a
class to hold the map and make that class the Bean or provide a name to the
Bean and make sure you use that when you inject the bean.
I think we are going to need a copy of code that does not work. I use Spring
Boot and Camel pretty extensively and while it is not without issues, it was
usually due to me doing something probably I shouldnt have or there was a
better way. :)
Also, as suggested, use constructor injection. The version of Boot you are
using makes it very easy - all you need is a single constructor on your managed
object (to include the RouteBuilder). And if you use Lombok, it is even easier.
Mark
On Wed, Oct 11, 2017 at 4:41 AM, Florian Posch <
florian.po...@performgroup.com> wrote:
> Hi,
>
> I'm using Camel 2.19.3 and Spring Boot 1.5.7 including the
> corresponding Camel starter for my application and am having some
> issues with a custom InflightRepository getting set up.
>
> It is defined as Spring @Component and should get applied to the
> context by auto-configuration but the problem is that the bean isn't
> properly initialized
>
> @Component
> public class MyInflightRepository extends
> DefaultInflightRepository {
>
> @Autowired
> private MyInjectedBean bean;
>
> // ...
> }
>
>
> The repository gets applied to the CamelContext as well as it is
> available but to me it looks as if premature bean access in
> CamelAutoConfiguration.afterPropertiesSet
> causes the bean not being initialiized properly (so @Autowired members
> remain null but Spring starts up without an error). As far as I
> understood this happens because Application.getBeansByType(...) gets
> called before all property resolvers are initialized.
>
> I was able to work around that by manually initializing my
> InflightRepository and setting it in a CamelConfiguration bean but for
> my taste this is a bit dangerous to scan for custom beans that might
> not get initialized properly during startup.
>
> There's a more detailed snippet given on my Stackoverflow post:
> https://stackoverflow.com/questions/46680985/spring-
> boot-camel-autowiring-issues-in-camel-components
>
> Any question, please let me know...
>
> Kind regards,
> Florian
>
> Florian Posch // Software Engineer
>
>
>
>
>
>
>
>
>
>
>
> T -
>
> RunningBall Sports Information GmbH
> M -
>
> Plabutscher Straße 63
> F -
>
> 8051 Graz
> E florian.po...@performgroup.com
>
> Austria
> Skype rb.fposch
>
>
>
>
> W www.performgroup.com
>
>
>
CONFIDENTIALITY - This email and any files transmitted with it, are
confidential, may be legally privileged and are intended solely for the use of
the individual or entity to whom they are addressed. If this has come to you in
error, you must not copy, distribute, disclose or use any of the information it
contains. Please notify the sender immediately and delete them from your system.
SECURITY - Please be aware that communication by email, by its very nature, is
not 100% secure and by communicating with Perform Group by email you consent to
us monitoring and reading any such correspondence.
VIRUSES - Although this email message has been scanned for the presence of
computer viruses, the sender accepts no liability for any damage sustained as a
result o
Camel Spring Boot: Bean initialization skipped in auto-configuration
Hi,
I'm using Camel 2.19.3 and Spring Boot 1.5.7 including the corresponding Camel
starter for my application and am having some issues with a custom
InflightRepository getting set up.
It is defined as Spring @Component and should get applied to the context by
auto-configuration but the problem is that the bean isn't properly initialized
@Component
public class MyInflightRepository extends DefaultInflightRepository {
@Autowired
private MyInjectedBean bean;
// ...
}
The repository gets applied to the CamelContext as well as it is available but
to me it looks as if premature bean access in
CamelAutoConfiguration.afterPropertiesSet causes the bean not being
initialiized properly (so @Autowired members remain null but Spring starts up
without an error). As far as I understood this happens because
Application.getBeansByType(...) gets called before all property resolvers are
initialized.
I was able to work around that by manually initializing my InflightRepository
and setting it in a CamelConfiguration bean but for my taste this is a bit
dangerous to scan for custom beans that might not get initialized properly
during startup.
There's a more detailed snippet given on my Stackoverflow post:
https://stackoverflow.com/questions/46680985/spring-boot-camel-autowiring-issues-in-camel-components
Any question, please let me know...
Kind regards,
Florian
Florian Posch // Software Engineer
T -
RunningBall Sports Information GmbH
M -
Plabutscher Straße 63
F -
8051 Graz
E florian.po...@performgroup.com
Austria
Skype rb.fposch
W www.performgroup.com
