Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
As Claus pointed out, the correct syntax is now #bean:. /Vojtech
Re: hash (pound sign) notation in Camel 3.x
Thanks. Works perfectly.
From: Claus Ibsen
Sent: Sunday, June 7, 2020 10:01
To: users@camel.apache.org
Subject: [EXTERNAL] - Re: hash (pound sign) notation in Camel 3.x
Hi
You should use #bean:xxx to refer to a bean in the registry by its id.
https://urldefense.com/v3/__https://github.com/apache/camel/commit/763bfeb999f1fa7a3a3f1ef96c7a7eefa2032734__;!!Obbck6kTJA!P57o44tdLPjkMBevUjgX2hR98fYAijroozT90KDqfGVRXVzdFKKgDHAxegBR-O_0$
On Fri, Jun 5, 2020 at 1:56 PM Vojtech Fried
wrote:
>
> Hi,
>
> in Camel 2.x I used a # notation in URIs for endpoints. I used the
> # for example for passwords, because passwords can contain any
> characters and I don't want any Camel processing on it. I used SimpleRegistry
> to add objects and somehow plugged it into Camel context and then looked
> those object by name with #. It does not seem to work in 3.x. The
> replacement for this seems to be ${ref:} notation. But it works
> differently, at least for my "toD" URIs. If I put a String into the registry
> and reference it with "ref:", it is somehow processed by Camel when it
> normalizes URIs, I think properties are looked up if the string contains
> something like {{name}}.
>
> Is there any documentation on "#" or "ref"? Was "#" replaced with "ref" in
> 3.x or is it some problem in my environment? I haven't found it in migration
> to 3 docs.
>
> /Vojtech
--
Claus Ibsen
-
https://urldefense.com/v3/__http://davsclaus.com__;!!Obbck6kTJA!P57o44tdLPjkMBevUjgX2hR98fYAijroozT90KDqfGVRXVzdFKKgDHAxevVC1KEJ$
@davsclaus
Camel in Action 2:
https://urldefense.com/v3/__https://www.manning.com/ibsen2__;!!Obbck6kTJA!P57o44tdLPjkMBevUjgX2hR98fYAijroozT90KDqfGVRXVzdFKKgDHAxejVYAWjR$
hash (pound sign) notation in Camel 3.x
Hi,
in Camel 2.x I used a # notation in URIs for endpoints. I used the
# for example for passwords, because passwords can contain any characters
and I don't want any Camel processing on it. I used SimpleRegistry to add
objects and somehow plugged it into Camel context and then looked those object
by name with #. It does not seem to work in 3.x. The replacement for this
seems to be ${ref:} notation. But it works differently, at least for my
"toD" URIs. If I put a String into the registry and reference it with "ref:",
it is somehow processed by Camel when it normalizes URIs, I think properties
are looked up if the string contains something like {{name}}.
Is there any documentation on "#" or "ref"? Was "#" replaced with "ref" in 3.x
or is it some problem in my environment? I haven't found it in migration to 3
docs.
/Vojtech
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
I also had this problem. Neither URI encoding nor RAW helped. My solution in
2.x Camel was to use # notation in the URI with a String stored as
in a registry. Then Camel took the string and did not process it. It required
some fiddling with Registries to make them writable, but it was doable.
However, it seems that it does not work in Camel 3. Something like this is
"${ref:}, but when used with Strings, they are still somehow process by
Camel and in the end it fails with some special characters.
/Vojtech
From: Florian Patzl
Sent: Wednesday, June 3, 2020 14:49
To: users@camel.apache.org
Subject: [EXTERNAL] - Passwords in Camel endpoint URIs and limitations of RAW
syntax
Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint
URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and
mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've
tried and found out on the topic.
The main problem is that the reserved URI characters '+' and '&' (plus and
ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next
parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2=pwd2%2B2
A relevant post is here:
https://urldefense.com/v3/__https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623*34926623__;Iw!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKtilg8Ud$
Now, the solution in documentation is using the RAW(...) syntax:
https://urldefense.com/v3/__https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html*HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues__;Iw!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKrPfyljz$
So for example:
pop3://localhost:3110/?username=test2=RAW(pwd2+2)
Using that feature means we can no longer treat Camel URIs as pure URIs in our
application, because the '+' of the password must not be escaped for this to
work.
But if there's no way around that, we can deal with that.
However, when trying the limits of the RAW(...) syntax, we noticed that it can
not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support
for an alternative syntax using curly braces: RAW{...}, that I didn't find in
documentation:
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-12982__;!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKkO9zER_$
The last comment provides a pretty detailed summary of the options and limits.
The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks
URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
entries like:
pop3://localhost:3110/?password=xx&2%7D=test2
The same problem existed for the RAW(...) syntax:
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-11269__;!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKinvp2KB$
So the fix should be rather simple, I will check whether there's already an
issue for that and might even be able to submit a PR for that.
But, more importantly: By checking the passwords for ')&' and '}&' and
dynamically deciding the RAW syntax to use, we should be able to support any
password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as
technical limitation.
As an alternative to all of this, I sometimes saw the suggestion to configure
the component with 'useRawUri':
* In DefaultComponent, useRawUri() is hardcoded to false. That means for
applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the
components to override the method?
* Setting useRawUri on endpoint level does not seem to be supported:
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-6230__;!!Obbck6kTJA!KGASSfrNiG19hrz3XFpkjkI-DfK5STncbGLPofkcoybTnb0mkrK4hbqAKiki1NMf$
I tried that for the Mail component and got an error for unknown parameter
useRawUri.
So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...}
- depending on the input - is the best approach for handling passwords in
endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints?
I've briefly read up on using property placeholders in URIs and saw that we'd
still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints
in plain Java code would also solve the issue, but that would require a couple
of major changes in our application.
Best regards,
Florian
Re: stream caching not configurable for rest consumer?
Still isn't it a bug that if stream caching is not enabled in the context, what
happens in rest consumer is that it first uses the cache and second it uploads
the whole input into memory? Shouldn't the default strategy be to spool to file
after some limit?
BTW: I have found a way how to disable stream caching explicitly (without the
fix):
restConfiguration()
...
.endpointProperty("disableStreamCache", "true")
________
From: Vojtech Fried
Sent: Wednesday, April 1, 2020 12:27
To: users@camel.apache.org
Subject: Re: [EXTERNAL] - Re: stream caching not configurable for rest consumer?
I have created
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-14823__;!!Obbck6kTJA!PlMm8fZT6I7H24xabbHpPa5z-_TkYnV_X5Z33-L0c8PxlT2rwmPaoB3RCnc8KGlN$
Thanks.
From: Claus Ibsen
Sent: Wednesday, April 1, 2020 12:11
To: users@camel.apache.org
Subject: [EXTERNAL] - Re: stream caching not configurable for rest consumer?
Hi
Ah yeah looks like the code should be identical for those 2 (or maybe
just one common method as its InputStream as input).
And then they should check the camel context for stream caching in use
or not. Then you can turn it on|off on context level and have it work.
Can you create a new JIRA ticket
On Wed, Apr 1, 2020 at 11:06 AM Vojtech Fried wrote:
>
> Hi,
>
> I am trying to fix upload of huge files via camel rest endpoint. The problem
> is that stream caching enables itself in this case (without being enabled in
> camel context or the route), but when camel asks a StreamCachingStrategy if
> the data should be moved from memory to file, it always says no, because it
> has "enabled" flag false (because no one enabled stream caching).
>
> There is a Jira issue
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-8821__;!!Obbck6kTJA!Mgt7NJHDilliMB-tmUd04UaESpJLL6kaGPRG0pn6alRqo2ZN25TD-AQTWe2DPxWy$
> that should have fixed it, but it actually did not, because as the last
> comment says, it calls a wrong method "readResponse..." instead of
> "readRequest...", the fix is only in "readRequest..." method. So I think it
> should be reopened.
>
> Is there any other way how to disable stream caching?
>
> My code is in configure method in a class that extends RouteBuilder:
>
> restConfiguration()
> .contextPath(...)
> .enableCORS(true)
> .component("servlet")
> rest("...")
> .id("...")
> .produces("application/json")
> .post("...")
> .to("direct:...");
>
> Vojtech
--
Claus Ibsen
-
https://urldefense.com/v3/__http://davsclaus.com__;!!Obbck6kTJA!Mgt7NJHDilliMB-tmUd04UaESpJLL6kaGPRG0pn6alRqo2ZN25TD-AQTWU2V8rSp$
@davsclaus
Camel in Action 2:
https://urldefense.com/v3/__https://www.manning.com/ibsen2__;!!Obbck6kTJA!Mgt7NJHDilliMB-tmUd04UaESpJLL6kaGPRG0pn6alRqo2ZN25TD-AQTWV_-2ajy$
Re: [EXTERNAL] - Re: stream caching not configurable for rest consumer?
I have created https://issues.apache.org/jira/browse/CAMEL-14823
Thanks.
From: Claus Ibsen
Sent: Wednesday, April 1, 2020 12:11
To: users@camel.apache.org
Subject: [EXTERNAL] - Re: stream caching not configurable for rest consumer?
Hi
Ah yeah looks like the code should be identical for those 2 (or maybe
just one common method as its InputStream as input).
And then they should check the camel context for stream caching in use
or not. Then you can turn it on|off on context level and have it work.
Can you create a new JIRA ticket
On Wed, Apr 1, 2020 at 11:06 AM Vojtech Fried wrote:
>
> Hi,
>
> I am trying to fix upload of huge files via camel rest endpoint. The problem
> is that stream caching enables itself in this case (without being enabled in
> camel context or the route), but when camel asks a StreamCachingStrategy if
> the data should be moved from memory to file, it always says no, because it
> has "enabled" flag false (because no one enabled stream caching).
>
> There is a Jira issue
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CAMEL-8821__;!!Obbck6kTJA!Mgt7NJHDilliMB-tmUd04UaESpJLL6kaGPRG0pn6alRqo2ZN25TD-AQTWe2DPxWy$
> that should have fixed it, but it actually did not, because as the last
> comment says, it calls a wrong method "readResponse..." instead of
> "readRequest...", the fix is only in "readRequest..." method. So I think it
> should be reopened.
>
> Is there any other way how to disable stream caching?
>
> My code is in configure method in a class that extends RouteBuilder:
>
> restConfiguration()
> .contextPath(...)
> .enableCORS(true)
> .component("servlet")
> rest("...")
> .id("...")
> .produces("application/json")
> .post("...")
> .to("direct:...");
>
> Vojtech
--
Claus Ibsen
-
https://urldefense.com/v3/__http://davsclaus.com__;!!Obbck6kTJA!Mgt7NJHDilliMB-tmUd04UaESpJLL6kaGPRG0pn6alRqo2ZN25TD-AQTWU2V8rSp$
@davsclaus
Camel in Action 2:
https://urldefense.com/v3/__https://www.manning.com/ibsen2__;!!Obbck6kTJA!Mgt7NJHDilliMB-tmUd04UaESpJLL6kaGPRG0pn6alRqo2ZN25TD-AQTWV_-2ajy$
stream caching not configurable for rest consumer?
Hi,
I am trying to fix upload of huge files via camel rest endpoint. The problem is
that stream caching enables itself in this case (without being enabled in camel
context or the route), but when camel asks a StreamCachingStrategy if the data
should be moved from memory to file, it always says no, because it has
"enabled" flag false (because no one enabled stream caching).
There is a Jira issue https://issues.apache.org/jira/browse/CAMEL-8821 that
should have fixed it, but it actually did not, because as the last comment
says, it calls a wrong method "readResponse..." instead of "readRequest...",
the fix is only in "readRequest..." method. So I think it should be reopened.
Is there any other way how to disable stream caching?
My code is in configure method in a class that extends RouteBuilder:
restConfiguration()
.contextPath(...)
.enableCORS(true)
.component("servlet")
rest("...")
.id("...")
.produces("application/json")
.post("...")
.to("direct:...");
Vojtech
URI encoding and RAW values
Hi, I am trying to understand how the query parameters in Camel component configuration URIs work. I am working specifically with smtp output. I have seen https://camel.apache.org/manual/latest/faq/how-do-i-configure-password-options-on-camel-endpoints-without-the-value-being-encoded.html and https://camel.apache.org/manual/latest/faq/why-cant-i-use-sign-in-my-password.html and I also saw some discussions about it https://camel.465427.n5.nabble.com/Is-RAW-required-in-message-endpoint-URI-with-percent-endcoded-special-characters-td5748784.html But it seems to me that Camel handles URIs in a wrong way. I would expect that I as a client have to URLEncoder.encode the query parameter values and Camel would then parse the URI, find the parameters and URLDecoder.decode the values. This is not what Camel does though. Camel seems to take the whole query string, decode it and them it starts to parse it (starts to look for parameters). Since it is already decoded, it then does not decode the values. But it is wrong when a parameter value contains e.g. '&' character because the parser has no way to know if it is meant to be a control character (separator of parameters) or just a part of a value. At least this is what I saw in Camel code while debugging. URISupport.parseParameters calls uri.getQuery() which fetches the decoded query part. To fix it, URISupport.parseParameters should take uri.getRawQuery() and in URIScanner.getDecodedValue() it should just decode and not encode the percent chars first. There is probably more to fix, but this is the idea. The FAQ recommends to use RAW(...). But I think that 1. it should not be needed if URI handling in Camel was correct 2. RAW(...) has limitations of its own. (Try this: 'RAW(%3d&%%))&}&\t=+?)') Am I missing something? Vojtech
