Having finally caught up with the discussion about header/property copying and propagation, I thought it might be interesting to circle back to the original question Claus posed (slightly reworded) - what is the purpose of the out message?
As a "flame generator" question - If the exchange only carried 1 message, where each component in turn got to use and optionally modify. - how would this be different than the current pipeline behavior ? Currently if a component sets an out message, it is effectively replacing the in message for the next one. - would it not simplify the whole issue of header copying/propagation? No copying is necessary since it is the same message that travels through the route As an optional augmentation of this model, an exchange could carry 2 messages - the "current" or "working" message, which will eventually be the output of the exchange - a readonly primordial input message, as provided by the user. (start the flame...) > -----Original Message----- > From: Claus Ibsen [mailto:claus.ib...@gmail.com] > Sent: Tuesday, January 27, 2009 1:23 AM > To: dev@camel.apache.org > Subject: Re: Components setting data on OUT > > On Tue, Jan 27, 2009 at 6:02 AM, William Tam <email.w...@gmail.com> wrote: > > On Mon, Jan 26, 2009 at 11:34 PM, Hadrian Zbarcea <hzbar...@gmail.com> > wrote: > >> I think having dots ('.') in property names is not a good idea as they > don't > >> get along iirc with some technologies. > >> > > > > Sorry one more thing, the property name is just key to exchange > > property map. I believe we already use dots in exchange property > > names in some places today. > > > We have, in the Camel 2.0 design > (http://camel.apache.org/camel-20-design.html) a bullet, that we > should resolve the header key name issue. > James suggested to use a new naming convention CamelComponentName that I > like. > We should really remove all dots and usage with long packagenames etc. > as the name names dont apply with JMS spec and some other transports. > > And I think its something we should do prior to 2.0 release as its a > change that affects many components and internal parts in Camel. > > I am glad that I kinda started this ball, this thread sure has proven > that we needed a little heads up and consensus on the > header/properties issue in Camel. Keep the ball rolling. > > This is early morning and I am not juiced up for long and deep thoughts :) > > > >> Secondly, my point was that at this time, I am against the idea of > >> separating the headers into two maps. I don't see a compelling reason > to do > >> so and we should properly use properties as Roman and I think Claus > >> suggested. I'm confident that we'll find a good way to do it. > >> > >> Hadrian > >> > >> > >> On Jan 26, 2009, at 11:09 PM, William Tam wrote: > >> > >>> On Mon, Jan 26, 2009 at 5:34 PM, Hadrian Zbarcea <hzbar...@gmail.com> > >>> wrote: > >>>> > >>>> On Jan 26, 2009, at 4:17 PM, William Tam wrote: > >>>> > >>>>> On Mon, Jan 26, 2009 at 1:37 PM, Hadrian Zbarcea > <hzbar...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> I don't disagree, I was just suggesting that they should then > travel as > >>>>>> properties. > >>>>> > >>>>> Why shouldn't they (protocol headers) travel as headers (to avoids > >>>>> unnecessary copying between exchange properties and protocol > headers)? > >>>> > >>>> There should be no unnecessary copying, I agree. If a header (such > as > >>>> username/password) has broadly known and accepted semantics, the > endpoint > >>>> or > >>>> associated policy could set it as a property from start. Similarly, > an > >>>> endpoint should look at both properties and headers when sending a > >>>> message. > >>> > >>> I think I finally get it. Your example of username/password are > >>> really properties in my view. E.g, we can have a property > >>> org.apache.camel.headers.username. It is broadly known in Camel. > >>> Sure, we can stick it in exchange properties and let endpoint read it > >>> and create a protocol header with it (like, USER=foo). I am fine with > >>> that idea. That does not make org.apache.camel.headers.username=foo a > >>> protocol header, though. It really isn't. > >>> > >>> I am concerned about true custom protocol header that is totally > >>> unknown to Camel (say, foo=bar). I want my app to be able to define > >>> and insert some header in protocol message header. I argue that > >>> foo=bar should be set in the header not in exchange properties. > >>> > >>>> > >>>>> > >>>>> > >>>>>> Whatever we name them, and whatever mechanism we decide to use, > >>>>>> as pointed out before, we need to distinguish between headers that > are > >>>>>> endpoint/protocol specific and have no semantics outside the > endpoint > >>>>>> and > >>>>>> headers (which we called properties and didn't use consistently) > that > >>>>>> must > >>>>>> be carried over the lifetime of the Exchange. > >>>>> > >>>>> Custom protocol header fits into "no semantics outside the endpoint" > >>>>> (or else it won't get propagated) and so it goes to the exchange > >>>>> properties. How does the next component (in the pipeline) know > what > >>>>> one of the properties are intended to be sent in protocol header? > >>>>> Wouldn't it be better to let protocol headers travel as headers so > >>>>> only the "header candidates" will be potentially be sent? > >>>> > >>>> Not sure I understand your question, but I think we should use > policies > >>>> for > >>>> handling custom headers. > >>> > >>> Sorry about the typos and grammars. All I am saying is if we put user > >>> defined header in message headers, filter strategy (or policy if you > >>> will) won't have to iterate through the exchange properties for it. > >>> It is already in headers where it belongs. All we need to do is to > >>> apply filters to headers. It is much cleaner. In your example, if > >>> endpoint wants to read a specific property (e.g. > >>> org.apache.camel.headers.username) and make a header out of a > >>> property, it can certainly do so. > >>> > >>> (However, I am not sure there are really that many Camel well known > >>> properties at least in the presence.) > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>>> > >>>>>> Hadrian > >>>>>> > >>>>>> > >>>>>> On Jan 26, 2009, at 1:07 PM, William Tam wrote: > >>>>>> > >>>>>>> On Mon, Jan 26, 2009 at 10:35 AM, Hadrian Zbarcea > <hzbar...@gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> This headers business is a bit of a tricky one. I hit it last > year > >>>>>>>> in > >>>>>>>> the > >>>>>>>> context of security. > >>>>>>>> > >>>>>>>> I agree with the view that headers should only exist in the > context > >>>>>>>> of > >>>>>>>> an > >>>>>>>> endpoint. I think outside of that there is no guarantee that the > >>>>>>>> semantics > >>>>>>>> of a header is preserved. I am not sure if headers should be > >>>>>>>> propagated > >>>>>>>> from one endpoint to another at all. > >>>>>>> > >>>>>>> There are certainly use cases that protocol headers DO need to be > >>>>>>> propagated between endpoints. If users want to integrate with > some > >>>>>>> management and/or security tools like Actional, users are required > to > >>>>>>> include custom headers in protocol headers. These custom headers > >>>>>>> travel with messages to allow trust zone enforcement and message > >>>>>>> correlation. They need to be preserved and propagated across hops > >>>>>>> which are potentially over different transport protocols. > >>>>>>> > >>>>>>>> Properties should be used instead. > >>>>>>>> Coming back to security, if http is used for instance there are > >>>>>>>> several > >>>>>>>> ways > >>>>>>>> of handling that. If basic auth is used for instance one gets a > >>>>>>>> user/pass, > >>>>>>>> but that may need to be translated to something else at endpoint > >>>>>>>> boundaries. > >>>>>>>> I don't think that the "Authorization" header should exist > outside > >>>>>>>> the > >>>>>>>> http > >>>>>>>> endpoint for instance. > >>>>>>>> > >>>>>>>> Yes, we do propagate properties today, no issue there. But then > some > >>>>>>>> policies need to be defined per endpoint to deal with known > >>>>>>>> headers/properties, and camel specific properties should be > defined > >>>>>>>> to > >>>>>>>> deal > >>>>>>>> with know headers. Even better, endpoints should set protocol > >>>>>>>> specific > >>>>>>>> headers that are known as required to propagate (such as the auth > >>>>>>>> stuff) > >>>>>>>> as > >>>>>>>> properties from start. > >>>>>>>> > >>>>>>>> My $0.02 > >>>>>>>> Hadrian > >>>>>>>> > >>>>>>>> > >>>>>>>> On Jan 26, 2009, at 9:44 AM, Claus Ibsen wrote: > >>>>>>>> > >>>>>>>>> On Mon, Jan 26, 2009 at 3:37 PM, Roman Kalukiewicz > >>>>>>>>> <roman.kalukiew...@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>> Why don't we talk about exchange properties here? My feeling > here > >>>>>>>>>> is > >>>>>>>>>> that properties should be used as user-headers, while headers > are > >>>>>>>>>> always protocol headers. In fact it works this way right now: > If I > >>>>>>>>>> want to keep some value through the whole flow I put it into > >>>>>>>>>> properties. > >>>>>>>>>> > >>>>>>>>>> By current convention if I put something on a header it is sent > as > >>>>>>>>>> protocol-specific header (JMS property, HTTP header), and out > >>>>>>>>>> headers > >>>>>>>>>> are filled also with protocol headers (JSM properties of out > >>>>>>>>>> emssage, > >>>>>>>>>> HTTP response headers). In this case headers shouldn't be > >>>>>>>>>> propagated, > >>>>>>>>>> as there is no way to distinguish things propagated, from > things > >>>>>>>>>> retrieved. And out headers ARE different than in headers. > >>>>>>>>>> > >>>>>>>>>> It is a matter of naming, but currently headers are (what you > call) > >>>>>>>>>> protocol/system headers, while properties are user-headers > (work as > >>>>>>>>>> variables). Do we really need to extend it further? If someone > mix > >>>>>>>>>> those two concepts then it is problem of documentation, but not > >>>>>>>>>> lack > >>>>>>>>>> of functionality. I would just extend DSL a little to be able > to > >>>>>>>>>> retrieve a property (instead using header()). > >>>>>>>>>> > >>>>>>>>>> What do you think, guys? Maybe we should clearly communicate > what > >>>>>>>>>> things are for and what are the consequences of using one or > >>>>>>>>>> another. > >>>>>>>>>> > >>>>>>>>>> Roman > >>>>>>>>>> > >>>>>>>>>> PS. Pipeline should propagate all headers of course, but I > believe > >>>>>>>>>> an > >>>>>>>>>> endpoint is a place where we shouldn't guarantee that headers > will > >>>>>>>>>> be > >>>>>>>>>> propagated by stating it clearly. > >>>>>>>>> > >>>>>>>>> Properties have just lived in the dark and end users does not > really > >>>>>>>>> know they exists. We have some builder methods to set/get > >>>>>>>>> properties. > >>>>>>>>> I guess we need to document and maybe make sure the Spring DSL > also > >>>>>>>>> has support for accessing properties as well. > >>>>>>>>> > >>>>>>>>> To my knowledge properties is always preserved so I doubt we > have an > >>>>>>>>> issue there. > >>>>>>>>> > >>>>>>>>> So users should just start learn using properties as well :) > >>>>>>>>> However then we have the ProducerTemplate that has one liners > for > >>>>>>>>> sending an Exchange. We dont have a sendBodyAndProperties > method. > >>>>>>>>> But > >>>>>>>>> yet again it has too many methods already. They can just use > >>>>>>>>> send("xxx", Exchange) and have exchange populated with the > >>>>>>>>> properties > >>>>>>>>> of choice. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 2009/1/26 Claus Ibsen <claus.ib...@gmail.com>: > >>>>>>>>>>> > >>>>>>>>>>> On Sat, Jan 24, 2009 at 9:08 PM, William Tam > >>>>>>>>>>> <email.w...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> What we have stored in Headers today in Camel is both: > >>>>>>>>>>>>> - user headers > >>>>>>>>>>>>> - and system headers (added by Camel itself). > >>>>>>>>>>>>> > >>>>>>>>>>>>> I am starting to be more and more convinced that we should > >>>>>>>>>>>>> separate > >>>>>>>>>>>>> the two. > >>>>>>>>>>>>> So any headers that a users has enforced to be set should be > >>>>>>>>>>>>> kept > >>>>>>>>>>>>> in > >>>>>>>>>>>>> one Map and the others that the components set internally > (such > >>>>>>>>>>>>> as > >>>>>>>>>>>>> SQL > >>>>>>>>>>>>> number of rows returned, or whatnot we have, there are many) > in > >>>>>>>>>>>>> another Map. > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> It means that a component would have to look for header in > more > >>>>>>>>>>>> than > >>>>>>>>>>>> one place. Besides, the distinction of user vs system > header is > >>>>>>>>>>>> not > >>>>>>>>>>>> always clear. For example, the operation name header for cxf > >>>>>>>>>>>> endpoint > >>>>>>>>>>>> can be set by user but it is also created by cxf component. > I > >>>>>>>>>>>> am > >>>>>>>>>>>> sure there are many more examples. There is another header > >>>>>>>>>>>> category: > >>>>>>>>>>>> protocol headers. A protocol header is not really a user or > >>>>>>>>>>>> system > >>>>>>>>>>>> header. Protocol headers are header propagated from protocol > >>>>>>>>>>>> like > >>>>>>>>>>>> HTTP, which we do want to preserve in message header. > >>>>>>>>>>>> > >>>>>>>>>>>>> The user headers is always preserved and copied along in the > >>>>>>>>>>>>> routing. > >>>>>>>>>>>>> User can always clear/remove unwanted headers. > >>>>>>>>>>>>> The system headers should be short lived as they are not > really > >>>>>>>>>>>>> useable. So they are "alive" in the next step (process) in > the > >>>>>>>>>>>>> route, > >>>>>>>>>>>>> and when the pipeline invokes next route thereafter these > >>>>>>>>>>>>> information > >>>>>>>>>>>>> is cleared. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Separating these will also make the routing/tracing a bit > easier > >>>>>>>>>>>>> as > >>>>>>>>>>>>> Users can recognize their own headers instead its mixed with > all > >>>>>>>>>>>>> the > >>>>>>>>>>>>> noise the Camel components add. > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> I wonder we can leverage/extend the HeaderFilterStrategy > >>>>>>>>>>>> mechanism. > >>>>>>>>>>>> Currently, it is only used for filtering unwanted headers (in > >>>>>>>>>>>> both > >>>>>>>>>>>> request and response direction) when we propagate headers > between > >>>>>>>>>>>> Camel and external messages (like HTTP). > HeaderFilterStrategy > >>>>>>>>>>>> is > >>>>>>>>>>>> (or > >>>>>>>>>>>> will be) associated with an endpoint. We could make > >>>>>>>>>>>> HeaderFilterStrategy available to the exchange object. So, > when > >>>>>>>>>>>> an > >>>>>>>>>>>> endpoint creates an exchange, the exchange gets a header > filter > >>>>>>>>>>>> strategy. Then, pipeline can do something like this to > filter > >>>>>>>>>>>> unwanted header: message.filterHeaders(). The header filter > >>>>>>>>>>>> strategy > >>>>>>>>>>>> is highly customizable for each endpoint (can have a > component > >>>>>>>>>>>> wide > >>>>>>>>>>>> default) and it can be looked up from registry. > >>>>>>>>>>>> > >>>>>>>>>>> Good pointers William. > >>>>>>>>>>> > >>>>>>>>>>> Yeah we can revist it after you have moved the header filters > to > >>>>>>>>>>> the > >>>>>>>>>>> endpoint. > >>>>>>>>>>> > >>>>>>>>>>> Then we can check up upon how to leverage it as you suggest. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Claus Ibsen > >>>>>>>>>>> Apache Camel Committer > >>>>>>>>>>> > >>>>>>>>>>> Open Source Integration: http://fusesource.com > >>>>>>>>>>> Blog: http://davsclaus.blogspot.com/ > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Claus Ibsen > >>>>>>>>> Apache Camel Committer > >>>>>>>>> > >>>>>>>>> Open Source Integration: http://fusesource.com > >>>>>>>>> Blog: http://davsclaus.blogspot.com/ > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >> > >> > > > > > > -- > Claus Ibsen > Apache Camel Committer > > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/
