Configuring custom Kafka producer/consumer properties on Camel Kafka component/endpoint?

[Erik Boye]

Hi,

Is there any way to configure custom Kafka producer/consumer properties on the 
Camel Kafka component/endpoint (without "breaking in")?
I specifically want to configure the Confluent Kafka property with key 
'value.subject.name.strategy': 
https://docs.confluent.io/current/schema-registry/serializer-formatter.html

Other Camel components (i.e Camel ActiveMQ; destination.consumer) are made 
extensible by providing users a way to set "additional properties". Why does 
not Camel Kafka provide that? Until schemaRegistryURL was added as a 
configurable property, we had to make our own Camel Kafka Component to make it 
work ("breaking in"). There should definitely be a way to set additional/custom 
properties on Kafka producers/consumers.

Just have a look at the Camel Kafka documentation and do a text search for 
Confluent: https://camel.apache.org/components/latest/kafka-component.html ;-)
Having "native" support for schemas on Kafka records is definitely a good idea, 
and I am pretty sure Apache Kafka will move in that direction at some point

Erik

"This email with attachments is solely for the use of the individual or entity 
to which it is addressed. It may contain confidential or privileged 
information. If you are not the addressee, please notify the sender and delete 
this message and all attachments from your files."

Re: [Camel-K]: It cannot start due to "exec: \"kamel\": executable file not found in $PATH"

[Luca Morandini]

> Hi,
> I've updated the master to use M3-SNAPSHOT, since we've cut a release of M2
> last saturday.

That did the trick, thanks a lot.

Cheers,

Luca Morandini

netty tcp consumer not reading large payload

[Rohan Emmanuel]

Hi,
I have problem reading large payload using the netty TCP component. The
payload is getting read in chunks, can you please guide me as to how can I
read the entire payload

-- 
Regards,
Rohan Emmanuel

Re: netty tcp consumer not reading large payload

[Omar Al-Safi]

I think there is some few information here, like how did you use it, how
does your route looks like ..etc. Also did you take a look at netty example
here
https://github.com/apache/camel/tree/master/examples/camel-example-netty-http
?

On Tue, 8 Oct 2019 at 13:49, Rohan Emmanuel 
wrote:

> Hi,
> I have problem reading large payload using the netty TCP component. The
> payload is getting read in chunks, can you please guide me as to how can I
> read the entire payload
>
> --
> Regards,
> Rohan Emmanuel
>

Re: too many objects

[Mantas Gridinas]

What is the actual route definition you're using? I would try swapping
out http4 component for regular http component.

On Mon, Oct 7, 2019 at 2:34 PM arshid  wrote:
>
>
>
> Hi all,
>
>
>
> I am reading a csv file around 300 gb and then splitting it and parsing it to 
> json and sending it to rest api using http4 provider.
>
> The problem whenever I use http4 provider it creates lots of objects of type 
> TempfileManager. Which results in slow down after running for few hours.
>
> I have attached the screenshot of jvisualvm
>
>
>
> Can anyone help me understand in what is happening
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>

Re: netty tcp consumer not reading large payload

[Rohan Emmanuel]

hi,
i am using camel as a standalone java application. This application is to
read input using TCP when client connects to it.
attaching the route below, i see that after reading initial 1024 bytes, the
route flow proceeded to other procesors with partail data. i want to be
able to read entire message and then proceed further in the route . can you
please advice.
Also adding extract of DEBUG logs for your reference..



  
 


..


DEBUG LOG:
 NettyEventExecutorGroup] DEBUG
org.apache.camel.component.netty4.NettyConsumer - Channel: [id: 0x54bae73e,
L:/15.213.85.104:9993 - R:/15.213.82.141:3039] received body:
UnpooledUnsafeNoCleanerDirectByteBuf(ridx: 0, widx: 1024, cap: 1024)
reading body
the bytebuf is direct
16:36:44.654 [Camel (camel-1) thread #3 - NettyEventExecutorGroup] INFO
 consumer - consumer route called .  .POST
..
..
16:36:44.684 [Camel (camel-1) thread #3 - NettyEventExecutorGroup] DEBUG
org.apache.camel.component.netty4.NettyConsumer - Channel: [id: 0x54bae73e,
L:/15.213.85.104:9993 - R:/15.213.82.141:3039] received body:
UnpooledUnsafeNoCleanerDirectByteBuf(ridx: 0, widx: 16384, cap: 16384)
reading body
the bytebuf is direct
16:36:44.827 [Camel (camel-1) thread #3 - NettyEventExecutorGroup] INFO
 consumer - consumer route called .  .POST . .

On Tue, Oct 8, 2019 at 5:38 PM Omar Al-Safi  wrote:

> I think there is some few information here, like how did you use it, how
> does your route looks like ..etc. Also did you take a look at netty example
> here
>
> https://github.com/apache/camel/tree/master/examples/camel-example-netty-http
> ?
>
> On Tue, 8 Oct 2019 at 13:49, Rohan Emmanuel 
> wrote:
>
> > Hi,
> > I have problem reading large payload using the netty TCP component. The
> > payload is getting read in chunks, can you please guide me as to how can
> I
> > read the entire payload
> >
> > --
> > Regards,
> > Rohan Emmanuel
> >
>


-- 
Regards,
Rohan Emmanuel

RE: netty tcp consumer not reading large payload [General Use]

[Walke, Martin (Leonardo, UK)]

Hi,

The example you probably want is:

https://github.com/apache/camel/tree/master/examples/camel-example-netty-custom-correlation

It's a good example of what's required. Don't get side-tracked by netty-http 
examples as they're not what you want for a TCP stream.

Does your message have an 'end-of-message' byte value? You need to create a 
decoder factory class to put the incoming stream into an effective delimited 
message .

Regards
Martin

From: Rohan Emmanuel [via Camel] [mailto:ml+s465427n5844754...@n5.nabble.com]
Sent: 08 October 2019 12:59
To: Walke, Martin (Leonardo, UK)
Subject: netty tcp consumer not reading large payload


*** WARNING ***

This message has originated outside your organisation,
  either from an external partner or the Global Internet.
  Keep this in mind if you answer this message.
Hi,
I have problem reading large payload using the netty TCP component. The
payload is getting read in chunks, can you please guide me as to how can I
read the entire payload

--
Regards,
Rohan Emmanuel


If you reply to this email, your message will be added to the discussion below:
https://camel.465427.n5.nabble.com/netty-tcp-consumer-not-reading-large-payload-tp5844754.html
To unsubscribe from Camel - Users, click 
here.
NAML
Leonardo MW Ltd
Registered Office: Sigma House, Christopher Martin Road, Basildon, Essex SS14 
3EL
A company registered in England & Wales.  Company no. 02426132

This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.

Re: netty tcp consumer not reading large payload

[Claus Ibsen]

Hi

You need to configure what codec you use so camel-netty knows what the
data protocol is, eg what is the condition for a message to start/end
etc.
Is it text based or binary etc. And if its text based what are the
terminating characters (new line or something else).

AFAIR the default is java object protocol which are very often not
what you want to use.

On Tue, Oct 8, 2019 at 4:02 PM Rohan Emmanuel
 wrote:
>
> hi,
> i am using camel as a standalone java application. This application is to
> read input using TCP when client connects to it.
> attaching the route below, i see that after reading initial 1024 bytes, the
> route flow proceeded to other procesors with partail data. i want to be
> able to read entire message and then proceed further in the route . can you
> please advice.
> Also adding extract of DEBUG logs for your reference..
>
>
>  uri="netty4:tcp://:9993?allowDefaultCodec=false&exchangePattern=InOut"/>
>   
>  
> 
> 
> ..
>
>
> DEBUG LOG:
>  NettyEventExecutorGroup] DEBUG
> org.apache.camel.component.netty4.NettyConsumer - Channel: [id: 0x54bae73e,
> L:/15.213.85.104:9993 - R:/15.213.82.141:3039] received body:
> UnpooledUnsafeNoCleanerDirectByteBuf(ridx: 0, widx: 1024, cap: 1024)
> reading body
> the bytebuf is direct
> 16:36:44.654 [Camel (camel-1) thread #3 - NettyEventExecutorGroup] INFO
>  consumer - consumer route called .  .POST
> ..
> ..
> 16:36:44.684 [Camel (camel-1) thread #3 - NettyEventExecutorGroup] DEBUG
> org.apache.camel.component.netty4.NettyConsumer - Channel: [id: 0x54bae73e,
> L:/15.213.85.104:9993 - R:/15.213.82.141:3039] received body:
> UnpooledUnsafeNoCleanerDirectByteBuf(ridx: 0, widx: 16384, cap: 16384)
> reading body
> the bytebuf is direct
> 16:36:44.827 [Camel (camel-1) thread #3 - NettyEventExecutorGroup] INFO
>  consumer - consumer route called .  .POST . .
>
> On Tue, Oct 8, 2019 at 5:38 PM Omar Al-Safi  wrote:
>
> > I think there is some few information here, like how did you use it, how
> > does your route looks like ..etc. Also did you take a look at netty example
> > here
> >
> > https://github.com/apache/camel/tree/master/examples/camel-example-netty-http
> > ?
> >
> > On Tue, 8 Oct 2019 at 13:49, Rohan Emmanuel 
> > wrote:
> >
> > > Hi,
> > > I have problem reading large payload using the netty TCP component. The
> > > payload is getting read in chunks, can you please guide me as to how can
> > I
> > > read the entire payload
> > >
> > > --
> > > Regards,
> > > Rohan Emmanuel
> > >
> >
>
>
> --
> Regards,
> Rohan Emmanuel



-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

How to set dynamic value to routeid

[Imran Raza Khan]

i tried following and its not working

  from("direct:test")
.routeId("${body.topicName}")

  from("direct:test")
.routeId(simple("${body.topicName}").toString())

Above both didnt work

Re: How to set dynamic value to routeid

[Claus Ibsen]

Hi

You cannot set a dynamic route id, its assigned once during starting the route.


On Tue, Oct 8, 2019 at 7:15 PM Imran Raza Khan  wrote:
>
> i tried following and its not working
>
>   from("direct:test")
> .routeId("${body.topicName}")
>
>   from("direct:test")
> .routeId(simple("${body.topicName}").toString())
>
> Above both didnt work



-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: too many objects

[Willem Jiang]

The TempfileManager is used in the CachedOutputStream.java[1].
If you can provide your route, we can see if disable the stream
cache[2] have any side effects on your route.

[1]https://github.com/apache/camel/blob/master/core/camel-support/src/main/java/org/apache/camel/converter/stream/CachedOutputStream.java
[2]https://camel.apache.org/manual/latest/stream-caching.html

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Mon, Oct 7, 2019 at 7:34 PM arshid  wrote:
>
>
>
> Hi all,
>
>
>
> I am reading a csv file around 300 gb and then splitting it and parsing it to 
> json and sending it to rest api using http4 provider.
>
> The problem whenever I use http4 provider it creates lots of objects of type 
> TempfileManager. Which results in slow down after running for few hours.
>
> I have attached the screenshot of jvisualvm
>
>
>
> Can anyone help me understand in what is happening
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>