Re: Kafka queue full configuration

2017-03-27 Thread Manikumar 
looks like you are referring to scala producer configs.
https://kafka.apache.org/082/documentation.html#producerconfigs

scala producer is deprecated now. It will be removed in future. you can use
java producer
http://kafka.apache.org/documentation/#producerconfigs
buffering related are configs are buffer.memory, block.on.buffer.full

On Mon, Mar 27, 2017 at 3:15 PM, Mohapatra, Sudhir (Nokia - IN/Gurgaon) <
sudhir.mohapa...@nokia.com> wrote:

> Can you please let me know what is the new parameter name for the same
> functionality to simulate the queue full scenario?
>
> Regards,
> Sudhir
>
> From: Mohapatra, Sudhir (Nokia - IN/Gurgaon)
> Sent: Thursday, March 23, 2017 11:01 AM
> To: 'users@kafka.apache.org' <users@kafka.apache.org>; '
> d...@kafka.apache.org' <d...@kafka.apache.org>
> Subject: Kafka queue full configuration
>
> Hi,
> We are trying to simulate the kafka queue full scenarios on kafka 0.10.0.
> I have seen in earlier versions there is a configuration parameter
> "queue.buffering.max.messages" which can be set to simulate the queue full
> scenario.
> But in the kafka 0.10.0 this parameter is not there.
> https://kafka.apache.org/0100/documentation.html
>
> Is this "queue.buffering.max.messages" config parameter has been changed
> in kafka 0.10.0 release?
> Can you please let me know what is the new parameter name for the same
> functionality?
>
> Regards,
> Sudhir
>
>

RE: Kafka queue full configuration

2017-03-27 Thread Mohapatra, Sudhir (Nokia - IN/Gurgaon) 
Can you please let me know what is the new parameter name for the same 
functionality to simulate the queue full scenario?

Regards,
Sudhir

From: Mohapatra, Sudhir (Nokia - IN/Gurgaon)
Sent: Thursday, March 23, 2017 11:01 AM
To: 'users@kafka.apache.org' <users@kafka.apache.org>; 'd...@kafka.apache.org' 
<d...@kafka.apache.org>
Subject: Kafka queue full configuration

Hi,
We are trying to simulate the kafka queue full scenarios on kafka 0.10.0.
I have seen in earlier versions there is a configuration parameter 
"queue.buffering.max.messages" which can be set to simulate the queue full 
scenario.
But in the kafka 0.10.0 this parameter is not there.
https://kafka.apache.org/0100/documentation.html

Is this "queue.buffering.max.messages" config parameter has been changed in 
kafka 0.10.0 release?
Can you please let me know what is the new parameter name for the same 
functionality?

Regards,
Sudhir

Kafka queue full configuration

2017-03-23 Thread Mohapatra, Sudhir (Nokia - IN/Gurgaon) 
Hi,
We are trying to simulate the kafka queue full scenarios on kafka 0.10.0.
I have seen in earlier versions there is a configuration parameter 
"queue.buffering.max.messages" which can be set to simulate the queue full 
scenario.
But in the kafka 0.10.0 this parameter is not there.
https://kafka.apache.org/0100/documentation.html

Is this "queue.buffering.max.messages" config parameter has been changed in 
kafka 0.10.0 release?
Can you please let me know what is the new parameter name for the same 
functionality?

Regards,
Sudhir

Re: Queue Full

2015-10-26 Thread 马哲超 
You can try to set "queue.buffering.max.messages" larger, for example:

confParam=["queue.buffering.max.messages=200",
"batch.num.messages=1000"]

And reference for you:
https://github.com/edenhill/librdkafka/issues/210

2015-10-27 1:19 GMT+08:00 Prabhjot Bharaj :

> Hi,
>
> This is a type of problem where you operating more than network capacity.
> This can be handled at two places (you decide whichever is useful/practical
> for use case)  :-
>
> 1. In case the bottleneck is because of the broker slowness, increase the
> number of partitions of your topic, balance them out to newer nodes (if
> required) and retry. Make sure your producer can write to the new
> partitions as well
> 2. In case this is because your producer is creating too many messages,
> there are 2 ways you can commit them (select the most appropriate according
> to your use case. I'm considering that you dont want to lose your
> messages):-
> --> a. increase the number of producer machines (i.e. balance your
> producer load among more machines)
> --->b. produce at a slower rate (in case your producers are not
> scalable, you are in a problem. you can solve this problem by sacrificing
> latency). Write to your queue in batches and wait for it to get cleared
> before you commit the next batch to your local queue.
>
> Regards,
> Prabcs
>
>
> On Mon, Oct 26, 2015 at 10:41 PM, Eduardo Costa Alfaia <
> e.costaalf...@unibs.it> wrote:
>
> > Hi Magnus
> > I think this answer
> > c) producing messages at a higher rate than the network or broker can
> > handle
> > How could I manager this?
> >
> >
> > > On 26 Oct 2015, at 17:45, Magnus Edenhill  wrote:
> > >
> > > c) producing messages at a higher rate than the network or broker can
> > > handle
> >
> >
> > --
> > Informativa sulla Privacy: http://www.unibs.it/node/8155
> >
>
>
>
> --
> -
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>

Queue Full

2015-10-26 Thread Eduardo Costa Alfaia 
Hi Guys,

How could I solving this problem?

% Failed to produce message: Local: Queue full
% Failed to produce message: Local: Queue full

Thanks

-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Queue Full

2015-10-26 Thread Eduardo Costa Alfaia 
Hi Magnus
I think this answer
c) producing messages at a higher rate than the network or broker can
handle
How could I manager this?


> On 26 Oct 2015, at 17:45, Magnus Edenhill  wrote:
> 
> c) producing messages at a higher rate than the network or broker can
> handle


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Queue Full

2015-10-26 Thread Prabhjot Bharaj 
Hi,

This is a type of problem where you operating more than network capacity.
This can be handled at two places (you decide whichever is useful/practical
for use case)  :-

1. In case the bottleneck is because of the broker slowness, increase the
number of partitions of your topic, balance them out to newer nodes (if
required) and retry. Make sure your producer can write to the new
partitions as well
2. In case this is because your producer is creating too many messages,
there are 2 ways you can commit them (select the most appropriate according
to your use case. I'm considering that you dont want to lose your
messages):-
--> a. increase the number of producer machines (i.e. balance your
producer load among more machines)
--->b. produce at a slower rate (in case your producers are not
scalable, you are in a problem. you can solve this problem by sacrificing
latency). Write to your queue in batches and wait for it to get cleared
before you commit the next batch to your local queue.

Regards,
Prabcs


On Mon, Oct 26, 2015 at 10:41 PM, Eduardo Costa Alfaia <
e.costaalf...@unibs.it> wrote:

> Hi Magnus
> I think this answer
> c) producing messages at a higher rate than the network or broker can
> handle
> How could I manager this?
>
>
> > On 26 Oct 2015, at 17:45, Magnus Edenhill  wrote:
> >
> > c) producing messages at a higher rate than the network or broker can
> > handle
>
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

Re: Queue Full

2015-10-26 Thread Magnus Edenhill 
That's librdkafka's producer queue that has run full because:
 a) client is not connected to broker
 b) no leader broker for relevant partitions
 c) producing messages at a higher rate than the network or broker can
handle


/Magnus




2015-10-26 16:49 GMT+01:00 Eduardo Costa Alfaia <e.costaalf...@unibs.it>:

> Hi Guys,
>
> How could I solving this problem?
>
> % Failed to produce message: Local: Queue full
> % Failed to produce message: Local: Queue full
>
> Thanks
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>