More like getting a feel from the community about using lz4 for compression?
Has anyone used in the kafka setup.
I am aware that gzip and snappy are more older implementation and regressed.
Given that lz4 has better compression/decompression cycles (though slightly
less compression ratio), was t
The topic level metadata is available with every broker in kafka and also in
zookeepers. Kafka producer API uses list of brokers while the high level
kafka consumer (scala) uses the zookeeper host list.
Is there a reason for this two different behavior. And would it suffice to use
the zookeep
I am in the process of testing and migrating our prod kafka from 0.8.1.1 to
0.8.2.1.
Wanted to do a quick check with the community if anyone has observed any issue
with writing/reading data to 0.8.2.1 kafka broker(s), using 0.8.1.1 producer
and consumer.
Any gotchas to watch for or any concerns?
e.org" ; Virendra Pratap Singh
Sent: Tuesday, March 3, 2015 4:29 PM
Subject: Re: reassign a topic partition which has no ISR and leader set to -1
I hate bringing bad news, but...
You can't really reassign replicas if the leader is not available.
Since the leader is gone, the replica
Ran into a situation where both the leader and replica nodes for a few
partitions of a given topic went down. So now these partitions have no in-sync
replicas and neither any leader (leader set to -1).
I tried to reassign these partitions to a different set of brokers using
partition reassignmen
Is there a supported C/C++ kafka client API?
Regards,
Virendra
No worries, its working. My deployment was flawed and it was running
0.8.0, while I was under the impression that its running 0.8.1.1.
After going to proper 0.8.1.1, the parameter started working properly.
On 7/10/14, 2:18 PM, "Virendra Pratap Singh" wrote:
>Has someon
(kafka.utils.VerifiableProperties)
[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
(kafka.utils.VerifiableProperties)
Virendra
On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
wrote:
>Thank you for pointing out to JIRA.
>
>On 7/9/14, 6:38 PM, "Jun Rao&qu
Thank you for pointing out to JIRA.
On 7/9/14, 6:38 PM, "Jun Rao" wrote:
>This is being worked on in
>https://issues.apache.org/jira/browse/KAFKA-1325
>
>Thanks,
>
>Jun
>
>
>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.
10 min retention, you need to
>configure log rolling such that log segments are rolled at least every 10
>mins.
>
>Thanks,
>
>Jun
>
>
>On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> That's correct. Th
;
>On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> By setting this property
>> log.retention.mins=10
>> in the server.properties file, which is passed as argument when starting
>> the broker.
>>
>>
t;Guozhang
>
>
>On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
>> the other 0.8.1.1. Should the retention of topics for partitions
>> o
A-918
>
>if you are running 0.8.0 then it will not be recognized.
>
>Guozhang
>
>
>
>On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
>>
Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
Directly using the tar ball, no local build used).
I have set the log.retention.minutes=10. However the broker is not
honoring the setting. I see its not cleanin
gt;
>Guozhang
>
>
>On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I am aware of lack of programmatic way of deleting topics in kafka
>>0.8.0.
>> So using the sledge hammer approach.
>> This is wha
uster, it will recreate the topics once the brokers
>come up.
>
>Thanks,
>Neha
>
>
>On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I am aware of lack of programmatic way of deleting topics in kafka
>>0
I am aware of lack of programmatic way of deleting topics in kafka 0.8.0. So
using the sledge hammer approach.
This is what I am doing:
1. Bring whole of my kafka cluster down.
2. Delete all the content on all the kafka clusters pointed via logs.dir
setting.
3. Delete the topic metadata from zoo
.1.1, which is the latest stable release.
>
>Thanks,
>Neha
>
>
>On Tue, Jun 24, 2014 at 4:44 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> Have a kafka cluster with 10 brokers (kafka 0.8.0). All of the brokers
>> were setup upf
Multiple URLS can be
>
> given to allow fail-over.
>
>Command must include exactly one action: --generate, --execute or --verify
>
>/***
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.st
Have a kafka cluster with 10 brokers (kafka 0.8.0). All of the brokers were
setup upfront. None was added later. Default number of partition is set to 4
and default replication to 2.
Have 3 topics in the system. None of these topics are manually created upfront,
when the cluster is setup. So re
in the existing partition is far more
>then the sync rate between earlier and newly reassigned partitions.
>
>This is probably due to a bug. Partition reassignment is stable in
>0.8.1.1.
>I recommend you upgrade and retry.
>
>Thanks
>Neha
>
>
>On Mon, Jun 23, 2014 a
I attempted bin/kafka-reassign-partitions.sh for the topics in my cluster. It
was automatic reassignment of topics to new brokers.
Kafka duly initiated the partition reassignment. I believe kafka controller
will do the new leader election once the re-assigned partitions are in sync.
However, in
Have a scenario and wanted to understand how Kafka will behave. Will be
trying this out but wanted to check if anyone else has already done this.
Running kafka 0.8.0. The way we launch kafka server is via invoking
kafka.Kafka in one of a separate thread of our daemon java service
process.
Kafka
23 matches
Mail list logo
