how to write kafka connect hdfs parquet sink.

Hi, I have read confluent kafka connect hdfs but I don't want to use schema registry from confluent. I have produced avro encoded bytes to kafka, at that time, I have written my own avro serializer, not used

Re: Kafka Streams: Merging of partial results

Hi Michael-Keith, Good question. Two answers: in the default case the same key (e.g., "world") would end up in the same partition, so you wouldn't have the example you describe here where the same key is in two different partitions of the same topic. E.g., this default case applies if you are

Re: release of 0.10.1

We basically need the regex(java-util regex) support for specifying source topics. On 7/23/16, 7:41 PM, "Ewen Cheslack-Postava" wrote: 0.10.1.0 is considered a major release. The release 0.10.0.0 might have a follow up 0.10.0.1 for critical bug fixes, but 0.10.1.0

Re: release of 0.10.1

So this begs the question -- why doesn't Kafka join the mainstream and have a 1.0 release? The 0.x versioning makes one think that this code is, at best, beta. Chris From: Ewen Cheslack-Postava To: "users@kafka.apache.org" Date:

Re: Kafka on DC/OS

Hi Abhimanyu, What errors are you seeing? And which version of DCOS are you running as well? Tim On Fri, Jul 22, 2016 at 6:14 AM, Chakrabarty, Abhimanyu wrote: > I had a question regarding Kafka on DC/OS because whenever we try to install > the Kafka package

Re: Maximum number of producers per topic per broker

Just found out what's was causing this. Which is quite dangerous. The additional 100 or 200 server had the wrong ssl certificate on them causing it to creat some kind of DOS on the broker. That was the reason why I was seeing that the thread dump was more pointing at the Ssl side. Sent from

Rebalances despite frequent heartbeats

I'm trying to track down an issue with one of our consumers. There are 4 threads in the same consumer group, which will run happily for a few hours before one of them crashes with the following exception: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to

Kafka Streams: Merging of partial results

Hello Kafka Users, I've been floating this question around the #apache-kafka IRC channel on Freenode for the last week or two and I still haven't reached a satisfying answer. The basic question is: How does Kafka Steams merge partial results? So let me expand on that a bit... Consider the

Re: "Powered By" - Kafka in iPinYou

The incremental data every day is more than 80T and the overall data processed every data is around 30P. On Thu, Jul 21, 2016 at 4:13 PM, Glen Cao wrote: > iPinYou (www.ipinyou.com.cn/?defaultLocale=en > ) is the largest DSP in China which has its HQ in Beijing and offices

"Powered By" - Kafka in iPinYou

iPinYou (www.ipinyou.com.cn/?defaultLocale=en) is the largest DSP in China which has its HQ in Beijing and offices in Shanghai, Guangzhou, Silicon Valley and Seattle. Kafka clusters are the central data hub in iPinYou. All kinds of Internet display advertising data, such as bid/no-bid,

MirrorMaker replication...

I need a solution that will mirror a topic to a remote location that avoids cycles and a topic per region issue. This solution would require EXACTLY two topics regardless of how many regions (datacenters) it replicates to. I would like the initial producer (P1) to write to only a local (L1)

Re:Kafka on DC/OS

I had a question regarding Kafka on DC/OS because whenever we try to install the Kafka package it always shows that it is deploying in the marathon UI and gives us an error when we search for it using "dcos kafka".We have also tried using "dcos package install kafka" and tried to add the

Increasing kafka broker throughput

Hello, Some numbers - There is a single topic with 3 producers, 4 brokers, 80 partitions and replication factor of 2. Average produce rate is 45 MB/sec (cumulative) and max is at around 250 MB/sec (cumulative), the producer goes out of memory during peak times (buffer memory set to 10G, batch

I had a problem of kerberos??I try to solve for a long time can not be resolved??Please help me??

HI, My Frends: Sorry,Just send pictures don't show.I change to text. I come from China?? I had a problem, I try to solve for a long time can not be resolved. I also found to a similar article, but did not answer.

Re: Topic naming convention and common message envelope.

Thanks Ewen! Do you have an example of your namings and maybe how your envelope looks? Just curious. > On 24 июля 2016 г., at 3:41, Ewen Cheslack-Postava wrote: > > On Tue, Jul 19, 2016 at 12:48 AM, Denis Mikhaylov > wrote: > >> Hi, I plan to use Kafka

Re: Kafka Replication Factor

Hello Avi, Yes, replication factor of 1 means just one lead replica gets assigned for the topic partitions. To achieve HA of messages there are more things to watch out, not only replication factor - metadata/ZK HA, producer acks, min-in-sync replicas, rack awareness for replica assignment to

Kafka Replication Factor

Hi All Just double checking If i have created a topic with replication factor 1 does it mean i dont have replication if one of my disks crashes do i loose data? Thanks Avi