Re: Kafka as a data ingest

> However, I'm trying to figure out if I can use Kafka to read Hadoop file. The question is a bit unclear as to whether you mean "use Kafka to send data to a Hadoop file" or "use Kafka to read a Hadoop file into a Kafka topic". But in both cases, Kafka Connect provides a good option. The more

Re: Taking a long time to roll a new log segment (~1 min)

I can't speak to the exact details of why fds would be kept open longer in that specific case, but are you aware that the recommendation for production clusters for open fd limits is much higher? It's been suggested to be 100,000 as a starting point for quite awhile:

Re: Kafka as a data ingest

If you want to know if "kafka" can read hadoop files, then no. But you can write your own producer that reads from hdfs any which way and pushes to kafka. We use kafka as the ingestion pipeline's main queue. Read from various sources and push everything to kafka. On Tue, Jan 10, 2017 at 6:26 AM,

Re: Kafka as a data ingest

Can you explain in more detail? Do you want to have files created in hdfs somehow broken into records and put into Kafka? > On Jan 9, 2017, at 19:57, Cas Apanowicz wrote: > > Hi, > > I have general understanding of main Kafka functionality as a streaming tool. >

Kafka as a data ingest

Hi, I have general understanding of main Kafka functionality as a streaming tool. However, I'm trying to figure out if I can use Kafka to read Hadoop file. Can you please advise? Thanks Cas

Re: Json to JDBC using Kafka JDBC connector Sink

unsubscribe On Mon, Jan 9, 2017 at 6:14 PM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi, > > I’m wondering if the following is feasible… > I have a json document with pretty much 0 schema. The only thing I know for > sure is that it’s a json document. > My goal is to pipe that

Taking a long time to roll a new log segment (~1 min)

Hey! I've run into something concerning in our production clusterI believe I've posted this question to the mailing list previously ( http://mail-archives.apache.org/mod_mbox/kafka-users/201609.mbox/browser) but the problem has become considerably more serious. We've been fighting issues

static replication quota support?

Is it currently possible to set a replication quota statically? It seems like its only possible to set it per broker dynamically (via {leader/follower}.replication.throttled.rate), in addition to having to configure it on the topic level. It seems like this is something that I should be able to

Re: kafka streams consumer partition assignment is uneven

I meant I have 7 topics and each has 12 partitions. Considering that I have 4 streaming threads per node, I was expecting to see each thread process 1 partition from each topics and 7 partitions total per streaming thread. But that’s not the case. Or perhaps you are saying the number of

Re: kafka streams consumer partition assignment is uneven

What does the processing topology of your Kafka Streams application look like, and what's the exact topic and partition configuration? You say you have 12 partitions in your cluster, presumably across 7 topics -- that means that most topics have just a single partition. Depending on your

kafka streams consumer partition assignment is uneven

Hi, I have 3 kafka brokers, each with 4 disks. I have 12 partitions. I have 3 kafka streams nodes. Each is configured to have 4 streaming threads. My topology is quite complex and I have 7 topics and lots of joins and states. What I have noticed is that each of the 3 kafka streams nodes gets

Re: Under-replicated Partitions while rolling Kafka nodes in AWS

On Thu, Jan 5, 2017 at 4:32 PM James Cheng wrote: > > FYI, zookeeper 3.4.8 fixes the issue where you have to restart zookeeper > nodes when their DNS mapping changes. I'm not sure how it affects > restarting kafka though, when the zookeeper DNS changes. > >

a question about kafka connect

Hi,I have a question。When using connect-distributed, I start some connectors those push the data to ES from kafka. But there are many errors in the log，why are these errors and how to solve the problem？ Thinks！ [2017-01-09 16:24:30,645] INFO Sink task

Re: Re: Problem with processor API partition assignments

Hi Brian, That is correct. Thanks, Damian On Mon, 9 Jan 2017 at 14:19 Brian Krahmer wrote: > Hi Damian, >I started to pull together enough code to show my topology, and have > a feeling I see what the problem is. Is it correct that when > configuring a processor

Re: Re: Problem with processor API partition assignments

Hi Damian, I started to pull together enough code to show my topology, and have a feeling I see what the problem is. Is it correct that when configuring a processor that it uses the sources applied to that processor to ensure the partitions are aligned for that task? thanks, brian On

Re: Apache Kafka integration using Apache Camel

Can you enable DEBUG logs ? It'll be helpful to debug. -- Kamal On Mon, Jan 9, 2017 at 5:37 AM, Gupta, Swati wrote: > Hello All, > > Any help on this would be appreciated. > There seems to be no error. Does it look like a version issue? > > I have updated my pom.xml with