Re: Add Nginx in front of Kafka cluster?

parties = ports * On Thu, Sep 14, 2017 at 8:04 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > I would try to put the SSL on different ports than what you're sending > kafka to. Make sure the kafka ports don't do anything except communicate in > plaintext, put all 3rd parties on di

Re: Add Nginx in front of Kafka cluster?

is where the > other Kafka broker listens on. It's an [info] message so I'm not sure how > serious it is, but I don't see messages sent from filebeat in Kafka. :( > > Thanks! > -Yongtao > > On Thursday, September 14, 2017, 8:31:31 PM GMT+8, Ali Akhtar < > ali.rac...@gmail

Re: Add Nginx in front of Kafka cluster?

t, but the Nginx > requires SSL. But I could be wrong. > > > Thanks! > > -Yongtao > > > On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar < > ali.rac...@gmail.com> wrote: > > > How do you know that the brokers don't talk to each other? > > On Th

Re: Kafka with kubernetes

Not too familiar with that error, but I do have Kafka working on Kubernetes. I'll share my files here in case that helps: Zookeeper: https://gist.github.com/aliakhtar/812974c35cf2658022fca55cc83f4b1d Kafka: https://gist.github.com/aliakhtar/724fbee6910dec7263ab70332386af33 Essentially I have 3

Re: Kafka behavior when consuming a topic which doesn't exist?

Oh gotcha, thanks. So a topic will be created if topic creation is enabled. On Sat, Jul 8, 2017 at 8:14 PM, M. Manna <manme...@gmail.com> wrote: > Please check my previous email. > > On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar <ali.rac...@gmail.com> wrote: > > >

Re: Kafka behavior when consuming a topic which doesn't exist?

t-does-not-exist On 7 Jul 2017 9:46 pm, "Ali Akhtar" <ali.rac...@gmail.com> wrote: > Sometimes I see warnings in my logs if i create a consumer for a topic > which doesn't exist. Such as: > > org.apache.kafka.clients.NetworkClient - Error while fetching metad

Kafka behavior when consuming a topic which doesn't exist?

Sometimes I see warnings in my logs if i create a consumer for a topic which doesn't exist. Such as: org.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} If later messages are posted to that topic (which will create

Re: Caching in Kafka Streams to ignore garbage message

oin to lookup the IDs. > > > -Matthias > > On 4/27/17 2:22 PM, Ali Akhtar wrote: > > I have a Kafka topic which will receive a large amount of data. > > > > This data has an 'id' field. I need to look up the id in an external db, > > see if we are tracking that id

Caching in Kafka Streams to ignore garbage message

I have a Kafka topic which will receive a large amount of data. This data has an 'id' field. I need to look up the id in an external db, see if we are tracking that id, and if yes, we process that message, if not, we ignore it. 99% of the data will be for ids which are not being tracked - 1% or

Calculating time elapsed using event start / stop notification messages

I have a tricky use case where a user initiates an event (by clicking a button) and then stops it (by clicking it again, losing connection, closing the browser, etc). Each time the event starts or stops, a notification is sent to a kafka topic, with the user's id as the message key and the

Leader not available error after kafka node goes down

I have a 3 node kafka cluster which is being managed via kubernetes, in docker containers. Recently, one of the 3 nodes went down, and was automatically re-created by kubernetes. However, now whenever I try to consume from one of my Kafka topics, thru Kafka Streaming, i get the error: >6687

Re: Out of order message processing with Kafka Streams

Hans, Which class's javadocs should i look at? From my initial look at the javadocs and discussion with Michael, it doesn't seem possible. On Tue, Mar 21, 2017 at 10:44 PM, Hans Jespersen <h...@confluent.io> wrote: > Yes, and yes! > > -hans > > > > > On Mar 21

Re: Out of order message processing with Kafka Streams

; >> Ali, > >> > >> what you describe is (roughly!) how Kafka Streams implements the > internal > >> state stores to support windowing. > >> > >> Some users have been following a similar approach as you outlined, using > >> the P

Re: Out of order message processing with Kafka Streams

roach as you outlined, using > the Processor API. > > > > On Mon, Mar 20, 2017 at 7:54 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > It would be helpful to know the 'start' and 'end' of the current > metadata, > > so if an out of order message arrives lat

Re: Out of order message processing with Kafka Streams

-arriving and out-of-order data is only treated specially for > > > windowed > > > > aggregations. > > > > > > > > For stateless operations such as `KStream#foreach()` or > > `KStream#map()`, > > > > records are processed in the order they arrive (per parti

Re: Processing multiple topics

Are you saying, that it should process all messages from topic 1, then topic 2, then topic 3, then 4? Or that they need to be processed exactly at the same time? On Mon, Mar 20, 2017 at 10:05 PM, Manasa Danda wrote: > Hi, > > I am Manasa, currently working on a project

Re: Out of order message processing with Kafka Streams

s only treated specially for > windowed > > aggregations. > > > > For stateless operations such as `KStream#foreach()` or `KStream#map()`, > > records are processed in the order they arrive (per partition). > > > > -Michael > > > > > > > > > >

Re: Out of order message processing with Kafka Streams

ndow as if message B were consumed in the timestamp order > >> before message A. The end result of this flow is that you eventually get > >> the same results you would get in a batch processing system but with the > >> added benefit of getting intermediary result at much lower latency. &

Re: Out of order message processing with Kafka Streams

> > added benefit of getting intermediary result at much lower latency. > > > > -hans > > > > /** > > * Hans Jespersen, Principal Systems Engineer, Confluent Inc. > > * h...@confluent.io (650)924-2670 <(650)%20924-2670> > > */ > > > &g

Out of order message processing with Kafka Streams

Is it possible to have Kafka Streams order messages correctly by their timestamps, even if they arrived out of order? E.g, say Message A with a timestamp of 5:00 PM and Message B with a timestamp of 5:15 PM, are sent. Message B arrives sooner than Message A, due to network issues. Is it

Re: Processing time series data in order

messages with the same key > get sent to the same partition. > > > On Dec 26, 2016, at 23:32, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > > How would I route the messages to a specific partition? > > > >> On 27 Dec 2016 10:25 a.m., "Asaf Mesik

Re: Processing time series data in order

om Cassandra, in case of a problem > you can reread data from Kafka. > > -Jesse > > > On Dec 21, 2016, at 7:24 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > > - I'm receiving a batch of messages to a Kafka topic. > > > > Each message has a timestamp, ho

Processing time series data in order

- I'm receiving a batch of messages to a Kafka topic. Each message has a timestamp, however the messages can arrive / get processed out of order. I.e event 1's timestamp could've been a few seconds before event 2, and event 2 could still get processed before event 1. - I know the number of

Re: [Streams] Threading Frustration

@Damian, In the Java equivalent of this, does each KStream / KStreamBuilder.stream() invocation create its own topic group, i.e its own thread? On Mon, Dec 12, 2016 at 10:29 PM, Damian Guy wrote: > Yep - that looks correct > > On Mon, 12 Dec 2016 at 17:18 Avi Flax

Re: Deleting a topic without delete.topic.enable=true?

You need to also delete / restart zookeeper, its probably storing the topics there. (Or yeah, just enable it and then delete the topic) On Fri, Dec 9, 2016 at 9:09 PM, Rodrigo Sandoval wrote: > Why did you do all those things instead of just setting >

Re: Tracking when a batch of messages has arrived?

isplay/KAFKA/KIP- > > > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging > > > > > > If you can include the 'batch_id' inside your messages, and define > custom > > > control messages with a control topic, then you would not need one > topic > >

Re: Adding topics to KafkaStreams after ingestion has been started?

minutes sounds ok. Question is, what would happen if the JVM goes down > before you delete the topic. > > > -Matthias > > On 12/3/16 2:07 AM, Ali Akhtar wrote: > > Is there a way to make sure the offsets got committed? Perhaps, after the > > last msg has been consu

Re: Adding topics to KafkaStreams after ingestion has been started?

tth...@confluent.io> wrote: > I guess yes. You might only want to make sure the topic offsets got > committed -- not sure if committing offsets of a deleted topic could > cause issue (ie, crashing you Streams app) > > -Matthias > > On 12/2/16 11:04 PM, Ali Akhtar wrote: >

Re: Adding topics to KafkaStreams after ingestion has been started?

> > "--topic", "TOPIC-TO-BE-DELETED"}); > > TopicCommand.deleteTopic(zkUtils, commandOptions); > > So you can delete a topic within your Streams app. > > -Matthias > > > > On 12/2/16 9:25 PM, Ali Akhtar wrote: > > Is there a way to del

Re: Adding topics to KafkaStreams after ingestion has been started?

ou delete completely processed topics, there will be no overhead. > > -Matthias > > On 12/2/16 3:58 PM, Ali Akhtar wrote: > > Hey Matthias, > > > > So I have a scenario where I need to batch a group of messages together. > > > > I'm considering creating a new topic for each b

Re: Tracking when a batch of messages has arrived?

ed one topic > per batch, and you would be very close to the essence of the above > proposal. > > Thanks, > Apurva > > On Fri, Dec 2, 2016 at 5:02 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > Heya, > > > > I need to send a group of messages, which are a

Re: Adding topics to KafkaStreams after ingestion has been started?

a look here for more details: > http://docs.confluent.io/current/streams/architecture.html > > > -Matthias > > On 12/2/16 6:23 AM, Ali Akhtar wrote: > > That's pretty useful to know - thanks. > > > > 1) If I listened too foo-.*, and there were 5 foo topics cr

Re: Adding topics to KafkaStreams after ingestion has been started?

the > original stream was defined with a regular expression, i.e, > kafka.stream(Pattern.compile("foo-.*"); > > If any new topics are added after start that match the pattern, then they > will also be consumed. > > Thanks, > Damian > > On Fri, 2 Dec 2016 at 13

Adding topics to KafkaStreams after ingestion has been started?

Heya, Normally, you add your topics and their callbacks to a StreamBuilder, and then call KafkaStreams.start() to start ingesting those topics. Is it possible to add a new topic to the StreamBuilder, and start ingesting that as well, after KafkaStreams.start() has been called? Thanks.

Tracking when a batch of messages has arrived?

Heya, I need to send a group of messages, which are all related, and then process those messages, only when all of them have arrived. Here is how I'm planning to do this. Is this the right way, and can any improvements be made to this? 1) Send a message to a topic called batch_start, with a

Re: Message order different each time stream is replayed?

al ordering over the entire topic, you need to create a topic with a single partition. On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > While I was connected to console-consumer.sh, I posted a few messages to a > Kafka topic, one message at a time, acro

Message order different each time stream is replayed?

While I was connected to console-consumer.sh, I posted a few messages to a Kafka topic, one message at a time, across a few hours. I'd post a message, see it arrive in console-consumer, a few mins later I'd post the next message, and so on. They all arrived in order. However, when I now try to

Re: Unit tests (in Java) take a *very* long time to 'clean up'?

(t == Thread.currentThread()) return; t.stop(); }); On Fri, Nov 11, 2016 at 9:52 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Oh, so it seems like there's no easy way to just Thread.stop() without > changing the internal kafka / zk code? :( > > Perhaps

Re: Unit tests (in Java) take a *very* long time to 'clean up'?

ZK and calls org.apache.kafka.streams.integration.utils.KafkaEmbedded > to start Kafka). > So these are embedded in the sense that it's not another process, just > threads within the main streams test process. > > Thanks > Eno > > > On 11 Nov 2016, at 16:26, Ali Akhtar <ali.rac...@gmail.com> wrote: > &

Re: Unit tests (in Java) take a *very* long time to 'clean up'?

For me, the startup doesn't take anywhere near as long as shutdown does. On Fri, Nov 11, 2016 at 9:37 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Unless I'm missing anything, there's no reason why these throwaway > processes should be shutdown gracefully. Just kill them as soon

Re: Unit tests (in Java) take a *very* long time to 'clean up'?

Unless I'm missing anything, there's no reason why these throwaway processes should be shutdown gracefully. Just kill them as soon as the test finishes. On Fri, Nov 11, 2016 at 9:26 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Hey Eno, > > Thanks for the quick reply. >

Re: Unit tests (in Java) take a *very* long time to 'clean up'?

just > annoying and it will go away. > > Thanks, > Eno > > > > On 11 Nov 2016, at 14:28, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > > I have some unit tests in which I create an embedded single broker kafka > > cluster, using : > > > > Embe

Unit tests (in Java) take a *very* long time to 'clean up'?

I have some unit tests in which I create an embedded single broker kafka cluster, using : EmbeddedSingleNodeKafkaCluster.java from https://github.com/confluentinc/examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/EmbeddedSingleNodeKafkaCluster.java That class

Re: Kafka UTF 8 encoding problem

Its probably not UTF-8 if it contains Turkish characters. That's why base64 encoding / decoding it might help. On Wed, Nov 9, 2016 at 4:22 PM, Radoslaw Gruchalski wrote: > Are you sure your string is in utf-8 in the first place? > What if you pass your string via something

Re: Kafka UTF 8 encoding problem

I would recommend base64 encoding the message on the producer side, and decoding it on the consumer side. On Wed, Nov 9, 2016 at 3:40 PM, Baris Akgun (Garanti Teknoloji) < barisa...@garanti.com.tr> wrote: > Hi All, > > We are using Kafka 0,9.0.0 and we want to send our messages to topic in >

Re: [VOTE] Add REST Server to Apache Kafka

And this will make adding health checks via Kubernetes easy. On Wed, Oct 26, 2016 at 4:12 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > +1. I hope there will be a corresponding Java library for doing admin > functionality. > > On Wed, Oct 26, 2016 at 4:10 AM, Jungtaek Lim

Re: [VOTE] Add REST Server to Apache Kafka

+1. I hope there will be a corresponding Java library for doing admin functionality. On Wed, Oct 26, 2016 at 4:10 AM, Jungtaek Lim wrote: > +1 > > > On Wed, 26 Oct 2016 at 8:00 AM craig w wrote: > > > -1 > > > > On Tuesday, October 25, 2016, Sriram

Re: Removing kafka topic contents from Java

There isn't a java API for this, you'd have to mess around with bash scripts which I haven't found to be worth it. Just let the data expire and get deleted. Set a short expiry time for the topic if necessary. On Mon, Oct 24, 2016 at 6:30 PM, Demian Calcaprina wrote: >

Re: How to block tests of Kafka Streams until messages processed?

aid. In > your situation, options like these are what you'd currently need to do > since you are writing directly from your Kafka Stream app to Cassandra, > rather than writing from your app to Kafka and then using Kafka Connect to > ingest into Cassandra. > > > > On Wed, Oct 19

Re: How to block tests of Kafka Streams until messages processed?

auz...@surescripts.com > wrote: > For similar queue related tests we put the check in a loop. Check every > second until either the result is found or a timeout happens. > > -Dave > > -Original Message- > From: Ali Akhtar [mailto:ali.rac...@gmail.com] > Sent: Wednesday, Octobe

Re: Embedded Kafka Cluster - Maven artifact?

Please change that. On Thu, Oct 20, 2016 at 1:53 AM, Eno Thereska <eno.there...@gmail.com> wrote: > I'm afraid we haven't released this as a maven artefact yet :( > > Eno > > > On 18 Oct 2016, at 13:22, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > >

How to block tests of Kafka Streams until messages processed?

I'm using Kafka Streams, and I'm attempting to write integration tests for a stream processor. The processor listens to a topic, processes incoming messages, and writes some data to Cassandra tables. I'm attempting to write a test which produces some test data, and then checks whether or not the

Embedded Kafka Cluster - Maven artifact?

Is there a maven artifact that can be used to create instances of EmbeddedSingleNodeKafkaCluster for unit / integration tests?

Re: Understanding out of order message processing w/ Streaming

t; computing the sum of numbers, then you do not need to anything special. > > > > > > On Wed, Oct 12, 2016 at 10:22 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > Thanks Matthias. > > > > So, if I'm understanding this right, Kafka will not discard

Re: Understanding out of order message processing w/ Streaming

the > timestamp of the current record. > > Does this makes sense? > > To fix you issue, you could add a .transformValue() before you KTable, > which allows you to access the timestamp of a record. If you add this > timestamp to you value and pass it to KTable afterwards, you can > ac

Re: How can I delete a topic programatically?

The last time I tried, I couldn't find a way to do it, other than to trigger the bash script for topic deletion programatically. On Wed, Oct 12, 2016 at 9:18 AM, Ratha v wrote: > Hi all; > > I have two topics(source and target). I do some processing on the message >

Re: Understanding out of order message processing w/ Streaming

P.S, does my scenario require using windows, or can it be achieved using just KTable? On Wed, Oct 12, 2016 at 8:56 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Heya, > > Say I'm building a live auction site, with different products. Different > users will bid on different

Re: In Kafka Streaming, Serdes should use Optionals

annot be simply > ignored; you can, though, apply a simple filter such like "filter((key, > value) => value != null)" before your processor lambda operator, if it > looks clearer in your code. > > Guozhang > > > On Sun, Oct 9, 2016 at 3:14 PM, Ali Akhtar &l

Understanding out of order message processing w/ Streaming

Heya, Say I'm building a live auction site, with different products. Different users will bid on different products. And each time they do, I want to update the product's price, so it should always have the latest price in place. Example: Person 1 bids $3 on Product A, and Person 2 bids $5 on

Safely deleting all existing topics?

In development, I often need to delete all existing data in all topics, and start over. My process for this currently is: stop zookeeper, stop kafka broker, rm -rf ~/kafka/data/* But when I bring the broker back on, it often prints a bunch of errors and needs to be restarted before it actually

KTable and KStream should share an interface

They both have a lot of the same methods, and yet they can't be used polymorphically because they don't share the same parent interface. I think KIterable or something like that should be used as their base interface w/ shared methods.

Re: Kafka null keys - OK or a problem?

So.. it should be okay to have null keys, I'm guessing. On Mon, Oct 10, 2016 at 11:51 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Hey Michael, > > We're using this one: https://github.com/Shopify/sarama > > Any ideas how that one works? > > On Mon, Oct 10, 2016 a

Re: Kafka null keys - OK or a problem?

eys > (https://github.com/confluentinc/confluent-kafka-go/). > > If you decide to generate keys and you want even spread, a random > number generator is probably your best bet. > > Gwen > > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: >

Kafka null keys - OK or a problem?

A kafka producer written elsewhere that I'm using, which uses the Go kafka driver, is sending messages where the key is null. Is this OK - or will this cause issues due to partitioning not happening correctly? What would be a good way to generate keys in this case, to ensure even partition

Re: In Kafka Streaming, Serdes should use Optionals

ecking null looks > fine to me. > > > Guozhang > > On Fri, Oct 7, 2016 at 3:12 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > Hey G, > > > > Looks like the only difference is a valueSerde parameter. > > > > How does that prevent having to look for

Re: Deleting a message after all consumer have consumed it

Also, you can set a retention period and have messages get auto deleted after a certain time (default 1 week) On Sat, Oct 8, 2016 at 3:21 AM, Hans Jespersen wrote: > Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging > rethought as a distributed commit log”.

Re: In Kafka Streaming, Serdes should use Optionals

@confluent.io> wrote: > > > Ali, the Apache Kafka project still targets Java 7, which means we can't > > use Java 8 features just yet. > > > > FYI: There's on ongoing conversation about when Kafka would move from > Java > > 7 to Java 8. > > > > On Fri, Oct 7

In Kafka Streaming, Serdes should use Optionals

Since we're using Java 8 in most cases anyway, Serdes / Serialiazers should use options, to avoid having to deal with the lovely nulls.

Re: Printing to stdin from KStreams?

ssible (yet), Ali. I am also not aware of any > plan to add such a feature in the short-term. > > > > On Fri, Oct 7, 2016 at 1:36 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > Is it possible to have kafka-streams-reset be automatically called during > > devel

Re: Printing to stdin from KStreams?

://docs.confluent.io/3.0.1/streams/developer-guide.html#application- > > reset-tool > > > > and > > http://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resett > > ing-a-streams-application/ > > > > > > About the timestamp issue: it seems that you

Re: Printing to stdin from KStreams?

gt; > Just use #print() or #writeAsText() > > > - -Matthias > > On 10/6/16 6:25 PM, Ali Akhtar wrote: > > What the subject says. For dev, it would be a lot easier if > > debugging info can be printed to stdin instead of another topic, > > where it will persist. > > &g

Printing to stdin from KStreams?

What the subject says. For dev, it would be a lot easier if debugging info can be printed to stdin instead of another topic, where it will persist. Any ideas if this is possible?

Re: Handling out of order messages without KTables

a custom TimestampExtractor that > returns the JSON embedded TS instead of record TS (as > DefaultTimestampExtractor does) > > See > http://docs.confluent.io/3.0.1/streams/developer-guide.html#timestamp-ex > tractor > > > - -Matthias > > On 10/6/16 2:59 PM, Ali Ak

Re: Handling out of order messages without KTables

'last one' can be determined using the timestamp in the json of the message) On Fri, Oct 7, 2016 at 2:54 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Thanks for the reply. > > Its not possible to provide keys, unfortunately. (Producer is written by a > colleague, and said co

Re: Handling out of order messages without KTables

ing the same product, go to > the same instance. You can ensure this, by given all records of the > same product the same key and "groupByKey" before processing the data. > > - -Matthias > > On 10/6/16 10:55 AM, Ali Akhtar wrote: > > Thank you, State Store seems promising

Re: Handling out of order messages without KTables

> > Thus, you can attach a state store to your processor and compare the > timestamps of the current record with the timestamp of the one in your > store. > > - -Matthias > > On 10/6/16 8:52 AM, Ali Akhtar wrote: > > Heya, > > > > I have some Kafka producers, whic

Handling out of order messages without KTables

Heya, I have some Kafka producers, which are listening to webhook events, and for each webhook event, they post its payload to a Kafka topic. Each payload contains a timestamp from the webhook source. This timestamp is the source of truth about which events happened first, which happened last,

Re: Kafka Streams dynamic partitioning

> It's often a good idea to over-partition your topics. For example, even if today 10 machines (and thus 10 partitions) would be sufficient, pick a higher number of partitions (say, 50) so you have some wiggle room to add more machines (11...50) later if need be. If you create e.g 30 partitions,

Re: difficulty to delete a topic because of its syntax

I don't see a space in that topic name On Wed, Oct 5, 2016 at 6:42 PM, Hamza HACHANI wrote: > Hi, > > I created a topic called device-connection-invert-key-value-the > metric-changelog. > > I insit that there is a space in it. > > > > Now that i want to delete it

Re: Snazzy new look to our website

Just noticed this on pulling up the documentation. Oh yeah! This new look is fantastic. On Wed, Oct 5, 2016 at 4:31 AM, Vahid S Hashemian wrote: > +1 > > Thank you for the much needed new design. > At first glance, it looks great, and more professional. > > --Vahid >

Re: Kafka Streams - Parallel by default or 1 thread per topic?

<3 On Wed, Oct 5, 2016 at 2:31 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > That's awesome. Thanks. > > On Wed, Oct 5, 2016 at 2:19 AM, Matthias J. Sax <matth...@confluent.io> > wrote: > >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA512 >> &g

Re: Kafka Streams - Parallel by default or 1 thread per topic?

consumer group fashion, ie, are > distributed over the running instances. > > Please see here for more details > http://docs.confluent.io/current/streams/architecture.html#parallelism-m > odel > > > - -Matthias > > On 10/4/16 1:27 PM, Ali Akhtar wrote: > > I need to co

Kafka Streams - Parallel by default or 1 thread per topic?

I need to consume a large number of topics, and handle each topic in a different way. I was thinking about creating a different KStream for each topic, and doing KStream.foreach for each stream, to process incoming messages. However, its unclear if this will be handled in a parallel way by

Re: New Idea for Kafka multiple consumers running parallel.

You may be able to control the starting offset, but if you try to control which instance gets offset 4.. you'll lose all benefits of parallelism. On 4 Oct 2016 3:02 pm, "Kaushil Rambhia/ MUM/CORP/ ENGINEERING" < kaushi...@pepperfry.com> wrote: > Hi guys, > i am using apache kafka with phprd

Re: offset topics growing huge

Newbie question, but what exactly does log.cleaner.enable=true do, and how do I know if I need to set it to be true? Also, if config changes like that need to be made once a cluster is up and running, what's the recommended way to do that? Do you killall -12 kafka and then make the change, and

Re: Architecture recommendations for a tricky use case

Avi, Why did you choose Druid over Postgres / Cassandra / Elasticsearch? On Fri, Sep 30, 2016 at 1:09 AM, Avi Flax <avi.f...@parkassist.com> wrote: > > > On Sep 29, 2016, at 09:54, Ali Akhtar <ali.rac...@gmail.com> wrote: > > > > I'd appreciate som

Re: Architecture recommendations for a tricky use case

gt; On Thu, Sep 29, 2016 at 10:40 AM, Deepak Sharma <deepakmc...@gmail.com> > wrote: > > If you use spark direct streams , it ensure end to end guarantee for > > messages. > > > > > > On Thu, Sep 29, 2016 at 9:05 PM, Ali Akhtar <ali.rac...@gmail.com> >

Re: Architecture recommendations for a tricky use case

o handle etl failures? Do you care about lost / > >> duplicated data? Are your writes idempotent? > >> > >> Absent any other information about the problem, I'd stay away from > >> cassandra/flume/hdfs/hbase/whatever, and use a spark direct stream > >> fee

Re: Architecture recommendations for a tricky use case

rom kafka. > > On Thu, Sep 29, 2016 at 8:24 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > >> I don't think I need a different speed storage and batch storage. Just >> taking in raw data from Kafka, standardizing, and storing it somewhere >> where the web UI

Re: Architecture recommendations for a tricky use case

m > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any

Re: Architecture recommendations for a tricky use case

nd all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destructio

Re: Architecture recommendations for a tricky use case

It needs to be able to scale to a very large amount of data, yes. On Thu, Sep 29, 2016 at 7:00 PM, Deepak Sharma <deepakmc...@gmail.com> wrote: > What is the message inflow ? > If it's really high , definitely spark will be of great use . > > Thanks > Deepak > >

Architecture recommendations for a tricky use case

I have a somewhat tricky use case, and I'm looking for ideas. I have 5-6 Kafka producers, reading various APIs, and writing their raw data into Kafka. I need to: - Do ETL on the data, and standardize it. - Store the standardized data somewhere (HBase / Cassandra / Raw HDFS / ElasticSearch /

Re: Zookeeper hostname/ip change

Perhaps if you add 1 node, take down existing node, etc? On Sun, Sep 25, 2016 at 10:37 PM, brenfield111 wrote: > I need to change the hostnames and ips for the Zookeeper ensemble > serving my Kafka cluster. > > Will Kafka carry on as usual, along with it's existing ZK

Re: java.nio.channels.ClosedChannelException in console-consumer.sh

other using the private IPs. Shouldn't that be enough? I don't want to expose kafka publicly. On Fri, Sep 16, 2016 at 10:48 PM, Ali Akhtar <ali.rac...@gmail.com> wrote: > I've created a 3 broker kafka cluster, changing only the config values for > broker id, log.dirs, and zookeeper con

java.nio.channels.ClosedChannelException in console-consumer.sh

I've created a 3 broker kafka cluster, changing only the config values for broker id, log.dirs, and zookeeper connect. I left the remaining fields as default. The broker ids are 1, 2, 3. I opened the port 9092 on AWS. I then created a topic 'test' with replication factor of 2, and 3 partitions.

Re: which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?

Examine server.properties and see which port you're using in there On Thu, Sep 15, 2016 at 3:52 PM, kant kodali wrote: > which port should I use 9091 or 9092 or 2181 to send messages through kafka > when using a client Library? > I start kafka as follows: > sudo

Re: Publish to 1 topic, consume from N

It sounds like you can implement the 'mapping service' component yourself using Kafka. Have all of your messages go to one kafka topic. Have one consumer group listening to this 'everything goes here' topic. This consumer group acts as your mapping service. It looks at each message, and based on

Re: hi

ent as possible. > Again the big question is What is the right setup for Kafka to be > comparable > with the other I mentioned in my previous email? > > > > > > > On Thu, Sep 15, 2016 1:47 AM, Ali Akhtar ali.rac...@gmail.com > wrote: > The issue is clearly that you'r

Re: hi

> and > receive throughput was 2K messages/secI am unable to push 300K messages > with > Kafka with the above configuration and environment so at this point my > biggest > question is what is the fair setup for Kafka so its comparable with NATS > and > NSQ? > kant > > > &g

  1   2   >