While I was connected to console-consumer.sh, I posted a few messages to a
Kafka topic, one message at a time, across a few hours.
I'd post a message, see it arrive in console-consumer, a few mins later I'd
post the next message, and so on.
They all arrived in order.
However, when I now try to v
topic, you need to create
a topic with a single partition.
On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar wrote:
> While I was connected to console-consumer.sh, I posted a few messages to a
> Kafka topic, one message at a time, across a few hours.
>
> I'd post a message, see it ar
Heya,
I need to send a group of messages, which are all related, and then process
those messages, only when all of them have arrived.
Here is how I'm planning to do this. Is this the right way, and can any
improvements be made to this?
1) Send a message to a topic called batch_start, with a batc
Heya,
Normally, you add your topics and their callbacks to a StreamBuilder, and
then call KafkaStreams.start() to start ingesting those topics.
Is it possible to add a new topic to the StreamBuilder, and start ingesting
that as well, after KafkaStreams.start() has been called?
Thanks.
was defined with a regular expression, i.e,
> kafka.stream(Pattern.compile("foo-.*");
>
> If any new topics are added after start that match the pattern, then they
> will also be consumed.
>
> Thanks,
> Damian
>
> On Fri, 2 Dec 2016 at 13:13 Ali Akhtar wrote:
more details:
> http://docs.confluent.io/current/streams/architecture.html
>
>
> -Matthias
>
> On 12/2/16 6:23 AM, Ali Akhtar wrote:
> > That's pretty useful to know - thanks.
> >
> > 1) If I listened too foo-.*, and there were 5 foo topics created after
> >
er batch, and you would be very close to the essence of the above
> proposal.
>
> Thanks,
> Apurva
>
> On Fri, Dec 2, 2016 at 5:02 AM, Ali Akhtar wrote:
>
> > Heya,
> >
> > I need to send a group of messages, which are all related, and then
> process
> &
l be no overhead.
>
> -Matthias
>
> On 12/2/16 3:58 PM, Ali Akhtar wrote:
> > Hey Matthias,
> >
> > So I have a scenario where I need to batch a group of messages together.
> >
> > I'm considering creating a new topic for each batch that arrives, i.e
> &g
uot;, "TOPIC-TO-BE-DELETED"});
> > TopicCommand.deleteTopic(zkUtils, commandOptions);
>
> So you can delete a topic within your Streams app.
>
> -Matthias
>
>
>
> On 12/2/16 9:25 PM, Ali Akhtar wrote:
> > Is there a way to delete the processed top
:
> I guess yes. You might only want to make sure the topic offsets got
> committed -- not sure if committing offsets of a deleted topic could
> cause issue (ie, crashing you Streams app)
>
> -Matthias
>
> On 12/2/16 11:04 PM, Ali Akhtar wrote:
> > Thank you very much
uestion is, what would happen if the JVM goes down
> before you delete the topic.
>
>
> -Matthias
>
> On 12/3/16 2:07 AM, Ali Akhtar wrote:
> > Is there a way to make sure the offsets got committed? Perhaps, after the
> > last msg has been consumed, I can setup a task to
s
> that batch is going to have?
>
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Sat
You need to also delete / restart zookeeper, its probably storing the
topics there. (Or yeah, just enable it and then delete the topic)
On Fri, Dec 9, 2016 at 9:09 PM, Rodrigo Sandoval wrote:
> Why did you do all those things instead of just setting
> delete.topic.enable=true?
>
> On Dec 9, 2016
@Damian,
In the Java equivalent of this, does each KStream / KStreamBuilder.stream()
invocation create its own topic group, i.e its own thread?
On Mon, Dec 12, 2016 at 10:29 PM, Damian Guy wrote:
> Yep - that looks correct
>
> On Mon, 12 Dec 2016 at 17:18 Avi Flax wrote:
>
> >
> > > On Dec 12,
- I'm receiving a batch of messages to a Kafka topic.
Each message has a timestamp, however the messages can arrive / get
processed out of order. I.e event 1's timestamp could've been a few seconds
before event 2, and event 2 could still get processed before event 1.
- I know the number of messag
a problem
> you can reread data from Kafka.
>
> -Jesse
>
> > On Dec 21, 2016, at 7:24 PM, Ali Akhtar wrote:
> >
> > - I'm receiving a batch of messages to a Kafka topic.
> >
> > Each message has a timestamp, however the messages can arrive / get
> p
wish for. Of course this won't work if your updates occur
> in different hosts.
> Also maybe Kafka streams can help shard the based on item Id to a second
> topic
> On Thu, 22 Dec 2016 at 4:31 Ali Akhtar wrote:
>
> > The batch size can be large, so in memory ordering isn&#
ent to the same partition.
>
> > On Dec 26, 2016, at 23:32, Ali Akhtar wrote:
> >
> > How would I route the messages to a specific partition?
> >
> >> On 27 Dec 2016 10:25 a.m., "Asaf Mesika" wrote:
> >>
> >> There is a much easier ap
101 - 118 of 118 matches
Mail list logo
