look at
number of rows in Cassandra for a particular key and see if we have that many
messages in Kafka and Snowflake for that key.
Thanks
Rajib
-Original Message-
From: Bill Bejeck
Sent: Tuesday, May 12, 2020 7:41 AM
To: users@kafka.apache.org
Subject: Re: Offset Management
Hi Rajib,
Generally, it's best to let Kafka handle the offset management.
Under normal circumstances, when you restart a consumer, it will start
reading records from the last committed offset, there's no need for you to
manage that process yourself.
If you need manually commit records
Hi, I wanted to know if it is a good practice to develop a custom offset
management method while consuming from Kafka. I am thinking to develop it as
below.
1. Create a PartitionInfo named tuple as below
PartitionInfo("PartitionInfo",["header","custom writer",
Hello Team,
I have few questions on consumer offset management done in latest release,
if you could spend few minutes and answer the below questions - (please
answer based on latest release)
1. Is there any dependency of zookeeper in storing consumer offset ?
2. Can you give some insight on the
I’m currently wondering if it’s possible to use the internal
`__consumer_offsets` topic to manage offsets outside the consumer group APIs.
I’m using the low-level API to manage the consumers but I’d still like to store
offsets in Kafka.
If it’s not possible to publish and fetch offsets from the
We have a requirement that consumer must be able to re-read the messages.
In High level consumer api,it looks like the re-start of consumer needed ,if
offset has to be reset.
The new consumer API seems to be of beta quality.
In stable version ,I guess the only option is to go for simple consu
RDBMS within the same transaction.
Then, on re-balance, we read the offsets from RDBMS.
So far so good. But we can see lot of auto-commit stuff in the log. So, we
don't need the Kafka offset management. Is there a clean way to disable it?
Is simple setting enable.auto.commit =false just e
ere I
> could find more information about Kafka's Offset management? Currently, I'm
> using the
>
> props.put("enable.auto.commit", "true")
> props.put("auto.commit.interval.ms", "1000")
>
> which I guess commits to the Zook
Could anyone point me to some code samples or some documentation on where I
could find more information about Kafka's Offset management? Currently, I'm
using the
props.put("enable.auto.commit", "true")
props.put("auto.commit.interval.ms", "1000&qu
Subject: Re: Regarding The Kafka Offset Management Issue In Direct Stream
Approach.
Questions about Spark-kafka integration are better directed to the Spark
user mailing list.
I'm not 100% sure what you're asking. The spark createDirectStream api
will not store any offsets internally,
Questions about Spark-kafka integration are better directed to the Spark
user mailing list.
I'm not 100% sure what you're asking. The spark createDirectStream api
will not store any offsets internally, unless you enable checkpointing.
On Sun, Nov 1, 2015 at 10:26 PM, Charan Ganga Phani Adabala
Hi All,
We are working in Apache spark with Kafka integration, in this use case we are
using DirectStream approach. we want to avoid the data loss in this approach
for actually we take offsets and saving that offset into MongoDB.
We want some clarification is Spark stores any offsets internally,
Questions about spark's kafka integration should probably be directed to
the spark user mailing list, not this one. I don't monitor kafka mailing
lists as closely, for instance.
For the direct stream, Spark doesn't keep any state regarding offsets,
unless you enable checkpointing. Have you read
Hi All,
We are working in Apache spark with Kafka integration, in this use case we are
using DirectStream approach. we want to avoid the data loss in this approach
for actually we take offsets and saving that offset into MongoDB.
We want some clarification is Spark stores any offsets internally,
This might help :
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
http://www.slideshare.net/jjkoshy/offset-management-in-kafka
Thanks,
Mayuresh
On Thu, Oct 15, 2015 at 5:39 AM, Kiran Singh wrote:
> Hi
>
> I am looking for offset m
Hi
I am looking for offset management in Kafka.
Is there any link which can explain how offset are updated in Kafka once a
consumer read a message.
to implement custom offset management on client
side (instead of using standard API to commit and retrieve offsets, one can
directly talk with custom offset store) but then problem arises that no
commercial or FOSS kafka monitoring solution support it out of the box.
I know I would, but the
mer APIs, then you can get the offsets
> > > either
> > > > from zookeeper or the __consumer_offsets__ topic. On the other hand,
> if
> > > you
> > > > use the simple consumer APIs, you mean to manage offsets yourself,
> then
> > > you
&g
r hand, if
> > you
> > > use the simple consumer APIs, you mean to manage offsets yourself, then
> > you
> > > should monitor them yourself, simple and plain, right?
> > >
> > > 2015-04-22 14:36 GMT+08:00 Stevo Slavić :
> > >
> > > &g
>
> > > Hello Apache Kafka community,
> > >
> > > Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset
> > > management responsibility is mainly client/consumer side
> responsibility.
> > >
> > > Wouldn't it be better
then you
> should monitor them yourself, simple and plain, right?
>
> 2015-04-22 14:36 GMT+08:00 Stevo Slavić :
>
> > Hello Apache Kafka community,
> >
> > Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset
> > management responsibility is mainly cl
consumer APIs, you mean to manage offsets yourself, then you
should monitor them yourself, simple and plain, right?
2015-04-22 14:36 GMT+08:00 Stevo Slavić :
> Hello Apache Kafka community,
>
> Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset
> management responsibili
Hello Apache Kafka community,
Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset
management responsibility is mainly client/consumer side responsibility.
Wouldn't it be better if it was broker side only responsibility?
E.g. now if one wants to use custom offset management
documentation also explain how to migrate from zookeeper offsets to
> > kafka offsets.
> >
> > Gwen
> >
> > On Wed, Apr 8, 2015 at 8:13 AM, Yosi Botzer
> wrote:
> >
> > > Hi,
> > >
> > > I have seen this presentation:
> > > http:
ffsets to
> kafka offsets.
>
> Gwen
>
> On Wed, Apr 8, 2015 at 8:13 AM, Yosi Botzer wrote:
>
> > Hi,
> >
> > I have seen this presentation:
> > http://www.slideshare.net/jjkoshy/offset-management-in-kafka
> >
> > describing a new approach for o
I have seen this presentation:
> > http://www.slideshare.net/jjkoshy/offset-management-in-kafka
> >
> > describing a new approach for offset management.
> >
> > I wanted to know from which kafka version this option will be available
> and
> > will it be available by default
> >
> > Thanks
> > Yosi
> >
>
ets.
Gwen
On Wed, Apr 8, 2015 at 8:13 AM, Yosi Botzer wrote:
> Hi,
>
> I have seen this presentation:
> http://www.slideshare.net/jjkoshy/offset-management-in-kafka
>
> describing a new approach for offset management.
>
> I wanted to know from which kafka version this opti
Hi,
I have seen this presentation:
http://www.slideshare.net/jjkoshy/offset-management-in-kafka
describing a new approach for offset management.
I wanted to know from which kafka version this option will be available and
will it be available by default
Thanks
Yosi
> 1) Does Offset Commit/Fetch API works with Simple Consumer ?
Yes - in 0.8.2.1. There is an example given at
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
> 2) With MM, can you selectively MM offset topic.
>
> Use Case:
> Let's suppose you
Hi Joel Koshy,
I have following questions:
1) Does Offset Commit/Fetch API works with Simple Consumer ?
Use Case: Camus dual commit offset on hdfs as well as to Kafka offset
topic. It will be only committed once when FILE Is committed to HDFS (Not
like auto commit)
2) With MM, can you selecti
Hi list,
I was under the impression that consumers still needed to interact with
zookeeper to track their offset. Going through recent Jiras to track the
progress I see that https://issues.apache.org/jira/browse/KAFKA-1000 and
https://issues.apache.org/jira/browse/KAFKA-1012 seem to indicate that
We store offsets in INT64, so you can go as high as:
9,223,372,036,854,775,807
messages per topic-partition before looping around :)
Gwen
On Fri, Feb 20, 2015 at 12:21 AM, Clement Dussieux | AT Internet <
clement.dussi...@atinternet.com> wrote:
> Hi,
>
>
> I am using Kafka_2.9.2-0.8.2 and play a
Hi,
I am using Kafka_2.9.2-0.8.2 and play a bit with offsets in my code.
I would like to know how is implemented the offset system for message posting.
The main question here is: for every message posted, it gets an offset greater
that the previous one, meaning that message1 gets offset x and m
Yes it is supported in 0.8.2-beta. It is documented on the site - you
will need to set offsets.storage to kafka.
On Thu, Feb 19, 2015 at 03:57:31PM -0500, Matthew Butt wrote:
> I'm having a hard time figuring out if the new Kafka-based offset
> management in the high-level Scala
I'm having a hard time figuring out if the new Kafka-based offset
management in the high-level Scala Consumer is implemented in the current
version of 0.8.2-beta. If I implement a high-level consumer, will it use
the new system, or will it still be storing in zookeeper? Do I need to wait
fo
On Thu, Feb 05, 2015 at 11:57:15PM -0800, Joel Koshy wrote:
> On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote:
> > I'm not sure what you mean by 'default' behavior 'only if' offset.storage
> > is kafka. Does that mean the 'default' behavior is 'false' if
> > offset.storage is 'zoo
On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote:
> I'm not sure what you mean by 'default' behavior 'only if' offset.storage
> is kafka. Does that mean the 'default' behavior is 'false' if
> offset.storage is 'zookeeper'? Can that be clarified in the config
> documentation section
On Thu, Feb 5, 2015 at 9:52 PM, Joel Koshy wrote:
> > Ok, so it looks like the default settings are:
> > offset.storage = zookeeper
> > dual.commit.enabled = true
> > The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
> > state) that it will only apply if offset.storage = kafka
> Ok, so it looks like the default settings are:
> offset.storage = zookeeper
> dual.commit.enabled = true
> The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
> state) that it will only apply if offset.storage = kafka. Is that right?
> (I'm guessing not)
dual.commit.enabled de
Ok, so it looks like the default settings are:
offset.storage = zookeeper
dual.commit.enabled = true
The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
state) that it will only apply if offset.storage = kafka. Is that right?
(I'm guessing not)
"*If you are using "kafka"* as o
This is documented in the official docs:
http://kafka.apache.org/documentation.html#distributionimpl
On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> What are the defaults for those settings (I assume it will be to continue
> using only zookeeper by default)?
>
> Also, if I hav
What are the defaults for those settings (I assume it will be to continue
using only zookeeper by default)?
Also, if I have a cluster of consumers sharing the same groupId, and I
update them via a rolling release, will it be a problem during the rolling
restart if there is inconsistency in the set
Thanks Jon. I updated the FAQ with your procedure:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
?
On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
jbringhu...@linkedin.com.invalid> wrote:
> There should probably be
This is what I've found so far.
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
The high-level consumer just worked for me by setting offsets.storage =
kafka.
Scroll down to the offsets.* config params.
http://kafka.apache.org/documentation.ht
There should probably be a wiki page started for this so we have the details in
one place. The same question was asked on Freenode IRC a few minutes ago. :)
A summary of the migration procedure is:
1) Upgrade your brokers and set dual.commit.enabled=false and
offsets.storage=zookeeper (Commit o
Hi,
For 0.8.2, one of the features listed is:
- Kafka-based offset storage.
Is there documentation on this (I've heard discussion of it of course)?
Also, is it something that will be used by existing consumers when they
migrate up to 0.8.2? What is the migration process?
Thanks,
Jason
There isn't much difference btw option 1 and 2 in terms of the offset
commit overhead to Zookeeper. In 0.8.2, we will have a Kafka-based offset
management, which is much more efficient than committing to Zookeeper.
Thanks,
Jun
On Tue, Jan 6, 2015 at 10:45 AM, Rafi Shamim wrote:
> Hell
t; On Wed, Jan 7, 2015 at 1:04 PM, Jay Kreps wrote:
> > > >
> > > > > Hey guys,
> > > > >
> > > > > We need to take the versioning of the protocol seriously. People are
> > > > > definitely using the offset commit functionality
> >
> > > On Wed, Jan 7, 2015 at 1:04 PM, Jay Kreps wrote:
> > >
> > > > Hey guys,
> > > >
> > > > We need to take the versioning of the protocol seriously. People are
> > > > definitely using the offset commit functionality in 0.8.
of the protocol seriously. People are
> > > definitely using the offset commit functionality in 0.8.1 and I really
> > > think we should treat this as a bug and revert the change to version 0.
> > >
> > > -Jay
> > >
> > > On Wed, Jan 7, 2015
t; > think we should treat this as a bug and revert the change to version 0.
> >
> > -Jay
> >
> > On Wed, Jan 7, 2015 at 9:24 AM, Jun Rao wrote:
> >
> > > Yes, we did make an incompatible change in OffsetCommitRequest in
> 0.8.2,
> > > which is a
an 7, 2015 at 9:24 AM, Jun Rao wrote:
>
> > Yes, we did make an incompatible change in OffsetCommitRequest in 0.8.2,
> > which is a mistake. The incompatible change was introduced in KAFKA-1012
> in
> > Mar, 2014 when we added the kafka-based offset management support.
>
ake an incompatible change in OffsetCommitRequest in 0.8.2,
> which is a mistake. The incompatible change was introduced in KAFKA-1012 in
> Mar, 2014 when we added the kafka-based offset management support. However,
> we didn't realize that this breaks the wire protocol until much later
Yes, we did make an incompatible change in OffsetCommitRequest in 0.8.2,
which is a mistake. The incompatible change was introduced in KAFKA-1012 in
Mar, 2014 when we added the kafka-based offset management support. However,
we didn't realize that this breaks the wire protocol until much
Hello,
I would like to write a multi-threaded consumer for the high-level
consumer in Kafka 0.8.1. I have found two ways that seem feasible
while keeping the guarantee that messages in a partition are processed
in order. I would appreciate any feedback this list has.
Option 1
- Create mu
estions about the current status and future
> of
> > the
> > >> >>> Kafka
> > >> >>> > > > consumers.
> > >> >>> > > >
> > >> >>> > > > We have been working to adding Kafka support in Spring XD
> [1],
gt; using the high level consumer via Spring Integration Kafka
> [2].
> >> We
> >> >>> are
> >> >>> > > > working on adding features such as:
> >> >>> > > > - the ability to control offsets/replay topics;
&g
>> consumers;
>> >>> > > >
>> >>> > > > We are currently at version 0.8.1.1, so using the simple consumer
>> > is
>> >>> a
>> >>> > > > pretty straightforward choice right now. However, in the light of
&g
; 1) With respect to the consumer redesign for 0.9, what is the
> > future
> >>> of
> >>> > > the
> >>> > > > Simple Consumer and High Level Consumer? To my best
> understanding,
> >>> the
> >>> > > > ex
gt; > the
>>> > > > Simple Consumer and High Level Consumer? To my best understanding,
>>> the
>>> > > > existing high level consumer API will be deprecated in favour of
> the
>>> new
>>> > > > consumer API. What
viding the building blocks for the new consumer, or will
>> it be
>> > > > deprecated as well?
>> > >
>> > > The new consumer will subsume both use-cases (simple and high-level).
>> > > You can still use the old SimpleConsumer if you wish - i.e., the wire
; > > > [3] and providing the building blocks for the new consumer, or will
>> it be
>> > > > deprecated as well?
>> > >
>> > > The new consumer will subsume both use-cases (simple and high-level).
>> > > You can still use the ol
t; >
> > > The new consumer will subsume both use-cases (simple and high-level).
> > > You can still use the old SimpleConsumer if you wish - i.e., the wire
> > > protocol for fetch and other requests will still be supported.
> > >
> > > >
> &g
ll be supported.
> >
> > >
> > > 2) Regarding the new consumer: the v0.8.2 codebase contains an early
> > > implementation of it, but since this a feature scheduled only for 0.9,
> > what
> > > is its status as well? Is it included only as a fu
d only for 0.9,
> what
> > is its status as well? Is it included only as a future reference and for
> > stabilizing the API?
>
> It is a WIP so you cannot really use it.
>
> > 3) Obviously, offset management is a concern if using the simple
> consumer,
> > so - w
w consumer: the v0.8.2 codebase contains an early
> implementation of it, but since this a feature scheduled only for 0.9, what
> is its status as well? Is it included only as a future reference and for
> stabilizing the API?
It is a WIP so you cannot really use it.
> 3) Obviously, off
scheduled only for 0.9, what
is its status as well? Is it included only as a future reference and for
stabilizing the API?
3) Obviously, offset management is a concern if using the simple consumer,
so - wondering about the Offset Management API as well. The Kafka protocol
document specifically indicates
Yes take a look at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon'twantmyconsumer'soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer'soffsets
?
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
ht
Hi All
Can we manage partition-offset commits separately instead of a
consumerConnector.commitOffsets() (commits offsets of all the related
broker partitions simultaneously) call ?
69 matches
Mail list logo
