Hi All,
I have a serialized avro binary data represented by byte[] where one of the
field is long with a logical type of TimeStamp.
Timestamp tsp = new Timestamp(1530228588182l);
Schema schema = SchemaBuilder.builder()
.record("hello")
.fields()
ask whether this should also be tuned or not.
> >
> >
> > Regards,
> > Abhimanyu
> >
> >
> >
> >
> > On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth...@gmail.com> wrote:
> >
> >> @Abhimanyu Why do you think you need to set
017 03:25, "Abhimanyu Nagrath" <abhimanyunagr...@gmail.com>
> > wrote:
> >
> >> Hi Kant,
> >>
> >> Thanks for the suggestion.
> >>
> >>
> >> Regards,
> >> Abhimanyu
> >>
> >> On Sun, May 21, 2017 a
@Abhimanyu You can try setting offset.retention = 30 (log.retention). At
most, you will have a storage overhead of 5 million msgs per day * 30
(days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a
TB of hard disk)
On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka
at 2:19 AM, kant kodali <kanth...@gmail.com> wrote:
> What is your average message size and network speed?
>
> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> abhimanyunagr...@gmail.com> wrote:
>
>> Hi Girish,
>>
>> I did not set any value for
What is your average message size and network speed?
On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
abhimanyunagr...@gmail.com> wrote:
> Hi Girish,
>
> I did not set any value for offsets.retention.minutes so therefore what I
> think is picking its default value i.e 1440 minutes so what do
gt; https://www.confluent.io/blog/how-to-choose-the-number-of-
> topicspartitions-in-a-kafka-cluster/
>
> -Sameer.
>
> On Tue, May 16, 2017 at 2:40 PM, kant kodali <kanth...@gmail.com> wrote:
>
> > Forgot to mention: The question in this thread is for one node which has
Forgot to mention: The question in this thread is for one node which has 8
CPU's 16GB RAM & 500GB hard disk space.
On Tue, May 16, 2017 at 2:06 AM, kant kodali <kanth...@gmail.com> wrote:
> Hi All,
>
> 1. I was wondering if anyone has seen or heard or able to create 1M or
Hi All,
1. I was wondering if anyone has seen or heard or able to create 1M or 10M
or 100M or 1B partitions in a topic? I understand lot of this depends on
filesystem limitations (we are using ext4) and the OS limitations but I
just would like to know what is the scale one had seen in production?
Hi All,
I have read enough blogs from Confluent and others and also books that
tried to talk about the differences between the two and while it is great
to know those differences I hardly find them any useful when it comes to
decision making process of which one to pick since I don't see the
, Matthias J. Sax <matth...@confluent.io>
wrote:
> Yes, you can do that.
>
> -Matthias
>
>
>
> On 3/30/17 6:09 PM, kant kodali wrote:
> > Hi All,
> >
> > Can multiple Kafka consumers read from the same partition of same topic
> by
> > default? B
Hi All,
Can multiple Kafka consumers read from the same partition of same topic by
default? By default, I mean since group.id is not mandatory I am wondering
if I spawn multiple kafka consumers without specifying any group.id and
give them the same topic and partition name will they be able to
Does the number of App instances and Zookeeper servers should be the same?
I understand the requirement of 2F+1 to tolerate F failures but this is to
tolerate failures of Zookeeper instances itself. But how about the number
of App instances ? For example say I have 3 zookeeper servers and I have 2
es. That way after a
> crash, they can check the external store (HSFS, JSDC, etc) retrieve the
> last committed offset and seek the the next message and continue processing
> with no duplicates and exactly once semantics.
>
> -hans
>
>
>
>
> > On Dec 21, 2016, at 4:
How does Kafka emulate exactly once processing currently? Does it require
the producer to send at least once and consumer to de dupe?
I did do my research but I feel like I am going all over the place so a
simple short answer would be great!
Thanks!
ecord with higher writetime will win.
>
> Using es's document _version field is just one option, you can use
> something from you domain or kafka's offset or machine timestamp (not
> recommended at all).
>
> I hope it could help
>
> From: ka
And there is this https://github.com/vroyer/elassandra which is still under
active development and not sure how they plan to keep up with Apache
Cassandra moving forward.
On Mon, Nov 7, 2016 at 9:36 AM, kant kodali <kanth...@gmail.com> wrote:
> Fixing typo's
>
> Hi Tauzell,
>
Fixing typo's
Hi Tauzell,
Yeah our users want to query, do aggregations on Elastic Search directly
and we cannot have inconsistent data because say the writes didn't make it
into Cassandra but made it to Elastic search then a simple aggregations
like count will lead to a wrong answer but again
; same set of messages from Kafka ... I don't think this is possible.
>
> -Dave
>
> -Original Message-
> From: kant kodali [mailto:kanth...@gmail.com]
> Sent: Monday, November 7, 2016 10:48 AM
> To: users@kafka.apache.org
> Subject: Re: is there a way to make sure two con
e elasticsearch _version field as cassandra's
> writetime?(_version is strictly increasing, cassandra uses writetime for
> applying LWW, so last write in elasticsearch will always win)
>
> It needs no transaction and makes databases convergent.
>
>
> _______
ngineer, Confluent Inc.
> * h...@confluent.io (650)924-2670
> */
>
> On Sun, Nov 6, 2016 at 7:08 PM, kant kodali <kanth...@gmail.com> wrote:
>
> > Hi Hans,
> >
> > The two storages we use are Cassandra and Elastic search and they are on
> > the same datacenter f
Hi Hans,
The two storages we use are Cassandra and Elastic search and they are on
the same datacenter for now.
The Programming Language we use is Java and OS would be Ubuntu or CentOS.
We get messages in JSON format so we insert into Elastic Search directly
and for Cassandra we transform JSON
Hi! Thanks. any pointers on how to do that?
On Sun, Nov 6, 2016 at 2:32 PM, Tauzell, Dave <dave.tauz...@surescripts.com>
wrote:
> You should have one consumer pull the message and submit the data to each
> storage using an XA transaction.
>
> > On Nov 5, 2016, at 19:4
here may be a better way to get you the functionality you want?
>
> -hans
>
>
>
>
> > On Nov 5, 2016, at 4:31 PM, kant kodali <kanth...@gmail.com> wrote:
> >
> > I am new to Kafka and reading this statement "write consumer 1 and
> consumer
> > 2 to
2016 at 4:15 PM, kant kodali <kanth...@gmail.com> wrote:
> Hi Hans,
>
> What do you mean by "write consumer 1 and consumer 2 to share a common
> external offset storage" ? can you please elaborate a bit more.
>
> Thanks!
>
> On Sat, Nov 5, 2016 at
his in Apache Kafka but if you can
> write consumer 1 and consumer 2 to share a common external offset storage
> then you may be able to build the functionality you seek.
>
> -hans
>
>
>
> > On Nov 5, 2016, at 3:55 PM, kant kodali <kanth...@gmail.com> wrote:
Sorry there is a typo. here is a restatement.
Is there a way to make sure two consumers receive the same message from the
kafka broker in a atomic way? such that if consumer 1 gets a message
consumer 2 should also get that message and if consumer 1 fails for
whatever reason consumer 2 should
is there a way to make sure two consumers receive the same message from the
kafka broker in a atomic way? such that if consumer 1 gets a message
consumer 2 should also get that message and in case one of the consumer
fails for whatever reason consumer 2 should also rollback to previous
offset or
@Fei Just curious why you guys are interested in using Kafka. I thought
alcatel-lucent usually create their own software no?
On Fri, Sep 23, 2016 10:36 PM, Kamal C kamaltar...@gmail.com
wrote:
Reduce the metadata refresh interval 'metadata.max.age.ms' from 5 min to
your desired time
to limit
latency, and that may also be enough to get a low enough latency. It would
certainly be interesting to compare the two, with the same hardware, and
with high load.
On Thu, Sep 22, 2016 at 6:01 PM kant kodali <kanth...@gmail.com> wrote:
> @Gerard Thanks for this. It looks
Does Kafka Sync/persist every message from a publisher by default? If not, What
settings should I change so I Sync every message?
real-time-datas-unifying
> >
> > –
> > Best regards,
> > Radek Gruchalski
> > ra...@gruchalski.com
> >
> >
> > On September 17, 2016 at 9:49:43 PM, kant kodali (kanth...@gmail.com)
> wrote:
> >
> > Still it should be possible to
independent of Zookeeper alltogether. Maybe not worth spending time on
migrating to Consul in that case.
Cheers,
Jens
On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfount...@meetme.com>
wrote:
> +2 watching.
>
> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <kanth..
Why does comcast needs to do better than 1-2 seconds?
On Sun, Sep 18, 2016 8:08 PM, Ghosh, Achintya (Contractor)
achintya_gh...@comcast.com
wrote:
Hi there,
We have an usecase where we do a lot of business logic to process each message
and sometime it takes 1-2 sec, so will be Kafka
your
thoughts.
On Sat, Sep 17, 2016 12:39 PM, Radoslaw Gruchalski ra...@gruchalski.com
wrote:
Kafka is not a queue. It’s a distributed commit log.
–
Best regards,
Radek Gruchalski
ra...@gruchalski.com
On September 17, 2016 at 9:23:09 PM, kant kodali (kanth...@gmail.com) wrote
regarding if this is the reason:
Pull is much more sensible when a lot of data is pushed through. It allows
consumers consuming at their own pace, slow consumers do not slow the complete
system down.
--
Best regards,
Rad
On Sat, Sep 17, 2016 at 11:18 AM +0200, "kant kodali&qu
why did Kafka choose pull instead of push for a consumer? push sounds like it
is more realtime to me than poll and also wouldn't poll just keeps polling even
when they are no messages in the broker causing more traffic? please enlighten
me
ker will be the leader of 1 of the 3 partitions.
P.S how did the benchmarking go?
On Sat, Sep 17, 2016 at 1:36 PM, kant kodali <kanth...@gmail.com> wrote:
> can one topic be registered in multiple brokers? if so, which component of
> kafka decides which broker should get the message for that particular
> topic?
> Thanks!
can one topic be registered in multiple brokers? if so, which component of
kafka decides which broker should get the message for that particular topic?
Thanks!
https://issues.apache.org/jira/browse/KAFKA-1793
It would be great to use Consul instead of Zookeeper for Kafka and I think it
would benefit Kafka a lot from the exponentially growing consul community.
.
On Thu, 15 Sep 2016 at 17:03 kant kodali <kanth...@gmail.com> wrote:
> I haven't changed anything from
> https://github.com/apache/kafka/blob/trunk/config/server.properties
> and it looks like it is pointing to zookeeper.
> Question:
> Does producer client need to po
?
On Thu, Sep 15, 2016 4:24 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Examine server.properties and see which port you're using in there
On Thu, Sep 15, 2016 at 3:52 PM, kant kodali <kanth...@gmail.com> wrote:
which port should I use 9091 or 9092 or 2181 to send messages through kafka
which port should I use 9091 or 9092 or 2181 to send messages through kafka
when using a client Library?
I start kafka as follows:
sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo
./bin/kafka-server-start.sh config/server.properties
and I dont see any process running on 9091 or
that a go..but can you
please confirm that one partition on one broker should be able to handle 300K
messages of 1KB data size for each message?
Thanks,kant
On Thu, Sep 15, 2016 2:28 AM, kant kodali kanth...@gmail.com
wrote:
Hi Ben,
I can give that a try but can you tell me the suspicion
:16 AM, Ben Davison ben.davi...@7digital.com
wrote:
Hi Kant,
I was following the other thread, can you try using a different
benchmarking client for a test.
https://grey-boundary.io/load-testing-apache-kafka-on-aws/
Ben
On Thursday, 15 September 2016, kant kodali <kanth...@gmail.
with Kafka I tried it with 10 messages with single broker and only one partiton
that looked instantaneous and ~5K messages/sec for the data size of 1KB
I tried it with 1000 messages that looked instantaneous as well ~5K messages/sec
for the data size of 1KBI tried it with 10K messages with single
.
Here's a case study that should help:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Good luck, let us know how it goes
On Thu, Sep 15, 2016 at 1:42 PM, kant kodali <kanth...@gmail.com> wrote:
yeah..
I
com
wrote:
Lower the workload gradually, start from 10 messages, increase to 100, then
1000, and so on. See if it slows down as the workload increases. If so, you
need more brokers + partitions to handle the workload.
On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth...@gmail.com>
m4.xlarge
On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
What's the instance size that you're using? With 300k messages your single
broker might not be able to handle it.
On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> wrote:
M
My goal is to test the throughput (#messages per second) given my setup and
with a data size of 1KB. if you guys already have some idea on these numbers
that would be helpful as well.
On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
wrote:
172.* is all private ip's for my
-producer.sh instead of the Node code?
On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> wrote:
> They are hosted on AWS and I dont think there are any network issues
> because I
> tried testing other Queuing systems with no issues however I am using a
>
tar ali.rac...@gmail.com
wrote:
It sounds like a network issue. Where are the 3 servers located / hosted?
On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com> wrote:
Hi,
I have the following setup.
Single Kafka broker and Zookeeper on Machine 1single Kafka
Hi,
I have the following setup.
Single Kafka broker and Zookeeper on Machine 1single Kafka producer on Machine 2
Single Kafka Consumer on Machine 3
When a producer client sends a message to the Kafka broker by pointing at the
Zookeeper Server the consumer doesn't seem to get the message right
Hi All,
I am trying to do a simple benchmark test for Kafka using single broker,
producer and consumer however my consumer doesn't seem to receive all the
messages produced by the producer so not sure what is going on any help?
Here is the full description of the problem.
54 matches
Mail list logo
