Hi Mudit,
Thanks a lot for your answer.
However, today we have set "advertised.host.name" on each kafka instance to
the specific IP address of each node. For example, by default kafka tries
to read the machine's hostname, specifically ip-10-1-83-5.ec2.internal -
now it's "10.1.83.5" (of course, w
Glad to hear that you issue is fixed now!
On 6/2/16, 2:11 PM, "Marco B." wrote:
>Hi Mudit,
>
>Thanks a lot for your answer.
>
>However, today we have set "advertised.host.name" on each kafka instance to
>the specific IP address of each node. For example, by default kafka tries
>to read the ma
Filesystem encryption is transparent to Kafka. You don't need to use SSL,
but your encryption requirements may cause you to need SSL as well.
With regards to compression, without adding at rest encryption to Kafka
(which is a very major piece of work, one that for sure requires a KIP and
has many,
Hi Avi,
Using the low-level streams API you can definitely read or write to arbitrary
locations inside the process() method.
However, back to your original question: even with the low-level streams API
the sources and sinks can only be Kafka topics for now. So, as Gwen mentioned,
Connect would
Hafsa,
since you specifically asked about non-free Kafka monitoring options as
well: As of version 3.0.0, the Confluent Platform provides a commercial
monitoring tool for Kafka called Confluent Control Center. (Disclaimer: I
work for Confluent.)
Quoting from the product page at
http://www.confl
Thanks Ben.
The comments on the Jira mention a pluggable component that will manage the
bootstrap list from a discovery service.
That's exactly what I need.
Was a Kip drafted for this enhancement?
-Danny
> On Jun 1, 2016, at 7:05 AM, Ben Stopford wrote:
>
> Hey Danny
>
> Currently the boo
Hi Danny,
A KIP has not been drafted for that yet. Would you be interested in working
on it?
Ismael
On Thu, Jun 2, 2016 at 1:15 PM, Danny Bahir wrote:
> Thanks Ben.
>
> The comments on the Jira mention a pluggable component that will manage
> the bootstrap list from a discovery service.
>
> Th
That looks like somebody is killing the process. I'd suspect either the
linux OOM killer or something else automatically killing the JVM for some
reason.
For the OOM killer, assuming you're on ubuntu, it's pretty easy to find in
/var/log/syslog (depending on your setup). I don't know about other
o
Hi Srikanth,
your third approach seems to be the best fit. It uses only one shuffle
of the data (which you cannot prevent in any case).
If you want to put everything into a single application, you could use a
"dummy" custom aggregation to convert the KStream into a KTable instead
of writing into
Hi Srikanth,
I am not exactly sure if I understand your question correctly.
One way to track the progress is to get the current record offset (you
can obtain it in the low lever Processor API via the provided Context
object).
Otherwise, on commit, all writes to intermediate topics are flushed to
Not that I have anything against paying for monitoring, or against
Confluent, but you will need your consumers to be using kafka 1.10 is you
want to make most out of the confluent solution. We currently are using
zabbix, it's free, and it has complete functionality in one product. It
does can be a
With the caveat that I’ve never tried this before...
I don’t see a reason why this wouldn’t work. There’s no topic information
that’s encoded in the log segments, as far as I’m aware. And there’s no
information about offsets stored in Zookeeper. So in theory, you should be
able to shut down the en
Hi there,
Firstly, a note that Kafka isn't really designed for this kind of large
message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/
covers a lot of tips around this use case however, and covers some tuning
that will likely improve your usage.
In particular, I expect tuning up
I am on the same situation. I use zookeeper to publish kafka broker
endpoints for dynamic discovery.
Il Gio 2 Giu 2016 14:33 Ismael Juma ha scritto:
> Hi Danny,
>
> A KIP has not been drafted for that yet. Would you be interested in working
> on it?
>
> Ismael
>
> On Thu, Jun 2, 2016 at 1:15 PM,
Hi all,
I'm looking at changing the Kafka default logging setup to use the
RollingFileAppender instead of the DailyRollingFileAppender in an effort to
accomplish two goals:
1) Avoid filling up users' disks if the log files grow unexpectedly
2) Move off the admittedly unreliable DailyRollingFileApp
I haven't started using this in production but this is how I will likely setup
the logging as it is easier to manage.
-Dave
-Original Message-
From: Dustin Cote [mailto:dus...@confluent.io]
Sent: Thursday, June 02, 2016 9:33 AM
To: users@kafka.apache.org; d...@kafka.apache.org
Subject: C
+1, this is what Wikimedia uses in production.
On Thu, Jun 2, 2016 at 10:38 AM, Tauzell, Dave wrote:
> I haven't started using this in production but this is how I will likely
> setup the logging as it is easier to manage.
>
> -Dave
>
> -Original Message-
> From: Dustin Cote [mailto:dus.
Just to clarify, do you mean you are using the RollingFileAppender in
production, or the naming convention for DailyRollingFileAppender is
required by your production systems?
On Thu, Jun 2, 2016 at 10:49 AM, Andrew Otto wrote:
> +1, this is what Wikimedia uses in production.
>
> On Thu, Jun 2,
I used Ambari automatic deployment and it does not require additional
configurations unless you need some changes.
On Wed, Jun 1, 2016 at 9:15 PM, Shaolu Xu wrote:
> Hi All,
>
> I used the latest HDP 2.4 version.
> Did you do some configuration before used HDP? I searched a solution that
> is ht
Thanks!
> On May 31, 2016, at 1:00 PM, Michael Noll wrote:
>
> FYI: I fixed the docs of schema registry (vProps -> props).
>
> Best, Michael
>
>
> On Tue, May 31, 2016 at 2:05 AM, Rick Mangi wrote:
>
>> That was exactly the problem, I found the example here to be very helpful
>> -
>> https
The RollingFileAppender is required to use in production.
-Dave
-Original Message-
From: Dustin Cote [mailto:dus...@confluent.io]
Sent: Thursday, June 02, 2016 9:51 AM
To: users@kafka.apache.org
Subject: Re: Changing default logger to RollingFileAppender (KAFKA-2394)
Just to clarify, do
In terms of performance there is not going to be much difference to+table
vs through+aggregateByKey rt?
Srikanth
On Thu, Jun 2, 2016 at 9:21 AM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> your third approach seems to be the best fit. It uses only one shuffle
> of the data (which you cannot prev
I would not expect a performance difference.
-Matthias
On 06/02/2016 06:15 PM, Srikanth wrote:
> In terms of performance there is not going to be much difference to+table
> vs through+aggregateByKey rt?
>
> Srikanth
>
>
> On Thu, Jun 2, 2016 at 9:21 AM, Matthias J. Sax
> wrote:
>
>> Hi Srika
Matthias,
"""bin/kafka-consumer-groups.sh --zookeeper localhost:2181/kafka10
--list""" output didn't show the group I used in streams app.
Also, AbstractTask.java had a commit() API. That made me wonder if offset
management was overridden too.
I'm trying out KafkaStreams for one new streaming app
I did try approach 3 yesterday with the following sample data.
topic1:
127339 538433
131933 626026
128072 536012
128074 546262
*123507 517631*
128073 542361
128073 560608
topic2:
128074 100282
131933 100394
127339 100445
128073 100710
*123507 100226*
I joined these and printed the re
Hi Tom,
That is one of the first things that i checked. Active memory never goes
above 50% of overall available. File cache uses the rest of the memory but
i do not think that causes OOM killer.
Either way there is no entries in /var/log/messages (centos) to show OOM is
happening.
Thanks
On Thu,
Hi
Thanks for the information .
I have one question :
Right now in my scenario maximum message size is around 800KB . did we
consider these messages in large size categories , because article told
about 10-100 MB data .
Regards
Prateek
On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford wrote:
The article says ideal is about 10KB, which holds up well with what we've
seen in practice as well.
On Thu, Jun 2, 2016 at 6:25 PM, prateek arora
wrote:
> Hi
> Thanks for the information .
>
> I have one question :
>
> Right now in my scenario maximum message size is around 800KB . did we
> con
Hi All,
Any suggestions or inputs on this ? Any help would be greatly appreciated.
Thanks,
Meghana
On Wed, Jun 1, 2016 at 3:01 PM, Meghana Narasimhan <
mnarasim...@bandwidth.com> wrote:
> Hi,
> I have a 3 node cluster with kafka version 0.9.0.1 with many topics having
> replication factor 3 and
Wondering if anyone has encountered similar issues.
Using Kafka 0.8.2.1.
Occasionally, we encounter a situation in which a consumer (including
kafka-console-consumer.sh) just hangs.
If I increment the offset to skip the offending message, things work fine again.
I have been able to identify the
Hi All,
We are using Kafka broker cluster in our data center.
Recently, It is realized that when a Kafka broker goes down then client try
to refresh the metadata but it get stale metadata upto near 30 seconds.
After near 30-35 seconds, updated metadata is obtained by client. This is
really a larg
What about in dmesg? I have run into this issue and it was the OOM
killer. I also ran into a heap issue using too much of the direct memory
(JVM). Reducing the fetcher threads helped with that problem.
On Jun 2, 2016 12:19 PM, "allen chan" wrote:
> Hi Tom,
>
> That is one of the first things t
Have you verified that the old leader of the partition is using the same
I'd as before? Check in zk /brokers/ids to get a list of available
brokers. I would use the reassignment tool to move partition 3 to brokers
in the list from zk (specifying only 3 brokers). Make sure to include
broker with
Client version?
On Fri, Jun 3, 2016, 4:44 AM safique ahemad wrote:
> Hi All,
>
> We are using Kafka broker cluster in our data center.
> Recently, It is realized that when a Kafka broker goes down then client try
> to refresh the metadata but it get stale metadata upto near 30 seconds.
>
> After
kafka version:0.9.0.0
go sarama client version: 1.8
On Thu, Jun 2, 2016 at 5:14 PM, Steve Tian wrote:
> Client version?
>
> On Fri, Jun 3, 2016, 4:44 AM safique ahemad wrote:
>
> > Hi All,
> >
> > We are using Kafka broker cluster in our data center.
> > Recently, It is realized that when a Kaf
So you are coming from https://github.com/Shopify/sarama/issues/661 ,
right? I'm not sure if anything from broker side can help but looks like
you already found DialTimeout on client side can help?
Cheers, Steve
On Fri, Jun 3, 2016, 8:33 AM safique ahemad wrote:
> kafka version:0.9.0.0
> go s
On 6/2/16, 07:03, "Eno Thereska" wrote:
> Using the low-level streams API you can definitely read or write to arbitrary
> locations inside the process() method.
Ah, good to know — thank you!
> However, back to your original question: even with the low-level streams
> API the sources and sinks c
you got it right...
But DialTimeout is not a concern here. Client try fetching metadata from
Kafka brokers but Kafka give them stale metadata near 30-40 sec.
It try to fetch 3-4 time in between until it get updated metadata.
This is completely different problem than
https://github.com/Shopify/sara
Hate to bring up "non-flashy" technology... but Apache Camel would be a
great fit for something like this. Two java libraries each with very strong
suits.
On Thu, Jun 2, 2016 at 6:09 PM, Avi Flax wrote:
> On 6/2/16, 07:03, "Eno Thereska" wrote:
>
> > Using the low-level streams API you can de
Yes, I'm in.
Sent from my iPhone
> On Jun 2, 2016, at 8:32 AM, Ismael Juma wrote:
>
> Hi Danny,
>
> A KIP has not been drafted for that yet. Would you be interested in working
> on it?
>
> Ismael
>
>> On Thu, Jun 2, 2016 at 1:15 PM, Danny Bahir wrote:
>>
>> Thanks Ben.
>>
>> The comments
quoting from https://cwiki.apache.org/confluence/display/KAFKA/FAQ
The high-level consumer will block if
the next message available is larger than the maximum fetch size you have
specified
- One possibility of a stalled consumer is that the fetch size in the
consumer is smaller than the lar
I see. I'm not sure if this is a known issue. Do you mind share the
brokers/topics setup and the steps to reproduce this issue?
Cheers, Steve
On Fri, Jun 3, 2016, 9:45 AM safique ahemad wrote:
> you got it right...
>
> But DialTimeout is not a concern here. Client try fetching metadata from
>
Thanks for the quick reply Danny.
The message size as per the DumpLogSegments is around 59KB
I used a very high message.max.size and a high fetchsize of 1 MB (that's the
message.max.size in the broker) and still the same hang behavior.
Also tried a max-wait-ms so that the consumer does not "hang
I'm hesitant to cite it because it wasn't really a proper benchmark, but
with the end-to-end encryption through Kafka proof of concept described at
http://symc.ly/1pC2CEG, doing the encryption added only 26% to the time
taken to send messages and only 6% to the time taken to consume messages.
This
Hello Srikanth,
When involved in joins, KTable need to pass both the old value as well as
the new value as a pair to the join operator since it is "an ever updating
table with the underlying changelog", for example, your topic1 stream have
key "128073" with updated values from 542361 to 560608. Th
45 matches
Mail list logo