Hi Kafka users,
I am trying to understand the behavior of compression in Kafka. Consider a
scenario, where producer sets compression.codec "snappy" and broker's
compression.code "lz4"?
In this scenario, what is the behavior of the compression?
As far as I have understood is the following,
The me
LoginType was in 0.10.x release.
This seems to indicate Kafka version mismatch.
Can you check the dependencies of your test ?
Thanks
On Tue, Feb 13, 2018 at 8:03 PM, Ian Ewing wrote:
> I have been trying to figure out how to unit test a kafka producer. Should
> take in a simple integer and pe
I have been trying to figure out how to unit test a kafka producer. Should
take in a simple integer and perform some addition. Followed what I could
find on spring kafka unit testing but keep running into this error:
19:53:12.788 [main] ERROR kafka.server.KafkaServer - [Kafka Server 0],
Fatal erro
We noticed that the replication factor is six for *__consumer_offsets* and
as we have been moving consumers over from zk to kf offsets latency for
FetchFollower on the group coordinator for a busy consumer_group increases
significantly (in hundreds of ms), but offset commit latency and cpu
increase
I have also check the commit-latency-avg, it's around 23 millis per commit.
That translate to about the same throughput that I'm getting now
(0.04message/millis). Does anyone got any benchmark for kafka stream's
commit-latency-avg? Is it possible to tune it to be faster? I just want to
veri
Hey Damian and folks,
I've also tried 1000 and 500 and the performance state is exactly the same.
Any other ideas?
Regards,
Brilly
-Original Message-
From: Damian Guy [mailto:damian@gmail.com]
Sent: Tuesday, February 13, 2018 4:48 PM
To: users@kafka.apache.org
Subject: Re: Kafka St
Yes. If a consumer when down, all the polled messages that
were not committed will be redelivered to another consumer.
regards
On Tue, Feb 13, 2018 at 9:31 AM, pradeep s
wrote:
> Hi All,
> I am running a Kafka consumer(Single threaded) on kubernetes . Application
> is polling the records and
+1
Ran tests, verified streams quickstart works
On Tue, 13 Feb 2018 at 17:52 Damian Guy wrote:
> Thanks Ewen - i had the staging repo set up as profile that i forgot to
> add to my maven command. All good.
>
> On Tue, 13 Feb 2018 at 17:41 Ewen Cheslack-Postava
> wrote:
>
>> Damian,
>>
>> Which
Thanks Ewen - i had the staging repo set up as profile that i forgot to add
to my maven command. All good.
On Tue, 13 Feb 2018 at 17:41 Ewen Cheslack-Postava
wrote:
> Damian,
>
> Which quickstart are you referring to? The streams quickstart only executes
> pre-built stuff afaict.
>
> In any case
Damian,
Which quickstart are you referring to? The streams quickstart only executes
pre-built stuff afaict.
In any case, if you're building a maven streams project, did you modify it
to point to the staging repository at
https://repository.apache.org/content/groups/staging/ in addition to the
def
Hi All,
I am running a Kafka consumer(Single threaded) on kubernetes . Application
is polling the records and accummulating in memory . There is a scheduled
write of these records to S3 . Only after that i am committing the offsets
back to Kafka.
There are 4 partitions and 4 consumers(4 kubernetes
Hi Kafka guys,
I faced the following Exception when I sent message to Kafka 1.0 -
2018-02-13 04:45:49,861 INFO [STDOUT] org.apache.kafka.common.KafkaException:
Cannot perform send because at least one previous transactional or idempotent
request has failed with errors.
2018-02-13 04:45:49,861
So...if you have a consumer running on A, and then you start another one on
B, I would think it would trigger a rebalance. I am not 100% sure whether A
would just keep being the active consumer or if it would switch to B. That
being said, yes, I think the general idea is correct.
In terms of offse
On Tue, Feb 13, 2018 at 2:59 PM, Subhash Sriram
wrote:
Hey Xavier,
>
> Within a consumer group, you can only have as many consumers as you have
> partitions in a topic. If you start more consumers than partitions within
> the same group, they will just be idle.
>
Excellent, I am reading the docs
1. Perhaps a human-readable log, being write-only, and which may
buffer on the user side or in the kernel, may be more efficient
because small writes are accumulated in a buffer (cheap) actually
pushed to disk (less cheap)? If you mmap'd this instead, how do you
feel it would behave?
2. Did you re
ah .. ok .. thanks for the clarification .. for reduce I guess the overload
with Materialized does the same thing ..
regards.
On Tue, Feb 13, 2018 at 2:24 PM, Damian Guy wrote:
> There is an overload `leftJoin(KTable, ValuJoiner, Joined)`
>
> Joined is where you specify the Serde for the KTable
Hey Xavier,
Within a consumer group, you can only have as many consumers as you have
partitions in a topic. If you start more consumers than partitions within the
same group, they will just be idle.
Thanks,
Subhash
Sent from my iPhone
> On Feb 13, 2018, at 8:30 AM, Xavier Noria wrote:
>
> C
Can a consumer group that listens to one topic, which has only one
partition, have multiple consumer instances in different
processes/machines? Or can you have at most as many consumer instances as
partitions?
If that is like what you said , why index file use the memory mapped file?
From: jan
Sent: Monday, February 12, 2018 9:26 PM
To: users@kafka.apache.org
Subject: Re: why kafka index file use memory mapped files ,however log file
doesn't
A human-readable log file
I logged https://issues.apache.org/jira/browse/KAFKA-6557
Best regards,
Wouter
On 13 February 2018 at 12:50, Jan Filipiak wrote:
> I would encourage you todo so.
> I also think its not reasonable behavior
>
> On 13.02.2018 11:28, Wouter Bancken wrote:
>
>> We have upgraded our Kafka version as
Hi,
Im using Kafka to collect Device Status and Log from IoT gateway which are
installed different location in users place. Im trying to setup
Authentication option between client (producer) and the broker., In
Password based authentication what if I want to change password frequently
as a best pr
I would encourage you todo so.
I also think its not reasonable behavior
On 13.02.2018 11:28, Wouter Bancken wrote:
We have upgraded our Kafka version as an attempt to solve this issue.
However, the issue is still present in Kafka 1.0.0.
Can I log a bug for this in JIRA?
Wouter
On 5 February 2
Hi Ewen,
I'm trying to run the streams quickstart and I'm getting:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate
(default-cli) on project standalone-pom: The desired archetype does not
exist (org.apache.kafka:streams-quickstart-java:1.0.1)
Something
We have upgraded our Kafka version as an attempt to solve this issue.
However, the issue is still present in Kafka 1.0.0.
Can I log a bug for this in JIRA?
Wouter
On 5 February 2018 at 09:22, Wouter Bancken
wrote:
> The consumers in consumer group 'X' do not have a regex subscription
> matchin
+1 (non-binding)
ran quick-start, unit tests on the src.
On Tue, Feb 13, 2018 at 5:31 AM, Ewen Cheslack-Postava
wrote:
> Thanks for the heads up, I forgot to drop the old ones, I've done that and
> rc1 artifacts should be showing up now.
>
> -Ewen
>
>
> On Mon, Feb 12, 2018 at 12:57 PM, Ted Y
There is an overload `leftJoin(KTable, ValuJoiner, Joined)`
Joined is where you specify the Serde for the KTable and for the resulting
type. We don't need the Serde for the stream at this point as the value has
already been deserialized.
HTH,
Damian
On Tue, 13 Feb 2018 at 05:39 Debasish Ghosh
w
Hi Brilly,
My initial guess is that it is the overhead of committing. Commit is
synchronous and you have the commit interval set to 50ms. Perhaps try
increasing it.
Thanks,
Damian
On Tue, 13 Feb 2018 at 07:49 TSANG, Brilly
wrote:
> Hi kafka users,
>
> I created a filtering stream with the Proc
27 matches
Mail list logo