ly the snapshot logic is decoupled from the log retention as of
> KIP-360.
>
> Justine
>
> On Mon, May 20, 2024 at 11:20 PM Claude Warren wrote:
>
> > The LRU cache is just that: a cache, so yes things expire from the cache
> > but they are not gone. As long as a s
piring
> > > active producers.
> > >
> > > I am willing to discuss some solutions that work with older clients,
> but
> > my
> > > concern is spending too much time on a complicated solution and not
> > > encouraging movement to newer and better clients.
&
;
> Fixing the client also allows us to fix some of the other issues we have
> with idempotent producers.
>
> Justine
>
> On Fri, May 17, 2024 at 12:46 AM Claude Warren wrote:
>
> > I think that the point here is that the design that assumes that you can
> > keep all t
r example
> PARAMETERIZED "(nl|de|cz)-accounts-". I'm lifting regex syntax here,
> but this isn't actually a regex, and wouldn't allow arbitrary numbers
> of characters, or the * or + operators.
> In the background it could evaluate exactly like the 3 individual
>
so in the rejection alternatives you listed an approved KIP which is a
> > bit confusing can you move this to motivations instead
> >
> > > On 15 May 2024, at 14:35, Claude Warren wrote:
> > >
> > > This is a proposal that should solve the OOM problem on the s
uthorizerBenchmark, we can
> drop this kip.
>
> And thank you Claude for the suggestion on the new implementation.
>
> On Tue, May 7, 2024 at 4:37 PM Claude Warren, Jr
> wrote:
>
> > I have updated KIP-1042 with a proposal for how to reduce the time spent
> >
This is a proposal that should solve the OOM problem on the servers without
some of the other proposed KIPs being active.
Full details in
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1044%3A+A+proposal+to+change+idempotent+producer+--+server+implementation
trary numbers
> of characters, or the * or + operators.
> In the background it could evaluate exactly like the 3 individual
> PREFIX rules, but be easier to evaluate on the backend, and support
> the intersection query I mentioned earlier. It could also support
> [a-zA-Z] notation i
ls` to support MATCH-type ACL
> without introducing performance issue?
>
>
> Thanks,
>
> 2024年5月3日(金) 19:51 Claude Warren, Jr :
>
> > As I wrote in [1], the ACL evaluation algorithm needs to be specified
> with
> > respect to the specificity of the pattern so
t; have avoided that solution is that there is no state stored for idempotent
> producers when grabbing an ID. My concern there is either storing too much
> state to track this or throttling before we need to.
>
> Justine
>
> On Thu, May 2, 2024 at 2:36 PM Claude W
ort+for+wildcard+when+creating+new+acls>
On Fri, May 3, 2024 at 12:18 PM Claude Warren wrote:
> Took me awhile to find it but the link to the KIP is
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1042%3A+Support+for+wildcard+when+creating+new+acls
>
> On Fri, May 3, 20
Took me awhile to find it but the link to the KIP is
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1042%3A+Support+for+wildcard+when+creating+new+acls
On Fri, May 3, 2024 at 10:13 AM Murali Basani
wrote:
> Hello,
>
> I'd like to propose a suggestion to our resource patterns in Kafka
I think that if this is introduced (and perhaps even if it is not) we need
a clear ACL evaluation process.
I know we have both allow and deny, and that deny takes precedence over
allow.
But let's consider two scenarios
1. Unintended access.
Let's assume we start with the 6 topics Murali used
There is some question about whether or not we need the configuration
options. My take on them is as follows:
producer.id.quota.window.num No opinion. I don't know what this is used
for, but I suspect that there is a good reason to have it. It is not used
within the Bloom filter caching
Quick note: I renamed the example code. It is now at
https://github.com/Claudenw/kafka/blob/KIP-936/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerIDQuotaManagerCache.java
On Thu, May 2, 2024 at 10:47 AM Claude Warren, Jr
wrote:
> Igor, thanks for taking the t
Igor, thanks for taking the time to look and to review the code. I regret
that I have not pushed the latest code, but I will do so and will see what
I can do about answering your Bloom filter related questions here.
How would an operator know or decide to change the configuration
> for the
My Confluence ID is "claude"
On Thu, Apr 25, 2024 at 8:40 PM Matthias J. Sax wrote:
> What's your wiki ID? We can grant write access on our side if you have
> already an account.
>
> -Matthias
>
> On 4/25/24 4:06 AM, Claude Warren wrote:
> > I would li
I would like to get edit access to the Kafka confluence so that I can work
on KIP-936. Can someone here do that or do I need to go through Infra?
Claude
at or below Shape.P
- Assuming Shape.N = 11 and Shape.P = 0.06 the Bloom filter at each
layer will consume 35 bytes. https://hur.st/bloomfilter provides a quick
calculator for other values.
Claude
On Tue, Apr 16, 2024 at 8:06 AM Claude Warren wrote:
> Let's put aside the CPC d
selector based
> on
> > the modulus of the system by the resolution of the Y' partitions. On each
> > call:
> I didn’t evaluate CPC datasketch or any counter solution as I explained
> above the aim is not to build a counter specially the Kafka Sensor can be
> enough to in
implementation.
[1] https://datasketches.apache.org/docs/CPC/CPC.html
On Fri, Apr 12, 2024 at 3:10 PM Claude Warren wrote:
> I think there is an issue in the KIP.
>
> Basically the kip says, if the PID is found in either of the Bloom filters
> then no action is taken
> If the P
Congrats Greg! All the hard work paid off.
On Mon, Apr 15, 2024 at 6:58 AM Ivan Yurchenko wrote:
> Congrats Greg!
>
> On Sun, Apr 14, 2024, at 22:51, Sophie Blee-Goldman wrote:
> > Congrats Greg! Happy to have you
> >
> > On Sun, Apr 14, 2024 at 9:26 AM Jorge Esteban Quilcate Otoya <
> >
minute window we
are trying to track. Or am I missing something? It seems like we need to
add each PID to the last bloom filter
On Fri, Apr 12, 2024 at 2:45 PM Claude Warren wrote:
> Initial code is available at
> https://github.com/Claudenw/kafka/blob/KIP-936/storage/src/main/java/org/
Initial code is available at
https://github.com/Claudenw/kafka/blob/KIP-936/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerIDQuotaManager.java
On Tue, Apr 9, 2024 at 2:37 PM Claude Warren wrote:
> I should also note that the probability of false positives does not f
gt; ACL would need to be provided to write to the DLQ if configured.
>
> Cheers,
> Damien
>
> On Fri, 12 Apr 2024 at 08:59, Claude Warren, Jr
> wrote:
> >
> > I am new to the Kafka codebase so please excuse any ignorance on my part.
> >
> > When a d
I am new to the Kafka codebase so please excuse any ignorance on my part.
When a dead letter queue is established is there a process to ensure that
it at least is defined with the same ACL as the original queue? Without
such a guarantee at the start it seems that managing dead letter queues
will
9, 2024 at 2:26 PM Claude Warren wrote:
> The overall design for KIP-936 seems sound to me. I would make the
> following changes:
>
> Replace the "TimedBloomFilter" with a "LayeredBloomFilter" from
> commons-collections v4.5
>
> Define the producer.id.q
The overall design for KIP-936 seems sound to me. I would make the
following changes:
Replace the "TimedBloomFilter" with a "LayeredBloomFilter" from
commons-collections v4.5
Define the producer.id.quota.window.size.seconds to be the length of time
that a Bloom filter of PIDs will exist.
Define
Is there test code, or initial POC code for this KIP somewhere? I would like
to help move this forward but need a few pointers to associated resources. I
have read KIP-853 and it is beginning to sink in, but code would be nice.
Thanks,
Claude
On 2024/03/21 18:41:04 José Armando García Sancio
ll need to replace the bloom for all
> principals however if they are separated like I suggested then replacing
> the LayedBloomFilter of max entries X with another one with max entries Y
> will only impact one user and not everyone. Does this make sense?
>
>
>
> On Fri, A
I misspoke before the LayedBloomFilterTest.testExpiration() uses
milliseconds to expire the data but it layout an example of how to expire
filters in time intervals.
On Fri, Aug 18, 2023 at 4:01 PM Claude Warren wrote:
> Sorry for taking so long to get back to you, somehow I missed your mess
yers
> > older than 1 hour. Since the layers are a time ordered list this is
> simply
> > removing the elderly layers from the front of the list.
>
> Maybe am missing something here but I can't find anything in the
> `LayerManager` code that point to how often will the eviction
new layer
This should be very fast and space efficient.
On Wed, Jun 21, 2023 at 11:13 AM Claude Warren wrote:
> I have an implementation of a layered Bloom filter in [1] (note the
> layered branch). This should handle the layering Bloom filter and allow
> for layers that
>
>1. Do
I have an implementation of a layered Bloom filter in [1] (note the layered
branch). This should handle the layering Bloom filter and allow for layers
that
1. Do not become over populated and thus yield too many false positives.
2. Expire and are removed automatically.
The layered Bloom
The link I thought I included did not carry over in the last post. The
paper can be found at:
https://webdocs.cs.ualberta.ca/~drafiei/papers/DupDet06Sigmod.pdf
On Thu, Jun 8, 2023 at 9:05 AM Claude Warren wrote:
>
> Have you considered using Stable Bloom Filters [1]. I think they d
Have you considered using Stable Bloom Filters [1]. I think they do what you
want without a lot of the overhead you propose for your solution. In addition,
you may want to look at Commons-Collections v4.5 [2] (currently snapshot) for
efficient Bloom filter code. I have a Stable Bloom
Claude Warren created KAFKA-14924:
-
Summary: Kafka DOAP file has an error
Key: KAFKA-14924
URL: https://issues.apache.org/jira/browse/KAFKA-14924
Project: Kafka
Issue Type: Bug
37 matches
Mail list logo
