Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2006

[Apache Jenkins Server]

See 

Re: Subscribe to mailing list

[Josep Prat]

Hi Mital,
Thanks for your interest in helping out with Apache Kafka.
You can find the instructions on how to subscribe to the mailing list here:
https://kafka.apache.org/contact.html

|>
*Developer mailing list: A list for discussion on Kafka® development. To
subscribe, send an email to dev-subscr...@kafka.apache.org
. Once subscribed, you can have discussion
on Kafka® development by mailing to dev@kafka.apache.org
. Archives are available here
. To unsubscribe,
send an email to dev-unsubscr...@kafka.apache.org
.*

Best,
———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Sun, Jul 16, 2023, 15:54 Mital Awachat  wrote:

> Hi,
>
> I would like to subscribe to the mailing list.
>
> --
> Regards
> Mital Awachat
>

Re: [DISCUSS] KIP-944 Support async runtimes in consumer

[Erik van Oosten]

Hi Colin, Philip, Kirk,

As far as I am aware all concerns about KIP-944 have been addressed. 
Including those about dirty reads between threads and green threads 
because of missing memory barriers. If you agree, I would like to open 
the vote again. If not, please let me know.


I'll open another KIP with a proposal on how to improve the consumer API 
so that we don't need any thread trickery anymore. I would rather not 
wait for that one because there will be a lot work before that can even 
be implemented.


Once KIP-944 has been accepted, I'll work on adding the unit tests that 
are described in the KIP.


Kind regards,
    Erik.


Op 30-06-2023 om 07:56 schreef Erik van Oosten:

[This is a resend with the correct KIP number.]

Hello developers of the Java based consumer,

I submitted https://github.com/apache/kafka/pull/13914 to fix a long 
standing problem that the Kafka consumer on the JVM is not usable from 
asynchronous runtimes such as Kotlin co-routines and ZIO. However, 
since it extends the public API I was requested to create a KIP.


So here it is:
KIP-944 Support async runtimes in consumer 
https://cwiki.apache.org/confluence/x/chw0Dw


Any questions, comments, ideas and other additions are welcome!

The KIP should be complete except for the testing section. As far as I 
am aware there are no tests for the current behavior. Any help in this 
area would be appreciated.


Kind regards,
    Erik.



--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com

Application for permission grants for contribution

[abhinav tripathi]

Hi Team,
I would like to contribute to the Kafka community. I want to work on open
issues and also have some ideas of my own.
Requesting access and grants required for myself.

My credentials:

JIRA username : shikamaru-96
Cwiki username: shikamaru-96

Email: imabhinav.tripa...@gmail.com

Thanks,
Abhinav

Subscribe to mailing list

[Mital Awachat]

Hi,

I would like to subscribe to the mailing list.

-- 
Regards
Mital Awachat

Requesting permissions to contribute to Apache Kafka

[Mital Awachat]

Hi Team,

I am requesting permission to contribute to Apache Kafka.

Wiki:
Username: mital.awachat
Email: mital.awacha...@gmail.com

Jira:
Username: mital.awachat
Email: mital.awacha...@gmail.com
Using https://selfserve.apache.org/jira-account.html

-- 
Regards
Mital Awachat

Request for review of my PR

[Owen Leung]

Hi there,

Can I ask for review for my PR below ? It's been a few weeks since I last
heard back from you guys.

https://github.com/apache/kafka/pull/13773

Thanks!
Owen

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #43

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 561134 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-8: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-7: SMOKE-TEST-CLIENT-CLOSED
streams-8: SMOKE-TEST-CLIENT-CLOSED
streams-7: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-6: SMOKE-TEST-CLIENT-CLOSED
streams-6: SMOKE-TEST-CLIENT-CLOSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':streams:upgrade-system-tests-23:unitTest'.
> Process 'Gradle Test Executor 15' finished with non-zero exit value 1
  This problem might be caused by incorrect test process configuration.
  Please refer to the test execution section in the User Manual at 
https://docs.gradle.org/8.0.2/userguide/java_testing.html#sec:test_execution

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 2h 53m 27s
230 actionable tasks: 124 executed, 106 up-to-date

See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.5/build/reports/profile/profile-2023-07-16-09-49-19.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 8 and Scala 2.13

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
---
* What 

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

[Omnia Ibrahim]

Thanks Claude for the feedback and the raising this implementation to
Apache commons-collections.
I had a look into your layered bloom filter and at first glance, I think it
would be a better improvement, however, regarding the following suggestion

> By hashing the principal and PID into the filter as a single hash only
one Bloom filter is required.

I am not sure how this will work when we have different producer-id-rate
for different KafkaPrincipal as proposed in the KIP.
For example `userA` had producer-id-rate of 1000 per hour while `user2` has
a quota of 100 producer ids per hour. How will we configure the max entries
for the Shape?

The only thing that comes to my mind to maintain this desired behavior in
the KIP is to NOT hash PID with KafkaPrincipal and keep a
Map
then each one of these bloom filters is controlled with
`Shape(, 0.1)`.

Does that make sense? WDYT?

Also regarding the eviction function
> (evict function)
> The evict function will determine if it has been a minute since the last
> time we created a new layer, if so create a new layer and evict all layers
> older than 1 hour.  Since the layers are a time ordered list this is
simply
> removing the elderly layers from the front of the list.

Maybe am missing something here but I can't find anything in the
`LayerManager` code that point to how often will the eviction function
runs. Do you mean that the eviction function runs every minute? If so, can
we control this?

Cheers,
Omnia

On Wed, Jun 21, 2023 at 11:43 AM Claude Warren  wrote:

> I think that the either using a Stable bloom filter or a Layered bloom
> filter constructed as follows:
>
>
>- Each layer is configured for the maximum number of principal-PID pairs
>expected in a single minute.
>- Expect 60 layers (one for each minute)
>- If the layer becomes fully populated add another layer.
>- When the time to insert into the current layer expires, remove the
>layers that are older than an hour.
>
> This will provide a sliding window of one hour with the ability to handle
> bursts above the expected rate of inserts without additional false
> positives.
>
> By hashing the principal and PID into the filter as a single hash only one
> Bloom filter is required.
>
> The code I pointed to earlier uses the common-collections4 Bloom filter
> implementation.  So a rough pseudo code for the implementation is:
>
> Shape shap = Shape.fromNP( 1000, 0.1 ); // 1000 entries, 0.1 false positive
> rate
> LayeredBloomFilter filter = LayeredBloomFilter( shape, 60, evictFunc ); //
> 60 initial layers, eviction function.
>
> (on PID)
>
> long[2] buff = Murmur3Hash.hash128x64( String.format("%s%s", principal, PID
> ).getBytes(java.nio.charset.Charset.UTF8));
> Hasher hasher = new EnhancedDoubleHasher( buff[0], buff[1] );
> if (filter.contains(hasher)) {
> // handle case where principal-pid was already seen
> }
> filter.merge( hasher ); // ensures that principal-pid remains in seen for
> the next hour.
>
> (evict function)
> The evict function will determine if it has been a minute since the last
> time we created a new layer, if so create a new layer and evict all layers
> older than 1 hour.  Since the layers are a time ordered list this is simply
> removing the elderly layers from the front of the list.
>
> if it has not been an hour and the current filter is full (e.g. has 1000
> entries) create a new layer
>
> This should be very fast and space efficient.
>
>
> On Wed, Jun 21, 2023 at 11:13 AM Claude Warren  wrote:
>
> > I have an implementation of a layered Bloom filter in [1] (note the
> > layered branch).  This should handle the layering Bloom filter and allow
> > for layers that
> >
> >1. Do not become over populated and thus yield too many false
> >positives.
> >2. Expire and are removed automatically.
> >
> > The layered Bloom filter also does not need another thread to remove the
> > old filters as this is amortized across all the inserts.
> >
> > In addition, the number of Layered Bloom filter instances can be reduced
> > by hashing the Kafka Principal and the ID together into the Bloom filter
> to
> > look for.
> >
> > [1] https://github.com/Claudenw/BloomFilters/tree/layered
> >
> > On Sun, Jun 18, 2023 at 10:18 PM Omnia Ibrahim 
> > wrote:
> >
> >> Hi Haruki, Thanks for having a look at the KIP.
> >> > 1. Do you have any memory-footprint estimation for
> >> TimeControlledBloomFilter?
> >> I don't at the moment have any estimate as I don't have a full
> >> implementation of this one at the moment. I can work on one if it's
> >> required.
> >>
> >> > * If I read the KIP correctly, TimeControlledBloomFilter will be
> >> > allocated per KafkaPrincipal so the size should be reasonably small
> >> > considering clusters which have a large number of users.
> >> The Map stored in the cache has 2 dimensions one is vertical which is
> >> KafkaPrincipal (producers only) and the second horizontal which is the
> >> time
> >> of the windows.
> >> - Horizontally we 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2005

[Apache Jenkins Server]

See