[jira] [Commented] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833804#comment-17833804 ] Aviral Srivastava commented on KAFKA-12506: --- [~ableegoldman] , I want to work on this. I am sorry this was abandoned. I am back. Thank you! > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Assignee: Arpit Goyal >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324391#comment-17324391 ] Aviral Srivastava commented on KAFKA-12506: --- Hi [~ableegoldman], upon running the command: `./gradlew streams:test`, I am getting a successful build: ``` Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 13m 31s 46 actionable tasks: 13 executed, 33 up-to-date ``` Full log: https://docs.google.com/document/d/1NwnEMFd926zbfM56fs54vjcFnqPXW5x_Rr2C8Qg4kvQ/edit?usp=sharing > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Assignee: Aviral Srivastava >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317457#comment-17317457 ] Aviral Srivastava commented on KAFKA-12506: --- Hey [~ableegoldman], actually I always rebase and run the complete tests. They pass and if they fail, I can see the reason but in here, it is hard to determine the failure as I wrote in my last comment - `This problem might be caused by incorrect test process configuration.` > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Assignee: Aviral Srivastava >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315899#comment-17315899 ] Aviral Srivastava commented on KAFKA-12506: --- Hi [~ableegoldman] , a test is failing due to wrong configurations and I was not able to pull up the file (as it is a local remote file on the server) The failure reason mentioned [here]([https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-10432/7/pipeline/)] is as follows: ``` > Process 'Gradle Test Executor 55' finished with non-zero exit value 134 [|https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-10432/7/pipeline/#step-157-log-1556][2021-04-06T20:20:47.883Z] This problem might be caused by incorrect test process configuration. [|https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-10432/7/pipeline/#step-157-log-1557][2021-04-06T20:20:47.883Z] Please refer to the test execution section in the User Manual at ``` I ran the exact Gradle commands mentioned in the Jenkins build as well as in the official home page documentation(readme of the main repo) multiple times before pushing the last commit. Can you help me debug here? Or any resourceful reading that you think I might have to go through? Thanks in advance! > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Assignee: Aviral Srivastava >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-12521) Expand coverage of integration tests with more realistic applications
[ https://issues.apache.org/jira/browse/KAFKA-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aviral Srivastava reassigned KAFKA-12521: - Assignee: Aviral Srivastava > Expand coverage of integration tests with more realistic applications > - > > Key: KAFKA-12521 > URL: https://issues.apache.org/jira/browse/KAFKA-12521 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: Aviral Srivastava >Priority: Major > Labels: integration-tests, newbie++ > > Many of the integration tests in Kafka Streams run a minimal topology, often > just consuming a single input topic without doing any processing or producing > to an output topic. Some of the complex concurrency bugs that we've found > only showed up when we had some actual data to process and a stateful > topology: [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] and > [KAFKA-12500|https://issues.apache.org/jira/browse/KAFKA-12500] > We should try to expand the Streams applications in these test to really flex > the tested feature or general stability of streams. These tests should > # Produce some data to the input topic so Streams will actually have > something to consume > # Include one or more stateful operators in the topology, with caching enabled > # Insert a repartition, or force one with a key-changing operation, to get > multiple subtopologies > # Write out some results to an output topic > # Verify the results in the output topic > We don't necessarily need to do all of these at the same time, any > incremental improvement would help. Perhaps we could even write a standard > framework for integration tests that includes all of the above, so we need > only plug our test into the framework when designing new integration tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310085#comment-17310085 ] Aviral Srivastava edited comment on KAFKA-12506 at 3/27/21, 10:22 PM: -- Hi [~ableegoldman], I have created a new [PR| https://github.com/apache/kafka/pull/10419] and waiting for your remarks. was (Author: kebab-mai-haddi): Hi [~ableegoldman], I have created a new PR and waiting for your remarks. > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Assignee: Aviral Srivastava >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310085#comment-17310085 ] Aviral Srivastava commented on KAFKA-12506: --- Hi [~ableegoldman], I have created a new PR and waiting for your remarks. > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Assignee: Aviral Srivastava >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12506) Expand AdjustStreamThreadCountTest
[ https://issues.apache.org/jira/browse/KAFKA-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306648#comment-17306648 ] Aviral Srivastava commented on KAFKA-12506: --- [~ableegoldman], hi! I am new here and would like to contribute to Kafka starting with this issue ( as it seems interesting and is tagged as a newbie). Can I start working on it? Thanks! > Expand AdjustStreamThreadCountTest > -- > > Key: KAFKA-12506 > URL: https://issues.apache.org/jira/browse/KAFKA-12506 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: A. Sophie Blee-Goldman >Priority: Major > Labels: newbie, newbie++ > > Right now the AdjustStreamThreadCountTest runs a minimal topology that just > consumes a single input topic, and doesn't produce any data to this topic. > Some of the complex concurrency bugs that we've found only showed up when we > had some actual data to process and a stateful topology: > [KAFKA-12503|https://issues.apache.org/jira/browse/KAFKA-12503] KAFKA-12500 > See the umbrella ticket for the list of improvements we need to make this a > more effective integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12517) Kafka's readme should provide Run configurations in the BUILD with IDE Section
Aviral Srivastava created KAFKA-12517: - Summary: Kafka's readme should provide Run configurations in the BUILD with IDE Section Key: KAFKA-12517 URL: https://issues.apache.org/jira/browse/KAFKA-12517 Project: Kafka Issue Type: Improvement Components: build Reporter: Aviral Srivastava I want to contribute to Kafka and for that, I want to set up IntelliJ IDEA for Kafka so that I can debug and run it(Kafka) using an IDE. However, I am not able to configure my "Run" settings and the documentation does not contain any such information. I dived deeper into how Kafka is normally executed and could not deduce the basics such as Classpath, etc. I have asked a specific question regarding the same here: [https://stackoverflow.com/questions/66739363/where-is-the-main-class-of-kafka-located] I am ready to contribute (add Run configs to the configuration) but I need some help from the community in setting up my own IDE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
