[jira] [Commented] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842717#comment-17842717 ] Hector Geraldino commented on KAFKA-16223: -- Thanks [~chia7712] [~cmukka20] for following up on this. As you've already noticed, the strategy I took was to create a separate [KafkaConfigBackingStoreMockitoTest|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/test/java/org/apache/kafka/connect/storage/KafkaConfigBackingStoreMockitoTest.java] and migrate the tests in batches to facilitate reviews. My original plan was to do this in 3 separate batches, the [first batch|https://github.com/apache/kafka/pull/15520] was merged a few weeks ago, and I have another set of tests already migrated and plan to open the second PR soon. If you guys want to pick up the remaining that's fine by me. I haven't had enough time these past weeks to work on this, but I'd love to see this completed (and the migration to JUnit5 started) before 3.8 > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino reopened KAFKA-16223: -- > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14683) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-14683. -- Reviewer: Greg Harris Resolution: Fixed > Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest > - > > Key: KAFKA-14683 > URL: https://issues.apache.org/jira/browse/KAFKA-14683 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14683) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-14683. -- Reviewer: Greg Harris Resolution: Fixed > Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest > - > > Key: KAFKA-14683 > URL: https://issues.apache.org/jira/browse/KAFKA-14683 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino reopened KAFKA-16223: -- > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-16223. -- Fix Version/s: 3.8.0 Reviewer: Greg Harris Resolution: Fixed > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-16223. -- Fix Version/s: 3.8.0 Reviewer: Greg Harris Resolution: Fixed > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.8.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16358) Update Connect Transformation documentation
[ https://issues.apache.org/jira/browse/KAFKA-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-16358: - Description: When reading the [Kafka Connect docs|https://kafka.apache.org/documentation/#connect_included_transformation] for transformations, there are a few gaps that should be covered: * The Flatten, Cast and TimestampConverter transformations are not listed * HeadersFrom should be HeaderFrom * -InsertHeader is not documented- was: When reading the [Kafka Connect docs|https://kafka.apache.org/documentation/#connect_included_transformation] for transformations, there are a few gaps that should be covered: * The Flatten, Cast and TimestampConverter transformations are not listed * HeadersFrom should be HeaderFrom * InsertHeader is not documented Should be relatively easy to fix > Update Connect Transformation documentation > --- > > Key: KAFKA-16358 > URL: https://issues.apache.org/jira/browse/KAFKA-16358 > Project: Kafka > Issue Type: Bug > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > When reading the [Kafka Connect > docs|https://kafka.apache.org/documentation/#connect_included_transformation] > for transformations, there are a few gaps that should be covered: > * The Flatten, Cast and TimestampConverter transformations are not listed > * HeadersFrom should be HeaderFrom > * -InsertHeader is not documented- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16358) Update Connect Transformation documentation
[ https://issues.apache.org/jira/browse/KAFKA-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino reassigned KAFKA-16358: Assignee: Hector Geraldino > Update Connect Transformation documentation > --- > > Key: KAFKA-16358 > URL: https://issues.apache.org/jira/browse/KAFKA-16358 > Project: Kafka > Issue Type: Bug > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > When reading the [Kafka Connect > docs|https://kafka.apache.org/documentation/#connect_included_transformation] > for transformations, there are a few gaps that should be covered: > * The Flatten, Cast and TimestampConverter transformations are not listed > * HeadersFrom should be HeaderFrom > * InsertHeader is not documented > Should be relatively easy to fix -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16358) Update Connect Transformation documentation
Hector Geraldino created KAFKA-16358: Summary: Update Connect Transformation documentation Key: KAFKA-16358 URL: https://issues.apache.org/jira/browse/KAFKA-16358 Project: Kafka Issue Type: Bug Components: connect Reporter: Hector Geraldino When reading the [Kafka Connect docs|https://kafka.apache.org/documentation/#connect_included_transformation] for transformations, there are a few gaps that should be covered: * The Flatten, Cast and TimestampConverter transformations are not listed * HeadersFrom should be HeaderFrom * InsertHeader is not documented Should be relatively easy to fix -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16358) Update Connect Transformation documentation
Hector Geraldino created KAFKA-16358: Summary: Update Connect Transformation documentation Key: KAFKA-16358 URL: https://issues.apache.org/jira/browse/KAFKA-16358 Project: Kafka Issue Type: Bug Components: connect Reporter: Hector Geraldino When reading the [Kafka Connect docs|https://kafka.apache.org/documentation/#connect_included_transformation] for transformations, there are a few gaps that should be covered: * The Flatten, Cast and TimestampConverter transformations are not listed * HeadersFrom should be HeaderFrom * InsertHeader is not documented Should be relatively easy to fix -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16311) [Debezium Informix Connector Unable to Commit Processed Log Position]
[ https://issues.apache.org/jira/browse/KAFKA-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-16311. -- Resolution: Invalid Please try reporting this to [Debezium|https://github.com/debezium/debezium-connector-informix] https://issues.redhat.com/browse/DBZ > [Debezium Informix Connector Unable to Commit Processed Log Position] > - > > Key: KAFKA-16311 > URL: https://issues.apache.org/jira/browse/KAFKA-16311 > Project: Kafka > Issue Type: Bug >Reporter: Maaheen Yasin >Priority: Blocker > Attachments: connect logs.out > > > I am using Debezium Informix Source connector and JDBC Sink connector and the > below versions of Informix database and KAFKA Connect. > Informix Dynamic Server > 14.10.FC10W1X2 > Informix JDBC Driver for Informix Dynamic Server > 4.50.JC10 > KAFKA Version: 7.4.1-ce > > *Expected Behavior:* > All tasks of the Informix source connector are running, and all messages are > being published in the topic. During the DDL Execution, the informix database > is put under single user mode and the DDL on the table on which CDC was > previously enabled was executed. After the database exits from the single > user mode, then the connector should be able to reconnect with the source > database and be able to publish messages in the topic for each new event. > *Actual Behavior:* > All tasks of the Informix source connector are running, and all messages are > being published in the topic. During the DDL Execution, the database is put > under single user mode and the DDL on the table on which CDC was previously > enabled was executed. After the database exits from the single user mode, the > source connector is able to reconnect with the database, however, no messages > are being published in the topic and the below error is being printed in the > KAFKA Connect Logs. > > *[2024-02-22 15:54:34,913] WARN [kafka_devs|task-0|offsets] Couldn't commit > processed log positions with the source database due to a concurrent > connector shutdown or restart > (io.debezium.connector.common.BaseSourceTask:349)* > > The complete KAFKA Connect logs has been attached. Kindly comment on why this > issue is occurring and what steps should be followed to avoid this issue or > to resolve this issue. > > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16311) [Debezium Informix Connector Unable to Commit Processed Log Position]
[ https://issues.apache.org/jira/browse/KAFKA-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-16311. -- Resolution: Invalid Please try reporting this to [Debezium|https://github.com/debezium/debezium-connector-informix] https://issues.redhat.com/browse/DBZ > [Debezium Informix Connector Unable to Commit Processed Log Position] > - > > Key: KAFKA-16311 > URL: https://issues.apache.org/jira/browse/KAFKA-16311 > Project: Kafka > Issue Type: Bug >Reporter: Maaheen Yasin >Priority: Blocker > Attachments: connect logs.out > > > I am using Debezium Informix Source connector and JDBC Sink connector and the > below versions of Informix database and KAFKA Connect. > Informix Dynamic Server > 14.10.FC10W1X2 > Informix JDBC Driver for Informix Dynamic Server > 4.50.JC10 > KAFKA Version: 7.4.1-ce > > *Expected Behavior:* > All tasks of the Informix source connector are running, and all messages are > being published in the topic. During the DDL Execution, the informix database > is put under single user mode and the DDL on the table on which CDC was > previously enabled was executed. After the database exits from the single > user mode, then the connector should be able to reconnect with the source > database and be able to publish messages in the topic for each new event. > *Actual Behavior:* > All tasks of the Informix source connector are running, and all messages are > being published in the topic. During the DDL Execution, the database is put > under single user mode and the DDL on the table on which CDC was previously > enabled was executed. After the database exits from the single user mode, the > source connector is able to reconnect with the database, however, no messages > are being published in the topic and the below error is being printed in the > KAFKA Connect Logs. > > *[2024-02-22 15:54:34,913] WARN [kafka_devs|task-0|offsets] Couldn't commit > processed log positions with the source database due to a concurrent > connector shutdown or restart > (io.debezium.connector.common.BaseSourceTask:349)* > > The complete KAFKA Connect logs has been attached. Kindly comment on why this > issue is occurring and what steps should be followed to avoid this issue or > to resolve this issue. > > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818856#comment-17818856 ] Hector Geraldino commented on KAFKA-16223: -- One thing that worked for me when migrating the WorkerSinkTaskTest was creating a separate *WorkerSinkTaskMockTest* test class and splitting the migration in smaller batches. Is that something you'd consider? > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817778#comment-17817778 ] Hector Geraldino commented on KAFKA-16223: -- Hey [~cmukka20], I haven't started on this one just yet, as I'm still working on getting https://issues.apache.org/jira/browse/KAFKA-14683 to the finish line. I'm ok with you working on this, but also we can divide an conquer to see if we can land this before/as part of the 3.8.0 release. wdyt? > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17814575#comment-17814575
]
Hector Geraldino commented on KAFKA-14132:
--
[~christo_lolov] I opened a separate Jira ticket to track the migration of
KafkaConfigBackingStoreTest. Maybe this ticket can be marked as done?
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.8.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#00875a}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> # -KafkaConfigBackingStoreTest- KAFKA-16223
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
Hector Geraldino created KAFKA-16223: Summary: Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest Key: KAFKA-16223 URL: https://issues.apache.org/jira/browse/KAFKA-16223 Project: Kafka Issue Type: Sub-task Components: connect Reporter: Hector Geraldino -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hector Geraldino updated KAFKA-14132:
-
Description:
{color:#de350b}Some of the tests below use EasyMock as well. For those migrate
both PowerMock and EasyMock to Mockito.{color}
Unless stated in brackets the tests are in the connect module.
A list of tests which still require to be moved from PowerMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}InReview{color}
{color:#00875a}Merged{color}
# {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
# {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
# {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
# {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
# {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
# {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
# {color:#00875a}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
# -KafkaConfigBackingStoreTest- KAFKA-16223
# {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
([https://github.com/apache/kafka/pull/12418])
# {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
# {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
# {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
# {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
*The coverage report for the above tests after the change should be >= to what
the coverage is now.*
was:
{color:#de350b}Some of the tests below use EasyMock as well. For those migrate
both PowerMock and EasyMock to Mockito.{color}
Unless stated in brackets the tests are in the connect module.
A list of tests which still require to be moved from PowerMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}InReview{color}
{color:#00875a}Merged{color}
# {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
# {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
# {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
# {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
# {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
# {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
# {color:#00875a}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
# KafkaConfigBackingStoreTest
# {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
([https://github.com/apache/kafka/pull/12418])
# {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
# {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
# {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
# {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
*The coverage report for the above tests after the change should be >= to what
the coverage is now.*
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.8.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#00875a}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> #
[jira] [Created] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
Hector Geraldino created KAFKA-16223: Summary: Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest Key: KAFKA-16223 URL: https://issues.apache.org/jira/browse/KAFKA-16223 Project: Kafka Issue Type: Sub-task Components: connect Reporter: Hector Geraldino -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
[ https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino reassigned KAFKA-16223: Assignee: Hector Geraldino > Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest > --- > > Key: KAFKA-16223 > URL: https://issues.apache.org/jira/browse/KAFKA-16223 > Project: Kafka > Issue Type: Sub-task > Components: connect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17814145#comment-17814145
]
Hector Geraldino commented on KAFKA-14132:
--
Hey [~bachmanity1] - as I'm starting to wrap-up the migration of the
{_}WorkerSinkTaskTest{_}, *KafkaConfigBackingStoreTest* will be the only
remaining test pending for migration.
Are you currently working on it? If not, I'm happy to pick this up. If you are,
I can lend a hand and we can divide & conquer. It'd be nice if we can have this
(and KAFKA-12199) ready for 3.8
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.8.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#00875a}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> # KafkaConfigBackingStoreTest (owner: [~bachmanity1])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776375#comment-17776375
]
Hector Geraldino commented on KAFKA-14132:
--
I think we can set the target to KAFKA-14683 as well. Now that KAFKA-14684 has
been merged, my plan is to tackle this last one next.
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.7.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (UNOWNED)
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15248) Add BooleanConverter to Kafka Connect
Hector Geraldino created KAFKA-15248: Summary: Add BooleanConverter to Kafka Connect Key: KAFKA-15248 URL: https://issues.apache.org/jira/browse/KAFKA-15248 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino KIP-959: Add BooleanConverter to Kafka Connect -> https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15248) Add BooleanConverter to Kafka Connect
Hector Geraldino created KAFKA-15248: Summary: Add BooleanConverter to Kafka Connect Key: KAFKA-15248 URL: https://issues.apache.org/jira/browse/KAFKA-15248 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino KIP-959: Add BooleanConverter to Kafka Connect -> https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14809) Connect incorrectly logs that no records were produced by source tasks
[
https://issues.apache.org/jira/browse/KAFKA-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700372#comment-17700372
]
Hector Geraldino commented on KAFKA-14809:
--
No that's perfect. Thanks [~ChrisEgerton]!
> Connect incorrectly logs that no records were produced by source tasks
> --
>
> Key: KAFKA-14809
> URL: https://issues.apache.org/jira/browse/KAFKA-14809
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Hector Geraldino
>Assignee: Hector Geraldino
>Priority: Minor
>
> There's an *{{if}}* condition when [committing
> offsets|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L219]
> that is referencing the wrong variable, so the statement always evaluates to
> {*}true{*}.
> This causes log statements like the following to be spuriously emitted:
> {quote}[2023-03-14 16:18:04,675] DEBUG WorkerSourceTask\{id=job-0} Either no
> records were produced by the task since the last offset commit, or every
> record has been filtered out by a transformation or dropped due to
> transformation or conversion errors.
> (org.apache.kafka.connect.runtime.WorkerSourceTask:220)
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14809) Fix logging conditional on WorkerSourceTask
Hector Geraldino created KAFKA-14809:
Summary: Fix logging conditional on WorkerSourceTask
Key: KAFKA-14809
URL: https://issues.apache.org/jira/browse/KAFKA-14809
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino
There's an *{{if}}* condition when [committing
offsets|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L219]
that is referencing the wrong variable, so the statement always evaluates to
{*}true{*}.
It's a subtle bug, which went undetected probably because its only used to log
information about pending committable offsets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14809) Fix logging conditional on WorkerSourceTask
Hector Geraldino created KAFKA-14809:
Summary: Fix logging conditional on WorkerSourceTask
Key: KAFKA-14809
URL: https://issues.apache.org/jira/browse/KAFKA-14809
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino
There's an *{{if}}* condition when [committing
offsets|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L219]
that is referencing the wrong variable, so the statement always evaluates to
{*}true{*}.
It's a subtle bug, which went undetected probably because its only used to log
information about pending committable offsets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-14059) Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino reassigned KAFKA-14059: Assignee: Hector Geraldino > Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest > --- > > Key: KAFKA-14059 > URL: https://issues.apache.org/jira/browse/KAFKA-14059 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Chris Egerton >Assignee: Hector Geraldino >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684947#comment-17684947
]
Hector Geraldino commented on KAFKA-14132:
--
Perfect. Entered https://issues.apache.org/jira/browse/KAFKA-14683 and
https://issues.apache.org/jira/browse/KAFKA-14684 to track progress
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#ff8b00}KafkaBasedLogTest{color} (owner: [~mdedetrich-aiven])
> # RetryUtilTest (owner: [~mdedetrich-aiven] )
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-14684) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskThreadedTest
Hector Geraldino created KAFKA-14684: Summary: Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskThreadedTest Key: KAFKA-14684 URL: https://issues.apache.org/jira/browse/KAFKA-14684 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hector Geraldino updated KAFKA-14132:
-
Description:
{color:#de350b}Some of the tests below use EasyMock as well. For those migrate
both PowerMock and EasyMock to Mockito.{color}
Unless stated in brackets the tests are in the connect module.
A list of tests which still require to be moved from PowerMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}InReview{color}
{color:#00875a}Merged{color}
# {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
# {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
# {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
# {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
# {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
# {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
# {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
([https://github.com/apache/kafka/pull/12728])
# KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
# {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
([https://github.com/apache/kafka/pull/12418])
# {color:#ff8b00}KafkaBasedLogTest{color} (owner: [~mdedetrich-aiven])
# RetryUtilTest (owner: [~mdedetrich-aiven] )
# {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
# {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
*The coverage report for the above tests after the change should be >= to what
the coverage is now.*
was:
{color:#de350b}Some of the tests below use EasyMock as well. For those migrate
both PowerMock and EasyMock to Mockito.{color}
Unless stated in brackets the tests are in the connect module.
A list of tests which still require to be moved from PowerMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}InReview{color}
{color:#00875a}Merged{color}
# {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
# {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
# {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
# WorkerSinkTaskTest (owner: ??)
# WorkerSinkTaskThreadedTest (owner: ??)
# {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
# {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
# {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
# {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
# {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
([https://github.com/apache/kafka/pull/12728])
# KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
# {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
([https://github.com/apache/kafka/pull/12418])
# {color:#ff8b00}KafkaBasedLogTest{color} (owner: [~mdedetrich-aiven])
# RetryUtilTest (owner: [~mdedetrich-aiven] )
# {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
# {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
*The coverage report for the above tests after the change should be >= to what
the coverage is now.*
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner:
[jira] [Created] (KAFKA-14684) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskThreadedTest
Hector Geraldino created KAFKA-14684: Summary: Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskThreadedTest Key: KAFKA-14684 URL: https://issues.apache.org/jira/browse/KAFKA-14684 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14683) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest
Hector Geraldino created KAFKA-14683: Summary: Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest Key: KAFKA-14683 URL: https://issues.apache.org/jira/browse/KAFKA-14683 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14683) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest
Hector Geraldino created KAFKA-14683: Summary: Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest Key: KAFKA-14683 URL: https://issues.apache.org/jira/browse/KAFKA-14683 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683961#comment-17683961
]
Hector Geraldino commented on KAFKA-14132:
--
Hey [~divijvaidya],
I'm putting together a KIP around
[https://github.com/apache/kafka/pull/13193|https://github.com/apache/kafka/pull/13193,]
in which I'll propose adding new metric counters for filtered (skipped)
records. The POC is code complete, but for it to be ready I need to add new
test cases on *WorkerSinkTaskTest* and *WorkerSinkTaskThreadedTest* unit tests.
These two are assigned to you, and I was wondering if you're OK with me picking
them up, this will unblock my other work. I did the same for
https://issues.apache.org/jira/browse/KAFKA-14659 (had to raise
[https://github.com/apache/kafka/pull/13191] before adding tests for that bug
fix)
Wdyt?
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # WorkerSinkTaskTest (owner: Divij) *WIP*
> # WorkerSinkTaskThreadedTest (owner: Divij) *WIP*
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#ff8b00}KafkaBasedLogTest{color} (owner: [~mdedetrich-aiven])
> # RetryUtilTest (owner: [~mdedetrich-aiven] )
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-14659) source-record-write-[rate|total] metrics include filtered records
[
https://issues.apache.org/jira/browse/KAFKA-14659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hector Geraldino reassigned KAFKA-14659:
Assignee: Hector Geraldino
> source-record-write-[rate|total] metrics include filtered records
> -
>
> Key: KAFKA-14659
> URL: https://issues.apache.org/jira/browse/KAFKA-14659
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Beard
>Assignee: Hector Geraldino
>Priority: Minor
>
> Source tasks in Kafka connect offer two sets of metrics (documented in
> [ConnectMetricsRegistry.java|https://github.com/apache/kafka/blob/72cfc994f5675be349d4494ece3528efed290651/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectMetricsRegistry.java#L173-L191]):
> ||Metric||Description||
> |source-record-poll-rate|The average per-second number of records
> produced/polled (before transformation) by this task belonging to the named
> source connector in this worker.|
> |source-record-write-rate|The average per-second number of records output
> from the transformations and written to Kafka for this task belonging to the
> named source connector in this worker. This is after transformations are
> applied and excludes any records filtered out by the transformations.|
> There are also corresponding "-total" metrics that capture the total number
> of records polled and written for the metrics above, respectively.
> In short, the "poll" metrics capture the number of messages sourced
> pre-transformation/filtering, and the "write" metrics should capture the
> number of messages ultimately written to Kafka post-transformation/filtering.
> However, the implementation of the {{source-record-write-*}} metrics
> _includes_ records filtered out by transformations (and also records that
> result in produce failures with the config {{{}errors.tolerance=all{}}}).
> h3. Details
> In
> [AbstractWorkerSourceTask.java|https://github.com/apache/kafka/blob/a382acd31d1b53cd8695ff9488977566083540b1/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L389-L397],
> each source record is passed through the transformation chain where it is
> potentially filtered out, checked to see if it was in fact filtered out, and
> if so it is accounted for in the internal metrics via
> {{{}counter.skipRecord(){}}}.
> {code:java}
> for (final SourceRecord preTransformRecord : toSend) {
> retryWithToleranceOperator.sourceRecord(preTransformRecord);
> final SourceRecord record =
> transformationChain.apply(preTransformRecord);
> final ProducerRecord producerRecord =
> convertTransformedRecord(record);
> if (producerRecord == null || retryWithToleranceOperator.failed()) {
>
> counter.skipRecord();
> recordDropped(preTransformRecord);
> continue;
> }
> ...
> {code}
> {{SourceRecordWriteCounter.skipRecord()}} is implemented as follows:
> {code:java}
>
> public SourceRecordWriteCounter(int batchSize, SourceTaskMetricsGroup
> metricsGroup) {
> assert batchSize > 0;
> assert metricsGroup != null;
> this.batchSize = batchSize;
> counter = batchSize;
> this.metricsGroup = metricsGroup;
> }
> public void skipRecord() {
> if (counter > 0 && --counter == 0) {
> finishedAllWrites();
> }
> }
>
> private void finishedAllWrites() {
> if (!completed) {
> metricsGroup.recordWrite(batchSize - counter);
> completed = true;
> }
> }
> {code}
> For example: If a batch starts with 100 records, {{batchSize}} and
> {{counter}} will both be initialized to 100. If all 100 records get filtered
> out, {{counter}} will be decremented 100 times, and
> {{{}finishedAllWrites(){}}}will record the value 100 to the underlying
> {{source-record-write-*}} metrics rather than 0, the correct value according
> to the documentation for these metrics.
> h3. Solutions
> Assuming the documentation correctly captures the intent of the
> {{source-record-write-*}} metrics, it seems reasonable to fix these metrics
> such that filtered records do not get counted.
> It may also be useful to add additional metrics to capture the rate and total
> number of records filtered out by transformations, which would require a KIP.
> I'm not sure what the best way of accounting for produce failures in the case
> of {{errors.tolerance=all}} is yet. Maybe these failures deserve their own
> new metrics?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14060) Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683418#comment-17683418 ] Hector Geraldino commented on KAFKA-14060: -- Hey [~ChrisEgerton], I wrote a patch for https://issues.apache.org/jira/browse/KAFKA-14659, but when I was about to write tests for it I found out that this test was still using PowerMock/EasyMock, which is a blocker. I assigned the ticket to myself, and plan to raise a PR in a day or two. > Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest > --- > > Key: KAFKA-14060 > URL: https://issues.apache.org/jira/browse/KAFKA-14060 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Chris Egerton >Assignee: Hector Geraldino >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-14060) Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino reassigned KAFKA-14060: Assignee: Hector Geraldino (was: Chris Egerton) > Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest > --- > > Key: KAFKA-14060 > URL: https://issues.apache.org/jira/browse/KAFKA-14060 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Chris Egerton >Assignee: Hector Geraldino >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14651) Add connectorDeleted flag when stopping tasks
[ https://issues.apache.org/jira/browse/KAFKA-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14651: - Description: Jira ticket for [KIP-901|https://cwiki.apache.org/confluence/display/KAFKA/KIP-901%3A+Add+connectorDeleted+flag+when+stopping+tasks] It would be useful for Connectors to know when its instance is being deleted. This will give a chance to connector tasks to perform any cleanup routines before as part of the connector removal process. was:It would be useful for Connectors to know when its instance is being deleted. This will give a chance to connectors to perform any cleanup tasks (e.g. deleting external resources, or deleting offsets) before the connector is completely removed from the cluster. > Add connectorDeleted flag when stopping tasks > - > > Key: KAFKA-14651 > URL: https://issues.apache.org/jira/browse/KAFKA-14651 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > Jira ticket for > [KIP-901|https://cwiki.apache.org/confluence/display/KAFKA/KIP-901%3A+Add+connectorDeleted+flag+when+stopping+tasks] > It would be useful for Connectors to know when its instance is being deleted. > This will give a chance to connector tasks to perform any cleanup routines > before as part of the connector removal process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14651) Add connectorDeleted flag when stopping tasks
Hector Geraldino created KAFKA-14651: Summary: Add connectorDeleted flag when stopping tasks Key: KAFKA-14651 URL: https://issues.apache.org/jira/browse/KAFKA-14651 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino It would be useful for Connectors to know when its instance is being deleted. This will give a chance to connectors to perform any cleanup tasks (e.g. deleting external resources, or deleting offsets) before the connector is completely removed from the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14651) Add connectorDeleted flag when stopping tasks
Hector Geraldino created KAFKA-14651: Summary: Add connectorDeleted flag when stopping tasks Key: KAFKA-14651 URL: https://issues.apache.org/jira/browse/KAFKA-14651 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Hector Geraldino Assignee: Hector Geraldino It would be useful for Connectors to know when its instance is being deleted. This will give a chance to connectors to perform any cleanup tasks (e.g. deleting external resources, or deleting offsets) before the connector is completely removed from the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14354) Add 'isDeleted' parameter when stopping a Connector
[ https://issues.apache.org/jira/browse/KAFKA-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14354: - Description: It would be useful for Connectors to know when its instance is being deleted. This will give a chance to connectors to perform any cleanup tasks (e.g. deleting external resources, or deleting offsets) before the connector is completely removed from the cluster. (was: It would be useful to have a callback method added to the Connector API, so connectors extending the SourceConnector and SinkConnector classes can be notified when their connector instance is being deleted. This will give a chance to connectors to perform any cleanup tasks (e.g. deleting external resources, or deleting offsets) before the connector is completely removed from the cluster.) > Add 'isDeleted' parameter when stopping a Connector > --- > > Key: KAFKA-14354 > URL: https://issues.apache.org/jira/browse/KAFKA-14354 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > It would be useful for Connectors to know when its instance is being deleted. > This will give a chance to connectors to perform any cleanup tasks (e.g. > deleting external resources, or deleting offsets) before the connector is > completely removed from the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14354) Add 'isDeleted' parameter when stopping a Connector
[ https://issues.apache.org/jira/browse/KAFKA-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14354: - Summary: Add 'isDeleted' parameter when stopping a Connector (was: Add 'destroyed()' callback method to Connector API) > Add 'isDeleted' parameter when stopping a Connector > --- > > Key: KAFKA-14354 > URL: https://issues.apache.org/jira/browse/KAFKA-14354 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > It would be useful to have a callback method added to the Connector API, so > connectors extending the SourceConnector and SinkConnector classes can be > notified when their connector instance is being deleted. This will give a > chance to connectors to perform any cleanup tasks (e.g. deleting external > resources, or deleting offsets) before the connector is completely removed > from the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14354) Add 'destroyed()' callback method to Connector API
[ https://issues.apache.org/jira/browse/KAFKA-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14354: - Summary: Add 'destroyed()' callback method to Connector API (was: Add delete callback method to Connector API) > Add 'destroyed()' callback method to Connector API > -- > > Key: KAFKA-14354 > URL: https://issues.apache.org/jira/browse/KAFKA-14354 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > It would be useful to have a callback method added to the Connector API, so > connectors extending the SourceConnector and SinkConnector classes can be > notified when their connector instance is being deleted. This will give a > chance to connectors to perform any cleanup tasks (e.g. deleting external > resources, or deleting offsets) before the connector is completely removed > from the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14354) Add delete callback method to Connector API
[ https://issues.apache.org/jira/browse/KAFKA-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14354: - Description: It would be useful to have a callback method added to the Connector API, so connectors extending the SourceConnector and SinkConnector classes can be notified when their connector instance is being deleted. This will give a chance to connectors to perform any cleanup tasks (e.g. deleting external resources, or deleting offsets) before the connector is completely removed from the cluster. (was: KIP-795: https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries) > Add delete callback method to Connector API > --- > > Key: KAFKA-14354 > URL: https://issues.apache.org/jira/browse/KAFKA-14354 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > It would be useful to have a callback method added to the Connector API, so > connectors extending the SourceConnector and SinkConnector classes can be > notified when their connector instance is being deleted. This will give a > chance to connectors to perform any cleanup tasks (e.g. deleting external > resources, or deleting offsets) before the connector is completely removed > from the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14354) Add delete callback method to Connector API
[ https://issues.apache.org/jira/browse/KAFKA-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14354: - Priority: Minor (was: Major) > Add delete callback method to Connector API > --- > > Key: KAFKA-14354 > URL: https://issues.apache.org/jira/browse/KAFKA-14354 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Minor > > KIP-795: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14354) Add delete callback method to Connector API
Hector Geraldino created KAFKA-14354: Summary: Add delete callback method to Connector API Key: KAFKA-14354 URL: https://issues.apache.org/jira/browse/KAFKA-14354 Project: Kafka Issue Type: Improvement Components: clients Reporter: Hector Geraldino Assignee: Hector Geraldino KIP-795: https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14354) Add delete callback method to Connector API
[ https://issues.apache.org/jira/browse/KAFKA-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-14354: - Component/s: KafkaConnect (was: clients) > Add delete callback method to Connector API > --- > > Key: KAFKA-14354 > URL: https://issues.apache.org/jira/browse/KAFKA-14354 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Major > > KIP-795: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14354) Add delete callback method to Connector API
Hector Geraldino created KAFKA-14354: Summary: Add delete callback method to Connector API Key: KAFKA-14354 URL: https://issues.apache.org/jira/browse/KAFKA-14354 Project: Kafka Issue Type: Improvement Components: clients Reporter: Hector Geraldino Assignee: Hector Geraldino KIP-795: https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-13434) Add a public API for AbstractCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-13434. -- Resolution: Won't Do KIP has been discarded > Add a public API for AbstractCoordinator > > > Key: KAFKA-13434 > URL: https://issues.apache.org/jira/browse/KAFKA-13434 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Major > > KIP-795: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-13434) Add a public API for AbstractCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino resolved KAFKA-13434. -- Resolution: Won't Do KIP has been discarded > Add a public API for AbstractCoordinator > > > Key: KAFKA-13434 > URL: https://issues.apache.org/jira/browse/KAFKA-13434 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Major > > KIP-795: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13521) Supress changelog schema version breaks migration
[
https://issues.apache.org/jira/browse/KAFKA-13521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hector Geraldino updated KAFKA-13521:
-
Issue Type: Bug (was: Improvement)
> Supress changelog schema version breaks migration
> -
>
> Key: KAFKA-13521
> URL: https://issues.apache.org/jira/browse/KAFKA-13521
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0, 2.5.0, 2.5.1
>Reporter: Hector Geraldino
>Priority: Major
>
> Hi,
> We recently updated the kafka-streams library in one of our apps from v2.5.0
> to v2.5.1. This upgrade changes the header format of the state store for
> suppress changelog topics (see
> https://issues.apache.org/jira/browse/KAFKA-10173 and
> [https://github.com/apache/kafka/pull/8905)]
> What we noticed was that, introducing a new version on the binary schema
> header breaks older clients. I.e. applications running on v2.5.1 can parse
> the v3, v2, v1 and 0 headers, while the ones running on 2.5.0 (and, I assume,
> previous versions) cannot read headers in v3 format.
> The logged exception is:
>
> {code:java}
> java.lang.IllegalArgumentException: Restoring apparently invalid changelog
> record: ConsumerRecord(topic =
> msgequator-kfns-msgequator-msgequator-suppress-buffer-store-changelog,
> partition = 8, leaderEpoch = 405, offset = 711400430, CreateTime =
> 1638828473341, serialized key size = 32, serialized value size = 90, headers
> = RecordHeaders(headers = [RecordHeader(key = v, value = [3])], isReadOnly =
> false), key = [B@5cf0e540, value = [B@40abc004)
> at
> org.apache.kafka.streams.state.internals.InMemoryTimeOrderedKeyValueBuffer.restoreBatch(InMemoryTimeOrderedKeyValueBuffer.java:372)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:92)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:350)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:94)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:401)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:779)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
> ~[msgequator-1.59.3.jar:1.59.3]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)
> ~[msgequator-1.59.3.jar:1.59.3] {code}
>
>
> There's obviously no clear solution for this other than stopping/starting all
> instances at once. A rolling bounce that takes some time to complete (in our
> case, days) will break instances that haven't been upgraded yet after a
> rebalance that causes older clients to pick up the newly encoded changelog
> partition(s)
>
> I don't know if adding a flag on the client side that lists the supported
> protocol versions (so it behaves like Kafka Consumers when picking the
> rebalance protocol - cooperative or eager), or if it just needs to be
> explicitly stated on the migration guide that a full stop/start migration is
> required in cases where the protocol version changes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (KAFKA-13521) Supress changelog schema version breaks migration
Hector Geraldino created KAFKA-13521:
Summary: Supress changelog schema version breaks migration
Key: KAFKA-13521
URL: https://issues.apache.org/jira/browse/KAFKA-13521
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.5.1, 2.5.0, 2.4.0
Reporter: Hector Geraldino
Hi,
We recently updated the kafka-streams library in one of our apps from v2.5.0 to
v2.5.1. This upgrade changes the header format of the state store for suppress
changelog topics (see https://issues.apache.org/jira/browse/KAFKA-10173 and
[https://github.com/apache/kafka/pull/8905)]
What we noticed was that, introducing a new version on the binary schema header
breaks older clients. I.e. applications running on v2.5.1 can parse the v3, v2,
v1 and 0 headers, while the ones running on 2.5.0 (and, I assume, previous
versions) cannot read headers in v3 format.
The logged exception is:
{code:java}
java.lang.IllegalArgumentException: Restoring apparently invalid changelog
record: ConsumerRecord(topic =
msgequator-kfns-msgequator-msgequator-suppress-buffer-store-changelog,
partition = 8, leaderEpoch = 405, offset = 711400430, CreateTime =
1638828473341, serialized key size = 32, serialized value size = 90, headers =
RecordHeaders(headers = [RecordHeader(key = v, value = [3])], isReadOnly =
false), key = [B@5cf0e540, value = [B@40abc004)
at
org.apache.kafka.streams.state.internals.InMemoryTimeOrderedKeyValueBuffer.restoreBatch(InMemoryTimeOrderedKeyValueBuffer.java:372)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:92)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:350)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:94)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:401)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:779)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)
~[msgequator-1.59.3.jar:1.59.3] {code}
There's obviously no clear solution for this other than stopping/starting all
instances at once. A rolling bounce that takes some time to complete (in our
case, days) will break instances that haven't been upgraded yet after a
rebalance that causes older clients to pick up the newly encoded changelog
partition(s)
I don't know if adding a flag on the client side that lists the supported
protocol versions (so it behaves like Kafka Consumers when picking the
rebalance protocol - cooperative or eager), or if it just needs to be
explicitly stated on the migration guide that a full stop/start migration is
required in cases where the protocol version changes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (KAFKA-13521) Supress changelog schema version breaks migration
Hector Geraldino created KAFKA-13521:
Summary: Supress changelog schema version breaks migration
Key: KAFKA-13521
URL: https://issues.apache.org/jira/browse/KAFKA-13521
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.5.1, 2.5.0, 2.4.0
Reporter: Hector Geraldino
Hi,
We recently updated the kafka-streams library in one of our apps from v2.5.0 to
v2.5.1. This upgrade changes the header format of the state store for suppress
changelog topics (see https://issues.apache.org/jira/browse/KAFKA-10173 and
[https://github.com/apache/kafka/pull/8905)]
What we noticed was that, introducing a new version on the binary schema header
breaks older clients. I.e. applications running on v2.5.1 can parse the v3, v2,
v1 and 0 headers, while the ones running on 2.5.0 (and, I assume, previous
versions) cannot read headers in v3 format.
The logged exception is:
{code:java}
java.lang.IllegalArgumentException: Restoring apparently invalid changelog
record: ConsumerRecord(topic =
msgequator-kfns-msgequator-msgequator-suppress-buffer-store-changelog,
partition = 8, leaderEpoch = 405, offset = 711400430, CreateTime =
1638828473341, serialized key size = 32, serialized value size = 90, headers =
RecordHeaders(headers = [RecordHeader(key = v, value = [3])], isReadOnly =
false), key = [B@5cf0e540, value = [B@40abc004)
at
org.apache.kafka.streams.state.internals.InMemoryTimeOrderedKeyValueBuffer.restoreBatch(InMemoryTimeOrderedKeyValueBuffer.java:372)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:92)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:350)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:94)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:401)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:779)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
~[msgequator-1.59.3.jar:1.59.3]
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)
~[msgequator-1.59.3.jar:1.59.3] {code}
There's obviously no clear solution for this other than stopping/starting all
instances at once. A rolling bounce that takes some time to complete (in our
case, days) will break instances that haven't been upgraded yet after a
rebalance that causes older clients to pick up the newly encoded changelog
partition(s)
I don't know if adding a flag on the client side that lists the supported
protocol versions (so it behaves like Kafka Consumers when picking the
rebalance protocol - cooperative or eager), or if it just needs to be
explicitly stated on the migration guide that a full stop/start migration is
required in cases where the protocol version changes.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (KAFKA-13434) Add a public API for AbstractCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-13434: - Description: KIP-795: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator|https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for+AbstractCoordinator] The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries was: KIP-784: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for+AbstractCoordinator] The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries > Add a public API for AbstractCoordinator > > > Key: KAFKA-13434 > URL: https://issues.apache.org/jira/browse/KAFKA-13434 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Major > > KIP-795: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator|https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for+AbstractCoordinator] > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KAFKA-13434) Add a public API for AbstractCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-13434: - Description: KIP-795: https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries was: KIP-795: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator|https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for+AbstractCoordinator] The AbstractCoordinator should have a companion public interface that is part of Kafka's public API, so backwards compatibility can be maintained in future versions of the client libraries > Add a public API for AbstractCoordinator > > > Key: KAFKA-13434 > URL: https://issues.apache.org/jira/browse/KAFKA-13434 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Major > > KIP-795: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (KAFKA-13434) Add a public API for AbstractCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Geraldino updated KAFKA-13434: - Summary: Add a public API for AbstractCoordinator (was: Add a public API for AbstractCoordinatos) > Add a public API for AbstractCoordinator > > > Key: KAFKA-13434 > URL: https://issues.apache.org/jira/browse/KAFKA-13434 > Project: Kafka > Issue Type: Improvement > Components: clients >Reporter: Hector Geraldino >Assignee: Hector Geraldino >Priority: Major > > KIP-784: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+public+APIs+for+AbstractCoordinator] > The AbstractCoordinator should have a companion public interface that is part > of Kafka's public API, so backwards compatibility can be maintained in future > versions of the client libraries -- This message was sent by Atlassian Jira (v8.3.4#803005)
