[jira] [Commented] (KAFKA-17584) Fix incorrect synonym handling for dynamic log configurations
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895570#comment-17895570
]
Christo Lolov commented on KAFKA-17584:
---
Heya [~mjsax], I have a preference for this to make it in 3.7.2. I have made an
attempt to port the changes https://github.com/apache/kafka/pull/17696, but I
would like reviews from [~cmccabe] (author of the original fix) and [~showuon]
(ported the fix to 3.8)
> Fix incorrect synonym handling for dynamic log configurations
> -
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
> Fix For: 3.9.0, 3.8.1
>
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the default value (source code)
> for log.retention.ms. This poses a durability issue if users have set their
> retention by using log.retention.hours or log.retention.minutes. In other
> words, if a user has set log.retention.hours=-1 (infinite retention) and they
> dynamically change `message.max.bytes` their retention will immediately
> change back to the default of 60480 ms (7 days) and data before this will
> be scheduled for deletion immediately.
> Steps to reproduce:
> 1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> 2. Start a single ZK or KRaft instance + a single Kafka instance
> 3. Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> 4. Create a few segments with the console producer
> 5. Observe that they are deleted after 1 minute
> 6. Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> 7. Create a few more segments with the console producer
> 8. Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-17584) Fix incorrect synonym handling for dynamic log configurations
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894524#comment-17894524
]
Christo Lolov commented on KAFKA-17584:
---
[~mjsax] Apologies for the delay, I was taking some time off from computers :).
At the time of discovering the bug 3.9.0 was undergoing its release candidates,
hence I put 3.9.0 as affected version. As Colin correctly mentioned, this is
present in all Kafka versions at least since 3.0.
> Fix incorrect synonym handling for dynamic log configurations
> -
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.9.0
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
> Fix For: 3.9.0, 3.8.1
>
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the default value (source code)
> for log.retention.ms. This poses a durability issue if users have set their
> retention by using log.retention.hours or log.retention.minutes. In other
> words, if a user has set log.retention.hours=-1 (infinite retention) and they
> dynamically change `message.max.bytes` their retention will immediately
> change back to the default of 60480 ms (7 days) and data before this will
> be scheduled for deletion immediately.
> Steps to reproduce:
> 1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> 2. Start a single ZK or KRaft instance + a single Kafka instance
> 3. Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> 4. Create a few segments with the console producer
> 5. Observe that they are deleted after 1 minute
> 6. Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> 7. Create a few more segments with the console producer
> 8. Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-17584) Changing dynamic configurations resets retention
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-17584:
--
Description:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the default value (source code) for
log.retention.ms. This poses a durability issue if users have set their
retention by using log.retention.hours or log.retention.minutes. In other
words, if a user has set log.retention.hours=-1 (infinite retention) and they
dynamically change `message.max.bytes` their retention will immediately change
back to the default of 60480 ms (7 days) and data before this will be
scheduled for deletion immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
was:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the value for log.retention.ms. This
poses a durability issue if users have set their retention by using
log.retention.hours or log.retention.minutes. In other words, if a user has set
log.retention.hours=-1 (infinite retention) and they dynamically change
`message.max.bytes` their retention will immediately change back to the default
of 60480 ms (7 days) and data before this will be scheduled for deletion
immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
> Changing dynamic configurations resets retention
>
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the default value (source code)
> for log.retention.ms. This poses a durability issue if users have set their
> retention by using log.retention.hours or log.retention.minutes. In other
> words, if a user has set log.retention.hours=-1 (infinite retention) and they
> dynamically change `message.max.bytes` their retention will immediately
> change back to the default of 60480 ms (7 days) and data before this will
> be scheduled for deletion immediately.
> Steps to reproduce:
> # Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> # Start a single ZK or KRaft instance + a single Kafka instance
> # Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> # Create a few segments with the console producer
> # Observe that they are deleted after 1 minute
> # Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> # Create a few more segments with the console producer
> # Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-17584) Changing dynamic configurations resets retention
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-17584:
--
Description:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the default value (source code) for
log.retention.ms. This poses a durability issue if users have set their
retention by using log.retention.hours or log.retention.minutes. In other
words, if a user has set log.retention.hours=-1 (infinite retention) and they
dynamically change `message.max.bytes` their retention will immediately change
back to the default of 60480 ms (7 days) and data before this will be
scheduled for deletion immediately.
Steps to reproduce:
1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
2. Start a single ZK or KRaft instance + a single Kafka instance
3. Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
4. Create a few segments with the console producer
5. Observe that they are deleted after 1 minute
6. Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
7. Create a few more segments with the console producer
8. Observe that segments are no longer deleted after 1 minute
was:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the default value (source code) for
log.retention.ms. This poses a durability issue if users have set their
retention by using log.retention.hours or log.retention.minutes. In other
words, if a user has set log.retention.hours=-1 (infinite retention) and they
dynamically change `message.max.bytes` their retention will immediately change
back to the default of 60480 ms (7 days) and data before this will be
scheduled for deletion immediately.
Steps to reproduce:
1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
2. Start a single ZK or KRaft instance + a single Kafka instance
3. Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
4. Create a few segments with the console producer
5. Observe that they are deleted after 1 minute
6. Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
7. Create a few more segments with the console producer
8. Observe that segments are no longer deleted after 1 minute
> Changing dynamic configurations resets retention
>
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the default value (source code)
> for log.retention.ms. This poses a durability issue if users have set their
> retention by using log.retention.hours or log.retention.minutes. In other
> words, if a user has set log.retention.hours=-1 (infinite retention) and they
> dynamically change `message.max.bytes` their retention will immediately
> change back to the default of 60480 ms (7 days) and data before this will
> be scheduled for deletion immediately.
> Steps to reproduce:
> 1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> 2. Start a single ZK or KRaft instance + a single Kafka instance
> 3. Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> 4. Create a few segments with the console producer
> 5. Observe that they are deleted after 1 minute
> 6. Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> 7. Create a few more segments with the console producer
> 8. Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-17584) Changing dynamic configurations resets retention
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-17584:
--
Description:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the default value (source code) for
log.retention.ms. This poses a durability issue if users have set their
retention by using log.retention.hours or log.retention.minutes. In other
words, if a user has set log.retention.hours=-1 (infinite retention) and they
dynamically change `message.max.bytes` their retention will immediately change
back to the default of 60480 ms (7 days) and data before this will be
scheduled for deletion immediately.
Steps to reproduce:
1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
2. Start a single ZK or KRaft instance + a single Kafka instance
3. Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
4. Create a few segments with the console producer
5. Observe that they are deleted after 1 minute
6. Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
7. Create a few more segments with the console producer
8. Observe that segments are no longer deleted after 1 minute
was:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the default value (source code) for
log.retention.ms. This poses a durability issue if users have set their
retention by using log.retention.hours or log.retention.minutes. In other
words, if a user has set log.retention.hours=-1 (infinite retention) and they
dynamically change `message.max.bytes` their retention will immediately change
back to the default of 60480 ms (7 days) and data before this will be
scheduled for deletion immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
> Changing dynamic configurations resets retention
>
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the default value (source code)
> for log.retention.ms. This poses a durability issue if users have set their
> retention by using log.retention.hours or log.retention.minutes. In other
> words, if a user has set log.retention.hours=-1 (infinite retention) and they
> dynamically change `message.max.bytes` their retention will immediately
> change back to the default of 60480 ms (7 days) and data before this will
> be scheduled for deletion immediately.
> Steps to reproduce:
> 1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> 2. Start a single ZK or KRaft instance + a single Kafka instance
> 3. Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> 4. Create a few segments with the console producer
> 5. Observe that they are deleted after 1 minute
> 6. Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> 7. Create a few more segments with the console producer
> 8. Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-17584) Changing dynamic configurations resets retention
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-17584:
--
Priority: Blocker (was: Major)
> Changing dynamic configurations resets retention
>
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the value for log.retention.ms.
> This poses a durability issue if users have set their retention by using
> log.retention.hours or log.retention.minutes. In other words, if a user has
> set log.retention.hours=-1 (infinite retention) and they dynamically change
> `message.max.bytes` their retention will immediately change back to the
> default of 60480 ms (7 days) and data before this will be scheduled for
> deletion immediately.
> Steps to reproduce:
> # Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> # Start a single ZK or KRaft instance + a single Kafka instance
> # Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> # Create a few segments with the console producer
> # Observe that they are deleted after 1 minute
> # Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> # Create a few more segments with the console producer
> # Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-17584) Changing dynamic configurations resets retention
[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-17584:
--
Description:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the value for log.retention.ms. This
poses a durability issue if users have set their retention by using
log.retention.hours or log.retention.minutes. In other words, if a user has set
log.retention.hours=-1 (infinite retention) and they dynamically change
`message.max.bytes` their retention will immediately change back to the default
of 60480 ms (7 days) and data before this will be scheduled for deletion
immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
was:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the value for log.retention.ms. This
poses a durability issue if users have set their retention by using
log.retention.hours or log.retention.minutes. In other words, if a user has set
log.retention.hours=-1 (infinite retention) and they dynamically change
`message.max.bytes` their retention will immediately change back to the default
of 60480 ms (7 days) and data before this will be scheduled for deletion
immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
```
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512
```
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
```
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609
```
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
> Changing dynamic configurations resets retention
>
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the value for log.retention.ms.
> This poses a durability issue if users have set their retention by using
> log.retention.hours or log.retention.minutes. In other words, if a user has
> set log.retention.hours=-1 (infinite retention) and they dynamically change
> `message.max.bytes` their retention will immediately change back to the
> default of 60480 ms (7 days) and data before this will be scheduled for
> deletion immediately.
> Steps to reproduce:
> # Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> # Start a single ZK or KRaft instance + a single Kafka instance
> # Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> # Create a few segments with the console producer
> # Observe that they are deleted after 1 minute
> # Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> # Create a few more segments with the console producer
> # Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17584) Changing dynamic configurations resets retention
Christo Lolov created KAFKA-17584: - Summary: Changing dynamic configurations resets retention Key: KAFKA-17584 URL: https://issues.apache.org/jira/browse/KAFKA-17584 Project: Kafka Issue Type: Bug Reporter: Christo Lolov Assignee: Christo Lolov Updating certain dynamic configurations (for example `message.max.bytes`) causes retention based on time to reset to the value for log.retention.ms. This poses a durability issue if users have set their retention by using log.retention.hours or log.retention.minutes. In other words, if a user has set log.retention.hours=-1 (infinite retention) and they dynamically change `message.max.bytes` their retention will immediately change back to the default of 60480 ms (7 days) and data before this will be scheduled for deletion immediately. Steps to reproduce: # Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to server.properties # Start a single ZK or KRaft instance + a single Kafka instance # Create a topic using ``` bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config segment.bytes=512 ``` # Create a few segments with the console producer # Observe that they are deleted after 1 minute # Use the following command ``` bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers --entity-default --alter --add-config message.max.bytes=1048609 ``` (the value of `message.max.bytes` is irrelevant) # Create a few more segments with the console producer # Observe that segments are no longer deleted after 1 minute -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17579) Dynamic LogCleaner configurations are not picked up upon restart
[ https://issues.apache.org/jira/browse/KAFKA-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-17579. --- Resolution: Fixed > Dynamic LogCleaner configurations are not picked up upon restart > > > Key: KAFKA-17579 > URL: https://issues.apache.org/jira/browse/KAFKA-17579 > Project: Kafka > Issue Type: Bug >Reporter: Christo Lolov >Priority: Major > > Dynamic configurations for the LogCleaner are not applied upon a restart. > > Reproduction steps: > # Create a 1-broker cluster > # Change the number of log.cleaner.threads to 2 > # Bounce the broker > # Observe the server.logs - you will notice that only 1 log cleaner thread > has been instantiated > > Proposed solution: > # Change immutable variables `val` to methods `def` > # Add unit/integration tests which test reconfiguration of the LogCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17579) Dynamic LogCleaner configurations are not picked up upon restart
Christo Lolov created KAFKA-17579: - Summary: Dynamic LogCleaner configurations are not picked up upon restart Key: KAFKA-17579 URL: https://issues.apache.org/jira/browse/KAFKA-17579 Project: Kafka Issue Type: Bug Reporter: Christo Lolov Dynamic configurations for the LogCleaner are not applied upon a restart. Reproduction steps: # Create a 1-broker cluster # Change the number of log.cleaner.threads to 2 # Bounce the broker # Observe the server.logs - you will notice that only 1 log cleaner thread has been instantiated Proposed solution: # Change immutable variables `val` to methods `def` # Add unit/integration tests which test reconfiguration of the LogCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17579) Dynamic LogCleaner configurations are not picked up upon restart
[ https://issues.apache.org/jira/browse/KAFKA-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882944#comment-17882944 ] Christo Lolov commented on KAFKA-17579: --- This has been picked up by Jason Taylor. His Jira account is still being reviewed hence I will assign the ticket once it is approved. > Dynamic LogCleaner configurations are not picked up upon restart > > > Key: KAFKA-17579 > URL: https://issues.apache.org/jira/browse/KAFKA-17579 > Project: Kafka > Issue Type: Bug >Reporter: Christo Lolov >Priority: Major > > Dynamic configurations for the LogCleaner are not applied upon a restart. > > Reproduction steps: > # Create a 1-broker cluster > # Change the number of log.cleaner.threads to 2 > # Bounce the broker > # Observe the server.logs - you will notice that only 1 log cleaner thread > has been instantiated > > Proposed solution: > # Change immutable variables `val` to methods `def` > # Add unit/integration tests which test reconfiguration of the LogCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16154) Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP
[ https://issues.apache.org/jira/browse/KAFKA-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866786#comment-17866786 ] Christo Lolov commented on KAFKA-16154: --- Yup, everything has been reverted. I will aim to open a PR by tomorrow morning so we can include this in 3.9 > Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP > --- > > Key: KAFKA-16154 > URL: https://issues.apache.org/jira/browse/KAFKA-16154 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Christo Lolov >Priority: Major > Fix For: 3.9.0 > > > A broker should start returning offsets when given a timestamp of -5, which > signifies a LATEST_TIERED_TIMESTAMP. > There are 3 cases. > Tiered Storage is not enabled. In such a situation asking for > LATEST_TIERED_TIMESTAMP should always return no offset. > Tiered Storage is enabled and there is nothing in remote storage. In such a > situation the offset returned should be 0. > Tiered Storage is enabled and there is something in remote storage. In such a > situation the offset returned should be the highest offset the broker is > aware of. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16154) Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP
[ https://issues.apache.org/jira/browse/KAFKA-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866513#comment-17866513 ] Christo Lolov commented on KAFKA-16154: --- Let me circle back tomorrow and see which parts have made it in and which haven't cause we had to revert some of the changes to unblock 3.8 > Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP > --- > > Key: KAFKA-16154 > URL: https://issues.apache.org/jira/browse/KAFKA-16154 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Christo Lolov >Priority: Major > Fix For: 3.8.0 > > > A broker should start returning offsets when given a timestamp of -5, which > signifies a LATEST_TIERED_TIMESTAMP. > There are 3 cases. > Tiered Storage is not enabled. In such a situation asking for > LATEST_TIERED_TIMESTAMP should always return no offset. > Tiered Storage is enabled and there is nothing in remote storage. In such a > situation the offset returned should be 0. > Tiered Storage is enabled and there is something in remote storage. In such a > situation the offset returned should be the highest offset the broker is > aware of. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16855) KRaft - Wire replaying a TopicRecord
[ https://issues.apache.org/jira/browse/KAFKA-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865408#comment-17865408 ] Christo Lolov commented on KAFKA-16855: --- Heya, [~muralibasani]! The general idea of replaying a TopicRecord is the following: * If a topic has tiering enabled then it should have both copy and expire tasks * If a topic doesn't have tiering enabled then it should have only expire tasks I believe a good starting point of how records are replayed is the ReplicaManager#applyDelta Does this give you enough information to start? > KRaft - Wire replaying a TopicRecord > > > Key: KAFKA-16855 > URL: https://issues.apache.org/jira/browse/KAFKA-16855 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Muralidhar Basani >Priority: Major > > *Summary* > Replaying a TopicRecord containing a new TieredEpoch and TieredState needs to > interact with the two thread pools in the RemoteLogManager to add/remove the > correct tasks from each -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool
[ https://issues.apache.org/jira/browse/KAFKA-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17861096#comment-17861096 ] Christo Lolov commented on KAFKA-16853: --- Heya [~abhijeetkumar], I hope you are well! If you haven't started on this should I try to have a stab at it? > Split RemoteLogManagerScheduledThreadPool > - > > Key: KAFKA-16853 > URL: https://issues.apache.org/jira/browse/KAFKA-16853 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Abhijeet Kumar >Priority: Major > > *Summary* > To begin with create just the RemoteDataExpirationThreadPool and move > expiration to it. Keep all settings as if the only thread pool was the > RemoteLogManagerScheduledThreadPool. Ensure that the new thread pool is wired > correctly to the RemoteLogManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16850) KRaft - Add v2 of TopicRecord
[ https://issues.apache.org/jira/browse/KAFKA-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17859944#comment-17859944 ] Christo Lolov commented on KAFKA-16850: --- I think this is addressed by [https://github.com/apache/kafka/pull/16257,] so I will proceed to mark this ticket as complete [~showuon]. Let me know if you think otherwise! > KRaft - Add v2 of TopicRecord > - > > Key: KAFKA-16850 > URL: https://issues.apache.org/jira/browse/KAFKA-16850 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Luke Chen >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-17031) Make configurations public
[ https://issues.apache.org/jira/browse/KAFKA-17031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17859878#comment-17859878 ] Christo Lolov commented on KAFKA-17031: --- Yup! > Make configurations public > -- > > Key: KAFKA-17031 > URL: https://issues.apache.org/jira/browse/KAFKA-17031 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Priority: Major > > *Summary* > We introduce 3 new configurations as part of this KIP. Up to now they have > been defined as internal only. We need to change said configurations to > public prior to the release. > > Also need to deprecate "remote.log.manager.thread.pool.size" config. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-17031) Make configurations public
[ https://issues.apache.org/jira/browse/KAFKA-17031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov updated KAFKA-17031: -- Description: *Summary* We introduce 3 new configurations as part of this KIP. Up to now they have been defined as internal only. We need to change said configurations to public prior to the release. was: ### Summary We introduce 3 new configurations as part of this KIP. Up to now they have been defined as internal only. We need to change said configurations to public prior to the release. > Make configurations public > -- > > Key: KAFKA-17031 > URL: https://issues.apache.org/jira/browse/KAFKA-17031 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Priority: Major > > *Summary* > We introduce 3 new configurations as part of this KIP. Up to now they have > been defined as internal only. We need to change said configurations to > public prior to the release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17031) Make configurations public
Christo Lolov created KAFKA-17031: - Summary: Make configurations public Key: KAFKA-17031 URL: https://issues.apache.org/jira/browse/KAFKA-17031 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov ### Summary We introduce 3 new configurations as part of this KIP. Up to now they have been defined as internal only. We need to change said configurations to public prior to the release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool
[ https://issues.apache.org/jira/browse/KAFKA-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17859742#comment-17859742 ] Christo Lolov commented on KAFKA-16853: --- Could I also be looped into the review please? > Split RemoteLogManagerScheduledThreadPool > - > > Key: KAFKA-16853 > URL: https://issues.apache.org/jira/browse/KAFKA-16853 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Abhijeet Kumar >Priority: Major > > *Summary* > To begin with create just the RemoteDataExpirationThreadPool and move > expiration to it. Keep all settings as if the only thread pool was the > RemoteLogManagerScheduledThreadPool. Ensure that the new thread pool is wired > correctly to the RemoteLogManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16856) Zookeeper - Add new exception
[ https://issues.apache.org/jira/browse/KAFKA-16856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-16856: - Assignee: Muralidhar Basani > Zookeeper - Add new exception > - > > Key: KAFKA-16856 > URL: https://issues.apache.org/jira/browse/KAFKA-16856 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Muralidhar Basani >Priority: Major > > *Summary* > Add TIERED_STORAGE_DISABLEMENT_IN_PROGRESS exception -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16852) Add *.thread.pool.size
[ https://issues.apache.org/jira/browse/KAFKA-16852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851064#comment-17851064 ] Christo Lolov commented on KAFKA-16852: --- Heya [~muralibasani] :) ! Thank you for putting yourself forward for this - I will assign the ticket to you and aim to provide a review on the PR by end of day today > Add *.thread.pool.size > -- > > Key: KAFKA-16852 > URL: https://issues.apache.org/jira/browse/KAFKA-16852 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Priority: Major > > *Summary* > Add the remote.log.manager.copier.thread.pool.size and > remote.log.manager.expiration.thread.pool.size configurations as internal-only -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16852) Add *.thread.pool.size
[ https://issues.apache.org/jira/browse/KAFKA-16852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-16852: - Assignee: Muralidhar Basani > Add *.thread.pool.size > -- > > Key: KAFKA-16852 > URL: https://issues.apache.org/jira/browse/KAFKA-16852 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Muralidhar Basani >Priority: Major > > *Summary* > Add the remote.log.manager.copier.thread.pool.size and > remote.log.manager.expiration.thread.pool.size configurations as internal-only -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16851) Add remote.log.disable.policy
[ https://issues.apache.org/jira/browse/KAFKA-16851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-16851: - Assignee: Christo Lolov > Add remote.log.disable.policy > - > > Key: KAFKA-16851 > URL: https://issues.apache.org/jira/browse/KAFKA-16851 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Christo Lolov >Priority: Major > > *Summary* > Add the configuration as internal-only to begin with. Do not wire it to > anything yet, just ensure that it is settable dynamically -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16854) Zookeeper - Add v5 of StopReplica
[ https://issues.apache.org/jira/browse/KAFKA-16854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-16854: - Assignee: Christo Lolov > Zookeeper - Add v5 of StopReplica > - > > Key: KAFKA-16854 > URL: https://issues.apache.org/jira/browse/KAFKA-16854 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Christo Lolov >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool
[ https://issues.apache.org/jira/browse/KAFKA-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850426#comment-17850426 ] Christo Lolov commented on KAFKA-16853: --- Heya [~abhijeetkumar], since you expressed an interest to pick this up I have assigned it to you. Let me know if you don't have the bandwidth to pick this up! > Split RemoteLogManagerScheduledThreadPool > - > > Key: KAFKA-16853 > URL: https://issues.apache.org/jira/browse/KAFKA-16853 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Abhijeet Kumar >Priority: Major > > *Summary* > To begin with create just the RemoteDataExpirationThreadPool and move > expiration to it. Keep all settings as if the only thread pool was the > RemoteLogManagerScheduledThreadPool. Ensure that the new thread pool is wired > correctly to the RemoteLogManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool
[ https://issues.apache.org/jira/browse/KAFKA-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-16853: - Assignee: Christo Lolov > Split RemoteLogManagerScheduledThreadPool > - > > Key: KAFKA-16853 > URL: https://issues.apache.org/jira/browse/KAFKA-16853 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Christo Lolov >Priority: Major > > *Summary* > To begin with create just the RemoteDataExpirationThreadPool and move > expiration to it. Keep all settings as if the only thread pool was the > RemoteLogManagerScheduledThreadPool. Ensure that the new thread pool is wired > correctly to the RemoteLogManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool
[ https://issues.apache.org/jira/browse/KAFKA-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-16853: - Assignee: Abhijeet Kumar (was: Christo Lolov) > Split RemoteLogManagerScheduledThreadPool > - > > Key: KAFKA-16853 > URL: https://issues.apache.org/jira/browse/KAFKA-16853 > Project: Kafka > Issue Type: Sub-task >Reporter: Christo Lolov >Assignee: Abhijeet Kumar >Priority: Major > > *Summary* > To begin with create just the RemoteDataExpirationThreadPool and move > expiration to it. Keep all settings as if the only thread pool was the > RemoteLogManagerScheduledThreadPool. Ensure that the new thread pool is wired > correctly to the RemoteLogManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-16857) Zookeeper - Add new ZNodes
[
https://issues.apache.org/jira/browse/KAFKA-16857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-16857:
--
Description:
*Summary*
Additional information needs to be stored in new ZNodes as part of disablement.
Ensure that said information makes it into Zookeeper.
{code:java}
/brokers/topics/{topic-name}/partitions
/tieredstorage
/tiered_epoch
/state {code}
was:
*Summary*
Additional information needs to be stored in new ZNodes as part of disablement.
Ensure that said information makes it into Zookeeper.
{code:java}
/brokers/topics/{topic-name}/partitions
/tieredstorage/ /tiered_epoch
/state {code}
> Zookeeper - Add new ZNodes
> --
>
> Key: KAFKA-16857
> URL: https://issues.apache.org/jira/browse/KAFKA-16857
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Priority: Major
>
> *Summary*
> Additional information needs to be stored in new ZNodes as part of
> disablement. Ensure that said information makes it into Zookeeper.
> {code:java}
> /brokers/topics/{topic-name}/partitions
>/tieredstorage
> /tiered_epoch
>
> /state {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16857) Zookeeper - Add new ZNodes
Christo Lolov created KAFKA-16857:
-
Summary: Zookeeper - Add new ZNodes
Key: KAFKA-16857
URL: https://issues.apache.org/jira/browse/KAFKA-16857
Project: Kafka
Issue Type: Sub-task
Reporter: Christo Lolov
*Summary*
Additional information needs to be stored in new ZNodes as part of disablement.
Ensure that said information makes it into Zookeeper.
{code:java}
/brokers/topics/{topic-name}/partitions
/tieredstorage/ /tiered_epoch
/state {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16856) Zookeeper - Add new exception
Christo Lolov created KAFKA-16856: - Summary: Zookeeper - Add new exception Key: KAFKA-16856 URL: https://issues.apache.org/jira/browse/KAFKA-16856 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov *Summary* Add TIERED_STORAGE_DISABLEMENT_IN_PROGRESS exception -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16855) KRaft - Wire replaying a TopicRecord
Christo Lolov created KAFKA-16855: - Summary: KRaft - Wire replaying a TopicRecord Key: KAFKA-16855 URL: https://issues.apache.org/jira/browse/KAFKA-16855 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov *Summary* Replaying a TopicRecord containing a new TieredEpoch and TieredState needs to interact with the two thread pools in the RemoteLogManager to add/remove the correct tasks from each -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16854) Zookeeper - Add v5 of StopReplica
Christo Lolov created KAFKA-16854: - Summary: Zookeeper - Add v5 of StopReplica Key: KAFKA-16854 URL: https://issues.apache.org/jira/browse/KAFKA-16854 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16853) Split RemoteLogManagerScheduledThreadPool
Christo Lolov created KAFKA-16853: - Summary: Split RemoteLogManagerScheduledThreadPool Key: KAFKA-16853 URL: https://issues.apache.org/jira/browse/KAFKA-16853 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov *Summary* To begin with create just the RemoteDataExpirationThreadPool and move expiration to it. Keep all settings as if the only thread pool was the RemoteLogManagerScheduledThreadPool. Ensure that the new thread pool is wired correctly to the RemoteLogManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16852) Add *.thread.pool.size
Christo Lolov created KAFKA-16852: - Summary: Add *.thread.pool.size Key: KAFKA-16852 URL: https://issues.apache.org/jira/browse/KAFKA-16852 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov *Summary* Add the remote.log.manager.copier.thread.pool.size and remote.log.manager.expiration.thread.pool.size configurations as internal-only -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16851) Add remote.log.disable.policy
Christo Lolov created KAFKA-16851: - Summary: Add remote.log.disable.policy Key: KAFKA-16851 URL: https://issues.apache.org/jira/browse/KAFKA-16851 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov *Summary* Add the configuration as internal-only to begin with. Do not wire it to anything yet, just ensure that it is settable dynamically -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16850) KRaft - Add v2 of TopicRecord
Christo Lolov created KAFKA-16850: - Summary: KRaft - Add v2 of TopicRecord Key: KAFKA-16850 URL: https://issues.apache.org/jira/browse/KAFKA-16850 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16790) Calls to RemoteLogManager are made before it is configured
[
https://issues.apache.org/jira/browse/KAFKA-16790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847829#comment-17847829
]
Christo Lolov commented on KAFKA-16790:
---
I have assigned the Jira ticket to you, [~muralibasani]! If you manage to get
this in until the 29th of May you will fix it in time for 3.8 :). Feel free to
tag me as a reviewer on any PRs you raise
> Calls to RemoteLogManager are made before it is configured
> --
>
> Key: KAFKA-16790
> URL: https://issues.apache.org/jira/browse/KAFKA-16790
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.8.0
>Reporter: Christo Lolov
>Assignee: Muralidhar Basani
>Priority: Major
>
> BrokerMetadataPublisher#onMetadataUpdate calls ReplicaManager#applyDelta (1)
> which in turn calls RemoteLogManager#onLeadershipChange (2), however, the
> RemoteLogManager is configured after the BrokerMetadataPublisher starts
> running (3, 4). This is incorrect, we either need to initialise the
> RemoteLogManager before we start the BrokerMetadataPublisher or we need to
> skip calls to onLeadershipChange if the RemoteLogManager is not initialised.
> (1)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L151]
> (2)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L2737]
> (3)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L432]
> (4)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L515]
> The way to reproduce the problem is by looking at the following changes
> {code:java}
> config/kraft/broker.properties | 10 ++
> .../main/java/kafka/log/remote/RemoteLogManager.java | 8 +++-
> core/src/main/scala/kafka/server/ReplicaManager.scala | 6 +-
> 3 files changed, 22 insertions(+), 2 deletions(-)diff --git
> a/config/kraft/broker.properties b/config/kraft/broker.properties
> index 2d15997f28..39d126cf87 100644
> --- a/config/kraft/broker.properties
> +++ b/config/kraft/broker.properties
> @@ -127,3 +127,13 @@ log.segment.bytes=1073741824
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=30
> +
> +remote.log.storage.system.enable=true
> +remote.log.metadata.manager.listener.name=PLAINTEXT
> +remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
> +remote.log.storage.manager.class.path=/home/ec2-user/kafka/storage/build/libs/kafka-storage-3.8.0-SNAPSHOT-test.jar
> +remote.log.storage.manager.impl.prefix=rsm.config.
> +remote.log.metadata.manager.impl.prefix=rlmm.config.
> +rsm.config.dir=/tmp/kafka-remote-storage
> +rlmm.config.remote.log.metadata.topic.replication.factor=1
> +log.retention.check.interval.ms=1000
> diff --git a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> index 6555b7c0cd..e84a072abc 100644
> --- a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> +++ b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> @@ -164,6 +164,7 @@ public class RemoteLogManager implements Closeable {
> // The endpoint for remote log metadata manager to connect to
> private Optional endpoint = Optional.empty();
> private boolean closed = false;
> + private boolean up = false;
>
> /**
> * Creates RemoteLogManager instance with the given arguments.
> @@ -298,6 +299,7 @@ public class RemoteLogManager implements Closeable {
> // in connecting to the brokers or remote storages.
> configureRSM();
> configureRLMM();
> + up = true;
> }
>
> public RemoteStorageManager storageManager() {
> @@ -329,7 +331,11 @@ public class RemoteLogManager implements Closeable {
> public void onLeadershipChange(Set partitionsBecomeLeader,
> Set partitionsBecomeFollower,
> Map topicIds) {
> - LOGGER.debug("Received leadership changes for leaders: {} and
> followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
> + if (!up) {
> + LOGGER.error("NullPointerException");
> + return;
> + }
> + LOGGER.error("Received leadership changes for leaders: {} and
> followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
>
> Map leaderPartitionsWithLeaderEpoch =
> filterPartitions(partitionsBecomeLeader)
> .collect(Collectors.toMap(
> diff --git a/core/src/main/scala/kafka/server/ReplicaMana
[jira] [Assigned] (KAFKA-16790) Calls to RemoteLogManager are made before it is configured
[
https://issues.apache.org/jira/browse/KAFKA-16790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov reassigned KAFKA-16790:
-
Assignee: Muralidhar Basani
> Calls to RemoteLogManager are made before it is configured
> --
>
> Key: KAFKA-16790
> URL: https://issues.apache.org/jira/browse/KAFKA-16790
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.8.0
>Reporter: Christo Lolov
>Assignee: Muralidhar Basani
>Priority: Major
>
> BrokerMetadataPublisher#onMetadataUpdate calls ReplicaManager#applyDelta (1)
> which in turn calls RemoteLogManager#onLeadershipChange (2), however, the
> RemoteLogManager is configured after the BrokerMetadataPublisher starts
> running (3, 4). This is incorrect, we either need to initialise the
> RemoteLogManager before we start the BrokerMetadataPublisher or we need to
> skip calls to onLeadershipChange if the RemoteLogManager is not initialised.
> (1)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L151]
> (2)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L2737]
> (3)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L432]
> (4)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L515]
> The way to reproduce the problem is by looking at the following changes
> {code:java}
> config/kraft/broker.properties | 10 ++
> .../main/java/kafka/log/remote/RemoteLogManager.java | 8 +++-
> core/src/main/scala/kafka/server/ReplicaManager.scala | 6 +-
> 3 files changed, 22 insertions(+), 2 deletions(-)diff --git
> a/config/kraft/broker.properties b/config/kraft/broker.properties
> index 2d15997f28..39d126cf87 100644
> --- a/config/kraft/broker.properties
> +++ b/config/kraft/broker.properties
> @@ -127,3 +127,13 @@ log.segment.bytes=1073741824
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=30
> +
> +remote.log.storage.system.enable=true
> +remote.log.metadata.manager.listener.name=PLAINTEXT
> +remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
> +remote.log.storage.manager.class.path=/home/ec2-user/kafka/storage/build/libs/kafka-storage-3.8.0-SNAPSHOT-test.jar
> +remote.log.storage.manager.impl.prefix=rsm.config.
> +remote.log.metadata.manager.impl.prefix=rlmm.config.
> +rsm.config.dir=/tmp/kafka-remote-storage
> +rlmm.config.remote.log.metadata.topic.replication.factor=1
> +log.retention.check.interval.ms=1000
> diff --git a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> index 6555b7c0cd..e84a072abc 100644
> --- a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> +++ b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> @@ -164,6 +164,7 @@ public class RemoteLogManager implements Closeable {
> // The endpoint for remote log metadata manager to connect to
> private Optional endpoint = Optional.empty();
> private boolean closed = false;
> + private boolean up = false;
>
> /**
> * Creates RemoteLogManager instance with the given arguments.
> @@ -298,6 +299,7 @@ public class RemoteLogManager implements Closeable {
> // in connecting to the brokers or remote storages.
> configureRSM();
> configureRLMM();
> + up = true;
> }
>
> public RemoteStorageManager storageManager() {
> @@ -329,7 +331,11 @@ public class RemoteLogManager implements Closeable {
> public void onLeadershipChange(Set partitionsBecomeLeader,
> Set partitionsBecomeFollower,
> Map topicIds) {
> - LOGGER.debug("Received leadership changes for leaders: {} and
> followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
> + if (!up) {
> + LOGGER.error("NullPointerException");
> + return;
> + }
> + LOGGER.error("Received leadership changes for leaders: {} and
> followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
>
> Map leaderPartitionsWithLeaderEpoch =
> filterPartitions(partitionsBecomeLeader)
> .collect(Collectors.toMap(
> diff --git a/core/src/main/scala/kafka/server/ReplicaManager.scala
> b/core/src/main/scala/kafka/server/ReplicaManager.scala
> index 35499430d6..bd3f41c3d6 100644
> --- a/core/src/main/scala/kafka/server/ReplicaManager.scala
> +++ b/core/src/main/scala/kafka/server/ReplicaMan
[jira] [Commented] (KAFKA-16790) Calls to RemoteLogManager are made before it is configured
[
https://issues.apache.org/jira/browse/KAFKA-16790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847824#comment-17847824
]
Christo Lolov commented on KAFKA-16790:
---
Heya [~muralibasani], thank you for picking this up on such a short notice!
Here's my reasoning. The real initialisation of a RemoteLogManager happens when
its startup method is called (that is what in turn calls the configure methods
on the RemoteStorageManager and the RemoteLogMetadataManager). You are correct
that the createRemoteLogManager method is called earlier than the
creation/running of the BrokerMetadataPublisher. However, that doesn't
configure the underlying managers. Let me know if there is still confusion on
what the problem is!
> Calls to RemoteLogManager are made before it is configured
> --
>
> Key: KAFKA-16790
> URL: https://issues.apache.org/jira/browse/KAFKA-16790
> Project: Kafka
> Issue Type: Bug
> Components: kraft
>Affects Versions: 3.8.0
>Reporter: Christo Lolov
>Priority: Major
>
> BrokerMetadataPublisher#onMetadataUpdate calls ReplicaManager#applyDelta (1)
> which in turn calls RemoteLogManager#onLeadershipChange (2), however, the
> RemoteLogManager is configured after the BrokerMetadataPublisher starts
> running (3, 4). This is incorrect, we either need to initialise the
> RemoteLogManager before we start the BrokerMetadataPublisher or we need to
> skip calls to onLeadershipChange if the RemoteLogManager is not initialised.
> (1)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L151]
> (2)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L2737]
> (3)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L432]
> (4)
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L515]
> The way to reproduce the problem is by looking at the following changes
> {code:java}
> config/kraft/broker.properties | 10 ++
> .../main/java/kafka/log/remote/RemoteLogManager.java | 8 +++-
> core/src/main/scala/kafka/server/ReplicaManager.scala | 6 +-
> 3 files changed, 22 insertions(+), 2 deletions(-)diff --git
> a/config/kraft/broker.properties b/config/kraft/broker.properties
> index 2d15997f28..39d126cf87 100644
> --- a/config/kraft/broker.properties
> +++ b/config/kraft/broker.properties
> @@ -127,3 +127,13 @@ log.segment.bytes=1073741824
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=30
> +
> +remote.log.storage.system.enable=true
> +remote.log.metadata.manager.listener.name=PLAINTEXT
> +remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
> +remote.log.storage.manager.class.path=/home/ec2-user/kafka/storage/build/libs/kafka-storage-3.8.0-SNAPSHOT-test.jar
> +remote.log.storage.manager.impl.prefix=rsm.config.
> +remote.log.metadata.manager.impl.prefix=rlmm.config.
> +rsm.config.dir=/tmp/kafka-remote-storage
> +rlmm.config.remote.log.metadata.topic.replication.factor=1
> +log.retention.check.interval.ms=1000
> diff --git a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> index 6555b7c0cd..e84a072abc 100644
> --- a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> +++ b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
> @@ -164,6 +164,7 @@ public class RemoteLogManager implements Closeable {
> // The endpoint for remote log metadata manager to connect to
> private Optional endpoint = Optional.empty();
> private boolean closed = false;
> + private boolean up = false;
>
> /**
> * Creates RemoteLogManager instance with the given arguments.
> @@ -298,6 +299,7 @@ public class RemoteLogManager implements Closeable {
> // in connecting to the brokers or remote storages.
> configureRSM();
> configureRLMM();
> + up = true;
> }
>
> public RemoteStorageManager storageManager() {
> @@ -329,7 +331,11 @@ public class RemoteLogManager implements Closeable {
> public void onLeadershipChange(Set partitionsBecomeLeader,
> Set partitionsBecomeFollower,
> Map topicIds) {
> - LOGGER.debug("Received leadership changes for leaders: {} and
> followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
> + if (!up) {
> + LOGGER.error("NullPointerException");
> + return;
> + }
> + LOGGER.error("Recei
[jira] [Updated] (KAFKA-16790) Calls to RemoteLogManager are made before it is configured
[
https://issues.apache.org/jira/browse/KAFKA-16790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-16790:
--
Description:
BrokerMetadataPublisher#onMetadataUpdate calls ReplicaManager#applyDelta (1)
which in turn calls RemoteLogManager#onLeadershipChange (2), however, the
RemoteLogManager is configured after the BrokerMetadataPublisher starts running
(3, 4). This is incorrect, we either need to initialise the RemoteLogManager
before we start the BrokerMetadataPublisher or we need to skip calls to
onLeadershipChange if the RemoteLogManager is not initialised.
(1)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L151]
(2)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L2737]
(3)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L432]
(4)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L515]
The way to reproduce the problem is by looking at the following changes
```
---
config/kraft/broker.properties | 10 ++
.../main/java/kafka/log/remote/RemoteLogManager.java | 8 +++-
core/src/main/scala/kafka/server/ReplicaManager.scala | 6 +-
3 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/config/kraft/broker.properties b/config/kraft/broker.properties
index 2d15997f28..39d126cf87 100644
--- a/config/kraft/broker.properties
+++ b/config/kraft/broker.properties
@@ -127,3 +127,13 @@ log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted
according
# to the retention policies
log.retention.check.interval.ms=30
+
+remote.log.storage.system.enable=true
+remote.log.metadata.manager.listener.name=PLAINTEXT
+remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
+remote.log.storage.manager.class.path=/home/ec2-user/kafka/storage/build/libs/kafka-storage-3.8.0-SNAPSHOT-test.jar
+remote.log.storage.manager.impl.prefix=rsm.config.
+remote.log.metadata.manager.impl.prefix=rlmm.config.
+rsm.config.dir=/tmp/kafka-remote-storage
+rlmm.config.remote.log.metadata.topic.replication.factor=1
+log.retention.check.interval.ms=1000
diff --git a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
index 6555b7c0cd..e84a072abc 100644
--- a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
+++ b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
@@ -164,6 +164,7 @@ public class RemoteLogManager implements Closeable {
// The endpoint for remote log metadata manager to connect to
private Optional endpoint = Optional.empty();
private boolean closed = false;
+ private boolean up = false;
/**
* Creates RemoteLogManager instance with the given arguments.
@@ -298,6 +299,7 @@ public class RemoteLogManager implements Closeable {
// in connecting to the brokers or remote storages.
configureRSM();
configureRLMM();
+ up = true;
}
public RemoteStorageManager storageManager() {
@@ -329,7 +331,11 @@ public class RemoteLogManager implements Closeable {
public void onLeadershipChange(Set partitionsBecomeLeader,
Set partitionsBecomeFollower,
Map topicIds) {
- LOGGER.debug("Received leadership changes for leaders: {} and
followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+ if (!up) {
+ LOGGER.error("NullPointerException");
+ return;
+ }
+ LOGGER.error("Received leadership changes for leaders: {} and
followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
Map leaderPartitionsWithLeaderEpoch =
filterPartitions(partitionsBecomeLeader)
.collect(Collectors.toMap(
diff --git a/core/src/main/scala/kafka/server/ReplicaManager.scala
b/core/src/main/scala/kafka/server/ReplicaManager.scala
index 35499430d6..bd3f41c3d6 100644
--- a/core/src/main/scala/kafka/server/ReplicaManager.scala
+++ b/core/src/main/scala/kafka/server/ReplicaManager.scala
@@ -2688,6 +2688,7 @@ class ReplicaManager(val config: KafkaConfig,
*/
def applyDelta(delta: TopicsDelta, newImage: MetadataImage): Unit = {
// Before taking the lock, compute the local changes
+ stateChangeLogger.error("ROBIN")
val localChanges = delta.localChanges(config.nodeId)
val metadataVersion = newImage.features().metadataVersion()
@@ -2734,7 +2735,10 @@ class ReplicaManager(val config: KafkaConfig,
replicaFetcherManager.shutdownIdleFetcherThreads()
replicaAlterLogDirsManager.shutdownIdleFetcherThreads()
- remoteLogManager.foreach(rlm =>
rlm.onLeadershipC
[jira] [Updated] (KAFKA-16790) Calls to RemoteLogManager are made before it is configured
[
https://issues.apache.org/jira/browse/KAFKA-16790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-16790:
--
Description:
BrokerMetadataPublisher#onMetadataUpdate calls ReplicaManager#applyDelta (1)
which in turn calls RemoteLogManager#onLeadershipChange (2), however, the
RemoteLogManager is configured after the BrokerMetadataPublisher starts running
(3, 4). This is incorrect, we either need to initialise the RemoteLogManager
before we start the BrokerMetadataPublisher or we need to skip calls to
onLeadershipChange if the RemoteLogManager is not initialised.
(1)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L151]
(2)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L2737]
(3)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L432]
(4)
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L515]
The way to reproduce the problem is by looking at the following changes
{code:java}
config/kraft/broker.properties | 10 ++
.../main/java/kafka/log/remote/RemoteLogManager.java | 8 +++-
core/src/main/scala/kafka/server/ReplicaManager.scala | 6 +-
3 files changed, 22 insertions(+), 2 deletions(-)diff --git
a/config/kraft/broker.properties b/config/kraft/broker.properties
index 2d15997f28..39d126cf87 100644
--- a/config/kraft/broker.properties
+++ b/config/kraft/broker.properties
@@ -127,3 +127,13 @@ log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted
according
# to the retention policies
log.retention.check.interval.ms=30
+
+remote.log.storage.system.enable=true
+remote.log.metadata.manager.listener.name=PLAINTEXT
+remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
+remote.log.storage.manager.class.path=/home/ec2-user/kafka/storage/build/libs/kafka-storage-3.8.0-SNAPSHOT-test.jar
+remote.log.storage.manager.impl.prefix=rsm.config.
+remote.log.metadata.manager.impl.prefix=rlmm.config.
+rsm.config.dir=/tmp/kafka-remote-storage
+rlmm.config.remote.log.metadata.topic.replication.factor=1
+log.retention.check.interval.ms=1000
diff --git a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
index 6555b7c0cd..e84a072abc 100644
--- a/core/src/main/java/kafka/log/remote/RemoteLogManager.java
+++ b/core/src/main/java/kafka/log/remote/RemoteLogManager.java
@@ -164,6 +164,7 @@ public class RemoteLogManager implements Closeable {
// The endpoint for remote log metadata manager to connect to
private Optional endpoint = Optional.empty();
private boolean closed = false;
+ private boolean up = false;
/**
* Creates RemoteLogManager instance with the given arguments.
@@ -298,6 +299,7 @@ public class RemoteLogManager implements Closeable {
// in connecting to the brokers or remote storages.
configureRSM();
configureRLMM();
+ up = true;
}
public RemoteStorageManager storageManager() {
@@ -329,7 +331,11 @@ public class RemoteLogManager implements Closeable {
public void onLeadershipChange(Set partitionsBecomeLeader,
Set partitionsBecomeFollower,
Map topicIds) {
- LOGGER.debug("Received leadership changes for leaders: {} and
followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+ if (!up) {
+ LOGGER.error("NullPointerException");
+ return;
+ }
+ LOGGER.error("Received leadership changes for leaders: {} and
followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
Map leaderPartitionsWithLeaderEpoch =
filterPartitions(partitionsBecomeLeader)
.collect(Collectors.toMap(
diff --git a/core/src/main/scala/kafka/server/ReplicaManager.scala
b/core/src/main/scala/kafka/server/ReplicaManager.scala
index 35499430d6..bd3f41c3d6 100644
--- a/core/src/main/scala/kafka/server/ReplicaManager.scala
+++ b/core/src/main/scala/kafka/server/ReplicaManager.scala
@@ -2688,6 +2688,7 @@ class ReplicaManager(val config: KafkaConfig,
*/
def applyDelta(delta: TopicsDelta, newImage: MetadataImage): Unit = {
// Before taking the lock, compute the local changes
+ stateChangeLogger.error("ROBIN")
val localChanges = delta.localChanges(config.nodeId)
val metadataVersion = newImage.features().metadataVersion()
@@ -2734,7 +2735,10 @@ class ReplicaManager(val config: KafkaConfig,
replicaFetcherManager.shutdownIdleFetcherThreads()
replicaAlterLogDirsManager.shutdownIdleFetcherThreads()
- remoteLogManager.foreach(rlm =>
rlm.onLeadership
[jira] [Created] (KAFKA-16790) Calls to RemoteLogManager are made before it is configured
Christo Lolov created KAFKA-16790: - Summary: Calls to RemoteLogManager are made before it is configured Key: KAFKA-16790 URL: https://issues.apache.org/jira/browse/KAFKA-16790 Project: Kafka Issue Type: Bug Components: kraft Affects Versions: 3.8.0 Reporter: Christo Lolov BrokerMetadataPublisher#onMetadataUpdate calls ReplicaManager#applyDelta (1) which in turn calls RemoteLogManager#onLeadershipChange (2), however, the RemoteLogManager is configured after the BrokerMetadataPublisher starts running (3, 4). This is incorrect, we either need to initialise the RemoteLogManager before we start the BrokerMetadataPublisher or we need to skip calls to onLeadershipChange if the RemoteLogManager is not initialised. (1) [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L151] (2) [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L2737] (3) [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L432] (4) [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L515] The way to reproduce the problem is by looking at the following branch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#00875a}TaskManagerTest{color} (owner: Christo)
[https://github.com/apache/kafka/pull/13874,
https://github.com/apache/kafka/pull/13897,
https://github.com/apache/kafka/pull/13873|https://github.com/apache/kafka/pull/13874]
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#00875a}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo) [https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StandbyTaskTest{color} (owner: Matthew) (takeover: Christo)
# {color:#00875a}StoreChangelogReaderTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#00875a}StreamTaskTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/14716]
# {color:#00875a}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StreamsMetricsImplTest{color} (owner: Dalibor) (takeover:
Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskC
[jira] [Created] (KAFKA-16480) ListOffsets change should have an associated API/IBP version update
Christo Lolov created KAFKA-16480: - Summary: ListOffsets change should have an associated API/IBP version update Key: KAFKA-16480 URL: https://issues.apache.org/jira/browse/KAFKA-16480 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov Assignee: Christo Lolov https://issues.apache.org/jira/browse/KAFKA-16154 introduced the changes to the ListOffsets API to accept latest-tiered-timestamp and return the corresponding offset. Those changes should have a) increased the version of the ListOffsets API b) increased the inter-broker protocol version c) hidden the latest version of the ListOffsets behind the latestVersionUnstable flag The purpose of this task is to remedy said miss -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14576) Move ConsoleConsumer to tools
[ https://issues.apache.org/jira/browse/KAFKA-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810066#comment-17810066 ] Christo Lolov commented on KAFKA-14576: --- Sure [~mimaison] , apologies for not unassigned myself sooner! > Move ConsoleConsumer to tools > - > > Key: KAFKA-14576 > URL: https://issues.apache.org/jira/browse/KAFKA-14576 > Project: Kafka > Issue Type: Sub-task >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16154) Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP
Christo Lolov created KAFKA-16154: - Summary: Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP Key: KAFKA-16154 URL: https://issues.apache.org/jira/browse/KAFKA-16154 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov Assignee: Christo Lolov Fix For: 3.8.0 A broker should start returning offsets when given a timestamp of -5, which signifies a LATEST_TIERED_TIMESTAMP. There are 3 cases. Tiered Storage is not enabled. In such a situation asking for LATEST_TIERED_TIMESTAMP should always return no offset. Tiered Storage is enabled and there is nothing in remote storage. In such a situation the offset returned should be 0. Tiered Storage is enabled and there is something in remote storage. In such a situation the offset returned should be the highest offset the broker is aware of. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15734) KRaft support in BaseConsumerTest
[
https://issues.apache.org/jira/browse/KAFKA-15734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov resolved KAFKA-15734.
---
Resolution: Fixed
> KRaft support in BaseConsumerTest
> -
>
> Key: KAFKA-15734
> URL: https://issues.apache.org/jira/browse/KAFKA-15734
> Project: Kafka
> Issue Type: Task
> Components: core
>Reporter: Sameer Tejani
>Assignee: Sushant Mahajan
>Priority: Minor
> Labels: kraft, kraft-test, newbie
>
> The following tests in BaseConsumerTest in
> core/src/test/scala/integration/kafka/api/BaseConsumerTest.scala need to be
> updated to support KRaft
> 38 : def testSimpleConsumption(): Unit = {
> 57 : def testClusterResourceListener(): Unit = {
> 78 : def testCoordinatorFailover(): Unit = {
> Scanned 125 lines. Found 0 KRaft tests out of 3 tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15734) KRaft support in BaseConsumerTest
[
https://issues.apache.org/jira/browse/KAFKA-15734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807177#comment-17807177
]
Christo Lolov commented on KAFKA-15734:
---
Resolving this ticket!
> KRaft support in BaseConsumerTest
> -
>
> Key: KAFKA-15734
> URL: https://issues.apache.org/jira/browse/KAFKA-15734
> Project: Kafka
> Issue Type: Task
> Components: core
>Reporter: Sameer Tejani
>Assignee: Sushant Mahajan
>Priority: Minor
> Labels: kraft, kraft-test, newbie
>
> The following tests in BaseConsumerTest in
> core/src/test/scala/integration/kafka/api/BaseConsumerTest.scala need to be
> updated to support KRaft
> 38 : def testSimpleConsumption(): Unit = {
> 57 : def testClusterResourceListener(): Unit = {
> 78 : def testCoordinatorFailover(): Unit = {
> Scanned 125 lines. Found 0 KRaft tests out of 3 tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15147) Measure pending and outstanding Remote Segment operations
[ https://issues.apache.org/jira/browse/KAFKA-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805734#comment-17805734 ] Christo Lolov commented on KAFKA-15147: --- Heya [~fvisconte], a couple of days ago [~showuon] also noticed discrepancies and hopefully they should be addressed as part of https://github.com/apache/kafka/pull/15133! > Measure pending and outstanding Remote Segment operations > - > > Key: KAFKA-15147 > URL: https://issues.apache.org/jira/browse/KAFKA-15147 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Christo Lolov >Priority: Major > Labels: tiered-storage > Fix For: 3.7.0 > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Upload+and+delete+lag+metrics+in+Tiered+Storage > > KAFKA-15833: RemoteCopyLagBytes > KAFKA-16002: RemoteCopyLagSegments, RemoteDeleteLagBytes, > RemoteDeleteLagSegments > KAFKA-16013: ExpiresPerSec > KAFKA-16014: RemoteLogSizeComputationTime, RemoteLogSizeBytes, > RemoteLogMetadataCount > KAFKA-15158: RemoteDeleteRequestsPerSec, RemoteDeleteErrorsPerSec, > BuildRemoteLogAuxStateRequestsPerSec, BuildRemoteLogAuxStateErrorsPerSec > > Remote Log Segment operations (copy/delete) are executed by the Remote > Storage Manager, and recorded by Remote Log Metadata Manager (e.g. default > TopicBasedRLMM writes to the internal Kafka topic state changes on remote log > segments). > As executions run, fail, and retry; it will be important to know how many > operations are pending and outstanding over time to alert operators. > Pending operations are not enough to alert, as values can oscillate closer to > zero. An additional condition needs to apply (running time > threshold) to > consider an operation outstanding. > Proposal: > RemoteLogManager could be extended with 2 concurrent maps > (pendingSegmentCopies, pendingSegmentDeletes) `Map[Uuid, Long]` to measure > segmentId time when operation started, and based on this expose 2 metrics per > operation: > * pendingSegmentCopies: gauge of pendingSegmentCopies map > * outstandingSegmentCopies: loop over pending ops, and if now - startedTime > > timeout, then outstanding++ (maybe on debug level?) > Is this a valuable metric to add to Tiered Storage? or better to solve on a > custom RLMM implementation? > Also, does it require a KIP? > Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
[https://github.com/apache/kafka/pull/13874,
https://github.com/apache/kafka/pull/13897,
https://github.com/apache/kafka/pull/13873|https://github.com/apache/kafka/pull/13874]
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#00875a}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo) [https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StandbyTaskTest{color} (owner: Matthew) (takeover: Christo)
# {color:#00875a}StoreChangelogReaderTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/14716]
# {color:#00875a}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StreamsMetricsImplTest{color} (owner: Dalibor) (takeover:
Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskC
[jira] [Commented] (KAFKA-15147) Measure pending and outstanding Remote Segment operations
[ https://issues.apache.org/jira/browse/KAFKA-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17802238#comment-17802238 ] Christo Lolov commented on KAFKA-15147: --- Heya [~enether], after a very quick search the metric documentation appears to actually be in the kafka codebase itself so I will open a pull request against the 3.7 branch tomorrow morning and tag you as a reviewer! > Measure pending and outstanding Remote Segment operations > - > > Key: KAFKA-15147 > URL: https://issues.apache.org/jira/browse/KAFKA-15147 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Christo Lolov >Priority: Major > Labels: tiered-storage > Fix For: 3.7.0 > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Upload+and+delete+lag+metrics+in+Tiered+Storage > > KAFKA-15833: RemoteCopyLagBytes > KAFKA-16002: RemoteCopyLagSegments, RemoteDeleteLagBytes, > RemoteDeleteLagSegments > KAFKA-16013: ExpiresPerSec > KAFKA-16014: RemoteLogSizeComputationTime, RemoteLogSizeBytes, > RemoteLogMetadataCount > KAFKA-15158: RemoteDeleteRequestsPerSec, RemoteDeleteErrorsPerSec, > BuildRemoteLogAuxStateRequestsPerSec, BuildRemoteLogAuxStateErrorsPerSec > > Remote Log Segment operations (copy/delete) are executed by the Remote > Storage Manager, and recorded by Remote Log Metadata Manager (e.g. default > TopicBasedRLMM writes to the internal Kafka topic state changes on remote log > segments). > As executions run, fail, and retry; it will be important to know how many > operations are pending and outstanding over time to alert operators. > Pending operations are not enough to alert, as values can oscillate closer to > zero. An additional condition needs to apply (running time > threshold) to > consider an operation outstanding. > Proposal: > RemoteLogManager could be extended with 2 concurrent maps > (pendingSegmentCopies, pendingSegmentDeletes) `Map[Uuid, Long]` to measure > segmentId time when operation started, and based on this expose 2 metrics per > operation: > * pendingSegmentCopies: gauge of pendingSegmentCopies map > * outstandingSegmentCopies: loop over pending ops, and if now - startedTime > > timeout, then outstanding++ (maybe on debug level?) > Is this a valuable metric to add to Tiered Storage? or better to solve on a > custom RLMM implementation? > Also, does it require a KIP? > Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800833#comment-17800833
]
Christo Lolov commented on KAFKA-14132:
---
Heya [~enether], thanks for checking in on this one! I need to do another pass
to see whether everything with PowerMock has indeed been moved - I still have a
couple of classes with EasyMock which need to be moved and given the timeline
that won't happen for 3.7. What is left here, which I haven't documented, is
that the PowerMock dependency needs to be removed from Kafka and the build
script needs to be changed!
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.8.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#00875a}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> # KafkaConfigBackingStoreTest (owner: [~bachmanity1])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15147) Measure pending and outstanding Remote Segment operations
[ https://issues.apache.org/jira/browse/KAFKA-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800832#comment-17800832 ] Christo Lolov commented on KAFKA-15147: --- Heya [~enether]! I believe all the work that we wanted to do in order to close KIP-963 is done. The only remaining bit is to update the JMX documentation on the Apache Kafka website, but as far as I am aware that is in a different repository and I was planning on doing it after New Years. Is this suitable for the 3.7 release or should I try to find time in the next couple of days to get the documentation sorted? > Measure pending and outstanding Remote Segment operations > - > > Key: KAFKA-15147 > URL: https://issues.apache.org/jira/browse/KAFKA-15147 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Christo Lolov >Priority: Major > Labels: tiered-storage > Fix For: 3.7.0 > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Upload+and+delete+lag+metrics+in+Tiered+Storage > > KAFKA-15833: RemoteCopyLagBytes > KAFKA-16002: RemoteCopyLagSegments, RemoteDeleteLagBytes, > RemoteDeleteLagSegments > KAFKA-16013: ExpiresPerSec > KAFKA-16014: RemoteLogSizeComputationTime, RemoteLogSizeBytes, > RemoteLogMetadataCount > KAFKA-15158: RemoteDeleteRequestsPerSec, RemoteDeleteErrorsPerSec, > BuildRemoteLogAuxStateRequestsPerSec, BuildRemoteLogAuxStateErrorsPerSec > > Remote Log Segment operations (copy/delete) are executed by the Remote > Storage Manager, and recorded by Remote Log Metadata Manager (e.g. default > TopicBasedRLMM writes to the internal Kafka topic state changes on remote log > segments). > As executions run, fail, and retry; it will be important to know how many > operations are pending and outstanding over time to alert operators. > Pending operations are not enough to alert, as values can oscillate closer to > zero. An additional condition needs to apply (running time > threshold) to > consider an operation outstanding. > Proposal: > RemoteLogManager could be extended with 2 concurrent maps > (pendingSegmentCopies, pendingSegmentDeletes) `Map[Uuid, Long]` to measure > segmentId time when operation started, and based on this expose 2 metrics per > operation: > * pendingSegmentCopies: gauge of pendingSegmentCopies map > * outstandingSegmentCopies: loop over pending ops, and if now - startedTime > > timeout, then outstanding++ (maybe on debug level?) > Is this a valuable metric to add to Tiered Storage? or better to solve on a > custom RLMM implementation? > Also, does it require a KIP? > Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16002) Implement RemoteCopyLagSegments, RemoteDeleteLagBytes and RemoteDeleteLagSegments
Christo Lolov created KAFKA-16002: - Summary: Implement RemoteCopyLagSegments, RemoteDeleteLagBytes and RemoteDeleteLagSegments Key: KAFKA-16002 URL: https://issues.apache.org/jira/browse/KAFKA-16002 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov Assignee: Christo Lolov Fix For: 3.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
[https://github.com/apache/kafka/pull/13874,
https://github.com/apache/kafka/pull/13897,
https://github.com/apache/kafka/pull/13873|https://github.com/apache/kafka/pull/13874]
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#00875a}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo) [https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StandbyTaskTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/14716]
# {color:#00875a}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StreamsMetricsImplTest{color} (owner: Dalibor) (takeover:
Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskC
[jira] [Created] (KAFKA-15883) Implement RemoteCopyLagBytes
Christo Lolov created KAFKA-15883: - Summary: Implement RemoteCopyLagBytes Key: KAFKA-15883 URL: https://issues.apache.org/jira/browse/KAFKA-15883 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov Assignee: Christo Lolov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.
[ https://issues.apache.org/jira/browse/KAFKA-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787965#comment-17787965 ] Christo Lolov commented on KAFKA-15857: --- I will take a stab at this one because of https://github.com/apache/kafka/pull/14788 > Introduce LocalLogStartOffset and TieredOffset in OffsetSpec. > - > > Key: KAFKA-15857 > URL: https://issues.apache.org/jira/browse/KAFKA-15857 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Christo Lolov >Priority: Major > Labels: need-kip, tiered-storage > > Introduce EarliestLocalOffset and TieredOffset in OffsetSpec which will help > in finding respective offsets while using AdminClient#listOffsets(). > EarliestLocalOffset - local log start offset of a topic partition. > TieredOffset - Highest offset up to which the segments were copied to remote > storage. > We can discuss further on naming and semantics of these offset specs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.
[ https://issues.apache.org/jira/browse/KAFKA-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov reassigned KAFKA-15857: - Assignee: Christo Lolov > Introduce LocalLogStartOffset and TieredOffset in OffsetSpec. > - > > Key: KAFKA-15857 > URL: https://issues.apache.org/jira/browse/KAFKA-15857 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Satish Duggana >Assignee: Christo Lolov >Priority: Major > Labels: need-kip, tiered-storage > > Introduce EarliestLocalOffset and TieredOffset in OffsetSpec which will help > in finding respective offsets while using AdminClient#listOffsets(). > EarliestLocalOffset - local log start offset of a topic partition. > TieredOffset - Highest offset up to which the segments were copied to remote > storage. > We can discuss further on naming and semantics of these offset specs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
[https://github.com/apache/kafka/pull/13874,
https://github.com/apache/kafka/pull/13897,
https://github.com/apache/kafka/pull/13873|https://github.com/apache/kafka/pull/13874]
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo) [https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StandbyTaskTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/14716]
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StreamsMetricsImplTest{color} (owner: Dalibor) (takeover:
Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTask
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
[https://github.com/apache/kafka/pull/13874,
https://github.com/apache/kafka/pull/13897,
https://github.com/apache/kafka/pull/13873|https://github.com/apache/kafka/pull/13874]
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo) [https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StandbyTaskTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/13932]
# {color:#00875a}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947) (takeover: Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleF
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
[https://github.com/apache/kafka/pull/13874,
https://github.com/apache/kafka/pull/13897,
https://github.com/apache/kafka/pull/13873|https://github.com/apache/kafka/pull/13874]
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo) [https://github.com/apache/kafka/pull/13932]
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524] (takeover: Christo)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
[https://github.com/apache/kafka/pull/12524]
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
[https://github.com/apache/kafka/pull/13932]
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947) (takeover: Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlus
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947) (takeover: Christo)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (own
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-15388) Handle topics that were having compaction as retention earlier are changed to delete only retention policy and onboarded to tiered storage.
[
https://issues.apache.org/jira/browse/KAFKA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15388:
--
Description:
Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
There are 3 paths I looked at:
* When data is moved to remote storage (1)
* When data is read from remote storage (2)
* When data is deleted from remote storage (3)
(1) Does not have a problem with compacted topics. Compacted segments are
uploaded and their metadata claims they contain offset from the baseOffset of
the segment until the next segment's baseOffset. There are no gaps in offsets.
(2) Does not have a problem if a customer is querying offsets which do not
exist within a segment, but there are offset after the queried offset within
the same segment. *However, it does have a problem when the next available
offset is in a subsequent segment.*
(3) For data deleted via DeleteRecords there is no problem. For data deleted
via retention there is no problem.
*I believe the proper solution to (2) is to make tiered storage continue
looking for the next greater offset in subsequent segments.*
Steps to reproduce the issue:
{code:java}
// TODO (christo)
{code}
was:
Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
There are 3 paths I looked at: * When data is moved to remote storage (1)
* When data is read from remote storage (2)
* When data is deleted from remote storage (3)
(1) Does not have a problem with compacted topics. Compacted segments are
uploaded and their metadata claims they contain offset from the baseOffset of
the segment until the next segment's baseOffset. There are no gaps in offsets.
(2) Does not have a problem if a customer is querying offsets which do not
exist within a segment, but there are offset after the queried offset within
the same segment. *However, it does have a problem when the next available
offset is in a subsequent segment.*
(3) For data deleted via DeleteRecords there is no problem. For data deleted
via retention there is no problem.
I believe the proper solution to (2) is to make tiered storage continue looking
for the next greater offset in subsequent segments.
Steps to reproduce the issue:
{code:java}
// TODO (christo)
{code}
> Handle topics that were having compaction as retention earlier are changed to
> delete only retention policy and onboarded to tiered storage.
>
>
> Key: KAFKA-15388
> URL: https://issues.apache.org/jira/browse/KAFKA-15388
> Project: Kafka
> Issue Type: Bug
>Reporter: Satish Duggana
>Priority: Blocker
> Fix For: 3.7.0
>
>
> Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
>
> There are 3 paths I looked at:
> * When data is moved to remote storage (1)
> * When data is read from remote storage (2)
> * When data is deleted from remote storage (3)
> (1) Does not have a problem with compacted topics. Compacted segments are
> uploaded and their metadata claims they contain offset from the baseOffset of
> the segment until the next segment's baseOffset. There are no gaps in offsets.
> (2) Does not have a problem if a customer is querying offsets which do not
> exist within a segment, but there are offset after the queried offset within
> the same segment. *However, it does have a problem when the next available
> offset is in a subsequent segment.*
> (3) For data deleted via DeleteRecords there is no problem. For data deleted
> via retention there is no problem.
>
> *I believe the proper solution to (2) is to make tiered storage continue
> looking for the next greater offset in subsequent segments.*
> Steps to reproduce the issue:
> {code:java}
> // TODO (christo)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15388) Handle topics that were having compaction as retention earlier are changed to delete only retention policy and onboarded to tiered storage.
[
https://issues.apache.org/jira/browse/KAFKA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15388:
--
Description:
Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
There are 3 paths I looked at: * When data is moved to remote storage (1)
* When data is read from remote storage (2)
* When data is deleted from remote storage (3)
(1) Does not have a problem with compacted topics. Compacted segments are
uploaded and their metadata claims they contain offset from the baseOffset of
the segment until the next segment's baseOffset. There are no gaps in offsets.
(2) Does not have a problem if a customer is querying offsets which do not
exist within a segment, but there are offset after the queried offset within
the same segment. *However, it does have a problem when the next available
offset is in a subsequent segment.*
(3) For data deleted via DeleteRecords there is no problem. For data deleted
via retention there is no problem.
I believe the proper solution to (2) is to make tiered storage continue looking
for the next greater offset in subsequent segments.
Steps to reproduce the issue:
{code:java}
// TODO (christo)
{code}
was:
Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
There are 3 paths I looked at: * When data is moved to remote storage (1)
* When data is read from remote storage (2)
* When data is deleted from remote storage (3)
(1) Does not have a problem with compacted topics. Compacted segments are
uploaded and their metadata claims they contain offset from the baseOffset of
the segment until the next segment's baseOffset. There are no gaps in offsets.
(2) Does not have a problem if a customer is querying offsets which do not
exist within a segment, but there are offset after the queried offset within
the same segment. *However, it does have a problem when the next available
offset is in a subsequent segment.*
(3) For data deleted via DeleteRecords there is no problem. For data deleted
via retention there is no problem.
I believe the proper solution to (2) is to make tiered storage continue looking
for the next greater offset in subsequent segments.
> Handle topics that were having compaction as retention earlier are changed to
> delete only retention policy and onboarded to tiered storage.
>
>
> Key: KAFKA-15388
> URL: https://issues.apache.org/jira/browse/KAFKA-15388
> Project: Kafka
> Issue Type: Bug
>Reporter: Satish Duggana
>Priority: Blocker
> Fix For: 3.7.0
>
>
> Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
>
> There are 3 paths I looked at: * When data is moved to remote storage (1)
> * When data is read from remote storage (2)
> * When data is deleted from remote storage (3)
>
> (1) Does not have a problem with compacted topics. Compacted segments are
> uploaded and their metadata claims they contain offset from the baseOffset of
> the segment until the next segment's baseOffset. There are no gaps in offsets.
> (2) Does not have a problem if a customer is querying offsets which do not
> exist within a segment, but there are offset after the queried offset within
> the same segment. *However, it does have a problem when the next available
> offset is in a subsequent segment.*
> (3) For data deleted via DeleteRecords there is no problem. For data deleted
> via retention there is no problem.
>
> I believe the proper solution to (2) is to make tiered storage continue
> looking for the next greater offset in subsequent segments.
>
> Steps to reproduce the issue:
> {code:java}
> // TODO (christo)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15388) Handle topics that were having compaction as retention earlier are changed to delete only retention policy and onboarded to tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov updated KAFKA-15388: -- Description: Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517] There are 3 paths I looked at: * When data is moved to remote storage (1) * When data is read from remote storage (2) * When data is deleted from remote storage (3) (1) Does not have a problem with compacted topics. Compacted segments are uploaded and their metadata claims they contain offset from the baseOffset of the segment until the next segment's baseOffset. There are no gaps in offsets. (2) Does not have a problem if a customer is querying offsets which do not exist within a segment, but there are offset after the queried offset within the same segment. *However, it does have a problem when the next available offset is in a subsequent segment.* (3) For data deleted via DeleteRecords there is no problem. For data deleted via retention there is no problem. I believe the proper solution to (2) is to make tiered storage continue looking for the next greater offset in subsequent segments. was:Context: https://github.com/apache/kafka/pull/13561#discussion_r1300055517 > Handle topics that were having compaction as retention earlier are changed to > delete only retention policy and onboarded to tiered storage. > > > Key: KAFKA-15388 > URL: https://issues.apache.org/jira/browse/KAFKA-15388 > Project: Kafka > Issue Type: Bug >Reporter: Satish Duggana >Priority: Blocker > Fix For: 3.7.0 > > > Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517] > > There are 3 paths I looked at: * When data is moved to remote storage (1) > * When data is read from remote storage (2) > * When data is deleted from remote storage (3) > > (1) Does not have a problem with compacted topics. Compacted segments are > uploaded and their metadata claims they contain offset from the baseOffset of > the segment until the next segment's baseOffset. There are no gaps in offsets. > (2) Does not have a problem if a customer is querying offsets which do not > exist within a segment, but there are offset after the queried offset within > the same segment. *However, it does have a problem when the next available > offset is in a subsequent segment.* > (3) For data deleted via DeleteRecords there is no problem. For data deleted > via retention there is no problem. > > I believe the proper solution to (2) is to make tiered storage continue > looking for the next greater offset in subsequent segments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15388) Handle topics that were having compaction as retention earlier are changed to delete only retention policy and onboarded to tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1739#comment-1739 ] Christo Lolov commented on KAFKA-15388: --- Heya [~jimmywang611], apologies for the delay. I am actively working on this, I was just gathering data, which I will share now. > Handle topics that were having compaction as retention earlier are changed to > delete only retention policy and onboarded to tiered storage. > > > Key: KAFKA-15388 > URL: https://issues.apache.org/jira/browse/KAFKA-15388 > Project: Kafka > Issue Type: Bug >Reporter: Satish Duggana >Priority: Blocker > Fix For: 3.7.0 > > > Context: https://github.com/apache/kafka/pull/13561#discussion_r1300055517 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15660) File-based Tiered Storage should delete folders upon topic deletion
Christo Lolov created KAFKA-15660:
-
Summary: File-based Tiered Storage should delete folders upon
topic deletion
Key: KAFKA-15660
URL: https://issues.apache.org/jira/browse/KAFKA-15660
Project: Kafka
Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Christo Lolov
We have added a quick-start guide for Tiered Storage as part of Apache Kafka
3.6 - [https://kafka.apache.org/documentation/#tiered_storage_config_ex.]
When interacting with it, however, it appears that when topics are deleted
while remote segments and their indecies are deleted the folders are not:
{code:java}
> ls /tmp/kafka-remote-storage/kafka-tiered-storage
A-0-ApBdPOE1SOOw-Ie8RQLuAA B-0-2omLZKw1Tiu2-EUKsIzj9Q
C-0-FXdccGWXQJCj-RQynsOK3Q D-0-vqfdYtYLSlWEyXp6cwwmpg
> ls /tmp/kafka-remote-storage/kafka-tiered-storage/A-0-ApBdPOE1SOOw-Ie8RQLuAA
{code}
I think that the file-based implementation shipping with Kafka should delete
the folders as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15660) File-based Tiered Storage should delete folders upon topic deletion
[
https://issues.apache.org/jira/browse/KAFKA-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15660:
--
Description:
We have added a quick-start guide for Tiered Storage as part of Apache Kafka
3.6 - [https://kafka.apache.org/documentation/#tiered_storage_config_ex.]
When interacting with it, however, it appears that when topics are deleted
while remote segments and their indecies are deleted the folders are not:
{code:java}
> ls /tmp/kafka-remote-storage/kafka-tiered-storage
A-0-ApBdPOE1SOOw-Ie8RQLuAA B-0-2omLZKw1Tiu2-EUKsIzj9Q
C-0-FXdccGWXQJCj-RQynsOK3Q D-0-vqfdYtYLSlWEyXp6cwwmpg
> ls /tmp/kafka-remote-storage/kafka-tiered-storage/A-0-ApBdPOE1SOOw-Ie8RQLuAA
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
__remote_log_metadata{code}
I think that the file-based implementation shipping with Kafka should delete
the folders as well.
was:
We have added a quick-start guide for Tiered Storage as part of Apache Kafka
3.6 - [https://kafka.apache.org/documentation/#tiered_storage_config_ex.]
When interacting with it, however, it appears that when topics are deleted
while remote segments and their indecies are deleted the folders are not:
{code:java}
> ls /tmp/kafka-remote-storage/kafka-tiered-storage
A-0-ApBdPOE1SOOw-Ie8RQLuAA B-0-2omLZKw1Tiu2-EUKsIzj9Q
C-0-FXdccGWXQJCj-RQynsOK3Q D-0-vqfdYtYLSlWEyXp6cwwmpg
> ls /tmp/kafka-remote-storage/kafka-tiered-storage/A-0-ApBdPOE1SOOw-Ie8RQLuAA
{code}
I think that the file-based implementation shipping with Kafka should delete
the folders as well.
> File-based Tiered Storage should delete folders upon topic deletion
> ---
>
> Key: KAFKA-15660
> URL: https://issues.apache.org/jira/browse/KAFKA-15660
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Christo Lolov
>Priority: Minor
>
> We have added a quick-start guide for Tiered Storage as part of Apache Kafka
> 3.6 - [https://kafka.apache.org/documentation/#tiered_storage_config_ex.]
> When interacting with it, however, it appears that when topics are deleted
> while remote segments and their indecies are deleted the folders are not:
> {code:java}
> > ls /tmp/kafka-remote-storage/kafka-tiered-storage
> A-0-ApBdPOE1SOOw-Ie8RQLuAA B-0-2omLZKw1Tiu2-EUKsIzj9Q
> C-0-FXdccGWXQJCj-RQynsOK3Q D-0-vqfdYtYLSlWEyXp6cwwmpg
> > ls /tmp/kafka-remote-storage/kafka-tiered-storage/A-0-ApBdPOE1SOOw-Ie8RQLuAA
> > bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
> __remote_log_metadata{code}
> I think that the file-based implementation shipping with Kafka should delete
> the folders as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Commented] (KAFKA-14132) Remaining PowerMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776221#comment-17776221
]
Christo Lolov commented on KAFKA-14132:
---
I support the target :). I will hunt down the remaining ones in the sibling
JIRA.
> Remaining PowerMock to Mockito tests
>
>
> Key: KAFKA-14132
> URL: https://issues.apache.org/jira/browse/KAFKA-14132
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.7.0
>
>
> {color:#de350b}Some of the tests below use EasyMock as well. For those
> migrate both PowerMock and EasyMock to Mockito.{color}
> Unless stated in brackets the tests are in the connect module.
> A list of tests which still require to be moved from PowerMock to Mockito as
> of 2nd of August 2022 which do not have a Jira issue and do not have pull
> requests I am aware of which are opened:
> {color:#ff8b00}InReview{color}
> {color:#00875a}Merged{color}
> # {color:#00875a}ErrorHandlingTaskTest{color} (owner: [~shekharrajak])
> # {color:#00875a}SourceTaskOffsetCommiterTest{color} (owner: Christo)
> # {color:#00875a}WorkerMetricsGroupTest{color} (owner: Divij)
> # {color:#00875a}WorkerTaskTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ErrorReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RetryWithToleranceOperatorTest{color} (owner: [~yash.mayya])
> # {color:#00875a}WorkerErrantRecordReporterTest{color} (owner: [~yash.mayya])
> # {color:#00875a}ConnectorsResourceTest{color} (owner: [~mdedetrich-aiven])
> # {color:#ff8b00}StandaloneHerderTest{color} (owner: [~mdedetrich-aiven])
> ([https://github.com/apache/kafka/pull/12728])
> # KafkaConfigBackingStoreTest (owner: [~mdedetrich-aiven])
> # {color:#00875a}KafkaOffsetBackingStoreTest{color} (owner: Christo)
> ([https://github.com/apache/kafka/pull/12418])
> # {color:#00875a}KafkaBasedLogTest{color} (owner: @bachmanity ])
> # {color:#00875a}RetryUtilTest{color} (owner: [~yash.mayya])
> # {color:#00875a}RepartitionTopicTest{color} (streams) (owner: Christo)
> # {color:#00875a}StateManagerUtilTest{color} (streams) (owner: Christo)
> *The coverage report for the above tests after the change should be >= to
> what the coverage is now.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15385) Replace EasyMock with Mockito for AbstractStreamTest
[ https://issues.apache.org/jira/browse/KAFKA-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15385. --- Resolution: Duplicate Closing this in favour of https://issues.apache.org/jira/browse/KAFKA-14133 > Replace EasyMock with Mockito for AbstractStreamTest > > > Key: KAFKA-15385 > URL: https://issues.apache.org/jira/browse/KAFKA-15385 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: Fei Xie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15384) Replace EasyMock with Mockito for KTableTransformValuesTest
[ https://issues.apache.org/jira/browse/KAFKA-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15384. --- Resolution: Fixed Closing this ticket in favour of https://issues.apache.org/jira/browse/KAFKA-14133 > Replace EasyMock with Mockito for KTableTransformValuesTest > --- > > Key: KAFKA-15384 > URL: https://issues.apache.org/jira/browse/KAFKA-15384 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: Fei Xie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15383) Replace EasyMock with Mockito for KTableImplTest
[ https://issues.apache.org/jira/browse/KAFKA-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15383. --- Resolution: Duplicate Closing this ticket in favour of https://issues.apache.org/jira/browse/KAFKA-14133 > Replace EasyMock with Mockito for KTableImplTest > > > Key: KAFKA-15383 > URL: https://issues.apache.org/jira/browse/KAFKA-15383 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: Fei Xie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15382) Replace EasyMock with Mockito for KStreamTransformValuesTest
[ https://issues.apache.org/jira/browse/KAFKA-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15382. --- Resolution: Duplicate Closing this ticket in favour of https://issues.apache.org/jira/browse/KAFKA-14133 > Replace EasyMock with Mockito for KStreamTransformValuesTest > > > Key: KAFKA-15382 > URL: https://issues.apache.org/jira/browse/KAFKA-15382 > Project: Kafka > Issue Type: Sub-task > Components: streams, unit tests >Reporter: Fei Xie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#00875a}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Commented] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test
[ https://issues.apache.org/jira/browse/KAFKA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761727#comment-17761727 ] Christo Lolov commented on KAFKA-15399: --- Heya [~pnee], [~lianetm] and [~showuon]! I had a look at the last 5 builds on trunk since [https://github.com/apache/kafka/pull/14319] was merged in and none of them contain the test failure detailed in https://issues.apache.org/jira/browse/KAFKA-15427: * [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2164/#showFailuresLink] * [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2165/] * [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2166/#showFailuresLink] * [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2167/] * [https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2168/] I believe the issue is resolved so I will keep this Jira ticket as resolved as well, but if you manage to find an occurrence I will gladly reopen it! > Enable OffloadAndConsumeFromLeader test > --- > > Key: KAFKA-15399 > URL: https://issues.apache.org/jira/browse/KAFKA-15399 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 3.6.0 >Reporter: Kamal Chandraprakash >Assignee: Kamal Chandraprakash >Priority: Blocker > Fix For: 3.6.0 > > > Build / JDK 17 and Scala 2.13 / initializationError – > org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test
[ https://issues.apache.org/jira/browse/KAFKA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christo Lolov resolved KAFKA-15399. --- Resolution: Fixed > Enable OffloadAndConsumeFromLeader test > --- > > Key: KAFKA-15399 > URL: https://issues.apache.org/jira/browse/KAFKA-15399 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 3.6.0 >Reporter: Kamal Chandraprakash >Assignee: Kamal Chandraprakash >Priority: Blocker > Fix For: 3.6.0 > > > Build / JDK 17 and Scala 2.13 / initializationError – > org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test
[ https://issues.apache.org/jira/browse/KAFKA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761301#comment-17761301 ] Christo Lolov commented on KAFKA-15399: --- I have detailed the reason for the failing integration test in https://issues.apache.org/jira/browse/KAFKA-15427. I will proceed to close this ticket, but if there is something I have missed feel free to reopen it! > Enable OffloadAndConsumeFromLeader test > --- > > Key: KAFKA-15399 > URL: https://issues.apache.org/jira/browse/KAFKA-15399 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 3.6.0 >Reporter: Kamal Chandraprakash >Assignee: Kamal Chandraprakash >Priority: Blocker > Fix For: 3.6.0 > > > Build / JDK 17 and Scala 2.13 / initializationError – > org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15427) Integration tests in TS test harness detect resource leaks
[
https://issues.apache.org/jira/browse/KAFKA-15427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15427:
--
Description:
The pull request ([https://github.com/apache/kafka/pull/14116]) for adding the
Tiered Storage test harness uncovered resource leaks as part of the build
([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
This can be reproduced locally by running the following command:
{code:java}
./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
--tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
--tests
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
--rerun{code}
Output:
{code:java}
> Task :storage:testGradle Test Run :storage:test > Gradle Test Executor 3 >
> RemoteLogMetadataManagerTest >
> testRemotePartitionDeletion(RemoteLogMetadataManager) >
> remoteLogMetadataManager =
> org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@4cc76301
> PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest >
testRemotePartitionDeletion(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@2ca47471
PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@ce12fbb
PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@69aabcb0
PASSED
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError
failed, log available in
/Users/lolovc/Documents/kafka/storage/build/reports/testOutput/org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError.test.stdoutGradle
Test Run :storage:test > Gradle Test Executor 3 >
OffloadAndConsumeFromLeaderTest > initializationError FAILED
org.opentest4j.AssertionFailedError: Found 2 unexpected threads during
@BeforeAll: `controller-event-thread,Test worker-EventThread` ==> expected:
but was:
... {code}
The point of this Jira ticket is to find the resource leak and fix it
was:
The pull request ([https://github.com/apache/kafka/pull/14116]) for adding the
Tiered Storage test harness uncovered resource leaks as part of the build
([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
This can be reproduced locally by running the following command:
{code:java}
./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
--tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
--tests
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
--rerun{code}
Output:
{code:java}
> Task :storage:testGradle Test Run :storage:test > Gradle Test Executor 3 >
> RemoteLogMetadataManagerTest >
> testRemotePartitionDeletion(RemoteLogMetadataManager) >
> remoteLogMetadataManager =
> org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@4cc76301
> PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest >
testRemotePartitionDeletion(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@2ca47471
PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@ce12fbb
PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@69aabcb0
PASSED
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError
failed, log available in
/Users/lolovc/Documents/kafka/storage/build/reports/testOutput/org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError.test.stdoutGradle
Test Run :storage:test > Gradle Test Executor 3 >
OffloadAndConsumeFromLe
[jira] [Updated] (KAFKA-15427) Integration tests in TS test harness detect resource leaks
[
https://issues.apache.org/jira/browse/KAFKA-15427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15427:
--
Description:
The pull request ([https://github.com/apache/kafka/pull/14116]) for adding the
Tiered Storage test harness uncovered resource leaks as part of the build
([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
This can be reproduced locally by running the following command:
{code:java}
./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
--tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
--tests
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
--rerun{code}
Output:
{code:java}
> Task :storage:testGradle Test Run :storage:test > Gradle Test Executor 3 >
> RemoteLogMetadataManagerTest >
> testRemotePartitionDeletion(RemoteLogMetadataManager) >
> remoteLogMetadataManager =
> org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@4cc76301
> PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest >
testRemotePartitionDeletion(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@2ca47471
PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@ce12fbb
PASSED
Gradle Test Run :storage:test > Gradle Test Executor 3 >
RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) >
remoteLogMetadataManager =
org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@69aabcb0
PASSED
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError
failed, log available in
/Users/lolovc/Documents/kafka/storage/build/reports/testOutput/org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError.test.stdoutGradle
Test Run :storage:test > Gradle Test Executor 3 >
OffloadAndConsumeFromLeaderTest > initializationError FAILED
org.opentest4j.AssertionFailedError: Found 2 unexpected threads during
@BeforeAll: `controller-event-thread,Test worker-EventThread` ==> expected:
but was:
... {code}
The point of this Jira ticket is to find the resource leak and fix it
was:
The pull request ([https://github.com/apache/kafka/pull/14116]) for adding the
Tiered Storage test harness uncovered resource leaks as part of the build
([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
This can be reproduced locally by running the following command:
{code:java}
./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
--tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
--tests
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
--rerun{code}
The point of this Jira ticket is to find the resource leak and fix it
> Integration tests in TS test harness detect resource leaks
> --
>
> Key: KAFKA-15427
> URL: https://issues.apache.org/jira/browse/KAFKA-15427
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.6.0
>
>
> The pull request ([https://github.com/apache/kafka/pull/14116]) for adding
> the Tiered Storage test harness uncovered resource leaks as part of the build
> ([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
>
> This can be reproduced locally by running the following command:
> {code:java}
> ./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
> --tests
> org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
> --tests
> org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
> --rerun{code}
> Output:
> {code:java}
> > Task :storage:testGradle Test Run :storage:test > Gradle Test Executor 3 >
> > RemoteLogMetadataManagerTest >
> > testRemotePartitionDeletion(RemoteLogMetadataManager) >
> > remoteLogMetadataManager =
> > org.apache.kafka.server.log.remote.storage.InmemoryRemoteLo
[jira] [Updated] (KAFKA-15427) Integration tests in TS test harness detect resource leaks
[
https://issues.apache.org/jira/browse/KAFKA-15427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15427:
--
Description:
The pull request ([https://github.com/apache/kafka/pull/14116]) for adding the
Tiered Storage test harness uncovered resource leaks as part of the build
([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
This can be reproduced locally by running the following command:
{code:java}
./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
--tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
--tests
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
--rerun{code}
The point of this Jira ticket is to find the resource leak and fix it
was:
The pull request (https://github.com/apache/kafka/pull/14116) for adding the
Tiered Storage test harness uncovered resource leaks as part of the build
([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
This can be reproduced locally by running the following command
```
./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
--tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
--tests
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
--rerun
```
The point of this Jira ticket is to find the resource leak and fix it
> Integration tests in TS test harness detect resource leaks
> --
>
> Key: KAFKA-15427
> URL: https://issues.apache.org/jira/browse/KAFKA-15427
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.6.0
>
>
> The pull request ([https://github.com/apache/kafka/pull/14116]) for adding
> the Tiered Storage test harness uncovered resource leaks as part of the build
> ([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)]
>
> This can be reproduced locally by running the following command:
> {code:java}
> ./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test
> --tests
> org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest
> --tests
> org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest
> --rerun{code}
>
> The point of this Jira ticket is to find the resource leak and fix it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15427) Integration tests in TS test harness detect resource leaks
Christo Lolov created KAFKA-15427: - Summary: Integration tests in TS test harness detect resource leaks Key: KAFKA-15427 URL: https://issues.apache.org/jira/browse/KAFKA-15427 Project: Kafka Issue Type: Sub-task Reporter: Christo Lolov Assignee: Christo Lolov Fix For: 3.6.0 The pull request (https://github.com/apache/kafka/pull/14116) for adding the Tiered Storage test harness uncovered resource leaks as part of the build ([https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)] This can be reproduced locally by running the following command ``` ./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test --tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest --tests org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest --rerun ``` The point of this Jira ticket is to find the resource leak and fix it -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#ff8b00}AbstractStreamTest{color} (owner: Christo)
# {color:#00875a}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KTableImplTest{color} (owner: Christo)
# {color:#00875a}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#ff8b00}AbstractStreamTest{color} (owner: Christo)
# {color:#ff8b00}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#ff8b00}KTableImplTest{color} (owner: Christo)
# {color:#ff8b00}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-15267) Cluster-wide disablement of Tiered Storage
[
https://issues.apache.org/jira/browse/KAFKA-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-15267:
--
Fix Version/s: 3.6.0
> Cluster-wide disablement of Tiered Storage
> --
>
> Key: KAFKA-15267
> URL: https://issues.apache.org/jira/browse/KAFKA-15267
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Labels: tiered-storage
> Fix For: 3.6.0
>
>
> h2. Summary
> KIP-405 defines the configuration {{remote.log.storage.system.enable}} which
> controls whether all resources needed for Tiered Storage to function are
> instantiated properly in Kafka. However, the interaction between remote data
> and Kafka if that configuration is set to false while there are still topics
> with {{{}remote.storage.enable is undefined{}}}. {color:#ff8b00}*We would
> like to give customers the ability to switch off Tiered Storage on a cluster
> level and as such would need to define the behaviour.*{color}
> {{remote.log.storage.system.enable}} is a read-only configuration. This means
> that it can only be changed by *modifying the server.properties* and
> restarting brokers. As such, the {*}validity of values contained in it is
> only checked at broker startup{*}.
> This JIRA proposes a few behaviours and a recommendation on a way forward.
> h2. Option 1: Change nothing
> Pros:
> * No operation.
> Cons:
> * We do not solve the problem of moving back to older (or newer) Kafka
> versions not supporting TS.
> h2. Option 2: Remove the configuration, enable Tiered Storage on a cluster
> level and do not allow it to be disabled
> Always instantiate all resources for tiered storage. If no special ones are
> selected use the default ones which come with Kafka.
> Pros:
> * We solve the problem for moving between versions not allowing TS to be
> disabled.
> Cons:
> * We do not solve the problem of moving back to older (or newer) Kafka
> versions not supporting TS.
> * We haven’t quantified how much computer resources (CPU, memory) idle TS
> components occupy.
> * TS is a feature not required for running Kafka. As such, while it is still
> under development we shouldn’t put it on the critical path of starting a
> broker. In this way, a stray memory leak won’t impact anything on the
> critical path of a broker.
> * We are potentially swapping one problem for another. How does TS behave if
> one decides to swap the TS plugin classes when data has already been written?
> h2. Option 3: Hide topics with tiering enabled
> Customers cannot interact with topics which have tiering enabled. They cannot
> create new topics with the same names. Retention (and compaction?) do not
> take effect on files already in local storage.
> Pros:
> * We do not force data-deletion.
> Cons:
> * This will be quite involved - the controller will need to know when a
> broker’s server.properties have been altered; the broker will need to not
> proceed to delete logs it is not the leader or follower for.
> h2. {color:#00875a}Option 4: Do not start the broker if there are topics with
> tiering enabled{color} - Recommended
> This option has 2 different sub-options. The first one is that TS cannot be
> disabled on cluster-level if there are *any* tiering topics - in other words
> all tiered topics need to be deleted. The second one is that TS cannot be
> disabled on a cluster-level if there are *any* topics with *tiering enabled*
> - they can have tiering disabled, but with a retention policy set to delete
> or retain (as per
> [KIP-950|https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement]).
> A topic can have tiering disabled and remain on the cluster as long as there
> is no *remote* data when TS is disabled cluster-wide.
> Pros:
> * We force the customer to be very explicit in disabling tiering of topics
> prior to disabling TS on the whole cluster.
> Cons:
> * You have to make certain that all data in remote is deleted (just a
> disablement of tired topic is not enough). How do you determine whether all
> remote has expired if policy is retain? If retain policy in KIP-950 knows
> that there is data in remote then this should also be able to figure it out.
> The common denominator is that there needs to be no *remote* data at the
> point of disabling TS. As such, the most straightforward option is to refuse
> to start brokers if there are topics with the {{remote.storage.enabled}}
> present. This in essence requires customers to clean any tiered topics before
> switching off TS, which is a fair ask. Should we wish to revise this later it
> should be possible.
> h2. Option 5: Make Kafka forget about all remote information
> Pros:
> * Clean cut
> Cons:
> * Data is lost the moment
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#ff8b00}AbstractStreamTest{color} (owner: Christo)
# {color:#ff8b00}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#ff8b00}KTableImplTest{color} (owner: Christo)
# {color:#ff8b00}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}RecordCollectorTest{color} (owner: Christo)
# {color:#00875a}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#ff8b00}AbstractStreamTest{color} (owner: Christo)
# {color:#ff8b00}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#ff8b00}KTableImplTest{color} (owner: Christo)
# {color:#ff8b00}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#ff8b00}RecordCollectorTest{color} (owner: Christo)
# {color:#ff8b00}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Commented] (KAFKA-15388) Handle topics that were having compaction as retention earlier are changed to delete only retention policy and onboarded to tiered storage.
[ https://issues.apache.org/jira/browse/KAFKA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757487#comment-17757487 ] Christo Lolov commented on KAFKA-15388: --- I would like to contribute to this ticket once the above pull request has been merged by providing the solution for correctly setting the logStartOffset! > Handle topics that were having compaction as retention earlier are changed to > delete only retention policy and onboarded to tiered storage. > > > Key: KAFKA-15388 > URL: https://issues.apache.org/jira/browse/KAFKA-15388 > Project: Kafka > Issue Type: Task >Reporter: Satish Duggana >Priority: Major > Fix For: 3.6.0 > > > Context: https://github.com/apache/kafka/pull/13561#discussion_r1300055517 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# {color:#ff8b00}AbstractStreamTest{color} (owner: Christo)
# {color:#ff8b00}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#ff8b00}KTableImplTest{color} (owner: Christo)
# {color:#ff8b00}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#ff8b00}RecordCollectorTest{color} (owner: Christo)
# {color:#ff8b00}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {col
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# AbstractStreamTest (owner: Christo)
# {color:#ff8b00}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#ff8b00}KTableImplTest{color} (owner: Christo)
# {color:#ff8b00}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#ff8b00}RecordCollectorTest{color} (owner: Christo)
# {color:#ff8b00}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {color:#ff8b00}StoreToProc
[jira] [Updated] (KAFKA-14133) Remaining EasyMock to Mockito tests
[
https://issues.apache.org/jira/browse/KAFKA-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-14133:
--
Description:
{color:#de350b}There are tests which use both PowerMock and EasyMock. I have
put those in https://issues.apache.org/jira/browse/KAFKA-14132. Tests here rely
solely on EasyMock.{color}
Unless stated in brackets the tests are in the streams module.
A list of tests which still require to be moved from EasyMock to Mockito as of
2nd of August 2022 which do not have a Jira issue and do not have pull requests
I am aware of which are opened:
{color:#ff8b00}In Review{color}
{color:#00875a}Merged{color}
# {color:#00875a}WorkerConnectorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}WorkerCoordinatorTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}RootResourceTest{color} (connect) (owner: [~yash.mayya] )
# {color:#00875a}ByteArrayProducerRecordEquals{color} (connect) (owner:
[~yash.mayya] )
# {color:#00875a}KStreamFlatTransformTest{color} (owner: Christo)
# {color:#00875a}KStreamFlatTransformValuesTest{color} (owner: Christo)
# {color:#00875a}KStreamPrintTest{color} (owner: Christo)
# {color:#00875a}KStreamRepartitionTest{color} (owner: Christo)
# {color:#00875a}MaterializedInternalTest{color} (owner: Christo)
# {color:#00875a}TransformerSupplierAdapterTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorMetricsTest{color} (owner: Christo)
# {color:#00875a}ClientUtilsTest{color} (owner: Christo)
# {color:#00875a}HighAvailabilityStreamsPartitionAssignorTest{color} (owner:
Christo)
# {color:#00875a}TopologyTest{color} (owner: Christo)
# {color:#00875a}KTableSuppressProcessorTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingSessionBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedWindowBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}ChangeLoggingWindowBytesStoreTest{color} (owner: Christo)
# {color:#00875a}MeteredTimestampedWindowStoreTest{color} (owner: Christo)
# {color:#00875a}StreamsRebalanceListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedKeyValueStoreMaterializerTest{color} (owner:
Christo)
# {color:#00875a}CachingInMemoryKeyValueStoreTest{color} (owner: Christo)
# {color:#00875a}CachingInMemorySessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentSessionStoreTest{color} (owner: Christo)
# {color:#00875a}CachingPersistentWindowStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingKeyValueBytesStoreTest{color} (owner: Christo)
# {color:#00875a}ChangeLoggingTimestampedKeyValueBytesStoreTest{color} (owner:
Christo)
# {color:#00875a}CompositeReadOnlyWindowStoreTest{color} (owner: Christo)
# {color:#00875a}KeyValueStoreBuilderTest{color} (owner: Christo)
# {color:#00875a}RocksDBStoreTest{color} (owner: Christo)
# {color:#00875a}StreamThreadStateStoreProviderTest{color} (owner: Christo)
# {color:#ff8b00}TaskManagerTest{color} (owner: Christo)
# {color:#00875a}InternalTopicManagerTest{color} (owner: Christo)
# {color:#00875a}ProcessorContextImplTest{color} (owner: Christo)
# {color:#00875a}WriteConsistencyVectorTest{color} (owner: Christo)
# {color:#00875a}StreamsAssignmentScaleTest{color} (owner: Christo)
# {color:#00875a}StreamsPartitionAssignorTest{color} (owner: Christo)
# {color:#00875a}AssignmentTestUtils{color} (owner: Christo)
# {color:#ff8b00}ProcessorStateManagerTest{color} (owner: Matthew) (takeover:
Christo)
# {color:#ff8b00}StandbyTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StoreChangelogReaderTest{color} (owner: Matthew)
# {color:#ff8b00}StreamTaskTest{color} (owner: Matthew)
# {color:#ff8b00}StreamThreadTest{color} (owner: Matthew) (takeover: Christo)
# {color:#ff8b00}StreamsMetricsImplTest{color} (owner: Dalibor) (Captured in
https://issues.apache.org/jira/browse/KAFKA-12947)
# {color:#00875a}TimeOrderedCachingPersistentWindowStoreTest{color} (owner:
[~shekharrajak])
# {color:#00875a}TimeOrderedWindowStoreTest{color} (owner: [~shekharrajak])
[https://github.com/apache/kafka/pull/12777]
# AbstractStreamTest
# {color:#ff8b00}KStreamTransformValuesTest{color} (owner: Christo)
# {color:#ff8b00}KTableImplTest{color} (owner: Christo)
# {color:#ff8b00}KTableTransformValuesTest{color} (owner: Christo)
# {color:#00875a}SessionCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedCacheFlushListenerTest{color} (owner: Christo)
# {color:#00875a}TimestampedTupleForwarderTest{color} (owner: Christo)
# {color:#00875a}ActiveTaskCreatorTest{color} (owner: Christo)
# {color:#00875a}ChangelogTopicsTest{color} (owner: Christo)
# {color:#00875a}GlobalProcessorContextImplTest{color} (owner: Christo)
# {color:#ff8b00}RecordCollectorTest{color} (owner: Christo)
# {color:#ff8b00}StateRestoreCallbackAdapterTest{color} (owner: Christo)
# {color:#ff8b00}StoreToProcessorContextAdapt
