[jira] [Updated] (KAFKA-15990) Add additional parameter which allows parameter remote.enabled and its relatives to be wiped if server tiering is disabled

2023-12-08 Thread Viktor Nikitash (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Nikitash updated KAFKA-15990:

Description: 
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.

In addition we have parameter unclean.leader.election which allows to wipe 
data, I encourage to create additional parameter which allows to do same to 
remote topic configs, as safe lever for migration

  was:
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.


> Add additional parameter which allows parameter remote.enabled and its 
> relatives to be wiped if server tiering is disabled
> --
>
> Key: KAFKA-15990
> URL: https://issues.apache.org/jira/browse/KAFKA-15990
> Project: Kafka
>  Issue Type: Improvement
>  Components: Tiered-Storage
>Reporter: Viktor Nikitash
>Priority: Major
>
> h2. Background
> There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
> which is not working on that versions at all, and when we upgrade kafka to 
> version 3.6.0 we have crash restart loop with message "You have to delete all 
> topics with the property remote.storage.enable=true before disabling tiered 
> storage cluster-wide".
> [https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
> h2. The Idea 
> The idea is to create new parameter which allows to wipe configurations 
> related to remote storage on topic level, with default value set to false 
> (which won't change logic from the current one), but in case we manually set 
> parameter to true, those remote topic parameters will be wiped.
> Of course additional checks required. Like do it only on Kafka startup and 
> ensure that there is no data on remote storage etc...
> h2. Motivation
> Since tiering in to working on version 3.6.0-, there is no any reason to keep 
> this configs during migration from older 3rd version to 3.6.0+. When it comes 
> to managing lots of clusters it can be a problem. So enabling this new 
> parameter before migration to 3.6.+ and disabling it after, makes migration 
> easier.
> In addition we have parameter unclean.leader.election which allows to wipe 
> data, I encourage to create additional parameter which allows to do same to 
> 

[jira] [Updated] (KAFKA-15990) Add additional parameter which allows parameter remote.enabled and its relatives to be wiped if server tiering is disabled

2023-12-08 Thread Viktor Nikitash (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Nikitash updated KAFKA-15990:

Description: 
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.

In addition we have parameter unclean.leader.election which could wipe data at 
some circumstances, I encourage to create additional parameter which allows to 
do same to remote topic configs, as safe lever for migration

  was:
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.

In addition we have parameter unclean.leader.election which allows to wipe 
data, I encourage to create additional parameter which allows to do same to 
remote topic configs, as safe lever for migration


> Add additional parameter which allows parameter remote.enabled and its 
> relatives to be wiped if server tiering is disabled
> --
>
> Key: KAFKA-15990
> URL: https://issues.apache.org/jira/browse/KAFKA-15990
> Project: Kafka
>  Issue Type: Improvement
>  Components: Tiered-Storage
>Reporter: Viktor Nikitash
>Priority: Major
>
> h2. Background
> There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
> which is not working on that versions at all, and when we upgrade kafka to 
> version 3.6.0 we have crash restart loop with message "You have to delete all 
> topics with the property remote.storage.enable=true before disabling tiered 
> storage cluster-wide".
> [https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
> h2. The Idea 
> The idea is to create new parameter which allows to wipe configurations 
> related to remote storage on topic level, with default value set to false 
> (which won't change logic from the current one), but in case we manually set 
> parameter to true, those remote topic parameters will be wiped.
> Of course additional checks required. Like do it only on Kafka startup and 
> ensure that there is no data on remote storage etc...
> h2. Motivation
> Since tiering in to working on version 3.6.0-, there is no any reason to keep 
> this configs during migration from older 3rd version to 3.6.0+. When it comes 
> to managing lots of clusters it can be a problem. So enabling this new 
> parameter before 

[jira] [Updated] (KAFKA-15990) Add additional parameter which allows parameter remote.enabled and its relatives to be wiped if server tiering is disabled

2023-12-08 Thread Viktor Nikitash (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Nikitash updated KAFKA-15990:

Description: 
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.

  was:
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) what we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.


> Add additional parameter which allows parameter remote.enabled and its 
> relatives to be wiped if server tiering is disabled
> --
>
> Key: KAFKA-15990
> URL: https://issues.apache.org/jira/browse/KAFKA-15990
> Project: Kafka
>  Issue Type: Improvement
>  Components: Tiered-Storage
>Reporter: Viktor Nikitash
>Priority: Major
>
> h2. Background
> There is a miss in Kafka versions [3.1.x; 3.6.0) that we can enable tiering, 
> which is not working on that versions at all, and when we upgrade kafka to 
> version 3.6.0 we have crash restart loop with message "You have to delete all 
> topics with the property remote.storage.enable=true before disabling tiered 
> storage cluster-wide".
> [https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
> h2. The Idea 
> The idea is to create new parameter which allows to wipe configurations 
> related to remote storage on topic level, with default value set to false 
> (which won't change logic from the current one), but in case we manually set 
> parameter to true, those remote topic parameters will be wiped.
> Of course additional checks required. Like do it only on Kafka startup and 
> ensure that there is no data on remote storage etc...
> h2. Motivation
> Since tiering in to working on version 3.6.0-, there is no any reason to keep 
> this configs during migration from older 3rd version to 3.6.0+. When it comes 
> to managing lots of clusters it can be a problem. So enabling this new 
> parameter before migration to 3.6.+ and disabling it after, makes migration 
> easier.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (KAFKA-15990) Add additional parameter which allows parameter remote.enabled and its relatives to be wiped if server tiering is disabled

2023-12-08 Thread Viktor Nikitash (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Nikitash updated KAFKA-15990:

Description: 
h2. Background

There is a miss in Kafka versions [3.1.x; 3.6.0) what we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
[https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
h2. The Idea 

The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.

Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h2. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.

  was:
h2. Background


There is a miss in Kafka versions [3.1.x; 3.6.0) what we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564
h3. 
The Idea 


The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.


Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h3. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.


> Add additional parameter which allows parameter remote.enabled and its 
> relatives to be wiped if server tiering is disabled
> --
>
> Key: KAFKA-15990
> URL: https://issues.apache.org/jira/browse/KAFKA-15990
> Project: Kafka
>  Issue Type: Improvement
>  Components: Tiered-Storage
>Reporter: Viktor Nikitash
>Priority: Major
>
> h2. Background
> There is a miss in Kafka versions [3.1.x; 3.6.0) what we can enable tiering, 
> which is not working on that versions at all, and when we upgrade kafka to 
> version 3.6.0 we have crash restart loop with message "You have to delete all 
> topics with the property remote.storage.enable=true before disabling tiered 
> storage cluster-wide".
> [https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564]
> h2. The Idea 
> The idea is to create new parameter which allows to wipe configurations 
> related to remote storage on topic level, with default value set to false 
> (which won't change logic from the current one), but in case we manually set 
> parameter to true, those remote topic parameters will be wiped.
> Of course additional checks required. Like do it only on Kafka startup and 
> ensure that there is no data on remote storage etc...
> h2. Motivation
> Since tiering in to working on version 3.6.0-, there is no any reason to keep 
> this configs during migration from older 3rd version to 3.6.0+. When it comes 
> to managing lots of clusters it can be a problem. So enabling this new 
> parameter before migration to 3.6.+ and disabling it after, makes migration 
> easier.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (KAFKA-15990) Add additional parameter which allows parameter remote.enabled and its relatives to be wiped if server tiering is disabled

2023-12-08 Thread Viktor Nikitash (Jira)
Viktor Nikitash created KAFKA-15990:
---

 Summary: Add additional parameter which allows parameter 
remote.enabled and its relatives to be wiped if server tiering is disabled
 Key: KAFKA-15990
 URL: https://issues.apache.org/jira/browse/KAFKA-15990
 Project: Kafka
  Issue Type: Improvement
  Components: Tiered-Storage
Reporter: Viktor Nikitash


h2. Background


There is a miss in Kafka versions [3.1.x; 3.6.0) what we can enable tiering, 
which is not working on that versions at all, and when we upgrade kafka to 
version 3.6.0 we have crash restart loop with message "You have to delete all 
topics with the property remote.storage.enable=true before disabling tiered 
storage cluster-wide".
https://github.com/apache/kafka/blob/43c635f3a48fcb16dd34eac16def379141912e77/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java#L564
h3. 
The Idea 


The idea is to create new parameter which allows to wipe configurations related 
to remote storage on topic level, with default value set to false (which won't 
change logic from the current one), but in case we manually set parameter to 
true, those remote topic parameters will be wiped.


Of course additional checks required. Like do it only on Kafka startup and 
ensure that there is no data on remote storage etc...
h3. Motivation

Since tiering in to working on version 3.6.0-, there is no any reason to keep 
this configs during migration from older 3rd version to 3.6.0+. When it comes 
to managing lots of clusters it can be a problem. So enabling this new 
parameter before migration to 3.6.+ and disabling it after, makes migration 
easier.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-15401) Segment with corrupted index should not be uploaded to remote storage

2023-08-24 Thread Viktor Nikitash (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758646#comment-17758646
 ] 

Viktor Nikitash commented on KAFKA-15401:
-

PR: https://github.com/apache/kafka/pull/14104

> Segment with corrupted index should not be uploaded to remote storage
> -
>
> Key: KAFKA-15401
> URL: https://issues.apache.org/jira/browse/KAFKA-15401
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 3.6.0
>Reporter: Viktor Nikitash
>Priority: Minor
>  Labels: KIP-405
> Fix For: 3.6.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> During data disk full event, there could be a situation with index 
> corruption. There are existing functions which perform sanity checks to 
> TimeIndex [1], TxnIndex [2], OffsetIndex [2]. The idea is performing same 
> checks before in RemoteLogManager before we upload segment to remote storage 
> [4].
> Resources:
> [1] 
> [TimeIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L73]
> [2] 
> [TransationIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TransactionIndex.java#L187]
> [3][OffsetIndex::sanityCheck()|[https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/OffsetIndex.java#L78]]
> [4][RemoteLogManager::copyLogSegmentsToRemote()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L649]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (KAFKA-15401) Segment with corrupted index should not be uploaded to remote storage

2023-08-24 Thread Viktor Nikitash (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Nikitash updated KAFKA-15401:

Description: 
During data disk full event, there could be a situation with index corruption. 
There are existing functions which perform sanity checks to TimeIndex [1], 
TxnIndex [2], OffsetIndex [2]. The idea is performing same checks before in 
RemoteLogManager before we upload segment to remote storage [4].

Resources:
[1][TimeIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L73]
[2][TransationIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TransactionIndex.java#L187]
[3][OffsetIndex::sanityCheck()|#L78]]
[4][RemoteLogManager::copyLogSegmentsToRemote()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L649]

  was:
During data disk full event, there could be a situation with index corruption. 
There are existing functions which perform sanity checks to TimeIndex [1], 
TxnIndex [2], OffsetIndex [2]. The idea is performing same checks before in 
RemoteLogManager before we upload segment to remote storage [4].

Resources:
[1] 
[TimeIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L73]
[2] 
[TransationIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TransactionIndex.java#L187]
[3][OffsetIndex::sanityCheck()|[https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/OffsetIndex.java#L78]]
[4][RemoteLogManager::copyLogSegmentsToRemote()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L649]


> Segment with corrupted index should not be uploaded to remote storage
> -
>
> Key: KAFKA-15401
> URL: https://issues.apache.org/jira/browse/KAFKA-15401
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 3.6.0
>Reporter: Viktor Nikitash
>Priority: Minor
>  Labels: KIP-405
> Fix For: 3.6.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> During data disk full event, there could be a situation with index 
> corruption. There are existing functions which perform sanity checks to 
> TimeIndex [1], TxnIndex [2], OffsetIndex [2]. The idea is performing same 
> checks before in RemoteLogManager before we upload segment to remote storage 
> [4].
> Resources:
> [1][TimeIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L73]
> [2][TransationIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TransactionIndex.java#L187]
> [3][OffsetIndex::sanityCheck()|#L78]]
> [4][RemoteLogManager::copyLogSegmentsToRemote()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L649]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (KAFKA-15401) Segment with corrupted index should not be uploaded to remote storage

2023-08-24 Thread Viktor Nikitash (Jira)
Viktor Nikitash created KAFKA-15401:
---

 Summary: Segment with corrupted index should not be uploaded to 
remote storage
 Key: KAFKA-15401
 URL: https://issues.apache.org/jira/browse/KAFKA-15401
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.6.0
Reporter: Viktor Nikitash
 Fix For: 3.6.0


During data disk full event, there could be a situation with index corruption. 
There are existing functions which perform sanity checks to TimeIndex [1], 
TxnIndex [2], OffsetIndex [2]. The idea is performing same checks before in 
RemoteLogManager before we upload segment to remote storage [4].

Resources:
[1] 
[TimeIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L73]
[2] 
[TransationIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TransactionIndex.java#L187]
[3][OffsetIndex::sanityCheck()|[https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/OffsetIndex.java#L78]]
[4][RemoteLogManager::copyLogSegmentsToRemote()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L649]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)