[jira] [Updated] (KAFKA-5240) Make persistent checkpointedOffsets optionable
[ https://issues.apache.org/jira/browse/KAFKA-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Gemela updated KAFKA-5240: Priority: Minor (was: Major) > Make persistent checkpointedOffsets optionable > -- > > Key: KAFKA-5240 > URL: https://issues.apache.org/jira/browse/KAFKA-5240 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Lukas Gemela >Priority: Minor > > By looking at the ProcessorStateManager class kafka streams library tries to > persist current offset for each partition to the file. It has to use locking > mechanism to be sure that there is no other thread/task running which is > modifying the .checkpoint file. It does that even if you don't use persistent > store in your topology (which is a bit confusing) > From my understanding this is because you want to make active state > restorations faster and not to seek from the beginning > (ProcessorStateManager:217) > We actually run everything in docker environment and we don't restart our > microservices - we just run another docker container and delete the old one. > We don't use persistent stores and we don't want to have our microservices to > write anything to the filesystem. > We always set aggressive [compact. delete] policy to get the kafka streams > internal topics to have them compacted as much as possible and therefore we > don't need a fast recovery either - we always have to replay whole topic no > matter what. > We also had some issues file file locking -see > https://issues.apache.org/jira/browse/KAFKA-5242 > Would it be possible to make writing to the file system optionable? > Thanks! > L. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5240) Make persistent checkpointedOffsets optionable
[ https://issues.apache.org/jira/browse/KAFKA-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Gemela updated KAFKA-5240: Description: By looking at the ProcessorStateManager class kafka streams library tries to persist current offset for each partition to the file. It has to use locking mechanism to be sure that there is no other thread/task running which is modifying the .checkpoint file. It does that even if you don't use persistent store in your topology (which is a bit confusing) >From my understanding this is because you want to make active state >restorations faster and not to seek from the beginning >(ProcessorStateManager:217) We actually run everything in docker environment and we don't restart our microservices - we just run another docker container and delete the old one. We don't use persistent stores and we don't want to have our microservices to write anything to the filesystem. We always set aggressive [compact. delete] policy to get the kafka streams internal topics to have them compacted as much as possible and therefore we don't need a fast recovery either - we always have to replay whole topic no matter what. Would it be possible to make writing to the file system optionable? Thanks! L. was: By looking at the ProcessorStateManager class kafka streams library tries to persist current offset for each partition to the file. It has to use locking mechanism to be sure that there is no other thread/task running which is modifying the .checkpoint file. It does that even if you don't use persistent store in your topology (which is a bit confusing) >From my understanding this is because you want to make active state >restorations faster and not to seek from the beginning >(ProcessorStateManager:217) We actually run everything in docker environment and we don't restart our microservices - we just run another docker container and delete the old one. We don't use persistent stores and we don't want to have our microservices to write anything to the filesystem. We always set aggressive [compact. delete] policy to get the kafka streams internal topics to have them compacted as much as possible and therefore we don't need a fast recovery either - we always have to replay whole topic no matter what. We also had some issues file file locking -see https://issues.apache.org/jira/browse/KAFKA-5242 Would it be possible to make writing to the file system optionable? Thanks! L. > Make persistent checkpointedOffsets optionable > -- > > Key: KAFKA-5240 > URL: https://issues.apache.org/jira/browse/KAFKA-5240 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Lukas Gemela >Priority: Minor > > By looking at the ProcessorStateManager class kafka streams library tries to > persist current offset for each partition to the file. It has to use locking > mechanism to be sure that there is no other thread/task running which is > modifying the .checkpoint file. It does that even if you don't use persistent > store in your topology (which is a bit confusing) > From my understanding this is because you want to make active state > restorations faster and not to seek from the beginning > (ProcessorStateManager:217) > We actually run everything in docker environment and we don't restart our > microservices - we just run another docker container and delete the old one. > We don't use persistent stores and we don't want to have our microservices to > write anything to the filesystem. > We always set aggressive [compact. delete] policy to get the kafka streams > internal topics to have them compacted as much as possible and therefore we > don't need a fast recovery either - we always have to replay whole topic no > matter what. > Would it be possible to make writing to the file system optionable? > Thanks! > L. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5240) Make persistent checkpointedOffsets optionable
[ https://issues.apache.org/jira/browse/KAFKA-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Gemela updated KAFKA-5240: Description: By looking at the ProcessorStateManager class kafka streams library tries to persist current offset for each partition to the file. It has to use locking mechanism to be sure that there is no other thread/task running which is modifying the .checkpoint file. It does that even if you don't use persistent store in your topology (which is a bit confusing) >From my understanding this is because you want to make active state >restorations faster and not to seek from the beginning >(ProcessorStateManager:217) We actually run everything in docker environment and we don't restart our microservices - we just run another docker container and delete the old one. We don't use persistent stores and we don't want to have our microservices to write anything to the filesystem. We always set aggressive [compact. delete] policy to get the kafka streams internal topics to have them compacted as much as possible and therefore we don't need a fast recovery either - we always have to replay whole topic no matter what. We also had some issues file file locking -see https://issues.apache.org/jira/browse/KAFKA-5242 Would it be possible to make writing to the file system optionable? Thanks! L. was: By looking at the ProcessorStateManager class kafka streams library tries to persist current offset for each partition to the file. It has to use locking mechanism to be sure that there is no other thread/task running which is modifying the .checkpoint file. It does that even if you don't use persistent store in your topology (which is a bit confusing) >From my understanding this is because you want to make active state >restorations faster and not to seek from the beginning >(ProcessorStateManager:217) We actually run everything in docker environment and we don't restart our microservices - we just run another docker container and delete the old one. We don't use persistent stores and we don't want to have our microservices to write anything to the filesystem. We always set aggressive [compact. delete] policy to get the kafka streams internal topics to have them compacted as much as possible and therefore we don't need a fast recovery either - we always have to replay whole topic no matter what. Would it be possible to make writing to the file system optionable? Thanks! L. > Make persistent checkpointedOffsets optionable > -- > > Key: KAFKA-5240 > URL: https://issues.apache.org/jira/browse/KAFKA-5240 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Lukas Gemela > > By looking at the ProcessorStateManager class kafka streams library tries to > persist current offset for each partition to the file. It has to use locking > mechanism to be sure that there is no other thread/task running which is > modifying the .checkpoint file. It does that even if you don't use persistent > store in your topology (which is a bit confusing) > From my understanding this is because you want to make active state > restorations faster and not to seek from the beginning > (ProcessorStateManager:217) > We actually run everything in docker environment and we don't restart our > microservices - we just run another docker container and delete the old one. > We don't use persistent stores and we don't want to have our microservices to > write anything to the filesystem. > We always set aggressive [compact. delete] policy to get the kafka streams > internal topics to have them compacted as much as possible and therefore we > don't need a fast recovery either - we always have to replay whole topic no > matter what. > We also had some issues file file locking -see > https://issues.apache.org/jira/browse/KAFKA-5242 > Would it be possible to make writing to the file system optionable? > Thanks! > L. -- This message was sent by Atlassian JIRA (v6.3.15#6346)