[jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013101#comment-16013101
 ] 

ASF GitHub Bot commented on KAFKA-5241:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3054


> GlobalKTable does not checkpoint offsets after restoring state
> --
>
> Key: KAFKA-5241
> URL: https://issues.apache.org/jira/browse/KAFKA-5241
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Tommy Becker
>Assignee: Tommy Becker
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> I'm experimenting with an application that uses a relatively large 
> GlobalKTable, and noticed that streams was not checkpointing its offsets on 
> close(). This is because although  
> {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}}
>  updates the checkpoint map, the actual checkpointing itself is guarded by a 
> check that the offsets passed from the {{GloablStateUpdateTask}} are not 
> empty. This is frustrating because if the topic backing the global table is 
> both large (therefore taking a long time to restore) and infrequently 
> written, then streams rebuilds the table from scratch every time the 
> application is started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012722#comment-16012722
 ] 

Guozhang Wang commented on KAFKA-5241:
--

[~twbecker] Added you to the contributor list and assigned this ticket to you.

> GlobalKTable does not checkpoint offsets after restoring state
> --
>
> Key: KAFKA-5241
> URL: https://issues.apache.org/jira/browse/KAFKA-5241
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Tommy Becker
>Assignee: Tommy Becker
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> I'm experimenting with an application that uses a relatively large 
> GlobalKTable, and noticed that streams was not checkpointing its offsets on 
> close(). This is because although  
> {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}}
>  updates the checkpoint map, the actual checkpointing itself is guarded by a 
> check that the offsets passed from the {{GloablStateUpdateTask}} are not 
> empty. This is frustrating because if the topic backing the global table is 
> both large (therefore taking a long time to restore) and infrequently 
> written, then streams rebuilds the table from scratch every time the 
> application is started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state

[Damian Guy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012270#comment-16012270
 ] 

Damian Guy commented on KAFKA-5241:
---

[~twbecker] Probably worth filing another JIRA for this as it is not just 
global stores that have this issue

> GlobalKTable does not checkpoint offsets after restoring state
> --
>
> Key: KAFKA-5241
> URL: https://issues.apache.org/jira/browse/KAFKA-5241
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Tommy Becker
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> I'm experimenting with an application that uses a relatively large 
> GlobalKTable, and noticed that streams was not checkpointing its offsets on 
> close(). This is because although  
> {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}}
>  updates the checkpoint map, the actual checkpointing itself is guarded by a 
> check that the offsets passed from the {{GloablStateUpdateTask}} are not 
> empty. This is frustrating because if the topic backing the global table is 
> both large (therefore taking a long time to restore) and infrequently 
> written, then streams rebuilds the table from scratch every time the 
> application is started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state

[Tommy Becker (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011360#comment-16011360
 ] 

Tommy Becker commented on KAFKA-5241:
-

I should also add that it seems broken that in the absence of checkpointed 
offsets in the store, the existing DB is simply opened and the contents of the 
topic written to it again. I would think that without a checkpoint the 
directory should be cleared and a new DB created since the contents of what is 
there are unknown. Should file a separate JIRA for this?

> GlobalKTable does not checkpoint offsets after restoring state
> --
>
> Key: KAFKA-5241
> URL: https://issues.apache.org/jira/browse/KAFKA-5241
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Tommy Becker
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> I'm experimenting with an application that uses a relatively large 
> GlobalKTable, and noticed that streams was not checkpointing its offsets on 
> close(). This is because although  
> {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}}
>  updates the checkpoint map, the actual checkpointing itself is guarded by a 
> check that the offsets passed from the {{GloablStateUpdateTask}} are not 
> empty. This is frustrating because if the topic backing the global table is 
> both large (therefore taking a long time to restore) and infrequently 
> written, then streams rebuilds the table from scratch every time the 
> application is started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010417#comment-16010417
 ] 

ASF GitHub Bot commented on KAFKA-5241:
---

GitHub user twbecker opened a pull request:

https://github.com/apache/kafka/pull/3054

KAFKA-5241: GlobalKTable does not checkpoint offsets after restoring state

Ensure checkpointable offsets for GlobalKTables are always written on close.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twbecker/kafka KAFKA-5241

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3054.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3054


commit 4f7263768ef73d18cf6b90894f4a0286bc79ea14
Author: Tommy Becker 
Date:   2017-05-15T12:13:07Z

Fix KAFKA-5241.

Ensure checkpointable offsets for GlobalKTables are always written on close.




> GlobalKTable does not checkpoint offsets after restoring state
> --
>
> Key: KAFKA-5241
> URL: https://issues.apache.org/jira/browse/KAFKA-5241
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Tommy Becker
>Priority: Minor
>
> I'm experimenting with an application that uses a relatively large 
> GlobalKTable, and noticed that streams was not checkpointing its offsets on 
> close(). This is because although  
> {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}}
>  updates the checkpoint map, the actual checkpointing itself is guarded by a 
> check that the offsets passed from the {{GloablStateUpdateTask}} are not 
> empty. This is frustrating because if the topic backing the global table is 
> both large (therefore taking a long time to restore) and infrequently 
> written, then streams rebuilds the table from scratch every time the 
> application is started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state

[Tommy Becker (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010389#comment-16010389
 ] 

Tommy Becker commented on KAFKA-5241:
-

I have a patch for this and will take the issue, if someone can grant me 
permission to assign to myself.

> GlobalKTable does not checkpoint offsets after restoring state
> --
>
> Key: KAFKA-5241
> URL: https://issues.apache.org/jira/browse/KAFKA-5241
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Tommy Becker
>Priority: Minor
>
> I'm experimenting with an application that uses a relatively large 
> GlobalKTable, and noticed that streams was not checkpointing its offsets on 
> close(). This is because although  
> {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}}
>  updates the checkpoint map, the actual checkpointing itself is guarded by a 
> check that the offsets passed from the {{GloablStateUpdateTask}} are not 
> empty. This is frustrating because if the topic backing the global table is 
> both large (therefore taking a long time to restore) and infrequently 
> written, then streams rebuilds the table from scratch every time the 
> application is started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)