[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=369543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-369543 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 10/Jan/20 02:00 Start Date: 10/Jan/20 02:00 Worklog Time Spent: 10m Work Description: udim commented on pull request #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 369543) Time Spent: 3.5h (was: 3h 20m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=367694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367694 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 07/Jan/20 19:34 Start Date: 07/Jan/20 19:34 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302#issuecomment-571737253 @udim should we merge this to the 2.18 branch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 367694) Time Spent: 3h 20m (was: 3h 10m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=365518&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-365518 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 02/Jan/20 22:25 Start Date: 02/Jan/20 22:25 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302#issuecomment-570377692 LGTM. Not merging to give release manager a chance to review and merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 365518) Time Spent: 3h 10m (was: 3h) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=365516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-365516 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 02/Jan/20 22:19 Start Date: 02/Jan/20 22:19 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302#issuecomment-570375446 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 365516) Time Spent: 3h (was: 2h 50m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=365515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-365515 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 02/Jan/20 22:17 Start Date: 02/Jan/20 22:17 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302#issuecomment-570374721 R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 365515) Time Spent: 2h 50m (was: 2h 40m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=355086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355086 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 06/Dec/19 10:40 Start Date: 06/Dec/19 10:40 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10301: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10301#issuecomment-562523439 It worked and Dataflow VR are blue again. Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355086) Time Spent: 2h 40m (was: 2.5h) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354779&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354779 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 23:27 Start Date: 05/Dec/19 23:27 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302#issuecomment-562364234 R: @ehudm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354779) Time Spent: 2.5h (was: 2h 20m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354778&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354778 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 23:27 Start Date: 05/Dec/19 23:27 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10302: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10302 This is a forward fix for https://github.com/apache/beam/pull/10258 to fix Dataflow VR postcommit suite. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_Po
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354768 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 23:17 Start Date: 05/Dec/19 23:17 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10301: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10301 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354768) Time Spent: 2h 10m (was: 2h) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[
https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354726&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354726
]
ASF GitHub Bot logged work on BEAM-4287:
Author: ASF GitHub Bot
Created on: 05/Dec/19 21:50
Start Date: 05/Dec/19 21:50
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10301: [BEAM-4287]
Fix to use the residual instead of the current restriction on process
continuations.
URL: https://github.com/apache/beam/pull/10301#discussion_r354565411
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
##
@@ -147,7 +147,7 @@ public void process(ProcessContext c, BoundedWindow w) {
ProcessContinuation continuation =
invoker.invokeProcessElement(new NestedProcessContext<>(fn, c,
element, w, tracker));
if (continuation.shouldResume()) {
- restriction = tracker.currentRestriction();
+ restriction = tracker.trySplit(0).getResidual();
Review comment:
The previous code here was tracker.checkpoint()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354726)
Time Spent: 2h (was: 1h 50m)
> SplittableDoFn: splitAtFraction() API for Java
> --
>
> Key: BEAM-4287
> URL: https://issues.apache.org/jira/browse/BEAM-4287
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> SDF currently only has a checkpoint() API. This Jira is about adding the
> splitAtFraction() API and its support in runners that support the respective
> feature for sources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354722&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354722 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 21:47 Start Date: 05/Dec/19 21:47 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10301: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10301#issuecomment-562333242 R: @lgajowy @boyuanzz I ran the same validates runner suite internally against this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354722) Time Spent: 1h 50m (was: 1h 40m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354713&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354713 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 21:46 Start Date: 05/Dec/19 21:46 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10301: [BEAM-4287] Fix to use the residual instead of the current restriction on process continuations. URL: https://github.com/apache/beam/pull/10301 This is a forward fix for https://github.com/apache/beam/pull/10258 to fix Dataflow VR postcommit suite. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCo
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354367&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354367 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 15:24 Start Date: 05/Dec/19 15:24 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10295: Revert "[BEAM-4287] Add trySplit API to Java restriction tracker matc… URL: https://github.com/apache/beam/pull/10295#issuecomment-562177107 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354367) Time Spent: 1.5h (was: 1h 20m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=354366&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354366 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 05/Dec/19 15:23 Start Date: 05/Dec/19 15:23 Worklog Time Spent: 10m Work Description: lgajowy commented on pull request #10295: Revert "[BEAM-4287] Add trySplit API to Java restriction tracker matc… URL: https://github.com/apache/beam/pull/10295 …hing Python SDK definition. Remove GetPartition()" This reverts commit 1ed36b430527e1c93c5763d23229379b1eb9f582, reversing changes made to 5143d408ebbbd044a024cfd84b87f70dd42b3ead. **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build Statu
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=353762&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353762 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 04/Dec/19 20:45 Start Date: 04/Dec/19 20:45 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10258: [BEAM-4287] Add trySplit API to Java restriction tracker matching Python SDK definition. Remove GetPartition() URL: https://github.com/apache/beam/pull/10258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353762) Time Spent: 1h 10m (was: 1h) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=353087&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353087 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 03/Dec/19 23:22 Start Date: 03/Dec/19 23:22 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10258: [BEAM-4287] Add trySplit API to Java restriction tracker matching Python SDK definition. Remove GetPartition() URL: https://github.com/apache/beam/pull/10258#issuecomment-561402185 The removal of partition is because it is unused and out of date wrt to Beam Python SDK but https://s.apache.org/beam-bundles-backlog-splitting and https://s.apache.org/beam-checkpoint-and-split-bundles provide the background. The fundamental idea is how does one aggregate the size being reported since the size can be per element and restriction (file range in a specific file) or shared across multiple elements and restrictions (pubsub reading multiple topics). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353087) Time Spent: 1h (was: 50m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=353086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353086 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 03/Dec/19 23:21 Start Date: 03/Dec/19 23:21 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10258: [BEAM-4287] Add trySplit API to Java restriction tracker matching Python SDK definition. Remove GetPartition() URL: https://github.com/apache/beam/pull/10258#issuecomment-561402185 The combination of https://s.apache.org/beam-bundles-backlog-splitting and https://s.apache.org/beam-checkpoint-and-split-bundles provide the background. They are out of date with respect to what was done in the Python SDK but the fundamental idea is how does one aggregate the size being reported since the size can be per element and restriction (file range in a specific file) or shared across multiple elements and restrictions (pubsub reading multiple topics). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353086) Time Spent: 50m (was: 40m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[
https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=352922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352922
]
ASF GitHub Bot logged work on BEAM-4287:
Author: ASF GitHub Bot
Created on: 03/Dec/19 19:35
Start Date: 03/Dec/19 19:35
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #10258: [BEAM-4287] Add
trySplit API to Java restriction tracker matching Python SDK definition. Remove
GetPartition()
URL: https://github.com/apache/beam/pull/10258#issuecomment-561322628
> Looks pretty good, just need to get it passing the Precommit. Looks like
you're missing some changes in RestrictionTrackersTest:
>
> ```
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:40:
error: is not
abstract and does not override abstract method trySplit(double) in
RestrictionTracker
> 10:25:53 new RestrictionTracker() {
> 10:25:53 ^
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:52:
error: method does not override or implement a method from a supertype
> 10:25:53 @Override
> 10:25:53 ^
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:87:
error: RestrictionTrackerWithSize is not abstract and does not override
abstract method trySplit(double) in RestrictionTracker
> 10:25:53 private static class RestrictionTrackerWithSize extends
RestrictionTracker
> 10:25:53 ^
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:105:
error: method does not override or implement a method from a supertype
> 10:25:53 @Override
> 10:25:53 ^
> ```
>
> Out of curiosity, what is the Partition stuff and why is it being removed?
Just an outdated part of the SDF spec?
It was related to how we report progress/backlog but we can now report it
via the sized elements that are being returned as part of the split.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 352922)
Time Spent: 0.5h (was: 20m)
> SplittableDoFn: splitAtFraction() API for Java
> --
>
> Key: BEAM-4287
> URL: https://issues.apache.org/jira/browse/BEAM-4287
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> SDF currently only has a checkpoint() API. This Jira is about adding the
> splitAtFraction() API and its support in runners that support the respective
> feature for sources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[
https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=352923&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352923
]
ASF GitHub Bot logged work on BEAM-4287:
Author: ASF GitHub Bot
Created on: 03/Dec/19 19:35
Start Date: 03/Dec/19 19:35
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #10258: [BEAM-4287] Add
trySplit API to Java restriction tracker matching Python SDK definition. Remove
GetPartition()
URL: https://github.com/apache/beam/pull/10258#issuecomment-561322628
> Looks pretty good, just need to get it passing the Precommit. Looks like
you're missing some changes in RestrictionTrackersTest:
>
> ```
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:40:
error: is not
abstract and does not override abstract method trySplit(double) in
RestrictionTracker
> 10:25:53 new RestrictionTracker() {
> 10:25:53 ^
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:52:
error: method does not override or implement a method from a supertype
> 10:25:53 @Override
> 10:25:53 ^
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:87:
error: RestrictionTrackerWithSize is not abstract and does not override
abstract method trySplit(double) in RestrictionTracker
> 10:25:53 private static class RestrictionTrackerWithSize extends
RestrictionTracker
> 10:25:53 ^
> 10:25:53
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java:105:
error: method does not override or implement a method from a supertype
> 10:25:53 @Override
> 10:25:53 ^
> ```
>
> Out of curiosity, what is the Partition stuff and why is it being removed?
Just an outdated part of the SDF spec?
It was related to how we report progress/backlog (specifically in streaming)
but we can now report it via the sized elements that are being returned as part
of the split.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 352923)
Time Spent: 40m (was: 0.5h)
> SplittableDoFn: splitAtFraction() API for Java
> --
>
> Key: BEAM-4287
> URL: https://issues.apache.org/jira/browse/BEAM-4287
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> SDF currently only has a checkpoint() API. This Jira is about adding the
> splitAtFraction() API and its support in runners that support the respective
> feature for sources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=352306&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352306 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 02/Dec/19 23:40 Start Date: 02/Dec/19 23:40 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10258: [BEAM-4287] Add trySplit API to Java restriction tracker matching Python SDK definition. Remove GetPartition() URL: https://github.com/apache/beam/pull/10258#issuecomment-560916860 R: @youngoli CC: @boyuanzz @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352306) Time Spent: 20m (was: 10m) > SplittableDoFn: splitAtFraction() API for Java > -- > > Key: BEAM-4287 > URL: https://issues.apache.org/jira/browse/BEAM-4287 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Eugene Kirpichov >Assignee: Eugene Kirpichov >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > SDF currently only has a checkpoint() API. This Jira is about adding the > splitAtFraction() API and its support in runners that support the respective > feature for sources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java
[ https://issues.apache.org/jira/browse/BEAM-4287?focusedWorklogId=352298&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352298 ] ASF GitHub Bot logged work on BEAM-4287: Author: ASF GitHub Bot Created on: 02/Dec/19 23:29 Start Date: 02/Dec/19 23:29 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10258: [BEAM-4287] Add trySplit API to Java restriction tracker matching Python SDK definition. URL: https://github.com/apache/beam/pull/10258 Modify all existing checkpoint() implementations to ignore the fraction and file TODOs. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompl
