[
https://issues.apache.org/jira/browse/BEAM-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443244#comment-16443244
]
Ismaël Mejía edited comment on BEAM-4016 at 4/18/18 9:56 PM:
-------------------------------------------------------------
I printed for SDFWithLifecycle in SplittableDoFnTest every call to the id of
the thread and the call to the method. Take a look at this commit
https://github.com/iemejia/beam/commit/75d8279858f2f4f0cd35af7a117ddb7ae7eb1e3a
This produces this output:
[code]
18 GetInitialRestriction c
19 GetInitialRestriction b
15 GetInitialRestriction a
13 Setup
14 Setup
18 Setup
18 SplitRestriction a - [0, 1)
14 SplitRestriction c - [0, 1)
13 SplitRestriction b - [0, 1)
18 Setup
17 Setup
13 Setup
17 ProcessElement b - OffsetRangeTracker{range=[0, 1), lastClaimedOffset=null,
lastAttemptedOffset=null}
18 ProcessElement c - OffsetRangeTracker{range=[0, 1), lastClaimedOffset=null,
lastAttemptedOffset=null}
13 ProcessElement a - OffsetRangeTracker{range=[0, 1), lastClaimedOffset=null,
lastAttemptedOffset=null}
17 Teardown
17 Teardown
17 Teardown
[code]
was (Author: iemejia):
I printed for SDFWithLifecycle in SplittableDoFnTest every call to the id of
the thread and the call to the method. Take a look at this commit
https://github.com/iemejia/beam/commit/75d8279858f2f4f0cd35af7a117ddb7ae7eb1e3a
This produces this output:
[code]
18 GetInitialRestriction c
19 GetInitialRestriction b
15 GetInitialRestriction a
13 Setup
14 Setup
18 Setup
18 SplitRestriction a - [0, 1)
14 SplitRestriction c - [0, 1)
13 SplitRestriction b - [0, 1)
18 Setup
17 Setup
13 Setup
17 ProcessElement b - OffsetRangeTracker{range=[0, 1), lastClaimedOffset=null,
lastAttemptedOffset=null}
18 ProcessElement c - OffsetRangeTracker{range=[0, 1), lastClaimedOffset=null,
lastAttemptedOffset=null}
13 ProcessElement a - OffsetRangeTracker{range=[0, 1), lastClaimedOffset=null,
lastAttemptedOffset=null}
17 Teardown
17 Teardown
17 Teardown
[code]
> Direct runner incorrect lifecycle, @SplitRestriction should execute after
> @Setup on SplittableDoFn
> --------------------------------------------------------------------------------------------------
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.4.0
> Reporter: Ismaël Mejía
> Assignee: Thomas Groh
> Priority: Major
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>
> The method annotated with @SplitRestriction is the method where we can define
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to
> execute this after the @Setup method given that usually connections are
> established at Setup and can be used to ask the different data stores about
> the partitioning strategy. I added a test for this in the
> SplittableDoFnTest.SDFWithLifecycle test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
