[GitHub] [beam] lukecwik commented on a change in pull request #12241: [BEAM-10420] Fix minor race condition related to splitting that will cause None has no method 'yyy'
lukecwik commented on a change in pull request #12241: URL: https://github.com/apache/beam/pull/12241#discussion_r454532875 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -842,29 +847,37 @@ def _invoke_process_per_window(self, def try_split(self, fraction): # type: (...) -> Optional[Tuple[SplitResultPrimary, SplitResultResidual]] -if self.threadsafe_restriction_tracker and self.current_windowed_value: +if not self.is_splittable: + return None + +with self.splitting_lock: + # Make a local reference to member variables that change references during + # processing under lock before attempting to split so we have a consistent + # view of all the references. + current_windowed_value = self.current_windowed_value + threadsafe_restriction_tracker = self.threadsafe_restriction_tracker + threadsafe_watermark_estimator = self.threadsafe_watermark_estimator + +if threadsafe_restriction_tracker: Review comment: I have a much larger rewrite here to support per window invocation where this will become a non-issue so I'll pass on this since we will need to hold the lock for the entire split call. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12241: [BEAM-10420] Fix minor race condition related to splitting that will cause None has no method 'yyy'
lukecwik commented on a change in pull request #12241: URL: https://github.com/apache/beam/pull/12241#discussion_r454532100 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -842,29 +847,37 @@ def _invoke_process_per_window(self, def try_split(self, fraction): # type: (...) -> Optional[Tuple[SplitResultPrimary, SplitResultResidual]] -if self.threadsafe_restriction_tracker and self.current_windowed_value: +if not self.is_splittable: + return None + +with self.splitting_lock: + # Make a local reference to member variables that change references during + # processing under lock before attempting to split so we have a consistent + # view of all the references. + current_windowed_value = self.current_windowed_value Review comment: The lock ends before the `if` since the `if` indentation level is not at the same level as `with lock` statement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12241: [BEAM-10420] Fix minor race condition related to splitting that will cause None has no method 'yyy'
lukecwik commented on a change in pull request #12241: URL: https://github.com/apache/beam/pull/12241#discussion_r454518553 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -842,29 +847,37 @@ def _invoke_process_per_window(self, def try_split(self, fraction): # type: (...) -> Optional[Tuple[SplitResultPrimary, SplitResultResidual]] -if self.threadsafe_restriction_tracker and self.current_windowed_value: +if not self.is_splittable: + return None + +with self.splitting_lock: + # Make a local reference to member variables that change references during + # processing under lock before attempting to split so we have a consistent + # view of all the references. + current_windowed_value = self.current_windowed_value + threadsafe_restriction_tracker = self.threadsafe_restriction_tracker + threadsafe_watermark_estimator = self.threadsafe_watermark_estimator + +if threadsafe_restriction_tracker: Review comment: There is an assumption that if one is set then the other is set. Similarly, if one is unset then the other is unset. ## File path: sdks/python/apache_beam/runners/common.py ## @@ -842,29 +847,37 @@ def _invoke_process_per_window(self, def try_split(self, fraction): # type: (...) -> Optional[Tuple[SplitResultPrimary, SplitResultResidual]] -if self.threadsafe_restriction_tracker and self.current_windowed_value: +if not self.is_splittable: + return None + +with self.splitting_lock: + # Make a local reference to member variables that change references during + # processing under lock before attempting to split so we have a consistent + # view of all the references. + current_windowed_value = self.current_windowed_value Review comment: The purpose of the lock is to get a consistent point in time copy of the references to the current objects. We don't need a deep copy. Splitting will fail if the restriction becomes finished and the main processing loop moves onto the next element. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #12241: [BEAM-10420] Fix minor race condition related to splitting that will cause None has no method 'yyy'
lukecwik commented on a change in pull request #12241: URL: https://github.com/apache/beam/pull/12241#discussion_r454518717 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -842,29 +847,37 @@ def _invoke_process_per_window(self, def try_split(self, fraction): # type: (...) -> Optional[Tuple[SplitResultPrimary, SplitResultResidual]] -if self.threadsafe_restriction_tracker and self.current_windowed_value: +if not self.is_splittable: + return None + +with self.splitting_lock: Review comment: Not deep copying objects. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org