[jira] [Resolved] (BEAM-7719) Ensure that publishing vendored artifacts first validates there contents
[ https://issues.apache.org/jira/browse/BEAM-7719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik resolved BEAM-7719. - Fix Version/s: Not applicable Resolution: Fixed > Ensure that publishing vendored artifacts first validates there contents > > > Key: BEAM-7719 > URL: https://issues.apache.org/jira/browse/BEAM-7719 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During the release of vendored guava 26.0, it was discovered that we don't > check the contents of the jars automatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9262) Update ApiServiceDescriptor to have open ended authentication method
[ https://issues.apache.org/jira/browse/BEAM-9262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik resolved BEAM-9262. - Fix Version/s: 2.20.0 Resolution: Fixed > Update ApiServiceDescriptor to have open ended authentication method > > > Key: BEAM-9262 > URL: https://issues.apache.org/jira/browse/BEAM-9262 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently the ApiServiceDescriptor uses a oneof field for authentication > which requires users to add additional types explicitly to the list instead > of using urn + payload mechanism allowing for anyone to extend this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik resolved BEAM-9344. - Fix Version/s: 2.20.0 Resolution: Fixed > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=391051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391051 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 22/Feb/20 04:57 Start Date: 22/Feb/20 04:57 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391051) Time Spent: 1h 20m (was: 1h 10m) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=391042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391042 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 22/Feb/20 03:47 Start Date: 22/Feb/20 03:47 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#issuecomment-589914414 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391042) Time Spent: 1h 10m (was: 1h) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9308) Optimize state cleanup at end-of-window
[ https://issues.apache.org/jira/browse/BEAM-9308?focusedWorklogId=391041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391041 ] ASF GitHub Bot logged work on BEAM-9308: Author: ASF GitHub Bot Created on: 22/Feb/20 03:36 Start Date: 22/Feb/20 03:36 Worklog Time Spent: 10m Work Description: steveniemitz commented on issue #10852: [BEAM-9308] Decorrelate state cleanup timers URL: https://github.com/apache/beam/pull/10852#issuecomment-589913494 > Why is this problem specific to the GC timer? How about the normal end-of-window timer that is used to fire windowed aggregations. For fixed windows there is one per key and those also fire all at the same time. heh, we already work around that on our own by using state + timers instead of the built-in combine transform. We already decorrelate our end-of-window triggering (and we're now using the watermark hold feature for timers which simplified things a lot), but can't work around the state GC w/o changing the worker itself. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391041) Time Spent: 1h 20m (was: 1h 10m) > Optimize state cleanup at end-of-window > --- > > Key: BEAM-9308 > URL: https://issues.apache.org/jira/browse/BEAM-9308 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Assignee: Steve Niemitz >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > When using state with a large keyspace, you can end up with a large amount of > state cleanup timers set to fire all 1ms after the end of a window. This can > cause a momentary (I've observed 1-3 minute) lag in processing while windmill > and the java harness fire and process these cleanup timers. > By spreading the firing over a short period after the end of the window, we > can decorrelate the firing of the timers and smooth the load out, resulting > in much less impact from state cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9308) Optimize state cleanup at end-of-window
[ https://issues.apache.org/jira/browse/BEAM-9308?focusedWorklogId=391040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391040 ] ASF GitHub Bot logged work on BEAM-9308: Author: ASF GitHub Bot Created on: 22/Feb/20 03:35 Start Date: 22/Feb/20 03:35 Worklog Time Spent: 10m Work Description: steveniemitz commented on issue #10852: [BEAM-9308] Decorrelate state cleanup timers URL: https://github.com/apache/beam/pull/10852#issuecomment-589913494 > Why is this problem specific to the GC timer? How about the normal end-of-window timer that is used to fire windowed aggregations. For fixed windows there is one per key and those also fire all at the same time. heh, we already work around that on our own by using state + timers instead of the built-in combine transform. We already decorrelate our end-of-window triggering (and we're now using the watermark hold feature for timers which simplified things a lot). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391040) Time Spent: 1h 10m (was: 1h) > Optimize state cleanup at end-of-window > --- > > Key: BEAM-9308 > URL: https://issues.apache.org/jira/browse/BEAM-9308 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Assignee: Steve Niemitz >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > When using state with a large keyspace, you can end up with a large amount of > state cleanup timers set to fire all 1ms after the end of a window. This can > cause a momentary (I've observed 1-3 minute) lag in processing while windmill > and the java harness fire and process these cleanup timers. > By spreading the firing over a short period after the end of the window, we > can decorrelate the firing of the timers and smooth the load out, resulting > in much less impact from state cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9308) Optimize state cleanup at end-of-window
[ https://issues.apache.org/jira/browse/BEAM-9308?focusedWorklogId=391039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391039 ] ASF GitHub Bot logged work on BEAM-9308: Author: ASF GitHub Bot Created on: 22/Feb/20 03:32 Start Date: 22/Feb/20 03:32 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #10852: [BEAM-9308] Decorrelate state cleanup timers URL: https://github.com/apache/beam/pull/10852#issuecomment-589913248 Why is this problem specific to the GC timer? How about the normal end-of-window timer that is used to fire windowed aggregations. For fixed windows there is one per key and those also fire all at the same time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391039) Time Spent: 1h (was: 50m) > Optimize state cleanup at end-of-window > --- > > Key: BEAM-9308 > URL: https://issues.apache.org/jira/browse/BEAM-9308 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Assignee: Steve Niemitz >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > When using state with a large keyspace, you can end up with a large amount of > state cleanup timers set to fire all 1ms after the end of a window. This can > cause a momentary (I've observed 1-3 minute) lag in processing while windmill > and the java harness fire and process these cleanup timers. > By spreading the firing over a short period after the end of the window, we > can decorrelate the firing of the timers and smooth the load out, resulting > in much less impact from state cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9308) Optimize state cleanup at end-of-window
[ https://issues.apache.org/jira/browse/BEAM-9308?focusedWorklogId=391029=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391029 ] ASF GitHub Bot logged work on BEAM-9308: Author: ASF GitHub Bot Created on: 22/Feb/20 02:24 Start Date: 22/Feb/20 02:24 Worklog Time Spent: 10m Work Description: steveniemitz commented on issue #10852: [BEAM-9308] Decorrelate state cleanup timers URL: https://github.com/apache/beam/pull/10852#issuecomment-589907688 Yay thanks for looking at this. I'll address your points in reverse order :P > Maybe we need a better prioritization strategy so that large #s of timers don't starve out elements? I think that'd be the best overall option, but ideally we'd have variable priority. ie, state cleanup timers should be low priority, while user timers should be the same priority as "normal" elements. In the end though, if we end up with state cleanup timers delayed by N minutes because they are deprioritized, that seems like we'd be in the same spot as explicitly decorrelating them here. > Delaying the timer will also prevent downstream aggregations from firing. 3 minutes could cause issues if the window itself is much smaller. Agreed, I sort of touched on this on my comment about letting the duration be configurable. Ideally it'd be some fraction of the window duration itself. I'm not sure it actually will delay the downstream aggregations from firing however, since the firing time it set to after the window closes (maxTimestamp + allowedLateness + 1ms), so once these begin firing, the watermark has already passed the end of the window. Or am I misunderstanding something here? > We want to reuse this timer for OnWindowExpiration, and this will delay all those callbacks as well. I'd actually argue that's preferable, since you'd have the same problem there was well (potentially millions of timers firing at the same time). > We currently rely on the state cleanup timer for watermark holds. Is this true? The state cleanup timer is already set past the end of the window, so by the time the timer fires the window has already closed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391029) Time Spent: 50m (was: 40m) > Optimize state cleanup at end-of-window > --- > > Key: BEAM-9308 > URL: https://issues.apache.org/jira/browse/BEAM-9308 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Assignee: Steve Niemitz >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > When using state with a large keyspace, you can end up with a large amount of > state cleanup timers set to fire all 1ms after the end of a window. This can > cause a momentary (I've observed 1-3 minute) lag in processing while windmill > and the java harness fire and process these cleanup timers. > By spreading the firing over a short period after the end of the window, we > can decorrelate the firing of the timers and smooth the load out, resulting > in much less impact from state cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=391028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391028 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 22/Feb/20 02:20 Start Date: 22/Feb/20 02:20 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919#discussion_r382877617 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -916,6 +919,9 @@ def _use_unified_worker(pipeline_options): debug_options = pipeline_options.view_as(DebugOptions) use_unified_worker_flag = 'use_unified_worker' + if debug_options.lookup_experiment(use_unified_worker_flag): Review comment: Ok, I think I understand the intent but I think it is confusing - can we move lines 925-926 to dataflow_runner.py, see: https://github.com/apache/beam/blob/1117508136ac5d0d66141c0e8f1c0830fb223a3a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L480, and replace 920-928 with line 928? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391028) Time Spent: 1.5h (was: 1h 20m) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=391026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391026 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 22/Feb/20 02:03 Start Date: 22/Feb/20 02:03 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919#discussion_r382876346 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -916,6 +919,9 @@ def _use_unified_worker(pipeline_options): debug_options = pipeline_options.view_as(DebugOptions) use_unified_worker_flag = 'use_unified_worker' + if debug_options.lookup_experiment(use_unified_worker_flag): Review comment: nvm, reading this again, looks like this logic is intenional This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391026) Time Spent: 1h 20m (was: 1h 10m) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=391025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391025 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 22/Feb/20 02:01 Start Date: 22/Feb/20 02:01 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919#discussion_r382876179 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -916,6 +919,9 @@ def _use_unified_worker(pipeline_options): debug_options = pipeline_options.view_as(DebugOptions) use_unified_worker_flag = 'use_unified_worker' + if debug_options.lookup_experiment(use_unified_worker_flag): Review comment: Do we need 922-924? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391025) Time Spent: 1h 10m (was: 1h) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[
https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=391024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391024
]
ASF GitHub Bot logged work on BEAM-9347:
Author: ASF GitHub Bot
Created on: 22/Feb/20 02:00
Start Date: 22/Feb/20 02:00
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #10919: [BEAM-9347]
Don't overwrite default runner harness for unified worker
URL: https://github.com/apache/beam/pull/10919#discussion_r382876017
##
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##
@@ -186,8 +186,11 @@ def __init__(self, packages, options,
environment_version, pipeline_url):
if job_type.startswith('FNAPI_'):
self.debug_options = self.debug_options or DebugOptions()
self.debug_options.experiments = self.debug_options.experiments or []
+ # Don't add the default image overwrite if user overwrites or
Review comment:
Also, should we be passing pipeline_options instead of debug options?
Perhaps it works either way but passing pipeline options would be cleaner since
that's what the signature of _use_unified_worker() expects.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391024)
Time Spent: 1h (was: 50m)
> Remove default image for Unified Worker
> ---
>
> Key: BEAM-9347
> URL: https://issues.apache.org/jira/browse/BEAM-9347
> Project: Beam
> Issue Type: Test
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The runner will choose the Runner Harness image for UW so we don't need to
> overwrite the image in default behavior.
> Also, this will help us distinguish between user requested overwrites for the
> default overwrites(which is not used).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[
https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=391022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391022
]
ASF GitHub Bot logged work on BEAM-9347:
Author: ASF GitHub Bot
Created on: 22/Feb/20 01:58
Start Date: 22/Feb/20 01:58
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #10919: [BEAM-9347]
Don't overwrite default runner harness for unified worker
URL: https://github.com/apache/beam/pull/10919#discussion_r382875832
##
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##
@@ -186,8 +186,11 @@ def __init__(self, packages, options,
environment_version, pipeline_url):
if job_type.startswith('FNAPI_'):
self.debug_options = self.debug_options or DebugOptions()
self.debug_options.experiments = self.debug_options.experiments or []
+ # Don't add the default image overwrite if user overwrites or
Review comment:
I think it will be easier to read if we restructure this:
```
if (self.debug_options.lookup_experiment('runner_harness_container_image')
or
_use_unified_worker(self.debug_options)):
# Comment on WHY we don't want to set the override
pass
else:
<...set the override...>
```
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391022)
Time Spent: 50m (was: 40m)
> Remove default image for Unified Worker
> ---
>
> Key: BEAM-9347
> URL: https://issues.apache.org/jira/browse/BEAM-9347
> Project: Beam
> Issue Type: Test
> Components: runner-dataflow
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The runner will choose the Runner Harness image for UW so we don't need to
> overwrite the image in default behavior.
> Also, this will help us distinguish between user requested overwrites for the
> default overwrites(which is not used).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9308) Optimize state cleanup at end-of-window
[ https://issues.apache.org/jira/browse/BEAM-9308?focusedWorklogId=391021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391021 ] ASF GitHub Bot logged work on BEAM-9308: Author: ASF GitHub Bot Created on: 22/Feb/20 01:55 Start Date: 22/Feb/20 01:55 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #10852: [BEAM-9308] Decorrelate state cleanup timers URL: https://github.com/apache/beam/pull/10852#issuecomment-589904689 As written, this is incorrect. We currently rely on the state cleanup timer for watermark holds. This PR will cause that hold to be pushed later, which can cause incorrect grouping for any downstream aggregations. This is something we might be able to address by using the new outputTimestamp. This requires some thought though. Delaying the timer will also prevent downstream aggregations from firing. 3 minutes could cause issues if the window itself is much smaller. We want to reuse this timer for OnWindowExpiration, and this will delay all those callbacks as well. I wonder if it would be better to first root cause why the GC timers caused issues for your pipeline. One possibility: I believe that today any timers for a key are always prioritized over any data for that key. Maybe we need a better prioritization strategy so that large #s of timers don't starve out elements? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391021) Time Spent: 40m (was: 0.5h) > Optimize state cleanup at end-of-window > --- > > Key: BEAM-9308 > URL: https://issues.apache.org/jira/browse/BEAM-9308 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Assignee: Steve Niemitz >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > When using state with a large keyspace, you can end up with a large amount of > state cleanup timers set to fire all 1ms after the end of a window. This can > cause a momentary (I've observed 1-3 minute) lag in processing while windmill > and the java harness fire and process these cleanup timers. > By spreading the firing over a short period after the end of the window, we > can decorrelate the firing of the timers and smooth the load out, resulting > in much less impact from state cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=391017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391017 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 22/Feb/20 01:50 Start Date: 22/Feb/20 01:50 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919#discussion_r382875122 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -916,6 +919,9 @@ def _use_unified_worker(pipeline_options): debug_options = pipeline_options.view_as(DebugOptions) use_unified_worker_flag = 'use_unified_worker' + if debug_options.lookup_experiment(use_unified_worker_flag): Review comment: PTAL at this method - we need to remove some of the statements in 922-928. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391017) Time Spent: 40m (was: 0.5h) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9353) ByteBuddy Schema code does not properly handle null values
[ https://issues.apache.org/jira/browse/BEAM-9353?focusedWorklogId=391011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391011 ] ASF GitHub Bot logged work on BEAM-9353: Author: ASF GitHub Bot Created on: 22/Feb/20 01:31 Start Date: 22/Feb/20 01:31 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #10926: [BEAM-9353] Fix bytebuddy nullable URL: https://github.com/apache/beam/pull/10926#issuecomment-589901975 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391011) Time Spent: 0.5h (was: 20m) > ByteBuddy Schema code does not properly handle null values > -- > > Key: BEAM-9353 > URL: https://issues.apache.org/jira/browse/BEAM-9353 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Reuven Lax >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7427) JmsCheckpointMark can not be correctly encoded
[
https://issues.apache.org/jira/browse/BEAM-7427?focusedWorklogId=391010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391010
]
ASF GitHub Bot logged work on BEAM-7427:
Author: ASF GitHub Bot
Created on: 22/Feb/20 01:29
Start Date: 22/Feb/20 01:29
Worklog Time Spent: 10m
Work Description: tannaga commented on issue #8757: [BEAM-7427] Fix
JmsCheckpointMark Avro Encoding
URL: https://github.com/apache/beam/pull/8757#issuecomment-589901680
@iemejia Do you happen to know when Beam 2.20.0 will be released? Is there a
snapshot version with this fix I can try locally to see if it fixes the issue I
am running into?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391010)
Time Spent: 9.5h (was: 9h 20m)
> JmsCheckpointMark can not be correctly encoded
> --
>
> Key: BEAM-7427
> URL: https://issues.apache.org/jira/browse/BEAM-7427
> Project: Beam
> Issue Type: Bug
> Components: io-java-jms
>Affects Versions: 2.12.0, 2.13.0, 2.14.0, 2.15.0, 2.16.0, 2.17.0, 2.18.0,
> 2.19.0
> Environment: Message Broker : solace
> JMS Client (Over AMQP) : "org.apache.qpid:qpid-jms-client:0.42.0
>Reporter: Mourad
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> I get the following exception when reading from unbounded JMS Source:
>
> {code:java}
> Caused by: org.apache.avro.SchemaParseException: Illegal character in: this$0
> at org.apache.avro.Schema.validateName(Schema.java:1151)
> at org.apache.avro.Schema.access$200(Schema.java:81)
> at org.apache.avro.Schema$Field.(Schema.java:403)
> at org.apache.avro.Schema$Field.(Schema.java:396)
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:622)
> at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:740)
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:604)
> at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:218)
> at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:215)
> at
> avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at
> avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at
> avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at
> avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> {code}
>
> The exception is thrown by Avro when introspecting {{JmsCheckpointMark}} to
> generate schema.
> JmsIO config :
>
> {code:java}
> PCollection messages = pipeline.apply("read messages from the
> events broker", JmsIO.readMessage()
> .withConnectionFactory(jmsConnectionFactory) .withTopic(options.getTopic())
> .withMessageMapper(new DFAMessageMapper())
> .withCoder(AvroCoder.of(DFAMessage.class)));
> {code}
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9359) Use DataCatalog client libraries rather than gRPC stubs
[ https://issues.apache.org/jira/browse/BEAM-9359?focusedWorklogId=391003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391003 ] ASF GitHub Bot logged work on BEAM-9359: Author: ASF GitHub Bot Created on: 22/Feb/20 01:12 Start Date: 22/Feb/20 01:12 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #10917: [BEAM-9359] Switch to Data Catalog client URL: https://github.com/apache/beam/pull/10917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391003) Time Spent: 40m (was: 0.5h) > Use DataCatalog client libraries rather than gRPC stubs > --- > > Key: BEAM-9359 > URL: https://issues.apache.org/jira/browse/BEAM-9359 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The [GCP docs|https://cloud.google.com/data-catalog/docs/reference/libraries] > indicate this is the preferred way to use the service. > The client library sets some headers in requests that ensure they are > consistently routed properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=391002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391002 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 22/Feb/20 01:11 Start Date: 22/Feb/20 01:11 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10937: [BEAM-8019] Branch on having multiple environments. URL: https://github.com/apache/beam/pull/10937 As discussed this'll allow us to test the new feature against staging and even prod before it's rollback-safe. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=390998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390998 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 22/Feb/20 00:51 Start Date: 22/Feb/20 00:51 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10936: [BEAM-9063] migrate docker images to apache - fix postcommit failure URL: https://github.com/apache/beam/pull/10936#issuecomment-589896095 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390998) Time Spent: 6.5h (was: 6h 20m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.20.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=390997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390997 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 22/Feb/20 00:50 Start Date: 22/Feb/20 00:50 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #10936: [BEAM-9063] migrate docker images to apache - fix postcommit failure URL: https://github.com/apache/beam/pull/10936 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9296) Add typing annotation to python SDF
[ https://issues.apache.org/jira/browse/BEAM-9296?focusedWorklogId=390996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390996 ] ASF GitHub Bot logged work on BEAM-9296: Author: ASF GitHub Bot Created on: 22/Feb/20 00:45 Start Date: 22/Feb/20 00:45 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10935: [BEAM-9296] Clean up and add type-hints to SDF API URL: https://github.com/apache/beam/pull/10935#issuecomment-589895222 cc: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390996) Time Spent: 20m (was: 10m) > Add typing annotation to python SDF > --- > > Key: BEAM-9296 > URL: https://issues.apache.org/jira/browse/BEAM-9296 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042305#comment-17042305
]
Luke Cwik commented on BEAM-9252:
-
Vote thread:
https://lists.apache.org/thread.html/r5d762577f9ad3a71b91c749a382603019e7b0a15ef875a459c921229%40%3Cdev.beam.apache.org%3E
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-9336) beam_PostCommit_Py_ValCont tests timeout
[ https://issues.apache.org/jira/browse/BEAM-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yichi Zhang closed BEAM-9336. - Fix Version/s: Not applicable Resolution: Duplicate > beam_PostCommit_Py_ValCont tests timeout > - > > Key: BEAM-9336 > URL: https://issues.apache.org/jira/browse/BEAM-9336 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Yichi Zhang >Priority: Minor > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 50m > Remaining Estimate: 0h > > > * [[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/]] > Initial investigation: > The tests seem to fail due to the pytest global timeout. > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9296) Add typing annotation to python SDF
[ https://issues.apache.org/jira/browse/BEAM-9296?focusedWorklogId=390988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390988 ] ASF GitHub Bot logged work on BEAM-9296: Author: ASF GitHub Bot Created on: 22/Feb/20 00:29 Start Date: 22/Feb/20 00:29 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10935: [BEAM-9296] Clean up and add type-hints to SDF API URL: https://github.com/apache/beam/pull/10935 R: @chadrik Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9357) Bump upper end of Google Bigquery dependencies for python
[ https://issues.apache.org/jira/browse/BEAM-9357?focusedWorklogId=390986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390986 ] ASF GitHub Bot logged work on BEAM-9357: Author: ASF GitHub Bot Created on: 22/Feb/20 00:24 Start Date: 22/Feb/20 00:24 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10929: [BEAM-9357] Bump google cloud bigquery to 1.24.0 URL: https://github.com/apache/beam/pull/10929#issuecomment-589891670 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390986) Time Spent: 0.5h (was: 20m) > Bump upper end of Google Bigquery dependencies for python > - > > Key: BEAM-9357 > URL: https://issues.apache.org/jira/browse/BEAM-9357 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness > Environment: Python >Reporter: David Rubinstein >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > I am trying to use Dataflow with Beam Python and a package that depends on > google-resumable-media 0.5.0. The current google-cloud-bigquery (which is > only used for testing) depends on google-resumable-media <= 0.4.1. The upper > bound on the google-cloud-bigquery version should be loosened to solve > possible transitive dependency issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik reassigned BEAM-9252:
---
Assignee: Luke Cwik (was: Tomo Suzuki)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390985
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 22/Feb/20 00:14
Start Date: 22/Feb/20 00:14
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10930: [BEAM-9252] Exclude
jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#issuecomment-589889905
@robinyqiu Thank you.
@lukecwik Would you take care of the release of this vendor grpc change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390985)
Time Spent: 2h 10m (was: 2h)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042276#comment-17042276
]
Luke Cwik commented on BEAM-9252:
-
Now we just need someone to perform the vendored grpc 1.26.0 0.2 release and
then the Beam version bump to 0.2
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390984
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 22/Feb/20 00:12
Start Date: 22/Feb/20 00:12
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390984)
Time Spent: 2h (was: 1h 50m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9228) _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
[
https://issues.apache.org/jira/browse/BEAM-9228?focusedWorklogId=390983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390983
]
ASF GitHub Bot logged work on BEAM-9228:
Author: ASF GitHub Bot
Created on: 22/Feb/20 00:11
Start Date: 22/Feb/20 00:11
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #10847:
[BEAM-9228] Support further partition for FnApi ListBuffer
URL: https://github.com/apache/beam/pull/10847#discussion_r381568266
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
##
@@ -994,7 +1069,13 @@ def input_for(transform_id, input_id):
# The worker will be waiting on these inputs as well.
for other_input in data_input:
if other_input not in deferred_inputs:
-deferred_inputs[other_input] = _ListBuffer([])
+outputs = process_bundle_descriptor.transforms[
+ other_input].outputs.values()
+coder_id = process_bundle_descriptor.pcollections[
+ only_element(outputs)].coder_id
+coder = context.coders[coder_id]
+deferred_inputs[other_input] = _ListBuffer(
+coder_impl=coder.get_impl())
Review comment:
As commented at L1082 (of the PR branch), deferred inputs cannot be parallel
processed for now. Is it better to set coder_impl to None to reduce unnecessary
processes for now and add it back later when parallel processing is supported
for deferred_inputs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390983)
Time Spent: 2h 50m (was: 2h 40m)
> _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
>
>
> Key: BEAM-9228
> URL: https://issues.apache.org/jira/browse/BEAM-9228
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.16.0, 2.18.0, 2.19.0
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> A user reported following issue.
> -
> I have a set of tfrecord files, obtained by converting parquet files with
> Spark. Each file is roughly 1GB and I have 11 of those.
> I would expect simple statistics gathering (ie counting number of items of
> all files) to scale linearly with respect to the number of cores on my system.
> I am able to reproduce the issue with the minimal snippet below
> {code:java}
> import apache_beam as beam
> from apache_beam.options.pipeline_options import PipelineOptions
> from apache_beam.runners.portability import fn_api_runner
> from apache_beam.portability.api import beam_runner_api_pb2
> from apache_beam.portability import python_urns
> import sys
> pipeline_options = PipelineOptions(['--direct_num_workers', '4'])
> file_pattern = 'part-r-00*
> runner=fn_api_runner.FnApiRunner(
> default_environment=beam_runner_api_pb2.Environment(
> urn=python_urns.SUBPROCESS_SDK,
> payload=b'%s -m apache_beam.runners.worker.sdk_worker_main'
> % sys.executable.encode('ascii')))
> p = beam.Pipeline(runner=runner, options=pipeline_options)
> lines = (p | 'read' >> beam.io.tfrecordio.ReadFromTFRecord(file_pattern)
> | beam.combiners.Count.Globally()
> | beam.io.WriteToText('/tmp/output'))
> p.run()
> {code}
> Only one combination of apache_beam revision / worker type seems to work (I
> refer to https://beam.apache.org/documentation/runners/direct/ for the worker
> types)
> * beam 2.16; neither multithread nor multiprocess achieve high cpu usage on
> multiple cores
> * beam 2.17: able to achieve high cpu usage on all 4 cores
> * beam 2.18: not tested the mulithreaded mode but the multiprocess mode fails
> when trying to serialize the Environment instance most likely because of a
> change from 2.17 to 2.18.
> I also tried briefly SparkRunner with version 2.16 but was no able to achieve
> any throughput.
> What is the recommnended way to achieve what I am trying to ? How can I
> troubleshoot ?
> --
> This is caused by [this
> PR|https://github.com/apache/beam/commit/02f8ad4eee3ec0ea8cbdc0f99c1dad29f00a9f60].
> A
[jira] [Work logged] (BEAM-9359) Use DataCatalog client libraries rather than gRPC stubs
[ https://issues.apache.org/jira/browse/BEAM-9359?focusedWorklogId=390982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390982 ] ASF GitHub Bot logged work on BEAM-9359: Author: ASF GitHub Bot Created on: 22/Feb/20 00:11 Start Date: 22/Feb/20 00:11 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10917: [BEAM-9359] Switch to Data Catalog client URL: https://github.com/apache/beam/pull/10917#issuecomment-589889031 the linkage checker results are stale and are picking up warnings due to a jackson version change so it seems like no new linage issues have been reported This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390982) Time Spent: 0.5h (was: 20m) > Use DataCatalog client libraries rather than gRPC stubs > --- > > Key: BEAM-9359 > URL: https://issues.apache.org/jira/browse/BEAM-9359 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The [GCP docs|https://cloud.google.com/data-catalog/docs/reference/libraries] > indicate this is the preferred way to use the service. > The client library sets some headers in requests that ensure they are > consistently routed properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hannah Jiang resolved BEAM-9063. Fix Version/s: (was: Not applicable) 2.20.0 Resolution: Fixed > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.20.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390981
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 22/Feb/20 00:07
Start Date: 22/Feb/20 00:07
Worklog Time Spent: 10m
Work Description: robinyqiu commented on issue #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#issuecomment-589888289
LGTM. (Verified this is working on Flink and Dataflow runner.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390981)
Time Spent: 1h 50m (was: 1h 40m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9359) Use DataCatalog client libraries rather than gRPC stubs
[ https://issues.apache.org/jira/browse/BEAM-9359?focusedWorklogId=390980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390980 ] ASF GitHub Bot logged work on BEAM-9359: Author: ASF GitHub Bot Created on: 22/Feb/20 00:07 Start Date: 22/Feb/20 00:07 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10917: [BEAM-9359] Switch to Data Catalog client URL: https://github.com/apache/beam/pull/10917#issuecomment-589888157 I pushed a change to replace the StatueRuntimeException with specific gax exceptions. Planning on merging when tests are green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390980) Time Spent: 20m (was: 10m) > Use DataCatalog client libraries rather than gRPC stubs > --- > > Key: BEAM-9359 > URL: https://issues.apache.org/jira/browse/BEAM-9359 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The [GCP docs|https://cloud.google.com/data-catalog/docs/reference/libraries] > indicate this is the preferred way to use the service. > The client library sets some headers in requests that ensure they are > consistently routed properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9359) Use DataCatalog client libraries rather than gRPC stubs
[ https://issues.apache.org/jira/browse/BEAM-9359?focusedWorklogId=390979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390979 ] ASF GitHub Bot logged work on BEAM-9359: Author: ASF GitHub Bot Created on: 22/Feb/20 00:06 Start Date: 22/Feb/20 00:06 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10917: [BEAM-9359] Switch to Data Catalog client URL: https://github.com/apache/beam/pull/10917#issuecomment-589887905 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390979) Remaining Estimate: 0h Time Spent: 10m > Use DataCatalog client libraries rather than gRPC stubs > --- > > Key: BEAM-9359 > URL: https://issues.apache.org/jira/browse/BEAM-9359 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Fix For: 2.20.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The [GCP docs|https://cloud.google.com/data-catalog/docs/reference/libraries] > indicate this is the preferred way to use the service. > The client library sets some headers in requests that ensure they are > consistently routed properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9359) Use DataCatalog client libraries rather than gRPC stubs
Brian Hulette created BEAM-9359: --- Summary: Use DataCatalog client libraries rather than gRPC stubs Key: BEAM-9359 URL: https://issues.apache.org/jira/browse/BEAM-9359 Project: Beam Issue Type: Improvement Components: dsl-sql Reporter: Brian Hulette Assignee: Brian Hulette Fix For: 2.20.0 The [GCP docs|https://cloud.google.com/data-catalog/docs/reference/libraries] indicate this is the preferred way to use the service. The client library sets some headers in requests that ensure they are consistently routed properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9358) BigQueryIO potential write speed regression
[ https://issues.apache.org/jira/browse/BEAM-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Kozlov updated BEAM-9358: Description: There is a drastic decrease in Megabytes/second write speeds (from ~50 to ~5) [1], as well as 10x increase in runtime [2] for python BigQueryIO in the PerfKit dashboard. Seems to be fairly recent, started on the Feb 20th and continued on the Feb 21st. Maybe a flake, but still worth investigating. [1] [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=1939451611=847031938] [2] [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=2088160722=15365888] was: There is a drastic decrease in Megabytes/second write speeds (from ~50 to ~5), as well as 10x increase in runtime [2] for python BigQueryIO in the PerfKit dashboard [1]. Seems to be fairly recent, started on the Feb 20th and continued on the Feb 21st. Maybe a flake, but still worth investigating. [1] [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=1939451611=847031938] [2] [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=2088160722=15365888] > BigQueryIO potential write speed regression > --- > > Key: BEAM-9358 > URL: https://issues.apache.org/jira/browse/BEAM-9358 > Project: Beam > Issue Type: Task > Components: io-py-gcp >Affects Versions: 2.19.0 >Reporter: Kirill Kozlov >Priority: Minor > > There is a drastic decrease in Megabytes/second write speeds (from ~50 to ~5) > [1], as well as 10x increase in runtime [2] for python BigQueryIO in the > PerfKit dashboard. > Seems to be fairly recent, started on the Feb 20th and continued on the Feb > 21st. Maybe a flake, but still worth investigating. > [1] > [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=1939451611=847031938] > [2] > [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=2088160722=15365888] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9358) BigQueryIO potential write speed regression
Kirill Kozlov created BEAM-9358: --- Summary: BigQueryIO potential write speed regression Key: BEAM-9358 URL: https://issues.apache.org/jira/browse/BEAM-9358 Project: Beam Issue Type: Task Components: io-py-gcp Affects Versions: 2.19.0 Reporter: Kirill Kozlov There is a drastic decrease in Megabytes/second write speeds (from ~50 to ~5), as well as 10x increase in runtime [2] for python BigQueryIO in the PerfKit dashboard [1]. Seems to be fairly recent, started on the Feb 20th and continued on the Feb 21st. Maybe a flake, but still worth investigating. [1] [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=1939451611=847031938] [2] [https://apache-beam-testing.appspot.com/explore?dashboard=5667383922393088=2088160722=15365888] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8487) Python typehints: support forward references
[
https://issues.apache.org/jira/browse/BEAM-8487?focusedWorklogId=390977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390977
]
ASF GitHub Bot logged work on BEAM-8487:
Author: ASF GitHub Bot
Created on: 21/Feb/20 23:50
Start Date: 21/Feb/20 23:50
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #10932: [BEAM-8487]
Handle nested forward references
URL: https://github.com/apache/beam/pull/10932#discussion_r382857475
##
File path: sdks/python/apache_beam/typehints/native_type_compatibility.py
##
@@ -163,8 +163,14 @@ def is_any(typ):
return typ is typing.Any
+try:
+ _ForwardRef = typing.ForwardRef
+except AttributeError:
Review comment:
Could you please add a comment which Python version requires the branching?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390977)
Time Spent: 1.5h (was: 1h 20m)
> Python typehints: support forward references
>
>
> Key: BEAM-8487
> URL: https://issues.apache.org/jira/browse/BEAM-8487
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Typehints may be given as string literals:
> https://www.python.org/dev/peps/pep-0484/#forward-references
> These are currently not evaluated and result in errors.
> Example 1:
> {code}
> def test_typed_callable_string_hints(self):
> def do_fn(element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > return issubclass(sub, base)
> E TypeError: issubclass() arg 2 must be a class or tuple of classes
> typehints.py:1168: TypeError
> {code}
> Example 2:
> {code}
> def test_typed_dofn_string_hints(self):
> class MyDoFn(beam.DoFn):
> def process(self, element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(MyDoFn())
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > raise ValueError('%s is not iterable' % type_hint)
> E ValueError: typehints.List[str] is not iterable
> typehints.py:1194: ValueError
> {code}
> where the non-iterable entity the error refers to is a string literal
> ("typehints.List[str]").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-1833) Restructure Python pipeline construction to better follow the Runner API
[ https://issues.apache.org/jira/browse/BEAM-1833?focusedWorklogId=390976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390976 ] ASF GitHub Bot logged work on BEAM-1833: Author: ASF GitHub Bot Created on: 21/Feb/20 23:45 Start Date: 21/Feb/20 23:45 Worklog Time Spent: 10m Work Description: rohdesamuel commented on issue #10934: [BEAM-1833] Broke some people, setting the default to have the experiment be disabled URL: https://github.com/apache/beam/pull/10934#issuecomment-589883541 > I was under the impression that you would update the pipelines to pass in the experiment and not change this default. I didn't want to burden others with modifying the PipelineOptions. And we don't have a good story on how to fix code if they are relying on the old implementation. I decided that the best would be to be disabled by default and then we enable it for InteractiveBeam. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390976) Time Spent: 3h (was: 2h 50m) > Restructure Python pipeline construction to better follow the Runner API > > > Key: BEAM-1833 > URL: https://issues.apache.org/jira/browse/BEAM-1833 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Sam Rohde >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h > Remaining Estimate: 0h > > The most important part is removing the runner.apply overrides, but there are > also various other improvements (e.g. all inputs and outputs should be named). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-1833) Restructure Python pipeline construction to better follow the Runner API
[ https://issues.apache.org/jira/browse/BEAM-1833?focusedWorklogId=390975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390975 ] ASF GitHub Bot logged work on BEAM-1833: Author: ASF GitHub Bot Created on: 21/Feb/20 23:41 Start Date: 21/Feb/20 23:41 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10934: [BEAM-1833] Broke some people, setting the default to have the experiment be disabled URL: https://github.com/apache/beam/pull/10934#issuecomment-589882650 I was under the impression that you would update the pipelines to pass in the experiment and not change this default. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390975) Time Spent: 2h 50m (was: 2h 40m) > Restructure Python pipeline construction to better follow the Runner API > > > Key: BEAM-1833 > URL: https://issues.apache.org/jira/browse/BEAM-1833 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Sam Rohde >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > The most important part is removing the runner.apply overrides, but there are > also various other improvements (e.g. all inputs and outputs should be named). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-1833) Restructure Python pipeline construction to better follow the Runner API
[ https://issues.apache.org/jira/browse/BEAM-1833?focusedWorklogId=390972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390972 ] ASF GitHub Bot logged work on BEAM-1833: Author: ASF GitHub Bot Created on: 21/Feb/20 23:40 Start Date: 21/Feb/20 23:40 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #10934: [BEAM-1833] Broke some people, setting the default to have the experiment be disabled URL: https://github.com/apache/beam/pull/10934 Change-Id: Ie83fefc63ee03c917028b994dd2f2487f06e8109 The fix for BEAM-1833 broke some people, so I am setting the experiment to be disabled by default. For the InteractiveRunner we enable it by default. Since the runner ordinarily doesn't have access to the PipelineOptions we set the experiment through the apply() method. This is so that users won't have to set the experiment manually. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-1833) Restructure Python pipeline construction to better follow the Runner API
[ https://issues.apache.org/jira/browse/BEAM-1833?focusedWorklogId=390974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390974 ] ASF GitHub Bot logged work on BEAM-1833: Author: ASF GitHub Bot Created on: 21/Feb/20 23:40 Start Date: 21/Feb/20 23:40 Worklog Time Spent: 10m Work Description: rohdesamuel commented on issue #10934: [BEAM-1833] Broke some people, setting the default to have the experiment be disabled URL: https://github.com/apache/beam/pull/10934#issuecomment-589882465 R: @davidyan74 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390974) Time Spent: 2h 40m (was: 2.5h) > Restructure Python pipeline construction to better follow the Runner API > > > Key: BEAM-1833 > URL: https://issues.apache.org/jira/browse/BEAM-1833 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Sam Rohde >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > The most important part is removing the runner.apply overrides, but there are > also various other improvements (e.g. all inputs and outputs should be named). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8537) Provide WatermarkEstimatorProvider for different types of WatermarkEstimator
[ https://issues.apache.org/jira/browse/BEAM-8537?focusedWorklogId=390973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390973 ] ASF GitHub Bot logged work on BEAM-8537: Author: ASF GitHub Bot Created on: 21/Feb/20 23:40 Start Date: 21/Feb/20 23:40 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10933: [BEAM-8537] Update docstring of ManualWatermarkEstimator.set_watermark() URL: https://github.com/apache/beam/pull/10933#issuecomment-589882455 Fix python formatting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390973) Time Spent: 17h 50m (was: 17h 40m) > Provide WatermarkEstimatorProvider for different types of WatermarkEstimator > > > Key: BEAM-8537 > URL: https://issues.apache.org/jira/browse/BEAM-8537 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 17h 50m > Remaining Estimate: 0h > > This is a follow up for in-progress PR: > https://github.com/apache/beam/pull/9794. > Current implementation in PR9794 provides a default implementation of > WatermarkEstimator. For further work, we want to let WatermarkEstimator to be > a pure Interface. We'll provide a WatermarkEstimatorProvider to be able to > create a custom WatermarkEstimator per windowed value. It should be similar > to how we track restriction for SDF: > WatermarkEstimator <---> RestrictionTracker > WatermarkEstimatorProvider <---> RestrictionTrackerProvider > WatermarkEstimatorParam <---> RestrictionDoFnParam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?focusedWorklogId=390970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390970 ] ASF GitHub Bot logged work on BEAM-8564: Author: ASF GitHub Bot Created on: 21/Feb/20 23:28 Start Date: 21/Feb/20 23:28 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10254: [BEAM-8564] Add LZO compression and decompression support URL: https://github.com/apache/beam/pull/10254#issuecomment-589879728 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390970) Time Spent: 12h 20m (was: 12h 10m) > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Time Spent: 12h 20m > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?focusedWorklogId=390969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390969 ] ASF GitHub Bot logged work on BEAM-8564: Author: ASF GitHub Bot Created on: 21/Feb/20 23:28 Start Date: 21/Feb/20 23:28 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10254: [BEAM-8564] Add LZO compression and decompression support URL: https://github.com/apache/beam/pull/10254#issuecomment-589879703 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390969) Time Spent: 12h 10m (was: 12h) > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Time Spent: 12h 10m > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8537) Provide WatermarkEstimatorProvider for different types of WatermarkEstimator
[ https://issues.apache.org/jira/browse/BEAM-8537?focusedWorklogId=390964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390964 ] ASF GitHub Bot logged work on BEAM-8537: Author: ASF GitHub Bot Created on: 21/Feb/20 23:20 Start Date: 21/Feb/20 23:20 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10933: [BEAM-8537] Update docstring of ManualWatermarkEstimator.set_watermark() URL: https://github.com/apache/beam/pull/10933 Follow up with Luke's comment: https://github.com/apache/beam/pull/10375#discussion_r382727370 R: @lukecwik Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?focusedWorklogId=390963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390963 ] ASF GitHub Bot logged work on BEAM-8564: Author: ASF GitHub Bot Created on: 21/Feb/20 23:16 Start Date: 21/Feb/20 23:16 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10254: [BEAM-8564] Add LZO compression and decompression support URL: https://github.com/apache/beam/pull/10254#issuecomment-589876952 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390963) Time Spent: 12h (was: 11h 50m) > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Time Spent: 12h > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?focusedWorklogId=390962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390962 ] ASF GitHub Bot logged work on BEAM-8564: Author: ASF GitHub Bot Created on: 21/Feb/20 23:13 Start Date: 21/Feb/20 23:13 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10254: [BEAM-8564] Add LZO compression and decompression support URL: https://github.com/apache/beam/pull/10254#issuecomment-589876072 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390962) Time Spent: 11h 50m (was: 11h 40m) > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Time Spent: 11h 50m > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9353) ByteBuddy Schema code does not properly handle null values
[ https://issues.apache.org/jira/browse/BEAM-9353?focusedWorklogId=390961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390961 ] ASF GitHub Bot logged work on BEAM-9353: Author: ASF GitHub Bot Created on: 21/Feb/20 23:07 Start Date: 21/Feb/20 23:07 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #10926: [BEAM-9353] Fix bytebuddy nullable URL: https://github.com/apache/beam/pull/10926#issuecomment-589874592 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390961) Time Spent: 20m (was: 10m) > ByteBuddy Schema code does not properly handle null values > -- > > Key: BEAM-9353 > URL: https://issues.apache.org/jira/browse/BEAM-9353 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Reuven Lax >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8487) Python typehints: support forward references
[
https://issues.apache.org/jira/browse/BEAM-8487?focusedWorklogId=390943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390943
]
ASF GitHub Bot logged work on BEAM-8487:
Author: ASF GitHub Bot
Created on: 21/Feb/20 22:13
Start Date: 21/Feb/20 22:13
Worklog Time Spent: 10m
Work Description: udim commented on issue #10932: [BEAM-8487] Handle
nested forward references
URL: https://github.com/apache/beam/pull/10932#issuecomment-589859233
R: @tvalentyn
CC: @robertwb
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390943)
Time Spent: 1h 20m (was: 1h 10m)
> Python typehints: support forward references
>
>
> Key: BEAM-8487
> URL: https://issues.apache.org/jira/browse/BEAM-8487
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Typehints may be given as string literals:
> https://www.python.org/dev/peps/pep-0484/#forward-references
> These are currently not evaluated and result in errors.
> Example 1:
> {code}
> def test_typed_callable_string_hints(self):
> def do_fn(element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > return issubclass(sub, base)
> E TypeError: issubclass() arg 2 must be a class or tuple of classes
> typehints.py:1168: TypeError
> {code}
> Example 2:
> {code}
> def test_typed_dofn_string_hints(self):
> class MyDoFn(beam.DoFn):
> def process(self, element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(MyDoFn())
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > raise ValueError('%s is not iterable' % type_hint)
> E ValueError: typehints.List[str] is not iterable
> typehints.py:1194: ValueError
> {code}
> where the non-iterable entity the error refers to is a string literal
> ("typehints.List[str]").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8487) Python typehints: support forward references
[ https://issues.apache.org/jira/browse/BEAM-8487?focusedWorklogId=390942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390942 ] ASF GitHub Bot logged work on BEAM-8487: Author: ASF GitHub Bot Created on: 21/Feb/20 22:12 Start Date: 21/Feb/20 22:12 Worklog Time Spent: 10m Work Description: udim commented on pull request #10932: [BEAM-8487] Handle nested forward references URL: https://github.com/apache/beam/pull/10932 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9357) Bump upper end of Google Bigquery dependencies for python
[ https://issues.apache.org/jira/browse/BEAM-9357?focusedWorklogId=390940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390940 ] ASF GitHub Bot logged work on BEAM-9357: Author: ASF GitHub Bot Created on: 21/Feb/20 22:10 Start Date: 21/Feb/20 22:10 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10929: [BEAM-9357] Bump google cloud bigquery to 1.24.0 URL: https://github.com/apache/beam/pull/10929#issuecomment-589858561 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390940) Time Spent: 20m (was: 10m) > Bump upper end of Google Bigquery dependencies for python > - > > Key: BEAM-9357 > URL: https://issues.apache.org/jira/browse/BEAM-9357 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness > Environment: Python >Reporter: David Rubinstein >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > I am trying to use Dataflow with Beam Python and a package that depends on > google-resumable-media 0.5.0. The current google-cloud-bigquery (which is > only used for testing) depends on google-resumable-media <= 0.4.1. The upper > bound on the google-cloud-bigquery version should be loosened to solve > possible transitive dependency issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390938
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 22:08
Start Date: 21/Feb/20 22:08
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #10930: [BEAM-9252] Exclude
jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#issuecomment-589857813
Lets hear back from @robinyqiu before merging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390938)
Time Spent: 1h 40m (was: 1.5h)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390937
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 22:07
Start Date: 21/Feb/20 22:07
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382828707
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -967,7 +967,7 @@ class BeamModulePlugin implements Plugin {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
+ exclude "META-INF/versions/*/module-info.class"
Review comment:
Reverting this change
https://github.com/apache/beam/pull/10463/files#r382787180 .
validateShadedJarDoesntLeakNonProjectClasses should fail upon root
module-info.class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390937)
Time Spent: 1.5h (was: 1h 20m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390936
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 22:06
Start Date: 21/Feb/20 22:06
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382828300
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390936)
Time Spent: 1h 20m (was: 1h 10m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390935
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 22:01
Start Date: 21/Feb/20 22:01
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382826597
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
Now I understand, misread what the exclusion was for. Thought it was for
excluding the file and not for excluding files during validation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390935)
Time Spent: 1h 10m (was: 1h)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8487) Python typehints: support forward references
[
https://issues.apache.org/jira/browse/BEAM-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042200#comment-17042200
]
Udi Meiri commented on BEAM-8487:
-
typing.get_type_hints() could probably do the resolution in from_callable.
> Python typehints: support forward references
>
>
> Key: BEAM-8487
> URL: https://issues.apache.org/jira/browse/BEAM-8487
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Typehints may be given as string literals:
> https://www.python.org/dev/peps/pep-0484/#forward-references
> These are currently not evaluated and result in errors.
> Example 1:
> {code}
> def test_typed_callable_string_hints(self):
> def do_fn(element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > return issubclass(sub, base)
> E TypeError: issubclass() arg 2 must be a class or tuple of classes
> typehints.py:1168: TypeError
> {code}
> Example 2:
> {code}
> def test_typed_dofn_string_hints(self):
> class MyDoFn(beam.DoFn):
> def process(self, element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(MyDoFn())
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > raise ValueError('%s is not iterable' % type_hint)
> E ValueError: typehints.List[str] is not iterable
> typehints.py:1194: ValueError
> {code}
> where the non-iterable entity the error refers to is a string literal
> ("typehints.List[str]").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9356) Flink python test logs are too noisy
[ https://issues.apache.org/jira/browse/BEAM-9356?focusedWorklogId=390934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390934 ] ASF GitHub Bot logged work on BEAM-9356: Author: ASF GitHub Bot Created on: 21/Feb/20 22:00 Start Date: 21/Feb/20 22:00 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10931: [BEAM-9356] reduce Flink test logs to warn URL: https://github.com/apache/beam/pull/10931 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=390933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390933 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 21/Feb/20 21:59 Start Date: 21/Feb/20 21:59 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #10612: [BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390933) Time Spent: 6h 10m (was: 6h) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 6h 10m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390931
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 21:57
Start Date: 21/Feb/20 21:57
Worklog Time Spent: 10m
Work Description: robinyqiu commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382824672
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
`**/module-info.class` is excluding the top level pattern. If I understand
it correctly, the problem is we *only* want the top level one to be excluded.
This was a regression introduced
[here](https://github.com/apache/beam/pull/10463/files#diff-23833058cbf2c1172b90e7764032aa59R959)
during the last grpc version change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390931)
Time Spent: 50m (was: 40m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390932
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 21:57
Start Date: 21/Feb/20 21:57
Worklog Time Spent: 10m
Work Description: robinyqiu commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382824672
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
`**/module-info.class` is excluding the top level pattern. If I understand
it correctly, the problem is we **only** want the top level one to be excluded.
This was a regression introduced
[here](https://github.com/apache/beam/pull/10463/files#diff-23833058cbf2c1172b90e7764032aa59R959)
during the last grpc version change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390932)
Time Spent: 1h (was: 50m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390930=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390930
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 21:56
Start Date: 21/Feb/20 21:56
Worklog Time Spent: 10m
Work Description: robinyqiu commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382824672
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
`**/module-info.class` is excluding the top level pattern. If I understand
it correctly, the problem is we only want the top level one to be excluded.
This was a regression introduced
[here](https://github.com/apache/beam/pull/10463/files#diff-23833058cbf2c1172b90e7764032aa59R959)
during the last grpc version change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390930)
Time Spent: 40m (was: 0.5h)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8487) Python typehints: support forward references
[
https://issues.apache.org/jira/browse/BEAM-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042195#comment-17042195
]
Udi Meiri commented on BEAM-8487:
-
Forward references can also be nested:
{code}
>>> typing.List['int']
typing.List[_ForwardRef('int')]
{code}
> Python typehints: support forward references
>
>
> Key: BEAM-8487
> URL: https://issues.apache.org/jira/browse/BEAM-8487
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Typehints may be given as string literals:
> https://www.python.org/dev/peps/pep-0484/#forward-references
> These are currently not evaluated and result in errors.
> Example 1:
> {code}
> def test_typed_callable_string_hints(self):
> def do_fn(element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > return issubclass(sub, base)
> E TypeError: issubclass() arg 2 must be a class or tuple of classes
> typehints.py:1168: TypeError
> {code}
> Example 2:
> {code}
> def test_typed_dofn_string_hints(self):
> class MyDoFn(beam.DoFn):
> def process(self, element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(MyDoFn())
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > raise ValueError('%s is not iterable' % type_hint)
> E ValueError: typehints.List[str] is not iterable
> typehints.py:1194: ValueError
> {code}
> where the non-iterable entity the error refers to is a string literal
> ("typehints.List[str]").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390915
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 21:32
Start Date: 21/Feb/20 21:32
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382813306
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
A similar change is needed in:
https://github.com/apache/beam/blob/1133f7b5ae5d10120563e267d7c65f669ccc2bf1/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L970
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390915)
Time Spent: 0.5h (was: 20m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390914
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 21:32
Start Date: 21/Feb/20 21:32
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10930: [BEAM-9252]
Exclude jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#discussion_r382815496
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -131,15 +131,15 @@ artifactId=${project.name}
inputs.files project.configurations.shadow.artifacts.files
doLast {
project.configurations.shadow.artifacts.files.each {
-FileTree exposedClasses = project.zipTree(it).matching {
+FileTree unexpectedlyExposedClasses = project.zipTree(it).matching
{
include "**/*.class"
exclude "org/apache/beam/vendor/**"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "**/module-info.class"
Review comment:
The glob matching semantics
[allegedly](https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/AbstractCopyTask.html#include-java.lang.String...-)
follow [ANT pattern
matching](https://ant.apache.org/manual/dirtasks.html#patterns) so I'm lost as
to why `**/module-info.class` didn't exclude the top level pattern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390914)
Time Spent: 0.5h (was: 20m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390912 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 21/Feb/20 21:26 Start Date: 21/Feb/20 21:26 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10908: [BEAM-9339] Declare capabilities for Python SDK. URL: https://github.com/apache/beam/pull/10908 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390912) Time Spent: 2h 50m (was: 2h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390898
]
ASF GitHub Bot logged work on BEAM-9252:
Author: ASF GitHub Bot
Created on: 21/Feb/20 20:50
Start Date: 21/Feb/20 20:50
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #10930: [BEAM-9252] Exclude
jboss's Main and module-info.java
URL: https://github.com/apache/beam/pull/10930#issuecomment-589829637
@robinyqiu I appreciate if you can confirm that this fix works in Cloud
Dataflow. I only confirmed shading plugin works.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390898)
Time Spent: 20m (was: 10m)
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/BEAM-9252?focusedWorklogId=390897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390897 ] ASF GitHub Bot logged work on BEAM-9252: Author: ASF GitHub Bot Created on: 21/Feb/20 20:47 Start Date: 21/Feb/20 20:47 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10930: [BEAM-9252] Exclude jboss's Main and module-info.java URL: https://github.com/apache/beam/pull/10930 CC: @robinyqiu ``` suztomo@suxtomo24:~/beam6/vendor/grpc-1_26_0$ ../../gradlew -PvendoredDependenciesOnly -Ppublishing publishToMavenLocal Configuration on demand is an incubating feature. Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 12s 4 actionable tasks: 4 executed suztomo@suxtomo24:~/beam6/vendor/grpc-1_26_0$ jar tf /usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.2-SNAPSHOT/beam-vendor-grpc-1_26_0-0.2-SNAPSHOT.jar |grep org/jboss/modules/Main suztomo@suxtomo24:~/beam6/vendor/grpc-1_26_0$ jar tf /usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.2-SNAPSHOT/beam-vendor-grpc-1_26_0-0.2-SNAPSHOT.jar |grep module-info suztomo@suxtomo24:~/beam6/vendor/grpc-1_26_0$ ``` The solution worked for jbonofre-beam-samples shading: https://gist.github.com/suztomo/929f82458c30176de669e0abf1a443a4. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build
[jira] [Work logged] (BEAM-2546) Create InfluxDbIO
[ https://issues.apache.org/jira/browse/BEAM-2546?focusedWorklogId=390896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390896 ] ASF GitHub Bot logged work on BEAM-2546: Author: ASF GitHub Bot Created on: 21/Feb/20 20:44 Start Date: 21/Feb/20 20:44 Worklog Time Spent: 10m Work Description: bipinupd commented on issue #10604: BEAM-2546 Beam IO for InfluxDB URL: https://github.com/apache/beam/pull/10604#issuecomment-589827867 > Oups forgot this one, I will take a look. Thanks @iemejia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390896) Time Spent: 2h 10m (was: 2h) > Create InfluxDbIO > - > > Key: BEAM-2546 > URL: https://issues.apache.org/jira/browse/BEAM-2546 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9198) BeamSQL aggregation analytics functionality
[
https://issues.apache.org/jira/browse/BEAM-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Wang updated BEAM-9198:
---
Description:
Mentor email: ruw...@google.com. Feel free to send emails for your questions.
Project Information
-
BeamSQL has a long list of of aggregation/aggregation analytics functionalities
to support.
To begin with, you will need to support this syntax:
{code:sql}
analytic_function_name ( [ argument_list ] )
OVER (
[ PARTITION BY partition_expression_list ]
[ ORDER BY expression [{ ASC | DESC }] [, ...] ]
[ window_frame_clause ]
)
{code}
As there is a long list of analytics functions, a good start point is support
rank() first.
This will requires touch core components of BeamSQL:
1. SQL parser to support the syntax above.
2. SQL core to implement physical relational operator.
3. Distributed algorithms to implement a list of functions in a distributed
manner.
4. Build benchmarks to measure performance of your implementation.
To understand what SQL analytics functionality is, you could check this great
explanation doc:
https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
To know about Beam's programming model, check:
https://beam.apache.org/documentation/programming-guide/#overview
was:
Mentor email: ruw...@google.com. Feel free to send emails for your questions.
Project Information
-
BeamSQL has a long list of of aggregation/aggregation analytics functionalities
to support.
To begin with, you will need to support this syntax:
{code:sql}
analytic_function_name ( [ argument_list ] )
OVER (
[ PARTITION BY partition_expression_list ]
[ ORDER BY expression [{ ASC | DESC }] [, ...] ]
[ window_frame_clause ]
)
{code}
This will requires touch core components of BeamSQL:
1. SQL parser to support the syntax above.
2. SQL core to implement physical relational operator.
3. Distributed algorithms to implement a list of functions in a distributed
manner.
4. Build benchmarks to measure performance of your implementation.
To understand what SQL analytics functionality is, you could check this great
explanation doc:
https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
To know about Beam's programming model, check:
https://beam.apache.org/documentation/programming-guide/#overview
> BeamSQL aggregation analytics functionality
>
>
> Key: BEAM-9198
> URL: https://issues.apache.org/jira/browse/BEAM-9198
> Project: Beam
> Issue Type: Task
> Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> Mentor email: ruw...@google.com. Feel free to send emails for your questions.
> Project Information
> -
> BeamSQL has a long list of of aggregation/aggregation analytics
> functionalities to support.
> To begin with, you will need to support this syntax:
> {code:sql}
> analytic_function_name ( [ argument_list ] )
> OVER (
> [ PARTITION BY partition_expression_list ]
> [ ORDER BY expression [{ ASC | DESC }] [, ...] ]
> [ window_frame_clause ]
> )
> {code}
> As there is a long list of analytics functions, a good start point is support
> rank() first.
> This will requires touch core components of BeamSQL:
> 1. SQL parser to support the syntax above.
> 2. SQL core to implement physical relational operator.
> 3. Distributed algorithms to implement a list of functions in a distributed
> manner.
> 4. Build benchmarks to measure performance of your implementation.
> To understand what SQL analytics functionality is, you could check this great
> explanation doc:
> https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
> To know about Beam's programming model, check:
> https://beam.apache.org/documentation/programming-guide/#overview
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=390883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390883
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 21/Feb/20 20:20
Start Date: 21/Feb/20 20:20
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10463: [BEAM-9030]
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r382787180
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -956,7 +956,7 @@ class BeamModulePlugin implements Plugin {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "META-INF/versions/*/module-info.class"
+ exclude "**/module-info.class"
Review comment:
This started ignoring "module-info.class".
Wiithout this change, we could have detected module-info.class sneaking into
the vendor jar.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390883)
Time Spent: 8h 20m (was: 8h 10m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=390884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390884
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 21/Feb/20 20:20
Start Date: 21/Feb/20 20:20
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10463: [BEAM-9030]
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r382787413
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -956,7 +956,7 @@ class BeamModulePlugin implements Plugin {
FileTree exposedClasses = project.zipTree(it).matching {
Review comment:
unexpectedlyExposedClasses
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390884)
Time Spent: 8.5h (was: 8h 20m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9357) Bump upper end of Google Bigquery dependencies for python
[ https://issues.apache.org/jira/browse/BEAM-9357?focusedWorklogId=390882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390882 ] ASF GitHub Bot logged work on BEAM-9357: Author: ASF GitHub Bot Created on: 21/Feb/20 20:19 Start Date: 21/Feb/20 20:19 Worklog Time Spent: 10m Work Description: drubinstein commented on pull request #10929: [BEAM-9357] Bump google cloud bigquery to 1.24.0 URL: https://github.com/apache/beam/pull/10929 I'm bumping the upper end of the google cloud bigquery package to the most recent package so it is easier to use other google-cloud-python packages. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Created] (BEAM-9357) Bump upper end of Google Bigquery dependencies for python
David Rubinstein created BEAM-9357: -- Summary: Bump upper end of Google Bigquery dependencies for python Key: BEAM-9357 URL: https://issues.apache.org/jira/browse/BEAM-9357 Project: Beam Issue Type: Bug Components: sdk-py-harness Environment: Python Reporter: David Rubinstein I am trying to use Dataflow with Beam Python and a package that depends on google-resumable-media 0.5.0. The current google-cloud-bigquery (which is only used for testing) depends on google-resumable-media <= 0.4.1. The upper bound on the google-cloud-bigquery version should be loosened to solve possible transitive dependency issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9356) Flink python test logs are too noisy
[ https://issues.apache.org/jira/browse/BEAM-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9356: -- Issue Type: Improvement (was: Bug) > Flink python test logs are too noisy > > > Key: BEAM-9356 > URL: https://issues.apache.org/jira/browse/BEAM-9356 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Minor > Labels: portability-flink, testing > > When running Python tests on the Flink runner, all the info logs from the > Flink local cluster are printed to the test log, which creates a lot of > noise. Especially severe for Flink Python PVR tests, which have 30+MB log > files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9356) Flink python test logs are too noisy
[ https://issues.apache.org/jira/browse/BEAM-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9356: -- Status: Open (was: Triage Needed) > Flink python test logs are too noisy > > > Key: BEAM-9356 > URL: https://issues.apache.org/jira/browse/BEAM-9356 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Minor > Labels: portability-flink, testing > > When running Python tests on the Flink runner, all the info logs from the > Flink local cluster are printed to the test log, which creates a lot of > noise. Especially severe for Flink Python PVR tests, which have 30+MB log > files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9356) Flink python test logs are too noisy
Kyle Weaver created BEAM-9356: - Summary: Flink python test logs are too noisy Key: BEAM-9356 URL: https://issues.apache.org/jira/browse/BEAM-9356 Project: Beam Issue Type: Bug Components: runner-flink Reporter: Kyle Weaver Assignee: Kyle Weaver When running Python tests on the Flink runner, all the info logs from the Flink local cluster are printed to the test log, which creates a lot of noise. Especially severe for Flink Python PVR tests, which have 30+MB log files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042133#comment-17042133
]
Tomo Suzuki commented on BEAM-9252:
---
For module-info.class, Beam used to have verification mechanism to detect
unexpected module-info.class:
This change disabled that
https://github.com/apache/beam/pull/10463/files#diff-23833058cbf2c1172b90e7764032aa59R959
.
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=390870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390870 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 21/Feb/20 19:44 Start Date: 21/Feb/20 19:44 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#discussion_r382771623 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java ## @@ -47,7 +50,9 @@ import org.apache.beam.sdk.testing.ValidatesRunner; import org.apache.beam.sdk.transforms.DoFn.BoundedPerElement; import org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement; +import org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker; import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker; +import org.apache.beam.sdk.transforms.splittabledofn.SplitResult; Review comment: fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390870) Time Spent: 1h (was: 50m) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=390868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390868 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 21/Feb/20 19:37 Start Date: 21/Feb/20 19:37 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10612: [BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-589803549 @ibzib, could you please take a look to [bed36ef](https://github.com/apache/beam/pull/10612/commits/bed36ef63f8e0b32f4d1be71d42d1f76ef9da9ec)? I merged spark changes to the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390868) Time Spent: 6h (was: 5h 50m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 6h > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=390862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390862 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 21/Feb/20 19:29 Start Date: 21/Feb/20 19:29 Worklog Time Spent: 10m Work Description: udim commented on issue #10927: [BEAM-8280] Fix IOTypeHints origin traceback on partials URL: https://github.com/apache/beam/pull/10927#issuecomment-589800324 CC: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390862) Time Spent: 6h 40m (was: 6.5h) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9355) Python typehints: support NewType
[ https://issues.apache.org/jira/browse/BEAM-9355?focusedWorklogId=390863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390863 ] ASF GitHub Bot logged work on BEAM-9355: Author: ASF GitHub Bot Created on: 21/Feb/20 19:29 Start Date: 21/Feb/20 19:29 Worklog Time Spent: 10m Work Description: udim commented on issue #10928: [BEAM-9355] Basic support for NewType URL: https://github.com/apache/beam/pull/10928#issuecomment-589800401 R: @kennknowles CC: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390863) Time Spent: 20m (was: 10m) > Python typehints: support NewType > - > > Key: BEAM-9355 > URL: https://issues.apache.org/jira/browse/BEAM-9355 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Udi Meiri >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > https://docs.python.org/3/library/typing.html#newtype -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=390861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390861 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 21/Feb/20 19:29 Start Date: 21/Feb/20 19:29 Worklog Time Spent: 10m Work Description: udim commented on issue #10927: [BEAM-8280] Fix IOTypeHints origin traceback on partials URL: https://github.com/apache/beam/pull/10927#issuecomment-589800163 R: @aaltay CC: @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390861) Time Spent: 6.5h (was: 6h 20m) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9355) Python typehints: support NewType
[ https://issues.apache.org/jira/browse/BEAM-9355?focusedWorklogId=390860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390860 ] ASF GitHub Bot logged work on BEAM-9355: Author: ASF GitHub Bot Created on: 21/Feb/20 19:27 Start Date: 21/Feb/20 19:27 Worklog Time Spent: 10m Work Description: udim commented on pull request #10928: [BEAM-9355] Basic support for NewType URL: https://github.com/apache/beam/pull/10928 As in: don't crash, convert to Any. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=390859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390859 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 21/Feb/20 19:25 Start Date: 21/Feb/20 19:25 Worklog Time Spent: 10m Work Description: udim commented on pull request #10927: [BEAM-8280] Fix IOTypeHints origin traceback on partials URL: https://github.com/apache/beam/pull/10927 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Created] (BEAM-9355) Python typehints: support NewType
Udi Meiri created BEAM-9355: --- Summary: Python typehints: support NewType Key: BEAM-9355 URL: https://issues.apache.org/jira/browse/BEAM-9355 Project: Beam Issue Type: New Feature Components: sdk-py-core Reporter: Udi Meiri https://docs.python.org/3/library/typing.html#newtype -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8537) Provide WatermarkEstimatorProvider for different types of WatermarkEstimator
[ https://issues.apache.org/jira/browse/BEAM-8537?focusedWorklogId=390857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390857 ] ASF GitHub Bot logged work on BEAM-8537: Author: ASF GitHub Bot Created on: 21/Feb/20 19:18 Start Date: 21/Feb/20 19:18 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10375: [BEAM-8537] Provide WatermarkEstimator to track watermark URL: https://github.com/apache/beam/pull/10375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390857) Time Spent: 17.5h (was: 17h 20m) > Provide WatermarkEstimatorProvider for different types of WatermarkEstimator > > > Key: BEAM-8537 > URL: https://issues.apache.org/jira/browse/BEAM-8537 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 17.5h > Remaining Estimate: 0h > > This is a follow up for in-progress PR: > https://github.com/apache/beam/pull/9794. > Current implementation in PR9794 provides a default implementation of > WatermarkEstimator. For further work, we want to let WatermarkEstimator to be > a pure Interface. We'll provide a WatermarkEstimatorProvider to be able to > create a custom WatermarkEstimator per windowed value. It should be similar > to how we track restriction for SDF: > WatermarkEstimator <---> RestrictionTracker > WatermarkEstimatorProvider <---> RestrictionTrackerProvider > WatermarkEstimatorParam <---> RestrictionDoFnParam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042106#comment-17042106
]
Tomo Suzuki edited comment on BEAM-9252 at 2/21/20 7:16 PM:
The jboss module's Main class seems incorrect.
{noformat}
suztomo@suxtomo24:~/jbonofre-beam-samples$ javap -verbose -cp
/usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
...
LocalVariableTable:
Start Length Slot Name Signature
562 6497415 cnt I
...
0 0 0 args [Ljava/lang/String;
3 65533 1 argsLen I
5 65531 2 deps Ljava/lang/String;
9 65527 3 moduleArgs [Ljava/lang/String;
12 65524 4 modulePath Ljava/lang/String;
15 65521 5 configPath Ljava/lang/String;
18 65518 6 classpath Ljava/lang/S
{noformat}
{noformat}
suztomo@suxtomo24:~/jbonofre-beam-samples$ java -cp
/usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.ClassFormatError: Invalid length 64974 in
LocalVariableTable in class file
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
{noformat}
jboss.modules.Main before vendoring is not broken:
{noformat}
suztomo@suxtomo24:~/jbonofre-beam-samples$ java -cp
~/.gradle/caches/modules-2/files-2.1/org.jboss.modules/jboss-modules/1.1.0.Beta1/96c3c0f0f56c263ae0897bf68b99ce3510cfe5b/jboss-modules-1.1.0.Beta1.jar
org/jboss/modules/Main
No module specified
Usage: java [-jvmoptions...] -jar jboss-modules.jar [-options...]
[args...]
java [-jvmoptions...] -jar jboss-modules.jar [-options...] -jar
[args...]
java [-jvmoptions...] -jar jboss-modules.jar [-options...] -cp
[args...]
...
{noformat}
was (Author: suztomo):
The jboss's Main class seems incorrect.
{noformat}
suztomo@suxtomo24:~/jbonofre-beam-samples$ javap -verbose -cp
/usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
...
LocalVariableTable:
Start Length Slot Name Signature
562 6497415 cnt I
588 2515 e Ljava/lang/IndexOutOfBoundsException;
52 56114 arg Ljava/lang/String;
36 6550012 i I
39 6549713 argsLength I
717 313 environmentLoader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleLoader;
749 6478712 loader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleLoader;
759 6477714 moduleIdentifier
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleIdentifier;
0 012 loader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleLoader;
0 014 moduleIdentifier
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleIdentifier;
0 020 i I
945 6459120 old Ljava/lang/ClassLoader;
0104517 reader Ljava/io/BufferedReader;
864 18118 name Ljava/lang/String;
872 17319 line Ljava/lang/String;
0 017 ignored Ljava/io/IOException;
1070 023 ignored Ljava/io/IOException;
0 015 classLoader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleClassLoader;
0 016 stream Ljava/io/InputStream;
0 016 e
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleNotFoundException;
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=390856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390856 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 21/Feb/20 19:15 Start Date: 21/Feb/20 19:15 Worklog Time Spent: 10m Work Description: Ardagan commented on pull request #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#discussion_r382727459 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java ## @@ -47,7 +50,9 @@ import org.apache.beam.sdk.testing.ValidatesRunner; import org.apache.beam.sdk.transforms.DoFn.BoundedPerElement; import org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement; +import org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker; import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker; +import org.apache.beam.sdk.transforms.splittabledofn.SplitResult; Review comment: At https://s.apache.org/splittable-do-fn>, second quote is missing that breaks all formatting in github and should break href. Can you fix typo please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390856) Time Spent: 50m (was: 40m) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomo Suzuki reassigned BEAM-9252:
-
Assignee: Tomo Suzuki
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Assignee: Tomo Suzuki
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9085) Performance regression in np.random.RandomState() skews performance test results across Python 2/3 on Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9085?focusedWorklogId=390854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390854 ] ASF GitHub Bot logged work on BEAM-9085: Author: ASF GitHub Bot Created on: 21/Feb/20 19:06 Start Date: 21/Feb/20 19:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10924: Revert "[BEAM-9085] Fix performance regression in SyntheticSource" URL: https://github.com/apache/beam/pull/10924#issuecomment-589791506 Thank you; some of the tests we have internally at Google also started timing out with this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390854) Time Spent: 3.5h (was: 3h 20m) > Performance regression in np.random.RandomState() skews performance test > results across Python 2/3 on Dataflow > -- > > Key: BEAM-9085 > URL: https://issues.apache.org/jira/browse/BEAM-9085 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Tests show that the performance of core Beam operations in Python 3.x on > Dataflow can be a few time slower than in Python 2.7. We should investigate > what's the cause of the problem. > Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A > dashboard with runtime results can be found here [2]. > [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py > [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9085) Performance regression in np.random.RandomState() skews performance test results across Python 2/3 on Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9085?focusedWorklogId=390855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390855 ] ASF GitHub Bot logged work on BEAM-9085: Author: ASF GitHub Bot Created on: 21/Feb/20 19:06 Start Date: 21/Feb/20 19:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10924: Revert "[BEAM-9085] Fix performance regression in SyntheticSource" URL: https://github.com/apache/beam/pull/10924 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390855) Time Spent: 3h 40m (was: 3.5h) > Performance regression in np.random.RandomState() skews performance test > results across Python 2/3 on Dataflow > -- > > Key: BEAM-9085 > URL: https://issues.apache.org/jira/browse/BEAM-9085 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Tests show that the performance of core Beam operations in Python 3.x on > Dataflow can be a few time slower than in Python 2.7. We should investigate > what's the cause of the problem. > Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A > dashboard with runtime results can be found here [2]. > [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py > [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9354) How long does PubSubIO message deduplication last?
Tianzi Cai created BEAM-9354:
Summary: How long does PubSubIO message deduplication last?
Key: BEAM-9354
URL: https://issues.apache.org/jira/browse/BEAM-9354
Project: Beam
Issue Type: Improvement
Components: io-java-gcp
Reporter: Tianzi Cai
GCP documentation heavily
[promotes|https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub]
Beam's PubSubIO for Pub/Sub message deduplication. Yet nowhere in the
documentation, including the [source
code|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java],
tells users how long this deduplication is supposed to last.
In
[`PubsubIO.java`|https://github.com/apache/beam/blob/a24bc3bae54f089b93bd66a118bd4bf09dbc9254/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L842-L853]:
{code:java}
/**
* When reading from Cloud Pub/Sub where unique record identifiers are
provided as Pub/Sub
* message attributes, specifies the name of the attribute containing the
unique identifier. The
* value of the attribute can be any string that uniquely identifies this
record.
*
* Pub/Sub cannot guarantee that no duplicate data will be delivered on
the Pub/Sub stream.
* If {@code idAttribute} is not provided, Beam cannot guarantee that no
duplicate data will be
* delivered, and deduplication of the stream will be strictly best effort.
*/
public Read withIdAttribute(String idAttribute) {
return toBuilder().setIdAttribute(idAttribute).build();
}
{code}
This information here isn't enough for users to know if a second message,
published with the same custom IdAttribute as that of a first message, which
was published `x` minutes ago, would be deduplicated by the Dataflow runner.
Better documentation will help. I imagine a lot of users will wonder about this
and may even ask how to configure this period, but that will probably need a
separate ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042106#comment-17042106
]
Tomo Suzuki commented on BEAM-9252:
---
The jboss's Main class seems incorrect.
{noformat}
suztomo@suxtomo24:~/jbonofre-beam-samples$ javap -verbose -cp
/usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
...
LocalVariableTable:
Start Length Slot Name Signature
562 6497415 cnt I
588 2515 e Ljava/lang/IndexOutOfBoundsException;
52 56114 arg Ljava/lang/String;
36 6550012 i I
39 6549713 argsLength I
717 313 environmentLoader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleLoader;
749 6478712 loader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleLoader;
759 6477714 moduleIdentifier
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleIdentifier;
0 012 loader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleLoader;
0 014 moduleIdentifier
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleIdentifier;
0 020 i I
945 6459120 old Ljava/lang/ClassLoader;
0104517 reader Ljava/io/BufferedReader;
864 18118 name Ljava/lang/String;
872 17319 line Ljava/lang/String;
0 017 ignored Ljava/io/IOException;
1070 023 ignored Ljava/io/IOException;
0 015 classLoader
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleClassLoader;
0 016 stream Ljava/io/InputStream;
0 016 e
Lorg/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/ModuleNotFoundException;
0114916 e
Ljava/lang/reflect/InvocationTargetException;
0 0 0 args [Ljava/lang/String;
3 65533 1 argsLen I
5 65531 2 deps Ljava/lang/String;
9 65527 3 moduleArgs [Ljava/lang/String;
12 65524 4 modulePath Ljava/lang/String;
15 65521 5 configPath Ljava/lang/String;
18 65518 6 classpath Ljava/lang/S
{noformat}
{noformat}
suztomo@suxtomo24:~/jbonofre-beam-samples$ java -cp
/usr/local/google/home/suztomo/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.ClassFormatError: Invalid length 64974 in
LocalVariableTable in class file
org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
{noformat}
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
>
[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=390833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390833 ] ASF GitHub Bot logged work on BEAM-5605: Author: ASF GitHub Bot Created on: 21/Feb/20 18:26 Start Date: 21/Feb/20 18:26 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10920: [BEAM-5605] Eagerly close the BoundedReader once we have read everything or have failed. URL: https://github.com/apache/beam/pull/10920 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390833) Time Spent: 17h 20m (was: 17h 10m) > Support Portable SplittableDoFn for batch > - > > Key: BEAM-5605 > URL: https://issues.apache.org/jira/browse/BEAM-5605 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Assignee: Luke Cwik >Priority: Major > Labels: portability > Time Spent: 17h 20m > Remaining Estimate: 0h > > Roll-up item tracking work towards supporting portable SplittableDoFn for > batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042088#comment-17042088
]
Tomo Suzuki edited comment on BEAM-9252 at 2/21/20 6:22 PM:
Looking into this.
!image-2020-02-21-13-22-01-341.png!
was (Author: suztomo):
Looking into this.
!ArrayIndexOutOfBoundsException.png.png!
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png,
> image-2020-02-21-13-22-01-341.png
>
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomo Suzuki updated BEAM-9252:
--
Attachment: ArrayIndexOutOfBoundsException.png
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Priority: Critical
> Fix For: 2.20.0
>
> Attachments: ArrayIndexOutOfBoundsException.png
>
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
