[jira] [Work logged] (BEAM-6404) FnAPI translation error
[
https://issues.apache.org/jira/browse/BEAM-6404?focusedWorklogId=187291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187291
]
ASF GitHub Bot logged work on BEAM-6404:
Author: ASF GitHub Bot
Created on: 19/Jan/19 02:12
Start Date: 19/Jan/19 02:12
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #7456: [BEAM-6404] Fix
issue with side inputs and flatten encoding.
URL: https://github.com/apache/beam/pull/7456#issuecomment-455740438
The PR break the build
https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/133/console
1st commit "Cleanup coder windowing." seems to be the cause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187291)
Time Spent: 1h (was: 50m)
> FnAPI translation error
> ---
>
> Key: BEAM-6404
> URL: https://issues.apache.org/jira/browse/BEAM-6404
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {code:java}
> def run(argv=None):
> parser = argparse.ArgumentParser()
> _, pipeline_args = parser.parse_known_args(argv)
> options = pipeline_options.PipelineOptions(pipeline_args)
> numbers = [1, 2]
> with beam.Pipeline(options=options) as p:
> sum_1 = (p
> | 'ReadNumber1' >> transforms.Create(numbers)
> | 'CalculateSum1' >> beam.CombineGlobally(fn_sum))
> sum_2 = (p
> | 'ReadNumber2' >> transforms.Create(numbers)
> | beam.ParDo(_copy_number, pvalue.AsSingleton(sum_1))
> | 'CalculateSum2' >> beam.CombineGlobally(fn_sum))
> _ = ((sum_1, sum_2)
> | beam.Flatten()
> | 'CalculateSum3' >> beam.CombineGlobally(fn_sum)
> | beam.io.WriteToText('out.txt'))
> run()
> {code}
>
> fails with
> KeyError: u'ref_Coder_FastPrimitivesCoder_4_windowed'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6439) Move Python Flink VR tests to PreCommit
[ https://issues.apache.org/jira/browse/BEAM-6439?focusedWorklogId=187296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187296 ] ASF GitHub Bot logged work on BEAM-6439: Author: ASF GitHub Bot Created on: 19/Jan/19 02:19 Start Date: 19/Jan/19 02:19 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7539: [BEAM-6439] Move Python Validates Runner Flink test to PreCommit URL: https://github.com/apache/beam/pull/7539#issuecomment-455740993 Ping: @tweise @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187296) Time Spent: 3h 50m (was: 3h 40m) > Move Python Flink VR tests to PreCommit > --- > > Key: BEAM-6439 > URL: https://issues.apache.org/jira/browse/BEAM-6439 > Project: Beam > Issue Type: New Feature > Components: runner-flink, testing >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: Not applicable > > Attachments: png.png > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Now that they're fast and stable, it would be good to catch changes that > break this earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6470) python tenacity is breaks the Python validates runner flink tests
[ https://issues.apache.org/jira/browse/BEAM-6470?focusedWorklogId=187293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187293 ] ASF GitHub Bot logged work on BEAM-6470: Author: ASF GitHub Bot Created on: 19/Jan/19 02:17 Start Date: 19/Jan/19 02:17 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7577: [BEAM-6470] Remove dependency on tenacity URL: https://github.com/apache/beam/pull/7577#issuecomment-455740882 Python Flink ValidatesRunner is failing because of https://github.com/apache/beam/pull/7456 We can get this in to fix at least one of the issue with the validates runner tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187293) Time Spent: 0.5h (was: 20m) > python tenacity is breaks the Python validates runner flink tests > - > > Key: BEAM-6470 > URL: https://issues.apache.org/jira/browse/BEAM-6470 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Python tenacity is only added for test cases. > Beam VR tests executes actual pipelines which do not have tenacity. > We already have apache_beam.util.retry which has the retry functionality so > replacing tenacity with it will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6471) Use support installation to python virtual environment
[ https://issues.apache.org/jira/browse/BEAM-6471?focusedWorklogId=187295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187295 ] ASF GitHub Bot logged work on BEAM-6471: Author: ASF GitHub Bot Created on: 19/Jan/19 02:18 Start Date: 19/Jan/19 02:18 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7572: [BEAM-6471] Make pip location configurable URL: https://github.com/apache/beam/pull/7572 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187295) Time Spent: 20m (was: 10m) > Use support installation to python virtual environment > --- > > Key: BEAM-6471 > URL: https://issues.apache.org/jira/browse/BEAM-6471 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > To use piputil form within a python virtual environment, we need to use pip > from python virtual environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6470) python tenacity is breaks the Python validates runner flink tests
[ https://issues.apache.org/jira/browse/BEAM-6470?focusedWorklogId=187294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187294 ] ASF GitHub Bot logged work on BEAM-6470: Author: ASF GitHub Bot Created on: 19/Jan/19 02:18 Start Date: 19/Jan/19 02:18 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7577: [BEAM-6470] Remove dependency on tenacity URL: https://github.com/apache/beam/pull/7577#issuecomment-455740914 cc: @robertwb @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187294) Time Spent: 40m (was: 0.5h) > python tenacity is breaks the Python validates runner flink tests > - > > Key: BEAM-6470 > URL: https://issues.apache.org/jira/browse/BEAM-6470 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Python tenacity is only added for test cases. > Beam VR tests executes actual pipelines which do not have tenacity. > We already have apache_beam.util.retry which has the retry functionality so > replacing tenacity with it will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6354) Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and SplittableDoFnTest#testLateData
[
https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746906#comment-16746906
]
Ruoyun Huang commented on BEAM-6354:
If I understand it correctly, my input is bounded, thus maybe not relevant? My
code is like this:
final PCollectionView view = pipeline.apply("Create47",
Create.of(47)).apply(View.asSingleton());
I was tracing down the code path based on your suggestion. I am 90% sure that
the reason for empty output is trigger not filed (evidence being, onTrigger()
function never been executed).
> Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and
> SplittableDoFnTest#testLateData
>
>
> Key: BEAM-6354
> URL: https://issues.apache.org/jira/browse/BEAM-6354
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.10.0
>
>
> It seems that they have a similar root cause because both of them use
> unbounded streams.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6404) FnAPI translation error
[
https://issues.apache.org/jira/browse/BEAM-6404?focusedWorklogId=187292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187292
]
ASF GitHub Bot logged work on BEAM-6404:
Author: ASF GitHub Bot
Created on: 19/Jan/19 02:12
Start Date: 19/Jan/19 02:12
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #7456: [BEAM-6404] Fix
issue with side inputs and flatten encoding.
URL: https://github.com/apache/beam/pull/7456#issuecomment-455740438
The PR break the build
https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/133/console
2nd commit "[BEAM-6404] Fix issue with side inputs and flatten encoding.."
seems to be the cause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187292)
Time Spent: 1h 10m (was: 1h)
> FnAPI translation error
> ---
>
> Key: BEAM-6404
> URL: https://issues.apache.org/jira/browse/BEAM-6404
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code:java}
> def run(argv=None):
> parser = argparse.ArgumentParser()
> _, pipeline_args = parser.parse_known_args(argv)
> options = pipeline_options.PipelineOptions(pipeline_args)
> numbers = [1, 2]
> with beam.Pipeline(options=options) as p:
> sum_1 = (p
> | 'ReadNumber1' >> transforms.Create(numbers)
> | 'CalculateSum1' >> beam.CombineGlobally(fn_sum))
> sum_2 = (p
> | 'ReadNumber2' >> transforms.Create(numbers)
> | beam.ParDo(_copy_number, pvalue.AsSingleton(sum_1))
> | 'CalculateSum2' >> beam.CombineGlobally(fn_sum))
> _ = ((sum_1, sum_2)
> | beam.Flatten()
> | 'CalculateSum3' >> beam.CombineGlobally(fn_sum)
> | beam.io.WriteToText('out.txt'))
> run()
> {code}
>
> fails with
> KeyError: u'ref_Coder_FastPrimitivesCoder_4_windowed'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6472) Artifact staging permissions are not set while staging python artifacts
[ https://issues.apache.org/jira/browse/BEAM-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746886#comment-16746886 ] Ankur Goenka commented on BEAM-6472: https://github.com/apache/beam/pull/7573 > Artifact staging permissions are not set while staging python artifacts > --- > > Key: BEAM-6472 > URL: https://issues.apache.org/jira/browse/BEAM-6472 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > > We need to pass the read permissions for the artifacts so that the artifacts > can be read by the pip and installed. > Inside docker this is not an issue as docker runs are root. However when > using other types of environment which do not run as root, artifact access > will not be possible without the right permissions. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6472) Artifact staging permissions are not set while staging python artifacts
Ankur Goenka created BEAM-6472: -- Summary: Artifact staging permissions are not set while staging python artifacts Key: BEAM-6472 URL: https://issues.apache.org/jira/browse/BEAM-6472 Project: Beam Issue Type: Bug Components: sdk-py-harness Reporter: Ankur Goenka Assignee: Ankur Goenka We need to pass the read permissions for the artifacts so that the artifacts can be read by the pip and installed. Inside docker this is not an issue as docker runs are root. However when using other types of environment which do not run as root, artifact access will not be possible without the right permissions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6471) Use support installation to python virtual environment
[ https://issues.apache.org/jira/browse/BEAM-6471?focusedWorklogId=187277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187277 ] ASF GitHub Bot logged work on BEAM-6471: Author: ASF GitHub Bot Created on: 19/Jan/19 01:20 Start Date: 19/Jan/19 01:20 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7572: [BEAM-6471] Make pip location configurable URL: https://github.com/apache/beam/pull/7572#issuecomment-455735733 Thanks lostluck. Updated the commit wording to add jira number. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187277) Time Spent: 10m Remaining Estimate: 0h > Use support installation to python virtual environment > --- > > Key: BEAM-6471 > URL: https://issues.apache.org/jira/browse/BEAM-6471 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > To use piputil form within a python virtual environment, we need to use pip > from python virtual environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6470) python tenacity is breaks the Python validates runner flink tests
[ https://issues.apache.org/jira/browse/BEAM-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746870#comment-16746870 ] Ankur Goenka commented on BEAM-6470: Failing test [https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/33/] > python tenacity is breaks the Python validates runner flink tests > - > > Key: BEAM-6470 > URL: https://issues.apache.org/jira/browse/BEAM-6470 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > > Python tenacity is only added for test cases. > Beam VR tests executes actual pipelines which do not have tenacity. > We already have apache_beam.util.retry which has the retry functionality so > replacing tenacity with it will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187259
]
ASF GitHub Bot logged work on BEAM-6469:
Author: ASF GitHub Bot
Created on: 19/Jan/19 01:09
Start Date: 19/Jan/19 01:09
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #7576: [BEAM-6469] Install
Beam test dependencies in Flink compatibility tests
URL: https://github.com/apache/beam/pull/7576#issuecomment-455734543
Tenacity is now installed, but there is another unrelated failure in the
logs.
```
Successfully installed ... tenacity-5.0.2 ...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187259)
Time Spent: 50m (was: 40m)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6471) Use support installation to python virtual environment
Ankur Goenka created BEAM-6471: -- Summary: Use support installation to python virtual environment Key: BEAM-6471 URL: https://issues.apache.org/jira/browse/BEAM-6471 Project: Beam Issue Type: Bug Components: sdk-py-harness Reporter: Ankur Goenka Assignee: Ankur Goenka To use piputil form within a python virtual environment, we need to use pip from python virtual environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187269
]
ASF GitHub Bot logged work on BEAM-6469:
Author: ASF GitHub Bot
Created on: 19/Jan/19 01:15
Start Date: 19/Jan/19 01:15
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #7576: [BEAM-6469] Install
Beam test dependencies in Flink compatibility tests
URL: https://github.com/apache/beam/pull/7576#issuecomment-455734543
Tenacity is now installed:
```
Successfully installed ... tenacity-5.0.2 ...
```
However there is another unrelated failure in the logs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187269)
Time Spent: 1h 10m (was: 1h)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187268
]
ASF GitHub Bot logged work on BEAM-6469:
Author: ASF GitHub Bot
Created on: 19/Jan/19 01:14
Start Date: 19/Jan/19 01:14
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #7576: [BEAM-6469] Install
Beam test dependencies in Flink compatibility tests
URL: https://github.com/apache/beam/pull/7576#issuecomment-455735097
Run Python Flink ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187268)
Time Spent: 1h (was: 50m)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746873#comment-16746873
]
Valentyn Tymofieiev commented on BEAM-6469:
---
PR7576 addressed the missing tenacity installation, the test suite however
failed with:
==
ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):(
File "apache_beam/runners/portability/fn_api_runner_test.py", line 205, in
test_flattened_side_input
label='CheckFlattenOfSideInput')
File "apache_beam/pipeline.py", line 425, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 349, in
wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
test_flattened_side_input_1547859357.36_07dcde9b-acfc-4e8d-b930-582f7637a07e
failed in state FAILED: java.lang.IllegalArgumentException: PCollectionNodes
[PCollectionNode{id=ref_PCollection_PCollection_12, PCollection=unique_name:
"17side3/Map(decode).None"
coder_id: "ref_Coder_BytesCoder_1"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}] were consumed but never produced
--
possibly a flake.
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6470) python tenacity is breaks the Python validates runner flink tests
[ https://issues.apache.org/jira/browse/BEAM-6470?focusedWorklogId=187261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187261 ] ASF GitHub Bot logged work on BEAM-6470: Author: ASF GitHub Bot Created on: 19/Jan/19 01:09 Start Date: 19/Jan/19 01:09 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7577: [BEAM-6470] Remove dependency on tenacity URL: https://github.com/apache/beam/pull/7577#issuecomment-455734643 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187261) Time Spent: 20m (was: 10m) > python tenacity is breaks the Python validates runner flink tests > - > > Key: BEAM-6470 > URL: https://issues.apache.org/jira/browse/BEAM-6470 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Python tenacity is only added for test cases. > Beam VR tests executes actual pipelines which do not have tenacity. > We already have apache_beam.util.retry which has the retry functionality so > replacing tenacity with it will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6470) python tenacity is breaks the Python validates runner flink tests
Ankur Goenka created BEAM-6470: -- Summary: python tenacity is breaks the Python validates runner flink tests Key: BEAM-6470 URL: https://issues.apache.org/jira/browse/BEAM-6470 Project: Beam Issue Type: Bug Components: test-failures Reporter: Ankur Goenka Assignee: Ankur Goenka Python tenacity is only added for test cases. Beam VR tests executes actual pipelines which do not have tenacity. We already have apache_beam.util.retry which has the retry functionality so replacing tenacity with it will solve this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6470) python tenacity is breaks the Python validates runner flink tests
[ https://issues.apache.org/jira/browse/BEAM-6470?focusedWorklogId=187260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187260 ] ASF GitHub Bot logged work on BEAM-6470: Author: ASF GitHub Bot Created on: 19/Jan/19 01:09 Start Date: 19/Jan/19 01:09 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7577: [BEAM-6470] Remove dependency on tenacity URL: https://github.com/apache/beam/pull/7577 **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187260) Time Spent: 10m Remaining Estimate: 0h > python tenacity is breaks the Python validates
[jira] [Work logged] (BEAM-6237) ULR ValidatesRunner tests not deleting artifacts.
[ https://issues.apache.org/jira/browse/BEAM-6237?focusedWorklogId=187258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187258 ] ASF GitHub Bot logged work on BEAM-6237: Author: ASF GitHub Bot Created on: 19/Jan/19 01:07 Start Date: 19/Jan/19 01:07 Worklog Time Spent: 10m Work Description: youngoli commented on issue #7571: [BEAM-6237] Fix ULR not deleting artifacts after running jobs. URL: https://github.com/apache/beam/pull/7571#issuecomment-455734393 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187258) Time Spent: 40m (was: 0.5h) > ULR ValidatesRunner tests not deleting artifacts. > - > > Key: BEAM-6237 > URL: https://issues.apache.org/jira/browse/BEAM-6237 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > When running ValidatesRunner tests with the ULR, artifacts are never deleted. > Since a new job is run per test, this uses up massive amounts of disk storage > quickly (over 20 Gigabytes per execution). This often causes the machine > running these tests to run out of disk space which means tests start failing. > The ULR should be modified to delete these artifacts after they have been > staged to avoid this issue. Flink already does this, so the infrastructure > exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187244
]
ASF GitHub Bot logged work on BEAM-6469:
Author: ASF GitHub Bot
Created on: 19/Jan/19 00:40
Start Date: 19/Jan/19 00:40
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7576: [BEAM-6469] Install Beam
test dependencies in Flink compatibility tests
URL: https://github.com/apache/beam/pull/7576#issuecomment-455731154
Thanks for investigating. Will merge if tests pass.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187244)
Time Spent: 40m (was: 0.5h)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=187247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187247 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 19/Jan/19 00:40 Start Date: 19/Jan/19 00:40 Worklog Time Spent: 10m Work Description: mxm commented on issue #7574: [BEAM-6440] Ensure state backend key is set for timer deduplication URL: https://github.com/apache/beam/pull/7574#issuecomment-455731218 Python ValidatesRunner tests currently broken: https://issues.apache.org/jira/browse/BEAM-6469 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187247) Time Spent: 3h 10m (was: 3h) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187240 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 00:38 Start Date: 19/Jan/19 00:38 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455730888 PreCommit failing due to #7575. https://builds.apache.org/job/beam_PreCommit_Java_Commit/3648/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187240) Time Spent: 8h 20m (was: 8h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187241=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187241 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 00:38 Start Date: 19/Jan/19 00:38 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455730934 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187241) Time Spent: 8.5h (was: 8h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187239 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 00:38 Start Date: 19/Jan/19 00:38 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455730888 PreCommit failing due to #7575. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187239) Time Spent: 8h 10m (was: 8h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187236
]
ASF GitHub Bot logged work on BEAM-6469:
Author: ASF GitHub Bot
Created on: 19/Jan/19 00:37
Start Date: 19/Jan/19 00:37
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #7576: [BEAM-6469] Install
Beam test dependencies in Flink compatibility tests
URL: https://github.com/apache/beam/pull/7576#issuecomment-455730686
@mxm @aaltay I am hoping this will fix the Flink suite. Looks like Jenkins
is a little slow, but if you see that the tests passed - free to merge the PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187236)
Time Spent: 0.5h (was: 20m)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746860#comment-16746860
]
Maximilian Michels commented on BEAM-6469:
--
Thanks for investigating!
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746850#comment-16746850
]
Valentyn Tymofieiev commented on BEAM-6469:
---
Hoping that [https://github.com/apache/beam/pull/7576] will address this,
waiting for tests result.
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187221 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 19/Jan/19 00:25 Start Date: 19/Jan/19 00:25 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568#issuecomment-455729188 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187221) Time Spent: 1h 10m (was: 1h) > Cannot create empty TestBoundedTable > > > Key: BEAM-6468 > URL: https://issues.apache.org/jira/browse/BEAM-6468 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187220 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 19/Jan/19 00:25 Start Date: 19/Jan/19 00:25 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568#issuecomment-455729185 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187220) Time Spent: 1h (was: 50m) > Cannot create empty TestBoundedTable > > > Key: BEAM-6468 > URL: https://issues.apache.org/jira/browse/BEAM-6468 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187219
]
ASF GitHub Bot logged work on BEAM-6469:
Author: ASF GitHub Bot
Created on: 19/Jan/19 00:25
Start Date: 19/Jan/19 00:25
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #7576: [BEAM-6469] Install
Beam test dependencies in Flink compatibility tests
URL: https://github.com/apache/beam/pull/7576#issuecomment-455729177
Run Python Flink ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187219)
Time Spent: 20m (was: 10m)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5446) SplittableDoFn: Remove runner time execution information from public API surface
[ https://issues.apache.org/jira/browse/BEAM-5446?focusedWorklogId=187208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187208 ] ASF GitHub Bot logged work on BEAM-5446: Author: ASF GitHub Bot Created on: 19/Jan/19 00:13 Start Date: 19/Jan/19 00:13 Worklog Time Spent: 10m Work Description: swegner commented on issue #6467: [BEAM-5446] SplittableDoFn: Remove "internal" methods for public API surface URL: https://github.com/apache/beam/pull/6467#issuecomment-455727421 FYI, this was reverted in #7540. Details are in [[BEAM-6352]](https://issues.apache.org/jira/browse/BEAM-6352). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187208) Time Spent: 2h 20m (was: 2h 10m) > SplittableDoFn: Remove runner time execution information from public API > surface > > > Key: BEAM-5446 > URL: https://issues.apache.org/jira/browse/BEAM-5446 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Scott Wegner >Priority: Minor > Fix For: 2.9.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Move the setting of "claim observers" within RestrictionTracker to another > location to clean up the RestrictionTracker interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-6050) SplittableDoFnTest is failing for many runners
[ https://issues.apache.org/jira/browse/BEAM-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Wegner closed BEAM-6050. -- Resolution: Fixed The tests are no longer failing because the RestrictionTracker API change was also rolled-back. > SplittableDoFnTest is failing for many runners > -- > > Key: BEAM-6050 > URL: https://issues.apache.org/jira/browse/BEAM-6050 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Luke Cwik >Priority: Critical > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > > For example, > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/2190/#showFailuresLink] > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/2180/#showFailuresLink] > > Seems to be due to following commit. > [https://github.com/apache/beam/commit/413d8524f58604a3062a8eea07c1dcd1301e2f83] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentyn Tymofieiev updated BEAM-6469:
--
Summary: Python Flink ValidatesRunner tests fail due to missing module
(was: Python ValidatesRunner tests fail due to missing module)
> Python Flink ValidatesRunner tests fail due to missing module
> -
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6469) Python Flink ValidatesRunner tests fail due to missing module
[ https://issues.apache.org/jira/browse/BEAM-6469?focusedWorklogId=187216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187216 ] ASF GitHub Bot logged work on BEAM-6469: Author: ASF GitHub Bot Created on: 19/Jan/19 00:21 Start Date: 19/Jan/19 00:21 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #7576: [BEAM-6469] Install Beam test dependencies in Flink compatibility tests URL: https://github.com/apache/beam/pull/7576 **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187216) Time Spent: 10m Remaining Estimate: 0h > Python Flink
[jira] [Work logged] (BEAM-6352) Watch PTransform is broken
[
https://issues.apache.org/jira/browse/BEAM-6352?focusedWorklogId=187206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187206
]
ASF GitHub Bot logged work on BEAM-6352:
Author: ASF GitHub Bot
Created on: 19/Jan/19 00:10
Start Date: 19/Jan/19 00:10
Worklog Time Spent: 10m
Work Description: swegner commented on issue #7575: [BEAM-6352] Revert
PR#6467 to fix Watch transform
URL: https://github.com/apache/beam/pull/7575#issuecomment-455726991
R: @kennknowles
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187206)
Time Spent: 1.5h (was: 1h 20m)
> Watch PTransform is broken
> --
>
> Key: BEAM-6352
> URL: https://issues.apache.org/jira/browse/BEAM-6352
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.9.0
>Reporter: Gleb Kanterov
>Assignee: Scott Wegner
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> List of affected tests:
> org.apache.beam.sdk.transforms.WatchTest >
> testSinglePollMultipleInputsWithSideInput FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithKeyExtractor
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationDueToTerminationCondition FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsStopAfterTimeSinceNewOutput
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > testReadWatchForNewFiles
> FAILED
> {code}
> java.lang.IllegalArgumentException:
> org.apache.beam.sdk.transforms.Watch$WatchGrowthFn, @ProcessElement
> process(ProcessContext, GrowthTracker): Has tracker type
> Watch.GrowthTracker, but the DoFn's tracker
> type must be of type RestrictionTracker.
> {code}
> Relevant pull requests:
> - https://github.com/apache/beam/pull/6467
> - https://github.com/apache/beam/pull/7374
> Now tests are marked with @Ignore referencing this JIRA issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=187203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187203 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 19/Jan/19 00:08 Start Date: 19/Jan/19 00:08 Worklog Time Spent: 10m Work Description: mxm commented on issue #7574: [BEAM-6440] Ensure state backend key is set for timer deduplication URL: https://github.com/apache/beam/pull/7574#issuecomment-455726743 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187203) Time Spent: 3h (was: 2h 50m) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Attachments: image-2019-01-15-10-31-43-618.png, > image-2019-01-15-10-41-08-200.png > > Time Spent: 3h > Remaining Estimate: 0h > > After running a portable streaming app for 2-3 days, we see heap space > exhausted. Memory analysis shows large number of TimerData objects referenced > by heap state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6352) Watch PTransform is broken
[
https://issues.apache.org/jira/browse/BEAM-6352?focusedWorklogId=187209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187209
]
ASF GitHub Bot logged work on BEAM-6352:
Author: ASF GitHub Bot
Created on: 19/Jan/19 00:15
Start Date: 19/Jan/19 00:15
Worklog Time Spent: 10m
Work Description: swegner commented on issue #7540: [BEAM-6352] Revert
PR#6467 to fix Watch transform
URL: https://github.com/apache/beam/pull/7540#issuecomment-455727691
Cherry-pick: #7575
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187209)
Time Spent: 1h 40m (was: 1.5h)
> Watch PTransform is broken
> --
>
> Key: BEAM-6352
> URL: https://issues.apache.org/jira/browse/BEAM-6352
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.9.0
>Reporter: Gleb Kanterov
>Assignee: Scott Wegner
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> List of affected tests:
> org.apache.beam.sdk.transforms.WatchTest >
> testSinglePollMultipleInputsWithSideInput FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithKeyExtractor
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationDueToTerminationCondition FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsStopAfterTimeSinceNewOutput
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > testReadWatchForNewFiles
> FAILED
> {code}
> java.lang.IllegalArgumentException:
> org.apache.beam.sdk.transforms.Watch$WatchGrowthFn, @ProcessElement
> process(ProcessContext, GrowthTracker): Has tracker type
> Watch.GrowthTracker, but the DoFn's tracker
> type must be of type RestrictionTracker.
> {code}
> Relevant pull requests:
> - https://github.com/apache/beam/pull/6467
> - https://github.com/apache/beam/pull/7374
> Now tests are marked with @Ignore referencing this JIRA issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (BEAM-6050) SplittableDoFnTest is failing for many runners
[ https://issues.apache.org/jira/browse/BEAM-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Wegner reopened BEAM-6050: This was reverted in PR#7540. Details are in [BEAM-6352] > SplittableDoFnTest is failing for many runners > -- > > Key: BEAM-6050 > URL: https://issues.apache.org/jira/browse/BEAM-6050 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Luke Cwik >Priority: Critical > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > > For example, > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/2190/#showFailuresLink] > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/2180/#showFailuresLink] > > Seems to be due to following commit. > [https://github.com/apache/beam/commit/413d8524f58604a3062a8eea07c1dcd1301e2f83] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6050) SplittableDoFnTest is failing for many runners
[ https://issues.apache.org/jira/browse/BEAM-6050?focusedWorklogId=187207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187207 ] ASF GitHub Bot logged work on BEAM-6050: Author: ASF GitHub Bot Created on: 19/Jan/19 00:13 Start Date: 19/Jan/19 00:13 Worklog Time Spent: 10m Work Description: swegner commented on issue #7016: [BEAM-6050] Use correct type on @ProcessElement method for SplittableDoFns URL: https://github.com/apache/beam/pull/7016#issuecomment-455727382 FYI, this was reverted in #7540. Details are in [[BEAM-6352]](https://issues.apache.org/jira/browse/BEAM-6352). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187207) Time Spent: 1h 20m (was: 1h 10m) > SplittableDoFnTest is failing for many runners > -- > > Key: BEAM-6050 > URL: https://issues.apache.org/jira/browse/BEAM-6050 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Chamikara Jayalath >Assignee: Luke Cwik >Priority: Critical > Fix For: Not applicable > > Time Spent: 1h 20m > Remaining Estimate: 0h > > For example, > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/2190/#showFailuresLink] > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/2180/#showFailuresLink] > > Seems to be due to following commit. > [https://github.com/apache/beam/commit/413d8524f58604a3062a8eea07c1dcd1301e2f83] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6352) Watch PTransform is broken
[ https://issues.apache.org/jira/browse/BEAM-6352?focusedWorklogId=187202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187202 ] ASF GitHub Bot logged work on BEAM-6352: Author: ASF GitHub Bot Created on: 19/Jan/19 00:08 Start Date: 19/Jan/19 00:08 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7575: [BEAM-6352] Revert PR#6467 to fix Watch transform URL: https://github.com/apache/beam/pull/7575 Cherry-pick of #7540 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187202) Time Spent: 1h 20m (was: 1h 10m) > Watch PTransform is broken > -- > > Key: BEAM-6352 >
[jira] [Work logged] (BEAM-6440) FlinkTimerInternals memory leak
[ https://issues.apache.org/jira/browse/BEAM-6440?focusedWorklogId=187201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187201 ] ASF GitHub Bot logged work on BEAM-6440: Author: ASF GitHub Bot Created on: 19/Jan/19 00:08 Start Date: 19/Jan/19 00:08 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7574: [BEAM-6440] Ensure state backend key is set for timer deduplication URL: https://github.com/apache/beam/pull/7574 During working on the Savepoint restore test a problem showed up for portable pipelines not having the state backend key set before cleaning up the pending timers map. This moves the cleanup to before firing the timer when the key is ensured to be set. CC @tweise Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187201) Time Spent: 2h 50m (was: 2h 40m) > FlinkTimerInternals memory leak > --- > > Key: BEAM-6440 > URL: https://issues.apache.org/jira/browse/BEAM-6440 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.9.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > >
[jira] [Work logged] (BEAM-6237) ULR ValidatesRunner tests not deleting artifacts.
[ https://issues.apache.org/jira/browse/BEAM-6237?focusedWorklogId=187195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187195 ] ASF GitHub Bot logged work on BEAM-6237: Author: ASF GitHub Bot Created on: 19/Jan/19 00:02 Start Date: 19/Jan/19 00:02 Worklog Time Spent: 10m Work Description: lhaiesp commented on issue #7571: [BEAM-6237] Fix ULR not deleting artifacts after running jobs. URL: https://github.com/apache/beam/pull/7571#issuecomment-455725834 I'm running into the same failure from java precommit (org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles). Seems pretty recent. Is anyone looking into that? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187195) Time Spent: 0.5h (was: 20m) > ULR ValidatesRunner tests not deleting artifacts. > - > > Key: BEAM-6237 > URL: https://issues.apache.org/jira/browse/BEAM-6237 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When running ValidatesRunner tests with the ULR, artifacts are never deleted. > Since a new job is run per test, this uses up massive amounts of disk storage > quickly (over 20 Gigabytes per execution). This often causes the machine > running these tests to run out of disk space which means tests start failing. > The ULR should be modified to delete these artifacts after they have been > staged to avoid this issue. Flink already does this, so the infrastructure > exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6237) ULR ValidatesRunner tests not deleting artifacts.
[ https://issues.apache.org/jira/browse/BEAM-6237?focusedWorklogId=187184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187184 ] ASF GitHub Bot logged work on BEAM-6237: Author: ASF GitHub Bot Created on: 18/Jan/19 23:51 Start Date: 18/Jan/19 23:51 Worklog Time Spent: 10m Work Description: youngoli commented on issue #7571: [BEAM-6237] Fix ULR not deleting artifacts after running jobs. URL: https://github.com/apache/beam/pull/7571#issuecomment-455724012 The Java PreCommit failure seems unrelated to this PR so I'll send it out for review regardless. R: @HuangLED This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187184) Time Spent: 20m (was: 10m) > ULR ValidatesRunner tests not deleting artifacts. > - > > Key: BEAM-6237 > URL: https://issues.apache.org/jira/browse/BEAM-6237 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > When running ValidatesRunner tests with the ULR, artifacts are never deleted. > Since a new job is run per test, this uses up massive amounts of disk storage > quickly (over 20 Gigabytes per execution). This often causes the machine > running these tests to run out of disk space which means tests start failing. > The ULR should be modified to delete these artifacts after they have been > staged to avoid this issue. Flink already does this, so the infrastructure > exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6469) Python ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746832#comment-16746832
]
Valentyn Tymofieiev commented on BEAM-6469:
---
Some observations.
Tenacity was added as a Beam Dependency in
[https://github.com/apache/beam/pull/7492,] and this didn't break neither Flink
VR suite nor Dataflow VR suite.
[https://github.com/apache/beam/pull/7455/] deflaked a fn_api_runner test with
tenacity, and it broke Flink VR suite only.
I suspect Flink VR suite may not be installing test dependencies of the SDK
(which include tenacity:
https://github.com/apache/beam/blob/56babf083635b5f6a0cf8a50f93efe1037f3d6bc/sdks/python/setup.py#L140),
looking further to confirm.
> Python ValidatesRunner tests fail due to missing module
> ---
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-6469) Python ValidatesRunner tests fail due to missing module
[
https://issues.apache.org/jira/browse/BEAM-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-6469:
-
Assignee: Valentyn Tymofieiev
> Python ValidatesRunner tests fail due to missing module
> ---
>
> Key: BEAM-6469
> URL: https://issues.apache.org/jira/browse/BEAM-6469
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.10.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
> {noformat}
> 17:50:38 Traceback (most recent call last):
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 17:50:38 "__main__", fname, loader, pkg_name)
> 17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 17:50:38 exec code in run_globals
> 17:50:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
> line 37, in
> 17:50:38 from apache_beam.runners.portability import portable_runner_test
> 17:50:38 File "apache_beam/runners/portability/portable_runner_test.py",
> line 42, in
> 17:50:38 from apache_beam.runners.portability import fn_api_runner_test
> 17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 29, in
> 17:50:38 from tenacity import retry
> 17:50:38 ImportError: No module named tenacity
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5933) PCollectionViews$SimplePCollectionView.hashCode allocates memory
[ https://issues.apache.org/jira/browse/BEAM-5933?focusedWorklogId=187161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187161 ] ASF GitHub Bot logged work on BEAM-5933: Author: ASF GitHub Bot Created on: 18/Jan/19 23:32 Start Date: 18/Jan/19 23:32 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #6909: BEAM-5933: avoid memory allocation in hashCode call URL: https://github.com/apache/beam/pull/6909#issuecomment-455721006 You may mean that anything performance-related in the DirectRunner is a non-issue. Sometimes it seems that way, and it is true that it is focused on just being a fake for testing. But it is so bad that we really do need to improve it. Please keep reporting issues! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187161) Time Spent: 1h 50m (was: 1h 40m) > PCollectionViews$SimplePCollectionView.hashCode allocates memory > > > Key: BEAM-5933 > URL: https://issues.apache.org/jira/browse/BEAM-5933 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 2.8.0 >Reporter: Vojtech Janota >Assignee: Vojtech Janota >Priority: Trivial > Fix For: 2.9.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > I'm currently profiling memory consumption of our Beam pipeline and have > noticed that > > org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.hashCode() > makes noticeable heap allocations. The implementation is: > return Objects.hash(tag); > That itself translates to: > return Arrays.hashCode(values); > Which performs implicit array creation in order to call: > public static int Arrays.hashCode(Object a[]); > Instead of the helper call, doing simple: > tag.hashCode(); > Seems more appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5933) PCollectionViews$SimplePCollectionView.hashCode allocates memory
[ https://issues.apache.org/jira/browse/BEAM-5933?focusedWorklogId=187159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187159 ] ASF GitHub Bot logged work on BEAM-5933: Author: ASF GitHub Bot Created on: 18/Jan/19 23:31 Start Date: 18/Jan/19 23:31 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #6909: BEAM-5933: avoid memory allocation in hashCode call URL: https://github.com/apache/beam/pull/6909#issuecomment-455720806 @janotav you are quite right that this hidden contract is very suspicious. I have looked into the type hierarchy to investigate. The issue is that there are two desires in conflict: (1) a runner can deserialize a protobuf PCollectionView using just the tag, into whatever its runner-specific representation and (2) you can use PCollectionView as a key to retrieve values. Together, these force any subclass of PCollectionView should be equal (and equal hashcode) if their tags are equal, since runner's create proxy views or whatever. IMO this contract is broken, since the same tag but different `ViewFn` should not ever be equal. But if you want to gain the performance back, I bet you can roll forward and also just change here to match: https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RunnerPCollectionView.java#L108 Even better would be to port things to use the tag as the key into any implementation map. There is not even equals and hashcode on these subclasses in the Dataflow worker so I think that implies the tag is used directly: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowPortabilityPCollectionView.java and https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FetchAndFilterStreamingSideInputsOperation.java#L99 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187159) Time Spent: 1h 40m (was: 1.5h) > PCollectionViews$SimplePCollectionView.hashCode allocates memory > > > Key: BEAM-5933 > URL: https://issues.apache.org/jira/browse/BEAM-5933 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 2.8.0 >Reporter: Vojtech Janota >Assignee: Vojtech Janota >Priority: Trivial > Fix For: 2.9.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > I'm currently profiling memory consumption of our Beam pipeline and have > noticed that > > org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.hashCode() > makes noticeable heap allocations. The implementation is: > return Objects.hash(tag); > That itself translates to: > return Arrays.hashCode(values); > Which performs implicit array creation in order to call: > public static int Arrays.hashCode(Object a[]); > Instead of the helper call, doing simple: > tag.hashCode(); > Seems more appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187147 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 23:21 Start Date: 18/Jan/19 23:21 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455719046 Blocked on https://jira.apache.org/jira/browse/BEAM-6469 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187147) Time Spent: 8h (was: 7h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6460) Jackson Cache may hold on to Classloader after pipeline restart
[
https://issues.apache.org/jira/browse/BEAM-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels updated BEAM-6460:
-
Summary: Jackson Cache may hold on to Classloader after pipeline restart
(was: PipelineOptionsFactory may hold on to Classloader after pipeline restart)
> Jackson Cache may hold on to Classloader after pipeline restart
> ---
>
> Key: BEAM-6460
> URL: https://issues.apache.org/jira/browse/BEAM-6460
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.7.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> It looks like Jackson has an internal cache which may continue to hold the
> Flink application classloader through its {{TypeFactory}} class. When the
> pipeline is restarted due to a failure, a new classloader is created which
> can result in too many classes being loaded.
> Reported on the user mailing list:
> https://lists.apache.org/thread.html/e201891684ef3dcffce48d20d1f9be0e19fc2294334362cc7092c0ff@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=187143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187143 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 18/Jan/19 23:17 Start Date: 18/Jan/19 23:17 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #7532: [BEAM-6184]Make checkstyle report error on missing javadocmethod URL: https://github.com/apache/beam/pull/7532#issuecomment-455718323 Also, this PR, when merging, most likely will require an update on the suppresions. Ping me beforehand, I can take care of the manual update. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187143) Time Spent: 9h (was: 8h 50m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 9h > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6460) PipelineOptionsFactory may hold on to Classloader after pipeline restart
[
https://issues.apache.org/jira/browse/BEAM-6460?focusedWorklogId=187141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187141
]
ASF GitHub Bot logged work on BEAM-6460:
Author: ASF GitHub Bot
Created on: 18/Jan/19 23:16
Start Date: 18/Jan/19 23:16
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7552: [BEAM-6460] Remove cached
class references upon start/shutdown
URL: https://github.com/apache/beam/pull/7552#issuecomment-455718167
I have yet to verify this with @djhworld, but from my analysis the reported
problem comes either from an incorrect packaging of the jar or from the use of
Flink's `parent-first` classloader. Both lead to using Flink's internal version
of Jackson, instead of the user-provided one. This can let the classloader leak.
That said, the fix in this PR has been verified to fix the @djhworld's
problem, so I'm leaning towards merging it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187141)
Time Spent: 1h 40m (was: 1.5h)
> PipelineOptionsFactory may hold on to Classloader after pipeline restart
>
>
> Key: BEAM-6460
> URL: https://issues.apache.org/jira/browse/BEAM-6460
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.7.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> It looks like Jackson has an internal cache which may continue to hold the
> Flink application classloader through its {{TypeFactory}} class. When the
> pipeline is restarted due to a failure, a new classloader is created which
> can result in too many classes being loaded.
> Reported on the user mailing list:
> https://lists.apache.org/thread.html/e201891684ef3dcffce48d20d1f9be0e19fc2294334362cc7092c0ff@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=187138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187138 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 18/Jan/19 23:15 Start Date: 18/Jan/19 23:15 Worklog Time Spent: 10m Work Description: HuangLED commented on pull request #7532: [BEAM-6184]Make checkstyle report error on missing javadocmethod URL: https://github.com/apache/beam/pull/7532#discussion_r249212651 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -30,4 +30,65 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Review comment: 3 numbers refer to 3 functions (in the same file) that all violate our check, each number for one distinct function. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187138) Time Spent: 8h 50m (was: 8h 40m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 8h 50m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6457) bigquery.py is too large, and some tools are better moved elsewhere
[ https://issues.apache.org/jira/browse/BEAM-6457?focusedWorklogId=187134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187134 ] ASF GitHub Bot logged work on BEAM-6457: Author: ASF GitHub Bot Created on: 18/Jan/19 23:04 Start Date: 18/Jan/19 23:04 Worklog Time Spent: 10m Work Description: pabloem commented on issue #7542: [BEAM-6457] Refactoring of a few BigQuery classes. URL: https://github.com/apache/beam/pull/7542#issuecomment-455716161 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187134) Time Spent: 1.5h (was: 1h 20m) > bigquery.py is too large, and some tools are better moved elsewhere > --- > > Key: BEAM-6457 > URL: https://issues.apache.org/jira/browse/BEAM-6457 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Need to do a bit of refactoring of that file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6271) initial support for portable api in samza runner
[ https://issues.apache.org/jira/browse/BEAM-6271?focusedWorklogId=187132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187132 ] ASF GitHub Bot logged work on BEAM-6271: Author: ASF GitHub Bot Created on: 18/Jan/19 23:01 Start Date: 18/Jan/19 23:01 Worklog Time Spent: 10m Work Description: lhaiesp commented on issue #7321: [BEAM-6271] SamzaRunner: initial support for portable api in samza runner URL: https://github.com/apache/beam/pull/7321#issuecomment-455715619 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187132) Time Spent: 5h 20m (was: 5h 10m) > initial support for portable api in samza runner > > > Key: BEAM-6271 > URL: https://issues.apache.org/jira/browse/BEAM-6271 > Project: Beam > Issue Type: Task > Components: runner-samza >Reporter: Hai Lu >Assignee: Hai Lu >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Support portable api in Samza runner. > This ticket tracks the initial effort to support portable api in Samza > runner, including job server inside samza runner, config translation for > portable pipeline, transform translation for portable pipeline, refactor of > existing codes to merge logic of portable api and java api as much as > possible, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6237) ULR ValidatesRunner tests not deleting artifacts.
[ https://issues.apache.org/jira/browse/BEAM-6237?focusedWorklogId=187129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187129 ] ASF GitHub Bot logged work on BEAM-6237: Author: ASF GitHub Bot Created on: 18/Jan/19 22:58 Start Date: 18/Jan/19 22:58 Worklog Time Spent: 10m Work Description: youngoli commented on pull request #7571: [BEAM-6237] Fix ULR not deleting artifacts after running jobs. URL: https://github.com/apache/beam/pull/7571 This change switches the ULR from using LocalFileSystemArtifact[Stager/Retrieval]Service to using BeamFileSystemArtifact[Staging/Retrieval]Service which has functionality to remove artifacts after running a job. With this change ValidatesRunner tests no longer leave huge amounts of artifacts when run with the ULR. Other code had to be changed to allow this switch. In particular, the old code would store the path to the staged files after creating the staging service. This code instead stores an artifact staging session token, to keep track of a specific staging session (since the job server may have multiple staging sessions from different jobs). The new code also has changes to correctly pass the artifact retrieval token (passed to the ReferenceRunner as part of a RunJobRequest) to the BeamFileSystemArtifactRetrievalService. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [![Build
[jira] [Work logged] (BEAM-6460) PipelineOptionsFactory may hold on to Classloader after pipeline restart
[
https://issues.apache.org/jira/browse/BEAM-6460?focusedWorklogId=187133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187133
]
ASF GitHub Bot logged work on BEAM-6460:
Author: ASF GitHub Bot
Created on: 18/Jan/19 23:04
Start Date: 18/Jan/19 23:04
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7552: [BEAM-6460]
Remove cached class references upon start/shutdown
URL: https://github.com/apache/beam/pull/7552#issuecomment-455716098
Have you confirmed whether this is a problem? Just let me know on the bug /
removing Fix Version from 2.10.0. I'll consider it blocking still until then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187133)
Time Spent: 1.5h (was: 1h 20m)
> PipelineOptionsFactory may hold on to Classloader after pipeline restart
>
>
> Key: BEAM-6460
> URL: https://issues.apache.org/jira/browse/BEAM-6460
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.7.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> It looks like Jackson has an internal cache which may continue to hold the
> Flink application classloader through its {{TypeFactory}} class. When the
> pipeline is restarted due to a failure, a new classloader is created which
> can result in too many classes being loaded.
> Reported on the user mailing list:
> https://lists.apache.org/thread.html/e201891684ef3dcffce48d20d1f9be0e19fc2294334362cc7092c0ff@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6469) Python ValidatesRunner tests fail due to missing module
Maximilian Michels created BEAM-6469:
Summary: Python ValidatesRunner tests fail due to missing module
Key: BEAM-6469
URL: https://issues.apache.org/jira/browse/BEAM-6469
Project: Beam
Issue Type: Test
Components: sdk-py-core
Reporter: Maximilian Michels
Fix For: 2.10.0
https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/130/console
{noformat}
17:50:38 Traceback (most recent call last):
17:50:38 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
17:50:38 "__main__", fname, loader, pkg_name)
17:50:38 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
17:50:38 exec code in run_globals
17:50:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink_PR/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py",
line 37, in
17:50:38 from apache_beam.runners.portability import portable_runner_test
17:50:38 File "apache_beam/runners/portability/portable_runner_test.py", line
42, in
17:50:38 from apache_beam.runners.portability import fn_api_runner_test
17:50:38 File "apache_beam/runners/portability/fn_api_runner_test.py", line
29, in
17:50:38 from tenacity import retry
17:50:38 ImportError: No module named tenacity
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187128
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 22:58
Start Date: 18/Jan/19 22:58
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455714998
Opened an issue: https://jira.apache.org/jira/browse/BEAM-6469
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187128)
Time Spent: 12h 20m (was: 12h 10m)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 12h 20m
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?focusedWorklogId=187124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187124 ] ASF GitHub Bot logged work on BEAM-6445: Author: ASF GitHub Bot Created on: 18/Jan/19 22:54 Start Date: 18/Jan/19 22:54 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7529: [BEAM-6445]: Release Guide changes for release process improvement URL: https://github.com/apache/beam/pull/7529 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187124) Time Spent: 3h 10m (was: 3h) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?focusedWorklogId=187119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187119 ] ASF GitHub Bot logged work on BEAM-6445: Author: ASF GitHub Bot Created on: 18/Jan/19 22:46 Start Date: 18/Jan/19 22:46 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #7529: [BEAM-6445]: Release Guide changes for release process improvement URL: https://github.com/apache/beam/pull/7529#discussion_r249207936 ## File path: website/src/contribute/release-guide.md ## @@ -428,6 +422,44 @@ Javadoc to the Javadoc for other modules that Beam depends on. the version number has changed, download a new version of the corresponding `-docs/package-list` file. +### Triage release-blocking issues in JIRA + +There could be outstanding release-blocking issues, which should be triaged before proceeding to build a release candidate. We track them by assigning a specific `Fix version` field even before the issue resolved. + +The list of release-blocking issues is available at the [version status page](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel). Triage each unresolved issue with one of the following resolutions: + +For all JIRA issues: + +* If the issue has been resolved and JIRA was not updated, resolve it accordingly. + +For JIRA issues with type "Bug" or labeled "flaky": + +* If the issue is a known continuously failing test, it is not acceptable to defer this until the next release. Please work with the Beam community to resolve the issue. +* If the issue is a known flaky test, make an attempt to fix the issue. However, if the issue may take too long to fix (to the discretion of the release manager): Review comment: Updated to be more clear that the release manager should try to delegate a fix first then try delegating manual testing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187119) Time Spent: 3h (was: 2h 50m) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=187121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187121 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 18/Jan/19 22:48 Start Date: 18/Jan/19 22:48 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7532: [BEAM-6184]Make checkstyle report error on missing javadocmethod URL: https://github.com/apache/beam/pull/7532#discussion_r249208445 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -30,4 +30,65 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Review comment: Got it. So the 3 numbers here refer to different method lengths? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187121) Time Spent: 8h 40m (was: 8.5h) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 8h 40m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4594) Implement Beam Python User State and Timer API
[ https://issues.apache.org/jira/browse/BEAM-4594?focusedWorklogId=187120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187120 ] ASF GitHub Bot logged work on BEAM-4594: Author: ASF GitHub Bot Created on: 18/Jan/19 22:47 Start Date: 18/Jan/19 22:47 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7558: [BEAM-4594] Remove timers restriction for FnApi runner. URL: https://github.com/apache/beam/pull/7558 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187120) Time Spent: 7h 20m (was: 7h 10m) > Implement Beam Python User State and Timer API > -- > > Key: BEAM-4594 > URL: https://issues.apache.org/jira/browse/BEAM-4594 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Charles Chen >Assignee: Charles Chen >Priority: Major > Labels: portability > Time Spent: 7h 20m > Remaining Estimate: 0h > > This issue tracks the implementation of the Beam Python User State and Timer > API, described here: [https://s.apache.org/beam-python-user-state-and-timers]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187118 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:45 Start Date: 18/Jan/19 22:45 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455712328 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187118) Time Spent: 7h 50m (was: 7h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187117 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:45 Start Date: 18/Jan/19 22:45 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455712294 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187117) Time Spent: 7h 40m (was: 7.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6138) Add User Metric Support to Java SDK
[ https://issues.apache.org/jira/browse/BEAM-6138?focusedWorklogId=187116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187116 ] ASF GitHub Bot logged work on BEAM-6138: Author: ASF GitHub Bot Created on: 18/Jan/19 22:43 Start Date: 18/Jan/19 22:43 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7482: [BEAM-6138] Refactor the start and finish function registration so that PTransform IDs can be properly injected onto user counters. URL: https://github.com/apache/beam/pull/7482 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187116) Time Spent: 6h 10m (was: 6h) > Add User Metric Support to Java SDK > --- > > Key: BEAM-6138 > URL: https://issues.apache.org/jira/browse/BEAM-6138 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Assignee: Alex Amato >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187115
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 22:41
Start Date: 18/Jan/19 22:41
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455711488
This is essentially the same as https://github.com/apache/beam/pull/6675. It
looks like the option parsing that is reverted here is required by the failing
tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187115)
Time Spent: 12h (was: 11h 50m)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 12h
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187114=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187114 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:39 Start Date: 18/Jan/19 22:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-45577 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187114) Time Spent: 7.5h (was: 7h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187113 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:39 Start Date: 18/Jan/19 22:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455711034 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187113) Time Spent: 7h 20m (was: 7h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187103
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 22:33
Start Date: 18/Jan/19 22:33
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455709174
> Should this also get into master?
No, as per mailing list discussion we decided to leave this in master as
some of our users depend on this. A solution is being worked on for the next
release.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187103)
Time Spent: 11h 50m (was: 11h 40m)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187101
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 22:30
Start Date: 18/Jan/19 22:30
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455709174
> Should this also get into master?
No, as per mailing list discussion we decided to leave this in master as
some of our users depend on this. A solution is being worked for the next
release.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187101)
Time Spent: 11h 40m (was: 11.5h)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11h 40m
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5732) expose runner mode to user through samza pipeline option
[ https://issues.apache.org/jira/browse/BEAM-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyu Liu reassigned BEAM-5732: --- Assignee: Daniel Chen (was: Xinyu Liu) > expose runner mode to user through samza pipeline option > > > Key: BEAM-5732 > URL: https://issues.apache.org/jira/browse/BEAM-5732 > Project: Beam > Issue Type: Improvement > Components: runner-samza >Reporter: Hai Lu >Assignee: Daniel Chen >Priority: Major > > We should expose runner mode to user through samza pipeline option so that > user can decide whether to start samza job as local mode or remote mode. > This should work consistently in both Java runner and Portable runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187089 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 18/Jan/19 22:18 Start Date: 18/Jan/19 22:18 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568#issuecomment-455706114 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187089) Time Spent: 50m (was: 40m) > Cannot create empty TestBoundedTable > > > Key: BEAM-6468 > URL: https://issues.apache.org/jira/browse/BEAM-6468 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6386) Add named variant of PTransform::compose
[
https://issues.apache.org/jira/browse/BEAM-6386?focusedWorklogId=187057=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187057
]
ASF GitHub Bot logged work on BEAM-6386:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:53
Start Date: 18/Jan/19 21:53
Worklog Time Spent: 10m
Work Description: jklukas commented on pull request #7437: [BEAM-6386]
Add named variant of PTransform::compose()
URL: https://github.com/apache/beam/pull/7437#discussion_r249196723
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PTransform.java
##
@@ -319,4 +319,16 @@ public OutputT expand(InputT input) {
}
};
}
+
+ /** Like {@link #compose(SerializableFunction)}, but with a custom name. */
+ @Experimental
+ public static
Review comment:
Your example is spot-on and represents probably the majority use case of
`PTransform.compose`. I agree with you that in cases where you're defining a
composed transform inline as part of a pipeline, passing the name to
`PCollection.apply` is sufficient.
The specific case where I've found the lack of name a nuisance is when I
want to return a transform from a method. Consider the following enum, which we
use as a pipeline option:
```
public enum InputType {
pubsub {
/** Return a PTransform that reads from a Pubsub subscription. */
public PTransform>>
read(
SinkOptions.Parsed options) {
return PTransform.compose(input -> input
.apply(PubsubIO.readMessagesWithAttributes().fromSubscription(options.getInput()))
.apply(MapElementsWithErrors.ToPubsubMessageFrom.identity()));
}
},
file {
/** Return a PTransform that reads from local or remote files. */
public PTransform>>
read(
SinkOptions.Parsed options) {
return PTransform.compose(input ->
input.apply(TextIO.read().from(options.getInput()))
.apply(options.getInputFileFormat().decode()));
}
};
public abstract PTransform>> read(
SinkOptions.Parsed options);
}
```
My main class then calls:
```
input.apply(options.getInputType().read())
```
I'd like the name of the transform to indicate what the input type was
rather than picking a static name that I pass into `apply`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187057)
Time Spent: 0.5h (was: 20m)
> Add named variant of PTransform::compose
>
>
> Key: BEAM-6386
> URL: https://issues.apache.org/jira/browse/BEAM-6386
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Jeff Klukas
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> BEAM- 5413 introduced PTransform::compose as a concise way of creating a
> composition of transforms as a lambda. We should add a variant to allow
> specifying a name for the returned transform in the same way that {{apply}}
> can take an explicit name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2928) ULR support for portable side input
[ https://issues.apache.org/jira/browse/BEAM-2928?focusedWorklogId=187060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187060 ] ASF GitHub Bot logged work on BEAM-2928: Author: ASF GitHub Bot Created on: 18/Jan/19 21:54 Start Date: 18/Jan/19 21:54 Worklog Time Spent: 10m Work Description: swegner commented on pull request #7557: [BEAM-2928]Minor code refactoring and improvements. URL: https://github.com/apache/beam/pull/7557 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187060) Time Spent: 1h 10m (was: 1h) > ULR support for portable side input > --- > > Key: BEAM-2928 > URL: https://issues.apache.org/jira/browse/BEAM-2928 > Project: Beam > Issue Type: Improvement > Components: runner-core, runner-direct >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Labels: portability > Time Spent: 1h 10m > Remaining Estimate: 0h > > Get side inputs working on the ULR. Since the ULR code is based on the direct > runner code there should already be some code that could be reused, but new > code will need to be written where side inputs would interact with > portability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6241) MongoDbIO - Add Limit and Aggregates Support
[
https://issues.apache.org/jira/browse/BEAM-6241?focusedWorklogId=187053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187053
]
ASF GitHub Bot logged work on BEAM-6241:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:44
Start Date: 18/Jan/19 21:44
Worklog Time Spent: 10m
Work Description: iemejia commented on issue #7293: [BEAM-6241] Added
limit and aggregates support to MongoDbIO
URL: https://github.com/apache/beam/pull/7293#issuecomment-455697617
Yes no issues, just too busy with unexpected stuff. Will take a look during
the weekend. (sorry)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187053)
Time Spent: 2h 40m (was: 2.5h)
> MongoDbIO - Add Limit and Aggregates Support
>
>
> Key: BEAM-6241
> URL: https://issues.apache.org/jira/browse/BEAM-6241
> Project: Beam
> Issue Type: Improvement
> Components: io-java-mongodb
>Affects Versions: 2.9.0
>Reporter: Ahmed El.Hussaini
>Assignee: Ahmed El.Hussaini
>Priority: Major
> Labels: easyfix
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> h2. Adds Support to Limit Results
>
> {code:java}
> MongoDbIO.read()
> .withUri("mongodb://localhost:" + port)
> .withDatabase(DATABASE)
> .withCollection(COLLECTION)
> .withFilter("{\"scientist\":\"Einstein\"}")
> .withLimit(5));{code}
> h2. Adds Support to Use Aggregates
>
> {code:java}
> List aggregates = new ArrayList();
> aggregates.add(
> new BsonDocument(
> "$match",
> new BsonDocument("country", new BsonDocument("$eq", new
> BsonString("England");
> PCollection output =
> pipeline.apply(
> MongoDbIO.read()
> .withUri("mongodb://localhost:" + port)
> .withDatabase(DATABASE)
> .withCollection(COLLECTION)
> .withAggregate(aggregates));
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6354) Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and SplittableDoFnTest#testLateData
[ https://issues.apache.org/jira/browse/BEAM-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746691#comment-16746691 ] Kenneth Knowles commented on BEAM-6354: --- [~HuangLED] I wonder if this is related to what you saw in the ULR. > Hanging BoundedReadFromUnboundedSourceTest#testTimeBound and > SplittableDoFnTest#testLateData > > > Key: BEAM-6354 > URL: https://issues.apache.org/jira/browse/BEAM-6354 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Gleb Kanterov >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > > It seems that they have a similar root cause because both of them use > unbounded streams. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (BEAM-5446) SplittableDoFn: Remove runner time execution information from public API surface
[ https://issues.apache.org/jira/browse/BEAM-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reopened BEAM-5446: --- The PR was rolled back since the Watch transform is not yet made compatible with the change. > SplittableDoFn: Remove runner time execution information from public API > surface > > > Key: BEAM-5446 > URL: https://issues.apache.org/jira/browse/BEAM-5446 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Fix For: 2.9.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Move the setting of "claim observers" within RestrictionTracker to another > location to clean up the RestrictionTracker interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5446) SplittableDoFn: Remove runner time execution information from public API surface
[ https://issues.apache.org/jira/browse/BEAM-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-5446: - Assignee: Scott Wegner (was: Luke Cwik) > SplittableDoFn: Remove runner time execution information from public API > surface > > > Key: BEAM-5446 > URL: https://issues.apache.org/jira/browse/BEAM-5446 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Scott Wegner >Priority: Minor > Fix For: 2.9.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Move the setting of "claim observers" within RestrictionTracker to another > location to clean up the RestrictionTracker interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6352) Watch PTransform is broken
[
https://issues.apache.org/jira/browse/BEAM-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-6352.
---
Resolution: Fixed
> Watch PTransform is broken
> --
>
> Key: BEAM-6352
> URL: https://issues.apache.org/jira/browse/BEAM-6352
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.9.0
>Reporter: Gleb Kanterov
>Assignee: Scott Wegner
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> List of affected tests:
> org.apache.beam.sdk.transforms.WatchTest >
> testSinglePollMultipleInputsWithSideInput FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithKeyExtractor
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationDueToTerminationCondition FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsStopAfterTimeSinceNewOutput
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > testReadWatchForNewFiles
> FAILED
> {code}
> java.lang.IllegalArgumentException:
> org.apache.beam.sdk.transforms.Watch$WatchGrowthFn, @ProcessElement
> process(ProcessContext, GrowthTracker): Has tracker type
> Watch.GrowthTracker, but the DoFn's tracker
> type must be of type RestrictionTracker.
> {code}
> Relevant pull requests:
> - https://github.com/apache/beam/pull/6467
> - https://github.com/apache/beam/pull/7374
> Now tests are marked with @Ignore referencing this JIRA issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187050
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:35
Start Date: 18/Jan/19 21:35
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455695045
I'm hoping to get green for the release merge. Should this also get into
`master`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187050)
Time Spent: 11.5h (was: 11h 20m)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187049
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:34
Start Date: 18/Jan/19 21:34
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455694954
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187049)
Time Spent: 11h 20m (was: 11h 10m)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187048
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:34
Start Date: 18/Jan/19 21:34
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455694925
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187048)
Time Spent: 11h 10m (was: 11h)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6352) Watch PTransform is broken
[
https://issues.apache.org/jira/browse/BEAM-6352?focusedWorklogId=187046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187046
]
ASF GitHub Bot logged work on BEAM-6352:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:33
Start Date: 18/Jan/19 21:33
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7540: [BEAM-6352]
Revert PR#6467 to fix Watch transform
URL: https://github.com/apache/beam/pull/7540#issuecomment-455694542
Would you prepare a cherry-pick against the release branch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187046)
Time Spent: 1h 10m (was: 1h)
> Watch PTransform is broken
> --
>
> Key: BEAM-6352
> URL: https://issues.apache.org/jira/browse/BEAM-6352
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.9.0
>Reporter: Gleb Kanterov
>Assignee: Scott Wegner
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> List of affected tests:
> org.apache.beam.sdk.transforms.WatchTest >
> testSinglePollMultipleInputsWithSideInput FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithKeyExtractor
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationDueToTerminationCondition FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsStopAfterTimeSinceNewOutput
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > testReadWatchForNewFiles
> FAILED
> {code}
> java.lang.IllegalArgumentException:
> org.apache.beam.sdk.transforms.Watch$WatchGrowthFn, @ProcessElement
> process(ProcessContext, GrowthTracker): Has tracker type
> Watch.GrowthTracker, but the DoFn's tracker
> type must be of type RestrictionTracker.
> {code}
> Relevant pull requests:
> - https://github.com/apache/beam/pull/6467
> - https://github.com/apache/beam/pull/7374
> Now tests are marked with @Ignore referencing this JIRA issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6352) Watch PTransform is broken
[
https://issues.apache.org/jira/browse/BEAM-6352?focusedWorklogId=187045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187045
]
ASF GitHub Bot logged work on BEAM-6352:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:33
Start Date: 18/Jan/19 21:33
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7540:
[BEAM-6352] Revert PR#6467 to fix Watch transform
URL: https://github.com/apache/beam/pull/7540
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187045)
Time Spent: 1h (was: 50m)
> Watch PTransform is broken
> --
>
> Key: BEAM-6352
> URL: https://issues.apache.org/jira/browse/BEAM-6352
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.9.0
>Reporter: Gleb Kanterov
>Assignee: Scott Wegner
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> List of affected tests:
> org.apache.beam.sdk.transforms.WatchTest >
> testSinglePollMultipleInputsWithSideInput FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithKeyExtractor
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationDueToTerminationCondition FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsStopAfterTimeSinceNewOutput
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > testReadWatchForNewFiles
> FAILED
> {code}
> java.lang.IllegalArgumentException:
> org.apache.beam.sdk.transforms.Watch$WatchGrowthFn, @ProcessElement
> process(ProcessContext, GrowthTracker): Has tracker type
> Watch.GrowthTracker, but the DoFn's tracker
> type must be of type RestrictionTracker.
> {code}
> Relevant pull requests:
> - https://github.com/apache/beam/pull/6467
> - https://github.com/apache/beam/pull/7374
> Now tests are marked with @Ignore referencing this JIRA issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6352) Watch PTransform is broken
[
https://issues.apache.org/jira/browse/BEAM-6352?focusedWorklogId=187044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187044
]
ASF GitHub Bot logged work on BEAM-6352:
Author: ASF GitHub Bot
Created on: 18/Jan/19 21:33
Start Date: 18/Jan/19 21:33
Worklog Time Spent: 10m
Work Description: kennknowles commented on issue #7540: [BEAM-6352]
Revert PR#6467 to fix Watch transform
URL: https://github.com/apache/beam/pull/7540#issuecomment-455694421
Nice. Thanks for putting this together!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187044)
Time Spent: 50m (was: 40m)
> Watch PTransform is broken
> --
>
> Key: BEAM-6352
> URL: https://issues.apache.org/jira/browse/BEAM-6352
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.9.0
>Reporter: Gleb Kanterov
>Assignee: Scott Wegner
>Priority: Blocker
> Fix For: 2.10.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> List of affected tests:
> org.apache.beam.sdk.transforms.WatchTest >
> testSinglePollMultipleInputsWithSideInput FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithKeyExtractor
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationDueToTerminationCondition FAILED
> org.apache.beam.sdk.transforms.WatchTest > testMultiplePollsWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults
> FAILED
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsStopAfterTimeSinceNewOutput
> org.apache.beam.sdk.transforms.WatchTest >
> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED
> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests >
> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED
> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED
> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > testReadWatchForNewFiles
> FAILED
> {code}
> java.lang.IllegalArgumentException:
> org.apache.beam.sdk.transforms.Watch$WatchGrowthFn, @ProcessElement
> process(ProcessContext, GrowthTracker): Has tracker type
> Watch.GrowthTracker, but the DoFn's tracker
> type must be of type RestrictionTracker.
> {code}
> Relevant pull requests:
> - https://github.com/apache/beam/pull/6467
> - https://github.com/apache/beam/pull/7374
> Now tests are marked with @Ignore referencing this JIRA issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?focusedWorklogId=187036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187036 ] ASF GitHub Bot logged work on BEAM-6445: Author: ASF GitHub Bot Created on: 18/Jan/19 21:14 Start Date: 18/Jan/19 21:14 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #7529: [BEAM-6445]: Release Guide changes for release process improvement URL: https://github.com/apache/beam/pull/7529#discussion_r249187128 ## File path: website/src/contribute/release-guide.md ## @@ -428,6 +422,44 @@ Javadoc to the Javadoc for other modules that Beam depends on. the version number has changed, download a new version of the corresponding `-docs/package-list` file. +### Triage release-blocking issues in JIRA + +There could be outstanding release-blocking issues, which should be triaged before proceeding to build a release candidate. We track them by assigning a specific `Fix version` field even before the issue resolved. + +The list of release-blocking issues is available at the [version status page](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel). Triage each unresolved issue with one of the following resolutions: + +For all JIRA issues: + +* If the issue has been resolved and JIRA was not updated, resolve it accordingly. + +For JIRA issues with type "Bug" or labeled "flaky": + +* If the issue is a known continuously failing test, it is not acceptable to defer this until the next release. Please work with the Beam community to resolve the issue. +* If the issue is a known flaky test, make an attempt to fix the issue. However, if the issue may take too long to fix (to the discretion of the release manager): Review comment: This might not be super clear - I think the release manager should make an attempt to _get the test fixed_ but often by delegating / asking someone else, just like the next bullet. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187036) Time Spent: 2h 40m (was: 2.5h) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6445) Improve Release Process
[ https://issues.apache.org/jira/browse/BEAM-6445?focusedWorklogId=187037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187037 ] ASF GitHub Bot logged work on BEAM-6445: Author: ASF GitHub Bot Created on: 18/Jan/19 21:14 Start Date: 18/Jan/19 21:14 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #7529: [BEAM-6445]: Release Guide changes for release process improvement URL: https://github.com/apache/beam/pull/7529#discussion_r249187468 ## File path: website/src/contribute/release-guide.md ## @@ -953,6 +986,7 @@ Once all issues have been resolved, you should go back and build a new release c ### Checklist to proceed to the next step 1. Issues identified during vote have been resolved, with fixes committed to the release branch. +1. All issues tagged with `Fix-Version` for the current release should be closed. Review comment: 2. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187037) Time Spent: 2h 50m (was: 2h 40m) > Improve Release Process > --- > > Key: BEAM-6445 > URL: https://issues.apache.org/jira/browse/BEAM-6445 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > This JIRA tracks the improvement of the Beam release process as [discussed in > the dev > list|https://lists.apache.org/thread.html/d52ffbfca21eee953a230100520bd56d947a359c0029d5c291b736a7@%3Cdev.beam.apache.org%3E]. > In summary, this change will hopefully increase the greenness of the build > by: increasing coverage, adding pre and post commits to release validation, > and adding a regular cadence to look at flaky and backlogged tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6271) initial support for portable api in samza runner
[ https://issues.apache.org/jira/browse/BEAM-6271?focusedWorklogId=187038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187038 ] ASF GitHub Bot logged work on BEAM-6271: Author: ASF GitHub Bot Created on: 18/Jan/19 21:14 Start Date: 18/Jan/19 21:14 Worklog Time Spent: 10m Work Description: lhaiesp commented on issue #7321: [BEAM-6271] SamzaRunner: initial support for portable api in samza runner URL: https://github.com/apache/beam/pull/7321#issuecomment-455689548 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187038) Time Spent: 5h 10m (was: 5h) > initial support for portable api in samza runner > > > Key: BEAM-6271 > URL: https://issues.apache.org/jira/browse/BEAM-6271 > Project: Beam > Issue Type: Task > Components: runner-samza >Reporter: Hai Lu >Assignee: Hai Lu >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > Support portable api in Samza runner. > This ticket tracks the initial effort to support portable api in Samza > runner, including job server inside samza runner, config translation for > portable pipeline, transform translation for portable pipeline, refactor of > existing codes to merge logic of portable api and java api as much as > possible, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187030 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 18/Jan/19 20:57 Start Date: 18/Jan/19 20:57 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568#issuecomment-455685180 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187030) Time Spent: 40m (was: 0.5h) > Cannot create empty TestBoundedTable > > > Key: BEAM-6468 > URL: https://issues.apache.org/jira/browse/BEAM-6468 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6465) Flink: State accumulation during restoring from a savepoint
[
https://issues.apache.org/jira/browse/BEAM-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels updated BEAM-6465:
-
Issue Type: Bug (was: Test)
> Flink: State accumulation during restoring from a savepoint
> ---
>
> Key: BEAM-6465
> URL: https://issues.apache.org/jira/browse/BEAM-6465
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.7.0
>Reporter: Pawel Bartoszek
>Assignee: Maximilian Michels
>Priority: Major
> Attachments: Screen Shot 2019-01-18 at 11.38.02.png, Screen Shot
> 2019-01-18 at 12.07.03 copy.png
>
>
> This ticket captures my findings when restoring a BEAM job from a savepoint
> on a Flink runner.
>
> *The problem*
> When job is restored from a savepoint taken a few hours ago than we see that
> checkpoint size starts growing rediciously high which leads to the job
> running out of heap space error. We use filesystem state backend, which keeps
> state on the heap.
>
> *Job structure*
> Job has two paths the data lake path and the aggregate data path.
> *Data lake path*
> Data lake path is a dumb sink of all records received by the job. The records
> are flushed to S3.
> Datalake trigger:
> {code:java}
> input.apply(
> WithTimestamps.of(extractTimestamp).withAllowedTimestampSkew(standardDays(7)))
> .apply(Window.into(FixedWindows.of(standardMinutes(1)))
> .triggering(
> AfterWatermark.pastEndOfWindow()
>
> .withLateFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(lateFiringsPeriod))
> )
> .withAllowedLateness(standardMinutes(30), FIRE_ALWAYS)
> .discardingFiredPanes()); <-- IMPORTANT {code}
>
> *Aggregate path*
> Aggregate path has some group by key, count etc transformations
> Aggregate trigger:
> {code:java}
> input.apply(
> WithTimestamps.of(extractTimestamp).withAllowedTimestampSkew(standardDays(7)))
> .apply(Window.into(FixedWindows.of(WINDOW_SIZE))
> .triggering(
> AfterWatermark.pastEndOfWindow()
>
> .withLateFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(lateFiringsPeriod))
> )
> .withAllowedLateness(standardMinutes(30), FIRE_ALWAYS)
> .accumulatingFiredPanes());{code}
>
> *My investigation*
> Our team has written a tool to collect input watermarks from the Flink API.
> It turned out that it's a common situation that for pretty much every
> operator some sub operators (running on particular Flink slot) are running
> slower thus watermark is falling behind other slots. Look at the graph below:
> !Screen Shot 2019-01-18 at 12.07.03 copy.png!
> Y axis represents an event time, X wall clock time. Graph shows input
> watermarks for some operator in the job. Each line represents an input
> watermark for a specific slot (job runs with parallelism 8) for which
> operator is running on.
> At 17:55 the difference between slowest slot and quickest one is already 20
> mins. This means that 20 mins of data will be buffered in memory until
> slowest slot's watermark will cross the end of the 1 minute window of the
> buffered data.
> Unfortunately it's very hard to tell why some slots are doing better then the
> other( I believe data is properly balanced when it comes to hashing etc)
>
> Below I present checkpoint size growing in size. Normally when not restoring
> from a job checkpoint size is around 3 GB
> !Screen Shot 2019-01-18 at 11.38.02.png!
>
> *My findings*
> If the state is being accumulated because of watermark slowing down on some
> operator slots (Flink specific) than introducing *early firings* *with
> discarding mode* should help ... and indeed helped.
>
> *My worry is that introducing an early firings as a way to avoid OOM error
> during catchup seems hacky to me. The other downside is that early firings
> introduce speculative results which might not be acceptable in some cases.*
>
> *Setup:*
> * Job reads records from 32 Kinesis shards.
> * Job parallelism 8
> * Running on Beam 2.7 Flink 1.5
> *Hardware:*
> ** Master:1 x m5.xlarge
> ** Core instances: 5 x r4.2xlarge
> * *YARN session configuration:*
> **
> {code:java}
> /usr/bin/flink run --class streaming.Main -m yarn-cluster --yarnstreaming
> --yarnjobManagerMemory 6272 --yarntaskManagerMemory 26000 -yD
> classloader.resolve-order=parent-first -yD parallelism.default=8 -yD
> containerized.heap-cutoff-ratio=0.15 -yD state.backend=filesystem -yD
> yarn.maximum-failed-containers=-1 -yD jobmanager.web.checkpoints.history=1000
> -yD akka.ask.timeout=60s -XX:GCLogFileSize=20M -XX:NumberOfGCLogFiles=2
>
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187029=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187029 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 18/Jan/19 20:54 Start Date: 18/Jan/19 20:54 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568#issuecomment-455684448 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187029) Time Spent: 0.5h (was: 20m) > Cannot create empty TestBoundedTable > > > Key: BEAM-6468 > URL: https://issues.apache.org/jira/browse/BEAM-6468 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner
[
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=187027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187027
]
ASF GitHub Bot logged work on BEAM-5442:
Author: ASF GitHub Bot
Created on: 18/Jan/19 20:43
Start Date: 18/Jan/19 20:43
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7564: [release] Revert
"[BEAM-5442] Store duplicate unknown options in a list argument"
URL: https://github.com/apache/beam/pull/7564#issuecomment-455681274
Portable Python fails with ```ImportError: No module named tenacity```. It
runs fine on my machine.
No idea why ```:beam-sdks-python-precommit-dataflow:precommitIT``` fails,
possible the same reason.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187027)
Time Spent: 11h (was: 10h 50m)
> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11h
> Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the
> PortableRunner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3342) Create a Cloud Bigtable Python connector
[ https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=187026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187026 ] ASF GitHub Bot logged work on BEAM-3342: Author: ASF GitHub Bot Created on: 18/Jan/19 20:38 Start Date: 18/Jan/19 20:38 Worklog Time Spent: 10m Work Description: juan-rael commented on issue #7367: [BEAM-3342] Create a Cloud Bigtable Python connector Write URL: https://github.com/apache/beam/pull/7367#issuecomment-455680113 @chamikaramj It's ready... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187026) Time Spent: 11h (was: 10h 50m) > Create a Cloud Bigtable Python connector > > > Key: BEAM-3342 > URL: https://issues.apache.org/jira/browse/BEAM-3342 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Solomon Duskis >Assignee: Solomon Duskis >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > > I would like to create a Cloud Bigtable python connector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187025 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 18/Jan/19 20:36 Start Date: 18/Jan/19 20:36 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568#issuecomment-455679630 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187025) Time Spent: 20m (was: 10m) > Cannot create empty TestBoundedTable > > > Key: BEAM-6468 > URL: https://issues.apache.org/jira/browse/BEAM-6468 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6468) Cannot create empty TestBoundedTable
[ https://issues.apache.org/jira/browse/BEAM-6468?focusedWorklogId=187021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187021 ] ASF GitHub Bot logged work on BEAM-6468: Author: ASF GitHub Bot Created on: 18/Jan/19 20:16 Start Date: 18/Jan/19 20:16 Worklog Time Spent: 10m Work Description: amaliujia commented on pull request #7568: [BEAM-6468] Allow creating empty TestBoundedTable URL: https://github.com/apache/beam/pull/7568 Allow creating empty TestBoundedTable Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187021) Time Spent: 10m Remaining Estimate: 0h > Cannot create empty TestBoundedTable >
[jira] [Created] (BEAM-6468) Cannot create empty TestBoundedTable
Rui Wang created BEAM-6468: -- Summary: Cannot create empty TestBoundedTable Key: BEAM-6468 URL: https://issues.apache.org/jira/browse/BEAM-6468 Project: Beam Issue Type: Bug Components: dsl-sql Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3342) Create a Cloud Bigtable Python connector
[
https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=187017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187017
]
ASF GitHub Bot logged work on BEAM-3342:
Author: ASF GitHub Bot
Created on: 18/Jan/19 20:10
Start Date: 18/Jan/19 20:10
Worklog Time Spent: 10m
Work Description: juan-rael commented on pull request #7367: [BEAM-3342]
Create a Cloud Bigtable Python connector Write
URL: https://github.com/apache/beam/pull/7367#discussion_r249171788
##
File path: sdks/python/apache_beam/io/gcp/bigtable_io_test.py
##
@@ -0,0 +1,190 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unittest for GCP Bigtable testing."""
+from __future__ import absolute_import
+
+import datetime
+import logging
+import random
+import string
+import unittest
+import uuid
+
+import pytz
+
+import apache_beam as beam
+from apache_beam.io.gcp.bigtable_io import BigtableConfiguration
+from apache_beam.io.gcp.bigtable_io import WriteToBigtable
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+
+# Protect against environments where bigtable library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from google.cloud._helpers import _datetime_from_microseconds
+ from google.cloud._helpers import _microseconds_from_datetime
+ from google.cloud._helpers import UTC
+ from google.cloud.bigtable import row, column_family, Client
+except ImportError:
+ Client = None
+ UTC = pytz.utc
+ _microseconds_from_datetime = lambda label_stamp: label_stamp
+ _datetime_from_microseconds = lambda micro: micro
+
+
+EXISTING_INSTANCES = []
+LABEL_KEY = u'python-bigtable-beam'
+label_stamp = datetime.datetime.utcnow().replace(tzinfo=UTC)
+label_stamp_micros = _microseconds_from_datetime(label_stamp)
+LABELS = {LABEL_KEY: str(label_stamp_micros)}
+
+
+def _retry_on_unavailable(exc):
+ """Retry only errors whose status code is 'UNAVAILABLE'."""
+ from grpc import StatusCode
+ return exc.code() == StatusCode.UNAVAILABLE
+
+
+class GenerateDirectRows(beam.PTransform):
+ """ Generates an iterator of DirectRow object to process on beam pipeline.
+
+ """
+ def __init__(self, number, **kwargs):
+super(GenerateDirectRows, self).__init__(**kwargs)
+self.number = number
+self.rand = random.choice(string.ascii_letters + string.digits)
+self.column_family_id = 'cf1'
+
+ def _generate(self):
+value = ''.join(self.rand for i in range(100))
+
+for index in range(self.number):
+ key = "beam_key%s" % ('{0:07}'.format(index))
+ direct_row = row.DirectRow(row_key=key)
+ for column_id in range(10):
+direct_row.set_cell(self.column_family_id,
+('field%s' % column_id).encode('utf-8'),
+value,
+datetime.datetime.now())
+ yield direct_row
+
+ def expand(self, pvalue):
+return (pvalue
+| beam.Create(self._generate()))
+
+
+@unittest.skipIf(Client is None, 'GCP Bigtable dependencies are not installed')
+class BigtableIOWriteIT(unittest.TestCase):
Review comment:
So, I will not add this IT in the tox.ini file?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187017)
Time Spent: 10h 50m (was: 10h 40m)
> Create a Cloud Bigtable Python connector
>
>
> Key: BEAM-3342
> URL: https://issues.apache.org/jira/browse/BEAM-3342
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=187008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187008 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 18/Jan/19 19:36 Start Date: 18/Jan/19 19:36 Worklog Time Spent: 10m Work Description: HuangLED commented on pull request #7532: [BEAM-6184]Make checkstyle report error on missing javadocmethod URL: https://github.com/apache/beam/pull/7532#discussion_r249162079 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -30,4 +30,65 @@ + + > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 8h 20m > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6184) PortableRunner dependency missed in wordcount example maven artifact
[ https://issues.apache.org/jira/browse/BEAM-6184?focusedWorklogId=187013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187013 ] ASF GitHub Bot logged work on BEAM-6184: Author: ASF GitHub Bot Created on: 18/Jan/19 19:42 Start Date: 18/Jan/19 19:42 Worklog Time Spent: 10m Work Description: HuangLED commented on pull request #7532: [BEAM-6184]Make checkstyle report error on missing javadocmethod URL: https://github.com/apache/beam/pull/7532#discussion_r249163592 ## File path: sdks/java/build-tools/src/main/resources/beam/suppressions.xml ## @@ -30,4 +30,65 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Review comment: For conciseness. The intention is indeed to have the full file name (in this particular case BigqueryClient.java is a full file name). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187013) Time Spent: 8.5h (was: 8h 20m) > PortableRunner dependency missed in wordcount example maven artifact > > > Key: BEAM-6184 > URL: https://issues.apache.org/jira/browse/BEAM-6184 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Fix For: Not applicable > > Time Spent: 8.5h > Remaining Estimate: 0h > > > > more context: > https://lists.apache.org/thread.html/8dd60395424425f7502d62888c49014430d1d3b06c026606f3db28ab@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
