[jira] [Work logged] (BEAM-5436) Improve docs for Go SDK - getting started for non-linux

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5436?focusedWorklogId=146239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146239
 ]

ASF GitHub Bot logged work on BEAM-5436:


Author: ASF GitHub Bot
Created on: 21/Sep/18 05:31
Start Date: 21/Sep/18 05:31
Worklog Time Spent: 10m 
  Work Description: vitaliytv opened a new pull request #557: [BEAM-5436] 
Improve docs for Go SDK
URL: https://github.com/apache/beam-site/pull/557
 
 
   - error go compiling on clean mac with dataflow runner (without installing 
golang.org/x/sys/unix)
   - error executing go sdk with dataflow runner (without set staging_location)
   
   Once your pull request has been opened and assigned a number, please edit the
   URL below, replacing `PULL_REQUEST_NUMBER` with the number of your pull 
request.
   
   
http://apache-beam-website-pull-requests.storage.googleapis.com/PULL_REQUEST_NUMBER/index.html
   
   Finally, it will help us expedite review of your Pull Request if you tag
   someone (e.g. @username) to look at it.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146239)
Time Spent: 40m  (was: 0.5h)

> Improve docs for Go SDK - getting started for non-linux
> ---
>
> Key: BEAM-5436
> URL: https://issues.apache.org/jira/browse/BEAM-5436
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go, website
>Affects Versions: 0.5.0
> Environment: darwin amd64
>Reporter: Vitalii Tverdokhlib
>Assignee: Melissa Pashniak
>Priority: Trivial
>  Labels: documentation
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Cross-compiling error on Mac and staging_location required (even if 
> temp_location present]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5436) Improve docs for Go SDK - getting started for non-linux

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5436?focusedWorklogId=146240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146240
 ]

ASF GitHub Bot logged work on BEAM-5436:


Author: ASF GitHub Bot
Created on: 21/Sep/18 05:31
Start Date: 21/Sep/18 05:31
Worklog Time Spent: 10m 
  Work Description: vitaliytv commented on issue #6442: [BEAM-5436] Improve 
docs for Go SDK
URL: https://github.com/apache/beam/pull/6442#issuecomment-423417984
 
 
   https://github.com/apache/beam-site/pull/557


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146240)
Time Spent: 50m  (was: 40m)

> Improve docs for Go SDK - getting started for non-linux
> ---
>
> Key: BEAM-5436
> URL: https://issues.apache.org/jira/browse/BEAM-5436
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go, website
>Affects Versions: 0.5.0
> Environment: darwin amd64
>Reporter: Vitalii Tverdokhlib
>Assignee: Melissa Pashniak
>Priority: Trivial
>  Labels: documentation
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Cross-compiling error on Mac and staging_location required (even if 
> temp_location present]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #90

[Apache Jenkins Server]

See 


Changes:

[markliu] [BEAM-5334] Fix Python benmchmark with Gradle build

--
[...truncated 566.62 KB...]
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 275, in 
test_combine_per_key
assert_that(res, equal_to([('a', 1.5), ('b', 3.0)]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_combine_per_key_1537507277.11_fba931b8-6e8f-4b03-a411-a72303b87f01 failed 
in state FAILED.

==
ERROR: test_create (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 63, in 
test_create
assert_that(p | beam.Create(['a', 'b']), equal_to(['a', 'b']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_create_1537507277.47_e0a57c22-76fb-4e5e-b4b1-ebfcedc4e1c5 failed in state 
FAILED.

==
ERROR: test_flatten (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 268, in 
test_flatten
assert_that(res, equal_to(['a', 'b', 'c', 'd']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flatten_1537507277.96_766bd7da-e646-4f6f-8e0b-7d0fba05fbe7 failed in state 
FAILED.

==
ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 191, in 
test_flattened_side_input
equal_to([(None, {'a': 1, 'b': 2})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flattened_side_input_1537507278.47_31e6782e-2fc6-4b29-91d9-48e9d07d97ec 
failed in state FAILED.

==
ERROR: test_gbk_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 199, in 
test_gbk_side_input
equal_to([(None, {'a': [1]})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_gbk_side_input_1537507278.97_4fdd221e-8ca2-4b7d-8c63-0dfe1ac887c2 failed 
in state FAILED.

==
ERROR: test_group_by_key (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 261, in 
test_group_by_key
assert_that(res, equal_to([('a', [1, 2]), ('b', [3])]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_group_by_key_1537507279.45_cbf5c907-801c-4a5d-90fc-9e19d30179a3 failed in 
state FAILED.

==
ERROR: test_large_elements (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 317, in 
test_large_elements
assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
  File "apache_beam/pipeline.py", line 414, in __exit__

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146235
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 21/Sep/18 05:16
Start Date: 21/Sep/18 05:16
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423415856
 
 
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146235)
Time Spent: 2h 50m  (was: 2h 40m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146234
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 21/Sep/18 05:15
Start Date: 21/Sep/18 05:15
Worklog Time Spent: 10m 
  Work Description: lgajowy closed pull request #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/CommonJobProperties.groovy 
b/.test-infra/jenkins/CommonJobProperties.groovy
index 3fdbdba7e33..428ac0de449 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -318,9 +318,7 @@ class CommonJobProperties {
 def pkbArgs = genPerformanceArgs(argMap)
 
 // Absolute path of project root and virtualenv path of Beam and Perfkit.
-def beam_root = makePathAbsolute(checkoutDir)
 def perfkit_root = makePathAbsolute("PerfKitBenchmarker")
-def beam_env = makePathAbsolute("env/.beam_env")
 def perfkit_env = makePathAbsolute("env/.perfkit_env")
 
 context.steps {
@@ -340,16 +338,6 @@ class CommonJobProperties {
 // Install Perfkit benchmark requirements.
 shell("${perfkit_env}/bin/pip install -r 
${perfkit_root}/requirements.txt")
 
-// Install Beam Python SDK requirements.
-if (language == "PYTHON") {
-  shell("rm -rf ${beam_env}")
-  shell("virtualenv ${beam_env}")
-  shell("${beam_env}/bin/pip install --upgrade setuptools pip 
grpcio-tools==1.3.5")
-  shell("${beam_env}/bin/pip install -e 
${beam_root}/sdks/python/[gcp,test]")
-  // Build PythonSDK tar ball.
-  shell("(cd ${beam_root}/sdks/python && ${beam_env}/bin/python 
setup.py sdist --dist-dir=target)")
-}
-
 // Launch performance test.
 shell("${perfkit_env}/bin/python ${perfkit_root}/pkb.py ${pkbArgs}")
 }
diff --git a/.test-infra/jenkins/job_PerformanceTests_Python.groovy 
b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
index 0b25f7098e6..7a856004190 100644
--- a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
@@ -47,12 +47,13 @@ job('beam_PerformanceTests_Python'){
   def pipelineArgsJoined = pipelineArgList.join(',')
 
   def argMap = [
-  beam_sdk   : 'python',
-  benchmarks : 'beam_integration_benchmark',
-  bigquery_table : 'beam_performance.wordcount_py_pkb_results',
-  beam_it_class  : 
'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
-  beam_prebuilt  : 'true',  // skip beam prebuild
-  beam_it_args   : pipelineArgsJoined,
+  beam_sdk : 'python',
+  benchmarks   : 'beam_integration_benchmark',
+  bigquery_table   : 'beam_performance.wordcount_py_pkb_results',
+  beam_it_class: 
'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+  beam_prebuilt: 'true',  // skip beam prebuild
+  beam_python_sdk_location : 'build/apache-beam.tar.gz',
+  beam_it_args : pipelineArgsJoined,
   ]
 
   commonJobProperties.buildPerformanceTest(delegate, argMap, 'PYTHON')


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146234)
Time Spent: 2h 40m  (was: 2.5h)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> 

[beam] branch master updated (6cef8ad -> d691a86)

[lgajowy]

This is an automated email from the ASF dual-hosted git repository.

lgajowy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6cef8ad  Merge pull request #6440: [BEAM-5412][BEAM-5408] Fixes a bug 
that limited the size of TFRecords
 add ad489f1  [BEAM-5334] Fix Python benmchmark with Gradle build
 add d691a86  Merge pull request #6456: [BEAM-5334] Fix Python benmchmark 
with Gradle build

No new revisions were added by this update.

Summary of changes:
 .test-infra/jenkins/CommonJobProperties.groovy | 12 
 .test-infra/jenkins/job_PerformanceTests_Python.groovy | 13 +++--
 2 files changed, 7 insertions(+), 18 deletions(-)

[jira] [Assigned] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu reassigned BEAM-5334:
--

Assignee: Mark Liu

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623075#comment-16623075
 ] 

Mark Liu commented on BEAM-5334:


[https://github.com/apache/beam/pull/6456] is out for review. It should fix 
this benchmark test.

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146228
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 21/Sep/18 05:08
Start Date: 21/Sep/18 05:08
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423414803
 
 
   @pabloem all benchmarks passed. It's ready to merge. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146228)
Time Spent: 2h 20m  (was: 2h 10m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5441?focusedWorklogId=146221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146221
 ]

ASF GitHub Bot logged work on BEAM-5441:


Author: ASF GitHub Bot
Created on: 21/Sep/18 05:00
Start Date: 21/Sep/18 05:00
Worklog Time Spent: 10m 
  Work Description: ryan-williams edited a comment on issue #6461: 
[BEAM-5441] extend primitive transform check to find subtransform leaves
URL: https://github.com/apache/beam/pull/6461#issuecomment-423406984
 
 
   when I run locally:
   
   ```
   python -m apache_beam.examples.wordcount --input=/etc/profile 
--output=/tmp/py-wordcount-direct --experiments=beam_fn_api 
--runner=PortableRunner --sdk_location=container --job_endpoint=localhost:8099
   ```
   I see failures like:
   
   ```
   java.lang.Exception: The user defined 'open()' method caused an exception: 
java.lang.IllegalStateException: The passed environment does not contain a 
DockerPayload.
   at 
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
   at 
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
   at java.lang.Thread.run(Thread.java:745)
   ```
   
   ([more 
output](https://gist.github.com/ryan-williams/653861032f4c6e12e6e1b4220f753348))
   
   have no leads on what could be causing that / haven't looked yet, but wanted 
to get this out for folks to look at
   
   **update**: this was basically due to not running from within `sdks/python` 
(or alternatively, not having installed my local python sdk so that it would be 
used by the `python` command above)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146221)
Time Spent: 40m  (was: 0.5h)

> Portable Wordcount fails in GreedyPipelineFuser
> ---
>
> Key: BEAM-5441
> URL: https://issues.apache.org/jira/browse/BEAM-5441
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python, sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Maximilian Michels
>Assignee: Ryan Williams
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The Python SDK wordcount with the PortableRunner throws the following 
> exception:
> {noformat}
> java.lang.IllegalArgumentException: A PCollectionNode should have exactly one 
> producing PTransformNode, PCollectionNode{id=ref_PCollection_PCollection_26, 
> PCollection=unique_name: 
> "60write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys.None"
> coder_id: "ref_Coder_FastPrimitivesCoder_2"
> is_bounded: BOUNDED
> windowing_strategy_id: "ref_Windowing_Windowing_1"
> } has 
> [PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle_32,
>  transform=spec {
>   urn: "beam:transform:generic_composite:v1"
>   payload: ""
> }
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/AddRandomKeys_33"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/ReshufflePerKey_34"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41"
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_19"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle"
> }, 
> PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41,
>  transform=spec {
>   urn: "urn:beam:transform:pardo:v1"
>   payload: "\n\317\006\n\255\006\n 
> 

[jira] [Work logged] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5441?focusedWorklogId=146217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146217
 ]

ASF GitHub Bot logged work on BEAM-5441:


Author: ASF GitHub Bot
Created on: 21/Sep/18 04:39
Start Date: 21/Sep/18 04:39
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #6461: [BEAM-5441] 
extend primitive transform check to find subtransform leaves
URL: https://github.com/apache/beam/pull/6461#issuecomment-423411079
 
 
   seems like `:beam-sdks-python:testPython3` is broken at `HEAD`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146217)
Time Spent: 0.5h  (was: 20m)

> Portable Wordcount fails in GreedyPipelineFuser
> ---
>
> Key: BEAM-5441
> URL: https://issues.apache.org/jira/browse/BEAM-5441
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python, sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Maximilian Michels
>Assignee: Ryan Williams
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The Python SDK wordcount with the PortableRunner throws the following 
> exception:
> {noformat}
> java.lang.IllegalArgumentException: A PCollectionNode should have exactly one 
> producing PTransformNode, PCollectionNode{id=ref_PCollection_PCollection_26, 
> PCollection=unique_name: 
> "60write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys.None"
> coder_id: "ref_Coder_FastPrimitivesCoder_2"
> is_bounded: BOUNDED
> windowing_strategy_id: "ref_Windowing_Windowing_1"
> } has 
> [PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle_32,
>  transform=spec {
>   urn: "beam:transform:generic_composite:v1"
>   payload: ""
> }
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/AddRandomKeys_33"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/ReshufflePerKey_34"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41"
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_19"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle"
> }, 
> PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41,
>  transform=spec {
>   urn: "urn:beam:transform:pardo:v1"
>   payload: "\n\317\006\n\255\006\n 
> beam:dofn:pickled_python_info:v1\032\210\006eNrFkttP1EAUxtsFZe1yUcEL4v3aRbdFUEFFQEHUEGpSnPhkxml32Gnstv3a6YZNbKIxJf7ZTvdB2ER9NZNMcs6cb3LO73zfaqbPEuYLTj3OupZMWZTtx2k3s/w45cYmC0PmhfxjypKEp1vxdmRAa36HXqBmkrEkjX2eZRjx20EYWrS6DeqnnElO9/PIl0GsFKPm0HsYszaV/YQbOEHqm3Gbf1ABTpYYc1E3d3R1arvTG2Tip6Z91bQfutbRtT2cckoYTaIfoFFinPRtkvE0s7vswN7iPbuaoCV5Ju0ej3p2GHh20pcijhatZTsLJG+pSb+wDs/sYzO3Fq0Va8Fq895CK+mrUot3OscL7CModgXFSvqYIPXVkHW9NlvD5G5jlGiYIrX9CKdLnGlKnHUx7VPq5UEog4hSo8MlkzI1MDNEIugmcSppN27noaJxjsz9Yxs4X+KCi4ukTpXcl5Ri9hCXXMyJSedPC/C5CnBZjJriN9W9z6SukLZ1bXYPV5wd/RBXFVKJayWu/w+kuQzCCukNMbm7XhNTTYXvpotbYkb8HUclwu0Sd1zcFQrCPRemguAUaJLGwFpUBJHMMD9sb/UwyKveFFEm4zQz3r2v3Pe2Shu4r7z9oECrgGWSRhAluRx8l8F2yHicy6PEgpMf4qGXSSy6WCrxyMXjEk8KLJtiXlQfrRR4WuCZKWxHDKqfe6o7lnayhPtUuWVVLOUSL1ysDXe9PpBvFHhZ4NWRfNMjI5VsS6zl3ie8LrDtOBJvrF+Bv0km\022\035ref_Environment_Environment_1"
> }
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_25"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys"
> }]
> at 
> org.apache.beam.repackaged.beam_runners_core_construction_java.com.google.common.base.Preconditions.checkArgument(Preconditions.java:416)
> at 
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.buildNetwork(QueryablePipeline.java:176)
> at 
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.(QueryablePipeline.java:119)
> at 
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.forPrimitivesIn(QueryablePipeline.java:82)
> at 
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.(GreedyPipelineFuser.java:67)
> at 
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:89)
> at 
> 

[jira] [Updated] (BEAM-5455) Don't info log for every bundle in the python sdk

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-5455:
---
Labels: portability-flink  (was: )

> Don't info log for every bundle in the python sdk
> -
>
> Key: BEAM-5455
> URL: https://issues.apache.org/jira/browse/BEAM-5455
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Micah Wylde
>Assignee: Micah Wylde
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently the python sdk logs a bunch of messages at INFO level for every 
> bundle. For the portable flink runner, which uses a bundle size of 1 in 
> streaming mode, this produces huge log spew:
> {code:java}
> [grpc-default-executor-1] INFO sdk_worker.run - Got work 4
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[timestamp.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[pair_with_one.out0, 
> coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
> 
> {code}
>  
> These log messages should probably be at debug instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=146212=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146212
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 21/Sep/18 04:07
Start Date: 21/Sep/18 04:07
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6447: [BEAM-5442] Add 
FlinkOptions to Python SDK
URL: https://github.com/apache/beam/pull/6447#issuecomment-423407276
 
 
   This change would unblock somewhat (we also need the parallelism for 
testing). But rather than limiting it to the 2 Flink parameters, why not just 
pass all options that the user specified to the runner?
   
   @robertwb can you take a look as well? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146212)
Time Spent: 20m  (was: 10m)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5441?focusedWorklogId=146210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146210
 ]

ASF GitHub Bot logged work on BEAM-5441:


Author: ASF GitHub Bot
Created on: 21/Sep/18 04:05
Start Date: 21/Sep/18 04:05
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #6461: [BEAM-5441] 
extend primitive transform check to find subtransform leaves
URL: https://github.com/apache/beam/pull/6461#issuecomment-423406984
 
 
   when I run locally:
   
   ```
   python -m apache_beam.examples.wordcount --input=/etc/profile 
--output=/tmp/py-wordcount-direct --experiments=beam_fn_api 
--runner=PortableRunner --sdk_location=container --job_endpoint=localhost:8099
   ```
   I see failures like:
   
   ```
   java.lang.Exception: The user defined 'open()' method caused an exception: 
java.lang.IllegalStateException: The passed environment does not contain a 
DockerPayload.
   at 
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
   at 
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
   at java.lang.Thread.run(Thread.java:745)
   ```
   
   ([more 
output](https://gist.github.com/ryan-williams/653861032f4c6e12e6e1b4220f753348))
   
   have no leads on what could be causing that / haven't looked yet, but wanted 
to get this out for folks to look at


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146210)
Time Spent: 20m  (was: 10m)

> Portable Wordcount fails in GreedyPipelineFuser
> ---
>
> Key: BEAM-5441
> URL: https://issues.apache.org/jira/browse/BEAM-5441
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python, sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Maximilian Michels
>Assignee: Ryan Williams
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The Python SDK wordcount with the PortableRunner throws the following 
> exception:
> {noformat}
> java.lang.IllegalArgumentException: A PCollectionNode should have exactly one 
> producing PTransformNode, PCollectionNode{id=ref_PCollection_PCollection_26, 
> PCollection=unique_name: 
> "60write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys.None"
> coder_id: "ref_Coder_FastPrimitivesCoder_2"
> is_bounded: BOUNDED
> windowing_strategy_id: "ref_Windowing_Windowing_1"
> } has 
> [PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle_32,
>  transform=spec {
>   urn: "beam:transform:generic_composite:v1"
>   payload: ""
> }
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/AddRandomKeys_33"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/ReshufflePerKey_34"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41"
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_19"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle"
> }, 
> PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41,
>  transform=spec {
>   urn: "urn:beam:transform:pardo:v1"
>   payload: "\n\317\006\n\255\006\n 
> beam:dofn:pickled_python_info:v1\032\210\006eNrFkttP1EAUxtsFZe1yUcEL4v3aRbdFUEFFQEHUEGpSnPhkxml32Gnstv3a6YZNbKIxJf7ZTvdB2ER9NZNMcs6cb3LO73zfaqbPEuYLTj3OupZMWZTtx2k3s/w45cYmC0PmhfxjypKEp1vxdmRAa36HXqBmkrEkjX2eZRjx20EYWrS6DeqnnElO9/PIl0GsFKPm0HsYszaV/YQbOEHqm3Gbf1ABTpYYc1E3d3R1arvTG2Tip6Z91bQfutbRtT2cckoYTaIfoFFinPRtkvE0s7vswN7iPbuaoCV5Ju0ej3p2GHh20pcijhatZTsLJG+pSb+wDs/sYzO3Fq0Va8Fq895CK+mrUot3OscL7CModgXFSvqYIPXVkHW9NlvD5G5jlGiYIrX9CKdLnGlKnHUx7VPq5UEog4hSo8MlkzI1MDNEIugmcSppN27noaJxjsz9Yxs4X+KCi4ukTpXcl5Ri9hCXXMyJSedPC/C5CnBZjJriN9W9z6SukLZ1bXYPV5wd/RBXFVKJayWu/w+kuQzCCukNMbm7XhNTTYXvpotbYkb8HUclwu0Sd1zcFQrCPRemguAUaJLGwFpUBJHMMD9sb/UwyKveFFEm4zQz3r2v3Pe2Shu4r7z9oECrgGWSRhAluRx8l8F2yHicy6PEgpMf4qGXSSy6WCrxyMXjEk8KLJtiXlQfrRR4WuCZKWxHDKqfe6o7lnayhPtUuWVVLOUSL1ysDXe9PpBvFHhZ4NWRfNMjI5VsS6zl3ie8LrDtOBJvrF+Bv0km\022\035ref_Environment_Environment_1"
> }
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_25"
> }
> outputs {
>   key: "None"
>   value: 

[jira] [Work logged] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5441?focusedWorklogId=146208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146208
 ]

ASF GitHub Bot logged work on BEAM-5441:


Author: ASF GitHub Bot
Created on: 21/Sep/18 04:02
Start Date: 21/Sep/18 04:02
Worklog Time Spent: 10m 
  Work Description: ryan-williams opened a new pull request #6461: 
[BEAM-5441] extend primitive transform check to find subtransform leaves
URL: https://github.com/apache/beam/pull/6461
 
 
   WIP / attempt at fixing portable python wordcount, which was broken by #6328 
   
   R: @angoenka
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146208)
Time Spent: 10m
Remaining Estimate: 0h

> Portable Wordcount fails in GreedyPipelineFuser
> ---
>
> Key: BEAM-5441
> URL: https://issues.apache.org/jira/browse/BEAM-5441
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python, sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Maximilian Michels
>Assignee: Ryan Williams
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The Python SDK wordcount with the PortableRunner throws the following 
> exception:
> {noformat}
> java.lang.IllegalArgumentException: A PCollectionNode should have exactly one 
> producing PTransformNode, PCollectionNode{id=ref_PCollection_PCollection_26, 
> PCollection=unique_name: 
> "60write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys.None"
> coder_id: "ref_Coder_FastPrimitivesCoder_2"
> is_bounded: BOUNDED
> windowing_strategy_id: "ref_Windowing_Windowing_1"

[jira] [Work logged] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4498?focusedWorklogId=146207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146207
 ]

ASF GitHub Bot logged work on BEAM-4498:


Author: ASF GitHub Bot
Created on: 21/Sep/18 04:02
Start Date: 21/Sep/18 04:02
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6458: [BEAM-4498] Sync 
javadoc and pydoc from apache/beam-site
URL: https://github.com/apache/beam/pull/6458#issuecomment-423406643
 
 
   @swegner @udim are we not considering an alternative process that does not 
involve pulling all this into the repository?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146207)
Time Spent: 0.5h  (was: 20m)

> Migrate release Javadocs / Pydocs to [asf-site] branch and update release 
> guide
> ---
>
> Key: BEAM-4498
> URL: https://issues.apache.org/jira/browse/BEAM-4498
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5455) Don't info log for every bundle in the python sdk

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5455?focusedWorklogId=146205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146205
 ]

ASF GitHub Bot logged work on BEAM-5455:


Author: ASF GitHub Bot
Created on: 21/Sep/18 03:59
Start Date: 21/Sep/18 03:59
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6459: [BEAM-5455] Log 
per-bundle messages at debug in python sdk
URL: https://github.com/apache/beam/pull/6459#issuecomment-423406323
 
 
   @robertwb this change adjusts the log level to avoid flooding the logs on 
the runner side. But it doesn't avoid the logging gRPC calls from harness to 
runner. Any thoughts on suppressing those? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146205)
Time Spent: 20m  (was: 10m)

> Don't info log for every bundle in the python sdk
> -
>
> Key: BEAM-5455
> URL: https://issues.apache.org/jira/browse/BEAM-5455
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Micah Wylde
>Assignee: Micah Wylde
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently the python sdk logs a bunch of messages at INFO level for every 
> bundle. For the portable flink runner, which uses a bundle size of 1 in 
> streaming mode, this produces huge log spew:
> {code:java}
> [grpc-default-executor-1] INFO sdk_worker.run - Got work 4
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[timestamp.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[pair_with_one.out0, 
> coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
> 
> {code}
>  
> These log messages should probably be at debug instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6042

[Apache Jenkins Server]

See 


Changes:

[chamikara] Fixes a bug that limited the size of records for TFRecord files 
that are

[chamikara] Addresses reviewer comments.

--
[...truncated 1.15 MB...]
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError

==
FAIL: test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 221, in test_apply_custom_transform
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
- >> end captured logging << -

==
FAIL: test_create (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 154, in test_create
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 

[beam] branch revert-6151-beam4780-2 deleted (was 7714ae8)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch revert-6151-beam4780-2
in repository https://gitbox.apache.org/repos/asf/beam.git.


 was 7714ae8  Revert "[BEAM-4780] Updating to DockerJobBundleFactory in 
ReferenceRunner."

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #1567

[Apache Jenkins Server]

See 


Changes:

[chamikara] Fixes a bug that limited the size of records for TFRecord files 
that are

[chamikara] Addresses reviewer comments.

--
[...truncated 762.19 MB...]
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (4/16) (207f94f0ba3bf327e800c0c3dfc1c103) 
switched from RUNNING to FINISHED.
Sep 21, 2018 3:17:22 AM org.apache.flink.runtime.taskexecutor.TaskExecutor 
unregisterTaskAndNotifyFinalState
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) a4a16facaccc63ed4fa9637a31d0a272.
Sep 21, 2018 3:17:22 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (6/16) (480fadf3e20ce3f3db2ae9a5f0527d47) 
switched from RUNNING to FINISHED.
Sep 21, 2018 3:17:22 AM org.apache.flink.runtime.taskexecutor.TaskExecutor 
unregisterTaskAndNotifyFinalState
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) 563356923dd06c117f897ae605201904.
Sep 21, 2018 3:17:22 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #89

[Apache Jenkins Server]

See 


Changes:

[chamikara] Fixes a bug that limited the size of records for TFRecord files 
that are

[chamikara] Addresses reviewer comments.

--
[...truncated 566.42 KB...]
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 275, in 
test_combine_per_key
assert_that(res, equal_to([('a', 1.5), ('b', 3.0)]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_combine_per_key_1537498864.63_c20e97eb-d45a-4b61-ace4-015db8dc8b6a failed 
in state FAILED.

==
ERROR: test_create (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 63, in 
test_create
assert_that(p | beam.Create(['a', 'b']), equal_to(['a', 'b']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_create_1537498864.99_2b40f1bc-e5b3-4f27-8b35-e0990643b0f0 failed in state 
FAILED.

==
ERROR: test_flatten (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 268, in 
test_flatten
assert_that(res, equal_to(['a', 'b', 'c', 'd']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flatten_1537498865.48_4b9df673-1854-4082-9c17-233ca4d548c6 failed in state 
FAILED.

==
ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 191, in 
test_flattened_side_input
equal_to([(None, {'a': 1, 'b': 2})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flattened_side_input_1537498865.98_4dfcdce2-c89b-42e8-8b38-68a09ebc13de 
failed in state FAILED.

==
ERROR: test_gbk_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 199, in 
test_gbk_side_input
equal_to([(None, {'a': [1]})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_gbk_side_input_1537498866.48_cffa9af4-dfd1-4d62-aac1-b5f1d2fce0f3 failed 
in state FAILED.

==
ERROR: test_group_by_key (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 261, in 
test_group_by_key
assert_that(res, equal_to([('a', [1, 2]), ('b', [3])]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_group_by_key_1537498866.96_9f76e8e7-7f4d-459d-9f75-9dc3e36c3174 failed in 
state FAILED.

==
ERROR: test_large_elements (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 317, in 
test_large_elements
assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
  File 

[jira] [Resolved] (BEAM-5408) (Java) Using Compression.GZIP with TFRecordIO

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-5408.
--
   Resolution: Fixed
Fix Version/s: 2.8.0

> (Java) Using Compression.GZIP with TFRecordIO
> -
>
> Key: BEAM-5408
> URL: https://issues.apache.org/jira/browse/BEAM-5408
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: haden lee
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: 2.8.0
>
>
> In short, `TFRecrdIO.read()` does not seem to work if the entry being read is 
> longer than 8,192 (in terms of byte[] length).  `TFRecordIO.write()` seems to 
> be OK with this though (based on some experiments). Perhaps there is some 
> hard-coded value for this specific length somewhere in the SDK, and I'm 
> wondering if it can be increased or parameterized. 
> [I've posted this on 
> StackOverflow|https://stackoverflow.com/questions/52284639/beam-java-sdk-with-tfrecord-and-compression-gzip],
>  but I was advised to report it here.
> Here are the details:
> We're using Beam Java SDK (and Google Cloud Dataflow to run batch jobs) a 
> lot, and we noticed something weird (possibly a bug?) when we tried to use 
> `TFRecordIO` with `Compression.GZIP`. We were able to come up with some 
> sample code that can reproduce the errors we face.
> To be clear, we are using Beam Java SDK 2.4.
> Suppose we have `PCollection` which can be a PC of proto messages, 
> for instance, in byte[] format.
>  We usually write this to GCS (Google Cloud Storage) using Base64 encoding 
> (newline delimited Strings) or using TFRecordIO (without compression). We 
> have had no issue reading the data from GCS in this manner for a very long 
> time (2.5+ years for the former and ~1.5 years for the latter).
> Recently, we tried `TFRecordIO` with `Compression.GZIP` option, and 
> *sometimes* we get an exception as the data is seen as invalid (while being 
> read). The data itself (the gzip files) is not corrupted, and we've tested 
> various things, and reached the following conclusion.
> When a `byte[]` that is being compressed under `TFRecordIO` is above certain 
> threshold (I'd say when at or above 8192), then 
> `TFRecordIO.read().withCompression(Compression.GZIP)` would not work.
>  Specifically, it will throw the following exception:
>  
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.IllegalStateException: Invalid data
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> at org.apache.beam.sdk.io.TFRecordIO$TFRecordCodec.read(TFRecordIO.java:642)
> at 
> org.apache.beam.sdk.io.TFRecordIO$TFRecordSource$TFRecordReader.readNextRecord(TFRecordIO.java:526)
> at 
> org.apache.beam.sdk.io.CompressedSource$CompressedReader.readNextRecord(CompressedSource.java:426)
> at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:473)
> at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:468)
> at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
> at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:141)
> at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:161)
> at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:125)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>  
> This can be reproduced easily, so you can refer to the code at the end. You 
> will also see comments about the byte array length (as I tested with various 
> sizes, I concluded that 8192 is the magic number).
> So I'm wondering if this is a bug or known issue – I couldn't find anything 
> close to this on Apache Beam's Issue Tracker [here][1] but if there is 
> another forum/site I need to check, please let me know!
>  If this is indeed a bug, what would be the right channel to report this?
> —
>  The following code can reproduce the error we have.
> A successful run (with parameters 1, 39, 100) would show the following 
> message at the end:
> {code:java}
> // code placeholder
>  counter metrics from CountDoFn
> [counter] plain_base64_proto_array_len: 8126
> [counter] plain_base64_proto_in: 1
> [counter] plain_base64_proto_val_cnt: 39
> [counter] 

[jira] [Resolved] (BEAM-5412) TFRecordIO fails with records larger than 8K

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-5412.
--
   Resolution: Fixed
Fix Version/s: 2.8.0

> TFRecordIO fails with records larger than 8K
> 
>
> Key: BEAM-5412
> URL: https://issues.apache.org/jira/browse/BEAM-5412
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-text
>Affects Versions: 2.4.0
>Reporter: Raghu Angadi
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> This was reported on 
> [Stackoverflow|https://stackoverflow.com/questions/52284639/beam-java-sdk-with-tfrecord-and-compression-gzip].
>  TFRecordIO reader assumes a single call to {{channel.read()}} returns as 
> much as can fit in the input buffer. {{read()}} can return fewer bytes than 
> requested. Assert failure : 
> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L642



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (9d0b03a -> 6cef8ad)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9d0b03a  [Lint] rename variable in graphx/translate.go
 add c585eba  Fixes a bug that limited the size of records for TFRecord 
files that are GZIP compressed.
 add 140ff77  Addresses reviewer comments.
 new 6cef8ad  Merge pull request #6440: [BEAM-5412][BEAM-5408] Fixes a bug 
that limited the size of TFRecords

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/io/TFRecordIO.java| 27 ++
 .../org/apache/beam/sdk/io/TFRecordIOTest.java | 24 ---
 2 files changed, 44 insertions(+), 7 deletions(-)

[jira] [Created] (BEAM-5456) Update google-api-client libraries to 1.25

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-5456:


 Summary: Update google-api-client libraries to 1.25
 Key: BEAM-5456
 URL: https://issues.apache.org/jira/browse/BEAM-5456
 Project: Beam
  Issue Type: Improvement
  Components: io-java-gcp
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath
 Fix For: 2.8.0


This version updates authentication URLs 
([https://github.com/googleapis/google-api-java-client/releases)] that is 
needed for certain features.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5412) TFRecordIO fails with records larger than 8K

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5412?focusedWorklogId=146197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146197
 ]

ASF GitHub Bot logged work on BEAM-5412:


Author: ASF GitHub Bot
Created on: 21/Sep/18 02:54
Start Date: 21/Sep/18 02:54
Worklog Time Spent: 10m 
  Work Description: chamikaramj closed pull request #6440: 
[BEAM-5412][BEAM-5408] Fixes a bug that limited the size of TFRecords
URL: https://github.com/apache/beam/pull/6440
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java
index a6ac486bba0..274df369d72 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java
@@ -619,20 +619,39 @@ public int recordLength(byte[] data) {
 return null;
   }
   checkState(headerBytes == HEADER_LEN, "Not a valid TFRecord. Fewer than 
12 bytes.");
+
   header.rewind();
   long length = header.getLong();
+  long lengthHash = hashLong(length);
   int maskedCrc32OfLength = header.getInt();
-  checkState(hashLong(length) == maskedCrc32OfLength, "Mismatch of length 
mask");
+  if (lengthHash != maskedCrc32OfLength) {
+throw new IOException(
+String.format(
+"Mistmatch of length mask when reading a record. Expected %d 
but received %d.",
+maskedCrc32OfLength, lengthHash));
+  }
 
   ByteBuffer data = ByteBuffer.allocate((int) length);
-  checkState(inChannel.read(data) == length, "Invalid data");
+  while (data.hasRemaining() && inChannel.read(data) >= 0) {}
+  if (data.hasRemaining()) {
+throw new IOException(
+String.format(
+"EOF while reading record of length %d. Read only %d bytes. 
Input might be truncated.",
+length, data.position()));
+  }
 
   footer.clear();
   inChannel.read(footer);
   footer.rewind();
-  int maskedCrc32OfData = footer.getInt();
 
-  checkState(hashBytes(data.array()) == maskedCrc32OfData, "Mismatch of 
data mask");
+  int maskedCrc32OfData = footer.getInt();
+  int dataHash = hashBytes(data.array());
+  if (dataHash != maskedCrc32OfData) {
+throw new IOException(
+String.format(
+"Mistmatch of data mask when reading a record. Expected %d but 
received %d.",
+maskedCrc32OfData, dataHash));
+  }
   return data.array();
 }
 
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
index 5ff5b1c880f..fbd69a3f898 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
@@ -84,7 +84,8 @@
   private static final String[] FOO_BAR_RECORDS = {"foo", "bar"};
 
   private static final Iterable EMPTY = Collections.emptyList();
-  private static final Iterable LARGE = makeLines(1000);
+  private static final Iterable LARGE = makeLines(1000, 4);
+  private static final Iterable LARGE_RECORDS = makeLines(100, 10);
 
   @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
 
@@ -287,6 +288,18 @@ public void runTestRoundTripZlibFilesWithAuto() throws 
IOException {
 runTestRoundTrip(LARGE, 10, ".tfrecords", DEFLATE, AUTO);
   }
 
+  @Test
+  @Category(NeedsRunner.class)
+  public void runTestRoundTripLargeRecords() throws IOException {
+runTestRoundTrip(LARGE_RECORDS, 10, ".tfrecords", UNCOMPRESSED, 
UNCOMPRESSED);
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void runTestRoundTripLargeRecordsGzip() throws IOException {
+runTestRoundTrip(LARGE_RECORDS, 10, ".tfrecords", GZIP, GZIP);
+  }
+
   private void runTestRoundTrip(
   Iterable elems,
   int numShards,
@@ -344,10 +357,15 @@ private void runTestRoundTrip(
 readPipeline.run();
   }
 
-  private static Iterable makeLines(int n) {
+  private static Iterable makeLines(int n, int minRecordSize) {
 List ret = Lists.newArrayList();
+StringBuilder recordBuilder = new StringBuilder();
+for (int i = 0; i < minRecordSize; i++) {
+  recordBuilder.append("x");
+}
+String record = recordBuilder.toString();
 for (int i = 0; i < n; ++i) {
-  ret.add("word" + i);
+  ret.add(record + " " + i);
 }
 return ret;
   }


 


This is an automated message 

[beam] 01/01: Merge pull request #6440: [BEAM-5412][BEAM-5408] Fixes a bug that limited the size of TFRecords

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6cef8ade91bd88b86a22c9e868d53752f880cc98
Merge: 9d0b03a 140ff77
Author: Chamikara Jayalath 
AuthorDate: Thu Sep 20 19:54:40 2018 -0700

Merge pull request #6440: [BEAM-5412][BEAM-5408] Fixes a bug that limited 
the size of TFRecords

 .../java/org/apache/beam/sdk/io/TFRecordIO.java| 27 ++
 .../org/apache/beam/sdk/io/TFRecordIOTest.java | 24 ---
 2 files changed, 44 insertions(+), 7 deletions(-)

[jira] [Comment Edited] (BEAM-5305) Timeout handling in JDBCIO

[Byambasuren Ganbaatar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622986#comment-16622986
 ] 

Byambasuren Ganbaatar edited comment on BEAM-5305 at 9/21/18 2:49 AM:
--

[~jbonofre] Could respond this issue here? thanks.


was (Author: bya):
[~jbonofre]Could respond this issue here? thanks.

> Timeout handling in JDBCIO
> --
>
> Key: BEAM-5305
> URL: https://issues.apache.org/jira/browse/BEAM-5305
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-jdbc
>Affects Versions: 2.5.0
>Reporter: Byambasuren Ganbaatar
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>  Labels: JdbcIO, oracle
>
> Hello, we have a batch pipeline that is reading data from Oracle, writing it 
> to BigQuery.
> Using JdbcIO, BigQueryIO and Oracle driver [3].
> We run this job 2 times in a day, the pipeline usually works fine, it 
> completed within 10 minutes.
> But sometimes we have a [1] warning from this job, it never finished until we 
> cancelling it.
> (sometimes means 4 times in last month)
> Even we set timeout configuration for our code [2], warning log says [1] 
> Processing stuck in step Read JDBC.
>  
> Would you explain and correct us this kind of problem? 
> We already contacted Google Support Team, they said our configuration code is 
> looked fine them. Then  they instructed us to ask here.
> Thanks,
> Bya
> [1] Warning Log
> {code:java}
> Processing stuck in step Read JDBC/JdbcIO.Read/JdbcIO.ReadAll/ParDo(Read) for 
> at least 13h40m00s without outputting or completing in state process at 
> sun.nio.ch.FileDispatcherImpl.read0(Native Method) at 
> sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at 
> sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at 
> sun.nio.ch.IOUtil.read(IOUtil.java:197) at 
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at 
> oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144) at 
> oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82) at 
> oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139) at 
> oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101) at 
> oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
>  at 
> oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98) 
> at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534) 
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485) at 
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252) at 
> oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612) at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
>  at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59) 
> at 
> oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:747)
>  at 
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:904)
>  at 
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1082)
>  at 
> oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
>  at 
> oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
>  at 
> oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3822)
>  at 
> oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.processElement(JdbcIO.java:601) 
> at 
> org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source)
> {code}
>  
> [2] Jdbc configure
> {code:java}
> import org.apache.beam.sdk.io.jdbc.JdbcIO;
> import oracle.jdbc.driver.OracleConnection;
> JdbcIO.DataSourceConfiguration  dataSource = JdbcIO
> 
> .DataSourceConfiguration.create("oracle.jdbc.driver.OracleDriver", oracleUrl)
> .withUsername(oracleUser)
> .withPassword(oraclePassword)
> .withConnectionProperties(String.format("%s=%s",
> 
> OracleConnection.CONNECTION_PROPERTY_THIN_NET_CONNECT_TIMEOUT, 1))
> .withConnectionProperties(String.format("%s=%s",
> 
> OracleConnection.CONNECTION_PROPERTY_THIN_READ_TIMEOUT, 

[jira] [Commented] (BEAM-5305) Timeout handling in JDBCIO

[Byambasuren Ganbaatar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622986#comment-16622986
 ] 

Byambasuren Ganbaatar commented on BEAM-5305:
-

[~jbonofre]Could respond this issue here? thanks.

> Timeout handling in JDBCIO
> --
>
> Key: BEAM-5305
> URL: https://issues.apache.org/jira/browse/BEAM-5305
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-jdbc
>Affects Versions: 2.5.0
>Reporter: Byambasuren Ganbaatar
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>  Labels: JdbcIO, oracle
>
> Hello, we have a batch pipeline that is reading data from Oracle, writing it 
> to BigQuery.
> Using JdbcIO, BigQueryIO and Oracle driver [3].
> We run this job 2 times in a day, the pipeline usually works fine, it 
> completed within 10 minutes.
> But sometimes we have a [1] warning from this job, it never finished until we 
> cancelling it.
> (sometimes means 4 times in last month)
> Even we set timeout configuration for our code [2], warning log says [1] 
> Processing stuck in step Read JDBC.
>  
> Would you explain and correct us this kind of problem? 
> We already contacted Google Support Team, they said our configuration code is 
> looked fine them. Then  they instructed us to ask here.
> Thanks,
> Bya
> [1] Warning Log
> {code:java}
> Processing stuck in step Read JDBC/JdbcIO.Read/JdbcIO.ReadAll/ParDo(Read) for 
> at least 13h40m00s without outputting or completing in state process at 
> sun.nio.ch.FileDispatcherImpl.read0(Native Method) at 
> sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at 
> sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at 
> sun.nio.ch.IOUtil.read(IOUtil.java:197) at 
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at 
> oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144) at 
> oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82) at 
> oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139) at 
> oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101) at 
> oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
>  at 
> oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98) 
> at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534) 
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485) at 
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252) at 
> oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612) at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
>  at 
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59) 
> at 
> oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:747)
>  at 
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:904)
>  at 
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1082)
>  at 
> oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
>  at 
> oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
>  at 
> oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3822)
>  at 
> oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at 
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
>  at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.processElement(JdbcIO.java:601) 
> at 
> org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source)
> {code}
>  
> [2] Jdbc configure
> {code:java}
> import org.apache.beam.sdk.io.jdbc.JdbcIO;
> import oracle.jdbc.driver.OracleConnection;
> JdbcIO.DataSourceConfiguration  dataSource = JdbcIO
> 
> .DataSourceConfiguration.create("oracle.jdbc.driver.OracleDriver", oracleUrl)
> .withUsername(oracleUser)
> .withPassword(oraclePassword)
> .withConnectionProperties(String.format("%s=%s",
> 
> OracleConnection.CONNECTION_PROPERTY_THIN_NET_CONNECT_TIMEOUT, 1))
> .withConnectionProperties(String.format("%s=%s",
> 
> OracleConnection.CONNECTION_PROPERTY_THIN_READ_TIMEOUT, 60))
> .withConnectionProperties(String.format("%s=%s",
> 
> 

[jira] [Assigned] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

[Ryan Williams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Williams reassigned BEAM-5441:
---

Assignee: Ryan Williams

> Portable Wordcount fails in GreedyPipelineFuser
> ---
>
> Key: BEAM-5441
> URL: https://issues.apache.org/jira/browse/BEAM-5441
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python, sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Maximilian Michels
>Assignee: Ryan Williams
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>
> The Python SDK wordcount with the PortableRunner throws the following 
> exception:
> {noformat}
> java.lang.IllegalArgumentException: A PCollectionNode should have exactly one 
> producing PTransformNode, PCollectionNode{id=ref_PCollection_PCollection_26, 
> PCollection=unique_name: 
> "60write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys.None"
> coder_id: "ref_Coder_FastPrimitivesCoder_2"
> is_bounded: BOUNDED
> windowing_strategy_id: "ref_Windowing_Windowing_1"
> } has 
> [PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle_32,
>  transform=spec {
>   urn: "beam:transform:generic_composite:v1"
>   payload: ""
> }
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/AddRandomKeys_33"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/ReshufflePerKey_34"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41"
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_19"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle"
> }, 
> PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41,
>  transform=spec {
>   urn: "urn:beam:transform:pardo:v1"
>   payload: "\n\317\006\n\255\006\n 
> beam:dofn:pickled_python_info:v1\032\210\006eNrFkttP1EAUxtsFZe1yUcEL4v3aRbdFUEFFQEHUEGpSnPhkxml32Gnstv3a6YZNbKIxJf7ZTvdB2ER9NZNMcs6cb3LO73zfaqbPEuYLTj3OupZMWZTtx2k3s/w45cYmC0PmhfxjypKEp1vxdmRAa36HXqBmkrEkjX2eZRjx20EYWrS6DeqnnElO9/PIl0GsFKPm0HsYszaV/YQbOEHqm3Gbf1ABTpYYc1E3d3R1arvTG2Tip6Z91bQfutbRtT2cckoYTaIfoFFinPRtkvE0s7vswN7iPbuaoCV5Ju0ej3p2GHh20pcijhatZTsLJG+pSb+wDs/sYzO3Fq0Va8Fq895CK+mrUot3OscL7CModgXFSvqYIPXVkHW9NlvD5G5jlGiYIrX9CKdLnGlKnHUx7VPq5UEog4hSo8MlkzI1MDNEIugmcSppN27noaJxjsz9Yxs4X+KCi4ukTpXcl5Ri9hCXXMyJSedPC/C5CnBZjJriN9W9z6SukLZ1bXYPV5wd/RBXFVKJayWu/w+kuQzCCukNMbm7XhNTTYXvpotbYkb8HUclwu0Sd1zcFQrCPRemguAUaJLGwFpUBJHMMD9sb/UwyKveFFEm4zQz3r2v3Pe2Shu4r7z9oECrgGWSRhAluRx8l8F2yHicy6PEgpMf4qGXSSy6WCrxyMXjEk8KLJtiXlQfrRR4WuCZKWxHDKqfe6o7lnayhPtUuWVVLOUSL1ysDXe9PpBvFHhZ4NWRfNMjI5VsS6zl3ie8LrDtOBJvrF+Bv0km\022\035ref_Environment_Environment_1"
> }
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_25"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys"
> }]
> at 
> org.apache.beam.repackaged.beam_runners_core_construction_java.com.google.common.base.Preconditions.checkArgument(Preconditions.java:416)
> at 
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.buildNetwork(QueryablePipeline.java:176)
> at 
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.(QueryablePipeline.java:119)
> at 
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.forPrimitivesIn(QueryablePipeline.java:82)
> at 
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.(GreedyPipelineFuser.java:67)
> at 
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:89)
> at 
> org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:96)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like it was caused by https://github.com/apache/beam/pull/6328



--

[jira] [Commented] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

[Ryan Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622983#comment-16622983
 ] 

Ryan Williams commented on BEAM-5441:
-

i'm debugging this a bit; here's the structure of the wordcount pipeline:

There's a transform named {{write/Write/WriteImpl/DoOnce/Read}} with URN 
"{{beam:transform:read:v1}}" and 4 subtransforms:
{code:java}
"ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Impulse_30"
"ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Split_31"
"ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle_32"
"ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/ReadSplits_42"{code}
The 3rd one, {{Reshuffle_32}} (URN {{beam:transform:generic_composite:v1}}) has 
3 subtransforms:
{code:java}
"ref_AppliedPTransform_read/Read/Reshuffle/AddRandomKeys_7"
"ref_AppliedPTransform_read/Read/Reshuffle/ReshufflePerKey_8"
"ref_AppliedPTransform_read/Read/Reshuffle/RemoveRandomKeys_15"{code}
The top-level {{Read}} transform would normally be considered "primitive" 
[based on its 
URN|https://github.com/apache/beam/blob/9d0b03ad736797c33fbd6a4eb13155fa91367779/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java#L150].

However, [there's a special 
case|https://github.com/apache/beam/blob/9d0b03ad736797c33fbd6a4eb13155fa91367779/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java#L131-L135]
 in the "is primitive?" logic that says: if the "primitive" transform has 
subtransforms, use all of them instead of their parent.

This was added to fix a unittest ([PR 
discussion|https://github.com/apache/beam/pull/6328#issuecomment-419740410]): 
DirectRunner [replaces a 
GBK|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java#L264-L266]
 (normally a primitive transform) with [a wrapper for two runner-specific 
transforms ("group by key only" and "group also by 
window")|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectGroupByKey.java#L63-L70],
 and the expectation is that the wrapped transforms are considered 
"primitives", but the wrapper is not; the current implementation achieves that.

However, in the wordcount case, the {{Reshuffle}} gets added as a primitive 
transform (being a child of the otherwise-primitive {{Read}}) and, separately, 
its child {{RemoveRandomKeys}} (URN {{urn:beam:transform:pardo:v1}}) gets 
marked primitive, so there are two "primitive" transforms that have the same 
output {{PCollection}}, failing [the 
assertion|https://github.com/apache/beam/blob/9d0b03ad736797c33fbd6a4eb13155fa91367779/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java#L176-L182].

I _think_ the right fix is to just recurse the subtransform check: "leaf 
descendents" of a transform with a primitive URN are the things we should pass 
along.

I'll work on that and report back here, am interested in anyone else's thoughts 
as well.

> Portable Wordcount fails in GreedyPipelineFuser
> ---
>
> Key: BEAM-5441
> URL: https://issues.apache.org/jira/browse/BEAM-5441
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python, sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>
> The Python SDK wordcount with the PortableRunner throws the following 
> exception:
> {noformat}
> java.lang.IllegalArgumentException: A PCollectionNode should have exactly one 
> producing PTransformNode, PCollectionNode{id=ref_PCollection_PCollection_26, 
> PCollection=unique_name: 
> "60write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys.None"
> coder_id: "ref_Coder_FastPrimitivesCoder_2"
> is_bounded: BOUNDED
> windowing_strategy_id: "ref_Windowing_Windowing_1"
> } has 
> [PTransformNode{id=ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle_32,
>  transform=spec {
>   urn: "beam:transform:generic_composite:v1"
>   payload: ""
> }
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/AddRandomKeys_33"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/ReshufflePerKey_34"
> subtransforms: 
> "ref_AppliedPTransform_write/Write/WriteImpl/DoOnce/Read/Reshuffle/RemoveRandomKeys_41"
> inputs {
>   key: "0"
>   value: "ref_PCollection_PCollection_19"
> }
> outputs {
>   key: "None"
>   value: "ref_PCollection_PCollection_26"
> }
> unique_name: "write/Write/WriteImpl/DoOnce/Read/Reshuffle"
> }, 
> 

[jira] [Assigned] (BEAM-5455) Don't info log for every bundle in the python sdk

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise reassigned BEAM-5455:
--

Assignee: Micah Wylde  (was: Robert Bradshaw)

> Don't info log for every bundle in the python sdk
> -
>
> Key: BEAM-5455
> URL: https://issues.apache.org/jira/browse/BEAM-5455
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Micah Wylde
>Assignee: Micah Wylde
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently the python sdk logs a bunch of messages at INFO level for every 
> bundle. For the portable flink runner, which uses a bundle size of 1 in 
> streaming mode, this produces huge log spew:
> {code:java}
> [grpc-default-executor-1] INFO sdk_worker.run - Got work 4
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[timestamp.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[pair_with_one.out0, 
> coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
> 
> {code}
>  
> These log messages should probably be at debug instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6041

[Apache Jenkins Server]

See 


Changes:

[github] [Lint] rename variable in graphx/translate.go

--
[...truncated 1.16 MB...]
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError

==
FAIL: test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 221, in test_apply_custom_transform
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
- >> end captured logging << -

==
FAIL: test_create (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 154, in test_create
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 

[jira] [Work logged] (BEAM-5455) Don't info log for every bundle in the python sdk

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5455?focusedWorklogId=146186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146186
 ]

ASF GitHub Bot logged work on BEAM-5455:


Author: ASF GitHub Bot
Created on: 21/Sep/18 01:25
Start Date: 21/Sep/18 01:25
Worklog Time Spent: 10m 
  Work Description: mwylde opened a new pull request #6459: [BEAM-5455] Log 
per-bundle messages at debug in python sdk
URL: https://github.com/apache/beam/pull/6459
 
 
   This PR changes several log statements in the python SDK from info to debug. 
Without this change, the sdk can log at info level 10+ messages per bundle. For 
the portable flink runtime this means many logs per message consumed by the 
pipeline.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146186)
Time Spent: 10m
Remaining Estimate: 0h

> Don't info log for every bundle in the python sdk
> -
>
> Key: BEAM-5455
> URL: https://issues.apache.org/jira/browse/BEAM-5455
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>

[jira] [Created] (BEAM-5455) Don't info log for every bundle in the python sdk

[Micah Wylde (JIRA)]

Micah Wylde created BEAM-5455:
-

 Summary: Don't info log for every bundle in the python sdk
 Key: BEAM-5455
 URL: https://issues.apache.org/jira/browse/BEAM-5455
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-harness
Reporter: Micah Wylde
Assignee: Robert Bradshaw


Currently the python sdk logs a bunch of messages at INFO level for every 
bundle. For the portable flink runner, which uses a bundle size of 1 in 
streaming mode, this produces huge log spew:
{code:java}
[grpc-default-executor-1] INFO sdk_worker.run - Got work 4
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

{code}
 

These log messages should probably be at debug instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #1566

[Apache Jenkins Server]

See 


--
[...truncated 763.36 MB...]
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) 618b00779f628f9ddc1e5d2dba4846d3.
Sep 21, 2018 12:29:18 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (1/16) (5bb7cee9e214d665d1f83ecdd9c4bf1a) 
switched from RUNNING to FINISHED.
Sep 21, 2018 12:29:18 AM org.apache.flink.runtime.taskexecutor.TaskExecutor 
unregisterTaskAndNotifyFinalState
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) 2223a35f9cacef0a276cc3bdec3e9b8b.
Sep 21, 2018 12:29:18 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (8/16) (21e71abfdc9629e39aeaed2ac6ff214b) 
switched from RUNNING to FINISHED.
Sep 21, 2018 12:29:18 AM org.apache.flink.runtime.taskexecutor.TaskExecutor 
unregisterTaskAndNotifyFinalState
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 

Build failed in Jenkins: beam_PreCommit_Python_Cron #370

[Apache Jenkins Server]

See 


Changes:

[herohde] Add Go quickstart link to the footer

[xiliu] [BEAM-3079]: Samza Runner docs and capability matrix

[xiliu] Update the option docs

[xinyuliu.us] Add Samza to the runner list

[klk] Add a pull request template with staged URL

[apilloud] Add SQL Lexical Structure from BigQuery

[mergebot] Add Apache license headers

[apilloud] Update keywords from BeamSqlParserImplConstants

[apilloud] Update CREATE TABLE from doc

[apilloud] Move CREATE TABLE and SELECT up a level

[mergebot] Update contact-us.md

[mergebot] Spelling correction

[apilloud] Add redirect to new team page

[apilloud] Add SQL Set page

[boyuanz] Update release guide for download page changes

[klk] Make Samza icon a reasonable size

[klk] Alphabetize runners

[mergebot] New blog post about 2.5.0 release

[amaliujia] Update doc

[echauchot] [BEAM-4731] update flink submission port because flink 1.5 new 
standard

[mergebot] Adding YouTube channel page

[mergebot] Processing feedback

[mergebot] YouTube url update

[lcwik] [BEAM-3940] Remove reference to updating container version in pom.xml

[lcwik] Fix typos in programming guide

[lcwik] Fix typos in mobile gaming example

[kirpichov] Updates SDF status in capability matrix

[mergebot] Added new design documents

[mergebot] [BEAM-4430] Improve Performance Testing Documentation

[mergebot] Rename RowType to Schema in SQL walkthrough

[mergebot] Minor tweak to Intellij instructions.

[mergebot] fixup

[melissapa] [BEAM-2977] Improve unbounded prose in wordcount example

[mergebot] [BEAM-4188] Fix 2.4.0 release notes link

[mergebot] Update code style section to mention Spotless.

[mergebot] Clarify that this is only for Java/Groovy and not Python/Go/...

[lcwik] [BEAM-4828] Updated documentation, added SQS BEAM-4828 to In-Progress

[github] Update Python quickstart with Windows instructions

[mergebot] Add courtesy unsubscribe links to mailing lists

[melissapa] Fix shell tabs, add new shell tab switcher

[mergebot] Add a page for release cadence

[mergebot] updated the language

[mergebot] fix the space

[mergebot] [BEAM-4827] Add Nexmark dashboards to nexmark page on the website

[mergebot] [BEAM-4827] Fix typos

[mergebot] Add Flink instructions to portability contrib page.

[mergebot] Add instructions for local Flink cluster.

[mergebot] Update Java Quickstart for Windows

[mergebot] Update quickstart-java.md

[github] Update quickstart-java.md

[mergebot] [BEAM-4802] Update the contribution guide (Dependency section)

[migryz] Add post-commit tests policies summary page.

[migryz] Fix links

[github] Remove [TOC]

[migryz] Update postcommits guides

[ehudm] Remove testing matrix and simplify intro.

[mergebot] Moves KuduIO from in progress to available

[timrobertson100] [BEAM-4260] Document HCatalogIO use with Hive 1.1

[mergebot] Remove old pydoc and javadoc for versions < 2.0.0.

[apilloud] [BEAM-4761] Add Dataflow Runner

[mergebot] Update beam-site for release 2.6.0

[mergebot] Update download page

[mergebot] Updating downloads page.

[melissapa] Fix broken 2.5.0 download links

[mergebot] go version

[mergebot] Add section for SQL shell

[mergebot] Adding 'editable mode' flag installation for Python.

[thw] fix job service startup instruction

[mergebot] Fix Apex Runner instructions for execution on YARN

[mergebot] Update SQL walkthrough for 2.6.0

[github] Adding Pablo Estrada as author for blog

[github] Blog post for Beam 2.6.0 release

[github] Added Rafael Fernandez as author.

[rfernand] Adds Jenkinns job pointer to #building--testing

[rfernand] Minor rewrite to help newcomers explore the project's building and

[thw] fixup

[mergebot] Add instructions of using automation scripts

[mergebot] Fix broken tests

[mergebot] Addressed Ahmet's comments

[mergebot] Fixing 2.6.0 blog post

[mergebot] Quick fixes to layout

[mergebot] Address readability reviews

[mergebot] Grammar pass + pretty print

[mergebot] All files have Apache License

[mergebot] Add instructions about how to build python wheels

[mergebot] Add long term support release concept

[mergebot] Edit suggestions

[mergebot] Add precommit policies and triage guide.

[mergebot] Add dashes to pre-commit and post-commit.

[mergebot] Update programming guide to suggest using NewDoFn approach to

[mergebot] Address comments.

[mergebot] Add compatibility warning.

[mergebot] Fix language tab tags

[mergebot] Add blog post "A review of input streaming connectors"

[mergebot] Add authors for blog post in

[mergebot] Fix typo for author's name in blog post

[mergebot] Fix other typo in author's name for blog post

[mergebot] Blog post updates based on @iemejia's feedback

[mergebot] Updates to streaming connectors blog post

[mergebot] Set publication date for streaming connectors blog post

[mergebot] Update doc links in blog post to point to latest release

[mergebot] Fix extraneous p tag and add table 

[jira] [Work logged] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4498?focusedWorklogId=146184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146184
 ]

ASF GitHub Bot logged work on BEAM-4498:


Author: ASF GitHub Bot
Created on: 21/Sep/18 00:19
Start Date: 21/Sep/18 00:19
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #6458: [BEAM-4498] Sync javadoc 
and pydoc from apache/beam-site
URL: https://github.com/apache/beam/pull/6458#issuecomment-423375047
 
 
   R: @swegner 
   
   The flow as I see it:
   - Release script generates 2 PRs:
 - The first one generates pydoc and javadoc dirs (like this one), and is 
to be merged to branch asf-site.
 - The second updates 
`website/documentation/sdks/javadoc/{index,current}.md`, and is to be merge to 
branch master.
   - Post-commit process generates new files from the second PR and puts them 
in asf-site.
   - Apache website publishing process picks up changes from asf-site branch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146184)
Time Spent: 20m  (was: 10m)

> Migrate release Javadocs / Pydocs to [asf-site] branch and update release 
> guide
> ---
>
> Key: BEAM-4498
> URL: https://issues.apache.org/jira/browse/BEAM-4498
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1463

[Apache Jenkins Server]

See 


Changes:

[herohde] Add Go quickstart link to the footer

[xiliu] [BEAM-3079]: Samza Runner docs and capability matrix

[xiliu] Update the option docs

[xinyuliu.us] Add Samza to the runner list

[klk] Add a pull request template with staged URL

[apilloud] Add SQL Lexical Structure from BigQuery

[mergebot] Add Apache license headers

[apilloud] Update keywords from BeamSqlParserImplConstants

[apilloud] Update CREATE TABLE from doc

[apilloud] Move CREATE TABLE and SELECT up a level

[mergebot] Update contact-us.md

[mergebot] Spelling correction

[apilloud] Add redirect to new team page

[apilloud] Add SQL Set page

[boyuanz] Update release guide for download page changes

[klk] Make Samza icon a reasonable size

[klk] Alphabetize runners

[mergebot] New blog post about 2.5.0 release

[amaliujia] Update doc

[echauchot] [BEAM-4731] update flink submission port because flink 1.5 new 
standard

[mergebot] Adding YouTube channel page

[mergebot] Processing feedback

[mergebot] YouTube url update

[lcwik] [BEAM-3940] Remove reference to updating container version in pom.xml

[lcwik] Fix typos in programming guide

[lcwik] Fix typos in mobile gaming example

[kirpichov] Updates SDF status in capability matrix

[mergebot] Added new design documents

[mergebot] [BEAM-4430] Improve Performance Testing Documentation

[mergebot] Rename RowType to Schema in SQL walkthrough

[mergebot] Minor tweak to Intellij instructions.

[mergebot] fixup

[melissapa] [BEAM-2977] Improve unbounded prose in wordcount example

[mergebot] [BEAM-4188] Fix 2.4.0 release notes link

[mergebot] Update code style section to mention Spotless.

[mergebot] Clarify that this is only for Java/Groovy and not Python/Go/...

[lcwik] [BEAM-4828] Updated documentation, added SQS BEAM-4828 to In-Progress

[github] Update Python quickstart with Windows instructions

[mergebot] Add courtesy unsubscribe links to mailing lists

[melissapa] Fix shell tabs, add new shell tab switcher

[mergebot] Add a page for release cadence

[mergebot] updated the language

[mergebot] fix the space

[mergebot] [BEAM-4827] Add Nexmark dashboards to nexmark page on the website

[mergebot] [BEAM-4827] Fix typos

[mergebot] Add Flink instructions to portability contrib page.

[mergebot] Add instructions for local Flink cluster.

[mergebot] Update Java Quickstart for Windows

[mergebot] Update quickstart-java.md

[github] Update quickstart-java.md

[mergebot] [BEAM-4802] Update the contribution guide (Dependency section)

[migryz] Add post-commit tests policies summary page.

[migryz] Fix links

[github] Remove [TOC]

[migryz] Update postcommits guides

[ehudm] Remove testing matrix and simplify intro.

[mergebot] Moves KuduIO from in progress to available

[timrobertson100] [BEAM-4260] Document HCatalogIO use with Hive 1.1

[mergebot] Remove old pydoc and javadoc for versions < 2.0.0.

[apilloud] [BEAM-4761] Add Dataflow Runner

[mergebot] Update beam-site for release 2.6.0

[mergebot] Update download page

[mergebot] Updating downloads page.

[melissapa] Fix broken 2.5.0 download links

[mergebot] go version

[mergebot] Add section for SQL shell

[mergebot] Adding 'editable mode' flag installation for Python.

[thw] fix job service startup instruction

[mergebot] Fix Apex Runner instructions for execution on YARN

[mergebot] Update SQL walkthrough for 2.6.0

[github] Adding Pablo Estrada as author for blog

[github] Blog post for Beam 2.6.0 release

[github] Added Rafael Fernandez as author.

[rfernand] Adds Jenkinns job pointer to #building--testing

[rfernand] Minor rewrite to help newcomers explore the project's building and

[thw] fixup

[mergebot] Add instructions of using automation scripts

[mergebot] Fix broken tests

[mergebot] Addressed Ahmet's comments

[mergebot] Fixing 2.6.0 blog post

[mergebot] Quick fixes to layout

[mergebot] Address readability reviews

[mergebot] Grammar pass + pretty print

[mergebot] All files have Apache License

[mergebot] Add instructions about how to build python wheels

[mergebot] Add long term support release concept

[mergebot] Edit suggestions

[mergebot] Add precommit policies and triage guide.

[mergebot] Add dashes to pre-commit and post-commit.

[mergebot] Update programming guide to suggest using NewDoFn approach to

[mergebot] Address comments.

[mergebot] Add compatibility warning.

[mergebot] Fix language tab tags

[mergebot] Add blog post "A review of input streaming connectors"

[mergebot] Add authors for blog post in

[mergebot] Fix typo for author's name in blog post

[mergebot] Fix other typo in author's name for blog post

[mergebot] Blog post updates based on @iemejia's feedback

[mergebot] Updates to streaming connectors blog post

[mergebot] Set publication date for streaming connectors blog post

[mergebot] Update doc links in blog post to point to latest release

[mergebot] Fix extraneous p tag and add 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #1565

[Apache Jenkins Server]

See 


Changes:

[github] [Lint] rename variable in graphx/translate.go

--
[...truncated 763.75 MB...]
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) 0d1b0654f3b038895a883f215fb7089b.
Sep 21, 2018 12:08:38 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (8/16) (6f52baf9979f27d4b47c53ab83891888) 
switched from RUNNING to FINISHED.
Sep 21, 2018 12:08:38 AM org.apache.flink.runtime.taskexecutor.TaskExecutor 
unregisterTaskAndNotifyFinalState
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) bf6b18674c26b3c70a7190f0dc60a455.
Sep 21, 2018 12:08:38 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (4/16) (1bff9adab47d334a91f22eff81997eb7) 
switched from RUNNING to FINISHED.
Sep 21, 2018 12:08:38 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #88

[Apache Jenkins Server]

See 


--
[...truncated 567.53 KB...]
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 275, in 
test_combine_per_key
assert_that(res, equal_to([('a', 1.5), ('b', 3.0)]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_combine_per_key_1537488390.18_9fc053a4-ba6f-47a6-a7f9-64eead7a1218 failed 
in state FAILED.

==
ERROR: test_create (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 63, in 
test_create
assert_that(p | beam.Create(['a', 'b']), equal_to(['a', 'b']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_create_1537488390.57_41ea647e-d73b-4585-a851-1689adf77b38 failed in state 
FAILED.

==
ERROR: test_flatten (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 268, in 
test_flatten
assert_that(res, equal_to(['a', 'b', 'c', 'd']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flatten_1537488391.17_f4cf3dcf-ace4-448b-b471-c973bfb83c72 failed in state 
FAILED.

==
ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 191, in 
test_flattened_side_input
equal_to([(None, {'a': 1, 'b': 2})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flattened_side_input_1537488391.72_a0c691b5-acd7-43f2-abf4-37e4ff9a1aad 
failed in state FAILED.

==
ERROR: test_gbk_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 199, in 
test_gbk_side_input
equal_to([(None, {'a': [1]})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_gbk_side_input_1537488392.26_27e4f66a-b077-432c-9a7e-d15e62d7d68a failed 
in state FAILED.

==
ERROR: test_group_by_key (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 261, in 
test_group_by_key
assert_that(res, equal_to([('a', [1, 2]), ('b', [3])]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_group_by_key_1537488392.79_865831ff-3564-4428-b4a5-e7c799c823f0 failed in 
state FAILED.

==
ERROR: test_large_elements (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 317, in 
test_large_elements
assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 

Jenkins build is back to normal : beam_PreCommit_Website_Cron #84

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #87

[Apache Jenkins Server]

See 


Changes:

[github] [Lint] rename variable in graphx/translate.go

--
[...truncated 566.33 KB...]
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 275, in 
test_combine_per_key
assert_that(res, equal_to([('a', 1.5), ('b', 3.0)]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_combine_per_key_1537487609.31_1556870b-672c-442e-83dd-6ba80bb368b7 failed 
in state FAILED.

==
ERROR: test_create (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 63, in 
test_create
assert_that(p | beam.Create(['a', 'b']), equal_to(['a', 'b']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_create_1537487609.69_29170804-3050-4dcf-8eb9-5fcc4947abf3 failed in state 
FAILED.

==
ERROR: test_flatten (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 268, in 
test_flatten
assert_that(res, equal_to(['a', 'b', 'c', 'd']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flatten_1537487610.21_4763bfe6-5bde-4f5c-a25c-cd2f5a2a2247 failed in state 
FAILED.

==
ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 191, in 
test_flattened_side_input
equal_to([(None, {'a': 1, 'b': 2})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flattened_side_input_1537487610.75_c815bec3-ce2a-471b-a526-9796b50c5de2 
failed in state FAILED.

==
ERROR: test_gbk_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 199, in 
test_gbk_side_input
equal_to([(None, {'a': [1]})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_gbk_side_input_1537487611.29_d3a835f8-8a72-4b0e-bc67-dfc350512d63 failed 
in state FAILED.

==
ERROR: test_group_by_key (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 261, in 
test_group_by_key
assert_that(res, equal_to([('a', [1, 2]), ('b', [3])]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_group_by_key_1537487611.82_90ab2c67-cbcc-4552-b606-ea205093b4c7 failed in 
state FAILED.

==
ERROR: test_large_elements (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 317, in 
test_large_elements
assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
 

[beam] branch master updated (90e6482 -> 9d0b03a)

[herohde]

This is an automated email from the ASF dual-hosted git repository.

herohde pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 90e6482  Merge pull request #6435: Migrate latest website changes from 
apache/beam-site to apache/beam
 add 9094c5b  [Lint] rename variable in graphx/translate.go
 add 9d0b03a  [Lint] rename variable in graphx/translate.go

No new revisions were added by this update.

Summary of changes:
 sdks/go/pkg/beam/core/runtime/graphx/translate.go | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

[jira] [Work logged] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4498?focusedWorklogId=146179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146179
 ]

ASF GitHub Bot logged work on BEAM-4498:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:45
Start Date: 20/Sep/18 23:45
Worklog Time Spent: 10m 
  Work Description: udim opened a new pull request #6458: [BEAM-4498] Sync 
javadoc and pydoc from apache/beam-site
URL: https://github.com/apache/beam/pull/6458
 
 
   Migrates generated SDK documentation over to apache/beam on the asf-site 
branch.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146179)
Time Spent: 10m
Remaining Estimate: 0h

> Migrate release Javadocs / Pydocs to [asf-site] branch and update release 
> guide
> ---
>
> Key: BEAM-4498
> URL: https://issues.apache.org/jira/browse/BEAM-4498
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #1564

[Apache Jenkins Server]

See 


Changes:

[herohde] Add Go quickstart link to the footer

[xiliu] [BEAM-3079]: Samza Runner docs and capability matrix

[xiliu] Update the option docs

[xinyuliu.us] Add Samza to the runner list

[klk] Add a pull request template with staged URL

[apilloud] Add SQL Lexical Structure from BigQuery

[mergebot] Add Apache license headers

[apilloud] Update keywords from BeamSqlParserImplConstants

[apilloud] Update CREATE TABLE from doc

[apilloud] Move CREATE TABLE and SELECT up a level

[mergebot] Update contact-us.md

[mergebot] Spelling correction

[apilloud] Add redirect to new team page

[apilloud] Add SQL Set page

[boyuanz] Update release guide for download page changes

[klk] Make Samza icon a reasonable size

[klk] Alphabetize runners

[mergebot] New blog post about 2.5.0 release

[amaliujia] Update doc

[echauchot] [BEAM-4731] update flink submission port because flink 1.5 new 
standard

[mergebot] Adding YouTube channel page

[mergebot] Processing feedback

[mergebot] YouTube url update

[lcwik] [BEAM-3940] Remove reference to updating container version in pom.xml

[lcwik] Fix typos in programming guide

[lcwik] Fix typos in mobile gaming example

[kirpichov] Updates SDF status in capability matrix

[mergebot] Added new design documents

[mergebot] [BEAM-4430] Improve Performance Testing Documentation

[mergebot] Rename RowType to Schema in SQL walkthrough

[mergebot] Minor tweak to Intellij instructions.

[mergebot] fixup

[melissapa] [BEAM-2977] Improve unbounded prose in wordcount example

[mergebot] [BEAM-4188] Fix 2.4.0 release notes link

[mergebot] Update code style section to mention Spotless.

[mergebot] Clarify that this is only for Java/Groovy and not Python/Go/...

[lcwik] [BEAM-4828] Updated documentation, added SQS BEAM-4828 to In-Progress

[github] Update Python quickstart with Windows instructions

[mergebot] Add courtesy unsubscribe links to mailing lists

[melissapa] Fix shell tabs, add new shell tab switcher

[mergebot] Add a page for release cadence

[mergebot] updated the language

[mergebot] fix the space

[mergebot] [BEAM-4827] Add Nexmark dashboards to nexmark page on the website

[mergebot] [BEAM-4827] Fix typos

[mergebot] Add Flink instructions to portability contrib page.

[mergebot] Add instructions for local Flink cluster.

[mergebot] Update Java Quickstart for Windows

[mergebot] Update quickstart-java.md

[github] Update quickstart-java.md

[mergebot] [BEAM-4802] Update the contribution guide (Dependency section)

[migryz] Add post-commit tests policies summary page.

[migryz] Fix links

[github] Remove [TOC]

[migryz] Update postcommits guides

[ehudm] Remove testing matrix and simplify intro.

[mergebot] Moves KuduIO from in progress to available

[timrobertson100] [BEAM-4260] Document HCatalogIO use with Hive 1.1

[mergebot] Remove old pydoc and javadoc for versions < 2.0.0.

[apilloud] [BEAM-4761] Add Dataflow Runner

[mergebot] Update beam-site for release 2.6.0

[mergebot] Update download page

[mergebot] Updating downloads page.

[melissapa] Fix broken 2.5.0 download links

[mergebot] go version

[mergebot] Add section for SQL shell

[mergebot] Adding 'editable mode' flag installation for Python.

[thw] fix job service startup instruction

[mergebot] Fix Apex Runner instructions for execution on YARN

[mergebot] Update SQL walkthrough for 2.6.0

[github] Adding Pablo Estrada as author for blog

[github] Blog post for Beam 2.6.0 release

[github] Added Rafael Fernandez as author.

[rfernand] Adds Jenkinns job pointer to #building--testing

[rfernand] Minor rewrite to help newcomers explore the project's building and

[thw] fixup

[mergebot] Add instructions of using automation scripts

[mergebot] Fix broken tests

[mergebot] Addressed Ahmet's comments

[mergebot] Fixing 2.6.0 blog post

[mergebot] Quick fixes to layout

[mergebot] Address readability reviews

[mergebot] Grammar pass + pretty print

[mergebot] All files have Apache License

[mergebot] Add instructions about how to build python wheels

[mergebot] Add long term support release concept

[mergebot] Edit suggestions

[mergebot] Add precommit policies and triage guide.

[mergebot] Add dashes to pre-commit and post-commit.

[mergebot] Update programming guide to suggest using NewDoFn approach to

[mergebot] Address comments.

[mergebot] Add compatibility warning.

[mergebot] Fix language tab tags

[mergebot] Add blog post "A review of input streaming connectors"

[mergebot] Add authors for blog post in

[mergebot] Fix typo for author's name in blog post

[mergebot] Fix other typo in author's name for blog post

[mergebot] Blog post updates based on @iemejia's feedback

[mergebot] Updates to streaming connectors blog post

[mergebot] Set publication date for streaming connectors blog post

[mergebot] Update doc links in blog post to point to latest release

[mergebot] Fix 

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146177
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:41
Start Date: 20/Sep/18 23:41
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423368407
 
 
   (just in case...) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146177)
Time Spent: 2h  (was: 1h 50m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146176
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:40
Start Date: 20/Sep/18 23:40
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423368265
 
 
   Run Java HadoopInputFormatIO Performance Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146176)
Time Spent: 1h 50m  (was: 1h 40m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146175
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:39
Start Date: 20/Sep/18 23:39
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423368155
 
 
   Run Java AvroIO Performance Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146175)
Time Spent: 1h 40m  (was: 1.5h)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146172
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:26
Start Date: 20/Sep/18 23:26
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423365942
 
 
   Let's just leave this link here because it is related to this PR: 
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1776 
   (just in case somebody looks for it in the future) ;)
   
   LGTM, thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146172)
Time Spent: 1.5h  (was: 1h 20m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5434) Issue with BigQueryIO in Template

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5434?focusedWorklogId=146171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146171
 ]

ASF GitHub Bot logged work on BEAM-5434:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:26
Start Date: 20/Sep/18 23:26
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #6457: [BEAM-5434] Improve 
error handling in the artifact staging service
URL: https://github.com/apache/beam/pull/6457#issuecomment-423365865
 
 
   I mean...
   
   R: @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146171)
Time Spent: 0.5h  (was: 20m)

> Issue with BigQueryIO in Template
> -
>
> Key: BEAM-5434
> URL: https://issues.apache.org/jira/browse/BEAM-5434
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.5.0
>Reporter: Amarendra Kumar
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I am trying to build a google Dataflow template to be run from a cloud 
> function.
> The issue is with BigQueryIO trying execute a SQL.
> The opening step for my Dataflow Template is
> {code:java}
> BigQueryIO.readTableRows().withQueryLocation("US").withoutValidation().fromQuery(options.getSql()).usingStandardSql()
> {code}
> When the template is triggered for the first time its running fine.
> But when its triggered for the second time, it fails with the following error.
> {code}
> // Some comments here
> java.io.FileNotFoundException: No files matched spec: 
> gs://test-notification/temp/Notification/BigQueryExtractTemp/34d42a122600416c9ea748a6e325f87a/.avro
>   at 
> org.apache.beam.sdk.io.FileSystems.maybeAdjustEmptyMatchResult(FileSystems.java:172)
>   at org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:158)
>   at 
> org.apache.beam.sdk.io.FileBasedSource.createReader(FileBasedSource.java:329)
>   at 
> com.google.cloud.dataflow.worker.WorkerCustomSources$1.iterator(WorkerCustomSources.java:360)
>   at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:177)
>   at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
>   at 
> com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
>   at 
> com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:391)
>   at 
> com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:360)
>   at 
> com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
>   at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
>   at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
>   at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> In the second run, why is the process expecting a file in the GCS location?
> This file does get created while the job is running at the first run, but it 
> also gets deleted after the job is complete. 
> How are the two jobs related?
>  Could you please let me know if I am missing something or this is a bug?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #86

[Apache Jenkins Server]

See 


Changes:

[herohde] Add Go quickstart link to the footer

[xiliu] [BEAM-3079]: Samza Runner docs and capability matrix

[xiliu] Update the option docs

[xinyuliu.us] Add Samza to the runner list

[klk] Add a pull request template with staged URL

[apilloud] Add SQL Lexical Structure from BigQuery

[mergebot] Add Apache license headers

[apilloud] Update keywords from BeamSqlParserImplConstants

[apilloud] Update CREATE TABLE from doc

[apilloud] Move CREATE TABLE and SELECT up a level

[mergebot] Update contact-us.md

[mergebot] Spelling correction

[apilloud] Add redirect to new team page

[apilloud] Add SQL Set page

[boyuanz] Update release guide for download page changes

[klk] Make Samza icon a reasonable size

[klk] Alphabetize runners

[mergebot] New blog post about 2.5.0 release

[amaliujia] Update doc

[echauchot] [BEAM-4731] update flink submission port because flink 1.5 new 
standard

[mergebot] Adding YouTube channel page

[mergebot] Processing feedback

[mergebot] YouTube url update

[lcwik] [BEAM-3940] Remove reference to updating container version in pom.xml

[lcwik] Fix typos in programming guide

[lcwik] Fix typos in mobile gaming example

[kirpichov] Updates SDF status in capability matrix

[mergebot] Added new design documents

[mergebot] [BEAM-4430] Improve Performance Testing Documentation

[mergebot] Rename RowType to Schema in SQL walkthrough

[mergebot] Minor tweak to Intellij instructions.

[mergebot] fixup

[melissapa] [BEAM-2977] Improve unbounded prose in wordcount example

[mergebot] [BEAM-4188] Fix 2.4.0 release notes link

[mergebot] Update code style section to mention Spotless.

[mergebot] Clarify that this is only for Java/Groovy and not Python/Go/...

[lcwik] [BEAM-4828] Updated documentation, added SQS BEAM-4828 to In-Progress

[github] Update Python quickstart with Windows instructions

[mergebot] Add courtesy unsubscribe links to mailing lists

[melissapa] Fix shell tabs, add new shell tab switcher

[mergebot] Add a page for release cadence

[mergebot] updated the language

[mergebot] fix the space

[mergebot] [BEAM-4827] Add Nexmark dashboards to nexmark page on the website

[mergebot] [BEAM-4827] Fix typos

[mergebot] Add Flink instructions to portability contrib page.

[mergebot] Add instructions for local Flink cluster.

[mergebot] Update Java Quickstart for Windows

[mergebot] Update quickstart-java.md

[github] Update quickstart-java.md

[mergebot] [BEAM-4802] Update the contribution guide (Dependency section)

[migryz] Add post-commit tests policies summary page.

[migryz] Fix links

[github] Remove [TOC]

[migryz] Update postcommits guides

[ehudm] Remove testing matrix and simplify intro.

[mergebot] Moves KuduIO from in progress to available

[timrobertson100] [BEAM-4260] Document HCatalogIO use with Hive 1.1

[mergebot] Remove old pydoc and javadoc for versions < 2.0.0.

[apilloud] [BEAM-4761] Add Dataflow Runner

[mergebot] Update beam-site for release 2.6.0

[mergebot] Update download page

[mergebot] Updating downloads page.

[melissapa] Fix broken 2.5.0 download links

[mergebot] go version

[mergebot] Add section for SQL shell

[mergebot] Adding 'editable mode' flag installation for Python.

[thw] fix job service startup instruction

[mergebot] Fix Apex Runner instructions for execution on YARN

[mergebot] Update SQL walkthrough for 2.6.0

[github] Adding Pablo Estrada as author for blog

[github] Blog post for Beam 2.6.0 release

[github] Added Rafael Fernandez as author.

[rfernand] Adds Jenkinns job pointer to #building--testing

[rfernand] Minor rewrite to help newcomers explore the project's building and

[thw] fixup

[mergebot] Add instructions of using automation scripts

[mergebot] Fix broken tests

[mergebot] Addressed Ahmet's comments

[mergebot] Fixing 2.6.0 blog post

[mergebot] Quick fixes to layout

[mergebot] Address readability reviews

[mergebot] Grammar pass + pretty print

[mergebot] All files have Apache License

[mergebot] Add instructions about how to build python wheels

[mergebot] Add long term support release concept

[mergebot] Edit suggestions

[mergebot] Add precommit policies and triage guide.

[mergebot] Add dashes to pre-commit and post-commit.

[mergebot] Update programming guide to suggest using NewDoFn approach to

[mergebot] Address comments.

[mergebot] Add compatibility warning.

[mergebot] Fix language tab tags

[mergebot] Add blog post "A review of input streaming connectors"

[mergebot] Add authors for blog post in

[mergebot] Fix typo for author's name in blog post

[mergebot] Fix other typo in author's name for blog post

[mergebot] Blog post updates based on @iemejia's feedback

[mergebot] Updates to streaming connectors blog post

[mergebot] Set publication date for streaming connectors blog post

[mergebot] Update doc links in blog post to point to latest release

[mergebot] Fix extraneous p tag and 

Jenkins build is back to normal : beam_PerformanceTests_Python #1462

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5434) Issue with BigQueryIO in Template

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5434?focusedWorklogId=146170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146170
 ]

ASF GitHub Bot logged work on BEAM-5434:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:25
Start Date: 20/Sep/18 23:25
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #6457: [BEAM-5434] Improve 
error handling in the artifact staging service
URL: https://github.com/apache/beam/pull/6457#issuecomment-423365647
 
 
   @angoenka PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146170)
Time Spent: 20m  (was: 10m)

> Issue with BigQueryIO in Template
> -
>
> Key: BEAM-5434
> URL: https://issues.apache.org/jira/browse/BEAM-5434
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.5.0
>Reporter: Amarendra Kumar
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I am trying to build a google Dataflow template to be run from a cloud 
> function.
> The issue is with BigQueryIO trying execute a SQL.
> The opening step for my Dataflow Template is
> {code:java}
> BigQueryIO.readTableRows().withQueryLocation("US").withoutValidation().fromQuery(options.getSql()).usingStandardSql()
> {code}
> When the template is triggered for the first time its running fine.
> But when its triggered for the second time, it fails with the following error.
> {code}
> // Some comments here
> java.io.FileNotFoundException: No files matched spec: 
> gs://test-notification/temp/Notification/BigQueryExtractTemp/34d42a122600416c9ea748a6e325f87a/.avro
>   at 
> org.apache.beam.sdk.io.FileSystems.maybeAdjustEmptyMatchResult(FileSystems.java:172)
>   at org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:158)
>   at 
> org.apache.beam.sdk.io.FileBasedSource.createReader(FileBasedSource.java:329)
>   at 
> com.google.cloud.dataflow.worker.WorkerCustomSources$1.iterator(WorkerCustomSources.java:360)
>   at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:177)
>   at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
>   at 
> com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
>   at 
> com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:391)
>   at 
> com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:360)
>   at 
> com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
>   at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
>   at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
>   at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> In the second run, why is the process expecting a file in the GCS location?
> This file does get created while the job is running at the first run, but it 
> also gets deleted after the job is complete. 
> How are the two jobs related?
>  Could you please let me know if I am missing something or this is a bug?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5434) Issue with BigQueryIO in Template

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5434?focusedWorklogId=146168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146168
 ]

ASF GitHub Bot logged work on BEAM-5434:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:21
Start Date: 20/Sep/18 23:21
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #6457: [BEAM-5434] 
Improve error handling in the artifact staging service
URL: https://github.com/apache/beam/pull/6457
 
 
   While trying to debug an artifact staging problem, I encountered an
   uncaught null pointer exception in the onCompleted method.  I also
   decided to add log lines when errors are encountered, instead of
   forwarding them to the client SDK.  This is because at present, some
   client SDKs cannot be trusted to print the correct error message or
   stack trace, leading to pain while debugging.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146168)
Time Spent: 10m
Remaining Estimate: 0h

> Issue with BigQueryIO in Template
> -
>
> Key: BEAM-5434
> URL: 

[jira] [Created] (BEAM-5454) Improve Beam Fn API Artifact Staging Service Error Handling

[Axel Magnuson (JIRA)]

Axel Magnuson created BEAM-5454:
---

 Summary: Improve Beam Fn API Artifact Staging Service Error 
Handling
 Key: BEAM-5454
 URL: https://issues.apache.org/jira/browse/BEAM-5454
 Project: Beam
  Issue Type: Improvement
  Components: java-fn-execution
Reporter: Axel Magnuson
Assignee: Axel Magnuson


I noticed that the onComplete logic is not wrapped in a try/catch block.  This 
led to an uncaught NPE in a problem I was debugging.  It also does not log 
errors, preferring to just use the observer's callback to pass these errors to 
the SDK.  However some SDKs such as golang's do not surface these errors, and 
they get lost. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6435: Migrate latest website changes from apache/beam-site to apache/beam

[scott]

This is an automated email from the ASF dual-hosted git repository.

scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 90e64826eafc3cb36a65cb7e0759cc5b21bb6578
Merge: a724f14 61945a7
Author: Scott Wegner 
AuthorDate: Thu Sep 20 16:20:48 2018 -0700

Merge pull request #6435: Migrate latest website changes from 
apache/beam-site to apache/beam

 build.gradle   |2 +-
 website/.github/PULL_REQUEST_TEMPLATE.md   |   10 +
 website/_config.yml|2 +-
 website/src/_data/authors.yml  |   23 +
 website/src/_data/capability-matrix.yml|  144 ++-
 website/src/_includes/footer.html  |1 +
 website/src/_includes/section-menu/community.html  |2 +
 website/src/_includes/section-menu/contribute.html |   10 +
 website/src/_includes/section-menu/runners.html|1 +
 website/src/_includes/section-menu/sdks.html   |   13 +-
 website/src/_layouts/post.html |5 +-
 website/src/_posts/2018-06-26-beam-2.5.0.md|   98 ++
 website/src/_posts/2018-08-10-beam-2.6.0.md|   88 ++
 ...2018-08-20-review-input-streaming-connectors.md |  238 +
 .../src/_posts/2018-08-21-beam-summit-europe.md|   53 +
 website/src/blog/index.md  |4 +-
 website/src/community/contact-us.md|7 +-
 website/src/community/policies.md  |   34 +
 website/src/community/youtube-channel.md   |   48 +
 website/src/contribute/dependencies.md |   15 +-
 website/src/contribute/design-documents.md |5 +-
 website/src/contribute/eclipse.md  |   10 +-
 website/src/contribute/index.md|  117 ++-
 website/src/contribute/intellij.md |   21 +-
 website/src/contribute/portability.md  |   32 +-
 website/src/contribute/postcommits-guides.md   |   74 ++
 .../src/contribute/postcommits-policies-details.md |   93 ++
 website/src/contribute/postcommits-policies.md |   82 ++
 website/src/contribute/precommit-policies.md   |   66 ++
 website/src/contribute/precommit-triage-guide.md   |  137 +++
 website/src/contribute/release-guide.md|  408 +---
 .../authors-list.md => contribute/team.md} |   11 +-
 website/src/contribute/testing.md  |  389 +++-
 website/src/documentation/dsls/sql/create-table.md |  351 +++
 .../documentation/dsls/sql/lexical-structure.md| 1048 
 website/src/documentation/dsls/sql/overview.md |2 +-
 .../dsls/sql/{statements => }/select.md|3 +-
 website/src/documentation/dsls/sql/set.md  |   55 +
 website/src/documentation/dsls/sql/shell.md|  153 +++
 .../dsls/sql/statements/create-table.md|   38 -
 website/src/documentation/dsls/sql/walkthrough.md  |   64 +-
 website/src/documentation/index.md |1 +
 website/src/documentation/io/built-in-hcatalog.md  |  160 +++
 website/src/documentation/io/built-in.md   |7 +-
 website/src/documentation/io/testing.md|  223 -
 website/src/documentation/programming-guide.md |  139 ++-
 website/src/documentation/runners/apex.md  |   32 +-
 website/src/documentation/runners/flink.md |   51 +-
 website/src/documentation/runners/samza.md |  165 +++
 website/src/documentation/sdks/nexmark.md  |   49 +
 website/src/get-started/beam-overview.md   |1 +
 website/src/get-started/downloads.md   |   44 +-
 website/src/get-started/mobile-gaming-example.md   |2 +-
 website/src/get-started/quickstart-go.md   |2 +-
 website/src/get-started/quickstart-java.md |  119 +++
 website/src/get-started/quickstart-py.md   |   38 +-
 website/src/get-started/wordcount-example.md   |   85 +-
 website/src/images/blog/Facebook-AD.png|  Bin 0 -> 1275709 bytes
 website/src/images/logo_samza.png  |  Bin 0 -> 11728 bytes
 website/src/images/logos/runners/samza.png |  Bin 0 -> 11728 bytes
 website/src/images/precommit_durations.png |  Bin 0 -> 45673 bytes
 .../src/images/precommit_graph_queuing_time.png|  Bin 0 -> 25809 bytes
 website/src/index.md   |3 +
 website/src/js/language-switch.js  |1 +
 64 files changed, 4392 insertions(+), 687 deletions(-)

[beam] branch master updated (a724f14 -> 90e6482)

[scott]

This is an automated email from the ASF dual-hosted git repository.

scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a724f14  Merge pull request #6455: Ensure all pre-commits are run on 
gradle upgrade.
 add b12070b  Exclude website PULL_REQUEST_TEMPLATE from Rat.
 add 6f1e296  Add SQL Lexical Structure from BigQuery
 add 2b016d1  This closes https://github.com/apache/beam-site/pull/479
 add 5d5c3ec  Add a pull request template with staged URL
 add f3d74a9  Merge https://github.com/apache/beam-site/pull/461: Add a 
pull request template with staged URL
 add 109e0cc  Add Apache license headers
 add 3751ae7  This closes https://github.com/apache/beam-site/pull/481
 add 860f5c2  Update CREATE TABLE from doc
 add 5812950  This closes https://github.com/apache/beam-site/pull/482
 add da1e9a6  Move CREATE TABLE and SELECT up a level
 add 294f5b2  Merge https://github.com/apache/beam-site/pull/483: Move 
CREATE TABLE and SELECT up a level
 add c607fa4  Update contact-us.md
 add 422fe29  This closes https://github.com/apache/beam-site/pull/484
 add 5fb9d48  Update keywords from BeamSqlParserImplConstants
 add 2385d95  Merge https://github.com/apache/beam-site/pull/480: Update 
keywords from BeamSqlParserImplConstants
 add baff0e2  Spelling correction
 add 8c5c7a5  This closes https://github.com/apache/beam-site/pull/466
 add 38ec7b3  Add Go quickstart link to the footer
 add 169e918  Merge https://github.com/apache/beam-site/pull/444: Add Go 
quickstart link to the footer
 add 3944437  Update release guide for download page changes
 add f227a2d  This closes https://github.com/apache/beam-site/pull/486
 add 47c2a54  Add SQL Set page
 add 531d819  Merge https://github.com/apache/beam-site/pull/487: Add SQL 
SET page
 add 36523c3  [BEAM-3079]: Samza Runner docs and capability matrix
 add 00e3eb0  Update the option docs
 add 609250e  Add Samza to the runner list
 add b3776e6  Merge https://github.com/apache/beam-site/pull/471: Add Samza 
runner to web site
 add 2f002b2  Make Samza icon a reasonable size
 add 54ccc08  Alphabetize runners
 add ce50d8e  Add redirect to new team page
 add 646245c  Merge https://github.com/apache/beam-site/pull/485: Add 
redirect to new team page
 add a32441f  New blog post about 2.5.0 release
 add a1ddf25  This closes https://github.com/apache/beam-site/pull/476
 add b44ed62  Update doc
 add ff5796c  This closes https://github.com/apache/beam-site/pull/488
 add df97760  [BEAM-4731] update flink submission port because flink 1.5 
new standard is port 8081
 add 61987df  This closes https://github.com/apache/beam-site/pull/489
 add 62ab49f  Adding YouTube channel page
 add 8c70283  Processing feedback
 add 56b6297  YouTube url update
 add fa3f433  This closes https://github.com/apache/beam-site/pull/477
 add c354b5d  [BEAM-3940] Remove reference to updating container version in 
pom.xml
 add 63740f0  Fix typos in programming guide
 add 0b295f8  Fix typos in mobile gaming example
 add 1e0e343  Updates SDF status in capability matrix
 add cab8242  This closes https://github.com/apache/beam-site/pull/496
 add 0295f4a  Added new design documents
 add c5f17df  This closes https://github.com/apache/beam-site/pull/492
 add 219dd37  [BEAM-4430] Improve Performance Testing Documentation
 add c445968  This closes https://github.com/apache/beam-site/pull/465
 add c0822fe  Rename RowType to Schema in SQL walkthrough
 add 5765451  This closes https://github.com/apache/beam-site/pull/495
 add 6de295e  Minor tweak to Intellij instructions.
 add dbfc336  fixup
 add 3d3f381  This closes https://github.com/apache/beam-site/pull/494
 add f67240a  [BEAM-2977] Improve unbounded prose in wordcount example
 add 7344ef1  This closes https://github.com/apache/beam-site/pull/377
 add d0d125a  [BEAM-4188] Fix 2.4.0 release notes link
 add 0f3c328  This closes https://github.com/apache/beam-site/pull/497
 add 01abfdf  Update code style section to mention Spotless.
 add 3d9af1c  Clarify that this is only for Java/Groovy and not 
Python/Go/...
 add 143cbd9  This closes https://github.com/apache/beam-site/pull/498
 add 4a96b99  [BEAM-4828] Updated documentation, added SQS BEAM-4828 to 
In-Progress
 add c9bb035  [BEAM-4828] Updated documentation, added SQS BEAM-4828 to 
In-Progress
 add 19cd24d  Update Python quickstart with Windows instructions
 add 89e79e1  This closes https://github.com/apache/beam-site/pull/505
 add 185a1bb  Add courtesy unsubscribe links to mailing lists
 add 6a9d278  This closes https://github.com/apache/beam-site/pull/504
 add d286767  Fix shell tabs, add new shell tab switcher
 add 63483e7  This closes https://github.com/apache/beam-site/pull/507
 add 

[jira] [Work logged] (BEAM-4568) Add Apache headers to website sources

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4568?focusedWorklogId=146167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146167
 ]

ASF GitHub Bot logged work on BEAM-4568:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:18
Start Date: 20/Sep/18 23:18
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #555: [BEAM-4568] Add Apache 
headers to new source pages.
URL: https://github.com/apache/beam-site/pull/555
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/_posts/2018-06-26-beam-2.5.0.md 
b/src/_posts/2018-06-26-beam-2.5.0.md
index b445d6b9e9..9ee57a526a 100644
--- a/src/_posts/2018-06-26-beam-2.5.0.md
+++ b/src/_posts/2018-06-26-beam-2.5.0.md
@@ -7,6 +7,19 @@ categories: blog
 authors:
   - aromanenko
 ---
+
 
 We are glad to present the new 2.5.0 release of Beam. This release includes
 multiple fixes and new functionalities.  For more information
@@ -82,4 +95,4 @@ Metrics are pushed by the runners to configurable sinks (Http 
REST sink availabl
 
 According to git shortlog, the following 84 people contributed to the 2.5.0 
release. Thank you to all contributors!
 
-Ahmet Altay, Alan Myrvold, Alex Amato, Alex Van Boxel, Alexander Dejanovski, 
Alexey Romanenko, Aljoscha Krettek, ananvay, Andreas Ehrencrona, Andrew 
Pilloud, Ankur Goenka, Anton Kedin, arkash, Austin Bennett, Axel Magnuson, Ben 
Chambers, Ben Sidhom, Bill Neubauer, Boyuan Zhang, Braden Bassingthwaite, Cade 
Markegard, cclauss, Chamikara Jayalath, Charles Chen, Chuan Yu Foo, Cody 
Schroeder, Colm O hEigeartaigh, Daniel Oliveira, Dariusz Aniszewski, David 
Cavazos, Dawid Wysakowicz, Eric Roshan-Eisner, Etienne Chauchot, Eugene 
Kirpichov, Flavio Fiszman, Geet Kumar, GlennAmmons, Grzegorz Kołakowski, 
Henning Rohde, Innocent Djiofack, Ismaël Mejía, Jack Hsueh, Jason Kuster, 
Javier Antonio Gonzalez Trejo, Jean-Baptiste Onofré, Kai Jiang, Kamil Szewczyk, 
Katarzyna Kucharczyk, Kenneth Jung, Kenneth Knowles, Kevin Peterson, Lukasz 
Cwik, Łukasz Gajowy, Mairbek Khadikov, Manu Zhang, Maria Garcia Herrero, Marian 
Dvorsky, Mark Liu, Matthias Feys, Matthias Wessendorf, mingmxu, Nathan Howell, 
Pablo Estrada, Paul Gerver, Raghu Angadi, rarokni, Reuven Lax, Rezan Achmad, 
Robbe Sneyders, Robert Bradshaw, Robert Burke, Romain Manni-Bucau, Sam 
Waggoner, Sam Whittle, Scott Wegner, Stephan Hoyer, Thomas Groh, Thomas Weise, 
Tim Robertson, Udi Meiri, Valentyn Tymofieiev, XuMingmin, Yifan Zou, Yunqing 
Zhou
\ No newline at end of file
+Ahmet Altay, Alan Myrvold, Alex Amato, Alex Van Boxel, Alexander Dejanovski, 
Alexey Romanenko, Aljoscha Krettek, ananvay, Andreas Ehrencrona, Andrew 
Pilloud, Ankur Goenka, Anton Kedin, arkash, Austin Bennett, Axel Magnuson, Ben 
Chambers, Ben Sidhom, Bill Neubauer, Boyuan Zhang, Braden Bassingthwaite, Cade 
Markegard, cclauss, Chamikara Jayalath, Charles Chen, Chuan Yu Foo, Cody 
Schroeder, Colm O hEigeartaigh, Daniel Oliveira, Dariusz Aniszewski, David 
Cavazos, Dawid Wysakowicz, Eric Roshan-Eisner, Etienne Chauchot, Eugene 
Kirpichov, Flavio Fiszman, Geet Kumar, GlennAmmons, Grzegorz Kołakowski, 
Henning Rohde, Innocent Djiofack, Ismaël Mejía, Jack Hsueh, Jason Kuster, 
Javier Antonio Gonzalez Trejo, Jean-Baptiste Onofré, Kai Jiang, Kamil Szewczyk, 
Katarzyna Kucharczyk, Kenneth Jung, Kenneth Knowles, Kevin Peterson, Lukasz 
Cwik, Łukasz Gajowy, Mairbek Khadikov, Manu Zhang, Maria Garcia Herrero, Marian 
Dvorsky, Mark Liu, Matthias Feys, Matthias Wessendorf, mingmxu, Nathan Howell, 
Pablo Estrada, Paul Gerver, Raghu Angadi, rarokni, Reuven Lax, Rezan Achmad, 
Robbe Sneyders, Robert Bradshaw, Robert Burke, Romain Manni-Bucau, Sam 
Waggoner, Sam Whittle, Scott Wegner, Stephan Hoyer, Thomas Groh, Thomas Weise, 
Tim Robertson, Udi Meiri, Valentyn Tymofieiev, XuMingmin, Yifan Zou, Yunqing 
Zhou
diff --git a/src/_posts/2018-08-10-beam-2.6.0.md 
b/src/_posts/2018-08-10-beam-2.6.0.md
index 247ff3ef2b..dbc2bcf0e3 100644
--- a/src/_posts/2018-08-10-beam-2.6.0.md
+++ b/src/_posts/2018-08-10-beam-2.6.0.md
@@ -9,6 +9,19 @@ authors:
 - rfernand
 
 ---
+
 
 We are glad to present the new 2.6.0 release of Beam.
 This release includes multiple fixes and new functionality, such as new 
features in SQL and portability.
diff --git a/src/_posts/2018-08-20-review-input-streaming-connectors.md 
b/src/_posts/2018-08-20-review-input-streaming-connectors.md
index 4d6f1048eb..d3a9c9aebc 100644
--- a/src/_posts/2018-08-20-review-input-streaming-connectors.md
+++ b/src/_posts/2018-08-20-review-input-streaming-connectors.md
@@ -8,6 +8,19 @@ authors:
   - lkuligin
   - jphalip
 ---
+
 
 In this post, you'll learn about the current state of support for input 

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146166
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:18
Start Date: 20/Sep/18 23:18
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423364439
 
 
   Run Python Performance Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146166)
Time Spent: 1h 20m  (was: 1h 10m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (c427295 -> 70bdfaf)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from c427295  Prepare repository for deployment.
 add 7b4d91f  [BEAM-4568] Add Apache headers to new source pages.
 add 6097f3c  This closes #555
 new 70bdfaf  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/blog/2018/06/26/beam-2.5.0.html| 16 +-
 content/blog/2018/08/10/beam-2.6.0.html| 16 +-
 .../08/20/review-input-streaming-connectors.html   | 16 +-
 content/blog/2018/08/21/beam-summit-europe.html| 16 +-
 content/blog/index.html| 56 +++
 content/community/youtube-channel/index.html   | 16 +-
 content/documentation/dsls/sql/lexical/index.html  | 16 +-
 .../documentation/io/built-in/hcatalog/index.html  | 16 +-
 content/documentation/runners/samza/index.html | 16 +-
 content/feed.xml   | 64 --
 src/_posts/2018-06-26-beam-2.5.0.md| 15 -
 src/_posts/2018-08-10-beam-2.6.0.md| 13 +
 ...2018-08-20-review-input-streaming-connectors.md | 13 +
 src/_posts/2018-08-21-beam-summit-europe.md| 13 +
 src/community/youtube-channel.md   | 13 +
 src/documentation/dsls/sql/lexical-structure.md| 13 +
 src/documentation/io/built-in-hcatalog.md  | 15 -
 src/documentation/runners/samza.md | 14 +
 18 files changed, 343 insertions(+), 14 deletions(-)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 70bdfaf46c74a0feab03946ac512b9c5e4712c8b
Author: Mergebot 
AuthorDate: Thu Sep 20 23:18:36 2018 +

Prepare repository for deployment.
---
 content/blog/2018/06/26/beam-2.5.0.html| 16 +-
 content/blog/2018/08/10/beam-2.6.0.html| 16 +-
 .../08/20/review-input-streaming-connectors.html   | 16 +-
 content/blog/2018/08/21/beam-summit-europe.html| 16 +-
 content/blog/index.html| 56 +++
 content/community/youtube-channel/index.html   | 16 +-
 content/documentation/dsls/sql/lexical/index.html  | 16 +-
 .../documentation/io/built-in/hcatalog/index.html  | 16 +-
 content/documentation/runners/samza/index.html | 16 +-
 content/feed.xml   | 64 --
 10 files changed, 236 insertions(+), 12 deletions(-)

diff --git a/content/blog/2018/06/26/beam-2.5.0.html 
b/content/blog/2018/06/26/beam-2.5.0.html
index 562cf68..e054124 100644
--- a/content/blog/2018/06/26/beam-2.5.0.html
+++ b/content/blog/2018/06/26/beam-2.5.0.html
@@ -144,7 +144,21 @@
   
 
   
-We are glad to present the new 2.5.0 release of Beam. This release 
includes
+
+
+We are glad to present the new 2.5.0 release of Beam. This release includes
 multiple fixes and new functionalities.  For more information
 please check the detailed release notes.
 
diff --git a/content/blog/2018/08/10/beam-2.6.0.html 
b/content/blog/2018/08/10/beam-2.6.0.html
index dc4b766..9c80b96 100644
--- a/content/blog/2018/08/10/beam-2.6.0.html
+++ b/content/blog/2018/08/10/beam-2.6.0.html
@@ -144,7 +144,21 @@
   
 
   
-We are glad to present the new 2.6.0 release of Beam.
+
+
+We are glad to present the new 2.6.0 release of Beam.
 This release includes multiple fixes and new functionality, such as new 
features in SQL and portability.
 We also spent a significant amount of time automating the release and fixing 
continuous integration. For more information, check the
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12343392;>release
 notes.
diff --git a/content/blog/2018/08/20/review-input-streaming-connectors.html 
b/content/blog/2018/08/20/review-input-streaming-connectors.html
index 1bdde6d..2e9c8bb 100644
--- a/content/blog/2018/08/20/review-input-streaming-connectors.html
+++ b/content/blog/2018/08/20/review-input-streaming-connectors.html
@@ -144,7 +144,21 @@
   
 
   
-In this post, you’ll learn about the current state of support for input 
streaming connectors in Apache Beam. For more context, you’ll 
also learn about the corresponding state of support in https://spark.apache.org/;>Apache Spark.
+
+
+In this post, you’ll learn about the current state of support for input 
streaming connectors in Apache Beam. For more context, you’ll 
also learn about the corresponding state of support in https://spark.apache.org/;>Apache Spark.
 
 With batch processing, you might load data from any source, including a 
database system. Even if there are no specific SDKs available for those 
database systems, you can often resort to using a https://en.wikipedia.org/wiki/Java_Database_Connectivity;>JDBC 
driver. With streaming, implementing a proper data pipeline is arguably more 
challenging as generally fewer source types are available. For that reason, 
this article particularly focuses on the streaming use case.
 
diff --git a/content/blog/2018/08/21/beam-summit-europe.html 
b/content/blog/2018/08/21/beam-summit-europe.html
index db5f28c..2b59791 100644
--- a/content/blog/2018/08/21/beam-summit-europe.html
+++ b/content/blog/2018/08/21/beam-summit-europe.html
@@ -144,7 +144,21 @@
   
 
   
-With a growing community of contributors and users, the Apache Beam 
project is organising the first European Beam Summit.
+
+
+With a growing community of contributors and users, the Apache Beam project 
is organising the first European Beam Summit.
 
 We are happy to invite you to this event, which will take place in 
London on October 1st and 2nd of 2018. 

 
diff --git a/content/blog/index.html b/content/blog/index.html
index 23508c8..f3c8774 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -144,6 +144,20 @@ for the project.
  Matthias Baetens [https://twitter.com/matthiasbaetens;>@matthiasbaetens]
 
 
+
+
 With a growing community of contributors and users, the Apache Beam project 
is organising the first European Beam Summit.
 
 We are happy to invite you to this event, which will take place in 
London on October 1st and 2nd of 2018.
@@ -163,6 +177,20 @@ Read more<
  Leonid Kuligin [https://twitter.com/lkulighin;>@lkulighin]  
Julien Phalip [https://twitter.com/julienphalip;>@julienphalip]
 
 
+
+
 In this post, you’ll learn about the current state of support 

[jira] [Assigned] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-4498:
---

Assignee: Udi Meiri  (was: Scott Wegner)

> Migrate release Javadocs / Pydocs to [asf-site] branch and update release 
> guide
> ---
>
> Key: BEAM-4498
> URL: https://issues.apache.org/jira/browse/BEAM-4498
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6039

[Apache Jenkins Server]

See 


Changes:

[scott] Ensure all pre-commits are run on gradle upgrade.

--
[...truncated 1.15 MB...]
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError

==
FAIL: test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 221, in test_apply_custom_transform
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
- >> end captured logging << -

==
FAIL: test_create (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 154, in test_create
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested

[beam-site] 02/02: This closes #555

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6097f3cf6493725aa6d59621b453aa29098111c3
Merge: c427295 7b4d91f
Author: Mergebot 
AuthorDate: Thu Sep 20 23:15:22 2018 +

This closes #555

 src/_posts/2018-06-26-beam-2.5.0.md   | 15 ++-
 src/_posts/2018-08-10-beam-2.6.0.md   | 13 +
 .../2018-08-20-review-input-streaming-connectors.md   | 13 +
 src/_posts/2018-08-21-beam-summit-europe.md   | 13 +
 src/community/youtube-channel.md  | 13 +
 src/documentation/dsls/sql/lexical-structure.md   | 13 +
 src/documentation/io/built-in-hcatalog.md | 15 ++-
 src/documentation/runners/samza.md| 14 ++
 8 files changed, 107 insertions(+), 2 deletions(-)

[jira] [Assigned] (BEAM-5438) why not add a Gemfire IO

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-5438:
--

Assignee: (was: Eugene Kirpichov)

> why not add a Gemfire IO
> 
>
> Key: BEAM-5438
> URL: https://issues.apache.org/jira/browse/BEAM-5438
> Project: Beam
>  Issue Type: Wish
>  Components: io-ideas
>Reporter: zhangbin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/02: [BEAM-4568] Add Apache headers to new source pages.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7b4d91f5763654e86353d4c254fafb775e99da13
Author: Scott Wegner 
AuthorDate: Wed Sep 19 13:32:29 2018 -0700

[BEAM-4568] Add Apache headers to new source pages.

This is required for Rat check when importing website content to the
apache/beam repository.
---
 src/_posts/2018-06-26-beam-2.5.0.md   | 15 ++-
 src/_posts/2018-08-10-beam-2.6.0.md   | 13 +
 .../2018-08-20-review-input-streaming-connectors.md   | 13 +
 src/_posts/2018-08-21-beam-summit-europe.md   | 13 +
 src/community/youtube-channel.md  | 13 +
 src/documentation/dsls/sql/lexical-structure.md   | 13 +
 src/documentation/io/built-in-hcatalog.md | 15 ++-
 src/documentation/runners/samza.md| 14 ++
 8 files changed, 107 insertions(+), 2 deletions(-)

diff --git a/src/_posts/2018-06-26-beam-2.5.0.md 
b/src/_posts/2018-06-26-beam-2.5.0.md
index b445d6b..9ee57a5 100644
--- a/src/_posts/2018-06-26-beam-2.5.0.md
+++ b/src/_posts/2018-06-26-beam-2.5.0.md
@@ -7,6 +7,19 @@ categories: blog
 authors:
   - aromanenko
 ---
+
 
 We are glad to present the new 2.5.0 release of Beam. This release includes
 multiple fixes and new functionalities.  For more information
@@ -82,4 +95,4 @@ Metrics are pushed by the runners to configurable sinks (Http 
REST sink availabl
 
 According to git shortlog, the following 84 people contributed to the 2.5.0 
release. Thank you to all contributors!
 
-Ahmet Altay, Alan Myrvold, Alex Amato, Alex Van Boxel, Alexander Dejanovski, 
Alexey Romanenko, Aljoscha Krettek, ananvay, Andreas Ehrencrona, Andrew 
Pilloud, Ankur Goenka, Anton Kedin, arkash, Austin Bennett, Axel Magnuson, Ben 
Chambers, Ben Sidhom, Bill Neubauer, Boyuan Zhang, Braden Bassingthwaite, Cade 
Markegard, cclauss, Chamikara Jayalath, Charles Chen, Chuan Yu Foo, Cody 
Schroeder, Colm O hEigeartaigh, Daniel Oliveira, Dariusz Aniszewski, David 
Cavazos, Dawid Wysakowicz, Eric Rosha [...]
\ No newline at end of file
+Ahmet Altay, Alan Myrvold, Alex Amato, Alex Van Boxel, Alexander Dejanovski, 
Alexey Romanenko, Aljoscha Krettek, ananvay, Andreas Ehrencrona, Andrew 
Pilloud, Ankur Goenka, Anton Kedin, arkash, Austin Bennett, Axel Magnuson, Ben 
Chambers, Ben Sidhom, Bill Neubauer, Boyuan Zhang, Braden Bassingthwaite, Cade 
Markegard, cclauss, Chamikara Jayalath, Charles Chen, Chuan Yu Foo, Cody 
Schroeder, Colm O hEigeartaigh, Daniel Oliveira, Dariusz Aniszewski, David 
Cavazos, Dawid Wysakowicz, Eric Rosha [...]
diff --git a/src/_posts/2018-08-10-beam-2.6.0.md 
b/src/_posts/2018-08-10-beam-2.6.0.md
index 247ff3e..dbc2bcf 100644
--- a/src/_posts/2018-08-10-beam-2.6.0.md
+++ b/src/_posts/2018-08-10-beam-2.6.0.md
@@ -9,6 +9,19 @@ authors:
 - rfernand
 
 ---
+
 
 We are glad to present the new 2.6.0 release of Beam.
 This release includes multiple fixes and new functionality, such as new 
features in SQL and portability.
diff --git a/src/_posts/2018-08-20-review-input-streaming-connectors.md 
b/src/_posts/2018-08-20-review-input-streaming-connectors.md
index 4d6f104..d3a9c9a 100644
--- a/src/_posts/2018-08-20-review-input-streaming-connectors.md
+++ b/src/_posts/2018-08-20-review-input-streaming-connectors.md
@@ -8,6 +8,19 @@ authors:
   - lkuligin
   - jphalip
 ---
+
 
 In this post, you'll learn about the current state of support for input 
streaming connectors in [Apache Beam]({{ site.baseurl }}/). For more context, 
you'll also learn about the corresponding state of support in [Apache 
Spark](https://spark.apache.org/).
 
diff --git a/src/_posts/2018-08-21-beam-summit-europe.md 
b/src/_posts/2018-08-21-beam-summit-europe.md
index 8287b08..c867c8c 100644
--- a/src/_posts/2018-08-21-beam-summit-europe.md
+++ b/src/_posts/2018-08-21-beam-summit-europe.md
@@ -7,6 +7,19 @@ categories: blog
 authors:
   - mbaetens
 ---
+
 
 With a growing community of contributors and users, the Apache Beam project is 
organising the first European Beam Summit. 
 
diff --git a/src/community/youtube-channel.md b/src/community/youtube-channel.md
index 4d09a1d..23100b5 100644
--- a/src/community/youtube-channel.md
+++ b/src/community/youtube-channel.md
@@ -4,6 +4,19 @@ title: 'Beam YouTube channel'
 section_menu: section-menu/community.html
 permalink: /community/youtube-channel/
 ---
+
 
 # Beam YouTube channel
 ## What is it and what are the goals?
diff --git a/src/documentation/dsls/sql/lexical-structure.md 
b/src/documentation/dsls/sql/lexical-structure.md
index 8f3cd01..423871c 100644
--- a/src/documentation/dsls/sql/lexical-structure.md
+++ b/src/documentation/dsls/sql/lexical-structure.md
@@ -4,6 +4,19 @@ title: "Beam SQL: Lexical Structure"
 section_menu: 

[jira] [Work logged] (BEAM-4568) Add Apache headers to website sources

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4568?focusedWorklogId=146165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146165
 ]

ASF GitHub Bot logged work on BEAM-4568:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:14
Start Date: 20/Sep/18 23:14
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #555: [BEAM-4568] Add Apache 
headers to new source pages.
URL: https://github.com/apache/beam-site/pull/555#issuecomment-423363889
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146165)
Time Spent: 50m  (was: 40m)

> Add Apache headers to website sources
> -
>
> Key: BEAM-4568
> URL: https://issues.apache.org/jira/browse/BEAM-4568
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently website source files are missing apache license headers. These are 
> necessary on import into apache/beam to pass RAT automation.
> Until this is done, we will exclude imported website/ sources from RAT. When 
> fixing this bug, the exclusion should also be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146164
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:11
Start Date: 20/Sep/18 23:11
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423363238
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146164)
Time Spent: 1h 10m  (was: 1h)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146163
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:10
Start Date: 20/Sep/18 23:10
Worklog Time Spent: 10m 
  Work Description: markflyhigh removed a comment on issue #6456: 
[BEAM-5334] Fix Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423363087
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146163)
Time Spent: 1h  (was: 50m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5334?focusedWorklogId=146162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146162
 ]

ASF GitHub Bot logged work on BEAM-5334:


Author: ASF GitHub Bot
Created on: 20/Sep/18 23:10
Start Date: 20/Sep/18 23:10
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6456: [BEAM-5334] Fix 
Python benmchmark with Gradle build
URL: https://github.com/apache/beam/pull/6456#issuecomment-423363087
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146162)
Time Spent: 50m  (was: 40m)

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4568) Add Apache headers to website sources

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4568?focusedWorklogId=146156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146156
 ]

ASF GitHub Bot logged work on BEAM-4568:


Author: ASF GitHub Bot
Created on: 20/Sep/18 22:45
Start Date: 20/Sep/18 22:45
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #555: [BEAM-4568] Add Apache 
headers to new source pages.
URL: https://github.com/apache/beam-site/pull/555#issuecomment-423358235
 
 
   R: @udim 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146156)
Time Spent: 40m  (was: 0.5h)

> Add Apache headers to website sources
> -
>
> Key: BEAM-4568
> URL: https://issues.apache.org/jira/browse/BEAM-4568
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently website source files are missing apache license headers. These are 
> necessary on import into apache/beam to pass RAT automation.
> Until this is done, we will exclude imported website/ sources from RAT. When 
> fixing this bug, the exclusion should also be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5450) Create user-facing documentation for running pipelines with the Universal Local Runner.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5450:
--
Priority: Minor  (was: Major)

> Create user-facing documentation for running pipelines with the Universal 
> Local Runner.
> ---
>
> Key: BEAM-5450
> URL: https://issues.apache.org/jira/browse/BEAM-5450
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>
> The ULR needs documentation explaining how users can run pipelines with it, 
> preferably on a visible platform such as the Beam website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5451) Add side input support to Universal Local Runner

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5451:
--
Priority: Minor  (was: Major)

> Add side input support to Universal Local Runner
> 
>
> Key: BEAM-5451
> URL: https://issues.apache.org/jira/browse/BEAM-5451
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Ruoyun Huang
>Priority: Minor
>
> Get side inputs working on the ULR. Since the ULR code is based on the direct 
> runner code there should already be some code that could be reused, but new 
> code will need to be written where side inputs would interact with 
> portability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146151
 ]

ASF GitHub Bot logged work on BEAM-5444:


Author: ASF GitHub Bot
Created on: 20/Sep/18 22:24
Start Date: 20/Sep/18 22:24
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6449: [BEAM-5444] Revert 
Flink PostCommit to parallelism 1
URL: https://github.com/apache/beam/pull/6449#issuecomment-423353838
 
 
   I realized that we can just *run* the Flink validates runner test to make 
sure this fixes the issue, rather than speculating. If it passes, LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146151)
Time Spent: 50m  (was: 40m)

> Post Commit Failure in Flink Runner
> ---
>
> Key: BEAM-5444
> URL: https://issues.apache.org/jira/browse/BEAM-5444
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, test-failures
>Reporter: Robert Burke
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The first instance of this failure included entirely your commits, could you 
> investigate?
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/]
> It appears expected exceptions are either being swallowed and not thrown.
> SEVERE: Pipeline execution failed 
> org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: 
> The user defined 'open()' method caused an exception: java.lang.Exception: 
> Hasn't yet thrown



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5430) Adjust implementation of CombineGroupedValues runner to use CombineFn.apply

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5430?focusedWorklogId=146150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146150
 ]

ASF GitHub Bot logged work on BEAM-5430:


Author: ASF GitHub Bot
Created on: 20/Sep/18 22:23
Start Date: 20/Sep/18 22:23
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on issue #6454: [BEAM-5430] Adjust 
implementation of CombineGroupedValues runner to use CombineFn.apply
URL: https://github.com/apache/beam/pull/6454#issuecomment-423353670
 
 
   @youngoli 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146150)
Time Spent: 20m  (was: 10m)

> Adjust implementation of CombineGroupedValues runner to use CombineFn.apply
> ---
>
> Key: BEAM-5430
> URL: https://issues.apache.org/jira/browse/BEAM-5430
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Daniel Oliveira
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In the [implementation for the runner of 
> Combine.GroupedValues|https://github.com/apache/beam/blob/bdd0081b49f8e7df6733dc8e8bc90dda3efc6621/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java#L217]
>  in the Java SDK it essentially re-implements what was already implemented in 
> [CombineFn.apply|https://github.com/apache/beam/blob/bdd0081b49f8e7df6733dc8e8bc90dda3efc6621/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java#L348].
> The implementation should instead just call CombineFn.apply for simplicity 
> and code reuse.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5452) Metrics collection and aggregation support on Universal Local Runner

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5452:
--
Priority: Minor  (was: Major)

> Metrics collection and aggregation support on Universal Local Runner
> 
>
> Key: BEAM-5452
> URL: https://issues.apache.org/jira/browse/BEAM-5452
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>
> As the APIs for metrics in portability are established, the ULR should add 
> support for portable metrics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5453) Add support for unlifted combines in the Universal Local Runner

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5453:
-

 Summary: Add support for unlifted combines in the Universal Local 
Runner
 Key: BEAM-5453
 URL: https://issues.apache.org/jira/browse/BEAM-5453
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


Now that some specs for unlifted combines in portability have been laid out, 
support for executing combines should be added to the ULR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146149
 ]

ASF GitHub Bot logged work on BEAM-5444:


Author: ASF GitHub Bot
Created on: 20/Sep/18 22:23
Start Date: 20/Sep/18 22:23
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6449: [BEAM-5444] Revert 
Flink PostCommit to parallelism 1
URL: https://github.com/apache/beam/pull/6449#issuecomment-423353598
 
 
   Run Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146149)
Time Spent: 40m  (was: 0.5h)

> Post Commit Failure in Flink Runner
> ---
>
> Key: BEAM-5444
> URL: https://issues.apache.org/jira/browse/BEAM-5444
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, test-failures
>Reporter: Robert Burke
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The first instance of this failure included entirely your commits, could you 
> investigate?
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/]
> It appears expected exceptions are either being swallowed and not thrown.
> SEVERE: Pipeline execution failed 
> org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: 
> The user defined 'open()' method caused an exception: java.lang.Exception: 
> Hasn't yet thrown



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5452) Metrics collection and aggregation support on Universal Local Runner

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5452:
-

 Summary: Metrics collection and aggregation support on Universal 
Local Runner
 Key: BEAM-5452
 URL: https://issues.apache.org/jira/browse/BEAM-5452
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


As the APIs for metrics in portability are established, the ULR should add 
support for portable metrics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146147
 ]

ASF GitHub Bot logged work on BEAM-5444:


Author: ASF GitHub Bot
Created on: 20/Sep/18 22:12
Start Date: 20/Sep/18 22:12
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6449: [BEAM-5444] Revert 
Flink PostCommit to parallelism 1
URL: https://github.com/apache/beam/pull/6449#issuecomment-423350982
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146147)
Time Spent: 0.5h  (was: 20m)

> Post Commit Failure in Flink Runner
> ---
>
> Key: BEAM-5444
> URL: https://issues.apache.org/jira/browse/BEAM-5444
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, test-failures
>Reporter: Robert Burke
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The first instance of this failure included entirely your commits, could you 
> investigate?
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/]
> It appears expected exceptions are either being swallowed and not thrown.
> SEVERE: Pipeline execution failed 
> org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: 
> The user defined 'open()' method caused an exception: java.lang.Exception: 
> Hasn't yet thrown



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5451) Add side input support to Universal Local Runner

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5451:
-

 Summary: Add side input support to Universal Local Runner
 Key: BEAM-5451
 URL: https://issues.apache.org/jira/browse/BEAM-5451
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Ruoyun Huang


Get side inputs working on the ULR. Since the ULR code is based on the direct 
runner code there should already be some code that could be reused, but new 
code will need to be written where side inputs would interact with portability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6038

[Apache Jenkins Server]

See 


Changes:

[scott] Upgrade gradle wrapper to latest (4.10)

[altay] change python fnapi container version

[scott] Revert "Merge pull request #6446: Upgrade gradle wrapper to latest

--
[...truncated 1.15 MB...]
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError

==
FAIL: test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 221, in test_apply_custom_transform
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
- >> end captured logging << -

==
FAIL: test_create (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 154, in test_create
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 

[jira] [Created] (BEAM-5450) Create user-facing documentation for running pipelines with the Universal Local Runner.

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5450:
-

 Summary: Create user-facing documentation for running pipelines 
with the Universal Local Runner.
 Key: BEAM-5450
 URL: https://issues.apache.org/jira/browse/BEAM-5450
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


The ULR needs documentation explaining how users can run pipelines with it, 
preferably on a visible platform such as the Beam website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5126) PreCommit filtering broken based upon PR contents

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622780#comment-16622780
 ] 

Scott Wegner commented on BEAM-5126:


[~lcwik] do you know if this is still an issue? When this was in development, 
we previously saw some issues if somebody was doing jenkins development and 
running the Seed Job without the filtering changes in their branch.

If this is still an issue, I can investigate. 

> PreCommit filtering broken based upon PR contents
> -
>
> Key: BEAM-5126
> URL: https://issues.apache.org/jira/browse/BEAM-5126
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Minor
>
> PR precommits used to be filtered by the contents of the PR.
>  
> Example PR that should have only spawned the Java PreCommit:
> https://github.com/apache/beam/pull/6159
>  
> This broke work done in BEAM-4445



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5380) [beam_PostCommit_Go_GradleBuild ][Flake] Flakes due to Gradle parallelization

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin resolved BEAM-5380.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> [beam_PostCommit_Go_GradleBuild ][Flake] Flakes due to Gradle parallelization
> -
>
> Key: BEAM-5380
> URL: https://issues.apache.org/jira/browse/BEAM-5380
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Fix For: Not applicable
>
>
> Synced with herohde, 
> Seems that the job fails to build tests properly. Suspect is parallelization.
> Disabling parallelization for now. Will monitor for couple of days for repro.
> Failing job url:
> https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/919/consoleFull



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5380) [beam_PostCommit_Go_GradleBuild ][Flake] Flakes due to Gradle parallelization

[Mikhail Gryzykhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622775#comment-16622775
 ] 

Mikhail Gryzykhin commented on BEAM-5380:
-

Disabling parallelization seem to fix issues. Closing this ticket.

> [beam_PostCommit_Go_GradleBuild ][Flake] Flakes due to Gradle parallelization
> -
>
> Key: BEAM-5380
> URL: https://issues.apache.org/jira/browse/BEAM-5380
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Fix For: Not applicable
>
>
> Synced with herohde, 
> Seems that the job fails to build tests properly. Suspect is parallelization.
> Disabling parallelization for now. Will monitor for couple of days for repro.
> Failing job url:
> https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/919/consoleFull



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #1563

[Apache Jenkins Server]

See 


Changes:

[scott] Ensure all pre-commits are run on gradle upgrade.

--
[...truncated 762.32 MB...]
INFO: Freeing task resources for 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (9/16) (aa61c052006ee5179251a58326e82e95).
Sep 20, 2018 10:02:52 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (5/16) (6cd0a12f80b9fb36873e48d751f1ade6) 
switched from RUNNING to FINISHED.
Sep 20, 2018 10:02:52 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (3/16) (531d4cfc4f4e0a9fc0d8a58bbdb5ff64) 
switched from RUNNING to FINISHED.
Sep 20, 2018 10:02:52 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 
GenerateSequence/Read(UnboundedCountingSource)/StripIds/ParMultiDo(StripIds) -> 
ParDo(Counting)/ParMultiDo(Counting) (5/16) (6cd0a12f80b9fb36873e48d751f1ade6).
Sep 20, 2018 10:02:52 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/GroupByKey 
-> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/ExpandIterable/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/ReifyTimestamps.RemoveWildcard/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Reshuffle/RestoreOriginalTimestamps/Reify.ExtractTimestampsFromValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
GenerateSequence/Read(UnboundedCountingSource)/Reshuffle/Values/Values/Map/ParMultiDo(Anonymous)
 -> GenerateSequence/Read(UnboundedCountingSource)/Read/ParMultiDo(Read) -> 

[jira] [Created] (BEAM-5449) Add ValidatesRunner tests for the ULR, including automated Pre- and Post-commits.

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5449:
-

 Summary: Add ValidatesRunner tests for the ULR, including 
automated Pre- and Post-commits.
 Key: BEAM-5449
 URL: https://issues.apache.org/jira/browse/BEAM-5449
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


The ULR should have working ValidatesRunner tests that can be used by devs, as 
well as automated testing support for them on the Apache Beam Jenkins.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5448) Support running user pipelines with the Universal Local Runner in Python.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira reassigned BEAM-5448:
-

Assignee: Ruoyun Huang  (was: Daniel Oliveira)

> Support running user pipelines with the Universal Local Runner in Python.
> -
>
> Key: BEAM-5448
> URL: https://issues.apache.org/jira/browse/BEAM-5448
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Ruoyun Huang
>Priority: Major
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the ULR. This task is for creating the build rules 
> needed to accomplish this for pipelines using the Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5448) Support running user pipelines with the Universal Local Runner in Python.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5448:
--
Description: In order to aid testing, devs should be able to write 
pipelines and then easily run them with the ULR. This task is for creating the 
build rules needed to accomplish this for pipelines using the Python SDK.  
(was: In order to aid testing, devs should be able to write pipelines and then 
easily run them with the ULR. The Flink runner seems to have this 
functionality, so the implementation could be based on that.)

> Support running user pipelines with the Universal Local Runner in Python.
> -
>
> Key: BEAM-5448
> URL: https://issues.apache.org/jira/browse/BEAM-5448
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the ULR. This task is for creating the build rules 
> needed to accomplish this for pipelines using the Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5448) Support running user pipelines with the Universal Local Runner in Python.

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5448:
-

 Summary: Support running user pipelines with the Universal Local 
Runner in Python.
 Key: BEAM-5448
 URL: https://issues.apache.org/jira/browse/BEAM-5448
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


In order to aid testing, devs should be able to write pipelines and then easily 
run them with the ULR. The Flink runner seems to have this functionality, so 
the implementation could be based on that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5447) Support running user pipelines with the Universal Local Runner in Go.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5447:
--
Description: In order to aid testing, devs should be able to write 
pipelines and then easily run them with the ULR. This should be generally 
trivial with Go as the existing local runner for Go uses a build rule that 
should be nearly identical to the build rule needed to complete this.  (was: In 
order to aid testing, devs should be able to write pipelines and then easily 
run them with the ULR. The Flink runner seems to have this functionality, so 
the implementation could be based on that.)

> Support running user pipelines with the Universal Local Runner in Go.
> -
>
> Key: BEAM-5447
> URL: https://issues.apache.org/jira/browse/BEAM-5447
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the ULR. This should be generally trivial with Go as the 
> existing local runner for Go uses a build rule that should be nearly 
> identical to the build rule needed to complete this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5447) Support running user pipelines with the Universal Local Runner in Go.

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-5447:
-

 Summary: Support running user pipelines with the Universal Local 
Runner in Go.
 Key: BEAM-5447
 URL: https://issues.apache.org/jira/browse/BEAM-5447
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


In order to aid testing, devs should be able to write pipelines and then easily 
run them with the ULR. The Flink runner seems to have this functionality, so 
the implementation could be based on that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5303) Get ULR ValidatesRunner tests passing

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5303:
--
Description: 
The ULR validatesRunner tests currently fail for the vast majority of tests. 
This bug is for tracking progress on getting all the tests to pass.

Gradle command to run the tests:
./gradlew -p runners/direct-java validatesPortableRunner

If you get any errors related to docker, make sure you have a docker container 
built:
./gradlew -p sdks/java/container docker

Bugs encountered while working on this should be tracked either as new 
sub-tasks to the parent issue or as seperate bugs that are marked as blocking 
this.

  was:
The Reference Runner validatesRunner tests currently fail for the vast majority 
of tests. This bug is for tracking progress on getting all the tests to pass.

Gradle command to run the tests:
./gradlew -p runners/direct-java validatesPortableRunner

If you get any errors related to docker, make sure you have a docker container 
built:
./gradlew -p sdks/java/container docker

Bugs encountered while working on this should be tracked either as new 
sub-tasks to the parent issue or as seperate bugs that are marked as blocking 
this.

Summary: Get ULR ValidatesRunner tests passing  (was: Get Reference 
Runner ValidatesRunner tests passing)

> Get ULR ValidatesRunner tests passing
> -
>
> Key: BEAM-5303
> URL: https://issues.apache.org/jira/browse/BEAM-5303
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> The ULR validatesRunner tests currently fail for the vast majority of tests. 
> This bug is for tracking progress on getting all the tests to pass.
> Gradle command to run the tests:
> ./gradlew -p runners/direct-java validatesPortableRunner
> If you get any errors related to docker, make sure you have a docker 
> container built:
> ./gradlew -p sdks/java/container docker
> Bugs encountered while working on this should be tracked either as new 
> sub-tasks to the parent issue or as seperate bugs that are marked as blocking 
> this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5093) Get Universal Local Runner to a state suitable for general use.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5093:
--
Description: The ULR currently cannot be used to run a pipeline directly, 
does not have adequate testing, and has bugs preventing pipelines from 
executing properly. This is a general bug for listing issues that must be fixed 
before the Reference Runner can be considered ready for general use.  (was: The 
Reference Runner currently cannot be used to run a pipeline directly, does not 
have adequate testing, and cannot run all pipelines properly. This is a general 
bug for holding issues that need to be fixed before the Reference Runner can be 
considered ready for general use.)

> Get Universal Local Runner to a state suitable for general use.
> ---
>
> Key: BEAM-5093
> URL: https://issues.apache.org/jira/browse/BEAM-5093
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> The ULR currently cannot be used to run a pipeline directly, does not have 
> adequate testing, and has bugs preventing pipelines from executing properly. 
> This is a general bug for listing issues that must be fixed before the 
> Reference Runner can be considered ready for general use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5093) Get Universal Local Runner to a state suitable for general use.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5093:
--
Summary: Get Universal Local Runner to a state suitable for general use.  
(was: Get ReferenceRunner to a state suitable for general use.)

> Get Universal Local Runner to a state suitable for general use.
> ---
>
> Key: BEAM-5093
> URL: https://issues.apache.org/jira/browse/BEAM-5093
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> The Reference Runner currently cannot be used to run a pipeline directly, 
> does not have adequate testing, and cannot run all pipelines properly. This 
> is a general bug for holding issues that need to be fixed before the 
> Reference Runner can be considered ready for general use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5304:
--
Description: In order to aid testing, devs should be able to write 
pipelines and then easily run them with the ULR. The Flink runner seems to have 
this functionality, so the implementation could be based on that.  (was: In 
order to aid testing, devs should be able to write pipelines and then easily 
run them with the Reference Runner. The Flink runner seems to have this 
functionality, so the implementation could be based on that.)

> Support running user pipelines with the Universal Local Runner in Java.
> ---
>
> Key: BEAM-5304
> URL: https://issues.apache.org/jira/browse/BEAM-5304
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the ULR. The Flink runner seems to have this 
> functionality, so the implementation could be based on that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #85

[Apache Jenkins Server]

See 


Changes:

[scott] Ensure all pre-commits are run on gradle upgrade.

--
[...truncated 566.38 KB...]
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 275, in 
test_combine_per_key
assert_that(res, equal_to([('a', 1.5), ('b', 3.0)]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_combine_per_key_1537479970.14_26a71cc8-7e63-4f0a-b4dd-4845c09f879c failed 
in state FAILED.

==
ERROR: test_create (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 63, in 
test_create
assert_that(p | beam.Create(['a', 'b']), equal_to(['a', 'b']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_create_1537479970.49_9775799c-10e6-4831-8266-88a4e54cfc19 failed in state 
FAILED.

==
ERROR: test_flatten (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 268, in 
test_flatten
assert_that(res, equal_to(['a', 'b', 'c', 'd']))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flatten_1537479970.97_97aea2a1-09d6-401e-a1a9-8a48aedcd07e failed in state 
FAILED.

==
ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 191, in 
test_flattened_side_input
equal_to([(None, {'a': 1, 'b': 2})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_flattened_side_input_1537479971.46_4ab8193e-c0e9-4a7f-a6f1-b46316f6df36 
failed in state FAILED.

==
ERROR: test_gbk_side_input (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 199, in 
test_gbk_side_input
equal_to([(None, {'a': [1]})]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_gbk_side_input_1537479971.98_72ccb4da-ef1d-41b1-b1ed-d8977a63ca2a failed 
in state FAILED.

==
ERROR: test_group_by_key (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 261, in 
test_group_by_key
assert_that(res, equal_to([('a', [1, 2]), ('b', [3])]))
  File "apache_beam/pipeline.py", line 414, in __exit__
self.run().wait_until_finish()
  File "apache_beam/runners/portability/portable_runner.py", line 209, in 
wait_until_finish
'Pipeline %s failed in state %s.' % (self._job_id, self._state))
RuntimeError: Pipeline 
test_group_by_key_1537479972.48_90f08857-e90a-44cd-be1d-cc1aa09d274e failed in 
state FAILED.

==
ERROR: test_large_elements (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 317, in 
test_large_elements
assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
  File "apache_beam/pipeline.py", line 414, in __exit__

[jira] [Updated] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-5304:
--
Summary: Support running user pipelines with the Universal Local Runner in 
Java.  (was: Create a gradle target to run arbitrary pipelines with the 
Reference Runner.)

> Support running user pipelines with the Universal Local Runner in Java.
> ---
>
> Key: BEAM-5304
> URL: https://issues.apache.org/jira/browse/BEAM-5304
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the Reference Runner. The Flink runner seems to have 
> this functionality, so the implementation could be based on that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)