[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=151041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151041
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 04/Oct/18 05:43
Start Date: 04/Oct/18 05:43
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on a change in pull request 
#6559: [BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#discussion_r222543106
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java
 ##
 @@ -83,13 +94,31 @@ private void runBigQueryIOReadPipeline() {
 
   @Test
   public void testBigQueryReadEmpty() throws Exception {
-setupTestEnvironment("empty");
+setupTestEnvironment("empty", false);
 runBigQueryIOReadPipeline();
   }
 
   @Test
-  public void testBigQueryRead1K() throws Exception {
-setupTestEnvironment("1K");
+  public void testBigQueryRead1G() throws Exception {
 
 Review comment:
   Missing 1M here as well?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151041)
Time Spent: 1h 20m  (was: 1h 10m)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5643) Fix broken user_score_it_test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5643?focusedWorklogId=151039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151039
 ]

ASF GitHub Bot logged work on BEAM-5643:


Author: ASF GitHub Bot
Created on: 04/Oct/18 05:27
Start Date: 04/Oct/18 05:27
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6567: [BEAM-5643] fix 
broken python it test
URL: https://github.com/apache/beam/pull/6567#issuecomment-426889886
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151039)
Time Spent: 40m  (was: 0.5h)

> Fix broken user_score_it_test
> -
>
> Key: BEAM-5643
> URL: https://issues.apache.org/jira/browse/BEAM-5643
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5596) Build BeamSQL as a Library

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5596:
---
Summary: Build BeamSQL as a Library  (was: BeamSQL as a Library)

> Build BeamSQL as a Library
> --
>
> Key: BEAM-5596
> URL: https://issues.apache.org/jira/browse/BEAM-5596
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>
> Build BeamSQL as a jar which does not include any extra dependencies. 
> Therefore we can use this jar to use BeamSQL as a lib.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5643) Fix broken user_score_it_test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5643?focusedWorklogId=151037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151037
 ]

ASF GitHub Bot logged work on BEAM-5643:


Author: ASF GitHub Bot
Created on: 04/Oct/18 05:23
Start Date: 04/Oct/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6567: [BEAM-5643] fix 
broken python it test
URL: https://github.com/apache/beam/pull/6567#issuecomment-426889143
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151037)
Time Spent: 0.5h  (was: 20m)

> Fix broken user_score_it_test
> -
>
> Key: BEAM-5643
> URL: https://issues.apache.org/jira/browse/BEAM-5643
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6163

[Apache Jenkins Server]

See 


Changes:

[github] Use standard format for python dependencies (#6560)

--
[...truncated 1.31 MB...]
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1068, in 
wrapper = lambda x: [fn(x)]
  File 
"
 line 185, in raise_error
raise RuntimeError('x')
RuntimeError: x

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"
 line 131, in _execute
response = task()
  File 
"
 line 166, in 
self._execute(lambda: worker.do_instruction(work), work)
  File 
"
 line 212, in do_instruction
request.instruction_id)
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
  File 
"
 line 349, in process_instruction_id
yield
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File 
"
 line 387, in process_bundle
input_op.process_encoded(data.data)
  File 
"
 line 123, in process_encoded
self.output(decoded_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 268, in process
self.output(windowed_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 414, in process
self.dofn_receiver.receive(o)
  File 
"
 line 673, in receive
self.process(windowed_value)
  File 
"
 line 679, in process
self._reraise_augmented(exn)
  File 
"
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 787, in process_outputs
self.main_receivers.receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 

[jira] [Updated] (BEAM-5644) make Planner switchable

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5644:
---
Description: 
We can make planner switchable here: 
[BeamQueryPlanner.java#L145|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145]

 

By doing so, we can have different planner implementation to support different 
SQL dialect.

  was:
We can make planner switchable here: 
[BeamQueryPlanner.java#L145|[https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145].]

 

By doing so, we can have different planner implementation to support different 
SQL dialect.


> make Planner switchable 
> 
>
> Key: BEAM-5644
> URL: https://issues.apache.org/jira/browse/BEAM-5644
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> We can make planner switchable here: 
> [BeamQueryPlanner.java#L145|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145]
>  
> By doing so, we can have different planner implementation to support 
> different SQL dialect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5644) make Planner switchable

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5644:
---
Issue Type: New Feature  (was: Improvement)

> make Planner switchable 
> 
>
> Key: BEAM-5644
> URL: https://issues.apache.org/jira/browse/BEAM-5644
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> We can make planner switchable here: 
> [BeamQueryPlanner.java#L145|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145]
>  
> By doing so, we can have different planner implementation to support 
> different SQL dialect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5644) make Planner switchable

[Rui Wang (JIRA)]

Rui Wang created BEAM-5644:
--

 Summary: make Planner switchable 
 Key: BEAM-5644
 URL: https://issues.apache.org/jira/browse/BEAM-5644
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang


We can make planner switchable here: 
[BeamQueryPlanner.java#L145|[https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L145].]

 

By doing so, we can have different planner implementation to support different 
SQL dialect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5643) Fix broken user_score_it_test

[yifan zou (JIRA)]

yifan zou created BEAM-5643:
---

 Summary: Fix broken user_score_it_test
 Key: BEAM-5643
 URL: https://issues.apache.org/jira/browse/BEAM-5643
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: yifan zou
Assignee: yifan zou






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5643) Fix broken user_score_it_test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5643?focusedWorklogId=151029=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151029
 ]

ASF GitHub Bot logged work on BEAM-5643:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:40
Start Date: 04/Oct/18 04:40
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6567: [BEAM-5643] fix 
broken python it test
URL: https://github.com/apache/beam/pull/6567#issuecomment-426883241
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151029)
Time Spent: 20m  (was: 10m)

> Fix broken user_score_it_test
> -
>
> Key: BEAM-5643
> URL: https://issues.apache.org/jira/browse/BEAM-5643
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5643) Fix broken user_score_it_test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5643?focusedWorklogId=151028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151028
 ]

ASF GitHub Bot logged work on BEAM-5643:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:40
Start Date: 04/Oct/18 04:40
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #6567: [BEAM-5643] 
fix broken python it test
URL: https://github.com/apache/beam/pull/6567
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151028)
Time Spent: 10m
Remaining Estimate: 0h

> Fix broken user_score_it_test
> -
>
> Key: BEAM-5643
> URL: https://issues.apache.org/jira/browse/BEAM-5643
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: yifan zou
>Assignee: yifan zou
>

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1242

[Apache Jenkins Server]

See 


Changes:

[github] Use standard format for python dependencies (#6560)

--
[...truncated 75.42 KB...]
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ERROR
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting mock (from -r 

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=151027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151027
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:34
Start Date: 04/Oct/18 04:34
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6559: Do Not Merge. 
[BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#issuecomment-426882361
 
 
   +R @jasonkuster 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151027)
Time Spent: 1h 10m  (was: 1h)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151026
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:22
Start Date: 04/Oct/18 04:22
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6563: 
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222534143
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java
 ##
 @@ -29,23 +31,35 @@
 import 
org.apache.beam.runners.fnexecution.environment.EmbeddedEnvironmentFactory;
 import 
org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory;
 import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineDebugOptions;
 
 /** Implementation of a {@link FlinkExecutableStageContext}. */
 class FlinkDefaultExecutableStageContext implements 
FlinkExecutableStageContext, AutoCloseable {
   private final JobBundleFactory jobBundleFactory;
 
   private static FlinkDefaultExecutableStageContext create(JobInfo jobInfo) {
-JobBundleFactory jobBundleFactory =
-DefaultJobBundleFactory.create(
-jobInfo,
-ImmutableMap.of(
-BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER),
-new DockerEnvironmentFactory.Provider(),
-BeamUrns.getUrn(StandardEnvironments.Environments.PROCESS),
-new ProcessEnvironmentFactory.Provider(),
-Environments.ENVIRONMENT_EMBEDDED, // Non Public urn for 
testing.
-new EmbeddedEnvironmentFactory.Provider()));
-return new FlinkDefaultExecutableStageContext(jobBundleFactory);
+try {
+  PipelineOptions pipelineOptions =
+  PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+  JobBundleFactory jobBundleFactory =
+  DefaultJobBundleFactory.create(
+  jobInfo,
+  ImmutableMap.of(
+  BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER),
+  new DockerEnvironmentFactory.Provider(
+  pipelineOptions
+  .as(PortablePipelineDebugOptions.class)
 
 Review comment:
   This line could be moved up to where the pipeline options are constructed. 
It may also be nice to take care of the IOException try/catch clutter in 
PipelineOptionsTranslation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151026)
Time Spent: 28h 20m  (was: 28h 10m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 28h 20m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5641) Move mock classes to sql src/main

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5641?focusedWorklogId=151025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151025
 ]

ASF GitHub Bot logged work on BEAM-5641:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:22
Start Date: 04/Oct/18 04:22
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6566: [BEAM-5641] Move 
mock classes to sql src/main
URL: https://github.com/apache/beam/pull/6566#issuecomment-426880611
 
 
   R: @apilloud @akedin @kennknowles 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151025)
Time Spent: 20m  (was: 10m)

> Move mock classes to sql src/main
> -
>
> Key: BEAM-5641
> URL: https://issues.apache.org/jira/browse/BEAM-5641
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #222

[Apache Jenkins Server]

See 


Changes:

[github] Use standard format for python dependencies (#6560)

--
[...truncated 51.05 MB...]
[ToKeyedWorkItem (12/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (12/16) 
(7797ec38f1d0fb227c0e8d7daec18d69).
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16) 
(aae5e033e4474f38475b40f9ff6ddc0b) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (12/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (12/16) 
(7797ec38f1d0fb227c0e8d7daec18d69) [FINISHED]
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (15/16) 
(2d641aead1c97185f32a8d45aa58325c).
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (15/16) 
(2d641aead1c97185f32a8d45aa58325c) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
4bd60e5184ef555a1486ec391d720b9d.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
754b714576026d1836f707ad6fd094e3.
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (13/16) (d6ecadf6ff2d591e60f2ac38db817714) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (13/16) 
(d6ecadf6ff2d591e60f2ac38db817714).
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16) 
(7775cc5f664e5bd98ae6b81a9feb40da) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (13/16) 
(d6ecadf6ff2d591e60f2ac38db817714) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
e8b2efb778e8bce6bfad597c36893874.
[ToKeyedWorkItem (3/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (3/16) (c6f53650472e6ae49323593aeca499ee) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (3/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (3/16) 
(c6f53650472e6ae49323593aeca499ee).
[ToKeyedWorkItem (3/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (3/16) 
(c6f53650472e6ae49323593aeca499ee) [FINISHED]
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (2/16) (aedf1c5c5da4961b4561ac32ec39afb5) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (2/16) 
(aedf1c5c5da4961b4561ac32ec39afb5).
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
68e24e43d4ed79dd669d86646558bdfc.
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (2/16) 
(aedf1c5c5da4961b4561ac32ec39afb5) [FINISHED]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16) 
(248ea559254ff7afad79d5b0cc282464) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (16/16) (e320e3c9b3e7f6143ef50c196ded4cee) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (16/16) 
(e320e3c9b3e7f6143ef50c196ded4cee).
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (16/16) 
(e320e3c9b3e7f6143ef50c196ded4cee) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 

[beam-site] branch asf-site updated (5122163 -> 214ffd0)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 5122163  Prepare repository for deployment.
 add 02d574a  Update release guide
 add ad9b65d  Edit suggestions
 add 4091c5d  This closes #572
 new 214ffd0  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/release-guide/index.html | 5 +++--
 src/contribute/release-guide.md | 5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 214ffd0a73207c0ac49f27910963316869d7dea6
Author: Mergebot 
AuthorDate: Thu Oct 4 04:15:01 2018 +

Prepare repository for deployment.
---
 content/contribute/release-guide/index.html | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/content/contribute/release-guide/index.html 
b/content/contribute/release-guide/index.html
index 2af7e88..5a0eec3 100644
--- a/content/contribute/release-guide/index.html
+++ b/content/contribute/release-guide/index.html
@@ -570,8 +570,9 @@ instructions.
 Attention: Only PMC has permission to perform this. If you 
are not a PMC, please ask for help in dev@ mailing list.
 
 
-  In JIRA, navigate to the https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions;>Beam  Administration  
Versions.
-  Add a new release: choose the next minor version number compared to the 
one currently underway, select today’s date as the Start Date, and choose Add.
+  In JIRA, navigate to https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions;>Beam  Administration  
Versions.
+  Add a new release. Choose the next minor version number after the 
version currently underway, select the release cut date (today’s date) as the 
Start Date, and choose Add.
+  At the end of the release, go to the same page and mark the recently 
released version as released. Use the ... menu and choose Release.
 
 
 Triage release-blocking issues 
in JIRA

[jira] [Work logged] (BEAM-5641) Move mock classes to sql src/main

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5641?focusedWorklogId=151020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151020
 ]

ASF GitHub Bot logged work on BEAM-5641:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:06
Start Date: 04/Oct/18 04:06
Worklog Time Spent: 10m 
  Work Description: amaliujia opened a new pull request #6566: [BEAM-5641] 
Move mock classes to sql src/main
URL: https://github.com/apache/beam/pull/6566
 
 
   Move mock classes to SQL src/main, which allows other module to use mock 
classes without depending SQL test jar but SQL jar.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151020)
Time Spent: 10m
Remaining Estimate: 0h

> Move mock classes to sql src/main
> -
>
> Key: BEAM-5641
> URL: https://issues.apache.org/jira/browse/BEAM-5641
> Project: Beam
>  Issue Type: Improvement
>  Components: 

[beam-site] branch mergebot updated (021f266 -> 4091c5d)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 021f266  This closes #570
 add 5122163  Prepare repository for deployment.
 new 02d574a  Update release guide
 new ad9b65d  Edit suggestions
 new 4091c5d  This closes #572

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../2018/10/03/beam-2.7.0.html}|  81 +++---
 content/blog/index.html|  32 
 content/feed.xml   | 174 ++---
 content/get-started/downloads/index.html   |   2 +
 content/index.html |  10 +-
 src/contribute/release-guide.md|   5 +-
 6 files changed, 188 insertions(+), 116 deletions(-)
 copy content/{beam/capability/2016/04/03/presentation-materials.html => 
blog/2018/10/03/beam-2.7.0.html} (78%)

[beam-site] 03/03: This closes #572

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4091c5d00e0c48c0bdb9f6dc71d421ddd4085fd4
Merge: 5122163 ad9b65d
Author: Mergebot 
AuthorDate: Thu Oct 4 04:11:38 2018 +

This closes #572

 src/contribute/release-guide.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

[beam-site] 01/03: Update release guide

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 02d574ab0ec1a1191a53b6c1b7d87da0f6cbdf1d
Author: Ahmet Altay 
AuthorDate: Wed Oct 3 16:50:45 2018 -0700

Update release guide
---
 src/contribute/release-guide.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md
index cddad39..b8cd62c 100644
--- a/src/contribute/release-guide.md
+++ b/src/contribute/release-guide.md
@@ -181,7 +181,8 @@ When contributors resolve an issue in JIRA, they are 
tagging it with a release t
 __Attention__: Only PMC has permission to perform this. If you are not a PMC, 
please ask for help in dev@ mailing list.
 
 1. In JIRA, navigate to the [`Beam > Administration > 
Versions`](https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions).
-1. Add a new release: choose the next minor version number compared to the one 
currently underway, select today’s date as the `Start Date`, and choose `Add`.
+1. Add a new release: choose the next minor version number compared to the one 
currently underway, select release cut date (today’s date) as the `Start Date`, 
and choose `Add`.
+1. At the end of the release, in the same page mark the recently released 
version as released. (Use the `...` menu and choose `Release`).
 
 ### Triage release-blocking issues in JIRA
 

[beam-site] 02/03: Edit suggestions

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ad9b65dbfdb0c98c4f381b05ede31416a5f7f95b
Author: Melissa Pashniak 
AuthorDate: Wed Oct 3 18:37:20 2018 -0700

Edit suggestions
---
 src/contribute/release-guide.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md
index b8cd62c..6a5e1c4 100644
--- a/src/contribute/release-guide.md
+++ b/src/contribute/release-guide.md
@@ -180,9 +180,9 @@ When contributors resolve an issue in JIRA, they are 
tagging it with a release t
 
 __Attention__: Only PMC has permission to perform this. If you are not a PMC, 
please ask for help in dev@ mailing list.
 
-1. In JIRA, navigate to the [`Beam > Administration > 
Versions`](https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions).
-1. Add a new release: choose the next minor version number compared to the one 
currently underway, select release cut date (today’s date) as the `Start Date`, 
and choose `Add`.
-1. At the end of the release, in the same page mark the recently released 
version as released. (Use the `...` menu and choose `Release`).
+1. In JIRA, navigate to [`Beam > Administration > 
Versions`](https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions).
+1. Add a new release. Choose the next minor version number after the version 
currently underway, select the release cut date (today’s date) as the `Start 
Date`, and choose `Add`.
+1. At the end of the release, go to the same page and mark the recently 
released version as released. Use the `...` menu and choose `Release`.
 
 ### Triage release-blocking issues in JIRA
 

[beam] branch asf-site updated: Publishing website 2018/10/04 04:11:17 at commit 63102f8

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 903aa3a  Publishing website 2018/10/04 04:11:17 at commit 63102f8
903aa3a is described below

commit 903aa3a9612d5bc795a638a09bd7fa7e2389408f
Author: jenkins 
AuthorDate: Thu Oct 4 04:11:18 2018 +

Publishing website 2018/10/04 04:11:17 at commit 63102f8

[jira] [Work logged] (BEAM-5529) Dataflow runner raises AssertionError if job takes > 50 seconds to go from PENDING to RUNNING

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5529?focusedWorklogId=151022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151022
 ]

ASF GitHub Bot logged work on BEAM-5529:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:09
Start Date: 04/Oct/18 04:09
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6535: [BEAM-5529] Before 
starting the final countdown, check that job is not in pre-running state
URL: https://github.com/apache/beam/pull/6535#issuecomment-426878821
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151022)
Time Spent: 2h 20m  (was: 2h 10m)

> Dataflow runner raises AssertionError if job takes > 50 seconds to go from 
> PENDING to RUNNING
> -
>
> Key: BEAM-5529
> URL: https://issues.apache.org/jira/browse/BEAM-5529
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Joar Wandborg
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> To reproduce:
>  - Try to run a Pipeline that takes longer than 50 seconds to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:" 
> (automatically executes {{pipeline.run().wait_until_finish()}} via 
> {{Pipeline.__exit__}}) on Dataflow.
>  - Your script will raise "AssertionError: Job did not reach to a terminal 
> state after waiting indefinitely." (from 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153])
> I think the issue is that the block at 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160]
>  unintentionally exits if it takes > 50 seconds for a job to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=151024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151024
 ]

ASF GitHub Bot logged work on BEAM-5417:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:09
Start Date: 04/Oct/18 04:09
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6560: [BEAM-5417] Use 
standard format for python dependencies
URL: https://github.com/apache/beam/pull/6560
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 2e955242701..254135b2efb 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -131,7 +131,7 @@ def get_version():
 
 REQUIRED_TEST_PACKAGES = [
 'nose>=1.3.7',
-'parameterized~=0.6.0',
+'parameterized>=0.6.0,<0.7.0',
 'numpy>=1.14.3,<2',
 'pyhamcrest>=1.9,<2.0',
 ]


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151024)
Time Spent: 7h  (was: 6h 50m)

> FileSystems.match behaviour diff between GCS and local file system
> --
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │   └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, 
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: 
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), 
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for 
> both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (66d7c86 -> 63102f8)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 66d7c86  Merge pull request #6558 from apilloud/decouple-agg
 add 63102f8  Use standard format for python dependencies (#6560)

No new revisions were added by this update.

Summary of changes:
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151023
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:09
Start Date: 04/Oct/18 04:09
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6563: 
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222532785
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineDebugOptions.java
 ##
 @@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Pipeline options for debug flags for portable runner. */
+@Experimental
+@Hidden
+public interface PortablePipelineDebugOptions extends ExperimentalOptions, 
PipelineOptions {
+
+  @Description("Delete dynamically created SDK Harness containers.")
 
 Review comment:
   After reading the email thread, shouldn't this flag be 
`retainDynamicSdkHarnessContainers` with default `false` - if so I agree it is 
a debug option.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151023)
Time Spent: 28h 10m  (was: 28h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 28h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5529) Dataflow runner raises AssertionError if job takes > 50 seconds to go from PENDING to RUNNING

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5529?focusedWorklogId=151021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151021
 ]

ASF GitHub Bot logged work on BEAM-5529:


Author: ASF GitHub Bot
Created on: 04/Oct/18 04:08
Start Date: 04/Oct/18 04:08
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6535: [BEAM-5529] Before 
starting the final countdown, check that job is not in pre-running state
URL: https://github.com/apache/beam/pull/6535#issuecomment-426878802
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151021)
Time Spent: 2h 10m  (was: 2h)

> Dataflow runner raises AssertionError if job takes > 50 seconds to go from 
> PENDING to RUNNING
> -
>
> Key: BEAM-5529
> URL: https://issues.apache.org/jira/browse/BEAM-5529
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Joar Wandborg
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> To reproduce:
>  - Try to run a Pipeline that takes longer than 50 seconds to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:" 
> (automatically executes {{pipeline.run().wait_until_finish()}} via 
> {{Pipeline.__exit__}}) on Dataflow.
>  - Your script will raise "AssertionError: Job did not reach to a terminal 
> state after waiting indefinitely." (from 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153])
> I think the issue is that the block at 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160]
>  unintentionally exits if it takes > 50 seconds for a job to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5642) test_pardo_state_only flaky (times out)

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-5642:
-

 Summary: test_pardo_state_only flaky (times out)
 Key: BEAM-5642
 URL: https://issues.apache.org/jira/browse/BEAM-5642
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Robert Bradshaw
 Fix For: 2.8.0


[https://builds.apache.org/job/beam_PreCommit_Python_Commit/1577/consoleFull]

 
*16:43:20* 
==*16:43:20*
 ERROR: test_pardo_state_only 
(apache_beam.runners.portability.portable_runner_test.PortableRunnerTest)*16:43:20*
 
--*16:43:20*
 Traceback (most recent call last):*16:43:20*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
 line 265, in test_pardo_state_only*16:43:20* equal_to(expected))*16:43:20* 
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/pipeline.py",
 line 423, in __exit__*16:43:20* self.run().wait_until_finish()*16:43:20*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
 line 242, in wait_until_finish*16:43:20* 
beam_job_api_pb2.GetJobStateRequest(job_id=self._job_id)):*16:43:20*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/target/.tox/py3/lib/python3.5/site-packages/grpc/_channel.py",
 line 363, in __next__*16:43:20* return self._next()*16:43:20*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/target/.tox/py3/lib/python3.5/site-packages/grpc/_channel.py",
 line 348, in _next*16:43:20* self._state.condition.wait()*16:43:20*   File 
"/usr/lib/python3.5/threading.py", line 293, in wait*16:43:20* 
waiter.acquire()*16:43:20*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/portable_runner_test.py",
 line 68, in handler*16:43:20* raise BaseException(msg)*16:43:20* 
BaseException: Timed out after 30 seconds.*16:43:20*  >> 
begin captured stdout << -



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5641) Move mock classes to sql src/main

[Rui Wang (JIRA)]

Rui Wang created BEAM-5641:
--

 Summary: Move mock classes to sql src/main
 Key: BEAM-5641
 URL: https://issues.apache.org/jira/browse/BEAM-5641
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4495) Create website pre-commits for apache/beam repository

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4495?focusedWorklogId=151019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151019
 ]

ASF GitHub Bot logged work on BEAM-4495:


Author: ASF GitHub Bot
Created on: 04/Oct/18 03:50
Start Date: 04/Oct/18 03:50
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6546: [BEAM-4495] Create precommit to stage website on GCS
URL: https://github.com/apache/beam/pull/6546#discussion_r222529496
 
 

 ##
 File path: .test-infra/jenkins/PrecommitJobBuilder.groovy
 ##
 @@ -103,6 +103,7 @@ class PrecommitJobBuilder {
   rootBuildScriptDir(commonJobProperties.checkoutDir)
   tasks(gradleTask)
   commonJobProperties.setGradleSwitches(delegate)
+  switches('-PpullId=${ghprbPullId}')
 
 Review comment:
   Done.  Named it githubPullRequestId


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151019)
Time Spent: 11h 50m  (was: 11h 40m)

> Create website pre-commits for apache/beam repository
> -
>
> Key: BEAM-4495
> URL: https://issues.apache.org/jira/browse/BEAM-4495
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing, website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: Not applicable
>
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5639) Add exception handling to single message transforms in Python SDK

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-5639:
-

Assignee: Jeff Klukas

> Add exception handling to single message transforms in Python SDK
> -
>
> Key: BEAM-5639
> URL: https://issues.apache.org/jira/browse/BEAM-5639
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Jeff Klukas
>Assignee: Jeff Klukas
>Priority: Minor
>
> Add methods to python SDK Map and FlatMap that allow users to specify 
> expected exceptions and tuple tags to associate with the with collections of 
> the successfully and unsuccessfully processed elements.
> See discussion on dev list:
> [https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-5638:
-

Assignee: Jeff Klukas  (was: Kenneth Knowles)

> Add exception handling to single message transforms in Java SDK
> ---
>
> Key: BEAM-5638
> URL: https://issues.apache.org/jira/browse/BEAM-5638
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Jeff Klukas
>Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Add methods to MapElements, FlatMapElements, and Filter that allow users to 
> specify expected exceptions and tuple tags to associate with the with 
> collections of the successfully and unsuccessfully processed elements.
> See discussion on dev list:
> https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4495) Create website pre-commits for apache/beam repository

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4495?focusedWorklogId=151018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151018
 ]

ASF GitHub Bot logged work on BEAM-4495:


Author: ASF GitHub Bot
Created on: 04/Oct/18 03:50
Start Date: 04/Oct/18 03:50
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6546: [BEAM-4495] Create precommit to stage website on GCS
URL: https://github.com/apache/beam/pull/6546#discussion_r222530865
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -184,3 +202,32 @@ task publishWebsite << {
 
 commitWebsite.dependsOn testWebsite
 publishWebsite.dependsOn commitWebsite
+
+/*
+ * Stages a pull request on GCS
+ * For example:
+ *   ./gradlew :beam-website:stageWebsite -PpullId=${ghprbPullId}
+ */
+task stageWebsite << {
+  assert project.hasProperty('pullId')
+  assert pullId.isInteger()
+
+  def gcs_path = "gs://apache-beam-website-pull-requests/${pullId}"
 
 Review comment:
   Added a -PwebsiteBucket to override it, but didn't make it empty by default 
because the PullRequestBuilder for jenkins jobs doesn't currently allow jobs to 
override flags. Perhaps it should.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151018)
Time Spent: 11h 40m  (was: 11.5h)

> Create website pre-commits for apache/beam repository
> -
>
> Key: BEAM-4495
> URL: https://issues.apache.org/jira/browse/BEAM-4495
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing, website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: Not applicable
>
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4495) Create website pre-commits for apache/beam repository

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4495?focusedWorklogId=151013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151013
 ]

ASF GitHub Bot logged work on BEAM-4495:


Author: ASF GitHub Bot
Created on: 04/Oct/18 03:37
Start Date: 04/Oct/18 03:37
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6546: [BEAM-4495] Create precommit to stage website on GCS
URL: https://github.com/apache/beam/pull/6546#discussion_r222529480
 
 

 ##
 File path: .test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
 ##
 @@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+scope: this,
+nameBase: 'Website_Stage_GCS',
 
 Review comment:
   Maybe not? Our naming for jenkins has been 
job_(PreCommit|PostCommit)_(Java|Python|Go|Website){_action}.groovy.
   There is already a legacy job_PreCommit_Website_Stage.groovy, which stages 
on beam-site.
   And a new job_PreCommit_Website_Stage_Test.groovy for verifying the website.
   Naming it job_StageWebsite would be inconsistent with the other jobs.
   When the change from beam-site to beam is done, then  
job_PreCommit_Website_Stage.groovy can be deleted and this one renamed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151013)
Time Spent: 11h 20m  (was: 11h 10m)

> Create website pre-commits for apache/beam repository
> -
>
> Key: BEAM-4495
> URL: https://issues.apache.org/jira/browse/BEAM-4495
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing, website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: Not applicable
>
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4495) Create website pre-commits for apache/beam repository

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4495?focusedWorklogId=151014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151014
 ]

ASF GitHub Bot logged work on BEAM-4495:


Author: ASF GitHub Bot
Created on: 04/Oct/18 03:37
Start Date: 04/Oct/18 03:37
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6546: [BEAM-4495] Create precommit to stage website on GCS
URL: https://github.com/apache/beam/pull/6546#discussion_r222529496
 
 

 ##
 File path: .test-infra/jenkins/PrecommitJobBuilder.groovy
 ##
 @@ -103,6 +103,7 @@ class PrecommitJobBuilder {
   rootBuildScriptDir(commonJobProperties.checkoutDir)
   tasks(gradleTask)
   commonJobProperties.setGradleSwitches(delegate)
+  switches('-PpullId=${ghprbPullId}')
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151014)
Time Spent: 11.5h  (was: 11h 20m)

> Create website pre-commits for apache/beam repository
> -
>
> Key: BEAM-4495
> URL: https://issues.apache.org/jira/browse/BEAM-4495
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing, website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: Not applicable
>
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

[Manu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637744#comment-16637744
 ] 

Manu Zhang commented on BEAM-5620:
--

[~tvalentyn], I've seen fixes like 
[https://github.com/apache/beam/blob/c34c367f5da6f9bef8a46471195470923a201af9/sdks/python/apache_beam/coders/coders_test_common.py#L61]
 where `assertItemsEqual` is assigned to `assertCountEqual` on Python 2. Do you 
think it's a good idea to put it in a subclass of `TestCase` ?

> Some tests use assertItemsEqual method, not available in Python 3
> -
>
> Key: BEAM-5620
> URL: https://issues.apache.org/jira/browse/BEAM-5620
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> See: 
> https://github.com/apache/beam/search?q=assertItemsEqual_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5089) Add an SDK harness container with Python 3 interpreter for portable pipelines.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5089?focusedWorklogId=151006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151006
 ]

ASF GitHub Bot logged work on BEAM-5089:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:43
Start Date: 04/Oct/18 01:43
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6565: [BEAM-5089] Update 
dependency versions and switch gradle build to use /target fol…
URL: https://github.com/apache/beam/pull/6565#issuecomment-426858065
 
 
   R: @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151006)
Time Spent: 0.5h  (was: 20m)

> Add an SDK harness container with Python 3 interpreter for portable pipelines.
> --
>
> Key: BEAM-5089
> URL: https://issues.apache.org/jira/browse/BEAM-5089
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=151009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151009
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 04/Oct/18 02:16
Start Date: 04/Oct/18 02:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6559: Do Not Merge. 
[BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#issuecomment-426862928
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151009)
Time Spent: 1h  (was: 50m)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5607) Typing library no longer provisional in Python 3.7

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5607?focusedWorklogId=151010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151010
 ]

ASF GitHub Bot logged work on BEAM-5607:


Author: ASF GitHub Bot
Created on: 04/Oct/18 02:42
Start Date: 04/Oct/18 02:42
Worklog Time Spent: 10m 
  Work Description: manuzhang commented on a change in pull request #6543: 
[BEAM-5607] Set python_version of typing library < 3.7.0
URL: https://github.com/apache/beam/pull/6543#discussion_r222523343
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -124,7 +124,7 @@ def get_version():
 'pytz>=2018.3,<=2018.4',
 'pyyaml>=3.12,<4.0.0',
 'pyvcf>=0.6.8,<0.7.0',
-'typing>=3.6.0,<3.7.0',
+'typing>=3.6.0,<3.7.0; python_version < "3.7.0"',
 
 Review comment:
   @aaltay @tvalentyn I'd love to make the cleanup in a separate PR. I've seen 
errors coming from `futures` package from Python 3.5 to 3.7 and interestingly 
*installing futures* solves it for me. I've logged the typing issue in 
[BEAM-5607](https://issues.apache.org/jira/browse/BEAM-5607) and feel free to 
add more info there. By the way, do you think it's worthwhile to set up a 
testing matrix for Python 2, 3.5, 3.6 and 3.7 on Jenkins respectively ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151010)
Time Spent: 40m  (was: 0.5h)

> Typing library no longer provisional in Python 3.7
> --
>
> Key: BEAM-5607
> URL: https://issues.apache.org/jira/browse/BEAM-5607
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This is an issue coming from future since the target version in the [initial 
> proposal|https://lists.apache.org/thread.html/5371469de567357b1431606f766217ef73a9098dc45046f51a6ecceb@%3Cdev.beam.apache.org%3E]
>  is up to 3.6. Running python tests in 3.7 would throw the following error 
> because the [typing library is no longer provisional in Python 
> 3.7|https://github.com/python/typing#important-dates]
> {code}
> sdks/python/.eggs/typing-3.6.6-py3.7.egg/typing.py", line 1004, in __new__
> self._abc_registry = extra._abc_registry
> AttributeError: type object 'Callable' has no attribute '_abc_registry'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5607) Typing library no longer provisional in Python 3.7

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-5607:
-
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-1251

> Typing library no longer provisional in Python 3.7
> --
>
> Key: BEAM-5607
> URL: https://issues.apache.org/jira/browse/BEAM-5607
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This is an issue coming from future since the target version in the [initial 
> proposal|https://lists.apache.org/thread.html/5371469de567357b1431606f766217ef73a9098dc45046f51a6ecceb@%3Cdev.beam.apache.org%3E]
>  is up to 3.6. Running python tests in 3.7 would throw the following error 
> because the [typing library is no longer provisional in Python 
> 3.7|https://github.com/python/typing#important-dates]
> {code}
> sdks/python/.eggs/typing-3.6.6-py3.7.egg/typing.py", line 1004, in __new__
> self._abc_registry = extra._abc_registry
> AttributeError: type object 'Callable' has no attribute '_abc_registry'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5011) Beam Dependency Update Request: org.springframework

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5011.
---
   Resolution: Won't Fix
 Assignee: (was: yifan zou)
Fix Version/s: Not applicable

> Beam Dependency Update Request: org.springframework
> ---
>
> Key: BEAM-5011
> URL: https://issues.apache.org/jira/browse/BEAM-5011
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
> 2018-07-25 20:34:52.205280
> Please review and upgrade the org.springframework to the latest 
> version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5593) Beam Dependency Update Request: org.springframework:spring-expression

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5593.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

The 5.* RELEASE version is incompatible. Won't upgrade it at this moment.

> Beam Dependency Update Request: org.springframework:spring-expression
> -
>
> Key: BEAM-5593
> URL: https://issues.apache.org/jira/browse/BEAM-5593
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:37.533954 
> -
> Please consider upgrading the dependency 
> org.springframework:spring-expression. 
> The current version is 4.3.18.RELEASE. The latest version is 
> 5.1.0.RELEASE 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5011) Beam Dependency Update Request: org.springframework

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637731#comment-16637731
 ] 

yifan zou commented on BEAM-5011:
-

[~altay] This parent issue should not have a fix version. The latest version is 
not compatible with beam. I will close it at this moment, and the dependency 
system will notify us once it has newer version available in the future.

> Beam Dependency Update Request: org.springframework
> ---
>
> Key: BEAM-5011
> URL: https://issues.apache.org/jira/browse/BEAM-5011
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-25 20:34:52.205280
> Please review and upgrade the org.springframework to the latest 
> version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5011) Beam Dependency Update Request: org.springframework

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-5011:

Fix Version/s: (was: 2.8.0)

> Beam Dependency Update Request: org.springframework
> ---
>
> Key: BEAM-5011
> URL: https://issues.apache.org/jira/browse/BEAM-5011
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-25 20:34:52.205280
> Please review and upgrade the org.springframework to the latest 
> version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151008
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 02:14
Start Date: 04/Oct/18 02:14
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6563: 
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222520317
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineDebugOptions.java
 ##
 @@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Pipeline options for debug flags for portable runner. */
+@Experimental
+@Hidden
+public interface PortablePipelineDebugOptions extends ExperimentalOptions, 
PipelineOptions {
+
+  @Description("Delete dynamically created SDK Harness containers.")
 
 Review comment:
   This doesn't seem to have anything to do with "debug". I think this should 
live in PortablePipelineOptions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151008)
Time Spent: 28h  (was: 27h 50m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 28h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5089) Add an SDK harness container with Python 3 interpreter for portable pipelines.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5089?focusedWorklogId=151005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151005
 ]

ASF GitHub Bot logged work on BEAM-5089:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:43
Start Date: 04/Oct/18 01:43
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6565: [BEAM-5089] Update 
dependency versions and switch gradle build to use /target fol…
URL: https://github.com/apache/beam/pull/6565#issuecomment-426858028
 
 
   Run Python Dataflow ValidatesContainer


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151005)
Time Spent: 20m  (was: 10m)

> Add an SDK harness container with Python 3 interpreter for portable pipelines.
> --
>
> Key: BEAM-5089
> URL: https://issues.apache.org/jira/browse/BEAM-5089
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151007
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 02:01
Start Date: 04/Oct/18 02:01
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup 
SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#issuecomment-426860694
 
 
   Run Python Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151007)
Time Spent: 27h 50m  (was: 27h 40m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 27h 50m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5089) Add an SDK harness container with Python 3 interpreter for portable pipelines.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5089?focusedWorklogId=151004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151004
 ]

ASF GitHub Bot logged work on BEAM-5089:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:42
Start Date: 04/Oct/18 01:42
Worklog Time Spent: 10m 
  Work Description: tvalentyn opened a new pull request #6565: [BEAM-5089] 
Update dependency versions and switch gradle build to use /target fol…
URL: https://github.com/apache/beam/pull/6565
 
 
   …ders for generated go artifacts as was before my last change.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151004)
Time Spent: 10m
Remaining Estimate: 0h

> Add an SDK harness container with Python 3 interpreter for portable pipelines.
> --
>
> Key: BEAM-5089
> URL: https://issues.apache.org/jira/browse/BEAM-5089
> Project: 

Build failed in Jenkins: beam_PostCommit_Python_Verify #6162

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-5112] Don't use BeamSqlInputRefExpression to pass Integer

--
[...truncated 1.32 MB...]
wrapper = lambda x: [fn(x)]
  File 
"
 line 185, in raise_error
raise RuntimeError('x')
RuntimeError: x

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"
 line 131, in _execute
response = task()
  File 
"
 line 166, in 
self._execute(lambda: worker.do_instruction(work), work)
  File 
"
 line 212, in do_instruction
request.instruction_id)
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
  File 
"
 line 349, in process_instruction_id
yield
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File 
"
 line 387, in process_bundle
input_op.process_encoded(data.data)
  File 
"
 line 123, in process_encoded
self.output(decoded_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 268, in process
self.output(windowed_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 414, in process
self.dofn_receiver.receive(o)
  File 
"
 line 673, in receive
self.process(windowed_value)
  File 
"
 line 679, in process
self._reraise_augmented(exn)
  File 
"
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 787, in process_outputs
self.main_receivers.receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 414, in process
self.dofn_receiver.receive(o)
  File 

[jira] [Commented] (BEAM-5513) Upgrade google-cloud-pubsub to 0.35.4

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637671#comment-16637671
 ] 

Udi Meiri commented on BEAM-5513:
-

https://github.com/apache/beam/pull/6564

> Upgrade google-cloud-pubsub to 0.35.4
> -
>
> Key: BEAM-5513
> URL: https://issues.apache.org/jira/browse/BEAM-5513
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> Version 0.35.4 is not the latest version out, but it is the latest supported 
> on Dataflow runner.
> [~markflyhigh]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=151002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151002
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:06
Start Date: 04/Oct/18 01:06
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6559: Do Not Merge. 
[BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#issuecomment-426852321
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151002)
Time Spent: 50m  (was: 40m)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151001
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:03
Start Date: 04/Oct/18 01:03
Worklog Time Spent: 10m 
  Work Description: angoenka removed a comment on issue #6563: [BEAM-4176] 
Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#issuecomment-426851724
 
 
   Python Flink ValidatesRunner Tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151001)
Time Spent: 27h 40m  (was: 27.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 27h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151000
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:03
Start Date: 04/Oct/18 01:03
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup 
SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#issuecomment-426851807
 
 
   Run Python Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151000)
Time Spent: 27.5h  (was: 27h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 27.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150999
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:02
Start Date: 04/Oct/18 01:02
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup 
SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#issuecomment-426851724
 
 
   Python Flink ValidatesRunner Tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150999)
Time Spent: 27h 20m  (was: 27h 10m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 27h 20m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150998
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 01:01
Start Date: 04/Oct/18 01:01
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup 
SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#issuecomment-426851550
 
 
   R: @tweise @mxm @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150998)
Time Spent: 27h 10m  (was: 27h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 27h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1241

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150997
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 04/Oct/18 00:53
Start Date: 04/Oct/18 00:53
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #6563: [BEAM-4176] 
Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563
 
 
   We should clean the containers as they quickly pile up.
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150997)
Time Spent: 27h  (was: 26h 50m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1240

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-5112] Don't use BeamSqlInputRefExpression to pass Integer

--
[...truncated 80.16 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 

Build failed in Jenkins: beam_PostCommit_Python_Verify #6161

[Apache Jenkins Server]

See 


Changes:

[huangry] Expose flag min_cpu_platform in PipelineOption.

[valentyn] Move SDK dependencies from Dockerfile to base_image_requirements.txt

[valentyn] Sort the dependencies alphabetically.

[valentyn] Update the versions of SDK dependencies to be installed into the

[valentyn] Add a Python container based on Python 3 interpreter.

[huangry] Update doc and log more info.

--
[...truncated 1.31 MB...]
wrapper = lambda x: [fn(x)]
  File 
"
 line 185, in raise_error
raise RuntimeError('x')
RuntimeError: x

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"
 line 131, in _execute
response = task()
  File 
"
 line 166, in 
self._execute(lambda: worker.do_instruction(work), work)
  File 
"
 line 212, in do_instruction
request.instruction_id)
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
  File 
"
 line 349, in process_instruction_id
yield
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File 
"
 line 387, in process_bundle
input_op.process_encoded(data.data)
  File 
"
 line 123, in process_encoded
self.output(decoded_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 268, in process
self.output(windowed_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 414, in process
self.dofn_receiver.receive(o)
  File 
"
 line 673, in receive
self.process(windowed_value)
  File 
"
 line 679, in process
self._reraise_augmented(exn)
  File 
"
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 787, in process_outputs
self.main_receivers.receive(windowed_value)
  File 
"
 line 87, in receive

[jira] [Created] (BEAM-5640) Portable python sdk worker leaks memory when PyOpenSSL package is present

[Micah Wylde (JIRA)]

Micah Wylde created BEAM-5640:
-

 Summary: Portable python sdk worker leaks memory when PyOpenSSL 
package is present
 Key: BEAM-5640
 URL: https://issues.apache.org/jira/browse/BEAM-5640
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-harness
Reporter: Micah Wylde
Assignee: Robert Bradshaw


When PyOpenSSL package is installed on a system (e.g., in a virtualenv) the 
python sdk_worker process leaks memory. I've validated this when using the 
flink portable runner in streaming mode, but it may occur in other 
configurations as well. The leak is pretty significant, amounting to tens of 
MBs/sec.

I've put together a reproduction for the issue 
[here|https://github.com/mwylde/beam/tree/micah_memory_leak]. That branch 
includes a flink streaming data source that generates data, as well as a python 
pipeline that demonstrates the issue.

To reproduce:
{code:java}
check out the branch:
$ git clone g...@github.com:mwylde/beam.git
$ git checkout micah_memory_leak

build the python docker container with pyopenssl installed:
$ cd beam
$ ./gradlew :beam-sdks-python-container:docker

start the job server with embedded flink cluster:
$ ./gradlew runShadow

run the pipeline:
$ ./gradlew :beam-sdks-python:streamingLeak{code}
 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=150995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150995
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 04/Oct/18 00:23
Start Date: 04/Oct/18 00:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6559: Do Not Merge. 
[BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#issuecomment-426845708
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150995)
Time Spent: 40m  (was: 0.5h)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=150994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150994
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 04/Oct/18 00:23
Start Date: 04/Oct/18 00:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6559: Do Not Merge. 
[BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#issuecomment-426845689
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150994)
Time Spent: 0.5h  (was: 20m)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4498?focusedWorklogId=150993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150993
 ]

ASF GitHub Bot logged work on BEAM-4498:


Author: ASF GitHub Bot
Created on: 04/Oct/18 00:21
Start Date: 04/Oct/18 00:21
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6556: 
[BEAM-4498] Add redirects and point javadoc/pydoc links to new location
URL: https://github.com/apache/beam/pull/6556#discussion_r222506522
 
 

 ##
 File path: website/src/.htaccess
 ##
 @@ -13,3 +13,12 @@ RewriteCond %{HTTPS} !on
 # * Redirect (R) permanently (301) to https://beam.apache.org/,
 # * Stop processing more rules (L).
 RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301]
+
+# Javadocs / pydocs are available only on the published website, published from
+# https://github.com/apache/beam-site/tree/release-docs
+# They were previously hosted within this repository, and published at the URL
+# path /documentation/sdks/(javadoc|pydoc)/..
+# The following redirect maintains the previously supported URLs.
+RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" 
"https://beam.apache.org/documentation/releases/$1$2;
+# Keep this updated to point to the current release.
+RedirectMatch "/releases/(javadoc|pydoc)/current(.*)" 
"https://beam.apache.org/documentation/releases/$1/2.6.0$2;
 
 Review comment:
   The matching rule could be future-proofed as 
`"/releases/([^/]+)/current(.*)"`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150993)
Time Spent: 2h 50m  (was: 2h 40m)

> Migrate release Javadocs / Pydocs to [asf-site] branch and update release 
> guide
> ---
>
> Key: BEAM-4498
> URL: https://issues.apache.org/jira/browse/BEAM-4498
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: Not applicable
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_VR_Flink #221

[Apache Jenkins Server]

See 

[beam] branch asf-site updated: Publishing website 2018/10/04 00:01:05 at commit 66d7c86

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 7798b01  Publishing website 2018/10/04 00:01:05 at commit 66d7c86
7798b01 is described below

commit 7798b0176667ae33f744ed4d7730cea33e65bad6
Author: jenkins 
AuthorDate: Thu Oct 4 00:01:06 2018 +

Publishing website 2018/10/04 00:01:05 at commit 66d7c86

Jenkins build is back to normal : beam_PreCommit_Website_Cron #136

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-5637) Python support for custom dataflow worker jar

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-5637:

Description: One of the slightly subtle aspects is that we would need to 
ignore one of the staged jars for portable Python jobs. That requires a change 
to the Python boot code: 
https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104

> Python support for custom dataflow worker jar
> -
>
> Key: BEAM-5637
> URL: https://issues.apache.org/jira/browse/BEAM-5637
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Priority: Major
>
> One of the slightly subtle aspects is that we would need to ignore one of the 
> staged jars for portable Python jobs. That requires a change to the Python 
> boot code: 
> https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5636) Java support for custom dataflow worker jar

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-5636:

Description: One of the slightly subtle aspects is that we would need to 
ignore one of the staged jars for portable Java jobs. That requires a change to 
the Java boot code: 
https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/java/container/boot.go#L107

> Java support for custom dataflow worker jar
> ---
>
> Key: BEAM-5636
> URL: https://issues.apache.org/jira/browse/BEAM-5636
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Boyuan Zhang
>Priority: Major
>
> One of the slightly subtle aspects is that we would need to ignore one of the 
> staged jars for portable Java jobs. That requires a change to the Java boot 
> code: 
> https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/java/container/boot.go#L107



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[Jeff Klukas (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637636#comment-16637636
 ] 

Jeff Klukas commented on BEAM-5638:
---

Work in progress PR: https://github.com/apache/beam/pull/6518

> Add exception handling to single message transforms in Java SDK
> ---
>
> Key: BEAM-5638
> URL: https://issues.apache.org/jira/browse/BEAM-5638
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Kenneth Knowles
>Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Add methods to MapElements, FlatMapElements, and Filter that allow users to 
> specify expected exceptions and tuple tags to associate with the with 
> collections of the successfully and unsuccessfully processed elements.
> See discussion on dev list:
> https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5639) Add exception handling to single message transforms in Python SDK

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-5639:
-

Assignee: (was: Ahmet Altay)

> Add exception handling to single message transforms in Python SDK
> -
>
> Key: BEAM-5639
> URL: https://issues.apache.org/jira/browse/BEAM-5639
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Jeff Klukas
>Priority: Minor
>
> Add methods to python SDK Map and FlatMap that allow users to specify 
> expected exceptions and tuple tags to associate with the with collections of 
> the successfully and unsuccessfully processed elements.
> See discussion on dev list:
> [https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5639) Add exception handling to single message transforms in Python SDK

[Jeff Klukas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Klukas updated BEAM-5639:
--

Related issue for Java SDK: https://issues.apache.org/jira/browse/BEAM-5638

> Add exception handling to single message transforms in Python SDK
> -
>
> Key: BEAM-5639
> URL: https://issues.apache.org/jira/browse/BEAM-5639
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Jeff Klukas
>Assignee: Ahmet Altay
>Priority: Minor
>
> Add methods to python SDK Map and FlatMap that allow users to specify 
> expected exceptions and tuple tags to associate with the with collections of 
> the successfully and unsuccessfully processed elements.
> See discussion on dev list:
> [https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5639) Add exception handling to single message transforms in Python SDK

[Jeff Klukas (JIRA)]

Jeff Klukas created BEAM-5639:
-

 Summary: Add exception handling to single message transforms in 
Python SDK
 Key: BEAM-5639
 URL: https://issues.apache.org/jira/browse/BEAM-5639
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Jeff Klukas
Assignee: Ahmet Altay


Add methods to python SDK Map and FlatMap that allow users to specify expected 
exceptions and tuple tags to associate with the with collections of the 
successfully and unsuccessfully processed elements.

See discussion on dev list:

[https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5638) Add exception handling to single message transforms in Java SDK

[Jeff Klukas (JIRA)]

Jeff Klukas created BEAM-5638:
-

 Summary: Add exception handling to single message transforms in 
Java SDK
 Key: BEAM-5638
 URL: https://issues.apache.org/jira/browse/BEAM-5638
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Jeff Klukas
Assignee: Kenneth Knowles


Add methods to MapElements, FlatMapElements, and Filter that allow users to 
specify expected exceptions and tuple tags to associate with the with 
collections of the successfully and unsuccessfully processed elements.

See discussion on dev list:

https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5625) PortableRunner.PipelineResult.cancel not working

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved BEAM-5625.

Resolution: Fixed

> PortableRunner.PipelineResult.cancel not working
> 
>
> Key: BEAM-5625
> URL: https://issues.apache.org/jira/browse/BEAM-5625
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Fails due to missing CancelJobRequest parameter:
>     self._job_service.Cancel()
> TypeError: __call__() takes at least 2 arguments (1 given)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637627#comment-16637627
 ] 

Thomas Weise commented on BEAM-5442:


Yes, the PR is open.

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5440) Add option to mount a directory inside SDK harness containers

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-5440:
---
Fix Version/s: (was: 2.8.0)

> Add option to mount a directory inside SDK harness containers
> -
>
> Key: BEAM-5440
> URL: https://issues.apache.org/jira/browse/BEAM-5440
> Project: Beam
>  Issue Type: New Feature
>  Components: java-fn-execution
>Reporter: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
>
> While experimenting with the Python SDK locally, I found it inconvenient that 
> I can't mount a host directory to the Docker containers, i.e. the input must 
> already be in the container and the results of a Write remain inside the 
> container. For local testing, users may want to mount a host directory.
> Since BEAM-5288 the {{Environment}} carries explicit environment information, 
> we could a) add volume args to the {{DockerPayload}}, or b) provide a general 
> Docker arguments field.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4518) Errors when running Python Game stats with a low fixed_window_duration in the DirectRunner

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-4518:
--
Fix Version/s: (was: 2.8.0)

> Errors when running Python Game stats with a low fixed_window_duration in the 
> DirectRunner
> --
>
> Key: BEAM-4518
> URL: https://issues.apache.org/jira/browse/BEAM-4518
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Ahmet Altay
>Priority: Major
>
> Using the injector and the following command to start the DirectRunner 
> pipeline:
> python -m apache_beam.examples.complete.game.game_stats \
> --project=google.com:clouddfe \
> --topic projects/google.com:clouddfe/topics/leader_board-$USER-topic-1 \
> --dataset ${USER}_test --fixed_window_duration 1
> Fails with:
> ValueError: PCollection of size 2 with more than one element accessed as a 
> singleton view. First two elements encountered are "13.93", "11.78". 
> [while running 'CalculateSpammyUsers/ProcessAndFilter']
> Offending code is here:
> global_mean_score = (
>  sum_scores
>  | beam.Values()
>  | beam.CombineGlobally(beam.combiners.MeanCombineFn())\
>  .as_singleton_view())
>  # Filter the user sums using the global mean.
>  filtered = (
>  sum_scores
>  # Use the derived mean total score (global_mean_score) as a side input.
>  | 'ProcessAndFilter' >> beam.Filter(
>  lambda key_score, global_mean:\
>  key_score[1] > global_mean * self.SCORE_WEIGHT,
>  global_mean_score))
> Since global_mean_score is the result of CombineGlobally, this is either an 
> issue with CombineGlobally or side inputs implementation in DirectRunner. The 
> latter is more likely since it works on DataflowRunner.
> cc: [~mariagh]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4750) Beam performance degraded significantly since 2.4

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-4750:
--
Fix Version/s: (was: 2.8.0)
   2.9.0

> Beam performance degraded significantly since 2.4
> -
>
> Key: BEAM-4750
> URL: https://issues.apache.org/jira/browse/BEAM-4750
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Vojtech Janota
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.9.0
>
>
> Starting from Beam 2.4 onwards the *InMemoryStateInternals* class in the 
> *beam-runners-core-java* module does an expensive Coder encode/decode 
> operation when copying object state. This has significant impact on 
> performance and pipelines that previously took low minutes do not finish 
> within hours in our case. Based on the discussion on the dev mailing list, 
> the main motivation for this change was to enforce Coder sanity, something 
> that should arguably remain within the realm of the DirectRunner and should 
> not leak into the core layer.
> Links to commits that introduced the new behaviour:
>  * [https://github.com/apache/beam/commit/32a427c]
>  * [https://github.com/apache/beam/commit/8151d82]
>  
> Additional details and surrounding discussion can be found here:
>  * 
> https://lists.apache.org/thread.html/1e329318a4dafe27b9ff304d9460d05d8966e5ceebaf4ebfb948e2b8@%3Cdev.beam.apache.org%3E
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4215) Update Python SDK dependencies for next release

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-4215:
--
Fix Version/s: (was: 2.8.0)
   2.9.0

> Update Python SDK dependencies for next release
> ---
>
> Key: BEAM-4215
> URL: https://issues.apache.org/jira/browse/BEAM-4215
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.5.0
>Reporter: Valentyn Tymofieiev
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.9.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5011) Beam Dependency Update Request: org.springframework

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-5011:
-

Assignee: yifan zou

> Beam Dependency Update Request: org.springframework
> ---
>
> Key: BEAM-5011
> URL: https://issues.apache.org/jira/browse/BEAM-5011
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.8.0
>
>
> 2018-07-25 20:34:52.205280
> Please review and upgrade the org.springframework to the latest 
> version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5011) Beam Dependency Update Request: org.springframework

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637624#comment-16637624
 ] 

Ahmet Altay commented on BEAM-5011:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> Beam Dependency Update Request: org.springframework
> ---
>
> Key: BEAM-5011
> URL: https://issues.apache.org/jira/browse/BEAM-5011
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: 2.8.0
>
>
> 2018-07-25 20:34:52.205280
> Please review and upgrade the org.springframework to the latest 
> version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5440) Add option to mount a directory inside SDK harness containers

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637621#comment-16637621
 ] 

Ahmet Altay commented on BEAM-5440:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

It does not sound like a release blocker.

> Add option to mount a directory inside SDK harness containers
> -
>
> Key: BEAM-5440
> URL: https://issues.apache.org/jira/browse/BEAM-5440
> Project: Beam
>  Issue Type: New Feature
>  Components: java-fn-execution
>Reporter: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>
> While experimenting with the Python SDK locally, I found it inconvenient that 
> I can't mount a host directory to the Docker containers, i.e. the input must 
> already be in the container and the results of a Write remain inside the 
> container. For local testing, users may want to mount a host directory.
> Since BEAM-5288 the {{Environment}} carries explicit environment information, 
> we could a) add volume args to the {{DockerPayload}}, or b) provide a general 
> Docker arguments field.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5637) Python support for custom dataflow worker jar

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-5637:
---

 Summary: Python support for custom dataflow worker jar
 Key: BEAM-5637
 URL: https://issues.apache.org/jira/browse/BEAM-5637
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Henning Rohde






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637623#comment-16637623
 ] 

Ahmet Altay commented on BEAM-5419:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> Build multiple versions of the Flink Runner against different Flink versions
> 
>
> Key: BEAM-5419
> URL: https://issues.apache.org/jira/browse/BEAM-5419
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system, runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.8.0
>
>
> Following up on a discussion on the mailing list.
> We want to keep the Flink version stable across different versions to avoid 
> upgrade pain for long-term users. At the same time, there are users out there 
> with newer Flink clusters and developers also want to utilize new Flink 
> features.
> It would be great to build multiple versions of the Flink Runner against 
> different Flink versions.
> When the upgrade is as simple as changing the version property in the build 
> script, this should be pretty straight-forward. If not, having a "base 
> version" and applying a patch during the build could be an option. We should 
> avoid duplicating any Runner code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5267) Update Flink Runner to Flink 1.6.x

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637622#comment-16637622
 ] 

Ahmet Altay commented on BEAM-5267:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> Update Flink Runner to Flink 1.6.x
> --
>
> Key: BEAM-5267
> URL: https://issues.apache.org/jira/browse/BEAM-5267
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For the next release, the Flink version should be bumped. As changes for 
> 2.7.0 are already frozen, it's going to be 2.8.0. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5326) SDK support for custom dataflow worker jar

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-5326:

Summary: SDK support for custom dataflow worker jar  (was: Support for 
custom dataflow worker jar)

> SDK support for custom dataflow worker jar
> --
>
> Key: BEAM-5326
> URL: https://issues.apache.org/jira/browse/BEAM-5326
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Boyuan Zhang
>Priority: Major
>
> Doc: 
> https://docs.google.com/document/d/1-m-GzkYWIODKOEl1ZSUNXYbcGRvRr3QkasfHsJxbuoA/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5528) Java PortableRunner pipeline fails on FlinkRunner due to CREATE_VIEW

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637619#comment-16637619
 ] 

Ahmet Altay commented on BEAM-5528:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> Java PortableRunner pipeline fails on FlinkRunner due to CREATE_VIEW
> 
>
> Key: BEAM-5528
> URL: https://issues.apache.org/jira/browse/BEAM-5528
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We recently removed the CREATE_VIEW translation which was a NOOP. Turns out 
> we still need to support it until it is removed. 
> {noformat}
> java.lang.IllegalArgumentException: Unknown type of URN 
> beam:transform:create_view:v1 for PTransform with id 
> TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/View.CreatePCollectionView.
> at 
> org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.urnNotFound(FlinkBatchPortablePipelineTranslator.java:578)
> at 
> org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:233)
> at 
> org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:112)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-1240) Create RabbitMqIO

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637620#comment-16637620
 ] 

Ahmet Altay commented on BEAM-1240:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 12h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5636) Java support for custom dataflow worker jar

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-5636:
---

 Summary: Java support for custom dataflow worker jar
 Key: BEAM-5636
 URL: https://issues.apache.org/jira/browse/BEAM-5636
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Henning Rohde
Assignee: Boyuan Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5625) PortableRunner.PipelineResult.cancel not working

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637618#comment-16637618
 ] 

Ahmet Altay commented on BEAM-5625:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> PortableRunner.PipelineResult.cancel not working
> 
>
> Key: BEAM-5625
> URL: https://issues.apache.org/jira/browse/BEAM-5625
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Fails due to missing CancelJobRequest parameter:
>     self._job_service.Cancel()
> TypeError: __call__() takes at least 2 arguments (1 given)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637617#comment-16637617
 ] 

Ahmet Altay commented on BEAM-5442:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5456) Update google-api-client libraries to 1.25

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637614#comment-16637614
 ] 

Ahmet Altay commented on BEAM-5456:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

> Update google-api-client libraries to 1.25
> --
>
> Key: BEAM-5456
> URL: https://issues.apache.org/jira/browse/BEAM-5456
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.8.0
>
>
> This version updates authentication URLs 
> ([https://github.com/googleapis/google-api-java-client/releases)] that is 
> needed for certain features.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637616#comment-16637616
 ] 

Ahmet Altay commented on BEAM-5367:
---

What is the plan for this JIRA? Are we targeting 2.8.0 with cut date 10/10?

This does not sound like a blocker for the release.

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #220

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-5112] Don't use BeamSqlInputRefExpression to pass Integer

--
[...truncated 51.05 MB...]
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16) 
(13271d71286b31361c47f22dcb4ef66f).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16) 
(0f7a306bdff04acd87b90959123d1f6e) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(11/16) (2b29ccf3e47aab1acb557c9a403f6647) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (13/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (13/16) 
(990a2f687ecaad96ec3ac86604acb981) [FINISHED]
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (13/16) (ac523097975bc52bd7f4326196430458) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (12/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (12/16) (c4fc2a165637dd25b6928a4f40678d0c) switched from 
RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16) 
(89510d54b0a726238fe7139284e616a8).
[ToKeyedWorkItem (14/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (14/16) 
(5929dfeea448e2cb790ecf1d0330567b) [FINISHED]
[ToKeyedWorkItem (12/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (12/16) 
(c4fc2a165637dd25b6928a4f40678d0c).
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (13/16) 
(ac523097975bc52bd7f4326196430458).
[ToKeyedWorkItem (8/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (8/16) (9337d7788c7eb2331dfa9f6596cffc6e) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (12/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (12/16) 
(c4fc2a165637dd25b6928a4f40678d0c) [FINISHED]
[ToKeyedWorkItem (8/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (8/16) 
(9337d7788c7eb2331dfa9f6596cffc6e).
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (10/16) 
(a718cac1e7e5a61fab8e719cd04f3d8d).
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16) 
(7483b692c45c69dd39b35ee634c8a611) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (16/16) (108d6704932a2dd52bc3cfe9055702a5) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (4/16) (8d4ffc679f1e1dd6bae12b2691de0962) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (4/16) 
(8d4ffc679f1e1dd6bae12b2691de0962).
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (10/16) 
(a718cac1e7e5a61fab8e719cd04f3d8d) [FINISHED]
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (6/16) 
(41f46603d910111ceaecc8499b028b65) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
2c7a00ed4f2f2e9cd0e7811bb3fa229b.
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (15/16) (afb885142a424ed750acadee9e28ec17) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (15/16) 
(afb885142a424ed750acadee9e28ec17).
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 

[jira] [Updated] (BEAM-5367) [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website test failure

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-5367:
--
Fix Version/s: (was: 2.7.0)

> [beam_Release_Gradle_NightlySnapshot] is broken due to apache/beam website 
> test failure
> ---
>
> Key: BEAM-5367
> URL: https://issues.apache.org/jira/browse/BEAM-5367
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> apache/website test is breaking nightly build. Since we don't have stable 
> apache/beam website, we should remove it from the build for now. Until we 
> have an actual plan to fully migrate from asf beam-site, we should disable 
> apache/website checks from Beam build.
> Failing job url: [https://scans.gradle.com/s/uxb6mdigqj4n4/console-log#L23465]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5112) Investigate if Calcite can generate functions that we need

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5112?focusedWorklogId=150981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150981
 ]

ASF GitHub Bot logged work on BEAM-5112:


Author: ASF GitHub Bot
Created on: 03/Oct/18 23:29
Start Date: 03/Oct/18 23:29
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #6562: [BEAM-5112] 
UdafImpl isn't part of interpreter
URL: https://github.com/apache/beam/pull/6562
 
 
   Move UdafImpl out of interpreter. #6417 will delete interpreter, and this 
code isn't related to it.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 150981)
Time Spent: 1h 10m  (was: 1h)

> Investigate if Calcite can generate functions that we need
> --
>
> Key: BEAM-5112
> URL: https://issues.apache.org/jira/browse/BEAM-5112
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>  

[jira] [Assigned] (BEAM-5635) FR: Enable Transactional writes with DatastoreIO

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-5635:
-

Assignee: Kenneth Knowles

> FR: Enable Transactional writes with DatastoreIO
> 
>
> Key: BEAM-5635
> URL: https://issues.apache.org/jira/browse/BEAM-5635
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Alex Amato
>Assignee: Kenneth Knowles
>Priority: Major
>
> I have seen a user who would like to use Datastore Transactions to rollback a 
> set of records if one of them fails to write. Let's consider this use case 
> for DatastoreIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/10/03 23:29:04 at commit 66d7c86

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 87d4d5e  Publishing website 2018/10/03 23:29:04 at commit 66d7c86
87d4d5e is described below

commit 87d4d5ea97fdb5a65ddf1cbd06cca97b0a52ee9b
Author: jenkins 
AuthorDate: Wed Oct 3 23:29:05 2018 +

Publishing website 2018/10/03 23:29:04 at commit 66d7c86