[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=310342&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310342
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 11/Sep/19 06:49
Start Date: 11/Sep/19 06:49
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-530246967
 
 
   Run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310342)
Time Spent: 4.5h  (was: 4h 20m)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310338&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310338
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 06:37
Start Date: 11/Sep/19 06:37
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9506: 
[BEAM-8105] Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r323076674
 
 

 ##
 File path: release/src/main/scripts/publish_docker_images.sh
 ##
 @@ -0,0 +1,77 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script will generate and publish docker images for each language 
version to Docker Hub:
+# 1. Generate images tagged with :{RELEASE}
+# 2. Publish images tagged with :{RELEASE}
+# 3. Tag images with :latest tag and publish.
+# 4. Clearn up images.
+
+set -e
+
+source release/src/main/scripts/build_release_candidate.sh
+
+echo "Publish SDK docker images to Docker Hub."
+echo "Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+  echo "Publishing SDK docker images on docker hub="
+  cd ~
+  if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
+rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+  fi
+  mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
+  cd ${LOCAL_PYTHON_STAGING_DIR}
+
+  echo '---Cloning Beam Release Branch-'
+  git clone ${GIT_REPO_URL}
+  cd ${BEAM_ROOT_DIR}
+  git checkout ${RELEASE_BRANCH}
+
+  echo '---Generating and Pushing Python 
images-'
+  ./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}
 
 Review comment:
   This build steps lgtm in general. However, I'm wondering is it possible to 
reuse existing staged rc image instead of building another one? Rerun gradle 
build takes longer time and potentially has risk to introduce differences. 
Probably docker pull + tag + push?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310338)
Time Spent: 3h 50m  (was: 3h 40m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310339&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310339
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 06:37
Start Date: 11/Sep/19 06:37
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9506: 
[BEAM-8105] Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r323055624
 
 

 ##
 File path: release/src/main/scripts/build_release_candidate.sh
 ##
 @@ -198,6 +200,45 @@ if [[ $confirmation = "y" ]]; then
   rm -rf ~/${PYTHON_ARTIFACTS_DIR}
 fi
 
+echo "[Current Step]: Stage SDK docker images"
+echo "Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+  echo "Staging SDK docker images on docker hub="
+  cd ~
+  if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
+rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+  fi
+  mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
+  cd ${LOCAL_PYTHON_STAGING_DIR}
+
+  echo '---Cloning Beam Release Branch-'
+  git clone ${GIT_REPO_URL}
+  cd ${BEAM_ROOT_DIR}
+  git checkout ${RELEASE_BRANCH}
+
+  echo '---Generating and Pushing Python 
images-'
+  ./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc${RC_NUM}
+  for ver in "${PYTHON_VER[@]}"; do
+ docker push apachebeam/${ver}_sdk:${RELEASE}_rc${RC_NUM} &
+  done
+
+  echo '---Generating and Pushing Java images-'
+  ./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}_rc${RC_NUM}
+
+  echo '---Generating and Pushing Go images-'
+  ./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}_rc${RC_NUM}
+
+  rm -rf ~/${PYTHON_ARTIFACTS_DIR}
 
 Review comment:
   Why do we need to remove `~/python` here?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310339)
Time Spent: 3h 50m  (was: 3h 40m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310312&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310312
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 05:20
Start Date: 11/Sep/19 05:20
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9487: 
[BEAM-8165]Change default docker images name
URL: https://github.com/apache/beam/pull/9487#issuecomment-530209453
 
 
   > OK. I am fine with this change. I will only call 'sdk_version', 
'python_sdk_version', since that is only  applicable to python. And let's plan 
to clean it after 2.16. - LGTM.
   
   Thanks, I addressed your comment.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310312)
Time Spent: 2h 50m  (was: 2h 40m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8201) clean up the current container API

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8201:
---
Status: Open  (was: Triage Needed)

> clean up the current container API
> --
>
> Key: BEAM-8201
> URL: https://issues.apache.org/jira/browse/BEAM-8201
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
>
> From [~robertwb]
> As part of this project, I propose we look at and clean up the current 
> container API before we "release" it as public and stable. IIRC, we currently 
> provide the worker arguments through a combination of (1) environment 
> variables (2) command line parameters to docker and (3) via the provisioning 
> API. It would be good to have a more principled approach to specifying 
> arguments (either all the same way, or if they vary, good reason for doing so 
> rather than by historical accident).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310296
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 04:05
Start Date: 11/Sep/19 04:05
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #9506: 
[BEAM-8105] Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r323051426
 
 

 ##
 File path: release/src/main/scripts/publish_docker_images.sh
 ##
 @@ -0,0 +1,77 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script will generate and publish docker images for each language 
version to Docker Hub:
+# 1. Generate images tagged with :{RELEASE}
+# 2. Publish images tagged with :{RELEASE}
+# 3. Tag images with :latest tag and publish.
+# 4. Clearn up images.
+
+set -e
+
+source release/src/main/scripts/build_release_candidate.sh
+
+echo "Publish SDK docker images to Docker Hub."
+echo "Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+  echo "Publishing SDK docker images on docker hub="
+  cd ~
+  if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
+rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+  fi
+  mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
+  cd ${LOCAL_PYTHON_STAGING_DIR}
+
+  echo '---Cloning Beam Release Branch-'
+  git clone ${GIT_REPO_URL}
+  cd ${BEAM_ROOT_DIR}
+  git checkout ${RELEASE_BRANCH}
+
+  echo '---Generating and Pushing Python 
images-'
+  ./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}
+  for ver in "${PYTHON_VER[@]}"; do
+ docker push apachebeam/${ver}_sdk:${RELEASE}
+ docker tag apachebeam/${ver}_sdk:${RELEASE} apachebeam/${ver}_sdk:latest
+ docker push apachebeam/${ver}_sdk:latest
+  done
+
+  echo '---Generating and Pushing Java images-'
+  ./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}
+  docker tag apachebeam/java_sdk:${RELEASE} apachebeam/java_sdk:latest
+  docker push apachebeam/java_sdk:latest
+
+  echo '---Generating and Pushing Go images-'
+  ./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}
+  docker tag apachebeam/go_sdk:${RELEASE} apachebeam/go_sdk:latest
+  docker push apachebeam/go_sdk:latest
 
 Review comment:
   `dockerPush` task at L60 will do both: build an image and push it to remote. 
(with {RELEASE} tag.)
   L62 is changing latest image to the most recent image, which is current one.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310296)
Time Spent: 3h 40m  (was: 3.5h)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310277
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 03:59
Start Date: 11/Sep/19 03:59
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9487: 
[BEAM-8165]Change default docker images name
URL: https://github.com/apache/beam/pull/9487#issuecomment-530209453
 
 
   > OK. I am fine with this change. I will only call 'sdk_version', 
'python_sdk_version', since that is only applicable to python. And let's plan 
to clean it after 2.16. - LGTM.
   Thanks, I addressed your comment.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310277)
Time Spent: 2h 40m  (was: 2.5h)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310265&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310265
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 03:04
Start Date: 11/Sep/19 03:04
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530199924
 
 
   Yes, and full error messages are 
[here](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/)
 and 
[here](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/testReport/).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310265)
Time Spent: 1h 20m  (was: 1h 10m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310264&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310264
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 02:26
Start Date: 11/Sep/19 02:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #9487: [BEAM-8165]Change 
default docker images name
URL: https://github.com/apache/beam/pull/9487#issuecomment-530192970
 
 
   OK. I am fine with this change. I will only call 'sdk_version', 
'python_sdk_version', since that is only applicable to python. And let's plan 
to clean it after 2.16. - LGTM.
   
   @tweise - What do you think?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310264)
Time Spent: 2.5h  (was: 2h 20m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310261&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310261
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 02:19
Start Date: 11/Sep/19 02:19
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #9506: [BEAM-8105] 
Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r323034999
 
 

 ##
 File path: release/src/main/scripts/set_version.sh
 ##
 @@ -78,6 +79,7 @@ else
   sed -i -e "s/version=.*/version=$TARGET_VERSION-SNAPSHOT/" gradle.properties
   sed -i -e "s/project.version = .*/project.version = '$TARGET_VERSION'/" 
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
   sed -i -e "s/^__version__ = .*/__version__ = '${TARGET_VERSION}.dev'/" 
sdks/python/apache_beam/version.py
+  sed -i -e "s/sdk_version=.*/sdk_version=$TARGET_VERSION.dev/" 
gradle.properties
 
 Review comment:
   Instead of sdk_version, you may want to call this python_sdk_version, this 
is python specific.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310261)
Time Spent: 3.5h  (was: 3h 20m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310262
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 02:19
Start Date: 11/Sep/19 02:19
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #9506: [BEAM-8105] 
Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r323034658
 
 

 ##
 File path: release/src/main/scripts/publish_docker_images.sh
 ##
 @@ -0,0 +1,77 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script will generate and publish docker images for each language 
version to Docker Hub:
+# 1. Generate images tagged with :{RELEASE}
+# 2. Publish images tagged with :{RELEASE}
+# 3. Tag images with :latest tag and publish.
+# 4. Clearn up images.
+
+set -e
+
+source release/src/main/scripts/build_release_candidate.sh
+
+echo "Publish SDK docker images to Docker Hub."
+echo "Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+  echo "Publishing SDK docker images on docker hub="
+  cd ~
+  if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
+rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+  fi
+  mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
+  cd ${LOCAL_PYTHON_STAGING_DIR}
+
+  echo '---Cloning Beam Release Branch-'
+  git clone ${GIT_REPO_URL}
+  cd ${BEAM_ROOT_DIR}
+  git checkout ${RELEASE_BRANCH}
+
+  echo '---Generating and Pushing Python 
images-'
+  ./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}
+  for ver in "${PYTHON_VER[@]}"; do
+ docker push apachebeam/${ver}_sdk:${RELEASE}
+ docker tag apachebeam/${ver}_sdk:${RELEASE} apachebeam/${ver}_sdk:latest
+ docker push apachebeam/${ver}_sdk:latest
+  done
+
+  echo '---Generating and Pushing Java images-'
+  ./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}
+  docker tag apachebeam/java_sdk:${RELEASE} apachebeam/java_sdk:latest
+  docker push apachebeam/java_sdk:latest
+
+  echo '---Generating and Pushing Go images-'
+  ./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}
+  docker tag apachebeam/go_sdk:${RELEASE} apachebeam/go_sdk:latest
+  docker push apachebeam/go_sdk:latest
 
 Review comment:
   should this be push go_sdk:${RELEASE} or does it not matter?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310262)
Time Spent: 3.5h  (was: 3h 20m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310256
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 01:52
Start Date: 11/Sep/19 01:52
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530185756
 
 
   I'm not really up to speed on what's going on here, but both Apex and Flink 
seem upset about coders of two different types, if that helps. From the [Flink 
run](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/testReport/):
   ```
   org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
org.apache.flink.api.common.InvalidProgramException: Cannot union inputs of 
different types. 
   ```
   and from the [Apex 
run](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/):
   ```
   org.apache.apex.api.Launcher$LauncherException: 
java.lang.UnsupportedOperationException: ParMultiDo(Anonymous): Multiple side 
inputs with different coders.
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310256)
Time Spent: 1h 10m  (was: 1h)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310250
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 01:45
Start Date: 11/Sep/19 01:45
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530184335
 
 
   How are they breaking those runners? This test should be working fine on
   those runners AFAIK.
   
   On Tue, Sep 10, 2019 at 5:53 PM Yueyang Qiu 
   wrote:
   
   > Run Apex ValidatesRunner
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310250)
Time Spent: 1h  (was: 50m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8156) Finish migration to standard Python typing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8156?focusedWorklogId=310234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310234
 ]

ASF GitHub Bot logged work on BEAM-8156:


Author: ASF GitHub Bot
Created on: 11/Sep/19 01:12
Start Date: 11/Sep/19 01:12
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9509: [BEAM-8156] Add 
convert_to_typing_type
URL: https://github.com/apache/beam/pull/9509#discussion_r323023802
 
 

 ##
 File path: sdks/python/apache_beam/typehints/native_type_compatibility.py
 ##
 @@ -198,6 +202,10 @@ def convert_to_beam_type(typ):
   arity=-1,
   beam_type=typehints.Tuple),
   _TypeMapEntry(match=_match_is_union, arity=-1, 
beam_type=typehints.Union),
+  _TypeMapEntry(
+  match=_match_issubclass(typing.Iterator),
 
 Review comment:
   Note that `GeneratorHint` has changed in my latest update. It now accepts 1 
or 3 arguments in `__getitem__`.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310234)
Remaining Estimate: 502.5h  (was: 502h 40m)
Time Spent: 1.5h  (was: 1h 20m)

> Finish migration to standard Python typing
> --
>
> Key: BEAM-8156
> URL: https://issues.apache.org/jira/browse/BEAM-8156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Major
>   Original Estimate: 504h
>  Time Spent: 1.5h
>  Remaining Estimate: 502.5h
>
> We should migrate all Python uses of types to the standard typing module, and 
> make the typehints.* ones aliases of the Python ones. 
>  
> There are three places where we use custom typehints behavior: 
> (1) is_compatible_with
> (2) bind_type_variables/match_type_variables
> (3) trivial type inference. 
>  
> I would propose that each of these be adapted to a (internal) public 
> interface that accepts and returns standard typing types, and internally 
> converts to our (nowhere else exposed) typehints types, performs the logic, 
> and converts back. Each of these in turn can then be updated, as needed and 
> orthogonally, to operate on the typing types natively (possibly via deference 
> to a third-party library). 
>  
> I think coder inference could be easily adopted to use typing types directly, 
> but it may be a fourth place where we do internal conversion first. Another 
> gotcha is special care may need to be taken if we ever need to pickle these 
> types (which IIRC may have issues). 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=310232&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310232
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 11/Sep/19 01:03
Start Date: 11/Sep/19 01:03
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9491: [BEAM-5539] Upgrade 
google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530176284
 
 
   There are no bigtable tests except the IT, and it's broken. I'll hold off on 
this PR for now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310232)
Time Spent: 3h 10m  (was: 3h)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  ---

[jira] [Commented] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[Yueyang Qiu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927162#comment-16927162
 ] 

Yueyang Qiu commented on BEAM-8204:
---

Sent a PR to exclude the tests on Apex and Flink runner to make the dashboard 
green. Could you please review? We can look into the cause in more detail later.

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running in any test suites

[Chamikara Jayalath (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927161#comment-16927161
 ] 

Chamikara Jayalath commented on BEAM-7611:
--

[~sduskis] can you please look into this or assign to the original author 
(couldn't find Jira account of original author Juan Rael) ? Seems like we are 
lacking test coverage for Python BigTable sink.

> Python BigTableIO IT is not running in any test suites
> --
>
> Key: BEAM-7611
> URL: https://issues.apache.org/jira/browse/BEAM-7611
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp, testing
>Reporter: Chamikara Jayalath
>Assignee: Solomon Duskis
>Priority: Major
>
> We added an integration test here: [https://github.com/apache/beam/pull/7367]
>  
> But this currently does not get picked up by any test suites (and get skipped 
> by some due to missing dependencies) hence BigTable sink is largely untested.
>  
> First attempt to enable it: [https://github.com/apache/beam/pull/8886]
>  
> Solomon assigning to you since I cannot find Juan's (PR author) Jira ID.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-7611) Python BigTableIO IT is not running in any test suites

[Chamikara Jayalath (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-7611:
-
Priority: Critical  (was: Major)

> Python BigTableIO IT is not running in any test suites
> --
>
> Key: BEAM-7611
> URL: https://issues.apache.org/jira/browse/BEAM-7611
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp, testing
>Reporter: Chamikara Jayalath
>Assignee: Solomon Duskis
>Priority: Critical
>
> We added an integration test here: [https://github.com/apache/beam/pull/7367]
>  
> But this currently does not get picked up by any test suites (and get skipped 
> by some due to missing dependencies) hence BigTable sink is largely untested.
>  
> First attempt to enable it: [https://github.com/apache/beam/pull/8886]
>  
> Solomon assigning to you since I cannot find Juan's (PR author) Jira ID.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310228
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:53
Start Date: 11/Sep/19 00:53
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530174383
 
 
   Run Apex ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310228)
Time Spent: 50m  (was: 40m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310227
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:52
Start Date: 11/Sep/19 00:52
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530174355
 
 
   Run Flink ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310227)
Time Spent: 40m  (was: 0.5h)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running in any test suites

[Udi Meiri (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927153#comment-16927153
 ] 

Udi Meiri commented on BEAM-7611:
-

I attempted to make the test run on DirectRunner, and ran into issues with 
pickling enums and self.client itself (client version 1.0.0).
The pipeline code needs to be refactored into its own module, so we can avoid 
pickling the client used for setting up the test.

Another question arises:
Why was this test placed in the examples directory? Was the intent to write one?


Pickling error:
{code}
==
ERROR: test_bigtable_write 
(apache_beam.examples.cookbook.bigtableio_it_test.BigtableIOWriteTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py",
 line 176, in test_bigtable_write
| 'Generate Direct Rows' >> GenerateTestRows(number, **config_data))
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py",
 line 905, in __ror__
return self.transform.__ror__(pvalueish, self.label)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py",
 line 514, in __ror__
result = p.apply(self, pvalueish, label)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py", 
line 481, in apply
return self.apply(transform, pvalueish)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py", 
line 517, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py",
 line 175, in apply
return m(transform, input, options)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py",
 line 181, in apply_PTransform
return transform.expand(input)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py",
 line 97, in expand
beam_options['table_id']))
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pvalue.py", line 
113, in __or__
return self.pipeline.apply(ptransform, self)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py", 
line 517, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py",
 line 175, in apply
return m(transform, input, options)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py",
 line 181, in apply_PTransform
return transform.expand(input)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/io/gcp/bigtableio.py",
 line 143, in expand
beam_options['table_id'])))
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/core.py",
 line 1107, in __init__
super(ParDo, self).__init__(fn, *args, **kwargs)
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py",
 line 702, in __init__
self.fn = pickler.loads(pickler.dumps(self.fn))
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/internal/pickler.py",
 line 237, in dumps
s = dill.dumps(o)
  File 
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py",
 line 294, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py",
 line 287, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/internal/pickler.py",
 line 198, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py",
 line 910, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310217
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:35
Start Date: 11/Sep/19 00:35
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530171315
 
 
   Run Flink ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310217)
Time Spent: 20m  (was: 10m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310218&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310218
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:35
Start Date: 11/Sep/19 00:35
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530171351
 
 
   Run Apex ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310218)
Time Spent: 0.5h  (was: 20m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310216&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310216
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:33
Start Date: 11/Sep/19 00:33
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541
 
 
   Exclude the test because they are currently breaking post-commit tests.
   
   @TheNeuralBit @reuvenlax 
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink

[jira] [Work logged] (BEAM-5820) Vendor Calcite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=310210&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310210
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:09
Start Date: 11/Sep/19 00:09
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on pull request #9189: [BEAM-5820] 
vendor calcite
URL: https://github.com/apache/beam/pull/9189#discussion_r323012683
 
 

 ##
 File path: sdks/java/extensions/sql/build.gradle
 ##
 @@ -22,48 +22,7 @@ plugins {
   id 'org.apache.beam.module'
   id 'ca.coglinc.javacc'
 }
-applyJavaNature(
-  // javacc generated code produces lint warnings
-  disableLintWarnings: ['dep-ann'],
-  testShadowJar: true,
-  enableStrictDependencies: true,
-  shadowClosure: {
-dependencies {
-  include(dependency(library.java.guava))
-  include(dependency(library.java.protobuf_java))
-  include(dependency(library.java.protobuf_java_util))
-  include(dependency("org.apache.calcite:.*"))
-  include(dependency("org.apache.calcite.avatica:.*"))
-  include(dependency("org.codehaus.janino:.*"))
-  include(dependency("com.google.zetasql:.*"))
-}
-// guava uses the com.google.common and com.google.thirdparty package 
namespaces
-relocate("com.google.common", 
project.getJavaRelocatedPath("com.google.common")) {
-  // com.google.common is too generic, need to exclude guava-testlib
-  exclude "com.google.common.collect.testing.**"
-  exclude "com.google.common.escape.testing.**"
-  exclude "com.google.common.testing.**"
-  exclude "com.google.common.util.concurrent.testing.**"
-}
-relocate "com.google.cloud", getJavaRelocatedPath("com.google.cloud")
-relocate "com.google.logging", getJavaRelocatedPath("com.google.logging")
-relocate "com.google.longrunning", 
getJavaRelocatedPath("com.google.longrunning")
-relocate "com.google.rpc", getJavaRelocatedPath("com.google.rpc")
-
-relocate "com.google.thirdparty", 
project.getJavaRelocatedPath("com.google.thirdparty")
-
-relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
-relocate "com.google.zetasql", getJavaRelocatedPath("com.google.zetasql")
-relocate "org.apache.calcite", getJavaRelocatedPath("org.apache.calcite")
-
-  // Looking up the compiler factory in Calcite depends on having a properties
-  // file in the right location. We package one that is shading compatible
-  // in src/main/resources. Note that if this shaded path changes, that
-  // files name and contents need to be updated as well. TODO, swap to use
-  // getJavaRelocatedPath once the Maven build is no longer also shading this
-  // module.
-  relocate "org.codehaus", 
"org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus"
-})
+applyJavaNature( disableLintWarnings: ['dep-ann'] )
 
 Review comment:
   disable javacc generated code produces lint warnings
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310210)
Time Spent: 7h 20m  (was: 7h 10m)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8205) Newly added Java ValidatesRunner tests failed on FlinkRunner

[Yueyang Qiu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927133#comment-16927133
 ] 

Yueyang Qiu commented on BEAM-8205:
---

cc: [~reuvenlax]

> Newly added Java ValidatesRunner tests failed on FlinkRunner
> 
>
> Key: BEAM-8205
> URL: https://issues.apache.org/jira/browse/BEAM-8205
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Flink runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[Yueyang Qiu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu updated BEAM-8204:
--
Component/s: sdk-java-core

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=310202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310202
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 11/Sep/19 00:06
Start Date: 11/Sep/19 00:06
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #9491: [BEAM-5539] 
Upgrade google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530165988
 
 
   LGTM.
   
   I'm not sure about current integration test coverage of Cloud BT sink. 
Please make sure this change is captured by some test suite.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310202)
Time Spent: 3h  (was: 2h 50m)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do

[jira] [Created] (BEAM-8205) Newly added Java ValidatesRunner tests failed on FlinkRunner

[Yueyang Qiu (Jira)]

Yueyang Qiu created BEAM-8205:
-

 Summary: Newly added Java ValidatesRunner tests failed on 
FlinkRunner
 Key: BEAM-8205
 URL: https://issues.apache.org/jira/browse/BEAM-8205
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core, test-failures
Reporter: Yueyang Qiu
Assignee: Brian Hulette


Jenkins link:

[https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/testReport/]

 

Initial investigation:

[https://github.com/apache/beam/pull/9454] added new ValidatesRunner tests. 
They have been tested on Dataflow runner, but are failing on Flink runner.

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[Yueyang Qiu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927127#comment-16927127
 ] 

Yueyang Qiu edited comment on BEAM-8204 at 9/11/19 12:00 AM:
-

cc: [~reuvenlax]


was (Author: robinyqiu):
cc: Reuven Lax

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310199&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310199
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:59
Start Date: 10/Sep/19 23:59
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9506: [BEAM-8105] 
Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#issuecomment-530164610
 
 
   kindly ping @markflyhigh 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310199)
Time Spent: 3h 20m  (was: 3h 10m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=310198&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310198
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:59
Start Date: 10/Sep/19 23:59
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9491: [BEAM-5539] Upgrade 
google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530164522
 
 
   > s/bigquery/bigtable in title.
   
   done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310198)
Time Spent: 2h 50m  (was: 2h 40m)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> --

[jira] [Commented] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[Yueyang Qiu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927127#comment-16927127
 ] 

Yueyang Qiu commented on BEAM-8204:
---

cc: Reuven Lax

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[Yueyang Qiu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu updated BEAM-8204:
--
Description: 
Jenkins link:

[https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]

 

Initial investigation:

[https://github.com/apache/beam/pull/9454] and 
[https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
They have been tested on Dataflow runner, but are failing on Apex runner.

 

  was:
Jenkins link:

[https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]

 

Initial investigation:

[https://github.com/apache/beam/pull/9454] and 
[https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
They as been tested on Dataflow runner, but they are failing on Apex Runner.

 


> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[Yueyang Qiu (Jira)]

Yueyang Qiu created BEAM-8204:
-

 Summary: Newly added Java ValidatesRunner tests failed on 
ApexRunner
 Key: BEAM-8204
 URL: https://issues.apache.org/jira/browse/BEAM-8204
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Yueyang Qiu
Assignee: Brian Hulette


Jenkins link:

[https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]

 

Initial investigation:

[https://github.com/apache/beam/pull/9454] and 
[https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
They as been tested on Dataflow runner, but they are failing on Apex Runner.

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-3713) Consider moving away from nose to nose2 or pytest.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3713?focusedWorklogId=310185&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310185
 ]

ASF GitHub Bot logged work on BEAM-3713:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:36
Start Date: 10/Sep/19 23:36
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9514: [BEAM-3713] 
Convert ITs to not use save_main_session
URL: https://github.com/apache/beam/pull/9514#discussion_r323006012
 
 

 ##
 File path: sdks/python/apache_beam/examples/complete/distribopt.py
 ##
 @@ -314,7 +314,7 @@ def format_output(element):
   return result
 
 
-def run(argv=None):
+def run(argv=None, save_main_session=True):
 
 Review comment:
   I think the best solution would be to remove save_main_session from examples 
altogether, even if that means splitting them into 2 modules. Since that is out 
of scope for the item I'm working on, I opted for the solution I proposed. I 
believe my solution is necessary and simpler than checking the internal 
variable `__name__`.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310185)
Time Spent: 6h 10m  (was: 6h)

> Consider moving away from nose to nose2 or pytest.
> --
>
> Key: BEAM-3713
> URL: https://issues.apache.org/jira/browse/BEAM-3713
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Per 
> [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,]
>  , nose is in maintenance mode.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-3713) Consider moving away from nose to nose2 or pytest.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3713?focusedWorklogId=310184&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310184
 ]

ASF GitHub Bot logged work on BEAM-3713:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:36
Start Date: 10/Sep/19 23:36
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9514: [BEAM-3713] 
Convert ITs to not use save_main_session
URL: https://github.com/apache/beam/pull/9514#discussion_r323006012
 
 

 ##
 File path: sdks/python/apache_beam/examples/complete/distribopt.py
 ##
 @@ -314,7 +314,7 @@ def format_output(element):
   return result
 
 
-def run(argv=None):
+def run(argv=None, save_main_session=True):
 
 Review comment:
   I think the best solution would be to remove save_main_session from examples 
altogether, even if that means splitting them into 2 modules. Since that is out 
of scope for the item I'm working on, I opted for the solution I proposed. I 
believe my solution is necessary and simpler than checking the magic variable 
`__name__`.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310184)
Time Spent: 6h  (was: 5h 50m)

> Consider moving away from nose to nose2 or pytest.
> --
>
> Key: BEAM-3713
> URL: https://issues.apache.org/jira/browse/BEAM-3713
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Per 
> [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,]
>  , nose is in maintenance mode.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8184) Allow asynchronous execution

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8184?focusedWorklogId=310182&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310182
 ]

ASF GitHub Bot logged work on BEAM-8184:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:31
Start Date: 10/Sep/19 23:31
Worklog Time Spent: 10m 
  Work Description: lostluck commented on pull request #9530: [BEAM-8184] 
Add --async flag to dataflow runner
URL: https://github.com/apache/beam/pull/9530
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310182)
Time Spent: 1.5h  (was: 1h 20m)

> Allow asynchronous execution
> 
>
> Key: BEAM-8184
> URL: https://issues.apache.org/jira/browse/BEAM-8184
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: 2.15.0
>Reporter: Jack Whelpton
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When developing streaming pipelines, it would be useful to have a means of 
> deploying a pipeline and exiting, without blocking on completion.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310180&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310180
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:28
Start Date: 10/Sep/19 23:28
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9510: [BEAM-8105] 
update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#issuecomment-530158509
 
 
   @markflyhigh , I addressed all your comments, PTAL.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310180)
Time Spent: 3h 10m  (was: 3h)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310179&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310179
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:28
Start Date: 10/Sep/19 23:28
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9510: [BEAM-8105] 
update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#issuecomment-530158509
 
 
   @markflyhigh , I fixed all your comments, PTAL.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310179)
Time Spent: 3h  (was: 2h 50m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8184) Allow asynchronous execution

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8184?focusedWorklogId=310178&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310178
 ]

ASF GitHub Bot logged work on BEAM-8184:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:25
Start Date: 10/Sep/19 23:25
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #9530: [BEAM-8184] Add 
--async flag to dataflow runner
URL: https://github.com/apache/beam/pull/9530#issuecomment-530158032
 
 
   Will merge after the precommit passes...
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310178)
Time Spent: 1h 20m  (was: 1h 10m)

> Allow asynchronous execution
> 
>
> Key: BEAM-8184
> URL: https://issues.apache.org/jira/browse/BEAM-8184
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: 2.15.0
>Reporter: Jack Whelpton
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> When developing streaming pipelines, it would be useful to have a means of 
> deploying a pipeline and exiting, without blocking on completion.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8195) Quota exceeded for create requests

[yifan zou (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927112#comment-16927112
 ] 

yifan zou commented on BEAM-8195:
-

Technically it's doable. [~alanmyrvold] Any thoughts?

> Quota exceeded for create requests
> --
>
> Key: BEAM-8195
> URL: https://issues.apache.org/jira/browse/BEAM-8195
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, testing
>Reporter: Ahmet Altay
>Assignee: yifan zou
>Priority: Critical
>
> Post commits failied with the following error:
> HttpError accessing 
> :
>  response: <{'server': 'ESF', '-content-encoding': 'gzip', 'content-type': 
> 'application/json; charset=UTF-8', 'content-length': '598', 
> 'transfer-encoding': 'chunked', 'cache-control': 'private', 
> 'x-xss-protection': '0', 'date': 'Tue, 10 Sep 2019 12:02:24 GMT', 'vary': 
> 'Origin, X-Origin, Referer', 'x-frame-options': 'SAMEORIGIN', 'status': 
> '429', 'x-content-type-options': 'nosniff'}>, content <{
>   "error": {
> "code": 429,
> "message": "Quota exceeded for quota metric 
> 'dataflow.googleapis.com/create_requests' and limit 
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for 
> consumer 'project_number:844138762903'.",
> "status": "RESOURCE_EXHAUSTED",
> "details": [
>   {
> Could we increase the quota?
> /cc [~alanmyrvold] [~kenn]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8184) Allow asynchronous execution

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8184?focusedWorklogId=310175&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310175
 ]

ASF GitHub Bot logged work on BEAM-8184:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:21
Start Date: 10/Sep/19 23:21
Worklog Time Spent: 10m 
  Work Description: lostluck commented on pull request #9530: [BEAM-8184] 
Add --async flag to dataflow runner
URL: https://github.com/apache/beam/pull/9530#discussion_r323002606
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflow.go
 ##
 @@ -62,6 +62,7 @@ var (
minCPUPlatform   = flag.String("min_cpu_platform", "", "GCE minimum 
cpu platform (optional)")
workerJar= flag.String("dataflow_worker_jar", "", "Dataflow 
worker jar (optional)")
 
+   async  = flag.Bool("async", false, "Asynchronous execution. 
Submit the job and return immediately.")
 
 Review comment:
   execute_async works for me!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310175)
Time Spent: 1h 10m  (was: 1h)

> Allow asynchronous execution
> 
>
> Key: BEAM-8184
> URL: https://issues.apache.org/jira/browse/BEAM-8184
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: 2.15.0
>Reporter: Jack Whelpton
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> When developing streaming pipelines, it would be useful to have a means of 
> deploying a pipeline and exiting, without blocking on completion.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8156) Finish migration to standard Python typing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8156?focusedWorklogId=310174&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310174
 ]

ASF GitHub Bot logged work on BEAM-8156:


Author: ASF GitHub Bot
Created on: 10/Sep/19 23:20
Start Date: 10/Sep/19 23:20
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9509: [BEAM-8156] Add 
convert_to_typing_type
URL: https://github.com/apache/beam/pull/9509#discussion_r323002366
 
 

 ##
 File path: sdks/python/apache_beam/typehints/native_type_compatibility.py
 ##
 @@ -198,6 +202,10 @@ def convert_to_beam_type(typ):
   arity=-1,
   beam_type=typehints.Tuple),
   _TypeMapEntry(match=_match_is_union, arity=-1, 
beam_type=typehints.Union),
+  _TypeMapEntry(
+  match=_match_issubclass(typing.Iterator),
 
 Review comment:
   PTAL. I don't see how I can implement bidirectional conversion of internal 
and external types without at least supporting all internal types. Otherwise 
we'll break existing users.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310174)
Remaining Estimate: 502h 40m  (was: 502h 50m)
Time Spent: 1h 20m  (was: 1h 10m)

> Finish migration to standard Python typing
> --
>
> Key: BEAM-8156
> URL: https://issues.apache.org/jira/browse/BEAM-8156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Major
>   Original Estimate: 504h
>  Time Spent: 1h 20m
>  Remaining Estimate: 502h 40m
>
> We should migrate all Python uses of types to the standard typing module, and 
> make the typehints.* ones aliases of the Python ones. 
>  
> There are three places where we use custom typehints behavior: 
> (1) is_compatible_with
> (2) bind_type_variables/match_type_variables
> (3) trivial type inference. 
>  
> I would propose that each of these be adapted to a (internal) public 
> interface that accepts and returns standard typing types, and internally 
> converts to our (nowhere else exposed) typehints types, performs the logic, 
> and converts back. Each of these in turn can then be updated, as needed and 
> orthogonally, to operate on the typing types natively (possibly via deference 
> to a third-party library). 
>  
> I think coder inference could be easily adopted to use typing types directly, 
> but it may be a fourth place where we do internal conversion first. Another 
> gotcha is special care may need to be taken if we ever need to pickle these 
> types (which IIRC may have issues). 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=310164&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310164
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 10/Sep/19 22:49
Start Date: 10/Sep/19 22:49
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #9491: [BEAM-5539] 
Upgrade google-cloud-pubsub and bigquery packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530150108
 
 
   s/bigquery/bigtable in title.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310164)
Time Spent: 2h 40m  (was: 2.5h)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> ---

[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=310163&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310163
 ]

ASF GitHub Bot logged work on BEAM-5878:


Author: ASF GitHub Bot
Created on: 10/Sep/19 22:47
Start Date: 10/Sep/19 22:47
Worklog Time Spent: 10m 
  Work Description: mmckerns commented on issue #9237: [BEAM-5878] support 
DoFns with Keyword-only arguments
URL: https://github.com/apache/beam/pull/9237#issuecomment-530149542
 
 
   I know (https://github.com/uqfoundation/dill/issues/223)  I attach a 
milestone to each issue, but that doesn't give a full summary as some features 
are not reflected in the issues.  It's on the todo.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310163)
Time Spent: 13h 40m  (was: 13.5h)

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
> Fix For: 2.16.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=310159&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310159
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 10/Sep/19 22:40
Start Date: 10/Sep/19 22:40
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9491: [BEAM-5539] Upgrade 
google-cloud-pubsub and bigquery packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530148053
 
 
   R: @chamikaramj 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310159)
Time Spent: 2.5h  (was: 2h 20m)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> -
> P

[jira] [Updated] (BEAM-5869) Need a more deeper data structure quality check for row.

[Rui Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5869:
---
Issue Type: Improvement  (was: Task)

> Need a more deeper data structure quality check for row.
> 
>
> Key: BEAM-5869
> URL: https://issues.apache.org/jira/browse/BEAM-5869
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> Have tests to test quality when Row contains list, map, row and their nested 
> combination.  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8202) Support ParquetTable Writer

[Rui Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-8202:
---
Status: Open  (was: Triage Needed)

> Support ParquetTable Writer
> ---
>
> Key: BEAM-8202
> URL: https://issues.apache.org/jira/browse/BEAM-8202
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> https://github.com/apache/beam/pull/9054 supported reader for Parquet Table 
> in BeamSQL. We can support writer as well.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8203) Support AvroTable

[Rui Wang (Jira)]

Rui Wang created BEAM-8203:
--

 Summary: Support AvroTable
 Key: BEAM-8203
 URL: https://issues.apache.org/jira/browse/BEAM-8203
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Rui Wang


Similar to https://github.com/apache/beam/pull/9054, we can support Avro format 
of table.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8203) Support AvroTable

[Rui Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-8203:
---
Status: Open  (was: Triage Needed)

> Support AvroTable
> -
>
> Key: BEAM-8203
> URL: https://issues.apache.org/jira/browse/BEAM-8203
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> Similar to https://github.com/apache/beam/pull/9054, we can support table 
> with Avro format.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8203) Support AvroTable

[Rui Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-8203:
---
Description: Similar to https://github.com/apache/beam/pull/9054, we can 
support table with Avro format.  (was: Similar to 
https://github.com/apache/beam/pull/9054, we can support Avro format of table.)

> Support AvroTable
> -
>
> Key: BEAM-8203
> URL: https://issues.apache.org/jira/browse/BEAM-8203
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> Similar to https://github.com/apache/beam/pull/9054, we can support table 
> with Avro format.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8202) Support ParquetTable Writer

[Rui Wang (Jira)]

Rui Wang created BEAM-8202:
--

 Summary: Support ParquetTable Writer
 Key: BEAM-8202
 URL: https://issues.apache.org/jira/browse/BEAM-8202
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Rui Wang


https://github.com/apache/beam/pull/9054 supported reader for Parquet Table in 
BeamSQL. We can support writer as well.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8119) The python RC validation script need cleanup the pubsub resources after finish.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8119?focusedWorklogId=310151&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310151
 ]

ASF GitHub Bot logged work on BEAM-8119:


Author: ASF GitHub Bot
Created on: 10/Sep/19 22:05
Start Date: 10/Sep/19 22:05
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9534: 
[BEAM-8119] Properly cleanup Pubsub in release script
URL: https://github.com/apache/beam/pull/9534
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310151)
Time Spent: 2h  (was: 1h 50m)

> The python RC validation script need cleanup the pubsub resources after 
> finish.
> ---
>
> Key: BEAM-8119
> URL: https://issues.apache.org/jira/browse/BEAM-8119
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: yifan zou
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> * Python staging artifacts were moved to ‘python/’ directory. This needs to 
> be fixed in the script, otherwise the script won’t download the sdks and hash 
> files correctly.
>  * If the test fails, it sometimes left some died pubsub topics. This might 
> cause failures in the reruns since the resources already exist: 
>  * ERROR: Failed to create topic 
> [projects/apache-beam-testing/topics/wordstream-python-topic-1]: Resource 
> already exists in the project (resource=wordstream-python-topic-1). 
>  * *Workaround*: I manually deleted the topics using cmd: gcloud alpha pubsub 
> topics delete wordstream-python-topic-1
>  * TODO: 
>  ** Change the download directory in the script.
>  ** Add cleanup if the job was interrupted.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8201) clean up the current container API

[Luke Cwik (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927066#comment-16927066
 ] 

Luke Cwik commented on BEAM-8201:
-

The container contract is defined in 
[_https://s.apache.org/beam-fn-api-container-contract_]

 

Updating that document is a requirement of this task as well.

> clean up the current container API
> --
>
> Key: BEAM-8201
> URL: https://issues.apache.org/jira/browse/BEAM-8201
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
>
> From [~robertwb]
> As part of this project, I propose we look at and clean up the current 
> container API before we "release" it as public and stable. IIRC, we currently 
> provide the worker arguments through a combination of (1) environment 
> variables (2) command line parameters to docker and (3) via the provisioning 
> API. It would be good to have a more principled approach to specifying 
> arguments (either all the same way, or if they vary, good reason for doing so 
> rather than by historical accident).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310149
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 22:03
Start Date: 10/Sep/19 22:03
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322982688
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -1135,6 +1183,20 @@ __NOTE__: Only PMC members have permissions to do it, 
ping [dev@](mailto:dev@bea
 
 Make sure the download address for last release version is upldaed, [example 
PR](https://github.com/apache/beam-site/pull/478).
 
+ Deploy SDK docker images to DockerHub
+TODO(hannahjiang): change link to master branch after #9560 is merged.
+
+* Script: 
[publish_docker_images.sh](https://github.com/Hannah-Jiang/beam/blob/release_script_for_containers/release/src/main/scripts/publish_docker_images.sh)
 
 Review comment:
   doesn't L1187 work?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310149)
Time Spent: 2h 50m  (was: 2h 40m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (BEAM-4193) [SQL] Support trigger definition in CLI

[Rui Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-4193.
--
Fix Version/s: Not applicable
   Resolution: Won't Fix

Current plan is EMIT syntax to control output of SQL. So close this JIRA.

> [SQL] Support trigger definition in CLI
> ---
>
> Key: BEAM-4193
> URL: https://issues.apache.org/jira/browse/BEAM-4193
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> We need to be able to define triggers for the sources in SQL CLI. Otherwise 
> it is impossible to reason about outputs.
> One approach is to come up with a simple JSON trigger definition syntax, 
> along the lines of
>  
> {code:java}
> SET OPT( '{ "trgger" : { "DefaultTrigger" }, "allowedLateness" : 0 }' )
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310129
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 21:23
Start Date: 10/Sep/19 21:23
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #9506: 
[BEAM-8105] Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r322959511
 
 

 ##
 File path: release/src/main/scripts/build_release_candidate.sh
 ##
 @@ -20,7 +20,8 @@
 # 1. Build and stage java artifacts
 # 2. Stage source release on dist.apache.org
 # 3. Stage python binaries
-# 4. Create a PR to update beam-site
+# 4. Stage SDK docker images
 
 Review comment:
   The images will be verified in two steps.
   1. Tests running by `verify_release_build.sh` will test images. Jenkins 
build images and use them for testings. Though the tests don't directly use 
staging images, the images are built from the exact same codebase by calling 
the same gradle command. 
   2. For staging images, we should validate them in the same way we verify 
other artifacts, when we ask for votes from community.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310129)
Time Spent: 2h 40m  (was: 2.5h)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8201) clean up the current container API

[Hannah Jiang (Jira)]

Hannah Jiang created BEAM-8201:
--

 Summary: clean up the current container API
 Key: BEAM-8201
 URL: https://issues.apache.org/jira/browse/BEAM-8201
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Hannah Jiang
Assignee: Hannah Jiang


>From [~robertwb]
As part of this project, I propose we look at and clean up the current 
container API before we "release" it as public and stable. IIRC, we currently 
provide the worker arguments through a combination of (1) environment variables 
(2) command line parameters to docker and (3) via the provisioning API. It 
would be good to have a more principled approach to specifying arguments 
(either all the same way, or if they vary, good reason for doing so rather than 
by historical accident).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310122&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310122
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 21:14
Start Date: 10/Sep/19 21:14
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on pull request #9506: [BEAM-8105] 
Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r322965091
 
 

 ##
 File path: release/src/main/scripts/build_release_candidate.sh
 ##
 @@ -20,7 +20,8 @@
 # 1. Build and stage java artifacts
 # 2. Stage source release on dist.apache.org
 # 3. Stage python binaries
-# 4. Create a PR to update beam-site
+# 4. Stage SDK docker images
 
 Review comment:
   SG. Please update the template in the release sign-off sheet if staging 
image validation is required. 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310122)
Time Spent: 2.5h  (was: 2h 20m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310121
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 21:13
Start Date: 10/Sep/19 21:13
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on pull request #9506: [BEAM-8105] 
Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r322965091
 
 

 ##
 File path: release/src/main/scripts/build_release_candidate.sh
 ##
 @@ -20,7 +20,8 @@
 # 1. Build and stage java artifacts
 # 2. Stage source release on dist.apache.org
 # 3. Stage python binaries
-# 4. Create a PR to update beam-site
+# 4. Stage SDK docker images
 
 Review comment:
   SG. Remember to update the template in the release sign-off sheet if staging 
image validation is required. 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310121)
Time Spent: 2h 20m  (was: 2h 10m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8119) The python RC validation script need cleanup the pubsub resources after finish.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8119?focusedWorklogId=310118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310118
 ]

ASF GitHub Bot logged work on BEAM-8119:


Author: ASF GitHub Bot
Created on: 10/Sep/19 21:06
Start Date: 10/Sep/19 21:06
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #9534: [BEAM-8119] 
Properly cleanup Pubsub in release script
URL: https://github.com/apache/beam/pull/9534#issuecomment-530119629
 
 
   Got an LGTM. I'll go ahead to merge it before 2.16 branch cut.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310118)
Time Spent: 1h 50m  (was: 1h 40m)

> The python RC validation script need cleanup the pubsub resources after 
> finish.
> ---
>
> Key: BEAM-8119
> URL: https://issues.apache.org/jira/browse/BEAM-8119
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: yifan zou
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> * Python staging artifacts were moved to ‘python/’ directory. This needs to 
> be fixed in the script, otherwise the script won’t download the sdks and hash 
> files correctly.
>  * If the test fails, it sometimes left some died pubsub topics. This might 
> cause failures in the reruns since the resources already exist: 
>  * ERROR: Failed to create topic 
> [projects/apache-beam-testing/topics/wordstream-python-topic-1]: Resource 
> already exists in the project (resource=wordstream-python-topic-1). 
>  * *Workaround*: I manually deleted the topics using cmd: gcloud alpha pubsub 
> topics delete wordstream-python-topic-1
>  * TODO: 
>  ** Change the download directory in the script.
>  ** Add cleanup if the job was interrupted.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8119) The python RC validation script need cleanup the pubsub resources after finish.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8119?focusedWorklogId=310117&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310117
 ]

ASF GitHub Bot logged work on BEAM-8119:


Author: ASF GitHub Bot
Created on: 10/Sep/19 21:05
Start Date: 10/Sep/19 21:05
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9534: 
[BEAM-8119] Properly cleanup Pubsub in release script
URL: https://github.com/apache/beam/pull/9534#discussion_r322961933
 
 

 ##
 File path: release/src/main/python-release/python_release_automation_utils.sh
 ##
 @@ -218,9 +218,10 @@ function create_pubsub() {
 #   None
 ###
 function cleanup_pubsub() {
-  gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1
-  gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2
-  gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION
+  # Suppress error since topic/subscription may not exist
+  gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1 2> /dev/null
 
 Review comment:
   Also the main purpose to suppress stderr here is not to break the script. 
Messages in stdout can help people know the status of current step.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310117)
Time Spent: 1h 40m  (was: 1.5h)

> The python RC validation script need cleanup the pubsub resources after 
> finish.
> ---
>
> Key: BEAM-8119
> URL: https://issues.apache.org/jira/browse/BEAM-8119
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: yifan zou
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> * Python staging artifacts were moved to ‘python/’ directory. This needs to 
> be fixed in the script, otherwise the script won’t download the sdks and hash 
> files correctly.
>  * If the test fails, it sometimes left some died pubsub topics. This might 
> cause failures in the reruns since the resources already exist: 
>  * ERROR: Failed to create topic 
> [projects/apache-beam-testing/topics/wordstream-python-topic-1]: Resource 
> already exists in the project (resource=wordstream-python-topic-1). 
>  * *Workaround*: I manually deleted the topics using cmd: gcloud alpha pubsub 
> topics delete wordstream-python-topic-1
>  * TODO: 
>  ** Change the download directory in the script.
>  ** Add cleanup if the job was interrupted.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310116
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:59
Start Date: 10/Sep/19 20:59
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #9506: 
[BEAM-8105] Docker images release scripts
URL: https://github.com/apache/beam/pull/9506#discussion_r322959511
 
 

 ##
 File path: release/src/main/scripts/build_release_candidate.sh
 ##
 @@ -20,7 +20,8 @@
 # 1. Build and stage java artifacts
 # 2. Stage source release on dist.apache.org
 # 3. Stage python binaries
-# 4. Create a PR to update beam-site
+# 4. Stage SDK docker images
 
 Review comment:
   The images will be verified in two steps.
   1. Tests running by `verify_release_build.sh` will test images. Jenkins 
build images and use them for testings. Though the tests don't directly use 
staging images, the images are build from the exact same codebase by calling 
the same gradle command. 
   2. For staging images, we should validate them in the same way we verify 
other artifacts, when we ask for votes from community.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310116)
Time Spent: 2h 10m  (was: 2h)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926887#comment-16926887
 ] 

Valentyn Tymofieiev commented on BEAM-8198:
---

[~thw] reported a bug in streaming stack, I think we should track it in a 
separate issue.

> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markflyhigh]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-8198:
--
Description: 
context: 
https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E

Setting fix version to 2.16.0 to understand the cause, hopefully before the 
vote.

cc: [~altay] [~thw] [~markflyhigh]

  was:
context: 
https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E

Setting fix version to 2.16.0 to understand the cause, hopefully before the 
vote.

cc: [~altay] [~thw] [~markflyhigh]]


> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markflyhigh]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=310111&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310111
 ]

ASF GitHub Bot logged work on BEAM-7013:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:53
Start Date: 10/Sep/19 20:53
Worklog Time Spent: 10m 
  Work Description: zfraa commented on pull request #9519: [BEAM-7013] Use 
a 0-length byte array to represent empty sketch in HllCount
URL: https://github.com/apache/beam/pull/9519#discussion_r322957148
 
 

 ##
 File path: 
sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
 ##
 @@ -302,12 +304,53 @@ public void testMergePartialGlobally_SingletonInput() {
 
   @Test
   @Category(NeedsRunner.class)
-  public void testMergePartialGlobally_EmptyInput() {
+  public void testMergePartialGlobally_SingletonInputEmptySketch() {
 PCollection result =
-p.apply(Create.empty(TypeDescriptor.of(byte[].class)))
+
p.apply(Create.of(EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeWithEmptySketch() {
+PCollection result =
+p.apply(Create.of(LONGS_SKETCH, 
EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeMultipleEmptySketches() {
+PCollection result =
+p.apply(Create.of(EMPTY_SKETCH, 
EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeWithSketchOfEmptySet() {
+PCollection result =
+p.apply(Create.of(LONGS_SKETCH, LONGS_SKETCH_OF_EMPTY_SET))
+.apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeEmptySketchWithSketchOfEmptySet() {
+PCollection result =
+p.apply(Create.of(EMPTY_SKETCH, LONGS_SKETCH_OF_EMPTY_SET))
 .apply(HllCount.MergePartial.globally());
 
-PAssert.that(result).empty();
+PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH_OF_EMPTY_SET);
 
 Review comment:
   Good point, SG. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310111)
Time Spent: 34h  (was: 33h 50m)

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 34h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310109&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310109
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322945525
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -769,13 +815,15 @@ Template:
 1. Maven artifacts deployed to the staging repository of 
[repository.apache.org](https://repository.apache.org/content/repositories/)
 1. Source distribution deployed to the dev repository of 
[dist.apache.org](https://dist.apache.org/repos/dist/dev/beam/)
 1. Website pull request proposed to list the [release]({{ site.baseurl 
}}/get-started/downloads/), publish the [Java API reference 
manual](https://beam.apache.org/releases/javadoc/), and publish the [Python API 
reference manual](https://beam.apache.org/releases/pydoc/).
+1. Docker images are published to 
[DockerHub](https://hub.docker.com/search?q=apachebeam&type=image) with tags 
:{RELEASE}_rc.
 
 You can (optionally) also do additional verification by:
 1. Check that Python zip file contains the `README.md`, `NOTICE`, and 
`LICENSE` files.
 1. Check hashes (e.g. `md5sum -c *.md5` and `sha1sum -c *.sha1`)
 1. Check signatures (e.g. `gpg --verify apache-beam-1.2.3-python.zip.asc 
apache-beam-1.2.3-python.zip`)
 1. `grep` for legal headers in each file.
 1. Run all jenkins suites and include links to passing tests in the voting 
email. (Select "Run with parameters")
+1. Pull docker images to make sure they are pullable.
 
 Review comment:
   add a command example?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310109)
Time Spent: 1h 50m  (was: 1h 40m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8200) Add performance benchmarks for Python streaming

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-8200:
-

 Summary: Add performance benchmarks for Python streaming 
 Key: BEAM-8200
 URL: https://issues.apache.org/jira/browse/BEAM-8200
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


Per discussion [1], we have a gap in python streaming benchmark coverage. We 
should look into adding benchmarks and alerting (see: BEAM-8199). 

cc: [~altay] [~angoenka] [~kamilwu] [~kasiak] [~thw]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310110
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322934426
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -208,6 +209,18 @@ Automation 
Reliability](https://s.apache.org/beam-site-automation).
 
 Release manager needs to have an account with PyPI. If you need one, [register 
at PyPI](https://pypi.python.org/account/register/). You also need to be a 
maintainer (or an owner) of the 
[apache-beam](https://pypi.python.org/pypi/apache-beam) package in order to 
push a new release. Ask on the mailing list for assistance.
 
+ Login to DockerHub
+Run following command manually, it will ask you to input your DockerHub ID and 
password if 
+authorization info cannot be found from ~/.docker/config.json file.
+```
+docker login docker.io
+```
+After successful login, authorization info will be stored at 
~/.docker/config.json file. For example,
+```aidl
 
 Review comment:
   Not sure if this annotation will work for Beam website. This block didn't 
show up in [preview 
page](http://apache-beam-website-pull-requests.storage.googleapis.com/9510/contribute/release-guide/index.html)
 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310110)
Time Spent: 2h  (was: 1h 50m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310103
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322939853
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -618,6 +632,38 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+ Stage SDK images on hub.docker.com
+Build Python images and push to DockerHub
+```aidl
+./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc
+
+PYTHON_VER=("python2.7" "python3.5" "python3.6" "python3.7")
+for ver in "${PYTHON_VER[@]}"; do
+   docker push apachebeam/${ver}_sdk:${RELEASE}_rc &
+done
+``` 
+
+Build Java images and push to DockerHub
+```aidl
+./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}_rc
+```
+
+Build Go images and push to DockerHub
+```aidl
+./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}_rc
+```
+
+Clean up images from local
+```aidl
+for ver in "${PYTHON_VER[@]}"; do
+   docker rmi -f apachebeam/${ver}_sdk:${RELEASE}_rc
+done
+docker rmi -f apachebeam/java_sdk:${RELEASE}_rc
+docker rmi -f apachebeam/go_sdk:${RELEASE}_rc
+```
+
+Verify all images are publish to DockerHub with tags :${RELEASE}_rc
 
 Review comment:
   Verify all images are publish to DockerHub with tag: `${RELEASE}_rc`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310103)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310101&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310101
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322943519
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -618,6 +632,38 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+ Stage SDK images on hub.docker.com
+Build Python images and push to DockerHub
+```aidl
+./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc
+
+PYTHON_VER=("python2.7" "python3.5" "python3.6" "python3.7")
+for ver in "${PYTHON_VER[@]}"; do
+   docker push apachebeam/${ver}_sdk:${RELEASE}_rc &
 
 Review comment:
   should we include rc number in tag to distinguish different build for rc?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310101)
Time Spent: 1h 20m  (was: 1h 10m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310102
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322956551
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -1135,6 +1183,20 @@ __NOTE__: Only PMC members have permissions to do it, 
ping [dev@](mailto:dev@bea
 
 Make sure the download address for last release version is upldaed, [example 
PR](https://github.com/apache/beam-site/pull/478).
 
+ Deploy SDK docker images to DockerHub
+TODO(hannahjiang): change link to master branch after #9560 is merged.
+
+* Script: 
[publish_docker_images.sh](https://github.com/Hannah-Jiang/beam/blob/release_script_for_containers/release/src/main/scripts/publish_docker_images.sh)
+* Usage
+```aidl
+./beam/release/src/main/scripts/publish_docker_images.sh
+```
+Verify following points:
+* Images are published at 
[DockerHub](https://hub.docker.com/search?q=apachebeam&type=image) with tags 
:{RELEASE} and :latest.
+* Images with :latest tag are pointing to current release by confirming 
+  1. Images with :latest tags were updated a moment ago.
+  1. Size of Images with :latest tags should be same as the size of images 
with tag :{RELEASE}.
 
 Review comment:
   Should `latest` and `${RELEASE}` actually are same image? If so, how about 
also check DIGEST.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310102)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310106
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322946137
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -1135,6 +1183,20 @@ __NOTE__: Only PMC members have permissions to do it, 
ping [dev@](mailto:dev@bea
 
 Make sure the download address for last release version is upldaed, [example 
PR](https://github.com/apache/beam-site/pull/478).
 
+ Deploy SDK docker images to DockerHub
 
 Review comment:
   should this title use `###` instead?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310106)
Time Spent: 1.5h  (was: 1h 20m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310100
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322942524
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -769,13 +815,15 @@ Template:
 1. Maven artifacts deployed to the staging repository of 
[repository.apache.org](https://repository.apache.org/content/repositories/)
 1. Source distribution deployed to the dev repository of 
[dist.apache.org](https://dist.apache.org/repos/dist/dev/beam/)
 1. Website pull request proposed to list the [release]({{ site.baseurl 
}}/get-started/downloads/), publish the [Java API reference 
manual](https://beam.apache.org/releases/javadoc/), and publish the [Python API 
reference manual](https://beam.apache.org/releases/pydoc/).
+1. Docker images are published to 
[DockerHub](https://hub.docker.com/search?q=apachebeam&type=image) with tags 
:{RELEASE}_rc.
 
 Review comment:
   with tags: `{RELEASE}_rc`.
   Just realized there is a `:` in the tag. If that's the standard way to 
express a tag, please add it back.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310100)
Time Spent: 1h 20m  (was: 1h 10m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310104
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322948245
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -1135,6 +1183,20 @@ __NOTE__: Only PMC members have permissions to do it, 
ping [dev@](mailto:dev@bea
 
 Make sure the download address for last release version is upldaed, [example 
PR](https://github.com/apache/beam-site/pull/478).
 
+ Deploy SDK docker images to DockerHub
+TODO(hannahjiang): change link to master branch after #9560 is merged.
+
+* Script: 
[publish_docker_images.sh](https://github.com/Hannah-Jiang/beam/blob/release_script_for_containers/release/src/main/scripts/publish_docker_images.sh)
+* Usage
+```aidl
+./beam/release/src/main/scripts/publish_docker_images.sh
+```
+Verify following points:
+* Images are published at 
[DockerHub](https://hub.docker.com/search?q=apachebeam&type=image) with tags 
:{RELEASE} and :latest.
 
 Review comment:
   with tags: `${RELEASE}` and `lastest`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310104)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310108&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310108
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322947548
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -1135,6 +1183,20 @@ __NOTE__: Only PMC members have permissions to do it, 
ping [dev@](mailto:dev@bea
 
 Make sure the download address for last release version is upldaed, [example 
PR](https://github.com/apache/beam-site/pull/478).
 
+ Deploy SDK docker images to DockerHub
+TODO(hannahjiang): change link to master branch after #9560 is merged.
+
+* Script: 
[publish_docker_images.sh](https://github.com/Hannah-Jiang/beam/blob/release_script_for_containers/release/src/main/scripts/publish_docker_images.sh)
 
 Review comment:
   Can you add todo here pointing to master once your changes merged?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310108)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310107&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310107
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322931816
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -81,6 +81,7 @@ Please have these credentials ready at hand, you will likely 
need to enter them
 * GPG pass phrase (see the next section);
 * Apache ID and Password;
 * GitHub ID and Password.
+* DockerHub ID and Password. (You should be a member of maintainer team, 
please ask for help at dev@ if you are not.)
 
 Review comment:
   Can you provide a detail guide instead of pointing to dev@?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310107)
Time Spent: 1h 40m  (was: 1.5h)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310105
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:52
Start Date: 10/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r322941692
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -618,6 +632,38 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+ Stage SDK images on hub.docker.com
+Build Python images and push to DockerHub
+```aidl
+./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc
+
+PYTHON_VER=("python2.7" "python3.5" "python3.6" "python3.7")
+for ver in "${PYTHON_VER[@]}"; do
+   docker push apachebeam/${ver}_sdk:${RELEASE}_rc &
+done
+``` 
+
+Build Java images and push to DockerHub
+```aidl
+./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}_rc
+```
+
+Build Go images and push to DockerHub
+```aidl
+./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}_rc
+```
+
+Clean up images from local
+```aidl
+for ver in "${PYTHON_VER[@]}"; do
+   docker rmi -f apachebeam/${ver}_sdk:${RELEASE}_rc
+done
+docker rmi -f apachebeam/java_sdk:${RELEASE}_rc
+docker rmi -f apachebeam/go_sdk:${RELEASE}_rc
+```
+
+Verify all images are publish to DockerHub with tags :${RELEASE}_rc
 
 Review comment:
   Can you also provide a link/simple instruction to find those images on 
hub.docker.com?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310105)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Ahmet Altay (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926883#comment-16926883
 ] 

Ahmet Altay commented on BEAM-8198:
---

[~thw] could you provide any additional context if possible?

> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markflyhigh]]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310095&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310095
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:45
Start Date: 10/Sep/19 20:45
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #9487: 
[BEAM-8165]Change default docker images name
URL: https://github.com/apache/beam/pull/9487#discussion_r322953450
 
 

 ##
 File path: sdks/python/container/py2/build.gradle
 ##
 @@ -51,8 +52,13 @@ task copyLauncherDependencies(type: Copy) {
   }
 }
 
+def sdk_version = '2.16.0'
 
 Review comment:
   Thanks for reviewing and comments.
   I wanted to make `sdk_version` consistent with the one from `version.py`, 
forgot to change it back after testing. Thanks for catching it.
   
   After evaluating many options, I think introducing a new variable 
`sdk_version` to `gradle.properties` is the easiest way for now. This version 
will be consistent to the one from `version.py`, so we need to update it when 
we release it. It would be better to read it from `version.py` and assign to 
`sdk_version`, however, as far as I know, `gradle.properties` is not supposed 
to read variables from another file. If it's possible, I am happy to try.
   
   I made a commit with this change, can you please review and make comments?
   
   Thanks,
   Hannah
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310095)
Time Spent: 2h 20m  (was: 2h 10m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7760) Interactive Beam Caching PCollections bound to user defined vars in notebook

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7760?focusedWorklogId=310094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310094
 ]

ASF GitHub Bot logged work on BEAM-7760:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:44
Start Date: 10/Sep/19 20:44
Worklog Time Spent: 10m 
  Work Description: KevinGG commented on pull request #9278: [BEAM-7760] 
Added Interactive Beam module
URL: https://github.com/apache/beam/pull/9278#discussion_r322953360
 
 

 ##
 File path: sdks/python/apache_beam/runners/interactive/interactive_beam.py
 ##
 @@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module of Interactive Beam features that can be used in notebook.
+
+The purpose of the module is to reduce the learning curve of Interactive Beam
+users, provide a single place for importing and add sugar syntax for all
+Interactive Beam components. It gives users capability to interact with 
existing
+environment/session/context for Interactive Beam and visualize PCollections as
+bounded dataset. In the meantime, it hides the interactivity implementation
+from users so that users can focus on developing Beam pipeline without worrying
+about how hidden states in the interactive session are managed.
+
+Note: Backward-compatibility of Interactive Beam is only guaranteed within this
+module. Please only invoke interfaces provided by this module in your notebook
+or application code if you want backward-compatibility.
+"""
+
+from apache_beam.runners.interactive import interactive_environment as ie
+
+
+def watch(watchable):
+  """Watches a watchable so that Interactive Beam understands your pipeline.
+
+  If you write Beam pipeline in a notebook or __main__ module directly, since
 
 Review comment:
   Thx for sharing the contact!
   Will wait for @rosetn 's feedback to improve the comments before merging. 
ETA by this Friday.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310094)
Time Spent: 4h 40m  (was: 4.5h)

> Interactive Beam Caching PCollections bound to user defined vars in notebook
> 
>
> Key: BEAM-7760
> URL: https://issues.apache.org/jira/browse/BEAM-7760
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Cache only PCollections bound to user defined variables in a pipeline when 
> running pipeline with interactive runner in jupyter notebooks.
> [Interactive 
> Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]]
>  has been caching and using caches of "leaf" PCollections for interactive 
> execution in jupyter notebooks.
> The interactive execution is currently supported so that when appending new 
> transforms to existing pipeline for a new run, executed part of the pipeline 
> doesn't need to be re-executed. 
> A PCollection is "leaf" when it is never used as input in any PTransform in 
> the pipeline.
> The problem with building caches and pipeline to execute around "leaf" is 
> that when a PCollection is consumed by a sink with no output, the pipeline to 
> execute built will miss the subgraph generating and consuming that 
> PCollection.
> An example, "ReadFromPubSub --> WirteToPubSub" will result in an empty 
> pipeline.
> Caching around PCollections bound to user defined variables and replacing 
> transforms with source and sink of caches could resolve the pipeline to 
> execute properly under the interactive execution scenario. Also, cached 
> PCollection now can trace back to user code and can be used for user data

[jira] [Created] (BEAM-8199) Improve tooling to enable alerts on performance regression based on data collected from Beam benchmarks

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-8199:
-

 Summary: Improve tooling to enable alerts on performance 
regression based on data collected from Beam benchmarks
 Key: BEAM-8199
 URL: https://issues.apache.org/jira/browse/BEAM-8199
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Valentyn Tymofieiev


Currently we have some performance benchmarks in Beam, however they are not 
always monitored and we may  not notice when a regression happens, see [1]. 

Filing this issue to improve the tooling. Note that if automation becomes too 
noisy (too many regressions) people likely won't be paying attention either.

As a first step, we could monitor regression in 50th percentile of performance 
metrics we collect for benchmarks that we run repeatedly.

cc:  [~kamilwu] [~kasiak] [~markflyhigh]

https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-8198:
--
Description: 
context: 
https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E

Setting fix version to 2.16.0 to understand the cause, hopefully before the 
vote.

cc: [~altay] [~thw] [~markflyhigh]]

  was:
context: 
https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E

Setting fix version to 2.16.0 to understand the cause, hopefully before the 
vote.

cc: [~altay] [~thw] [~markliu]


> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markflyhigh]]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8170) datatype_inference_test break windows tests

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8170?focusedWorklogId=310093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310093
 ]

ASF GitHub Bot logged work on BEAM-8170:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:44
Start Date: 10/Sep/19 20:44
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #9522: [BEAM-8170] 
Don't import PyArrow on Python 2 + Windows
URL: https://github.com/apache/beam/pull/9522#discussion_r322953060
 
 

 ##
 File path: sdks/python/apache_beam/testing/datatype_inference.py
 ##
 @@ -34,6 +35,9 @@
   from avro.schema import parse as Parse  # avro library for python2
 # pylint: enable=wrong-import-order, wrong-import-position
 
+if not (platform.system() == 'Windows' and sys.version_info[0] == 2):
 
 Review comment:
   Thank you!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310093)
Time Spent: 1h 10m  (was: 1h)

> datatype_inference_test break windows tests
> ---
>
> Key: BEAM-8170
> URL: https://issues.apache.org/jira/browse/BEAM-8170
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Affects Versions: 2.16.0
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Blocker
> Fix For: Not applicable
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> These tests were added in https://github.com/apache/beam/pull/9281/files
> Windows + Py2 doesn't not support pyarrow.
> Please avoid importing pyarrow in this test. Example: 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio_test.py#L49-L58



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8195) Quota exceeded for create requests

[Ahmet Altay (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926882#comment-16926882
 ] 

Ahmet Altay commented on BEAM-8195:
---

Could we consider splitting into more projects for different types of tests?

> Quota exceeded for create requests
> --
>
> Key: BEAM-8195
> URL: https://issues.apache.org/jira/browse/BEAM-8195
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, testing
>Reporter: Ahmet Altay
>Assignee: yifan zou
>Priority: Critical
>
> Post commits failied with the following error:
> HttpError accessing 
> :
>  response: <{'server': 'ESF', '-content-encoding': 'gzip', 'content-type': 
> 'application/json; charset=UTF-8', 'content-length': '598', 
> 'transfer-encoding': 'chunked', 'cache-control': 'private', 
> 'x-xss-protection': '0', 'date': 'Tue, 10 Sep 2019 12:02:24 GMT', 'vary': 
> 'Origin, X-Origin, Referer', 'x-frame-options': 'SAMEORIGIN', 'status': 
> '429', 'x-content-type-options': 'nosniff'}>, content <{
>   "error": {
> "code": 429,
> "message": "Quota exceeded for quota metric 
> 'dataflow.googleapis.com/create_requests' and limit 
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for 
> consumer 'project_number:844138762903'.",
> "status": "RESOURCE_EXHAUSTED",
> "details": [
>   {
> Could we increase the quota?
> /cc [~alanmyrvold] [~kenn]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310091
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:37
Start Date: 10/Sep/19 20:37
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9487: 
[BEAM-8165]Change default docker images name
URL: https://github.com/apache/beam/pull/9487#issuecomment-530109210
 
 
   > `project.version` does not match Python version in master branch. It will 
be `2.16.0-SNAPSHOT` vs `2.16.0.dev`. I am guessing we can still use it and 
with a simple string raplace change '-SNAPSHOT' to '.dev' for python.
   
   I also investigated this option during weekend and there is a scenarios we 
cannot support with string replacement. 
   For released versions, it would be `{version}-SNAPSHOT` at 
`gradle.properties`, but `{version}` at `version.py`. So if a user build an 
image from released version, it would be tagged with `{version}.dev`, however, 
an image with `{version}` tag will be used as a default image, which would make 
difficult to investigate for users who are not familiar with how tagging works 
with Beam.
   This is the same issue as I mentioned previously, given 
`{version}-SNAPSHOT`, we cannot tell if it's developing version or released 
version. I also tried to change `{version}-SNAPSHOT` to `{version}` for 
released versions, and `{version}.dev` for developing versions, but 2.16 cutoff 
date is so close and the impact with the change would be wide, so I decided to 
introduce a new variable for now and make the change later.
   
   We have the version defined at several different locations and suffixes are 
not consistent, which should be improved. I changed scope of [the Jira 
ticket](https://issues.apache.org/jira/browse/BEAM-8192) to include this work.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310091)
Time Spent: 2h 10m  (was: 2h)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8192) Improve version definition

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8192:
---
Description: 
Version info is defined at several different locations and suffixes are not 
consistent.
For example, from version, we should be able to tell developing version vs 
released version.
Ideally, version is defined at a single location, and can be read from gradle 
and Py, Go, Java.

  was:Version info 


> Improve version definition
> --
>
> Key: BEAM-8192
> URL: https://issues.apache.org/jira/browse/BEAM-8192
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Hannah Jiang
>Priority: Major
>
> Version info is defined at several different locations and suffixes are not 
> consistent.
> For example, from version, we should be able to tell developing version vs 
> released version.
> Ideally, version is defined at a single location, and can be read from gradle 
> and Py, Go, Java.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=310088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310088
 ]

ASF GitHub Bot logged work on BEAM-7013:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:34
Start Date: 10/Sep/19 20:34
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #9519: [BEAM-7013] 
Use a 0-length byte array to represent empty sketch in HllCount
URL: https://github.com/apache/beam/pull/9519#discussion_r322949267
 
 

 ##
 File path: 
sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
 ##
 @@ -302,12 +304,53 @@ public void testMergePartialGlobally_SingletonInput() {
 
   @Test
   @Category(NeedsRunner.class)
-  public void testMergePartialGlobally_EmptyInput() {
+  public void testMergePartialGlobally_SingletonInputEmptySketch() {
 PCollection result =
-p.apply(Create.empty(TypeDescriptor.of(byte[].class)))
+
p.apply(Create.of(EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeWithEmptySketch() {
+PCollection result =
+p.apply(Create.of(LONGS_SKETCH, 
EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeMultipleEmptySketches() {
+PCollection result =
+p.apply(Create.of(EMPTY_SKETCH, 
EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeWithSketchOfEmptySet() {
+PCollection result =
+p.apply(Create.of(LONGS_SKETCH, LONGS_SKETCH_OF_EMPTY_SET))
+.apply(HllCount.MergePartial.globally());
+
+PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+p.run();
+  }
+
+  @Test
+  @Category(NeedsRunner.class)
+  public void testMergePartialGlobally_MergeEmptySketchWithSketchOfEmptySet() {
+PCollection result =
+p.apply(Create.of(EMPTY_SKETCH, LONGS_SKETCH_OF_EMPTY_SET))
 .apply(HllCount.MergePartial.globally());
 
-PAssert.that(result).empty();
+PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH_OF_EMPTY_SET);
 
 Review comment:
   > this is testing an implementation detail (there are other valid return 
values here).
   
   Yes. But the result is exposed to user anyway. So I prefer keeping it as it 
is. If we decide to return other values (byte[0]) in the future, we will change 
this unit test as well.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310088)
Time Spent: 33h 50m  (was: 33h 40m)

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 33h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8192) Improve version definition

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8192:
---
Summary: Improve version definition  (was: Remove hardcoded version for 
docker tasks)

> Improve version definition
> --
>
> Key: BEAM-8192
> URL: https://issues.apache.org/jira/browse/BEAM-8192
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Hannah Jiang
>Priority: Major
>
> Version info 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8192) Improve version definition

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8192:
---
Docs Text:   (was: For quick workaround, sdk_version is hardcoded for 
Python docker tasks. 
files: sdks/python/container/py*/build.gradle)

> Improve version definition
> --
>
> Key: BEAM-8192
> URL: https://issues.apache.org/jira/browse/BEAM-8192
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Hannah Jiang
>Priority: Major
>
> Version info 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-8198:
--
Status: Open  (was: Triage Needed)

> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markliu]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-8198:
-

 Summary: Investigate possible performance regression of Wordcount 
1GB batch benchmark on Py3.
 Key: BEAM-8198
 URL: https://issues.apache.org/jira/browse/BEAM-8198
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core, testing
Reporter: Valentyn Tymofieiev
Assignee: Valentyn Tymofieiev
 Fix For: 2.16.0


context: 
https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E

Setting fix version to 2.16.0 to understand the cause, hopefully before the 
vote.

cc: [~altay] [~thw] [~markliu]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=310085&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310085
 ]

ASF GitHub Bot logged work on BEAM-7013:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:30
Start Date: 10/Sep/19 20:30
Worklog Time Spent: 10m 
  Work Description: zfraa commented on issue #9519: [BEAM-7013] Use a 
0-length byte array to represent empty sketch in HllCount
URL: https://github.com/apache/beam/pull/9519#issuecomment-530106641
 
 
   Still LGTM! On the test naming with underscores: Yep, zetasketch does this 
-- we follow the principle outlined e.g. here: 
https://osherove.com/blog/2005/4/3/naming-standards-for-unit-tests.html, which 
is also consistent with internal Java Style. But let's follow Beam style here! 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310085)
Time Spent: 33h 40m  (was: 33.5h)

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 33h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=310084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310084
 ]

ASF GitHub Bot logged work on BEAM-7013:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:29
Start Date: 10/Sep/19 20:29
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #9519: [BEAM-7013] 
Use a 0-length byte array to represent empty sketch in HllCount
URL: https://github.com/apache/beam/pull/9519#discussion_r322947030
 
 

 ##
 File path: 
sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountMergePartialFn.java
 ##
 @@ -54,10 +54,15 @@ private HllCountMergePartialFn() {}
 return null;
   }
 
+  @Nullable
   @Override
   public HyperLogLogPlusPlus addInput(
   @Nullable HyperLogLogPlusPlus accumulator, byte[] input) {
 
 Review comment:
   @boyuanzz PTAL at the added logging
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310084)
Time Spent: 33.5h  (was: 33h 20m)

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 33.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=310083&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310083
 ]

ASF GitHub Bot logged work on BEAM-7013:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:27
Start Date: 10/Sep/19 20:27
Worklog Time Spent: 10m 
  Work Description: zfraa commented on pull request #9519: [BEAM-7013] Use 
a 0-length byte array to represent empty sketch in HllCount
URL: https://github.com/apache/beam/pull/9519#discussion_r322946215
 
 

 ##
 File path: 
sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountMergePartialFn.java
 ##
 @@ -54,10 +54,15 @@ private HllCountMergePartialFn() {}
 return null;
   }
 
+  @Nullable
   @Override
   public HyperLogLogPlusPlus addInput(
   @Nullable HyperLogLogPlusPlus accumulator, byte[] input) {
 
 Review comment:
   LGTM! Maybe Boyuan can also have a quick look at the logging? Not familiar 
with what's usual in Beam. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310083)
Time Spent: 33h 20m  (was: 33h 10m)

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 33h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8155) Add PubsubIO.readMessagesWithCoderAndParseFn method

[Chamikara Jayalath (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926870#comment-16926870
 ] 

Chamikara Jayalath commented on BEAM-8155:
--

Closing since PR was merged. Thanks!

> Add PubsubIO.readMessagesWithCoderAndParseFn method
> ---
>
> Key: BEAM-8155
> URL: https://issues.apache.org/jira/browse/BEAM-8155
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Claire McGinty
>Assignee: Claire McGinty
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Complements [#8879|https://github.com/apache/beam/pull/8879]; the constructor 
> for a generic-typed PubsubIO.Read is private, so some kind of public 
> constructor is needed to use custom parse functions with generic types.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (BEAM-8155) Add PubsubIO.readMessagesWithCoderAndParseFn method

[Chamikara Jayalath (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-8155.
--
Resolution: Fixed

> Add PubsubIO.readMessagesWithCoderAndParseFn method
> ---
>
> Key: BEAM-8155
> URL: https://issues.apache.org/jira/browse/BEAM-8155
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Claire McGinty
>Assignee: Claire McGinty
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Complements [#8879|https://github.com/apache/beam/pull/8879]; the constructor 
> for a generic-typed PubsubIO.Read is private, so some kind of public 
> constructor is needed to use custom parse functions with generic types.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8170) datatype_inference_test break windows tests

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8170?focusedWorklogId=310073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310073
 ]

ASF GitHub Bot logged work on BEAM-8170:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:12
Start Date: 10/Sep/19 20:12
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #9522: [BEAM-8170] 
Don't import PyArrow on Python 2 + Windows
URL: https://github.com/apache/beam/pull/9522#discussion_r322939896
 
 

 ##
 File path: sdks/python/apache_beam/testing/datatype_inference.py
 ##
 @@ -34,6 +35,9 @@
   from avro.schema import parse as Parse  # avro library for python2
 # pylint: enable=wrong-import-order, wrong-import-position
 
+if not (platform.system() == 'Windows' and sys.version_info[0] == 2):
 
 Review comment:
   https://issues.apache.org/jira/browse/BEAM-8188 opened for the cleanup
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310073)
Time Spent: 1h  (was: 50m)

> datatype_inference_test break windows tests
> ---
>
> Key: BEAM-8170
> URL: https://issues.apache.org/jira/browse/BEAM-8170
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Affects Versions: 2.16.0
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Blocker
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> These tests were added in https://github.com/apache/beam/pull/9281/files
> Windows + Py2 doesn't not support pyarrow.
> Please avoid importing pyarrow in this test. Example: 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio_test.py#L49-L58



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-1251) Python 3 Support

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=310072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310072
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 10/Sep/19 20:10
Start Date: 10/Sep/19 20:10
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #9537: [BEAM-1251] 
Warn Python 2 users about Beam becoming Python 3-only in the future to 
encourage migration.
URL: https://github.com/apache/beam/pull/9537
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310072)
Time Spent: 30h 10m  (was: 30h)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Time Spent: 30h 10m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian Jira
(v8.3.2#803003)