[GitHub] [beam] robinyqiu opened a new pull request #12163: [BEAM-9712] Refactor ZetaSQL planner code and support setting default timezone

2020-07-01 Thread GitBox
robinyqiu opened a new pull request #12163: URL: https://github.com/apache/beam/pull/12163 Enable setting default timezone option on ZetaSQL analyzer. Default timezone is used both at - query analysis time (e.g. `CAST` literal to `TIMESTAMP`), and - expression execution time (e.g. `TI

[GitHub] [beam] kennknowles opened a new pull request #12162: [BEAM-10402] Enable checker framework and eliminate nullability errors in some modules, disable it in other modules

2020-07-01 Thread GitBox
kennknowles opened a new pull request #12162: URL: https://github.com/apache/beam/pull/12162 I enabled the checker framework and immediately discovered over 100 nullability errors in the main Java SDK. Opening a PR as the simple way to publicize the breadth of errors in Beam. I will fix er

[GitHub] [beam] lukecwik commented on pull request #12016: [BEAM-10341] Support drain in python and java SDF

2020-07-01 Thread GitBox
lukecwik commented on pull request #12016: URL: https://github.com/apache/beam/pull/12016#issuecomment-652762695 Note to self, make sure that capabilities are added for SDKs that support drain. This is an automated message f

[GitHub] [beam] lukecwik commented on a change in pull request #12016: [BEAM-10341] Support drain in python and java SDF

2020-07-01 Thread GitBox
lukecwik commented on a change in pull request #12016: URL: https://github.com/apache/beam/pull/12016#discussion_r448727139 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java ## @@ -1053,6 +1056,51 @@ public Duration getAllowedTimestampSkew() {

[GitHub] [beam] lukecwik commented on a change in pull request #12016: [BEAM-10341] Support drain in python and java SDF

2020-07-01 Thread GitBox
lukecwik commented on a change in pull request #12016: URL: https://github.com/apache/beam/pull/12016#discussion_r448725242 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokersTest.java ## @@ -847,44 +848,26 @@ public Object restrictio

[GitHub] [beam] ettirapp opened a new pull request #12161: Adding Azure IO FileSystem

2020-07-01 Thread GitBox
ettirapp opened a new pull request #12161: URL: https://github.com/apache/beam/pull/12161 Adding an AzureBlobStoreFileSystem to the Java SDK. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and ea

[GitHub] [beam] robinyqiu merged pull request #12154: [BEAM-9709] Add test for timestamp from date in default timezone

2020-07-01 Thread GitBox
robinyqiu merged pull request #12154: URL: https://github.com/apache/beam/pull/12154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] robinyqiu commented on pull request #12154: [BEAM-9709] Add test for timestamp from date in default timezone

2020-07-01 Thread GitBox
robinyqiu commented on pull request #12154: URL: https://github.com/apache/beam/pull/12154#issuecomment-652744681 Thanks for your review. Spotless passed. This is an automated message from the Apache Git Service. To respond t

[GitHub] [beam] robinyqiu commented on a change in pull request #12154: [BEAM-9709] Add test for timestamp from date in default timezone

2020-07-01 Thread GitBox
robinyqiu commented on a change in pull request #12154: URL: https://github.com/apache/beam/pull/12154#discussion_r448711593 ## File path: sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java ## @@ -2791,56 +2791

[GitHub] [beam] ajamato commented on a change in pull request #12083: [BEAM-10317] Java - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-01 Thread GitBox
ajamato commented on a change in pull request #12083: URL: https://github.com/apache/beam/pull/12083#discussion_r448693178 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java ## @@ -216,11 +221,11 @@ publ

[GitHub] [beam] TheNeuralBit commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-652720997 > Unfortunately the python test is taken by tox precommit suites - I suppose it shouldn't? I think that's actually ok. The precommit will uses the fn api runner, which

[GitHub] [beam] TheNeuralBit commented on pull request #12145: [BEAM-10136] [BEAM-10135] Add JdbcIO for cross-language with python wrapper

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12145: URL: https://github.com/apache/beam/pull/12145#issuecomment-652719638 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit merged pull request #12160: Fix typo

2020-07-01 Thread GitBox
TheNeuralBit merged pull request #12160: URL: https://github.com/apache/beam/pull/12160 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652715930 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12090: [BEAM-10336,BEAM-10337] Add SchemaIO abstraction and implement for PubSub

2020-07-01 Thread GitBox
TheNeuralBit commented on a change in pull request #12090: URL: https://github.com/apache/beam/pull/12090#discussion_r448684950 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubSchemaCapableIOProvider.java ## @@ -0,0 +1,286

[GitHub] [beam] lukecwik commented on pull request #12160: Fix typo

2020-07-01 Thread GitBox
lukecwik commented on pull request #12160: URL: https://github.com/apache/beam/pull/12160#issuecomment-652715470 R: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] lukecwik opened a new pull request #12160: Fix typo

2020-07-01 Thread GitBox
lukecwik opened a new pull request #12160: URL: https://github.com/apache/beam/pull/12160 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://b

[GitHub] [beam] ajamato commented on pull request #12084: [BEAM-10317] Python - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-01 Thread GitBox
ajamato commented on pull request #12084: URL: https://github.com/apache/beam/pull/12084#issuecomment-652714875 Thanks Pablo. I addressed the lint issues and pushed the changes. Hopefully it passes This is an automated messa

[GitHub] [beam] apilloud commented on pull request #12159: [BEAM-10093] Run ZetaSQL Nexmark in postcommit

2020-07-01 Thread GitBox
apilloud commented on pull request #12159: URL: https://github.com/apache/beam/pull/12159#issuecomment-652710780 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] apilloud commented on pull request #12159: [BEAM-10093] Run ZetaSQL Nexmark in postcommit

2020-07-01 Thread GitBox
apilloud commented on pull request #12159: URL: https://github.com/apache/beam/pull/12159#issuecomment-652710628 run direct Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [beam] apilloud commented on pull request #12159: [BEAM-10093] Run ZetaSQL Nexmark in postcommit

2020-07-01 Thread GitBox
apilloud commented on pull request #12159: URL: https://github.com/apache/beam/pull/12159#issuecomment-652710594 run Dataflow Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [beam] lukecwik commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.

2020-07-01 Thread GitBox
lukecwik commented on pull request #12137: URL: https://github.com/apache/beam/pull/12137#issuecomment-652709644 > Thanks, looks good! > > There is still an issue with the tests: > > ``` > java.lang.Exception: Unexpected exception, expected but was > ``` > > htt

[GitHub] [beam] lostluck merged pull request #12113: Add Branching to Go SDK katas

2020-07-01 Thread GitBox
lostluck merged pull request #12113: URL: https://github.com/apache/beam/pull/12113 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] apilloud commented on pull request #12159: [BEAM-10093] Run ZetaSQL Nexmark in postcommit

2020-07-01 Thread GitBox
apilloud commented on pull request #12159: URL: https://github.com/apache/beam/pull/12159#issuecomment-652704860 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] apilloud opened a new pull request #12159: [BEAM-10093] Run ZetaSQL Nexmark in postcommit

2020-07-01 Thread GitBox
apilloud opened a new pull request #12159: URL: https://github.com/apache/beam/pull/12159 Run tests in postcommit. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choo

[GitHub] [beam] pabloem merged pull request #12085: [BEAM-10318] fix uninitialized grpc_server in FnApiRunner

2020-07-01 Thread GitBox
pabloem merged pull request #12085: URL: https://github.com/apache/beam/pull/12085 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] tvalentyn commented on pull request #12150: [BEAM-10398] Use GitHub Actions in wheels release process for Python

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12150: URL: https://github.com/apache/beam/pull/12150#issuecomment-652701878 Thanks, @aaltay and @ibzib for your comments. @TobKed we can cherry-pick this to the release branch once merged, and I can give it a try on 2.23.0. ---

[GitHub] [beam] TheNeuralBit commented on pull request #12090: [BEAM-10336,BEAM-10337] Add SchemaIO abstraction and implement for PubSub

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12090: URL: https://github.com/apache/beam/pull/12090#issuecomment-652701140 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] TheNeuralBit commented on pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12067: URL: https://github.com/apache/beam/pull/12067#issuecomment-652700056 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] angoenka commented on a change in pull request #12042: [BEAM-9962] Jenkins Plugin

2020-07-01 Thread GitBox
angoenka commented on a change in pull request #12042: URL: https://github.com/apache/beam/pull/12042#discussion_r448673087 ## File path: beam-ci/src/main/resources/index.jelly ## @@ -0,0 +1,7 @@ + Review comment: Let's add the license here.

[GitHub] [beam] angoenka commented on a change in pull request #12042: [BEAM-9962] Jenkins Plugin

2020-07-01 Thread GitBox
angoenka commented on a change in pull request #12042: URL: https://github.com/apache/beam/pull/12042#discussion_r448671816 ## File path: beam-ci/src/main/java/io/jenkins/plugins/PipelineLauncher.java ## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [beam] apilloud commented on a change in pull request #12154: [BEAM-9709] Add test for timestamp from date in default timezone

2020-07-01 Thread GitBox
apilloud commented on a change in pull request #12154: URL: https://github.com/apache/beam/pull/12154#discussion_r448672050 ## File path: sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java ## @@ -3724,6 +3690,5

[GitHub] [beam] angoenka commented on pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
angoenka commented on pull request #12042: URL: https://github.com/apache/beam/pull/12042#issuecomment-652697557 cc: @TobKed PTAL This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] lukecwik commented on a change in pull request #12016: [BEAM-10341] Support drain in python and java SDF

2020-07-01 Thread GitBox
lukecwik commented on a change in pull request #12016: URL: https://github.com/apache/beam/pull/12016#discussion_r448663612 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -1079,6 +1079,13 @@ def construct_bundle_application(self,

[GitHub] [beam] tvalentyn commented on pull request #11709: Throw an error when main output has the same tag as side outputs

2020-07-01 Thread GitBox
tvalentyn commented on pull request #11709: URL: https://github.com/apache/beam/pull/11709#issuecomment-652692320 Ping please. This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] lukecwik commented on a change in pull request #12016: [BEAM-10341] Support drain in python and java SDF

2020-07-01 Thread GitBox
lukecwik commented on a change in pull request #12016: URL: https://github.com/apache/beam/pull/12016#discussion_r448655800 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py ## @@ -1453,14 +1447,18 @@ def split_manager(num_elements):

[GitHub] [beam] tvalentyn commented on pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12156: URL: https://github.com/apache/beam/pull/12156#issuecomment-652691544 There is one unrelated failure due to https://issues.apache.org/jira/browse/BEAM-9975 in one py37 suite, the rest of the tests passed, merging this change. -

[GitHub] [beam] tvalentyn merged pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn merged pull request #12156: URL: https://github.com/apache/beam/pull/12156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tvalentyn commented on a change in pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn commented on a change in pull request #12156: URL: https://github.com/apache/beam/pull/12156#discussion_r448663663 ## File path: sdks/python/setup.py ## @@ -259,11 +259,17 @@ def run(self): python_requires = '>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*' -if sys

[GitHub] [beam] ibzib merged pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib merged pull request #12153: URL: https://github.com/apache/beam/pull/12153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [beam] ibzib commented on a change in pull request #12157: [BEAM-7587] Spark portable streaming

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12157: URL: https://github.com/apache/beam/pull/12157#discussion_r448656410 ## File path: runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java ## @@ -40,9 +41,7 @@ import org.apache.beam.runners.jobsub

[GitHub] [beam] tvalentyn edited a comment on pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
tvalentyn edited a comment on pull request #12042: URL: https://github.com/apache/beam/pull/12042#issuecomment-652683956 Also, please add a Jira issue to the PR title. This is an automated message from the Apache Git Service.

[GitHub] [beam] damondouglas commented on pull request #12113: Add Branching to Go SDK katas

2020-07-01 Thread GitBox
damondouglas commented on pull request #12113: URL: https://github.com/apache/beam/pull/12113#issuecomment-652684093 @lostluck / @henryken I've updated [the stepik course](https://stepik.org/course/70387) and committed the modified `*-remote.yaml` files to this PR. It is ready to merge.

[GitHub] [beam] tvalentyn commented on pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12042: URL: https://github.com/apache/beam/pull/12042#issuecomment-652683956 Also, please add the Jira to the PR title. This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] tvalentyn commented on pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12042: URL: https://github.com/apache/beam/pull/12042#issuecomment-652683753 High-level note: I might have missed a discussion or a design but from skimming through the PR description it is not clear what the purpose of Jenkins plugin is. It would be he

[GitHub] [beam] JustineKoa commented on a change in pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
JustineKoa commented on a change in pull request #12042: URL: https://github.com/apache/beam/pull/12042#discussion_r448522058 ## File path: beam-ci/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker ## @@ -0,0 +1 @@ +mock-maker-inline Review comment: O

[GitHub] [beam] ibzib commented on pull request #12157: [BEAM-7587] Spark portable streaming

2020-07-01 Thread GitBox
ibzib commented on pull request #12157: URL: https://github.com/apache/beam/pull/12157#issuecomment-652683027 > I currently have streaming context as a subclass of the original spark translation context, but thinking some more I'm not sure if that totally works, and it might be better to h

[GitHub] [beam] tvalentyn commented on pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12156: URL: https://github.com/apache/beam/pull/12156#issuecomment-652681040 @pabloem Yes, it's intended, because we our Py3.8 test coverage is not yet up to the same standard as other versions, but we are working on that (adding Postcommit tests in htt

[GitHub] [beam] tvalentyn edited a comment on pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn edited a comment on pull request #12156: URL: https://github.com/apache/beam/pull/12156#issuecomment-652681040 @pabloem Yes, it's intended, because our Py3.8 test coverage is not yet up to the same standard as other versions, but we are working on that (adding Postcommit tests in

[GitHub] [beam] ettirapp opened a new pull request #12158: [BEAM-10378] Adding Azure IO module

2020-07-01 Thread GitBox
ettirapp opened a new pull request #12158: URL: https://github.com/apache/beam/pull/12158 Adding an new module for Azure IO's. Work ongoing to implement Azure filesystem in the Java SDK. Thank you for your contribution! Follow this checklist to help u

[GitHub] [beam] annaqin418 commented on pull request #12157: [BEAM-7587] Spark portable streaming

2020-07-01 Thread GitBox
annaqin418 commented on pull request #12157: URL: https://github.com/apache/beam/pull/12157#issuecomment-652678870 R: @ibzib Creating this PR for now to track changes as I go. I set up some of the new interfaces, could you check if they look reasonable? I currently have streaming conte

[GitHub] [beam] pabloem commented on a change in pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
pabloem commented on a change in pull request #12156: URL: https://github.com/apache/beam/pull/12156#discussion_r448649324 ## File path: sdks/python/setup.py ## @@ -259,11 +259,17 @@ def run(self): python_requires = '>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*' -if sys.v

[GitHub] [beam] lostluck commented on a change in pull request #12113: Add Branching to Go SDK katas

2020-07-01 Thread GitBox
lostluck commented on a change in pull request #12113: URL: https://github.com/apache/beam/pull/12113#discussion_r448648297 ## File path: learning/katas/go/Core Transforms/Branching/Branching/task.md ## @@ -0,0 +1,36 @@ + + +# Branching + +The Beam SDK allows you to apply multi

[GitHub] [beam] kennknowles commented on a change in pull request #12067: [BEAM-10308] Make component ID assignments consistent across PipelineContext instances

2020-07-01 Thread GitBox
kennknowles commented on a change in pull request #12067: URL: https://github.com/apache/beam/pull/12067#discussion_r448646153 ## File path: sdks/python/apache_beam/pipeline.py ## @@ -221,6 +221,14 @@ def __init__(self, runner=None, options=None, argv=None): # then the tra

[GitHub] [beam] annaqin418 opened a new pull request #12157: [BEAM-7587] Spark portable streaming

2020-07-01 Thread GitBox
annaqin418 opened a new pull request #12157: URL: https://github.com/apache/beam/pull/12157 These changes will allow streaming data on the portable Spark runner. Thank you for your contribution! Follow this checklist to help us incorporate your contri

[GitHub] [beam] tvalentyn commented on a change in pull request #12115: [BEAM-7672] dynamically setup acceptable wheel specs according to installed python version

2020-07-01 Thread GitBox
tvalentyn commented on a change in pull request #12115: URL: https://github.com/apache/beam/pull/12115#discussion_r448642475 ## File path: sdks/python/container/boot.go ## @@ -170,6 +176,28 @@ func main() { log.Fatalf("Python exited: %v", execx.Execute("python", args...

[GitHub] [beam] pabloem commented on pull request #12084: [BEAM-10317] Python - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-01 Thread GitBox
pabloem commented on pull request #12084: URL: https://github.com/apache/beam/pull/12084#issuecomment-652671801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [beam] tvalentyn commented on pull request #12115: [BEAM-7672] dynamically setup acceptable wheel specs according to installed python version

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12115: URL: https://github.com/apache/beam/pull/12115#issuecomment-652669351 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] tvalentyn commented on pull request #12115: [BEAM-7672] dynamically setup acceptable wheel specs according to installed python version

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12115: URL: https://github.com/apache/beam/pull/12115#issuecomment-652669205 Run Dataflow Python ValidatesContainer This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] tvalentyn commented on pull request #12115: [BEAM-7672] dynamically setup acceptable wheel specs according to installed python version

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12115: URL: https://github.com/apache/beam/pull/12115#issuecomment-652669110 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] lostluck commented on pull request #12141: [BEAM-9615] Add Row coder functions.

2020-07-01 Thread GitBox
lostluck commented on pull request #12141: URL: https://github.com/apache/beam/pull/12141#issuecomment-652664890 I won't merge this yet, since jenkin's seems to be disconnected for now. I'm hoping I caught the last of the go v1.12 nits (around bit shift restrictions) -

[GitHub] [beam] tvalentyn commented on pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12156: URL: https://github.com/apache/beam/pull/12156#issuecomment-652664328 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] boyuanzz commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-01 Thread GitBox
boyuanzz commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-652661101 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] boyuanzz commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-01 Thread GitBox
boyuanzz commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-652660708 Looking. Sorry for the late. This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] davidyan74 commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-01 Thread GitBox
davidyan74 commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-652659503 @boyuanzz If possible, please take a look at this soon so that we can make the cut for 2.23.0. Thanks! This i

[GitHub] [beam] tvalentyn commented on pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn commented on pull request #12156: URL: https://github.com/apache/beam/pull/12156#issuecomment-652655334 R: @aaltay cc: @epicfaace @lazylynx @kamilwu This is an automated message from the Apache Git Service. To

[GitHub] [beam] tvalentyn opened a new pull request #12156: [BEAM-9754] Add Py 3.8 support to Dataflow runner.

2020-07-01 Thread GitBox
tvalentyn opened a new pull request #12156: URL: https://github.com/apache/beam/pull/12156 Adds Py3.8 support to Dataflow runner now that Dataflow provides Py 3.8 containers. Also add a user warning for Py3.8 and newer versions for which Apache Beam does not yet have robust test cover

[GitHub] [beam] ibzib commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448621802 ## File path: sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java ## @@ -2841,6 +2842,131

[GitHub] [beam] amaliujia commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
amaliujia commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448619519 ## File path: sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java ## @@ -2841,6 +2842,

[GitHub] [beam] amaliujia commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
amaliujia commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448618515 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java ## @@ -920,

[GitHub] [beam] amaliujia commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
amaliujia commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448615855 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -130,30 +135,58 @@

[GitHub] [beam] ibzib commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448614825 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -130,30 +135,58 @@ pub

[GitHub] [beam] ibzib commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448613591 ## File path: sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java ## @@ -2841,6 +2842,129

[GitHub] [beam] ibzib commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448613319 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java ## @@ -920,8 +9

[GitHub] [beam] Ardagan commented on pull request #12139: [DO NOT REVIEW] Scd py bq test

2020-07-01 Thread GitBox
Ardagan commented on pull request #12139: URL: https://github.com/apache/beam/pull/12139#issuecomment-652639984 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] ibzib commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448608405 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java ## @@ -107,6 +128,43 @@ ResolvedSta

[GitHub] [beam] ibzib commented on a change in pull request #12153: [BEAM-9953] [ZetaSQL] Implement CREATE FUNCTION and scalar UDF.

2020-07-01 Thread GitBox
ibzib commented on a change in pull request #12153: URL: https://github.com/apache/beam/pull/12153#discussion_r448607767 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java ## @@ -130,30 +135,57 @@ pub

[GitHub] [beam] JustineKoa commented on a change in pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
JustineKoa commented on a change in pull request #12042: URL: https://github.com/apache/beam/pull/12042#discussion_r448536674 ## File path: beam-ci/src/main/java/io/jenkins/plugins/ExecuteBeamPipelineOnDataflowBuilder.java ## @@ -0,0 +1,206 @@ +/* + * Licensed to the Apache So

[GitHub] [beam] angoenka commented on a change in pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
angoenka commented on a change in pull request #12042: URL: https://github.com/apache/beam/pull/12042#discussion_r448534456 ## File path: beam-ci/src/main/java/io/jenkins/plugins/ExecuteBeamPipelineOnDataflowBuilder.java ## @@ -0,0 +1,206 @@ +/* + * Licensed to the Apache Soft

[GitHub] [beam] je-ik commented on pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik commented on pull request #12155: URL: https://github.com/apache/beam/pull/12155#issuecomment-652632861 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] JustineKoa commented on a change in pull request #12042: Jenkins Plugin

2020-07-01 Thread GitBox
JustineKoa commented on a change in pull request #12042: URL: https://github.com/apache/beam/pull/12042#discussion_r448599468 ## File path: beam-ci/src/main/resources/index.jelly ## @@ -0,0 +1,4 @@ + + +TODO Review comment: I found out that "This view is used to ren

[GitHub] [beam] ajamato commented on a change in pull request #12084: [BEAM-10317] Python - Update BigQueryIO to tag BigQuery Jobs with the Dataflow Job ID

2020-07-01 Thread GitBox
ajamato commented on a change in pull request #12084: URL: https://github.com/apache/beam/pull/12084#discussion_r448551682 ## File path: sdks/python/apache_beam/io/gcp/bigquery_io_metadata.py ## @@ -0,0 +1,55 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one o

[GitHub] [beam] tvalentyn commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-01 Thread GitBox
tvalentyn commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-652628585 @epicfaace , Actually, I will send out Dataflow changes first, then we can add postcommit tests, this way it would be easier to separate and merge the changes, especially if so

[GitHub] [beam] davidyan74 commented on pull request #12143: [BEAM-10291] Adding full thread dump upon lull detection for Dataflow…

2020-07-01 Thread GitBox
davidyan74 commented on pull request #12143: URL: https://github.com/apache/beam/pull/12143#issuecomment-652626188 Test failure is on the flink runner and appears to have nothing to do with this PR. This is an automated mess

[GitHub] [beam] epicfaace commented on pull request #11824: [BEAM-10101] Add HttpIO / HttpFileSystem (Python)

2020-07-01 Thread GitBox
epicfaace commented on pull request #11824: URL: https://github.com/apache/beam/pull/11824#issuecomment-652626019 I'll take a look! This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] TheNeuralBit merged pull request #12136: [BEAM-10391] Add attribute map null check in dataflow worker

2020-07-01 Thread GitBox
TheNeuralBit merged pull request #12136: URL: https://github.com/apache/beam/pull/12136 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] TheNeuralBit commented on pull request #12136: [BEAM-10391] Add attribute map null check in dataflow worker

2020-07-01 Thread GitBox
TheNeuralBit commented on pull request #12136: URL: https://github.com/apache/beam/pull/12136#issuecomment-652625495 > Are we sure that gcp.pubsub.PubsubMessage is right and it is Dataflow that is wrong? There's also PubsubClient, which checks for null in getAttributeMap: https://gi

[GitHub] [beam] epicfaace commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-01 Thread GitBox
epicfaace commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-652621627 @tvalentyn I believe it should be ready now! This is an automated message from the Apache Git Service. To respo

[GitHub] [beam] je-ik commented on pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik commented on pull request #12155: URL: https://github.com/apache/beam/pull/12155#issuecomment-652621561 Run Direct ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [beam] aaltay commented on pull request #11959: refactor HCLS IO ITs to support stores in other projects

2020-07-01 Thread GitBox
aaltay commented on pull request #11959: URL: https://github.com/apache/beam/pull/11959#issuecomment-652620523 @jaketf - what is the next step on this PR? This is an automated message from the Apache Git Service. To respond t

[GitHub] [beam] tvalentyn commented on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-01 Thread GitBox
tvalentyn commented on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-652618865 Hey @epicfaace, I rebased your first 3 commits on top of masters and did some changes to enable Dataflow postcommits: https://github.com/apache/beam/compare/master...tvalentyn:

[GitHub] [beam] tvalentyn edited a comment on pull request #11788: [BEAM-9785] Add Python 3.8 postcommit tests

2020-07-01 Thread GitBox
tvalentyn edited a comment on pull request #11788: URL: https://github.com/apache/beam/pull/11788#issuecomment-652618865 Hey @epicfaace, I rebased your first 3 commits on top of master and did some changes to enable Dataflow postcommits: https://github.com/apache/beam/compare/master...tval

[GitHub] [beam] je-ik commented on a change in pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik commented on a change in pull request #12155: URL: https://github.com/apache/beam/pull/12155#discussion_r448579752 ## File path: runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java ## @@ -1222,6 +1209,7 @@ public synchronized void refre

[GitHub] [beam] je-ik commented on a change in pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik commented on a change in pull request #12155: URL: https://github.com/apache/beam/pull/12155#discussion_r448579090 ## File path: runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java ## @@ -270,11 +177,13 @@ public void proce

[GitHub] [beam] je-ik commented on a change in pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik commented on a change in pull request #12155: URL: https://github.com/apache/beam/pull/12155#discussion_r448578900 ## File path: runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java ## @@ -128,19 +108,6 @@ public void cleanu

[GitHub] [beam] je-ik commented on a change in pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik commented on a change in pull request #12155: URL: https://github.com/apache/beam/pull/12155#discussion_r448578714 ## File path: runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java ## @@ -70,6 +71,8 @@ public static ExecutionDriver crea

[GitHub] [beam] je-ik opened a new pull request #12155: [BEAM-10400] Fix race condition in DirectRunner watermark update

2020-07-01 Thread GitBox
je-ik opened a new pull request #12155: URL: https://github.com/apache/beam/pull/12155 Fixes [BEAM-10400] Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose revie

[GitHub] [beam] je-ik closed pull request #12152: Fix direct runner race during watermark update

2020-07-01 Thread GitBox
je-ik closed pull request #12152: URL: https://github.com/apache/beam/pull/12152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [beam] je-ik commented on pull request #12152: [WIP] Fix direct runner race during watermark update

2020-07-01 Thread GitBox
je-ik commented on pull request #12152: URL: https://github.com/apache/beam/pull/12152#issuecomment-652605959 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] aaltay merged pull request #12129: [BEAM-6696] Change GroupIntoBatches to group for real

2020-07-01 Thread GitBox
aaltay merged pull request #12129: URL: https://github.com/apache/beam/pull/12129 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

  1   2   >