[GitHub] [beam] vectorijk commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-29 Thread GitBox
vectorijk commented on a change in pull request #12962: URL: https://github.com/apache/beam/pull/12962#discussion_r496966029 ## File path: vendor/calcite-1_25_0/build.gradle ## @@ -18,29 +18,33 @@ plugins { id 'org.apache.beam.vendor-java' } -description = "Apache Beam ::

[GitHub] [beam] youngoli commented on pull request #12945: [BEAM-10971] Redirect contribute/design-documents to cwiki.

2020-09-29 Thread GitBox
youngoli commented on pull request #12945: URL: https://github.com/apache/beam/pull/12945#issuecomment-701121286 Nice! Good find with that tool This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [beam] codecov[bot] edited a comment on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701034584 # [Codecov](https://codecov.io/gh/apache/beam/pull/12969?src=pr&el=h1) Report > Merging [#12969](https://codecov.io/gh/apache/beam/pull/12969?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12972: Revert "[BEAM-8024] Add JPMS E2E test"

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12972: URL: https://github.com/apache/beam/pull/12972#issuecomment-701095326 # [Codecov](https://codecov.io/gh/apache/beam/pull/12972?src=pr&el=h1) Report > Merging [#12972](https://codecov.io/gh/apache/beam/pull/12972?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] commented on pull request #12972: Revert "[BEAM-8024] Add JPMS E2E test"

2020-09-29 Thread GitBox
codecov[bot] commented on pull request #12972: URL: https://github.com/apache/beam/pull/12972#issuecomment-701095326 # [Codecov](https://codecov.io/gh/apache/beam/pull/12972?src=pr&el=h1) Report > Merging [#12972](https://codecov.io/gh/apache/beam/pull/12972?src=pr&el=desc) into [master

[GitHub] [beam] TheNeuralBit commented on pull request #12505: [BEAM-8106] Add version to java container image name

2020-09-29 Thread GitBox
TheNeuralBit commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-701081225 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [beam] robertwb merged pull request #12873: Remove experimental declarations from fileio.

2020-09-29 Thread GitBox
robertwb merged pull request #12873: URL: https://github.com/apache/beam/pull/12873 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298 # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr&el=h1) Report > Merging [#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr&el=desc) into

[GitHub] [beam] robertwb commented on a change in pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
robertwb commented on a change in pull request #12819: URL: https://github.com/apache/beam/pull/12819#discussion_r497179044 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -428,7 +482,106 @@ def print_partition(indent, desc, n, total): print() -def te

[GitHub] [beam] codecov[bot] edited a comment on pull request #12873: Remove experimental declarations from fileio.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12873: URL: https://github.com/apache/beam/pull/12873#issuecomment-701056996 # [Codecov](https://codecov.io/gh/apache/beam/pull/12873?src=pr&el=h1) Report > Merging [#12873](https://codecov.io/gh/apache/beam/pull/12873?src=pr&el=desc) into

[GitHub] [beam] TheNeuralBit commented on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
TheNeuralBit commented on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701061894 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] TheNeuralBit merged pull request #12520: [BEAM-10677] Fix @SchemaFieldName in AutoValueSchema

2020-09-29 Thread GitBox
TheNeuralBit merged pull request #12520: URL: https://github.com/apache/beam/pull/12520 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] codecov[bot] edited a comment on pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12971: URL: https://github.com/apache/beam/pull/12971#issuecomment-701054283 # [Codecov](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=h1) Report > Merging [#12971](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=desc) into

[GitHub] [beam] emilymye commented on pull request #12505: [BEAM-8106] Add version to java container image name

2020-09-29 Thread GitBox
emilymye commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-701060291 @TheNeuralBit composing an email now This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298 # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr&el=h1) Report > Merging [#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12873: Remove experimental declarations from fileio.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12873: URL: https://github.com/apache/beam/pull/12873#issuecomment-701056996 # [Codecov](https://codecov.io/gh/apache/beam/pull/12873?src=pr&el=h1) Report > Merging [#12873](https://codecov.io/gh/apache/beam/pull/12873?src=pr&el=desc) into

[GitHub] [beam] kileys opened a new pull request #12972: Revert "[BEAM-8024] Add JPMS E2E test"

2020-09-29 Thread GitBox
kileys opened a new pull request #12972: URL: https://github.com/apache/beam/pull/12972 Reverts apache/beam#12899 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[GitHub] [beam] codecov[bot] edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-692856347 # [Codecov](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=h1) Report > Merging [#12779](https://codecov.io/gh/apache/beam/pull/12779?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] commented on pull request #12873: Remove experimental declarations from fileio.

2020-09-29 Thread GitBox
codecov[bot] commented on pull request #12873: URL: https://github.com/apache/beam/pull/12873#issuecomment-701056996 # [Codecov](https://codecov.io/gh/apache/beam/pull/12873?src=pr&el=h1) Report > Merging [#12873](https://codecov.io/gh/apache/beam/pull/12873?src=pr&el=desc) into [master

[GitHub] [beam] codecov[bot] edited a comment on pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12971: URL: https://github.com/apache/beam/pull/12971#issuecomment-701054283 # [Codecov](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=h1) Report > Merging [#12971](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=desc) into

[GitHub] [beam] robertwb commented on a change in pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
robertwb commented on a change in pull request #12819: URL: https://github.com/apache/beam/pull/12819#discussion_r497159403 ## File path: sdks/python/apache_beam/dataframe/pandas_docs_test.py ## @@ -0,0 +1,91 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one o

[GitHub] [beam] robertwb commented on a change in pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
robertwb commented on a change in pull request #12819: URL: https://github.com/apache/beam/pull/12819#discussion_r497158944 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -428,7 +482,106 @@ def print_partition(indent, desc, n, total): print() -def te

[GitHub] [beam] codecov[bot] commented on pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-09-29 Thread GitBox
codecov[bot] commented on pull request #12971: URL: https://github.com/apache/beam/pull/12971#issuecomment-701054283 # [Codecov](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=h1) Report > Merging [#12971](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=desc) into [master

[GitHub] [beam] robertwb commented on a change in pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
robertwb commented on a change in pull request #12819: URL: https://github.com/apache/beam/pull/12819#discussion_r497156063 ## File path: sdks/python/apache_beam/runners/direct/evaluation_context.py ## @@ -102,7 +102,8 @@ def __init__(self, side_inputs): list ) #

[GitHub] [beam] robertwb commented on a change in pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
robertwb commented on a change in pull request #12819: URL: https://github.com/apache/beam/pull/12819#discussion_r497155488 ## File path: sdks/python/apache_beam/dataframe/doctests.py ## @@ -350,16 +355,57 @@ def fake_pandas_module(self): def summarize(self): super(Be

[GitHub] [beam] codecov[bot] edited a comment on pull request #12819: [BEAM-9561] Initial framework for testing pandas website docs.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12819: URL: https://github.com/apache/beam/pull/12819#issuecomment-691336298 # [Codecov](https://codecov.io/gh/apache/beam/pull/12819?src=pr&el=h1) Report > Merging [#12819](https://codecov.io/gh/apache/beam/pull/12819?src=pr&el=desc) into

[GitHub] [beam] kennknowles commented on a change in pull request #12970: [BEAM-8024] Add dataflow and flink runner to JPMS test

2020-09-29 Thread GitBox
kennknowles commented on a change in pull request #12970: URL: https://github.com/apache/beam/pull/12970#discussion_r497145208 ## File path: sdks/java/testing/jpms-tests/build.gradle ## @@ -31,13 +34,66 @@ enableJavaPerformanceTesting() description = "Apache Beam :: SDKs :: Ja

[GitHub] [beam] robertwb commented on pull request #12873: Remove experimental declarations from fileio.

2020-09-29 Thread GitBox
robertwb commented on pull request #12873: URL: https://github.com/apache/beam/pull/12873#issuecomment-701051368 Thanks. Re-instated the annotation on WriteToFiles. This is an automated message from the Apache Git Service. T

[GitHub] [beam] pabloem merged pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
pabloem merged pull request #12967: URL: https://github.com/apache/beam/pull/12967 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] robertwb merged pull request #12959: [BEAM-10978] Fix bug with map type inference.

2020-09-29 Thread GitBox
robertwb merged pull request #12959: URL: https://github.com/apache/beam/pull/12959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] robertwb commented on pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-09-29 Thread GitBox
robertwb commented on pull request #12971: URL: https://github.com/apache/beam/pull/12971#issuecomment-701048994 R: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

2020-09-29 Thread GitBox
pabloem commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-701049001 @epicfaace LMK if you can take a look at the lint issues This is an automated message from the Apache Git Service

[GitHub] [beam] pabloem commented on pull request #12873: Remove experimental declarations from fileio.

2020-09-29 Thread GitBox
pabloem commented on pull request #12873: URL: https://github.com/apache/beam/pull/12873#issuecomment-701049196 feel free to merge @robertwb except for the change to WriteToFiles all others lgtm This is an automated message

[GitHub] [beam] robertwb opened a new pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-09-29 Thread GitBox
robertwb opened a new pull request #12971: URL: https://github.com/apache/beam/pull/12971 Sizes are directly measured in preceeding stages and estimated from there to avoid expensive fusion breaks due to using the global size as a side input. A cross-partition re-batching step is add

[GitHub] [beam] kileys commented on pull request #12946: [BEAM-10957] Compile Java 11 validates runner jobs with Java 11

2020-09-29 Thread GitBox
kileys commented on pull request #12946: URL: https://github.com/apache/beam/pull/12946#issuecomment-701046289 > I think there is an overarching issue: we want users to be able to use Java 11 authoring their pipelines but still ship Beam's jars that we release re built with Java 8. So the

[GitHub] [beam] boyuanzz commented on pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-29 Thread GitBox
boyuanzz commented on pull request #12572: URL: https://github.com/apache/beam/pull/12572#issuecomment-701043637 > Thanks, it looks fine in general for me. I left several questions, ptal. > > My main concern is the following: > > * Can we have a dataloss in case of failures dur

[GitHub] [beam] codecov[bot] edited a comment on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701034584 # [Codecov](https://codecov.io/gh/apache/beam/pull/12969?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@7c43ab6`). [Click

[GitHub] [beam] boyuanzz commented on a change in pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-29 Thread GitBox
boyuanzz commented on a change in pull request #12572: URL: https://github.com/apache/beam/pull/12572#discussion_r497131119 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java ## @@ -1620,14 +1635,43 @@ public void processElement( Co

[GitHub] [beam] kileys commented on pull request #12970: [BEAM-8024] Add dataflow and flink runner to JPMS test

2020-09-29 Thread GitBox
kileys commented on pull request #12970: URL: https://github.com/apache/beam/pull/12970#issuecomment-701041474 R: @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [beam] TheNeuralBit commented on pull request #12505: [BEAM-8106] Add version to java container image name

2020-09-29 Thread GitBox
TheNeuralBit commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-701039855 FYI: @lostluck 2.26.0 will be the first release with a Java container published at beam_java8_sdk rather than beam_java_sdk @emilymye you're planning on sending a mess

[GitHub] [beam] TheNeuralBit merged pull request #12505: [BEAM-8106] Add version to java container image name

2020-09-29 Thread GitBox
TheNeuralBit merged pull request #12505: URL: https://github.com/apache/beam/pull/12505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

2020-09-29 Thread GitBox
TheNeuralBit commented on a change in pull request #12318: URL: https://github.com/apache/beam/pull/12318#discussion_r497125141 ## File path: sdks/python/apache_beam/runners/portability/stager_test.py ## @@ -167,6 +167,9 @@ def test_no_main_session(self): # xdist adds unpi

[GitHub] [beam] boyuanzz commented on a change in pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-29 Thread GitBox
boyuanzz commented on a change in pull request #12572: URL: https://github.com/apache/beam/pull/12572#discussion_r497125319 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCommitOffset.java ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache So

[GitHub] [beam] ibzib commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-29 Thread GitBox
ibzib commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-701039441 Python precommit failures seem to be unrelated: BEAM-10985 I won't merge until it passes, since it seems the cause of failure is that no unit tests are running. ---

[GitHub] [beam] TheNeuralBit commented on pull request #12505: [BEAM-8106] Add version to java container image name

2020-09-29 Thread GitBox
TheNeuralBit commented on pull request #12505: URL: https://github.com/apache/beam/pull/12505#issuecomment-701039304 Filed a jira for the flake: BEAM-10987 This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] robertwb merged pull request #12961: Fix failed GroupIntoBatchesOverrideTest

2020-09-29 Thread GitBox
robertwb merged pull request #12961: URL: https://github.com/apache/beam/pull/12961 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] codecov[bot] edited a comment on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701034584 # [Codecov](https://codecov.io/gh/apache/beam/pull/12969?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@7c43ab6`). [Click

[GitHub] [beam] codecov[bot] edited a comment on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701034584 # [Codecov](https://codecov.io/gh/apache/beam/pull/12969?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@7c43ab6`). [Click

[GitHub] [beam] kileys opened a new pull request #12970: [BEAM-8024] Add dataflow and flink runner to JPMS test

2020-09-29 Thread GitBox
kileys opened a new pull request #12970: URL: https://github.com/apache/beam/pull/12970 Run JPMS test on dataflow and flink runner as well Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easil

[GitHub] [beam] codecov[bot] commented on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
codecov[bot] commented on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701034584 # [Codecov](https://codecov.io/gh/apache/beam/pull/12969?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@7c43ab6`). [Click here to

[GitHub] [beam] TheNeuralBit commented on pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
TheNeuralBit commented on pull request #12969: URL: https://github.com/apache/beam/pull/12969#issuecomment-701029918 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] TheNeuralBit opened a new pull request #12969: [BEAM-10986] Rollback to shadow 4.0.3

2020-09-29 Thread GitBox
TheNeuralBit opened a new pull request #12969: URL: https://github.com/apache/beam/pull/12969 Partial rollback of https://github.com/apache/beam/pull/12821 Post-Commit Tests Status (on master branch) -

[GitHub] [beam] tvalentyn commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

2020-09-29 Thread GitBox
tvalentyn commented on a change in pull request #12927: URL: https://github.com/apache/beam/pull/12927#discussion_r497096331 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -627,11 +627,19 @@ def write_record(self, writer, value): writer.append(value) +class _

[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr&el=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr&el=desc) into

[GitHub] [beam] kennknowles commented on a change in pull request #12946: [BEAM-10957] Compile Java 11 validates runner jobs with Java 11

2020-09-29 Thread GitBox
kennknowles commented on a change in pull request #12946: URL: https://github.com/apache/beam/pull/12946#discussion_r497091754 ## File path: .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy ## @@ -27,30 +27,17 @@ PostcommitJobBuilder.postCommitJob(

[GitHub] [beam] tvalentyn commented on a change in pull request #12927: Update some of the Python dependencies from Beam Dependency Check Report

2020-09-29 Thread GitBox
tvalentyn commented on a change in pull request #12927: URL: https://github.com/apache/beam/pull/12927#discussion_r497092701 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -627,11 +627,19 @@ def write_record(self, writer, value): writer.append(value) +class _

[GitHub] [beam] kileys commented on pull request #12946: [BEAM-10957] Compile Java 11 validates runner jobs with Java 11

2020-09-29 Thread GitBox
kileys commented on pull request #12946: URL: https://github.com/apache/beam/pull/12946#issuecomment-701021832 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] boyuanzz commented on a change in pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-29 Thread GitBox
boyuanzz commented on a change in pull request #12572: URL: https://github.com/apache/beam/pull/12572#discussion_r497084530 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCommitOffset.java ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache So

[GitHub] [beam] ibzib merged pull request #12907: [BEAM-10953] Add errorprone-slf4j plugin.

2020-09-29 Thread GitBox
ibzib merged pull request #12907: URL: https://github.com/apache/beam/pull/12907 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [beam] TheNeuralBit commented on pull request #12821: Bump versions of protobuf, shadow, other gradle plugins.

2020-09-29 Thread GitBox
TheNeuralBit commented on pull request #12821: URL: https://github.com/apache/beam/pull/12821#issuecomment-701008400 @runningcode Unfortunately it looks like the shadow upgrade is breaking our build. It makes it so `./gradlew :sdks:java:io:expansion-service:build` doesn't build the shade

[GitHub] [beam] ibzib commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-29 Thread GitBox
ibzib commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-701008304 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] codecov[bot] edited a comment on pull request #12959: [BEAM-10978] Fix bug with map type inference.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12959: URL: https://github.com/apache/beam/pull/12959#issuecomment-700340096 # [Codecov](https://codecov.io/gh/apache/beam/pull/12959?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`release-2.25.0@167385b`).

[GitHub] [beam] ibzib commented on pull request #11403: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-09-29 Thread GitBox
ibzib commented on pull request #11403: URL: https://github.com/apache/beam/pull/11403#issuecomment-700991335 hello world This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [beam] rohdesamuel commented on pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
rohdesamuel commented on pull request #12967: URL: https://github.com/apache/beam/pull/12967#issuecomment-700984206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [beam] boyuanzz commented on a change in pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-29 Thread GitBox
boyuanzz commented on a change in pull request #12572: URL: https://github.com/apache/beam/pull/12572#discussion_r497052782 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCommitOffset.java ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache So

[GitHub] [beam] kileys commented on pull request #12946: [BEAM-10957] Compile Java 11 validates runner jobs with Java 11

2020-09-29 Thread GitBox
kileys commented on pull request #12946: URL: https://github.com/apache/beam/pull/12946#issuecomment-700981917 R: @kennknowles, @lukecwik Ran the tests locally, unless you guys can get the seed job working to test the jenkins stuff on this branch Direct runner: https://gradle

[GitHub] [beam] ibzib commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page

2020-09-29 Thread GitBox
ibzib commented on a change in pull request #12963: URL: https://github.com/apache/beam/pull/12963#discussion_r497018651 ## File path: website/www/site/content/en/get-started/from-spark.md ## @@ -0,0 +1,245 @@ +--- +title: "Getting started from Apache Spark" +--- + + +# Getting

[GitHub] [beam] boyuanzz commented on pull request #12572: [BEAM-10123] Add Kafka Commit transform.

2020-09-29 Thread GitBox
boyuanzz commented on pull request #12572: URL: https://github.com/apache/beam/pull/12572#issuecomment-700980738 > Thanks! I did a quick look on this - are you going to add some tests for this addition? Sorry for the late and thanks for your help! Yeah I also want to have some tests

[GitHub] [beam] codecov[bot] edited a comment on pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12967: URL: https://github.com/apache/beam/pull/12967#issuecomment-700964375 # [Codecov](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=h1) Report > Merging [#12967](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=desc) into

[GitHub] [beam] lostluck commented on a change in pull request #12941: [BEAM-9681] Add textio.Read lesson to Go SDK katas

2020-09-29 Thread GitBox
lostluck commented on a change in pull request #12941: URL: https://github.com/apache/beam/pull/12941#discussion_r497031640 ## File path: learning/katas/go/io/textio/read/testdata/path.go ## @@ -0,0 +1,40 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] codecov[bot] edited a comment on pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12967: URL: https://github.com/apache/beam/pull/12967#issuecomment-700964375 # [Codecov](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=h1) Report > Merging [#12967](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=desc) into

[GitHub] [beam] guangstick commented on a change in pull request #12968: [BEAM-7195] Fix BQ BatchLoad not creating new tables issue

2020-09-29 Thread GitBox
guangstick commented on a change in pull request #12968: URL: https://github.com/apache/beam/pull/12968#discussion_r497018910 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java ## @@ -74,7 +74,7 @@ *

[GitHub] [beam] codecov[bot] edited a comment on pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12967: URL: https://github.com/apache/beam/pull/12967#issuecomment-700964375 # [Codecov](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=h1) Report > Merging [#12967](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=desc) into

[GitHub] [beam] guangstick edited a comment on pull request #12968: [BEAM-7195] Fix BQ BatchLoad not creating new tables issue

2020-09-29 Thread GitBox
guangstick edited a comment on pull request #12968: URL: https://github.com/apache/beam/pull/12968#issuecomment-700964600 R: @reuvenlax @chamikaramj Question: will the PTransform renaming cause compatibility issues when users **update** their GCP Dataflow pipelines? --

[GitHub] [beam] guangstick commented on pull request #12968: [BEAM-7195] Fix BQ BatchLoad not creating new tables issue

2020-09-29 Thread GitBox
guangstick commented on pull request #12968: URL: https://github.com/apache/beam/pull/12968#issuecomment-700964600 R: @reuvenlax @chamikaramj Question: will the PTransform renaming cause compatibility issues when users *update* their GCP Dataflow pipelines? ---

[GitHub] [beam] codecov[bot] commented on pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
codecov[bot] commented on pull request #12967: URL: https://github.com/apache/beam/pull/12967#issuecomment-700964375 # [Codecov](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=h1) Report > Merging [#12967](https://codecov.io/gh/apache/beam/pull/12967?src=pr&el=desc) into [master

[GitHub] [beam] codecov[bot] edited a comment on pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12966: URL: https://github.com/apache/beam/pull/12966#issuecomment-700931656 # [Codecov](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=h1) Report > Merging [#12966](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=desc) into

[GitHub] [beam] guangstick commented on a change in pull request #12968: [BEAM-7195] Fix BQ BatchLoad not creating new tables issue

2020-09-29 Thread GitBox
guangstick commented on a change in pull request #12968: URL: https://github.com/apache/beam/pull/12968#discussion_r497018910 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java ## @@ -74,7 +74,7 @@ *

[GitHub] [beam] guangstick opened a new pull request #12968: [BEAM-7195] Fix BQ BatchLoad not creating new tables issue

2020-09-29 Thread GitBox
guangstick opened a new pull request #12968: URL: https://github.com/apache/beam/pull/12968 In WriteTables and WriteRename, the `createDisposition` is set to `CREATE_NEVER` when the pane index is great than 0. https://github.com/apache/beam/blob/ecedd3e654352f1b51ab2caae0fd4665403bd0eb

[GitHub] [beam] rohdesamuel opened a new pull request #12967: [BEAM-10956] Adds duration string to the ib.show and ib.collect

2020-09-29 Thread GitBox
rohdesamuel opened a new pull request #12967: URL: https://github.com/apache/beam/pull/12967 Change-Id: I29df95dc29c67116d7bc81ee999173b5c932afb8 Adds a duration string to ib.show and ib.collect. This still allows for integer seconds and "inf".

[GitHub] [beam] kennknowles merged pull request #12899: [BEAM-8024] Add JPMS E2E test

2020-09-29 Thread GitBox
kennknowles merged pull request #12899: URL: https://github.com/apache/beam/pull/12899 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] codecov[bot] edited a comment on pull request #12959: [BEAM-10978] Fix bug with map type inference.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12959: URL: https://github.com/apache/beam/pull/12959#issuecomment-700340096 # [Codecov](https://codecov.io/gh/apache/beam/pull/12959?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`release-2.25.0@167385b`).

[GitHub] [beam] kennknowles commented on a change in pull request #12899: [BEAM-8024] Add JPMS E2E test

2020-09-29 Thread GitBox
kennknowles commented on a change in pull request #12899: URL: https://github.com/apache/beam/pull/12899#discussion_r497007055 ## File path: sdks/java/testing/jpms-tests/build.gradle ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

[GitHub] [beam] kennknowles commented on pull request #12899: [BEAM-8024] Add JPMS E2E test

2020-09-29 Thread GitBox
kennknowles commented on pull request #12899: URL: https://github.com/apache/beam/pull/12899#issuecomment-700950966 Coming back to my review, I just want to merge: 1. Having this checked in and running against even a single runner is an improvement and leaves the codebase in a consis

[GitHub] [beam] kennknowles commented on pull request #12899: [BEAM-8024] Add JPMS E2E test

2020-09-29 Thread GitBox
kennknowles commented on pull request #12899: URL: https://github.com/apache/beam/pull/12899#issuecomment-700950350 The Jenkins failure is due to Jenkins not finding a test report file. Obviously this doesn't touch Python. T

[GitHub] [beam] codecov[bot] edited a comment on pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12966: URL: https://github.com/apache/beam/pull/12966#issuecomment-700931656 # [Codecov](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=h1) Report > Merging [#12966](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=desc) into

[GitHub] [beam] robertwb commented on pull request #12959: [BEAM-10978] Fix bug with map type inference.

2020-09-29 Thread GitBox
robertwb commented on pull request #12959: URL: https://github.com/apache/beam/pull/12959#issuecomment-700944671 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] codecov[bot] edited a comment on pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12966: URL: https://github.com/apache/beam/pull/12966#issuecomment-700931656 # [Codecov](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=h1) Report > Merging [#12966](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=desc) into

[GitHub] [beam] aaltay commented on pull request #12963: [BEAM-10983] Add getting started from Spark page

2020-09-29 Thread GitBox
aaltay commented on pull request #12963: URL: https://github.com/apache/beam/pull/12963#issuecomment-700939028 @ibzib or @iemejia would be able to review this better. This is an automated message from the Apache Git Service.

[GitHub] [beam] codecov[bot] edited a comment on pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12966: URL: https://github.com/apache/beam/pull/12966#issuecomment-700931656 # [Codecov](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=h1) Report > Merging [#12966](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=desc) into

[GitHub] [beam] kennknowles merged pull request #12947: [Beam-10903] Fix Java nightly snapshot build.

2020-09-29 Thread GitBox
kennknowles merged pull request #12947: URL: https://github.com/apache/beam/pull/12947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] kennknowles commented on a change in pull request #12907: [BEAM-10953] Add errorprone-slf4j plugin.

2020-09-29 Thread GitBox
kennknowles commented on a change in pull request #12907: URL: https://github.com/apache/beam/pull/12907#discussion_r496987502 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -955,6 +955,7 @@ class BeamModulePlugin implements Plugin

[GitHub] [beam] codecov[bot] commented on pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
codecov[bot] commented on pull request #12966: URL: https://github.com/apache/beam/pull/12966#issuecomment-700931656 # [Codecov](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=h1) Report > Merging [#12966](https://codecov.io/gh/apache/beam/pull/12966?src=pr&el=desc) into [master

[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr&el=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr&el=desc) into

[GitHub] [beam] pabloem opened a new pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
pabloem opened a new pull request #12966: URL: https://github.com/apache/beam/pull/12966 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickl

[GitHub] [beam] pabloem commented on pull request #12966: Passing project properly in BQSource

2020-09-29 Thread GitBox
pabloem commented on pull request #12966: URL: https://github.com/apache/beam/pull/12966#issuecomment-700923975 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [beam] vectorijk commented on a change in pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-29 Thread GitBox
vectorijk commented on a change in pull request #12962: URL: https://github.com/apache/beam/pull/12962#discussion_r496964747 ## File path: vendor/calcite-1_25_0/build.gradle ## @@ -18,29 +18,33 @@ plugins { id 'org.apache.beam.vendor-java' } -description = "Apache Beam ::

[GitHub] [beam] vectorijk commented on pull request #12962: [WIP][BEAM-9379] Update calcite to 1.25

2020-09-29 Thread GitBox
vectorijk commented on pull request #12962: URL: https://github.com/apache/beam/pull/12962#issuecomment-700912215 > It might be a huge amount of work. We can just add the new vendored module and then release it, but we will want to know that we have enough people available to also port to

[GitHub] [beam] ibzib commented on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-29 Thread GitBox
ibzib commented on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-700907115 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] codecov[bot] edited a comment on pull request #12576: [BEAM-10671] Add environment configuration fields as first-class pipeline options.

2020-09-29 Thread GitBox
codecov[bot] edited a comment on pull request #12576: URL: https://github.com/apache/beam/pull/12576#issuecomment-692353567 # [Codecov](https://codecov.io/gh/apache/beam/pull/12576?src=pr&el=h1) Report > Merging [#12576](https://codecov.io/gh/apache/beam/pull/12576?src=pr&el=desc) into

  1   2   >