[GitHub] [beam] angoenka commented on pull request #12044: [BEAM-9707] Remove hardcoded unified harness image

2020-06-22 Thread GitBox
angoenka commented on pull request #12044: URL: https://github.com/apache/beam/pull/12044#issuecomment-647948340 Run Python Dataflow V2 ValidatesRunner This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] davidak09 commented on pull request #12050: [BEAM-10292] DefaultFilenamePolicy.ParamsCoder uses ResourceIdCoder

2020-06-22 Thread GitBox
davidak09 commented on pull request #12050: URL: https://github.com/apache/beam/pull/12050#issuecomment-647928655 Thanks for the review @dmvk ! You're right, I forgot to ensure backward compatibility. I tried to implement the requested changes, will you take a look at it please? --

[GitHub] [beam] lukecwik commented on pull request #12055: [BEAM-9272] Remove obsolete portable SDF urns

2020-06-22 Thread GitBox
lukecwik commented on pull request #12055: URL: https://github.com/apache/beam/pull/12055#issuecomment-647912036 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] bamnet commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.

2020-06-22 Thread GitBox
bamnet commented on pull request #12061: URL: https://github.com/apache/beam/pull/12061#issuecomment-647897629 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [beam] bamnet opened a new pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.

2020-06-22 Thread GitBox
bamnet opened a new pull request #12061: URL: https://github.com/apache/beam/pull/12061 Currently, the Go SDK `bigquery` is stuck in LegacySQL mode. This limits the type of queries developers can write the and types of tables they can query. To allow developers to switch to Standard SQL m

[GitHub] [beam] boyuanzz commented on a change in pull request #11749: [BEAM-9977] Implement ReadFromKafkaViaSDF

2020-06-22 Thread GitBox
boyuanzz commented on a change in pull request #11749: URL: https://github.com/apache/beam/pull/11749#discussion_r443937048 ## File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadViaSDF.java ## @@ -0,0 +1,861 @@ +/* + * Licensed to the Apache Software

[GitHub] [beam] lukecwik commented on pull request #12055: [BEAM-9272] Remove obsolete portable SDF urns

2020-06-22 Thread GitBox
lukecwik commented on pull request #12055: URL: https://github.com/apache/beam/pull/12055#issuecomment-647876714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] ihji commented on pull request #12060: [BEAM-10218] Add FnApiRunner to cross-language validate runner test

2020-06-22 Thread GitBox
ihji commented on pull request #12060: URL: https://github.com/apache/beam/pull/12060#issuecomment-647862334 R: @chamikaramj @robertwb This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] ihji opened a new pull request #12060: [BEAM-10218] Add FnApiRunner to cross-language validate runner test

2020-06-22 Thread GitBox
ihji opened a new pull request #12060: URL: https://github.com/apache/beam/pull/12060 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apac

[GitHub] [beam] youngoli commented on pull request #12059: Adding a helper function for making custom coders.

2020-06-22 Thread GitBox
youngoli commented on pull request #12059: URL: https://github.com/apache/beam/pull/12059#issuecomment-647858941 R: @lostluck As an aside, and not in this PR, what do you think about refactoring the locations of all the coder code in the SDK right now. I noticed while trying to fig

[GitHub] [beam] youngoli opened a new pull request #12059: Adding a helper function for making custom coders.

2020-06-22 Thread GitBox
youngoli opened a new pull request #12059: URL: https://github.com/apache/beam/pull/12059 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://b

[GitHub] [beam] robinyqiu commented on a change in pull request #12054: [Work in progress] TIME type

2020-06-22 Thread GitBox
robinyqiu commented on a change in pull request #12054: URL: https://github.com/apache/beam/pull/12054#discussion_r443906131 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java ## @@ -315,7 +315,7 @@ private static

[GitHub] [beam] robinyqiu commented on a change in pull request #12054: [Work in progress] TIME type

2020-06-22 Thread GitBox
robinyqiu commented on a change in pull request #12054: URL: https://github.com/apache/beam/pull/12054#discussion_r443904994 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java ## @@ -315,7 +315,7 @@ private static

[GitHub] [beam] robinyqiu commented on a change in pull request #12054: [Work in progress] TIME type

2020-06-22 Thread GitBox
robinyqiu commented on a change in pull request #12054: URL: https://github.com/apache/beam/pull/12054#discussion_r443904994 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java ## @@ -315,7 +315,7 @@ private static

[GitHub] [beam] aaltay closed pull request #12057: Use bz2 to compress pickler output for better compression.

2020-06-22 Thread GitBox
aaltay closed pull request #12057: URL: https://github.com/apache/beam/pull/12057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] pabloem merged pull request #12058: Format PULL_REQUEST_TEMPLATE.md

2020-06-22 Thread GitBox
pabloem merged pull request #12058: URL: https://github.com/apache/beam/pull/12058 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] pabloem commented on pull request #11850: [BEAM-1438] Allow 0 shards on WriteFiles streaming

2020-06-22 Thread GitBox
pabloem commented on pull request #11850: URL: https://github.com/apache/beam/pull/11850#issuecomment-647835759 @chamikaramj PTAL This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] pabloem opened a new pull request #12058: Format PULL_REQUEST_TEMPLATE.md

2020-06-22 Thread GitBox
pabloem opened a new pull request #12058: URL: https://github.com/apache/beam/pull/12058 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickl

[GitHub] [beam] aaltay opened a new pull request #12057: Use bz2 to compress pickler output for better compression.

2020-06-22 Thread GitBox
aaltay opened a new pull request #12057: URL: https://github.com/apache/beam/pull/12057 bz2 has better compression ration than zlib, but it is slower. (on behalf of @katsiapis) Thank you for your contribution! Follow this checklist to help us in

[GitHub] [beam] jhnmora000 commented on pull request #11975: [BEAM-9198] BeamSQL aggregation analytics functionality

2020-06-22 Thread GitBox
jhnmora000 commented on pull request #11975: URL: https://github.com/apache/beam/pull/11975#issuecomment-647809650 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] lukecwik commented on pull request #12055: [BEAM-9272] Remove obsolete portable SDF urns

2020-06-22 Thread GitBox
lukecwik commented on pull request #12055: URL: https://github.com/apache/beam/pull/12055#issuecomment-647809109 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] tysonjh commented on pull request #11990: [BEAM-9999] Remove Apex runner.

2020-06-22 Thread GitBox
tysonjh commented on pull request #11990: URL: https://github.com/apache/beam/pull/11990#issuecomment-647799401 > Just noticed that we miss the mention in the CHANGES.md file can you please add that @tysonjh thanks. Done in #12056 --

[GitHub] [beam] tysonjh commented on pull request #12056: Add Apex runner deprecation note.

2020-06-22 Thread GitBox
tysonjh commented on pull request #12056: URL: https://github.com/apache/beam/pull/12056#issuecomment-647799181 R: @iemejia This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [beam] tysonjh opened a new pull request #12056: Add Apex runner deprecation note.

2020-06-22 Thread GitBox
tysonjh opened a new pull request #12056: URL: https://github.com/apache/beam/pull/12056 Add Apex runner deprecation note missed in previous commit. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly

[GitHub] [beam] apilloud merged pull request #12041: [BEAM-10286] Fix race in direct runner

2020-06-22 Thread GitBox
apilloud merged pull request #12041: URL: https://github.com/apache/beam/pull/12041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] apilloud commented on a change in pull request #12041: [BEAM-10286] Fix race in direct runner

2020-06-22 Thread GitBox
apilloud commented on a change in pull request #12041: URL: https://github.com/apache/beam/pull/12041#discussion_r443853108 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java ## @@ -201,7 +204,9 @@ priv

[GitHub] [beam] lukecwik commented on pull request #12055: [BEAM-9272] Remove obsolete portable SDF urns

2020-06-22 Thread GitBox
lukecwik commented on pull request #12055: URL: https://github.com/apache/beam/pull/12055#issuecomment-647792541 R: @youngoli CC: @boyuanzz @robertwb This is an automated message from the Apache Git Service. To respond t

[GitHub] [beam] lukecwik opened a new pull request #12055: [BEAM-9272] Remove obsolete portable SDF urns

2020-06-22 Thread GitBox
lukecwik opened a new pull request #12055: URL: https://github.com/apache/beam/pull/12055 This also cleaned up the implementation logic within FnApiDoFnRunner since there isn't a sized and non-sized variant anymore. Thank you for your contribution! Fol

[GitHub] [beam] pabloem commented on pull request #11850: [BEAM-1438] Allow 0 shards on WriteFiles streaming

2020-06-22 Thread GitBox
pabloem commented on pull request #11850: URL: https://github.com/apache/beam/pull/11850#issuecomment-647791219 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] chamikaramj commented on pull request #12053: [BEAM-10301] Ignoring negative estimated sizes when creating Dataflow split responses

2020-06-22 Thread GitBox
chamikaramj commented on pull request #12053: URL: https://github.com/apache/beam/pull/12053#issuecomment-647787098 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] chamikaramj merged pull request #12053: [BEAM-10301] Ignoring negative estimated sizes when creating Dataflow split responses

2020-06-22 Thread GitBox
chamikaramj merged pull request #12053: URL: https://github.com/apache/beam/pull/12053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] iemejia commented on pull request #11405: [BEAM-9747] Deprecate RedisIO.readAll() and add RedisIO.readKeyPatterns as a replacement

2020-06-22 Thread GitBox
iemejia commented on pull request #11405: URL: https://github.com/apache/beam/pull/11405#issuecomment-647782646 Thanks Alexey! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] pabloem commented on a change in pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

2020-06-22 Thread GitBox
pabloem commented on a change in pull request #11582: URL: https://github.com/apache/beam/pull/11582#discussion_r443832544 ## File path: sdks/python/apache_beam/io/iobase.py ## @@ -1529,3 +1530,180 @@ def display_data(self): 'source': DisplayDataItem(self.source.__clas

[GitHub] [beam] ZijieSong946 opened a new pull request #12054: [Work in progress] TIME type

2020-06-22 Thread GitBox
ZijieSong946 opened a new pull request #12054: URL: https://github.com/apache/beam/pull/12054 cc: @robinyqiu Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose r

[GitHub] [beam] tvalentyn commented on pull request #12047: [BEAM-10291] Adding full thread dump upon lull detection

2020-06-22 Thread GitBox
tvalentyn commented on pull request #12047: URL: https://github.com/apache/beam/pull/12047#issuecomment-647765612 Thank you, @davidyan74 ! Does the added codepath have tests coverage? If not, could we add it? Thank you. ---

[GitHub] [beam] angoenka commented on pull request #12044: [BEAM-9707] Remove hardcoded unified harness image

2020-06-22 Thread GitBox
angoenka commented on pull request #12044: URL: https://github.com/apache/beam/pull/12044#issuecomment-647763274 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] pabloem commented on pull request #11850: [BEAM-1438] Allow 0 shards on WriteFiles streaming

2020-06-22 Thread GitBox
pabloem commented on pull request #11850: URL: https://github.com/apache/beam/pull/11850#issuecomment-647754899 Run JavaPortabilityApiJava11 PreCommit This is an automated message from the Apache Git Service. To respond to th

[GitHub] [beam] saavan-google-intern commented on a change in pull request #12009: [BEAM-10258] Support type hint annotations on PTransform's expand()

2020-06-22 Thread GitBox
saavan-google-intern commented on a change in pull request #12009: URL: https://github.com/apache/beam/pull/12009#discussion_r443808200 ## File path: sdks/python/apache_beam/pvalue.py ## @@ -158,6 +158,10 @@ def __ne__(self, other): def __hash__(self): return hash((self

[GitHub] [beam] kennknowles commented on pull request #11977: Remove "whitelist" and "blacklist" terminology from repository where possible

2020-06-22 Thread GitBox
kennknowles commented on pull request #11977: URL: https://github.com/apache/beam/pull/11977#issuecomment-647748753 Finally green! This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] kennknowles merged pull request #11977: Remove "whitelist" and "blacklist" terminology from repository where possible

2020-06-22 Thread GitBox
kennknowles merged pull request #11977: URL: https://github.com/apache/beam/pull/11977 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] kennknowles commented on a change in pull request #12041: [BEAM-10286] Fix race in direct runner

2020-06-22 Thread GitBox
kennknowles commented on a change in pull request #12041: URL: https://github.com/apache/beam/pull/12041#discussion_r443800518 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java ## @@ -201,7 +204,9 @@ p

[GitHub] [beam] lukecwik commented on a change in pull request #11808: [BEAM-10072] Fix RequiresTimeSortedInput for stateless DoFns

2020-06-22 Thread GitBox
lukecwik commented on a change in pull request #11808: URL: https://github.com/apache/beam/pull/11808#discussion_r443789083 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java ## @@ -510,15 +509,13 @@ public RawUnionV

[GitHub] [beam] chamikaramj opened a new pull request #12053: [BEAM-10301] Ignoring negative estimated sizes when creating Dataflow split responses

2020-06-22 Thread GitBox
chamikaramj opened a new pull request #12053: URL: https://github.com/apache/beam/pull/12053 Negative estimate sizes could result in Dataflow backend creating unsplittable sources. Thank you for your contribution! Follow this checklist to help us inco

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443779727 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/AddUuidsTransform.java ## @@ -0,0 +1,50 @@ +

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443779376 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/AddUuidsTransform.java ## @@ -0,0 +1,50 @@ +

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443779242 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/UuidDeduplicationOptions.java ## @@ -0,0 +1,

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443778735 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java ## @@ -0,0 +1,79 @@ +/* +

[GitHub] [beam] pabloem commented on a change in pull request #12035: [BEAM-10220] Add support for REQUIRE_MISSING in RowJsonDeserializer

2020-06-22 Thread GitBox
pabloem commented on a change in pull request #12035: URL: https://github.com/apache/beam/pull/12035#discussion_r443774541 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java ## @@ -159,9 +159,16 @@ public String toString() { private static f

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443755807 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/UuidDeduplicationTransform.java ## @@ -0,0 +

[GitHub] [beam] dhodun opened a new pull request #12052: Update walkthrough.md

2020-06-22 Thread GitBox
dhodun opened a new pull request #12052: URL: https://github.com/apache/beam/pull/12052 Fairly sure .setRowSchema() is required in this case. b/155306887 This is an automated message from the Apache Git Service. To

[GitHub] [beam] je-ik commented on pull request #11808: [BEAM-10072] Fix RequiresTimeSortedInput for stateless DoFns

2020-06-22 Thread GitBox
je-ik commented on pull request #11808: URL: https://github.com/apache/beam/pull/11808#issuecomment-647699712 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] davidyan74 commented on pull request #12047: [BEAM-10291] Adding full thread dump upon lull detection

2020-06-22 Thread GitBox
davidyan74 commented on pull request #12047: URL: https://github.com/apache/beam/pull/12047#issuecomment-647699609 The 2 test failures in Python PreCommit do not seem to be related to the changes in this PR. This is an auto

[GitHub] [beam] je-ik commented on pull request #11808: [BEAM-10072] Fix RequiresTimeSortedInput for stateless DoFns

2020-06-22 Thread GitBox
je-ik commented on pull request #11808: URL: https://github.com/apache/beam/pull/11808#issuecomment-647699486 Run DIrect ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [beam] je-ik commented on pull request #11808: [BEAM-10072] Fix RequiresTimeSortedInput for stateless DoFns

2020-06-22 Thread GitBox
je-ik commented on pull request #11808: URL: https://github.com/apache/beam/pull/11808#issuecomment-647699425 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] y1chi commented on pull request #11916: [BEAM-10189] Add ReadModifyWriteState user state to python sdk

2020-06-22 Thread GitBox
y1chi commented on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-647696671 > It is true that the fn protocol supports one cache token per handler (e.g. user state or side input handler). Those handler do not change for the lifetime of the application. I'm

[GitHub] [beam] kennknowles commented on pull request #11977: Remove "whitelist" and "blacklist" terminology from repository where possible

2020-06-22 Thread GitBox
kennknowles commented on pull request #11977: URL: https://github.com/apache/beam/pull/11977#issuecomment-647693686 Filed https://issues.apache.org/jira/browse/BEAM-10300 for flake/failure at https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/2339/testReport/junit/org.apache.beam.sdk

[GitHub] [beam] mxm commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink

2020-06-22 Thread GitBox
mxm commented on a change in pull request #12030: URL: https://github.com/apache/beam/pull/12030#discussion_r443742088 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy ## @@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [

[GitHub] [beam] lostluck commented on pull request #12024: [BEAM-9679] Add Side Input to Core Transform Go SDK katas

2020-06-22 Thread GitBox
lostluck commented on pull request #12024: URL: https://github.com/apache/beam/pull/12024#issuecomment-647686584 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [beam] lostluck merged pull request #12024: [BEAM-9679] Add Side Input to Core Transform Go SDK katas

2020-06-22 Thread GitBox
lostluck merged pull request #12024: URL: https://github.com/apache/beam/pull/12024 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] mxm commented on pull request #11916: [BEAM-10189] Add ReadModifyWriteState user state to python sdk

2020-06-22 Thread GitBox
mxm commented on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-647685479 It is true that the fn protocol supports one cache token per handler (e.g. user state or side input handler). Those handler do not change for the lifetime of the application. I'm sti

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443725277 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Pl

[GitHub] [beam] dpmills commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpmills commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443711084 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java ## @@ -0,0 +1,79 @@ +/* + * License

[GitHub] [beam] RyanSkraba commented on a change in pull request #11794: [BEAM-9894] Add batch SnowflakeIO.Write to Java SDK

2020-06-22 Thread GitBox
RyanSkraba commented on a change in pull request #11794: URL: https://github.com/apache/beam/pull/11794#discussion_r443687101 ## File path: sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java ## @@ -447,6 +513,346 @@ public void populateDispl

[GitHub] [beam] y1chi commented on pull request #11916: [BEAM-10189] Add ReadModifyWriteState user state to python sdk

2020-06-22 Thread GitBox
y1chi commented on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-647623292 > > @mxm Seems like flink runner is sending cache token per state, instead of one user state token per bundle. This will cause python sdk to fail > > https://github.com/apache/b

[GitHub] [beam] y1chi removed a comment on pull request #11916: [BEAM-10189] Add ReadModifyWriteState user state to python sdk

2020-06-22 Thread GitBox
y1chi removed a comment on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-647616682 > > @mxm Seems like flink runner is sending cache token per state, instead of one user state token per bundle. This will cause python sdk to fail > > https://github.com/

[GitHub] [beam] suztomo commented on pull request #12051: [BEAM-10298] beam-linkage-check.sh should not swallow errors

2020-06-22 Thread GitBox
suztomo commented on pull request #12051: URL: https://github.com/apache/beam/pull/12051#issuecomment-647617102 R: @lukecwik CC: @dpcollins-google This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] y1chi commented on pull request #11916: [BEAM-10189] Add ReadModifyWriteState user state to python sdk

2020-06-22 Thread GitBox
y1chi commented on pull request #11916: URL: https://github.com/apache/beam/pull/11916#issuecomment-647616682 > > @mxm Seems like flink runner is sending cache token per state, instead of one user state token per bundle. This will cause python sdk to fail > > https://github.com/apache/b

[GitHub] [beam] y1chi closed pull request #11916: [BEAM-10189] Add ReadModifyWriteState user state to python sdk

2020-06-22 Thread GitBox
y1chi closed pull request #11916: URL: https://github.com/apache/beam/pull/11916 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [beam] suztomo commented on a change in pull request #12051: [BEAM-10298] beam-linkage-check.sh should not swallow errors

2020-06-22 Thread GitBox
suztomo commented on a change in pull request #12051: URL: https://github.com/apache/beam/pull/12051#discussion_r443662571 ## File path: sdks/java/build-tools/beam-linkage-check.sh ## @@ -72,10 +72,6 @@ function runLinkageCheck () { COMMIT=$1 BRANCH=$2 MODE=$3 # "basel

[GitHub] [beam] suztomo opened a new pull request #12051: [BEAM-10298] beam-linkage-check.sh should not swallow errors

2020-06-22 Thread GitBox
suztomo opened a new pull request #12051: URL: https://github.com/apache/beam/pull/12051 sdks/java/build-tools/beam-linkage-check.sh does not show errors when it fails. It's because it sends errors to dev/null. https://github.com/apache/beam/pull/11919#discussion_r443612255 F

[GitHub] [beam] suztomo commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
suztomo commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443651699 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Plugin {

[GitHub] [beam] aromanenko-dev merged pull request #11405: [BEAM-9747] Deprecate RedisIO.readAll() and add RedisIO.readKeyPatterns as a replacement

2020-06-22 Thread GitBox
aromanenko-dev merged pull request #11405: URL: https://github.com/apache/beam/pull/11405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] suztomo commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
suztomo commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443628609 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Plugin {

[GitHub] [beam] lukecwik commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
lukecwik commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443623528 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Plugin {

[GitHub] [beam] lukecwik commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
lukecwik commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443623528 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Plugin {

[GitHub] [beam] lukecwik commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
lukecwik commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443623528 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Plugin {

[GitHub] [beam] dmvk commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink

2020-06-22 Thread GitBox
dmvk commented on a change in pull request #12030: URL: https://github.com/apache/beam/pull/12030#discussion_r443618694 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy ## @@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443612255 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Pl

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647561544 Run PubsubIO Performance Test Python This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647557852 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443599390 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -480,6 +479,7 @@ class BeamModulePlugin implements Pl

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443598030 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/UuidDeduplicationTransform.java ## @@ -0,0 +

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443596917 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/CloudPubsubTransforms.java ## @@ -0,0 +1,95

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443597419 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/CloudPubsubTransforms.java ## @@ -0,0 +1,95

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443596554 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java ## @@ -0,0 +1,75 @@ +/* +

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443596403 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java ## @@ -0,0 +1,75 @@ +/* +

[GitHub] [beam] mf2199 edited a comment on pull request #11295: [BEAM-3342] CBT `Read` IO Connector Wrapper

2020-06-22 Thread GitBox
mf2199 edited a comment on pull request #11295: URL: https://github.com/apache/beam/pull/11295#issuecomment-647196061 @chamikaramj @aaltay The build errors point to 'missing' arguments that were made so by design, e.g.: ``` 15:41:25 E ValueError: Pipeline has validations errors:

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647537661 Run PubsubIO Performance Test Python This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647534571 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443574688 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/UuidDeduplicationOptions.java ## @@ -0,0 +1,

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647533840 Run PubsubIO Performance Test Python This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] dpcollins-google commented on a change in pull request #11919: [BEAM-10114] Copy Pub/Sub Lite IO from Pub/Sub Lite github to beam.

2020-06-22 Thread GitBox
dpcollins-google commented on a change in pull request #11919: URL: https://github.com/apache/beam/pull/11919#discussion_r443571912 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/AddUuidsTransform.java ## @@ -0,0 +1,50 @@ +

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-64753 Run PubsubIO Performance Test Python This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] davidak09 commented on pull request #12050: [BEAM-10292] DefaultFilenamePolicy.ParamsCoder uses ResourceIdCoder

2020-06-22 Thread GitBox
davidak09 commented on pull request #12050: URL: https://github.com/apache/beam/pull/12050#issuecomment-647530984 R: @dmvk This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] davidak09 opened a new pull request #12050: [BEAM-10292] DefaultFilenamePolicy.ParamsCoder uses ResourceIdCoder

2020-06-22 Thread GitBox
davidak09 opened a new pull request #12050: URL: https://github.com/apache/beam/pull/12050 `DefaultFilenamePolicy.ParamsCoder` used `StringUtf8Coder` for encoding/decoding `baseFilename` and therefore information whether `baseFilename resourceID` is file or directory was lost. Using `Res

[GitHub] [beam] kamilwu commented on a change in pull request #12030: [BEAM-10270] Disable Pardo test with 200 iterations on Flink

2020-06-22 Thread GitBox
kamilwu commented on a change in pull request #12030: URL: https://github.com/apache/beam/pull/12030#discussion_r443569325 ## File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy ## @@ -79,29 +79,31 @@ def batchScenarios = { datasetName -> [

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647529090 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647528157 Run PubsubIO Performance Test Python This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] mwalenia commented on pull request #12048: [BEAM-10258] Fix Dataflow-based Jenkins jobs

2020-06-22 Thread GitBox
mwalenia commented on pull request #12048: URL: https://github.com/apache/beam/pull/12048#issuecomment-647523137 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] damgad commented on pull request #12038: [BEAM-5853] Trigger autodeploy on commit ID and image tag mismatch

2020-06-22 Thread GitBox
damgad commented on pull request #12038: URL: https://github.com/apache/beam/pull/12038#issuecomment-647510854 LGTM :+1: This is an automated message from the Apache Git Service. To respond to the message, please log on to G

  1   2   >