[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery

pabloem commented on pull request #11560: URL: https://github.com/apache/beam/pull/11560#issuecomment-622670110 PAssing precommit: https://builds.apache.org/job/beam_PostCommit_Python37_PR/132/ This is an automated message f

[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery

pabloem commented on pull request #11560: URL: https://github.com/apache/beam/pull/11560#issuecomment-622669830 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [beam] rahul8383 commented on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

rahul8383 commented on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-622661943 I will make the necessary changes as suggested by @TheNeuralBit in https://lists.apache.org/thread.html/r281e2913379c9733f6ac5baa08f361cc4ebe880a9880b2d54d6095b0%40%3Cdev.beam.a

[GitHub] [beam] jaketf commented on a change in pull request #11596: [BEAM-9856] [*WIP DO NOT MERGE*] Optimization/hl7v2 io list messages

jaketf commented on a change in pull request #11596: URL: https://github.com/apache/beam/pull/11596#discussion_r418843954 ## File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOTestUtil.java ## @@ -59,7 +59,7 @@

[GitHub] [beam] jaketf commented on a change in pull request #11596: [BEAM-9856] [*WIP DO NOT MERGE*] Optimization/hl7v2 io list messages

jaketf commented on a change in pull request #11596: URL: https://github.com/apache/beam/pull/11596#discussion_r418843954 ## File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOTestUtil.java ## @@ -59,7 +59,7 @@

[GitHub] [beam] jaketf opened a new pull request #11596: [BEAM-9856] [*WIP DO NOT MERGE*] Optimization/hl7v2 io list messages

jaketf opened a new pull request #11596: URL: https://github.com/apache/beam/pull/11596 This PR is an experiment to illustrate a potential strategy for implementing HL7v2IO.ListMessages as a splittable DoFn that carves the sendTime dimension up into time ranges and uses the Messages.List A

[GitHub] [beam] rahul8383 edited a comment on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

rahul8383 edited a comment on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-622652488 JdbcIO.Read -> SqlTransform.query(SELECT COUNT(*) FROM PCOLLECTION /*Any query*/ ) throws NPE if the input PCollection to SqlTransform has JdbcIO specific Logical Types(

[GitHub] [beam] rahul8383 commented on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

rahul8383 commented on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-622652488 JdbcIO.Read -> SqlTransform.query(SELECT COUNT(*) FROM PCOLLECTION //Any query ) would throw NPE if the input PCollection to SqlTransform JdbcIO specific Logical Types in its S

[GitHub] [beam] reuvenlax commented on pull request #11559: [BEAM-9836] Excluding spark runner for KeyTests

reuvenlax commented on pull request #11559: URL: https://github.com/apache/beam/pull/11559#issuecomment-622649935 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] reuvenlax commented on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

reuvenlax commented on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-622649872 Where do you see the NPE? This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] ibzib commented on a change in pull request #11492: [BEAM-9801] Pass in fire timestamp to timer callback

ibzib commented on a change in pull request #11492: URL: https://github.com/apache/beam/pull/11492#discussion_r418813369 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@ pub

[GitHub] [beam] ibzib commented on pull request #11595: [BEAM-9801] Fire timers set within timers in Spark.

ibzib commented on pull request #11595: URL: https://github.com/apache/beam/pull/11595#issuecomment-622648458 Run Python Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [beam] ibzib opened a new pull request #11595: [BEAM-9801] Fire timers set within timers in Spark.

ibzib opened a new pull request #11595: URL: https://github.com/apache/beam/pull/11595 I refactored a bit first because there were so many nested `for` and `try` blocks it hurt my eyes, in addition to unnecessary abstraction. Probably we should consider fixing up Spark Python validat

[GitHub] [beam] ananvay commented on pull request #11593: [BEAM-9870] Always generate Dataflow-compatible FnApi protos.

ananvay commented on pull request #11593: URL: https://github.com/apache/beam/pull/11593#issuecomment-622631226 Thanks Robert, LGTM. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] ibzib commented on a change in pull request #11593: [BEAM-9870] Always generate Dataflow-compatible FnApi protos.

ibzib commented on a change in pull request #11593: URL: https://github.com/apache/beam/pull/11593#discussion_r418781167 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -345,7 +345,7 @@ def visit_transform(self, transform_node): for ix

[GitHub] [beam] ibzib commented on a change in pull request #11492: [BEAM-9801] Pass in fire timestamp to timer callback

ibzib commented on a change in pull request #11492: URL: https://github.com/apache/beam/pull/11492#discussion_r418780573 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@ pub

[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery

pabloem commented on pull request #11560: URL: https://github.com/apache/beam/pull/11560#issuecomment-622611048 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] lukecwik commented on a change in pull request #11492: [BEAM-9801] Pass in fire timestamp to timer callback

lukecwik commented on a change in pull request #11492: URL: https://github.com/apache/beam/pull/11492#discussion_r418778828 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@

[GitHub] [beam] lukecwik commented on a change in pull request #11492: [BEAM-9801] Pass in fire timestamp to timer callback

lukecwik commented on a change in pull request #11492: URL: https://github.com/apache/beam/pull/11492#discussion_r418778740 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@

[GitHub] [beam] rohdesamuel opened a new pull request #11594: [BEAM-9692] Replace apply_WriteToBigQuery with PTransformOverride

rohdesamuel opened a new pull request #11594: URL: https://github.com/apache/beam/pull/11594 Replace apply_WriteToBigQuery with PTransformOverride Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly a

[GitHub] [beam] robertwb commented on a change in pull request #11593: [BEAM-9870] Always generate Dataflow-compatible FnApi protos.

robertwb commented on a change in pull request #11593: URL: https://github.com/apache/beam/pull/11593#discussion_r418777832 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -345,7 +345,7 @@ def visit_transform(self, transform_node): for

[GitHub] [beam] apilloud commented on a change in pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

apilloud commented on a change in pull request #11272: URL: https://github.com/apache/beam/pull/11272#discussion_r418773897 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/Date.java ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software

[GitHub] [beam] ibzib commented on a change in pull request #11593: [BEAM-9870] Always generate Dataflow-compatible FnApi protos.

ibzib commented on a change in pull request #11593: URL: https://github.com/apache/beam/pull/11593#discussion_r418771952 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -345,7 +345,7 @@ def visit_transform(self, transform_node): for ix

[GitHub] [beam] iemejia commented on pull request #11470: [BEAM-9791] Add precommit for dataflow runner v2

iemejia commented on pull request #11470: URL: https://github.com/apache/beam/pull/11470#issuecomment-622600483 Actually I have the same question, what is dataflow runner 2? The FnApi version? Even if Dataflow is a non open source service it would be convinient for the community to have mo

[GitHub] [beam] ibzib commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide

ibzib commented on a change in pull request #11584: URL: https://github.com/apache/beam/pull/11584#discussion_r418767547 ## File path: website/src/contribute/release-guide.md ## @@ -688,8 +688,20 @@ Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).

[GitHub] [beam] robertwb commented on pull request #11593: [BEAM-9870] Always generate Dataflow-compatible FnApi protos.

robertwb commented on pull request #11593: URL: https://github.com/apache/beam/pull/11593#issuecomment-622596915 R: @ibzib @ananvay CC: @lukecwik This is an automated message from the Apache Git Service. To respond to th

[GitHub] [beam] robertwb opened a new pull request #11593: [BEAM-9870] Always generate Dataflow-compatible FnApi protos.

robertwb opened a new pull request #11593: URL: https://github.com/apache/beam/pull/11593 For various reasons, Dataflow patches up the proto representations of side inputs. This change ensures these mutations are reflected regardless of whether FnAPI was explicitly requested.

[GitHub] [beam] Hannah-Jiang commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide

Hannah-Jiang commented on a change in pull request #11584: URL: https://github.com/apache/beam/pull/11584#discussion_r418766264 ## File path: website/src/contribute/release-guide.md ## @@ -688,8 +688,20 @@ Verify that files are [present](https://dist.apache.org/repos/dist/dev/

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

TheNeuralBit commented on a change in pull request #11272: URL: https://github.com/apache/beam/pull/11272#discussion_r418765847 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/Date.java ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Soft

[GitHub] [beam] Hannah-Jiang commented on pull request #11551: Cherrypick to release-2.21.0

Hannah-Jiang commented on pull request #11551: URL: https://github.com/apache/beam/pull/11551#issuecomment-622595455 > @Hannah-Jiang I'm not sure what, if anything really needs to be cherry-picked here. Java licenses were already cherry-picked in #11421, and the remainder of the changes ar

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

TheNeuralBit commented on a change in pull request #11272: URL: https://github.com/apache/beam/pull/11272#discussion_r418764453 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/Date.java ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Soft

[GitHub] [beam] boyuanzz commented on pull request #11592: [DO NOT REVIEW] Test only

boyuanzz commented on pull request #11592: URL: https://github.com/apache/beam/pull/11592#issuecomment-622592085 Run Dataflow PortabilityApi ValidatesRunner This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] ibzib commented on a change in pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

ibzib commented on a change in pull request #11591: URL: https://github.com/apache/beam/pull/11591#discussion_r418762563 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@ pub

[GitHub] [beam] ibzib commented on a change in pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

ibzib commented on a change in pull request #11591: URL: https://github.com/apache/beam/pull/11591#discussion_r418761275 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@ pub

[GitHub] [beam] ibzib commented on pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

ibzib commented on pull request #11591: URL: https://github.com/apache/beam/pull/11591#issuecomment-622589471 Run Python Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [beam] ibzib commented on pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

ibzib commented on pull request #11591: URL: https://github.com/apache/beam/pull/11591#issuecomment-622589337 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond t

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11272: [BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

TheNeuralBit commented on a change in pull request #11272: URL: https://github.com/apache/beam/pull/11272#discussion_r418759689 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java ## @@ -427,17 +430,12 @@ private st

[GitHub] [beam] lukecwik commented on a change in pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

lukecwik commented on a change in pull request #11591: URL: https://github.com/apache/beam/pull/11591#discussion_r418756561 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@

[GitHub] [beam] lukecwik commented on a change in pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

lukecwik commented on a change in pull request #11591: URL: https://github.com/apache/beam/pull/11591#discussion_r418756561 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java ## @@ -247,25 +247,27 @@

[GitHub] [beam] boyuanzz commented on pull request #11592: [DO NOT REVIEW] Test only

boyuanzz commented on pull request #11592: URL: https://github.com/apache/beam/pull/11592#issuecomment-622583832 Run Dataflow PortabilityApi ValidatesRunner This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] boyuanzz opened a new pull request #11592: [DO NOT REVIEW] Test only

boyuanzz opened a new pull request #11592: URL: https://github.com/apache/beam/pull/11592 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quick

[GitHub] [beam] ibzib opened a new pull request #11591: [BEAM-9801] [cherry-pick] Pass in fire timestamp to timer callback

ibzib opened a new pull request #11591: URL: https://github.com/apache/beam/pull/11591 I had to resolve a minor merge conflict between 325e0f1 and 8de324f22ca04b3716abf58ba77c2a3c117263a2. Thank you for your contribution! Follow this checklist to help

[GitHub] [beam] allenpradeep commented on pull request #11570: [BEAM-9822] Merge the stages 'Gather and Sort' and 'Create Batches'

allenpradeep commented on pull request #11570: URL: https://github.com/apache/beam/pull/11570#issuecomment-622580239 This is great niel. With these changes, there are 3 modes of using SpannerIO write. a) Use the conventional way(as it was till now) with a grouping factor where data is

[GitHub] [beam] ibzib commented on pull request #11492: [BEAM-9801] Pass in fire timestamp to timer callback

ibzib commented on pull request #11492: URL: https://github.com/apache/beam/pull/11492#issuecomment-622576973 Flakes are BEAM-9767 and BEAM-8912. I'm going to merge this so we can go ahead with the release. This is an automa

[GitHub] [beam] ibzib commented on pull request #11585: [BEAM-9860] Make job_endpoint required for PortableRunner

ibzib commented on pull request #11585: URL: https://github.com/apache/beam/pull/11585#issuecomment-622575793 Test flake: BEAM-9767 This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] lukecwik opened a new pull request #11590: [BEAM-8944] Improve UnboundedThreadPoolExecutor performance

lukecwik opened a new pull request #11590: URL: https://github.com/apache/beam/pull/11590 Existing performance suffered because of the use of timed waits and also due to the increase in number of "threading" objects being invoked. Using the benchmark from https://issues.apache.org/j

[GitHub] [beam] lukecwik commented on pull request #11590: [BEAM-8944] Improve UnboundedThreadPoolExecutor performance

lukecwik commented on pull request #11590: URL: https://github.com/apache/beam/pull/11590#issuecomment-62261 R: @mxm @pabloem CC: @robertwb This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11477: [BEAM-9650] Add PeriodicSequence generator.

TheNeuralBit commented on a change in pull request #11477: URL: https://github.com/apache/beam/pull/11477#discussion_r418724351 ## File path: examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java ## @@ -785,4 +787,73 @@ public static void main(String[] ar

[GitHub] [beam] TheNeuralBit commented on a change in pull request #10384: [BEAM-8933] Utilities for converting Arrow schemas and reading Arrow batches as Rows

TheNeuralBit commented on a change in pull request #10384: URL: https://github.com/apache/beam/pull/10384#discussion_r418716155 ## File path: sdks/java/extensions/arrow/src/main/java/org/apache/beam/sdk/extensions/arrow/ArrowConversion.java ## @@ -0,0 +1,448 @@ +/* + * License

[GitHub] [beam] allenpradeep commented on a change in pull request #11532: [BEAM-9822] Disable grouping when streaming

allenpradeep commented on a change in pull request #11532: URL: https://github.com/apache/beam/pull/11532#discussion_r418711729 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java ## @@ -1066,7 +1079,12 @@ public Spa

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11456: [BEAM-7554] Add MillisInstant logical type to replace DATETIME

TheNeuralBit commented on a change in pull request #11456: URL: https://github.com/apache/beam/pull/11456#discussion_r418708682 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/MillisInstant.java ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Ap

[GitHub] [beam] robertwb commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform

robertwb commented on pull request #11575: URL: https://github.com/apache/beam/pull/11575#issuecomment-622533616 The change looks good to me, once all its prerequisites get in. This is an automated message from the Apache Gi

[GitHub] [beam] robertwb commented on a change in pull request #11452: [BEAM-9692] Move apply_Read to PTransformOverride

robertwb commented on a change in pull request #11452: URL: https://github.com/apache/beam/pull/11452#discussion_r418694513 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -117,13 +117,15 @@ class DataflowRunner(PipelineRunner): # TODO: Remove

[GitHub] [beam] ibzib commented on pull request #11403: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

ibzib commented on pull request #11403: URL: https://github.com/apache/beam/pull/11403#issuecomment-622523122 Run Dataflow PortabilityApi ValidatesRunner This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] allenpradeep commented on pull request #11528: [BEAM-9821] Populate all SpannerIO batching parameters in display data.

allenpradeep commented on pull request #11528: URL: https://github.com/apache/beam/pull/11528#issuecomment-622521247 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] ibzib commented on pull request #11551: Cherrypick to release-2.21.0

ibzib commented on pull request #11551: URL: https://github.com/apache/beam/pull/11551#issuecomment-622517286 @Hannah-Jiang I'm not sure what, if anything really needs to be cherry-picked here. Java licenses were already cherry-picked in #11421, and the remainder of the changes are just us

[GitHub] [beam] Ardagan commented on pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

Ardagan commented on pull request #11582: URL: https://github.com/apache/beam/pull/11582#issuecomment-622514637 @rezarokni FYI This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] TheNeuralBit commented on pull request #11589: WIP: [BEAM-9623] Support for SQL TableProviders in Python SqlTransform

TheNeuralBit commented on pull request #11589: URL: https://github.com/apache/beam/pull/11589#issuecomment-622511334 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] TheNeuralBit opened a new pull request #11589: WIP: [BEAM-9623] Support for SQL TableProviders in Python SqlTransform

TheNeuralBit opened a new pull request #11589: URL: https://github.com/apache/beam/pull/11589 Currently this just hard-codes support for DataCatalogTableProvider. We should instead make it possible to specify and configure HCatalog instances as well. Post-Commit Tests Status (on mas

[GitHub] [beam] Ardagan commented on pull request #11582: [BEAM-9650] Add ReadAllFromBigQuery PTransform

Ardagan commented on pull request #11582: URL: https://github.com/apache/beam/pull/11582#issuecomment-622510185 > This is looking good. Something I think we should do is change the underlying implementation of ReadFromBigQuery This would be a bigger change than what I'd like to put i

[GitHub] [beam] Ardagan commented on a change in pull request #11477: [BEAM-9650] Add PeriodicSequence generator.

Ardagan commented on a change in pull request #11477: URL: https://github.com/apache/beam/pull/11477#discussion_r418673674 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PeriodicSequence.java ## @@ -21,33 +21,69 @@ import static org.apache.beam.ven

[GitHub] [beam] robertwb commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment

robertwb commented on a change in pull request #11039: URL: https://github.com/apache/beam/pull/11039#discussion_r418651280 ## File path: model/pipeline/src/main/proto/beam_runner_api.proto ## @@ -1271,6 +1271,11 @@ message DeferredArtifactPayload { message ArtifactStagingToRo

[GitHub] [beam] ibzib commented on pull request #11585: [BEAM-9860] Make job_endpoint required for PortableRunner

ibzib commented on pull request #11585: URL: https://github.com/apache/beam/pull/11585#issuecomment-622500284 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [beam] tvalentyn commented on pull request #11470: [BEAM-9791] Add precommit for dataflow runner v2

tvalentyn commented on pull request #11470: URL: https://github.com/apache/beam/pull/11470#issuecomment-622495831 nvm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [beam] tvalentyn commented on pull request #11470: [BEAM-9791] Add precommit for dataflow runner v2

tvalentyn commented on pull request #11470: URL: https://github.com/apache/beam/pull/11470#issuecomment-622494788 What is Dataflow runner V2? There are no details here or in the JIRA. This is an automated message from the Ap

[GitHub] [beam] chamikaramj commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

chamikaramj commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-622489105 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-622489303 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] chamikaramj commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

chamikaramj commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-622489032 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] mxm commented on pull request #11492: [BEAM-9801] Pass in fire timestamp to timer callback

mxm commented on pull request #11492: URL: https://github.com/apache/beam/pull/11492#issuecomment-622488737 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] TheNeuralBit commented on a change in pull request #10384: [BEAM-8933] Utilities for converting Arrow schemas and reading Arrow batches as Rows

TheNeuralBit commented on a change in pull request #10384: URL: https://github.com/apache/beam/pull/10384#discussion_r418649632 ## File path: sdks/java/extensions/arrow/src/main/java/org/apache/beam/sdk/extensions/arrow/ArrowConversion.java ## @@ -0,0 +1,448 @@ +/* + * License

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-622481501 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-622480730 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [beam] tvalentyn commented on pull request #11444: [BEAM-9701] loosen fastavro version restriction for Python 3.8 support

tvalentyn commented on pull request #11444: URL: https://github.com/apache/beam/pull/11444#issuecomment-622469723 You would need to install beam from HEAD though, since those changes are not yet included in released versions. -

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11456: [BEAM-7554] Add MillisInstant logical type to replace DATETIME

TheNeuralBit commented on a change in pull request #11456: URL: https://github.com/apache/beam/pull/11456#discussion_r418628471 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java ## @@ -257,10 +257,8 @@ priv

[GitHub] [beam] TheNeuralBit commented on a change in pull request #11456: [BEAM-7554] Add MillisInstant logical type to replace DATETIME

TheNeuralBit commented on a change in pull request #11456: URL: https://github.com/apache/beam/pull/11456#discussion_r418625619 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java ## @@ -419,7 +420,6 @@ public int hashCode() { FLOAT, DOU

[GitHub] [beam] tvalentyn commented on a change in pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

tvalentyn commented on a change in pull request #11086: URL: https://github.com/apache/beam/pull/11086#discussion_r418609817 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -45,8 +45,8 @@ may use some caching techniques to share the side inputs between calls in

[GitHub] [beam] tvalentyn commented on pull request #11086: [BEAM-8910] Make custom BQ source read from Avro

tvalentyn commented on pull request #11086: URL: https://github.com/apache/beam/pull/11086#issuecomment-622455437 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] mxm commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

mxm commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-622450697 > Good point about needing to remove Kafka, etc. for Flink. (Not sure how that would interact with its use via an embedded environment; I'll let that be a later PR.) I think t

[GitHub] [beam] mxm commented on a change in pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

mxm commented on a change in pull request #11557: URL: https://github.com/apache/beam/pull/11557#discussion_r418609854 ## File path: sdks/python/apache_beam/pipeline.py ## @@ -211,6 +211,8 @@ def __init__(self, runner=None, options=None, argv=None): experiments.append(

[GitHub] [beam] suztomo commented on pull request #11586: [BEAM-9444] GCP Library BOM as a Map

suztomo commented on pull request #11586: URL: https://github.com/apache/beam/pull/11586#issuecomment-622447857 @iemejia Thank you. This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] tvalentyn commented on pull request #11444: [BEAM-9701] loosen fastavro version restriction for Python 3.8 support

tvalentyn commented on pull request #11444: URL: https://github.com/apache/beam/pull/11444#issuecomment-622447887 @vishalaj1, yes, please try again. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] robertwb commented on a change in pull request #11554: Website - Migrated Jekyll to Hugo

robertwb commented on a change in pull request #11554: URL: https://github.com/apache/beam/pull/11554#discussion_r418604535 ## File path: website/www/site/content/en/blog/beam-2.19.0.md ## @@ -25,7 +24,7 @@ limitations under the License. --> We are happy to present the new

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-622442663 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] robertwb commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

robertwb commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-622441649 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] robertwb commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

robertwb commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-622441715 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] vishalaj1 commented on pull request #11444: [BEAM-9701] loosen fastavro version restriction for Python 3.8 support

vishalaj1 commented on pull request #11444: URL: https://github.com/apache/beam/pull/11444#issuecomment-622430319 Hi, I had this problem when i tried installing 3 days back. https://github.com/fastavro/fastavro/issues/427 Can you please let me know if this is fixed as we

[GitHub] [beam] kennknowles commented on pull request #11588: [BEAM-9776] Fixes filesystem not found error

kennknowles commented on pull request #11588: URL: https://github.com/apache/beam/pull/11588#issuecomment-622427418 @apilloud any pitfalls here? This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [beam] sjvanrossum opened a new pull request #11588: [BEAM-9776] Fixes filesystem not found error

sjvanrossum opened a new pull request #11588: URL: https://github.com/apache/beam/pull/11588 Registers standard filesystems. This fixes the linked JIRA issue, which occurs if a virtual filesystem table is queried before a local filesystem table is queried. This assumes that Beam SQL

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-622401634 Run Python Load Tests ParDo Flink Streaming This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] mxm commented on pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks

mxm commented on pull request #11558: URL: https://github.com/apache/beam/pull/11558#issuecomment-622401502 I've created a new dashboard here includes data from the batch tests and the two new streaming tests: https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056

[GitHub] [beam] suztomo commented on pull request #11586: [BEAM-9444] GCP Library BOM as a Map

suztomo commented on pull request #11586: URL: https://github.com/apache/beam/pull/11586#issuecomment-622398262 Java precommit check failed on org.apache.beam.runners.flink.FlinkSavepointTest.testSavepointRestorePortable “test timed out after 60 seconds“. This seems Transient failure.

[GitHub] [beam] iemejia commented on pull request #11586: [BEAM-9444] GCP Library BOM as a Map

iemejia commented on pull request #11586: URL: https://github.com/apache/beam/pull/11586#issuecomment-622362982 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] iemejia commented on pull request #11586: [BEAM-9444] GCP Library BOM as a Map

iemejia commented on pull request #11586: URL: https://github.com/apache/beam/pull/11586#issuecomment-622362502 retest this please This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] nielm commented on pull request #11438: [BEAM-9505] Remove spurious error message in SpannerIO when streaming.

nielm commented on pull request #11438: URL: https://github.com/apache/beam/pull/11438#issuecomment-622348150 @chamikaramj can you merge please :) This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [beam] mxm commented on a change in pull request #11578: [BEAM-8025] Increase the number of retrials en retrial delay in case of load in CassandraIOTest

mxm commented on a change in pull request #11578: URL: https://github.com/apache/beam/pull/11578#discussion_r418487133 ## File path: sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java ## @@ -153,18 +153,22 @@ public static void beforeCla

[GitHub] [beam] rahul8383 commented on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

rahul8383 commented on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-622308296 I agree that this is not a scalable solution. Providing a Calcite RelDataType Mapping for every Logical Type defined(which is the solution presented in this PR) by every IO is

[GitHub] [beam] robertwb commented on pull request #11557: [BEAM-9845] Stage artifacts over expansion service.

robertwb commented on pull request #11557: URL: https://github.com/apache/beam/pull/11557#issuecomment-622304615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] henryken commented on a change in pull request #11564: [Beam-9679] Add Core Transforms section / Map lesson to the Go SDK katas

henryken commented on a change in pull request #11564: URL: https://github.com/apache/beam/pull/11564#discussion_r418443043 ## File path: learning/katas/go/Core Transforms/Map/ParDo/pkg/task/task.go ## @@ -18,8 +18,9 @@ package task import "github.com/apache/beam/sdks/go/pkg/b

[GitHub] [beam] reuvenlax commented on a change in pull request #11456: [BEAM-7554] Add MillisInstant logical type to replace DATETIME

reuvenlax commented on a change in pull request #11456: URL: https://github.com/apache/beam/pull/11456#discussion_r418443258 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java ## @@ -257,10 +257,8 @@ private

[GitHub] [beam] rahul8383 commented on pull request #11581: [BEAM-8307] NPE in Calcite dialect when input PCollection has logical…

rahul8383 commented on pull request #11581: URL: https://github.com/apache/beam/pull/11581#issuecomment-622277282 @reuvenlax I have provided the source table schema and attached the NPE that I have faced in [BEAM-8307](https://issues.apache.org/jira/browse/BEAM-8307) NPE is throw

  1   2   >