[GitHub] [beam] bntnam commented on pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

bntnam commented on pull request #13406: URL: https://github.com/apache/beam/pull/13406#issuecomment-732687624 Added comments and resolved conflicts This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] bntnam removed a comment on pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

bntnam removed a comment on pull request #13406: URL: https://github.com/apache/beam/pull/13406#issuecomment-732653238 Added comments and resolved conflicts! This is an automated message from the Apache Git Service. To respon

[GitHub] [beam] bntnam commented on pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

bntnam commented on pull request #13406: URL: https://github.com/apache/beam/pull/13406#issuecomment-732653238 Added comments and resolved conflicts! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] bntnam commented on a change in pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

bntnam commented on a change in pull request #13406: URL: https://github.com/apache/beam/pull/13406#discussion_r529202882 ## File path: website/www/site/i18n/home/calendar/en.yaml ## @@ -0,0 +1,42 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may

[GitHub] [beam] bntnam commented on a change in pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

bntnam commented on a change in pull request #13406: URL: https://github.com/apache/beam/pull/13406#discussion_r529199295 ## File path: website/www/site/i18n/home/calendar/en.yaml ## @@ -0,0 +1,42 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may

[GitHub] [beam] xinbinhuang edited a comment on pull request #13239: Update doc/examples: BigQuerySource to ReadFromBigQuery

xinbinhuang edited a comment on pull request #13239: URL: https://github.com/apache/beam/pull/13239#issuecomment-732640055 > hi @xinbinhuang ! Thanks a lot for the contribution. I would be super grateful if you would fix all the user-facing documentation-type spots where we need to update

[GitHub] [beam] xinbinhuang edited a comment on pull request #13239: Update doc: BigQuerySource to ReadFromBigQuery

xinbinhuang edited a comment on pull request #13239: URL: https://github.com/apache/beam/pull/13239#issuecomment-732640055 > hi @xinbinhuang ! Thanks a lot for the contribution. I would be super grateful if you would fix all the user-facing documentation-type spots where we need to update

[GitHub] [beam] xinbinhuang commented on pull request #13239: Update doc: BigQuerySource to ReadFromBigQuery

xinbinhuang commented on pull request #13239: URL: https://github.com/apache/beam/pull/13239#issuecomment-732640055 > hi @xinbinhuang ! Thanks a lot for the contribution. I would be super grateful if you would fix all the user-facing documentation-type spots where we need to update - thoug

[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

aaltay commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-732610284 > > > BeamAssertException > > > > > > This one looks like a test on the asserts. > > The actual error (which is likely a flake) is: > > logs: [#12779 (checks)](htt

[GitHub] [beam] robinyqiu merged pull request #13381: [BEAM-10438] Support ZetaSQL IN operator

robinyqiu merged pull request #13381: URL: https://github.com/apache/beam/pull/13381 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit commented on pull request #13401: [BEAM-11324] Add additional verification in PartitioningSession

TheNeuralBit commented on pull request #13401: URL: https://github.com/apache/beam/pull/13401#issuecomment-732581287 It looks like I've introduced some flakiness.. I'll have to take a look in the morning though This is an au

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13410: [BEAM-11327] Replace Charset.defaultCharset() with StandardCharsets.UTF_8

TheNeuralBit commented on a change in pull request #13410: URL: https://github.com/apache/beam/pull/13410#discussion_r529153011 ## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactory.jav

[GitHub] [beam] tvalentyn merged pull request #13393: [BEAM-10593] Add Jenkins job for creating regular snapshots of the Beam SDK Harness Container Images

tvalentyn merged pull request #13393: URL: https://github.com/apache/beam/pull/13393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tvalentyn commented on a change in pull request #13393: [BEAM-10593] Add Jenkins job for creating regular snapshots of the Beam SDK Harness Container Images

tvalentyn commented on a change in pull request #13393: URL: https://github.com/apache/beam/pull/13393#discussion_r529135483 ## File path: .test-infra/jenkins/PythonTestProperties.groovy ## @@ -17,6 +17,7 @@ */ class PythonTestProperties { + final static List SUPPORTED_CO

[GitHub] [beam] tvalentyn commented on pull request #13330: [BEAM-9980] do not hardcode Python version for dataflow validate runner tests and make it switchable

tvalentyn commented on pull request #13330: URL: https://github.com/apache/beam/pull/13330#issuecomment-732535644 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] KevinGG commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

KevinGG commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-732535245 > > BeamAssertException > > This one looks like a test on the asserts. > > The actual error (which is likely a flake) is: > > logs: https://github.com/apache/

[GitHub] [beam] TheNeuralBit commented on pull request #13401: Add additional verification in PartitioningSession

TheNeuralBit commented on pull request #13401: URL: https://github.com/apache/beam/pull/13401#issuecomment-732532147 > as it seems this hasn't exposed bugs outside of verifying preservation of Singleton, right? There was one other case. The `map_index` expression used when grouping b

[GitHub] [beam] omarismail94 edited a comment on pull request #13410: [BEAM-11327] Replace Charset.defaultCharset() with StandardCharsets.UTF_8

omarismail94 edited a comment on pull request #13410: URL: https://github.com/apache/beam/pull/13410#issuecomment-732527680 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] omarismail94 commented on a change in pull request #12837: [BEAM-10891] Dockerfile for development container

omarismail94 commented on a change in pull request #12837: URL: https://github.com/apache/beam/pull/12837#discussion_r529110761 ## File path: Dockerfile ## @@ -0,0 +1,80 @@ +### +# Licensed to the Apac

[GitHub] [beam] omarismail94 commented on pull request #13410: [BEAM-11327] Replace Charset.defaultCharset() with StandardCharsets.UTF_8

omarismail94 commented on pull request #13410: URL: https://github.com/apache/beam/pull/13410#issuecomment-732527680 R: @TheNeuralBit @nielsbasjes This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13401: Add additional verification in PartitioningSession

TheNeuralBit commented on a change in pull request #13401: URL: https://github.com/apache/beam/pull/13401#discussion_r529106679 ## File path: sdks/python/apache_beam/dataframe/frames.py ## @@ -1401,17 +1403,20 @@ def replace(self, limit, **kwargs): def reset_index(self, leve

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13401: Add additional verification in PartitioningSession

TheNeuralBit commented on a change in pull request #13401: URL: https://github.com/apache/beam/pull/13401#discussion_r529106077 ## File path: sdks/python/apache_beam/dataframe/expressions.py ## @@ -48,10 +48,15 @@ def lookup(self, expr): # type: (Expression) -> Any class Par

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13401: Add additional verification in PartitioningSession

TheNeuralBit commented on a change in pull request #13401: URL: https://github.com/apache/beam/pull/13401#discussion_r529105654 ## File path: sdks/python/apache_beam/dataframe/partitionings.py ## @@ -115,6 +115,23 @@ def partition_fn(self, df, num_partitions): for key in r

[GitHub] [beam] omarismail94 opened a new pull request #13410: [BEAM-11327] Replace Charset.defaultCharset() with StandardCharsets.UTF_8

omarismail94 opened a new pull request #13410: URL: https://github.com/apache/beam/pull/13410 **Please** add a meaningful description for your change here Replace Charset.defaultCharset() with StandardCharsets.UTF_8 so that reliance of encoding set in `locale` is not used. The

[GitHub] [beam] tvalentyn commented on pull request #13390: [DO NOT MERGE] Clone of pr-13330

tvalentyn commented on pull request #13390: URL: https://github.com/apache/beam/pull/13390#issuecomment-732517547 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] boyuanzz commented on pull request #13396: Add use_unified_worker to pipeline options.

boyuanzz commented on pull request #13396: URL: https://github.com/apache/beam/pull/13396#issuecomment-732516770 Thanks for the quick review! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] aaltay merged pull request #13396: Add use_unified_worker to pipeline options.

aaltay merged pull request #13396: URL: https://github.com/apache/beam/pull/13396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] aaltay merged pull request #13350: [BEAM-11266] Python IO MongoDB: add bucket_auto aggregation option for bundling in Atlas.

aaltay merged pull request #13350: URL: https://github.com/apache/beam/pull/13350 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] tvalentyn commented on pull request #13390: [DO NOT MERGE] Clone of pr-13330

tvalentyn commented on pull request #13390: URL: https://github.com/apache/beam/pull/13390#issuecomment-732511858 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] tvalentyn closed pull request #13397: DO NOT MERGE - Clone of [pr-13393] add Jenkins auto-job for creating regular snapshots of the Beam SDKs

tvalentyn closed pull request #13397: URL: https://github.com/apache/beam/pull/13397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732511621 I agree with @omarismail94's assessment This is an automated message from the Apache Git Service. To respon

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12837: [BEAM-10891] Dockerfile for development container

TheNeuralBit commented on a change in pull request #12837: URL: https://github.com/apache/beam/pull/12837#discussion_r529090102 ## File path: Dockerfile ## @@ -0,0 +1,80 @@ +### +# Licensed to the Apac

[GitHub] [beam] tvalentyn commented on a change in pull request #13382: [BEAM-9980] do not use hardcoded python versions in validate container dataflow jobs

tvalentyn commented on a change in pull request #13382: URL: https://github.com/apache/beam/pull/13382#discussion_r529087404 ## File path: .test-infra/jenkins/PythonTestProperties.groovy ## @@ -23,8 +23,14 @@ class PythonTestProperties { LOWEST_SUPPORTED, HIGHEST_SUPP

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

TheNeuralBit commented on a change in pull request #13406: URL: https://github.com/apache/beam/pull/13406#discussion_r529086949 ## File path: website/www/site/i18n/home/calendar/en.yaml ## @@ -0,0 +1,42 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# yo

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13406: [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components

TheNeuralBit commented on a change in pull request #13406: URL: https://github.com/apache/beam/pull/13406#discussion_r529086949 ## File path: website/www/site/i18n/home/calendar/en.yaml ## @@ -0,0 +1,42 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# yo

[GitHub] [beam] robertwb commented on a change in pull request #13401: Add additional verification in PartitioningSession

robertwb commented on a change in pull request #13401: URL: https://github.com/apache/beam/pull/13401#discussion_r529082689 ## File path: sdks/python/apache_beam/dataframe/expressions.py ## @@ -48,10 +48,15 @@ def lookup(self, expr): # type: (Expression) -> Any class Partiti

[GitHub] [beam] tvalentyn commented on a change in pull request #13393: [BEAM-10593] Add Jenkins job for creating regular snapshots of the Beam SDKs

tvalentyn commented on a change in pull request #13393: URL: https://github.com/apache/beam/pull/13393#discussion_r529082732 ## File path: .test-infra/jenkins/job_Publish_Docker_Snapshots.groovy ## @@ -33,15 +34,15 @@ job('beam_Publish_Docker_Snapshots') { ) // Runs

[GitHub] [beam] robertwb commented on pull request #13401: Add additional verification in PartitioningSession

robertwb commented on pull request #13401: URL: https://github.com/apache/beam/pull/13401#issuecomment-732503837 Fair point. In fact, every operation preserves indexing by singleton, because when you apply to the whole input you get the whole output. But `df` being logically partitioned by

[GitHub] [beam] emilymye commented on a change in pull request #13393: [BEAM-10593] Add Jenkins job for creating regular snapshots of the Beam SDKs

emilymye commented on a change in pull request #13393: URL: https://github.com/apache/beam/pull/13393#discussion_r529081789 ## File path: .test-infra/jenkins/job_Publish_Docker_Snapshots.groovy ## @@ -33,15 +34,15 @@ job('beam_Publish_Docker_Snapshots') { ) // Runs

[GitHub] [beam] apilloud commented on a change in pull request #13377: [BEAM-10896] Support ZetaSQL UNNEST syntax on array of struct type

apilloud commented on a change in pull request #13377: URL: https://github.com/apache/beam/pull/13377#discussion_r529080156 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUnnest.java ## @@ -0,0 +1,140 @@ +/* + * L

[GitHub] [beam] apilloud commented on a change in pull request #13377: [BEAM-10896] Support ZetaSQL UNNEST syntax on array of struct type

apilloud commented on a change in pull request #13377: URL: https://github.com/apache/beam/pull/13377#discussion_r529080156 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUnnest.java ## @@ -0,0 +1,140 @@ +/* + * L

[GitHub] [beam] apilloud commented on a change in pull request #13377: [BEAM-10896] Support ZetaSQL UNNEST syntax on array of struct type

apilloud commented on a change in pull request #13377: URL: https://github.com/apache/beam/pull/13377#discussion_r529063802 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUnnest.java ## @@ -0,0 +1,140 @@ +/* + * L

[GitHub] [beam] tvalentyn commented on a change in pull request #13393: [BEAM-10593] Add Jenkins job for creating regular snapshots of the Beam SDKs

tvalentyn commented on a change in pull request #13393: URL: https://github.com/apache/beam/pull/13393#discussion_r529073679 ## File path: .test-infra/jenkins/job_Publish_Docker_Snapshots.groovy ## @@ -33,15 +34,15 @@ job('beam_Publish_Docker_Snapshots') { ) // Runs

[GitHub] [beam] omarismail94 edited a comment on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

omarismail94 edited a comment on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732486982 @nielsbasjes I am not familiar with how this works, and would love your thoughts on this: In the status quo (without docker containers), if you have multiple c

[GitHub] [beam] omarismail94 commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

omarismail94 commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732486982 @nielsbasjes I am not familiar with how this works, and would love your thoughts on this: In the status quo (without docker containers), if you have multiple copies

[GitHub] [beam] kennknowles merged pull request #13374: [BEAM-8106] Add Java 11 container to release and Java 11+V2+Dataflow testing

kennknowles merged pull request #13374: URL: https://github.com/apache/beam/pull/13374 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] boyuanzz merged pull request #13338: [BEAM-11070] Use self-checkpoint to enforce finalization happens.

boyuanzz merged pull request #13338: URL: https://github.com/apache/beam/pull/13338 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] boyuanzz commented on pull request #13338: [BEAM-11070] Use self-checkpoint to enforce finalization happens.

boyuanzz commented on pull request #13338: URL: https://github.com/apache/beam/pull/13338#issuecomment-732475522 Thanks for the quick review! I'm going to merge this PR. This is an automated message from the Apache Git Servic

[GitHub] [beam] nielsbasjes commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

nielsbasjes commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732470396 I'm in doubt about something. Right now I put the `~/.gradle` of the docker image in the real directory `~/.beam_docker_build_env/.gradle/` The reason for that i

[GitHub] [beam] nielsbasjes commented on a change in pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

nielsbasjes commented on a change in pull request #13308: URL: https://github.com/apache/beam/pull/13308#discussion_r529042179 ## File path: dev-support/docker/Dockerfile ## @@ -0,0 +1,141 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [beam] nielsbasjes commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

nielsbasjes commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732461325 What I ran into is that if you run `go get github.com/linkedin/goavro` while building the docker image the contents of `${HOME}/.cache/go-build` will be owned by root. This c

[GitHub] [beam] nielsbasjes commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

nielsbasjes commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732450092 I found a very subtle permission problem in the go part. Working on it. This is an automated message from

[GitHub] [beam] omarismail94 commented on pull request #12837: [BEAM-10891] Dockerfile for development container

omarismail94 commented on pull request #12837: URL: https://github.com/apache/beam/pull/12837#issuecomment-732440720 Yeah, I think @nielsbasjes 's PR is the way to go! I tested it and it seems to work. I left my comments in #13308

[GitHub] [beam] omarismail94 edited a comment on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

omarismail94 edited a comment on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732438422 I just pulled the branch and ran ``` ./gradlew check ``` Only 2 tests failed, but I think their cause is known: * `Task :sdks:java:extensions:ml:

[GitHub] [beam] omarismail94 commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

omarismail94 commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732438422 I just pulled the branch and ran ``` ./gradlew check ``` Only 2 tests failed, but I think their cause is known: * Task :sdks:java:extensions:ml:test. Th

[GitHub] [beam] omarismail94 commented on a change in pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

omarismail94 commented on a change in pull request #13308: URL: https://github.com/apache/beam/pull/13308#discussion_r528434588 ## File path: dev-support/docker/Dockerfile ## @@ -0,0 +1,141 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributo

[GitHub] [beam] amaliujia commented on a change in pull request #13319: [BEAM-11173] Add Bigtable table with read operation

amaliujia commented on a change in pull request #13319: URL: https://github.com/apache/beam/pull/13319#discussion_r529004528 ## File path: website/www/site/content/en/documentation/dsls/sql/extensions/create-external-table.md ## @@ -204,6 +205,131 @@ TYPE bigquery LOCATION 't

[GitHub] [beam] boyuanzz commented on pull request #13409: [WIP] test only

boyuanzz commented on pull request #13409: URL: https://github.com/apache/beam/pull/13409#issuecomment-732422078 Run Java Dataflow V2 ValidatesRunner Streaming This is an automated message from the Apache Git Service. To resp

[GitHub] [beam] boyuanzz opened a new pull request #13409: [WIP] test only

boyuanzz opened a new pull request #13409: URL: https://github.com/apache/beam/pull/13409 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quick

[GitHub] [beam] y1chi commented on pull request #13350: [BEAM-11266] Python IO MongoDB: add bucket_auto aggregation option for bundling in Atlas.

y1chi commented on pull request #13350: URL: https://github.com/apache/beam/pull/13350#issuecomment-732410290 @chamikaramj Cham do you mind help merging the PR？ This is an automated message from the Apache Git Service. To res

[GitHub] [beam] boyuanzz merged pull request #13372: Enable more runner v2 tests.

boyuanzz merged pull request #13372: URL: https://github.com/apache/beam/pull/13372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] boyuanzz commented on pull request #13227: [BEAM-10480] Add splittable dofn as the recommended way of building connectors.

boyuanzz commented on pull request #13227: URL: https://github.com/apache/beam/pull/13227#issuecomment-732399816 Updated the PR with suggestions. This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] boyuanzz commented on a change in pull request #13227: [BEAM-10480] Add splittable dofn as the recommended way of building connectors.

boyuanzz commented on a change in pull request #13227: URL: https://github.com/apache/beam/pull/13227#discussion_r528964281 ## File path: website/www/site/content/en/documentation/io/developing-io-overview.md ## @@ -46,33 +46,32 @@ are the recommended steps to get started: Fo

[GitHub] [beam] amaliujia edited a comment on pull request #13387: Decode data channel elements after, rather than before, placing them in the queue.

amaliujia edited a comment on pull request #13387: URL: https://github.com/apache/beam/pull/13387#issuecomment-732394173 Thanks @robertwb! Before merging this PR, let me run benchmarks on this PR to understand perf improvements. R @amaliujia

[GitHub] [beam] amaliujia commented on pull request #13387: Decode data channel elements after, rather than before, placing them in the queue.

amaliujia commented on pull request #13387: URL: https://github.com/apache/beam/pull/13387#issuecomment-732394173 Thanks @robertwb! Before merging this PR, let me run benchmarks on this PR to understand perf improvements.

[GitHub] [beam] TheNeuralBit commented on pull request #13401: Add additional verification in PartitioningSession

TheNeuralBit commented on pull request #13401: URL: https://github.com/apache/beam/pull/13401#issuecomment-732390090 So making `Index.check` more aggressive caused a new class of failures - expressions with `preserves=partitionings.Index` with a Singleton partitioned input should produce a

[GitHub] [beam] nielsbasjes commented on a change in pull request #12837: [BEAM-10891] Dockerfile for development container

nielsbasjes commented on a change in pull request #12837: URL: https://github.com/apache/beam/pull/12837#discussion_r528957173 ## File path: Dockerfile ## @@ -0,0 +1,80 @@ +### +# Licensed to the Apach

[GitHub] [beam] aaltay edited a comment on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

aaltay edited a comment on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-732388291 > BeamAssertException This one looks like a test on the asserts. The actual error (which is likely a flake) is: logs: https://github.com/apache/beam/pul

[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

aaltay commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-732388291 > BeamAssertException This one looks like a test on the asserts. The actual error (which is likely a flake) is: logs: https://github.com/apache/beam/pull/12779

[GitHub] [beam] rHermes commented on a change in pull request #13403: Fix small things in nexmark

rHermes commented on a change in pull request #13403: URL: https://github.com/apache/beam/pull/13403#discussion_r528951556 ## File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java ## @@ -349,7 +349,7 @@ void setPubsubMessageSerializ

[GitHub] [beam] boyuanzz commented on pull request #13338: [BEAM-11070] Use self-checkpoint to enforce finalization happens.

boyuanzz commented on pull request #13338: URL: https://github.com/apache/beam/pull/13338#issuecomment-732369405 The latest commit added more comments. PTAL @ibzib This is an automated message from the Apache Git Service. To

[GitHub] [beam] rohdesamuel commented on pull request #13262: [BEAM-11151] Adds the ToStringFn Bundle Processor to Python

rohdesamuel commented on pull request #13262: URL: https://github.com/apache/beam/pull/13262#issuecomment-732365928 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] manavgarg commented on a change in pull request #13112: [BEAM-11065] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub

manavgarg commented on a change in pull request #13112: URL: https://github.com/apache/beam/pull/13112#discussion_r528933023 ## File path: examples/templates/java/kafka-to-pubsub/src/test/java/org/apache/beam/templates/KafkaToPubsubTest.java ## @@ -0,0 +1,90 @@ +/* + * License

[GitHub] [beam] manavgarg commented on a change in pull request #13112: [BEAM-11065] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub

manavgarg commented on a change in pull request #13112: URL: https://github.com/apache/beam/pull/13112#discussion_r528928098 ## File path: examples/templates/java/kafka-to-pubsub/src/test/java/org/apache/beam/templates/KafkaToPubsubTest.java ## @@ -0,0 +1,54 @@ +/* + * License

[GitHub] [beam] emilymye commented on pull request #13374: [BEAM-8106] Add Java 11 container to release and Java 11+V2+Dataflow testing

emilymye commented on pull request #13374: URL: https://github.com/apache/beam/pull/13374#issuecomment-732358628 R: @TheNeuralBit R: @kennknowles After some investigation, I think this is actually the more reliable way created to run Java 11 tests against Java 8 Beam (based on c

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13401: Add additional verification in PartitioningSession

TheNeuralBit commented on a change in pull request #13401: URL: https://github.com/apache/beam/pull/13401#discussion_r528923757 ## File path: sdks/python/apache_beam/dataframe/expressions.py ## @@ -48,10 +48,15 @@ def lookup(self, expr): # type: (Expression) -> Any class Par

[GitHub] [beam] y1chi commented on pull request #13350: [BEAM-11266] Python IO MongoDB: add bucket_auto aggregation option for bundling in Atlas.

y1chi commented on pull request #13350: URL: https://github.com/apache/beam/pull/13350#issuecomment-732355028 Run Python MongoDBIO_IT This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] sjvanrossum opened a new pull request #13408: [WIP][BEAM-11323] Add support for Date logical type to Beam Schema

sjvanrossum opened a new pull request #13408: URL: https://github.com/apache/beam/pull/13408 This change modifies the AVRO logical type date from being mapped to DATETIME to being mapped to LOGICAL_TYPE Date. Thank you for your contribution! Follow thi

[GitHub] [beam] davidyan74 commented on a change in pull request #13335: [BEAM-10921]: Fix BEAM-10921 and underlying issues

davidyan74 commented on a change in pull request #13335: URL: https://github.com/apache/beam/pull/13335#discussion_r528883529 ## File path: sdks/python/apache_beam/pipeline.py ## @@ -504,10 +504,12 @@ def run(self, test_runner_api='AUTO'): if test_runner_api == 'AUTO':

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12837: [BEAM-10891] Dockerfile for development container

TheNeuralBit commented on a change in pull request #12837: URL: https://github.com/apache/beam/pull/12837#discussion_r528885075 ## File path: Dockerfile ## @@ -0,0 +1,80 @@ +### +# Licensed to the Apac

[GitHub] [beam] nehsyc commented on pull request #13405: [BEAM-10703, BEAM-10475] Add GroupIntoBatches to runner API; add Dataflow override in Python SDK

nehsyc commented on pull request #13405: URL: https://github.com/apache/beam/pull/13405#issuecomment-732316906 CC: @robertwb @reuvenlax @aaltay This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [beam] nehsyc commented on pull request #13405: [BEAM-10703, BEAM-10475] Add GroupIntoBatches to runner API; add Dataflow override in Python SDK

nehsyc commented on pull request #13405: URL: https://github.com/apache/beam/pull/13405#issuecomment-732316473 R: @boyuanzz @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-732315237 > One I'm kinda stuck on how to fix it is this one: Aha! I ran into this same error just now, its because I had JAVA_HOME set to use a Java 11 JDK. Switching back to J

[GitHub] [beam] TheNeuralBit commented on pull request #12837: [BEAM-10891] Dockerfile for development container

TheNeuralBit commented on pull request #12837: URL: https://github.com/apache/beam/pull/12837#issuecomment-732307669 Thank you for the analysis @nielsbasjes! > So I propose to make a combination of your image (+gcc) and my scripting to create something people can easily use. T

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12837: [BEAM-10891] Dockerfile for development container

TheNeuralBit commented on a change in pull request #12837: URL: https://github.com/apache/beam/pull/12837#discussion_r528870145 ## File path: Dockerfile ## @@ -0,0 +1,80 @@ +### +# Licensed to the Apac

[GitHub] [beam] tysonjh commented on a change in pull request #12915: [BEAM-7386] Introduce EventTimeBoundedEquijoin.

tysonjh commented on a change in pull request #12915: URL: https://github.com/apache/beam/pull/12915#discussion_r524920768 ## File path: sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java ## @@ -350,6 +368,255 @@ public void pr

[GitHub] [beam] kamilwu commented on a change in pull request #13388: Go SDK Group by key load tests

kamilwu commented on a change in pull request #13388: URL: https://github.com/apache/beam/pull/13388#discussion_r528834920 ## File path: .test-infra/jenkins/job_LoadTests_GBK_Flink_Go.groovy ## @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] kamilwu commented on a change in pull request #13388: Go SDK Group by key load tests

kamilwu commented on a change in pull request #13388: URL: https://github.com/apache/beam/pull/13388#discussion_r528832366 ## File path: .test-infra/jenkins/job_LoadTests_GBK_Flink_Go.groovy ## @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [beam] echauchot commented on pull request #13403: Fix small things in nexmark

echauchot commented on pull request #13403: URL: https://github.com/apache/beam/pull/13403#issuecomment-732258723 Direct Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] echauchot commented on pull request #13403: Fix small things in nexmark

echauchot commented on pull request #13403: URL: https://github.com/apache/beam/pull/13403#issuecomment-732258472 Direct Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [beam] echauchot commented on pull request #13403: Fix small things in nexmark

echauchot commented on pull request #13403: URL: https://github.com/apache/beam/pull/13403#issuecomment-732255801 Run Direct Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] echauchot commented on a change in pull request #13403: Fix small things in nexmark

echauchot commented on a change in pull request #13403: URL: https://github.com/apache/beam/pull/13403#discussion_r528760112 ## File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkOptions.java ## @@ -349,7 +349,7 @@ void setPubsubMessageSerial

[GitHub] [beam] kamilwu commented on pull request #13407: [DO NOT MERGE - this is a clone of #13388]

kamilwu commented on pull request #13407: URL: https://github.com/apache/beam/pull/13407#issuecomment-732210861 Run Load Tests Go GBK Flink Batch This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] kamilwu commented on pull request #13407: [DO NOT MERGE - this is a clone of #13388]

kamilwu commented on pull request #13407: URL: https://github.com/apache/beam/pull/13407#issuecomment-732207690 Run Load Tests Go GBK Flink Batch This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] kamilwu commented on pull request #13407: [DO NOT MERGE - this is a clone of #13388]

kamilwu commented on pull request #13407: URL: https://github.com/apache/beam/pull/13407#issuecomment-732203863 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [beam] kamilwu commented on pull request #13407: [DO NOT MERGE - this is a clone of #13388]

kamilwu commented on pull request #13407: URL: https://github.com/apache/beam/pull/13407#issuecomment-732202047 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit

piotr-szuberski commented on pull request #13026: URL: https://github.com/apache/beam/pull/13026#issuecomment-732199535 @aromanenko-dev Ok, despite printing all the time the same version it looks like it's working. I tried to put Kafka 0.9 among the Kafka versions and it indeed failed (sti

[GitHub] [beam] nielsbasjes commented on a change in pull request #12837: [BEAM-10891] Dockerfile for development container

nielsbasjes commented on a change in pull request #12837: URL: https://github.com/apache/beam/pull/12837#discussion_r528742322 ## File path: Dockerfile ## @@ -0,0 +1,80 @@ +### +# Licensed to the Apach

[GitHub] [beam] piotr-szuberski commented on pull request #13319: [BEAM-11173] Add Bigtable table with read operation

piotr-szuberski commented on pull request #13319: URL: https://github.com/apache/beam/pull/13319#issuecomment-732197328 @amaliujia This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] kamilwu commented on pull request #13407: [DO NOT MERGE - this is a clone of #13388]

kamilwu commented on pull request #13407: URL: https://github.com/apache/beam/pull/13407#issuecomment-732184861 Run Load Tests Go GBK Flink Batch This is an automated message from the Apache Git Service. To respond to the mes

  1   2   >