Re: [VOTE] Release 2.41.0, release candidate #1

[Jan Lukavský]

+1 (non-binding)

Validated Java SDK with classical Flink Runner.

On 8/15/22 23:06, Chamikara Jayalath via dev wrote:

+1 as well
(I believe Kiley is addressing the container tags issue)

Thanks,
Cham

On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw  
wrote:


+1 (binding).

I verified the release artifacts and signatures, and ran a couple of
simple Python pipelines.

On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
 wrote:
>
>
>
> On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok 
wrote:
>>
>> Thanks everyone!
>>
>> @Chamikara Jayalath The Spark issue is running successfully for
me, could you try it again? I'll look into the container tags.
>
>
> Thanks. Regarding the Spark issue, it could just be my setup
then. Feel free to close the Github issue.
>
> - Cham
>
>>
>>
>> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada
 wrote:
>>>
>>> +1 - I validated tests/build with existing Dataflow Templates
>>> Best
>>> -P.
>>>
>>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev
 wrote:

 +1 - I validated python quickstarts on direct runner.

 Thank you Kiley!



 On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev
 wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #1 for the
version 2.41.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific
comments)
>
>
> Reviewers are encouraged to test their own use cases with
the release candidate, and vote +1 if no issues are found.
>
> The complete staging area is available for your review,
which includes:
> * GitHub Release notes [1],
> * the official Apache source release to be deployed to
dist.apache.org  [2], which is signed with
the key with fingerprint 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
> * all artifacts to be deployed to the Maven Central
Repository [4],
> * source code tag "v2.41.0-RC1" [5],
> * website pull request listing the release [6], the blog
post [6], and publishing the API reference manual [7].
> * Java artifacts were built with Gradle 7.4 and
OpenJDK/Oracle JDK 1.8.0_232.
> * Python artifacts are deployed along with the source
release to the dist.apache.org  [2] and
PyPI[8].
> * Validation sheet with a tab for 2.41.0 release to help
with validation [9].
> * Docker images published to Docker Hub [10].
>
> The vote will be open for at least 72 hours. It is adopted
by majority approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects,
check out our blog post at
https://beam.apache.org/blog/validate-beam-release/.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/beam/milestone/3
> [2] https://dist.apache.org/repos/dist/dev/beam/2.41.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4]
https://repository.apache.org/content/repositories/orgapachebeam-1282/
> [5] https://github.com/apache/beam/tree/v2.41.0-RC1
> [6] https://github.com/apache/beam/pull/22706
> [7] https://github.com/apache/beam-site/pull/633
> [8] https://pypi.org/project/apache-beam/2.41.0rc1/
> [9]

https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=331459080
> [10]
https://hub.docker.com/search?q=apache%2Fbeam&type=image

Beam High Priority Issue Report (68)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/22642 [Bug]: Dataflow fails to drain a 
job when using BigQuery (java sdk v.2.38)
https://github.com/apache/beam/issues/22440 [Bug]: Python Batch Dataflow 
SideInput LoadTests failing
https://github.com/apache/beam/issues/22321 
PortableRunnerTestWithExternalEnv.test_pardo_large_input is regularly failing 
on jenkins
https://github.com/apache/beam/issues/22303 [Task]: Add tests to Kafka SDF and 
fix known and discovered issues
https://github.com/apache/beam/issues/22299 [Bug]: JDBCIO Write freeze at 
getConnection() in WriteFn
https://github.com/apache/beam/issues/22283 [Bug]: Python Lots of fn runner 
test items cost exactly 5 seconds to run
https://github.com/apache/beam/issues/21794 Dataflow runner creates a new timer 
whenever the output timestamp is change
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output 
to Failed Inserts PCollection
https://github.com/apache/beam/issues/21704 beam_PostCommit_Java_DataflowV2 
failures parent bug
https://github.com/apache/beam/issues/21703 pubsublite.ReadWriteIT failing in 
beam_PostCommit_Java_DataflowV1 and V2
https://github.com/apache/beam/issues/21702 SpannerWriteIT failing in beam 
PostCommit Java V1
https://github.com/apache/beam/issues/21701 beam_PostCommit_Java_DataflowV1 
failing with a variety of flakes and errors
https://github.com/apache/beam/issues/21700 
--dataflowServiceOptions=use_runner_v2 is broken
https://github.com/apache/beam/issues/21696 Flink Tests failure :  
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.runners.core.construction.SerializablePipelineOptions 
https://github.com/apache/beam/issues/21695 DataflowPipelineResult does not 
raise exception for unsuccessful states.
https://github.com/apache/beam/issues/21694 BigQuery Storage API insert with 
writeResult retry and write to error table
https://github.com/apache/beam/issues/21480 flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
https://github.com/apache/beam/issues/21472 Dataflow streaming tests failing 
new AfterSynchronizedProcessingTime test
https://github.com/apache/beam/issues/21471 Flakes: Failed to load cache entry
https://github.com/apache/beam/issues/21470 Test flake: test_split_half_sdf
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21468 
beam_PostCommit_Python_Examples_Dataflow failing
https://github.com/apache/beam/issues/21467 GBK and CoGBK streaming Java load 
tests failing
https://github.com/apache/beam/issues/21465 Kafka commit offset drop data on 
failure for runners that have non-checkpointing shuffle
https://github.com/apache/beam/issues/21463 NPE in Flink Portable 
ValidatesRunner streaming suite
https://github.com/apache/beam/issues/21462 Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use
https://github.com/apache/beam/issues/21271 pubsublite.ReadWriteIT flaky in 
beam_PostCommit_Java_DataflowV2  
https://github.com/apache/beam/issues/21270 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2
https://github.com/apache/beam/issues/21268 Race between member variable being 
accessed due to leaking uninitialized state via OutboundObserverFactory
https://github.com/apache/beam/issues/21267 WriteToBigQuery submits a duplicate 
BQ load job if a 503 error code is returned from googleapi
https://github.com/apache/beam/issues/21266 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite.
https://github.com/apache/beam/issues/21265 
apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally
 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible
https://github.com/apache/beam/issues/21263 (Broken Pipe induced) Bricked 
Dataflow Pipeline 
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21261 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21257 Either Create or DirectRunner fails 
to produce all elements to the following transform
https://github.com/apache/beam/issues/21123 Multiple jobs running on Flink 
session cluster reuse the persistent Python environment.
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCount

Benchmark tests for the Beam RunInference API

[Anand Inguva via dev]

Hi,

I created a doc
[1]
which outlines the plan for the RunInference API[2] benchmark/performance
tests. I would appreciate feedback on the following,

   - Models used for the benchmark tests.
   - Metrics calculated as part of the benchmark tests.


If you have any inputs or any suggestions on additional metrics/models that
would be helpful for the Beam ML community as part of the benchmark tests,
please let us know.

[1]
https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#
[2]
 
https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269



Thanks,
Anand

Beam BigtableIO versus Google CloudBigtableIO

[Sahith Nallapareddy via dev]

Hello,

I see that there are two implementations of reading and writing from
Bigtable, one in beam and one that is references in Google cloud
documentation. Is one preferred over the other? We often use the Beam
BigtableIO to write to bigtable but I have found that sometimes the default
configuration can lead to a lot of write requests (which can lead to having
more nodes as well it seems, more cost associated). I am about to try
messing around with the bulk options to see if that can raise the batching
of mutations, but is there anything else I should try, like switching the
actual transform we use?

Thanks,

Sahith

Re: [VOTE] Release 2.41.0, release candidate #1

[Ritesh Ghorse via dev]

+1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow runner


On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský  wrote:

> +1 (non-binding)
>
> Validated Java SDK with classical Flink Runner.
> On 8/15/22 23:06, Chamikara Jayalath via dev wrote:
>
> +1 as well
> (I believe Kiley is addressing the container tags issue)
>
> Thanks,
> Cham
>
> On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw 
> wrote:
>
>> +1 (binding).
>>
>> I verified the release artifacts and signatures, and ran a couple of
>> simple Python pipelines.
>>
>> On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
>>  wrote:
>> >
>> >
>> >
>> > On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok  wrote:
>> >>
>> >> Thanks everyone!
>> >>
>> >> @Chamikara Jayalath The Spark issue is running successfully for me,
>> could you try it again? I'll look into the container tags.
>> >
>> >
>> > Thanks. Regarding the Spark issue, it could just be my setup then. Feel
>> free to close the Github issue.
>> >
>> > - Cham
>> >
>> >>
>> >>
>> >> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada 
>> wrote:
>> >>>
>> >>> +1 - I validated tests/build with existing Dataflow Templates
>> >>> Best
>> >>> -P.
>> >>>
>> >>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev <
>> dev@beam.apache.org> wrote:
>> 
>>  +1 - I validated python quickstarts on direct runner.
>> 
>>  Thank you Kiley!
>> 
>> 
>> 
>>  On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev <
>> dev@beam.apache.org> wrote:
>> >
>> > Hi everyone,
>> > Please review and vote on the release candidate #1 for the version
>> 2.41.0, as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >
>> >
>> > Reviewers are encouraged to test their own use cases with the
>> release candidate, and vote +1 if no issues are found.
>> >
>> > The complete staging area is available for your review, which
>> includes:
>> > * GitHub Release notes [1],
>> > * the official Apache source release to be deployed to
>> dist.apache.org [2], which is signed with the key with fingerprint
>> 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "v2.41.0-RC1" [5],
>> > * website pull request listing the release [6], the blog post [6],
>> and publishing the API reference manual [7].
>> > * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle JDK
>> 1.8.0_232.
>> > * Python artifacts are deployed along with the source release to
>> the dist.apache.org [2] and PyPI[8].
>> > * Validation sheet with a tab for 2.41.0 release to help with
>> validation [9].
>> > * Docker images published to Docker Hub [10].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by
>> majority approval, with at least 3 PMC affirmative votes.
>> >
>> > For guidelines on how to try the release in your projects, check
>> out our blog post at https://beam.apache.org/blog/validate-beam-release/.
>> >
>> > Thanks,
>> > Release Manager
>> >
>> > [1] https://github.com/apache/beam/milestone/3
>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.41.0/
>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1282/
>> > [5] https://github.com/apache/beam/tree/v2.41.0-RC1
>> > [6] https://github.com/apache/beam/pull/22706
>> > [7] https://github.com/apache/beam-site/pull/633
>> > [8] https://pypi.org/project/apache-beam/2.41.0rc1/
>> > [9]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=331459080
>> > [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>
>

Re: Beam BigtableIO versus Google CloudBigtableIO

[Diego Gomez via dev]

Hello Sahith,

We recommend using BigtableIO over CloudBigtableIO. Both of them have
similar performances and main differences being than CloudBigtableIO uses
HBase Result and Puts, while BigtableIO uses protos to read results and
mutations.

The two connectors should result in similar spending on Bigtable's side,
more write requests doesn't necessarily mean more cost/nodes. What version
of CloudBigtableIO are you using and are you using an autoscaling CBT
cluster?

-Diego

On Tue, Aug 16, 2022 at 11:55 AM Sahith Nallapareddy via dev <
dev@beam.apache.org> wrote:

> Hello,
>
> I see that there are two implementations of reading and writing from
> Bigtable, one in beam and one that is references in Google cloud
> documentation. Is one preferred over the other? We often use the Beam
> BigtableIO to write to bigtable but I have found that sometimes the default
> configuration can lead to a lot of write requests (which can lead to having
> more nodes as well it seems, more cost associated). I am about to try
> messing around with the bulk options to see if that can raise the batching
> of mutations, but is there anything else I should try, like switching the
> actual transform we use?
>
> Thanks,
>
> Sahith
>

Re: [VOTE] Release 2.41.0, release candidate #1

[Alexey Romanenko]

I tested with "beam-samples" [1] and found that a rather simple test pipeline 
fails [2] with this runtime error:

Error: 
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.776 s <<< 
FAILURE! - in SerializationTest

5809
Error: 
 SerializationTest.nonSerilizableTest  Time elapsed: 2.708 s  <<< ERROR!

5810
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
 java.lang.IllegalStateException: Invisible parameter type of 
SerializationTest$1 arg0 for public 
SerializationTest$1$DoFnInvoker(SerializationTest$1)

5811
Caused by: java.lang.IllegalStateException: Invisible parameter type of 
SerializationTest$1 arg0 for public 
SerializationTest$1$DoFnInvoker(SerializationTest$1)


Seems like that it’s caused by “bytebuddy” dependency update [3] from version 
1.11.0 to 1.12.9 and it was vendored before (not sure if it’s related).

Downgrading the “bytebuddy” version to 1.11.0 fixes an error. 

I’ve not yet gone deep into a cause of this problem but maybe someone knows 
some details?

[1] https://github.com/Talend/beam-samples/ 

[2] 
https://github.com/Talend/beam-samples/runs/7856722514?check_suite_focus=true 
[3] https://github.com/apache/beam/pull/17317 


—
Alexey

> On 16 Aug 2022, at 17:54, Ritesh Ghorse via dev  wrote:
> 
> +1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow runner
> 
> 
> On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský  > wrote:
> +1 (non-binding)
> 
> Validated Java SDK with classical Flink Runner.
> 
> On 8/15/22 23:06, Chamikara Jayalath via dev wrote:
>> +1 as well
>> (I believe Kiley is addressing the container tags issue)
>> 
>> Thanks,
>> Cham
>> 
>> On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw > > wrote:
>> +1 (binding).
>> 
>> I verified the release artifacts and signatures, and ran a couple of
>> simple Python pipelines.
>> 
>> On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
>> mailto:dev@beam.apache.org>> wrote:
>> >
>> >
>> >
>> > On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok > > > wrote:
>> >>
>> >> Thanks everyone!
>> >>
>> >> @Chamikara Jayalath The Spark issue is running successfully for me, could 
>> >> you try it again? I'll look into the container tags.
>> >
>> >
>> > Thanks. Regarding the Spark issue, it could just be my setup then. Feel 
>> > free to close the Github issue.
>> >
>> > - Cham
>> >
>> >>
>> >>
>> >> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada > >> > wrote:
>> >>>
>> >>> +1 - I validated tests/build with existing Dataflow Templates
>> >>> Best
>> >>> -P.
>> >>>
>> >>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev > >>> > wrote:
>> 
>>  +1 - I validated python quickstarts on direct runner.
>> 
>>  Thank you Kiley!
>> 
>> 
>> 
>>  On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev >  > wrote:
>> >
>> > Hi everyone,
>> > Please review and vote on the release candidate #1 for the version 
>> > 2.41.0, as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> >
>> > Reviewers are encouraged to test their own use cases with the release 
>> > candidate, and vote +1 if no issues are found.
>> >
>> > The complete staging area is available for your review, which includes:
>> > * GitHub Release notes [1],
>> > * the official Apache source release to be deployed to dist.apache.org 
>> >  [2], which is signed with the key with 
>> > fingerprint 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "v2.41.0-RC1" [5],
>> > * website pull request listing the release [6], the blog post [6], and 
>> > publishing the API reference manual [7].
>> > * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle JDK 
>> > 1.8.0_232.
>> > * Python artifacts are deployed along with the source release to the 
>> > dist.apache.org  [2] and PyPI[8].
>> > * Validation sheet with a tab for 2.41.0 release to help with 
>> > validation [9].
>> > * Docker images published to Docker Hub [10].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority 
>> > approval, with at least 3 PMC affirmative votes.
>> >
>> > For guidelines on how to try the release in your projects, check out 
>> > our blog post at https://beam.apache.org/blog/validate-beam-release/ 
>> > .
>> >
>> > Thanks,
>> > Release Manager
>> >
>> > [1] https://github.com/apache/beam/milestone/3 
>> > 

Re: Beam BigtableIO versus Google CloudBigtableIO

[Sahith Nallapareddy via dev]

Hello Diego,

Right now we are using BigtableIO so I will continue to use that one!

For the second part, Ill explain a bit more what we saw as I simplified a
bit in my original email. At some point we had two streaming pipelines
writing to bigtable and we decided to combine these into one pipeline that
writes to multiple Bigtables. What we found is that our network traffic to
bigtable did go up by a bit more than 3x than when the pipelines separated.
Our nodes were about the same now looking back I think I misremembered that
part. We opened a google ticket at the time to see what we could do to
remedy this as we didnt expect that much of a cost increase and they told
us that this was due to the new implementation batching less mutations
(causing more write requests) than the old. We were advised to mess with
the bulk options, but we did not really get a chance to yet so I will try
that at some point. I was wondering if anyone could shed light if that is
the best way to configure how much bigtable batches requests or is there
more that could be done.

Thanks,

Sahith

On Tue, Aug 16, 2022 at 1:04 PM Diego Gomez  wrote:

> Hello Sahith,
>
> We recommend using BigtableIO over CloudBigtableIO. Both of them have
> similar performances and main differences being than CloudBigtableIO uses
> HBase Result and Puts, while BigtableIO uses protos to read results and
> mutations.
>
> The two connectors should result in similar spending on Bigtable's side,
> more write requests doesn't necessarily mean more cost/nodes. What version
> of CloudBigtableIO are you using and are you using an autoscaling CBT
> cluster?
>
> -Diego
>
> On Tue, Aug 16, 2022 at 11:55 AM Sahith Nallapareddy via dev <
> dev@beam.apache.org> wrote:
>
>> Hello,
>>
>> I see that there are two implementations of reading and writing from
>> Bigtable, one in beam and one that is references in Google cloud
>> documentation. Is one preferred over the other? We often use the Beam
>> BigtableIO to write to bigtable but I have found that sometimes the default
>> configuration can lead to a lot of write requests (which can lead to having
>> more nodes as well it seems, more cost associated). I am about to try
>> messing around with the bulk options to see if that can raise the batching
>> of mutations, but is there anything else I should try, like switching the
>> actual transform we use?
>>
>> Thanks,
>>
>> Sahith
>>
>

Re: Beam BigtableIO versus Google CloudBigtableIO

[Diego Gomez via dev]

Sounds good!

In regards to the second paragraph, it is true that there was a recent
change to the amount of mutations in a batch. I would still recommend using
bulkOptions and withBigtableOptionsConfigurator(), I believe that the field
'BIGTABLE_BULK_MAX_ROW_KEY_COUNT_DEFAULT' may be what you are looking for.
I can't really advise a specific number since each use case is different,
but just experiment and determine which size is best suited for your
workload.

More information on the different bulkOptions here:
https://cloud.google.com/bigtable/docs/hbase-client/javadoc/com/google/cloud/bigtable/config/BulkOptions.html

-Diego

On Tue, Aug 16, 2022 at 1:47 PM Sahith Nallapareddy 
wrote:

> Hello Diego,
>
> Right now we are using BigtableIO so I will continue to use that one!
>
> For the second part, Ill explain a bit more what we saw as I simplified a
> bit in my original email. At some point we had two streaming pipelines
> writing to bigtable and we decided to combine these into one pipeline that
> writes to multiple Bigtables. What we found is that our network traffic to
> bigtable did go up by a bit more than 3x than when the pipelines separated.
> Our nodes were about the same now looking back I think I misremembered that
> part. We opened a google ticket at the time to see what we could do to
> remedy this as we didnt expect that much of a cost increase and they told
> us that this was due to the new implementation batching less mutations
> (causing more write requests) than the old. We were advised to mess with
> the bulk options, but we did not really get a chance to yet so I will try
> that at some point. I was wondering if anyone could shed light if that is
> the best way to configure how much bigtable batches requests or is there
> more that could be done.
>
> Thanks,
>
> Sahith
>
> On Tue, Aug 16, 2022 at 1:04 PM Diego Gomez  wrote:
>
>> Hello Sahith,
>>
>> We recommend using BigtableIO over CloudBigtableIO. Both of them have
>> similar performances and main differences being than CloudBigtableIO uses
>> HBase Result and Puts, while BigtableIO uses protos to read results and
>> mutations.
>>
>> The two connectors should result in similar spending on Bigtable's side,
>> more write requests doesn't necessarily mean more cost/nodes. What version
>> of CloudBigtableIO are you using and are you using an autoscaling CBT
>> cluster?
>>
>> -Diego
>>
>> On Tue, Aug 16, 2022 at 11:55 AM Sahith Nallapareddy via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hello,
>>>
>>> I see that there are two implementations of reading and writing from
>>> Bigtable, one in beam and one that is references in Google cloud
>>> documentation. Is one preferred over the other? We often use the Beam
>>> BigtableIO to write to bigtable but I have found that sometimes the default
>>> configuration can lead to a lot of write requests (which can lead to having
>>> more nodes as well it seems, more cost associated). I am about to try
>>> messing around with the bulk options to see if that can raise the batching
>>> of mutations, but is there anything else I should try, like switching the
>>> actual transform we use?
>>>
>>> Thanks,
>>>
>>> Sahith
>>>
>>

Re: [VOTE] Release 2.41.0, release candidate #1

[Kiley Sok via dev]

cc: @Liam Miller-Cushon , who worked on the bytebuddy
update.

Liam, do you have any context on this error?

On Tue, Aug 16, 2022 at 10:11 AM Alexey Romanenko 
wrote:

> I tested with "beam-samples" [1] and found that a rather simple test
> pipeline fails [2] with this runtime error:
>
> Error:
>  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.776 s
> <<< FAILURE! - in SerializationTest
>
> 5809
> Error:
>  SerializationTest.nonSerilizableTest  Time elapsed: 2.708 s  <<< ERROR!
>
> 5810
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: Invisible parameter type of
> SerializationTest$1 arg0 for public
> SerializationTest$1$DoFnInvoker(SerializationTest$1)
>
> 5811
> Caused by: java.lang.IllegalStateException: Invisible parameter type of
> SerializationTest$1 arg0 for public
> SerializationTest$1$DoFnInvoker(SerializationTest$1)
>
>
> Seems like that it’s caused by “bytebuddy” dependency update [3] from
> version 1.11.0 to 1.12.9 and it was vendored before (not sure if it’s
> related).
>
> Downgrading the “bytebuddy” version to 1.11.0 fixes an error.
>
> I’ve not yet gone deep into a cause of this problem but maybe someone
> knows some details?
>
> [1] https://github.com/Talend/beam-samples/
> [2]
> https://github.com/Talend/beam-samples/runs/7856722514?check_suite_focus=true
>
> [3] https://github.com/apache/beam/pull/17317
>
> —
> Alexey
>
> On 16 Aug 2022, at 17:54, Ritesh Ghorse via dev 
> wrote:
>
> +1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow runner
>
>
> On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský  wrote:
>
>> +1 (non-binding)
>>
>> Validated Java SDK with classical Flink Runner.
>> On 8/15/22 23:06, Chamikara Jayalath via dev wrote:
>>
>> +1 as well
>> (I believe Kiley is addressing the container tags issue)
>>
>> Thanks,
>> Cham
>>
>> On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw 
>> wrote:
>>
>>> +1 (binding).
>>>
>>> I verified the release artifacts and signatures, and ran a couple of
>>> simple Python pipelines.
>>>
>>> On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
>>>  wrote:
>>> >
>>> >
>>> >
>>> > On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok 
>>> wrote:
>>> >>
>>> >> Thanks everyone!
>>> >>
>>> >> @Chamikara Jayalath The Spark issue is running successfully for me,
>>> could you try it again? I'll look into the container tags.
>>> >
>>> >
>>> > Thanks. Regarding the Spark issue, it could just be my setup then.
>>> Feel free to close the Github issue.
>>> >
>>> > - Cham
>>> >
>>> >>
>>> >>
>>> >> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada 
>>> wrote:
>>> >>>
>>> >>> +1 - I validated tests/build with existing Dataflow Templates
>>> >>> Best
>>> >>> -P.
>>> >>>
>>> >>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev <
>>> dev@beam.apache.org> wrote:
>>> 
>>>  +1 - I validated python quickstarts on direct runner.
>>> 
>>>  Thank you Kiley!
>>> 
>>> 
>>> 
>>>  On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev <
>>> dev@beam.apache.org> wrote:
>>> >
>>> > Hi everyone,
>>> > Please review and vote on the release candidate #1 for the version
>>> 2.41.0, as follows:
>>> > [ ] +1, Approve the release
>>> > [ ] -1, Do not approve the release (please provide specific
>>> comments)
>>> >
>>> >
>>> > Reviewers are encouraged to test their own use cases with the
>>> release candidate, and vote +1 if no issues are found.
>>> >
>>> > The complete staging area is available for your review, which
>>> includes:
>>> > * GitHub Release notes [1],
>>> > * the official Apache source release to be deployed to
>>> dist.apache.org [2], which is signed with the key with fingerprint
>>> 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
>>> > * all artifacts to be deployed to the Maven Central Repository [4],
>>> > * source code tag "v2.41.0-RC1" [5],
>>> > * website pull request listing the release [6], the blog post [6],
>>> and publishing the API reference manual [7].
>>> > * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle JDK
>>> 1.8.0_232.
>>> > * Python artifacts are deployed along with the source release to
>>> the dist.apache.org [2] and PyPI[8].
>>> > * Validation sheet with a tab for 2.41.0 release to help with
>>> validation [9].
>>> > * Docker images published to Docker Hub [10].
>>> >
>>> > The vote will be open for at least 72 hours. It is adopted by
>>> majority approval, with at least 3 PMC affirmative votes.
>>> >
>>> > For guidelines on how to try the release in your projects, check
>>> out our blog post at https://beam.apache.org/blog/validate-beam-release/
>>> .
>>> >
>>> > Thanks,
>>> > Release Manager
>>> >
>>> > [1] https://github.com/apache/beam/milestone/3
>>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.41.0/
>>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> > 

Re: [VOTE] Release 2.41.0, release candidate #1

[Kenneth Knowles]

Two options to unblock the release are:

1. Roll back https://github.com/apache/beam/pull/17317/files
2. Downgrade unvendored bytebuddy to 1.11.0 (if the above doesn't work or
is too high risk)

And as a follow up we should make sure there is some test that would
exercise this, since that PR was green and was a while ago too and our
postcommits did not catch it either.

Kenn

On Tue, Aug 16, 2022 at 12:50 PM Kiley Sok via dev 
wrote:

> cc: @Liam Miller-Cushon , who worked on the bytebuddy
> update.
>
> Liam, do you have any context on this error?
>
> On Tue, Aug 16, 2022 at 10:11 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> I tested with "beam-samples" [1] and found that a rather simple test
>> pipeline fails [2] with this runtime error:
>>
>> Error:
>>  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.776 s
>> <<< FAILURE! - in SerializationTest
>>
>> 5809
>> Error:
>>  SerializationTest.nonSerilizableTest  Time elapsed: 2.708 s  <<< ERROR!
>>
>> 5810
>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>> java.lang.IllegalStateException: Invisible parameter type of
>> SerializationTest$1 arg0 for public
>> SerializationTest$1$DoFnInvoker(SerializationTest$1)
>>
>> 5811
>> Caused by: java.lang.IllegalStateException: Invisible parameter type of
>> SerializationTest$1 arg0 for public
>> SerializationTest$1$DoFnInvoker(SerializationTest$1)
>>
>>
>> Seems like that it’s caused by “bytebuddy” dependency update [3] from
>> version 1.11.0 to 1.12.9 and it was vendored before (not sure if it’s
>> related).
>>
>> Downgrading the “bytebuddy” version to 1.11.0 fixes an error.
>>
>> I’ve not yet gone deep into a cause of this problem but maybe someone
>> knows some details?
>>
>> [1] https://github.com/Talend/beam-samples/
>> [2]
>> https://github.com/Talend/beam-samples/runs/7856722514?check_suite_focus=true
>>
>> [3] https://github.com/apache/beam/pull/17317
>>
>> —
>> Alexey
>>
>> On 16 Aug 2022, at 17:54, Ritesh Ghorse via dev 
>> wrote:
>>
>> +1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow
>> runner
>>
>>
>> On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský  wrote:
>>
>>> +1 (non-binding)
>>>
>>> Validated Java SDK with classical Flink Runner.
>>> On 8/15/22 23:06, Chamikara Jayalath via dev wrote:
>>>
>>> +1 as well
>>> (I believe Kiley is addressing the container tags issue)
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw 
>>> wrote:
>>>
 +1 (binding).

 I verified the release artifacts and signatures, and ran a couple of
 simple Python pipelines.

 On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
  wrote:
 >
 >
 >
 > On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok 
 wrote:
 >>
 >> Thanks everyone!
 >>
 >> @Chamikara Jayalath The Spark issue is running successfully for me,
 could you try it again? I'll look into the container tags.
 >
 >
 > Thanks. Regarding the Spark issue, it could just be my setup then.
 Feel free to close the Github issue.
 >
 > - Cham
 >
 >>
 >>
 >> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada 
 wrote:
 >>>
 >>> +1 - I validated tests/build with existing Dataflow Templates
 >>> Best
 >>> -P.
 >>>
 >>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev <
 dev@beam.apache.org> wrote:
 
  +1 - I validated python quickstarts on direct runner.
 
  Thank you Kiley!
 
 
 
  On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev <
 dev@beam.apache.org> wrote:
 >
 > Hi everyone,
 > Please review and vote on the release candidate #1 for the
 version 2.41.0, as follows:
 > [ ] +1, Approve the release
 > [ ] -1, Do not approve the release (please provide specific
 comments)
 >
 >
 > Reviewers are encouraged to test their own use cases with the
 release candidate, and vote +1 if no issues are found.
 >
 > The complete staging area is available for your review, which
 includes:
 > * GitHub Release notes [1],
 > * the official Apache source release to be deployed to
 dist.apache.org [2], which is signed with the key with fingerprint
 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
 > * all artifacts to be deployed to the Maven Central Repository
 [4],
 > * source code tag "v2.41.0-RC1" [5],
 > * website pull request listing the release [6], the blog post
 [6], and publishing the API reference manual [7].
 > * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle
 JDK 1.8.0_232.
 > * Python artifacts are deployed along with the source release to
 the dist.apache.org [2] and PyPI[8].
 > * Validation sheet with a tab for 2.41.0 release to help with
 validation [9].
 >

Re: [VOTE] Release 2.41.0, release candidate #1

[Kiley Sok via dev]

If we roll back, do we need to roll back +Lukasz Cwik
's change
[1] on master as well?

Liam, are we okay to roll back this change for this release?

[1] https://github.com/apache/beam/pull/22594

On Tue, Aug 16, 2022 at 2:25 PM Kenneth Knowles  wrote:

> Two options to unblock the release are:
>
> 1. Roll back https://github.com/apache/beam/pull/17317/files
> 2. Downgrade unvendored bytebuddy to 1.11.0 (if the above doesn't work or
> is too high risk)
>
> And as a follow up we should make sure there is some test that would
> exercise this, since that PR was green and was a while ago too and our
> postcommits did not catch it either.
>
> Kenn
>
> On Tue, Aug 16, 2022 at 12:50 PM Kiley Sok via dev 
> wrote:
>
>> cc: @Liam Miller-Cushon , who worked on the bytebuddy
>> update.
>>
>> Liam, do you have any context on this error?
>>
>> On Tue, Aug 16, 2022 at 10:11 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> I tested with "beam-samples" [1] and found that a rather simple test
>>> pipeline fails [2] with this runtime error:
>>>
>>> Error:
>>>  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.776 s
>>> <<< FAILURE! - in SerializationTest
>>>
>>> 5809
>>> Error:
>>>  SerializationTest.nonSerilizableTest  Time elapsed: 2.708 s  <<< ERROR!
>>>
>>> 5810
>>> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>>> java.lang.IllegalStateException: Invisible parameter type of
>>> SerializationTest$1 arg0 for public
>>> SerializationTest$1$DoFnInvoker(SerializationTest$1)
>>>
>>> 5811
>>> Caused by: java.lang.IllegalStateException: Invisible parameter type of
>>> SerializationTest$1 arg0 for public
>>> SerializationTest$1$DoFnInvoker(SerializationTest$1)
>>>
>>>
>>> Seems like that it’s caused by “bytebuddy” dependency update [3] from
>>> version 1.11.0 to 1.12.9 and it was vendored before (not sure if it’s
>>> related).
>>>
>>> Downgrading the “bytebuddy” version to 1.11.0 fixes an error.
>>>
>>> I’ve not yet gone deep into a cause of this problem but maybe someone
>>> knows some details?
>>>
>>> [1] https://github.com/Talend/beam-samples/
>>> [2]
>>> https://github.com/Talend/beam-samples/runs/7856722514?check_suite_focus=true
>>>
>>> [3] https://github.com/apache/beam/pull/17317
>>>
>>> —
>>> Alexey
>>>
>>> On 16 Aug 2022, at 17:54, Ritesh Ghorse via dev 
>>> wrote:
>>>
>>> +1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow
>>> runner
>>>
>>>
>>> On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský  wrote:
>>>
 +1 (non-binding)

 Validated Java SDK with classical Flink Runner.
 On 8/15/22 23:06, Chamikara Jayalath via dev wrote:

 +1 as well
 (I believe Kiley is addressing the container tags issue)

 Thanks,
 Cham

 On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw 
 wrote:

> +1 (binding).
>
> I verified the release artifacts and signatures, and ran a couple of
> simple Python pipelines.
>
> On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
>  wrote:
> >
> >
> >
> > On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok 
> wrote:
> >>
> >> Thanks everyone!
> >>
> >> @Chamikara Jayalath The Spark issue is running successfully for me,
> could you try it again? I'll look into the container tags.
> >
> >
> > Thanks. Regarding the Spark issue, it could just be my setup then.
> Feel free to close the Github issue.
> >
> > - Cham
> >
> >>
> >>
> >> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada 
> wrote:
> >>>
> >>> +1 - I validated tests/build with existing Dataflow Templates
> >>> Best
> >>> -P.
> >>>
> >>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev <
> dev@beam.apache.org> wrote:
> 
>  +1 - I validated python quickstarts on direct runner.
> 
>  Thank you Kiley!
> 
> 
> 
>  On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev <
> dev@beam.apache.org> wrote:
> >
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the
> version 2.41.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific
> comments)
> >
> >
> > Reviewers are encouraged to test their own use cases with the
> release candidate, and vote +1 if no issues are found.
> >
> > The complete staging area is available for your review, which
> includes:
> > * GitHub Release notes [1],
> > * the official Apache source release to be deployed to
> dist.apache.org [2], which is signed with the key with fingerprint
> 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
> > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > * source code tag "v2.41.0-RC1" [5],
> > * website pull r

Re: [VOTE] Release 2.41.0, release candidate #1

[Kenneth Knowles]

I don't think so. The vendored version that Beam depends on has been
published and shouldn't need to be built again. Not sure if our build has
changed so that it builds the vendored stuff as part of the main build as
well, vs pulling it from maven central.

Kenn

On Tue, Aug 16, 2022 at 2:42 PM Kiley Sok  wrote:

> If we roll back, do we need to roll back +Lukasz Cwik 's 
> change
> [1] on master as well?
>
> Liam, are we okay to roll back this change for this release?
>
> [1] https://github.com/apache/beam/pull/22594
>
> On Tue, Aug 16, 2022 at 2:25 PM Kenneth Knowles  wrote:
>
>> Two options to unblock the release are:
>>
>> 1. Roll back https://github.com/apache/beam/pull/17317/files
>> 2. Downgrade unvendored bytebuddy to 1.11.0 (if the above doesn't work or
>> is too high risk)
>>
>> And as a follow up we should make sure there is some test that would
>> exercise this, since that PR was green and was a while ago too and our
>> postcommits did not catch it either.
>>
>> Kenn
>>
>> On Tue, Aug 16, 2022 at 12:50 PM Kiley Sok via dev 
>> wrote:
>>
>>> cc: @Liam Miller-Cushon , who worked on the
>>> bytebuddy update.
>>>
>>> Liam, do you have any context on this error?
>>>
>>> On Tue, Aug 16, 2022 at 10:11 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 I tested with "beam-samples" [1] and found that a rather simple test
 pipeline fails [2] with this runtime error:

 Error:
  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.776
 s <<< FAILURE! - in SerializationTest

 5809
 Error:
  SerializationTest.nonSerilizableTest  Time elapsed: 2.708 s  <<< ERROR!

 5810
 org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
 java.lang.IllegalStateException: Invisible parameter type of
 SerializationTest$1 arg0 for public
 SerializationTest$1$DoFnInvoker(SerializationTest$1)

 5811
 Caused by: java.lang.IllegalStateException: Invisible parameter type of
 SerializationTest$1 arg0 for public
 SerializationTest$1$DoFnInvoker(SerializationTest$1)


 Seems like that it’s caused by “bytebuddy” dependency update [3] from
 version 1.11.0 to 1.12.9 and it was vendored before (not sure if it’s
 related).

 Downgrading the “bytebuddy” version to 1.11.0 fixes an error.

 I’ve not yet gone deep into a cause of this problem but maybe someone
 knows some details?

 [1] https://github.com/Talend/beam-samples/
 [2]
 https://github.com/Talend/beam-samples/runs/7856722514?check_suite_focus=true

 [3] https://github.com/apache/beam/pull/17317

 —
 Alexey

 On 16 Aug 2022, at 17:54, Ritesh Ghorse via dev 
 wrote:

 +1 (non-binding), Validated Go SDK Quickstart on Direct and Dataflow
 runner


 On Tue, Aug 16, 2022 at 4:26 AM Jan Lukavský  wrote:

> +1 (non-binding)
>
> Validated Java SDK with classical Flink Runner.
> On 8/15/22 23:06, Chamikara Jayalath via dev wrote:
>
> +1 as well
> (I believe Kiley is addressing the container tags issue)
>
> Thanks,
> Cham
>
> On Mon, Aug 15, 2022 at 1:00 PM Robert Bradshaw 
> wrote:
>
>> +1 (binding).
>>
>> I verified the release artifacts and signatures, and ran a couple of
>> simple Python pipelines.
>>
>> On Mon, Aug 15, 2022 at 12:40 PM Chamikara Jayalath via dev
>>  wrote:
>> >
>> >
>> >
>> > On Mon, Aug 15, 2022 at 11:37 AM Kiley Sok 
>> wrote:
>> >>
>> >> Thanks everyone!
>> >>
>> >> @Chamikara Jayalath The Spark issue is running successfully for
>> me, could you try it again? I'll look into the container tags.
>> >
>> >
>> > Thanks. Regarding the Spark issue, it could just be my setup then.
>> Feel free to close the Github issue.
>> >
>> > - Cham
>> >
>> >>
>> >>
>> >> On Mon, Aug 15, 2022 at 11:04 AM Pablo Estrada 
>> wrote:
>> >>>
>> >>> +1 - I validated tests/build with existing Dataflow Templates
>> >>> Best
>> >>> -P.
>> >>>
>> >>> On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev <
>> dev@beam.apache.org> wrote:
>> 
>>  +1 - I validated python quickstarts on direct runner.
>> 
>>  Thank you Kiley!
>> 
>> 
>> 
>>  On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev <
>> dev@beam.apache.org> wrote:
>> >
>> > Hi everyone,
>> > Please review and vote on the release candidate #1 for the
>> version 2.41.0, as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >
>> >
>> > Reviewers are encouraged to test their own use cases with the
>> release candidate, and vote +1 if no issues are found.
>> >
>> >>>

Forward StackOverflow questions with the apache-beam tag to a new mailing list

[Chamikara Jayalath via dev]

Hi folks,

It seems like many of the questions posted to StackOverflow with the
apache-beam tag [1] go unanswered or take more than they should to receive
an acceptable answer.

What do you all think about creating a new mailing list,
stackoverf...@beam.apache.org (assuming Apache Infra is OK with this and
it's technically feasible to do so), where notifications regarding such
questions will be forwarded to ?

This should allow folks who are interested in answering related questions
to get notified early. Hopefully getting more eyeballs on these questions
will increase the response rate (and the quality of the answers).

Another option might be to post such notifications (or aggregations) to dev@
but this might unnecessarily spam all members of the dev list.

Thanks,
Cham

[1] https://stackoverflow.com/questions/tagged/apache-beam