Re: bhulette stepping back (for now)

[Robin Qiu via dev]

Thanks for your contribution Brian! Hope you enjoy your new team!

Best,
Robin

On Fri, Nov 11, 2022 at 10:27 AM Kenneth Knowles  wrote:

> Your contributions have been huge. You will be missed! But have a fabulous
> time with BigQuery. And thank you so much for letting us know [1]
>
> Kenn
>
> [1] See "stepping down considerately" from
> https://www.apache.org/foundation/policies/conduct.html
>
> On Thu, Nov 10, 2022 at 4:00 PM Brian Hulette  wrote:
>
>> Hi dev@beam,
>>
>> I just wanted to let the community know that I will be stepping back from
>> Beam development for now. I'm switching to a different team within Google
>> next week - I will be working on BigQuery.
>>
>> I'm removing myself from automated code review assignments [1], and won't
>> actively monitor the beam lists anymore. That being said, I'm happy to
>> contribute to discussions or code reviews when it would be particularly
>> helpful, e.g. for anything relating to DataFrames/Schemas/SQL. I can always
>> be reached at bhule...@apache.org, and @TheNeuralBit [2] on GitHub.
>>
>> Brian
>>
>> [1] https://github.com/apache/beam/pull/24108
>> [2] https://github.com/TheNeuralBit
>>
>

Re: [ANNOUNCE] New committer: Tomo Suzuki

[Robin Qiu]

Congrats Tomo, well deserved!

On Fri, Apr 2, 2021 at 2:28 PM Ismaël Mejía  wrote:

> Congrats Tomo, so well deserved. It has been a pleasure to work with you!
>
>
> On Fri, Apr 2, 2021 at 8:29 PM Tyson Hamilton  wrote:
>
>> Congrats!
>>
>> On Fri, Apr 2, 2021 at 11:02 AM Pablo Estrada  wrote:
>>
>>> Thank you Tomo! And congrats : )
>>>
>>> On Fri, Apr 2, 2021 at 10:24 AM Robert Bradshaw 
>>> wrote:
>>>
 Congratulations!

 On Fri, Apr 2, 2021 at 10:19 AM Chamikara Jayalath <
 chamik...@google.com> wrote:

> Congrats Tomo!
>
> On Fri, Apr 2, 2021 at 9:54 AM Brian Hulette 
> wrote:
>
>> Congratulations Tomo! Well deserved :)
>>
>> On Fri, Apr 2, 2021 at 9:51 AM Yichi Zhang  wrote:
>>
>>> Congratulations!
>>>
>>> On Fri, Apr 2, 2021 at 9:42 AM Ahmet Altay  wrote:
>>>
 Congratulations! 

 On Fri, Apr 2, 2021 at 9:38 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Tomo Suzuki
>
> Since joining the Beam community in 2019, Tomo has done lots of
> critical work on Beam's dependencies: maintaining the dependency 
> checker
> that files Jiras and sends emails, upgrading dependencies, fixing
> dependency configuration errors, maintaining our linkage checker. Most
> recently, an epic upgrade of gRPC.
>
> Considering these highlighted contributions and the rest, the Beam
> PMC trusts Tomo with the responsibilities of a Beam committer [1].
>
> Thank you, Tomo, for your contributions.
>
> Kenn
>
> [1] https://beam.apache.org/contribute/become-a-committer
> /#an-apache-beam-committer
>

Re: [ANNOUNCE] New committer: Ning Kang

[Robin Qiu]

Congratulations Ning!

On Tue, Mar 23, 2021 at 12:56 PM Ahmet Altay  wrote:

> Congratulations Ning!
>
> On Tue, Mar 23, 2021 at 12:38 PM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Congrats, Ning Kang! Well deserved!
>> Thank you for your contributions and users support!
>>
>> Alexey
>>
>> On 23 Mar 2021, at 20:35, Pablo Estrada  wrote:
>>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new committer:
>> Ning Kang.
>>
>> Ning has been working in Beam for a while. He has contributed to the
>> interactive experience of the Pyhton SDK, and developed a sidebar
>> component, along with a release process for it. Ning has also helped users
>> on StackOverflow and user@, especially when it comes to Interactive Beam.
>>
>> Considering these contributions, the Beam PMC trusts Ning with the
>> responsibilities of a Beam committer.[1]
>>
>> Thanks Ning!
>> -P.
>>
>> [1] https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
>>
>>

Re: [ANNOUNCE] New PMC Member: Chamikara Jayalath

[Robin Qiu]

Congratulations, Cham!

On Thu, Jan 21, 2021 at 3:05 PM Tyson Hamilton  wrote:

> Woo! Congrats Cham!
>
> On Thu, Jan 21, 2021 at 3:02 PM Robert Burke  wrote:
>
>> Congratulations! That's fantastic news.
>>
>> On Thu, Jan 21, 2021, 2:59 PM Reza Rokni  wrote:
>>
>>> Congratulations!
>>>
>>> On Fri, Jan 22, 2021 at 6:58 AM Ankur Goenka  wrote:
>>>
 Congrats Cham!

 On Thu, Jan 21, 2021 at 2:57 PM Ahmet Altay  wrote:

> Hi all,
>
> Please join me and the rest of Beam PMC in welcoming Chamikara
> Jayalath as our
> newest PMC member.
>
> Cham has been part of the Beam community from its early days and
> contributed to the project in significant ways, including contributing new
> features and improvements especially related Beam IOs, advocating for
> users, and mentoring new community members.
>
> Congratulations Cham! And thanks for being a part of Beam!
>
> Ahmet
>

Re: BeamSql query failure on complex data structures

[Robin Qiu]

Yeah this looks like a Calcite issue. Thank you Kyle for marking
https://issues.apache.org/jira/browse/BEAM-9379 as a blocker!

On Tue, Dec 15, 2020 at 6:44 PM Robin Qiu  wrote:

> No. That seems new to me. I can take a look. It says it's an issue on
> Calcite side.
>
> On Tue, Dec 15, 2020 at 5:13 PM Andrew Pilloud 
> wrote:
>
>> Are these issues you are aware of?
>>
>> -- Forwarded message -
>> From: Aspen Barnes 
>> Date: Tue, Dec 15, 2020 at 12:42 PM
>> Subject: BeamSql query failure on complex data structures
>> To: 
>>
>>
>> Hello, I have recently encountered an issue with BeamSql queries against
>> complex data structures. See
>> https://issues.apache.org/jira/browse/BEAM-11140 for info. Essentially,
>> the query code generation is failing past a certain depth, which is not
>> hard to achieve with subqueries and nested data structures. Is there any
>> chance this issue could be fixed? It seems as though it has been solved on
>> Calcite's side.
>>
>> Aspen Barnes
>> Security Engineer II
>> Pronouns | they/them
>> Etsy
>>
>

Re: Implementing ARR_AGG

[Robin Qiu]

Hi Sonam, I replied directly to your draft PR. Please see me comments there
and let me know if that is helpful.

On Mon, Dec 7, 2020 at 4:37 AM Sonam Ramchand <
sonam.ramch...@venturedive.com> wrote:

> Hi Devs,
> I have tried to implement the ARR_AGG function for Zetasql dialect by
> following the STRING_AGG implementation (
> https://github.com/apache/beam/pull/11895).
> Draft PR for ARR_AGG is (https://github.com/apache/beam/pull/13483). When
> i try to run the test,
>
> @Test
> public void testArrayAggregation() {
>   String sql =
>   "SELECT ARRAY_AGG(x) AS array_agg\n" +
>   "FROM UNNEST([2, 1, -2, 3, -2, 1, 2]) AS x";  
> ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
>   BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
>   PCollection stream = BeamSqlRelUtils.toPCollection(pipeline, 
> beamRelNode);  Schema schema = Schema.builder().addArrayField("array_field", 
> FieldType.of(Schema.TypeName.ARRAY)).build();
>   PAssert.that(stream)
>   .containsInAnyOrder(Row.withSchema(schema).addArray(2, 1, -2, 3, 
> -2, 1, 2).build());  
> pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
> }
>
> I am getting an error,
> type mismatch:
> aggCall type:
> BIGINT NOT NULL ARRAY NOT NULL
> inferred type:
> ARRAY NOT NULL
> java.lang.AssertionError: type mismatch:
> aggCall type:
> BIGINT NOT NULL ARRAY NOT NULL
> inferred type:
> ARRAY NOT NULL at org.apache.beam.vendor.calcite.v1_20_0.org
> .apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at org.apache.beam.vendor.calcite.v1_20_0.org
> .apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:1958)
> at org.apache.beam.vendor.calcite.v1_20_0.org
> .apache.calcite.rel.core.Aggregate.typeMatchesInferred(Aggregate.java:434)
> at org.apache.beam.vendor.calcite.v1_20_0.org
> .apache.calcite.rel.core.Aggregate.(Aggregate.java:159)
> at org.apache.beam.vendor.calcite.v1_20_0.org
> .apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:65)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.AggregateScanConverter.convert(AggregateScanConverter.java:113)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.AggregateScanConverter.convert(AggregateScanConverter.java:50)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:102)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4719)
> at java.util.Collections$2.forEachRemaining(Collections.java:4727)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:101)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:89)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:55)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:141)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:180)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:168)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:152)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlDialectSpecTest.testArrayAggregation(ZetaSqlDialectSpecTest.java:4071)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:322)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
> at 

Re: Question about LOGICAL_AND

[Robin Qiu]

.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:266)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
> at
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
> at java.lang.Thread.run(Thread.java:748)
>
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlDialectSpecTest >
> testLogicalAndZetaSQL FAILED
> java.lang.ClassCastException at ZetaSqlDialectSpecTest.java:4334
> 1 test completed, 1 failed
> > Task :sdks:java:extensions:sql:zetasql:test FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':sdks:java:extensions:sql:zetasql:test'.
> > There were failing tests. See the report at:
> file:///home/vend/ApacheBeam/beam/sdks/java/extensions/sql/zetasql/build/reports/tests/test/index.html
>
> Do you have any idea why this is happening?
>
>
> On Fri, Nov 13, 2020 at 11:42 PM Robin Qiu  wrote:
>
>> Oh I see. Thanks for the

Re: Question about LOGICAL_AND

[Robin Qiu]

Oh I see. Thanks for the clarification, Kenn! Yeah the CombineFn
LOGICAL_AND is to be implemented.

On Fri, Nov 13, 2020 at 10:00 AM Kenneth Knowles  wrote:

> Some clarification: LOGICAL_AND is a ZetaSQL/BigQuery aggregate function:
> https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#logical_and
>
> So it needs to be implemented as a CombineFn. Here are some example PRs
> that do similar things: https://github.com/apache/beam/pulls?q=BIT_OR
>
> Kenn
>
> On Thu, Nov 12, 2020 at 12:48 PM Rui Wang  wrote:
>
>> Or the question is, which BeamSQL  dialect you are using?
>>
>>
>> -Rui
>>
>> On Thu, Nov 12, 2020 at 12:41 PM Robin Qiu  wrote:
>>
>>> Hi Sonam, AND operator is already defined by Calcite and the mapping is
>>> here:
>>> https://github.com/apache/beam/blob/816017e44e3209d334f4f3b2bc3fa829663c530e/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperatorMappingTable.java#L39.
>>> Is this what you are looking for?
>>>
>>> On Thu, Nov 12, 2020 at 12:00 PM Kyle Weaver 
>>> wrote:
>>>
>>>> If you're defining a new built-in function in ZetaSQL, you can define
>>>> an operator for it here:
>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java
>>>>
>>>> Then add the operator the table here:
>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperatorMappingTable.java
>>>>
>>>> On Thu, Nov 12, 2020 at 11:14 AM Sonam Ramchand <
>>>> sonam.ramch...@venturedive.com> wrote:
>>>>
>>>>> There is no LOGICAL_AND operator in SqlStdOperatorTable, is there any
>>>>> other way to implement LOGICAL_AND?
>>>>>
>>>>> --
>>>>>
>>>>> Regards,
>>>>> *Sonam*
>>>>> Software Engineer
>>>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>>>
>>>>> <http://venturedive.com/>
>>>>>
>>>>

Re: Question about LOGICAL_AND

[Robin Qiu]

Hi Sonam, AND operator is already defined by Calcite and the mapping is
here:
https://github.com/apache/beam/blob/816017e44e3209d334f4f3b2bc3fa829663c530e/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperatorMappingTable.java#L39.
Is this what you are looking for?

On Thu, Nov 12, 2020 at 12:00 PM Kyle Weaver  wrote:

> If you're defining a new built-in function in ZetaSQL, you can define an
> operator for it here:
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java
>
> Then add the operator the table here:
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperatorMappingTable.java
>
> On Thu, Nov 12, 2020 at 11:14 AM Sonam Ramchand <
> sonam.ramch...@venturedive.com> wrote:
>
>> There is no LOGICAL_AND operator in SqlStdOperatorTable, is there any
>> other way to implement LOGICAL_AND?
>>
>> --
>>
>> Regards,
>> *Sonam*
>> Software Engineer
>> Mobile: +92 3088337296 <+92%20308%208337296>
>>
>> 
>>
>

[ANNOUNCE] Beam 2.25.0 Released

[Robin Qiu]

The Apache Beam team is pleased to announce the release of version 2.25.0.

Apache Beam is an open source unified programming model to define and
execute data processing pipelines, including ETL, batch and stream
(continuous) processing. See: https://beam.apache.org

You can download the release here:
https://beam.apache.org/get-started/downloads/

This release includes bug fixes, features, and improvements detailed on
the Beam blog: https://beam.apache.org/blog/beam-2.25.0/

Thanks to everyone who contributed to this release, and we hope you enjoy
using Beam 2.25.0.

[RESULT] [VOTE] Release 2.25.0, release candidate #2

[Robin Qiu]

I'm happy to announce that we have unanimously approved this release.

There are 6 approving votes, 3 of which are binding:
* Ahmet Altay
* Pablo Estrada
* Robert Bradshaw

There are no disapproving votes.

Thanks everyone!

Re: [VOTE] Release 2.25.0, release candidate #2

[Robin Qiu]

Hi everyone, we now have 3 +1's from PMC members and no -1. The vote has
passed 72 hours and I am closing it now.

On Fri, Oct 23, 2020 at 1:47 PM Robert Bradshaw  wrote:

> +1 (binding).
>
> I verified the release artifacts and signatures, and tried a couple of
> Python pipelines from an install of a wheel in a fresh virtual
> environment. All looks good to me.
>
> On Thu, Oct 22, 2020 at 4:54 PM Tyson Hamilton  wrote:
> >
> > +1
> >
> > I went through the Nexmark queries and validated the results.
> >
> > On Thu, Oct 22, 2020 at 4:43 PM Pablo Estrada 
> wrote:
> >>
> >> +1 (binding)
> >> Validated Java quickstart for Direct, Dataflow, Spark runners.
> >> Tried out a few interactive queries on InteractiveRunner on Ipython.
> >> Best
> >> -P.
> >>
> >> On Thu, Oct 22, 2020 at 4:27 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >>>
> >>> +1.
> >>>
> >>> Verified the internal container images for Dataflow and verified that
> the release artifacts are not installable on Python 2 and Python 3.5 (which
> could otherwise break Beam Py2/Py3.5 users who don't set an upper bound on
> Beam).
> >>>
> >>> On Wed, Oct 21, 2020 at 1:14 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
> >>>>
> >>>> +1 (non-binding).
> >>>>
> >>>> Validated Java quickstart for Direct/Dataflow runners and x-lang
> Kafka/SQL.
> >>>>
> >>>> Thanks,
> >>>> Cham
> >>>>
> >>>> On Wed, Oct 21, 2020 at 6:03 AM Ismaël Mejía 
> wrote:
> >>>>>
> >>>>> Unrelated to the vote, but related to the Java 8/11 issue.
> >>>>>
> >>>>> We have some 'forward' compatibility tests that rely on the Beam
> daily SNAPSHOT jars
> >>>>> and they starting failing two days ago, it seems the SNAPSHOTs are
> now built also with Java 11  (not sure if related)
> >>>>>
> >>>>> Filled https://issues.apache.org/jira/browse/BEAM-11080 in case
> someone can take a look
> >>>>>
> >>>>> The SNAPSHOTs should be built with Java 8 too. We use these for
> forward
> >>>>> compatibility tests and they have helped us find multiple
> regressions in the
> >>>>> past.
> >>>>>
> >>>>> On Wed, Oct 21, 2020 at 3:42 AM Ahmet Altay 
> wrote:
> >>>>>>
> >>>>>> +1 - I verified python quickstarts.
> >>>>>>
> >>>>>> On Tue, Oct 20, 2020 at 11:36 AM Robin Qiu 
> wrote:
> >>>>>>>
> >>>>>>> Hi everyone,
> >>>>>>> Please review and vote on the release candidate #2 for the version
> 2.25.0, as follows:
> >>>>>>> [ ] +1, Approve the release
> >>>>>>> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>>>>>>
> >>>>>>>
> >>>>>>> The complete staging area is available for your review, which
> includes:
> >>>>>>> * JIRA release notes [1],
> >>>>>>> * the official Apache source release to be deployed to
> dist.apache.org [2], which is signed with the key with fingerprint
> AD70476B9D1AF3EFEC2208165952E71AACAF911D [3],
> >>>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>>>>>> * source code tag "v2.25.0-RC2" [5],
> >>>>>>> * website pull request listing the release [6], publishing the API
> reference manual [7], and the blog post [8].
> >>>>>>> * Java artifacts were built with Maven 3.5.3 and OpenJDK 1.8.0
> >>>>>>> * Python artifacts are deployed along with the source release to
> the dist.apache.org [2].
> >>>>>>> * Validation sheet with a tab for 2.25.0 release to help with
> validation [9].
> >>>>>>> * Docker images published to Docker Hub [10].
> >>>>>>>
> >>>>>>> The vote will be open for at least 72 hours. It is adopted by
> majority approval, with at least 3 PMC affirmative votes.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Robin
> >>>>>>>
> >>>>>>> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12347147
> >>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.25.0/
> >>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>>>>> [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1142
> >>>>>>> [5] https://github.com/apache/beam/tree/v2.25.0-RC2
> >>>>>>> [6] https://github.com/apache/beam/pull/13130
> >>>>>>> [7] https://github.com/apache/beam-site/pull/608
> >>>>>>> [8] https://github.com/apache/beam/pull/13131
> >>>>>>> [9]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1494345946
> >>>>>>> [10] https://hub.docker.com/search?q=apache%2Fbeam=image
>

[VOTE] Release 2.25.0, release candidate #2

[Robin Qiu]

Hi everyone,
Please review and vote on the release candidate #2 for the version 2.25.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint
AD70476B9D1AF3EFEC2208165952E71AACAF911D [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.25.0-RC2" [5],
* website pull request listing the release [6], publishing the API
reference manual [7], and the blog post [8].
* Java artifacts were built with Maven 3.5.3 and OpenJDK 1.8.0
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
* Validation sheet with a tab for 2.25.0 release to help with validation
[9].
* Docker images published to Docker Hub [10].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Robin

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12347147
[2] https://dist.apache.org/repos/dist/dev/beam/2.25.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1142
[5] https://github.com/apache/beam/tree/v2.25.0-RC2
[6] https://github.com/apache/beam/pull/13130
[7] https://github.com/apache/beam-site/pull/608
[8] https://github.com/apache/beam/pull/13131
[9]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1494345946
[10] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: [VOTE] Release 2.25.0, release candidate #1

[Robin Qiu]

Thank you all for the feedback! I will work on a RC2 to address these
problems.

On Mon, Oct 19, 2020 at 7:38 AM Ismaël Mejía  wrote:

> -1
>
> > * Java artifacts were built with Maven 3.5.3 and OpenJDK/Oracle JDK
> 11.0.8.
>
> As from discussion on 2.24.0 RC1 we MUST build Java artifacts with Java 8
> otherwise we will not have guaranteed compatibility with Java 8.
> We should update the release guide to make this explicit for the person
> preparing the release so this does not happen again and eventually include
> some validation for this in the build.
>
> I validated that this is broken the same way as before by running a
> pipeline with Direct runner using the 2.25.0 jars inside of a Java 8 docker.
> The Exception is the same.
>
> 2020-10-19 16:14:23,427 [direct-runner-worker] ERROR
> org.apache.beam.runners.direct.DirectTransformExecutor  - Error occurred
> within org.apache.beam.runners.direct.DirectTransformExecutor@6babef80
> java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
> at
> org.apache.beam.sdk.util.BufferedElementCountingOutputStream.outputBuffer(BufferedElementCountingOutputStream.java:197)
> at
> org.apache.beam.sdk.util.BufferedElementCountingOutputStream.flush(BufferedElementCountingOutputStream.java:180)
> at
> org.apache.beam.sdk.util.BufferedElementCountingOutputStream.finish(BufferedElementCountingOutputStream.java:119)
> at
> org.apache.beam.sdk.coders.IterableLikeCoder.encode(IterableLikeCoder.java:127)
> at
> org.apache.beam.sdk.coders.IterableLikeCoder.encode(IterableLikeCoder.java:60)
> at org.apache.beam.sdk.coders.Coder.encode(Coder.java:136)
>
>
> On Sat, Oct 17, 2020 at 3:57 AM Ahmet Altay  wrote:
>
>> I verified python quickstarts. There is a minor issue and I will update
>> my vote after that.
>>
>> Python batch pipelines on Dataflow are failing with the following error:
>> "RuntimeError: Beam SDK base version 2.25.0 does not match Dataflow Python
>> worker version 2.25.0.dev. Please check Dataflow worker startup logs and
>> make sure that correct version of Beam SDK is installed."
>>
>> Same issue happened during 2.24.0 and was fixed quickly. We may need to
>> update the release guide to prevent this error in the future. (/cc +Daniel
>> Oliveira  and +Valentyn Tymofieiev
>>  fixed the issue for 2.24.0).
>>
>> Ahmet
>>
>> On Fri, Oct 16, 2020 at 2:36 PM Robin Qiu  wrote:
>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version
>>> 2.25.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with fingerprint
>>> AD70476B9D1AF3EFEC2208165952E71AACAF911D [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.25.0-RC1" [5],
>>> * website pull request listing the release [6], publishing the API
>>> reference manual [7], and the blog post [8].
>>> * Java artifacts were built with Maven 3.5.3 and OpenJDK/Oracle JDK
>>> 11.0.8.
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2].
>>> * Validation sheet with a tab for 2.25.0 release to help with validation
>>> [9].
>>> * Docker images published to Docker Hub [10].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Robin
>>>
>>> [1]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12347147
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.25.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1139/
>>> [5] https://github.com/apache/beam/tree/v2.25.0-RC1
>>> [6] https://github.com/apache/beam/pull/13130
>>> [7] https://github.com/apache/beam-site/pull/608
>>> [8] https://github.com/apache/beam/pull/13131
>>> [9]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1494345946
>>> [10] https://hub.docker.com/search?q=apache%2Fbeam=image
>>>
>>

[VOTE] Release 2.25.0, release candidate #1

[Robin Qiu]

Hi everyone,
Please review and vote on the release candidate #1 for the version 2.25.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint
AD70476B9D1AF3EFEC2208165952E71AACAF911D [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.25.0-RC1" [5],
* website pull request listing the release [6], publishing the API
reference manual [7], and the blog post [8].
* Java artifacts were built with Maven 3.5.3 and OpenJDK/Oracle JDK 11.0.8.
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
* Validation sheet with a tab for 2.25.0 release to help with validation
[9].
* Docker images published to Docker Hub [10].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Robin

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12347147
[2] https://dist.apache.org/repos/dist/dev/beam/2.25.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1139/
[5] https://github.com/apache/beam/tree/v2.25.0-RC1
[6] https://github.com/apache/beam/pull/13130
[7] https://github.com/apache/beam-site/pull/608
[8] https://github.com/apache/beam/pull/13131
[9]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1494345946
[10] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: beam-sdks-java-bom.pom cannot be signed after upgrade to Gradle 6

[Robin Qiu]

For now to get the release unblocked, I will try to manually generate the
missing signature. I created a JIRA (
https://issues.apache.org/jira/browse/BEAM-11068) for this issue. @Kenneth
Knowles  Could you help find an owner?

Garrett, I tried to hack to generate the asc file at the right stage as you
mentioned but wasn't successful. Any help on how to do this would be
appreciated.

On Wed, Oct 14, 2020 at 9:25 PM Garrett Jones 
wrote:

> My knowledge of this stuff has gotten rusty. What I remember: I had to do
> special work since Gradle didn't have support for generating BOMs. The
> generation process is kind of finicky because various stages need to run in
> order and that order isn't obvious based on the structure of the build
> rules; it must be that the signing stage doesn't see the generated pom
> file. If you just want to get the release unblocked, you might have to
> either 1) add a hack to generate the asc file at the right stage, or 2)
> downgrade Gradle. Also, it's worth checking to see if Gradle 6 now has
> support for generating BOMs, which it didn't used to have, but that would
> be a bigger change than you might want to accept for unblocking the release.
>
> -- Garrett Jones [ go/garrettjones-user-manual
> <https://goto.google.com/garrettjones-user-manual> |
> go/reconsider-inline-replies
> <https://goto.google.com/reconsider-inline-replies> ]
>
>
> On Wed, Oct 14, 2020 at 8:07 PM Kenneth Knowles  wrote:
>
>> +Garrett Jones  who appears to have been
>> involved and +Michael Luckey  who has touched this
>> build.gradle (according to git) and is a bit of a build wizard.
>>
>> Kenn
>>
>> On Wed, Oct 14, 2020 at 7:34 PM Robin Qiu  wrote:
>>
>>> Hi all,
>>>
>>> I am working on creating Beam 2.25.0 RC1. The repo I created (
>>> https://repository.apache.org/#stagingRepositories) failed to close
>>> because
>>>
>>> Missing Signature:
>>>> '/org/apache/beam/beam-sdks-java-bom/2.25.0/beam-sdks-java-bom-2.25.0.pom.asc'
>>>> does not exist for 'beam-sdks-java-bom-2.25.0.pom'.
>>>
>>>
>>> I checked pom files in other modules and their signatures are present,
>>> so I think this problem only happens to beam-sdks-java-bom-2.25.0.pom. Also
>>> this has not happened in previous releases. I suspect this is caused by the
>>> recent upgrade to Gradle 6.
>>>
>>> I found that
>>> https://github.com/apache/beam/blob/master/sdks/java/bom/build.gradle
>>> does something special. It does not use a generated pom, instead it uses
>>> its own template and copies that
>>> to sdks/java/bom/build/publications/mavenJava/ as pom-default.xml. When I
>>> run the publish task locally, I found in
>>> sdks/java/bom/build/publications/mavenJava/ that the pom-default.xml is
>>> signed (i.e. pom-default.xml.asc is present), but
>>> beam-sdks-java-bom-2.25.0.pom is not signed (i.e.
>>> beam-sdks-java-bom-2.25.0.pom.asc is not present) in the output repository.
>>>
>>> I tried to understand how the Gradle plugins (maven-publish and signing)
>>> work and changed a few different configurations in
>>> https://github.com/apache/beam/blob/master/sdks/java/bom/build.gradle
>>> but that didn't help. Does anyone have context on this issue or have any
>>> suggestions that I could try? This is currently blocking the 2.25.0 release.
>>>
>>> Thanks,
>>> Robin
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "datapls-plat-team" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datapls-plat-team+unsubscr...@google.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/google.com/d/msgid/datapls-plat-team/CA%2BVw4HU8wfux9OUf9Kt_V5q221WrEhHqYwaq_fCSk4ViqddJuA%40mail.gmail.com
>>> <https://groups.google.com/a/google.com/d/msgid/datapls-plat-team/CA%2BVw4HU8wfux9OUf9Kt_V5q221WrEhHqYwaq_fCSk4ViqddJuA%40mail.gmail.com?utm_medium=email_source=footer>
>>> .
>>>
>>

beam-sdks-java-bom.pom cannot be signed after upgrade to Gradle 6

[Robin Qiu]

Hi all,

I am working on creating Beam 2.25.0 RC1. The repo I created (
https://repository.apache.org/#stagingRepositories) failed to close because

Missing Signature:
> '/org/apache/beam/beam-sdks-java-bom/2.25.0/beam-sdks-java-bom-2.25.0.pom.asc'
> does not exist for 'beam-sdks-java-bom-2.25.0.pom'.


I checked pom files in other modules and their signatures are present, so I
think this problem only happens to beam-sdks-java-bom-2.25.0.pom. Also this
has not happened in previous releases. I suspect this is caused by the
recent upgrade to Gradle 6.

I found that
https://github.com/apache/beam/blob/master/sdks/java/bom/build.gradle does
something special. It does not use a generated pom, instead it uses its own
template and copies that to sdks/java/bom/build/publications/mavenJava/ as
pom-default.xml. When I run the publish task locally, I found in
sdks/java/bom/build/publications/mavenJava/ that the pom-default.xml is
signed (i.e. pom-default.xml.asc is present), but
beam-sdks-java-bom-2.25.0.pom is not signed (i.e.
beam-sdks-java-bom-2.25.0.pom.asc is not present) in the output repository.

I tried to understand how the Gradle plugins (maven-publish and signing)
work and changed a few different configurations in
https://github.com/apache/beam/blob/master/sdks/java/bom/build.gradle but
that didn't help. Does anyone have context on this issue or have any
suggestions that I could try? This is currently blocking the 2.25.0 release.

Thanks,
Robin

Re: [UPDATE] Beam 2.25.0 release progress update

[Robin Qiu]

Thank you all for helping! I have been added to the beammaintainers team in
Docker Hub.

I am working on building the containers for the Dataflow worker now. After
that I will work on creating RC1.

On Tue, Oct 6, 2020 at 1:19 PM Kyle Weaver  wrote:

> > +1 to the idea. We discussed it in the dev list [1]. I do not believe we
> discussed it with INFRA.
>
> We would have to collect everyone's Docker hub usernames first.
>
> > Could someone help Robin on this ticket? Would reaching out to infra on
> slack help?
>
> I messaged #asfinfra on Slack.
>

Re: [UPDATE] Beam 2.25.0 release progress update

[Robin Qiu]

One thing that could block my work soon is that I am still not added to
the beammaintainers team in Docker Hub. Pablo has helped me created a infra
ticket (https://issues.apache.org/jira/browse/INFRA-20900) but it has not
made progress. Could someone help with expediting this? Thanks!

On Mon, Oct 5, 2020 at 3:39 PM Robin Qiu  wrote:

> Update: the release blockers are all resolved and the fixes are
> cherry-picked into the release branch. I will work on creating a release
> candidate soon.
>
> On Fri, Sep 25, 2020 at 11:36 AM Robin Qiu  wrote:
>
>> Update: here is the list of release blockers to be resolved
>> https://issues.apache.org/jira/projects/BEAM/versions/12347147
>>
>> On Thu, Sep 24, 2020 at 5:00 PM Robin Qiu  wrote:
>>
>>> I see. Thanks for letting me know!
>>>
>>> On Thu, Sep 24, 2020 at 4:59 PM Robert Bradshaw 
>>> wrote:
>>>
>>>> FYI, I'm waiting for Daniel's CLs to go in before starting my own.
>>>>
>>>> On Thu, Sep 24, 2020 at 9:29 AM Robert Bradshaw 
>>>> wrote:
>>>>
>>>>> OK, I'll get that started.
>>>>>
>>>>> On Thu, Sep 24, 2020 at 9:26 AM Robin Qiu  wrote:
>>>>>
>>>>>> No I haven't. Could you please do that? I wanted to sync with you
>>>>>> yesterday but I saw you were ooo.
>>>>>>
>>>>>> On Thu, Sep 24, 2020 at 9:03 AM Robert Bradshaw 
>>>>>> wrote:
>>>>>>
>>>>>>> [just you]
>>>>>>>
>>>>>>> Have you started an import at this commit, or should I?
>>>>>>>
>>>>>>> On Wed, Sep 23, 2020 at 5:51 PM Robin Qiu 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Update: The 2.25.0 release branch has been cut.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 23, 2020 at 3:34 PM Robin Qiu 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello everyone,
>>>>>>>>>
>>>>>>>>> I am starting this email thread to share with you updates on Beam
>>>>>>>>> 2.25.0 release.
>>>>>>>>>
>>>>>>>>> Update: I plan to cut the release branch in a few hours, according
>>>>>>>>> to the release calendar [1].
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Robin
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>

Re: [UPDATE] Beam 2.25.0 release progress update

[Robin Qiu]

Update: here is the list of release blockers to be resolved
https://issues.apache.org/jira/projects/BEAM/versions/12347147

On Thu, Sep 24, 2020 at 5:00 PM Robin Qiu  wrote:

> I see. Thanks for letting me know!
>
> On Thu, Sep 24, 2020 at 4:59 PM Robert Bradshaw 
> wrote:
>
>> FYI, I'm waiting for Daniel's CLs to go in before starting my own.
>>
>> On Thu, Sep 24, 2020 at 9:29 AM Robert Bradshaw 
>> wrote:
>>
>>> OK, I'll get that started.
>>>
>>> On Thu, Sep 24, 2020 at 9:26 AM Robin Qiu  wrote:
>>>
>>>> No I haven't. Could you please do that? I wanted to sync with you
>>>> yesterday but I saw you were ooo.
>>>>
>>>> On Thu, Sep 24, 2020 at 9:03 AM Robert Bradshaw 
>>>> wrote:
>>>>
>>>>> [just you]
>>>>>
>>>>> Have you started an import at this commit, or should I?
>>>>>
>>>>> On Wed, Sep 23, 2020 at 5:51 PM Robin Qiu  wrote:
>>>>>
>>>>>> Update: The 2.25.0 release branch has been cut.
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 23, 2020 at 3:34 PM Robin Qiu  wrote:
>>>>>>
>>>>>>> Hello everyone,
>>>>>>>
>>>>>>> I am starting this email thread to share with you updates on Beam
>>>>>>> 2.25.0 release.
>>>>>>>
>>>>>>> Update: I plan to cut the release branch in a few hours, according
>>>>>>> to the release calendar [1].
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Robin
>>>>>>>
>>>>>>> [1]
>>>>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>

DockerHub Push Permission

[Robin Qiu]

Hi DockerHub Admins,

I need push permission to proceed with release, can you please add me to
maintainer team?
My docker hub ID is: robinyqiu

Thanks,
Robin

Re: [UPDATE] Beam 2.25.0 release progress update

[Robin Qiu]

Update: The 2.25.0 release branch has been cut.


On Wed, Sep 23, 2020 at 3:34 PM Robin Qiu  wrote:

> Hello everyone,
>
> I am starting this email thread to share with you updates on Beam 2.25.0
> release.
>
> Update: I plan to cut the release branch in a few hours, according to the
> release calendar [1].
>
> Thanks,
> Robin
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>

[UPDATE] Beam 2.25.0 release progress update

[Robin Qiu]

Hello everyone,

I am starting this email thread to share with you updates on Beam 2.25.0
release.

Update: I plan to cut the release branch in a few hours, according to the
release calendar [1].

Thanks,
Robin

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com

[PROPOSAL] Preparing for Beam 2.25.0 release

[Robin Qiu]

Hello everyone,

The next Beam release (2.25.0) is scheduled to be cut on September 23
according to the release calendar [1].

I'd like to volunteer myself to handle this release. I plan on cutting the
branch on that date and cherry-picking in release-blocking fixes
afterwards. So unresolved release blocking JIRA issues should have
their "Fix Version/s" marked as "2.25.0".

Any comments or objections?

Thanks,
Robin Qiu

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com

Re: [ANNOUNCE] New committer: Heejong Lee

[Robin Qiu]

Congrats, Heejong!

On Tue, Sep 8, 2020 at 11:23 AM Valentyn Tymofieiev 
wrote:

> Congratulations, Heejong!
>
> On Tue, Sep 8, 2020 at 11:14 AM Ahmet Altay  wrote:
>
>> Hi everyone,
>>
>> Please join me and the rest of the Beam PMC in welcoming
>> a new committer: Heejong Lee .
>>
>> Heejong has been active in the community for more than 2 years, worked on
>> various IOs (parquet, kafka, file, pubsub) and most recently worked on
>> adding cross language transforms feature to Beam [1].
>>
>> In consideration of his contributions, the Beam PMC trusts him with the
>> responsibilities of a Beam committer [2].
>>
>> Thank you for your contributions Heejong!
>>
>> -Ahmet, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://issues.apache.org/jira/browse/BEAM-10634?jql=project%20%3D%20BEAM%20AND%20assignee%20in%20(heejong)%20ORDER%20BY%20resolved%20DESC%2C%20affectedVersion%20ASC%2C%20priority%20DESC%2C%20updated%20DESC
>> [2]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [Discuss] Propose Calcite Vendor Release (1.22.0)

[Robin Qiu]

+1

Thanks Rui for proposing this. Bringing in the newest version of Calcite
will also simplify our codebase [1] and resolve some existing issues [2]

[1] https://issues.apache.org/jira/browse/BEAM-9190
[2] https://issues.apache.org/jira/browse/BEAM-9191

On Thu, Mar 5, 2020 at 11:42 AM Xinyu Liu  wrote:

> Thanks, Rui! We've been waiting for the new version of Calcite which has
> the fix to unflatten the fields. Seems this version will come with it.
>
> Thanks,
> Xinyu
>
> On Thu, Mar 5, 2020 at 12:41 AM Ismaël Mejía  wrote:
>
>> The calcite vote already passed so this is good to go, thanks for
>> volunteering Rui.
>>
>> https://lists.apache.org/thread.html/r4962a4a2bacf481f2ee1064806b78829d96385c2e4a3c0ecb24a55a2%40%3Cdev.calcite.apache.org%3E
>>
>> On Thu, Mar 5, 2020 at 8:10 AM Kai Jiang  wrote:
>> >
>> > Thanks, Rui! Big +1 for calcite vendor release (1.22.0)
>> > Curious, what's the progress of Calcite 1.22.0 official release? I saw
>> Calcite community just passes the vote for 1.22.0 rc3.
>> >
>> > Best,
>> > Kai
>> >
>> >
>> > On Wed, Mar 4, 2020 at 9:24 PM Rui Wang  wrote:
>> >>
>> >> Hi Community,
>> >>
>> >> As Calcite is closing to finish their 1.22.0 release, I want to
>> propose a Calcite vendor release and I am volunteer to be the release
>> manager.
>> >>
>> >> I will wait until next Monday(03/09) to kick off the release if there
>> is no objection.
>> >>
>> >>
>> >> Best,
>> >> Rui Wang
>>
>

Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.2 for BEAM-9252

[Robin Qiu]

+1 (verified)

On Fri, Feb 21, 2020 at 4:55 PM Robert Bradshaw  wrote:

> +1 (binding)
>
>
> On Fri, Feb 21, 2020 at 4:48 PM Ahmet Altay  wrote:
> >
> > +1
> >
> > On Fri, Feb 21, 2020 at 4:39 PM Luke Cwik  wrote:
> >>
> >> +1 (binding)
> >> I diffed the binary contents of the 0.1 jar and 0.2 jar with no changes
> to the contents of the files and can confirm that module-info.class the
> offending Main.class and Main$1.class have been removed as well.
> >>
> >> On Fri, Feb 21, 2020 at 4:38 PM Luke Cwik  wrote:
> >>>
> >>> Please review the release of the following artifacts that we vendor:
> >>>  * beam-vendor-grpc-1_26_0
> >>>
> >>> Hi everyone,
> >>> Please review and vote on the release candidate #1 for the version
> 0.2, as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * the official Apache source release to be deployed to dist.apache.org
> [1], which is signed with the key with fingerprint
> EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],
> >>> * all artifacts to be deployed to the Maven Central Repository [3],
> >>> * commit hash "91125d1d1fc1fe8c5684a486c9b6163c4ec41549" [4],
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Release Manager
> >>>
> >>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
> >>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>> [3]
> https://repository.apache.org/content/repositories/orgapachebeam-1092/
> >>> [4]
> https://github.com/apache/beam/commit/91125d1d1fc1fe8c5684a486c9b6163c4ec41549
>

Re: [ANNOUNCE] New committer: Alex Van Boxel

[Robin Qiu]

Congratulations, Alex!

On Tue, Feb 18, 2020 at 1:48 PM Valentyn Tymofieiev 
wrote:

> Congratulations!
>
> On Tue, Feb 18, 2020 at 10:38 AM Alex Van Boxel  wrote:
>
>> Thank you everyone!
>>
>>  _/
>> _/ Alex Van Boxel
>>
>>
>> On Tue, Feb 18, 2020 at 7:05 PM  wrote:
>>
>>> Congrats Alex!
>>> Jan
>>>
>>>
>>> Dne 18. 2. 2020 18:46 napsal uživatel Thomas Weise :
>>>
>>> Congratulations!
>>>
>>>
>>> On Tue, Feb 18, 2020 at 8:33 AM Ismaël Mejía  wrote:
>>>
>>> Congrats Alex! Well done!
>>>
>>> On Tue, Feb 18, 2020 at 5:10 PM Gleb Kanterov  wrote:
>>>
>>> Congratulations!
>>>
>>> On Tue, Feb 18, 2020 at 5:02 PM Brian Hulette 
>>> wrote:
>>>
>>> Congratulations Alex! Well deserved!
>>>
>>> On Tue, Feb 18, 2020 at 7:49 AM Pablo Estrada 
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming
>>> a new committer: Alex Van Boxel
>>>
>>> Alex has contributed to Beam in many ways - as an organizer for Beam
>>> Summit, and meetups - and also with the Protobuf extensions for schemas.
>>>
>>> In consideration of his contributions, the Beam PMC trusts him with the
>>> responsibilities of a Beam committer[1].
>>>
>>> Thanks for your contributions Alex!
>>>
>>> Pablo, on behalf of the Apache Beam PMC.
>>>
>>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>>
>>>

Re: [ANNOUNCE] New committer: Hannah Jiang

[Robin Qiu]

Congratulations, Hannah!

On Tue, Jan 28, 2020 at 4:50 PM Alan Myrvold  wrote:

> Congrats, Hannah
>
> On Tue, Jan 28, 2020 at 4:46 PM Connell O'Callaghan 
> wrote:
>
>> Thank you for sharing Luke!!!
>>
>> Well done and congratulations Hannah!!
>>
>> On Tue, Jan 28, 2020 at 4:45 PM Heejong Lee  wrote:
>>
>>> Congratulations! :)
>>>
>>> On Tue, Jan 28, 2020 at 4:43 PM Yichi Zhang  wrote:
>>>
 Congrats Hannah!

 On Tue, Jan 28, 2020 at 3:57 PM Yifan Zou  wrote:

> Congratulations Hannah!!
>
> On Tue, Jan 28, 2020 at 3:55 PM Boyuan Zhang 
> wrote:
>
>> Thanks for all your contributions! Congratulations~
>>
>> On Tue, Jan 28, 2020 at 3:44 PM Pablo Estrada 
>> wrote:
>>
>>> yoooho : D
>>>
>>> On Tue, Jan 28, 2020 at 3:21 PM Luke Cwik  wrote:
>>>
 Hi everyone,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Hannah Jiang

 Hannah has contributed to Beam in many ways, including work on
 building and releasing the Apache Beam SDK containers.

 In consideration of their contributions, the Beam PMC trusts them
 with the responsibilities of a Beam committer[1].

 Thanks for your contributions Hannah!

 Luke, on behalf of the Apache Beam PMC.

 [1]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>

Default values not supported in Combine.globally() if not windowed by GlobalWindows

[Robin Qiu]

Hi all,

When I run a Combine.globally() transform with a default value set, on a
PCollection windowed by a FixedWindow, I got this error message:

java.lang.IllegalStateException: Default values are not supported in
Combine.globally() if the input PCollection is not windowed by
GlobalWindows ...
(from here

)

Is this unsupported because it is not implemented yet, or is this a design
decision to not support default values with windows other than GlobalWIndow
(e.g. because the semantics is not well defined)? Where can I learn more
about this?

Best,
Robin

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Robin Qiu]

Congrats, Daniel!

On Thu, Nov 21, 2019 at 10:11 AM Gleb Kanterov  wrote:

> Congratulations!
>
> On Thu, Nov 21, 2019 at 6:24 AM Thomas Weise  wrote:
>
>> Congratulations!
>>
>>
>> On Wed, Nov 20, 2019, 7:56 PM Chamikara Jayalath 
>> wrote:
>>
>>> Congrats!!
>>>
>>> On Wed, Nov 20, 2019 at 5:21 PM Daniel Oliveira 
>>> wrote:
>>>
 Thank you everyone! I won't let you down. o7

 On Wed, Nov 20, 2019 at 2:12 PM Ruoyun Huang  wrote:

> Congrats Daniel!
>
> On Wed, Nov 20, 2019 at 1:58 PM Robert Burke 
> wrote:
>
>> Congrats Daniel! Much deserved.
>>
>> On Wed, Nov 20, 2019, 12:49 PM Udi Meiri  wrote:
>>
>>> Congrats Daniel!
>>>
>>> On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver 
>>> wrote:
>>>
 Congrats Dan! Keep up the good work :)

 On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden 
 wrote:

> Congratulations! This is great news.
>
> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang 
> wrote:
>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congrats, Daniel!
>>>
>>> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles <
>>> k...@apache.org> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Daniel Oliveira

 Daniel introduced himself to dev@ over two years ago and has
 contributed in many ways since then. Daniel has contributed to 
 general
 project health, the portability framework, and all three 
 languages: Java,
 Python SDK, and Go. I would like to particularly highlight how he 
 deleted
 12k lines of dead reference runner code [1].

 In consideration of Daniel's contributions, the Beam PMC trusts
 him with the responsibilities of a Beam committer [2].

 Thank you, Daniel, for your contributions and looking forward
 to many more!

 Kenn, on behalf of the Apache Beam PMC

 [1] https://github.com/apache/beam/pull/8380
 [2]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Robin Qiu]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ X] Angler fish

On Wed, Nov 20, 2019 at 2:45 PM Alexey Romanenko 
wrote:

> [ ] Beaver
> [x] Hedgehog
> [ ] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [x] Firefly
> [ ] Cuttlefish
> [x] Dumbo Octopus
> [ ] Angler fish
>
> On 20 Nov 2019, at 14:38, Manu Zhang  wrote:
>
> [] Beaver
> [] Hedgehog
> [x] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> On Wed, Nov 20, 2019 at 8:38 PM Maximilian Michels  wrote:
>
>> [ ] Beaver
>> [ ] Hedgehog
>> [x] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [x] Firefly
>> [x] Cuttlefish
>> [ ] Dumbo Octopus
>> [x] Angler fish
>>
>> On 20.11.19 08:18, Alex Van Boxel wrote:
>> > [ ] Beaver
>> > [ ] Hedgehog
>> > [ ] Lemur
>> > [ ] Owl
>> > [ ] Salmon
>> > [ ] Trout
>> > [ ] Robot dinosaur
>> > [ X] Firefly
>> > [ ] Cuttlefish
>> > [ ] Dumbo Octopus
>> > [ X] Angler fish
>> >
>> >   _/
>> > _/ Alex Van Boxel
>> >
>> >
>> > On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni > > > wrote:
>> >
>> > [ ] Beaver
>> > [ ] Hedgehog
>> > [ ] Lemur
>> > [ ] Owl
>> > [X] Salmon
>> > [ ] Trout
>> > [ ] Robot dinosaur
>> > [ ] Firefly
>> > [ ] Cuttlefish
>> > [X] Dumbo Octopus
>> > [X] Angler fish
>> >
>> > On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles > > > wrote:
>> >
>> > Please cast your votes of approval [1] for animals you would
>> > support as Beam mascot. The animal with the most approval will
>> > be identified as the favorite.
>> >
>> > *** Vote for as many as you like, using this checklist as a
>> > template 
>> >
>> > [ ] Beaver
>> > [ ] Hedgehog
>> > [ ] Lemur
>> > [ ] Owl
>> > [ ] Salmon
>> > [ ] Trout
>> > [ ] Robot dinosaur
>> > [ ] Firefly
>> > [ ] Cuttlefish
>> > [ ] Dumbo Octopus
>> > [ ] Angler fish
>> >
>> > This vote will remain open for at least 72 hours.
>> >
>> > Kenn
>> >
>> > [1] See
>> > https://en.wikipedia.org/wiki/Approval_voting#Description and
>> > https://www.electionscience.org/library/approval-voting/
>> >
>> >
>> >
>> > --
>> >
>> > This email may be confidential and privileged. If you received this
>> > communication by mistake, please don't forward it to anyone else,
>> > please erase all copies and attachments, and please let me know that
>> > it has gone to the wrong person.
>> >
>> > The above terms reflect a potential business arrangement, are
>> > provided solely as a basis for further discussion, and are not
>> > intended to be and do not constitute a legally binding obligation.
>> > No legally binding obligations will be created, implied, or inferred
>> > until an agreement in final form is executed in writing by all
>> > parties involved.
>> >
>>
>
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Robin Qiu]

Congrats, Brian!

On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira 
wrote:

> Congratulations Brian! It's well deserved.
>
> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko 
> wrote:
>
>> Congratulations, Brian!
>>
>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>
>>> Congratulations!
>>>
>>>
>>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan 
>>> wrote:
>>>
 Well done Brian!!!

 Kenn thank you for sharing

 On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
> wrote:
>
>> Congratulations Brian!
>> Happy to see this happening and eager to see more of your work!
>>
>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>> wrote:
>> >
>> > Congrats Brian!
>> >
>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
>> >>
>> >> Congrats Brian!
>> >>
>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>> >>
>> >> Great news!
>> >>
>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>> wrote:
>> >>>
>> >>> Congratulations!
>> >>>
>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> 
>>  Congratulations, Brian!
>> 
>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>> sunjincheng...@gmail.com> wrote:
>> >
>> > Congratulation Brian!
>> >
>> > Best,
>> > Jincheng
>> >
>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>> >>
>> >> Thanks for your contributions and congrats Brian!
>> >>
>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Brian Hulette
>> >>>
>> >>> Brian introduced himself to dev@ earlier this year and has
>> been contributing since then. His contributions to Beam include
>> explorations of integration with Arrow, standardizing coders, portability
>> for schemas, and presentations at Beam events.
>> >>>
>> >>> In consideration of Brian's contributions, the Beam PMC
>> trusts him with the responsibilities of a Beam committer [1].
>> >>>
>> >>> Thank you, Brian, for your contributions and looking forward
>> to many more!
>> >>>
>> >>> Kenn, on behalf of the Apache Beam PMC
>> >>>
>> >>> [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> This email may be confidential and privileged. If you received
>> this communication by mistake, please don't forward it to anyone else,
>> please erase all copies and attachments, and please let me know that it 
>> has
>> gone to the wrong person.
>> >>
>> >> The above terms reflect a potential business arrangement, are
>> provided solely as a basis for further discussion, and are not intended 
>> to
>> be and do not constitute a legally binding obligation. No legally binding
>> obligations will be created, implied, or inferred until an agreement in
>> final form is executed in writing by all parties involved.
>>
>
>>

Re: [VOTE] Release 2.16.0, release candidate #1

[Robin Qiu]

+1

Verified the new module sdks/java/extensions/zetasketch works (on direct
runner)

On Fri, Oct 4, 2019 at 12:41 PM Kenneth Knowles  wrote:

> +1 (binding)
>
>  - Reviewed what verifications had been done. Nice.
>  - Also did a gradle build of some targets in the archival source release
>
> The source release still does not build as a whole, as it has not since
> 2.9.0 it seems. It is not as simple as excluding website from the build,
> because it fails at configure time. Since particular artifacts can build,
> it is not a blocker, but I've taken
> https://issues.apache.org/jira/browse/BEAM-6228 and upgraded to critical
> and put 2.17.0 as Release Version.
>
> Kenn
>
> On Fri, Oct 4, 2019 at 10:27 AM Pablo Estrada  wrote:
>
>> Hi all,
>> I looked at https://issues.apache.org/jira/browse/BEAM-8303, and it
>> seems like the user has a workaround - is that correct?
>> If that's the case, then I vote +1.
>>
>> @Max - lmk if you'd like to discuss further, but for now my vote is on +1.
>> Best
>> -P.
>>
>> On Fri, Oct 4, 2019 at 9:29 AM Mark Liu  wrote:
>>
>>> +1 (forgot to vote)
>>>
>>> I also triggered Java Nexmark on direct, dataflow, spark and flink
>>> runner. Didn't saw performance regression from the dashboard (
>>> https://apache-beam-testing.appspot.com/dashboard-admin)
>>>
>>> On Fri, Oct 4, 2019 at 8:23 AM Mark Liu  wrote:
>>>
 Thanks for the validation work! I validated following:

 - Java Quickstart on direct, dataflow,spark local, flink local runner
 - Java mobile gaming on direct and dataflow runner
 - Python Quickstart in batch and streaming in py2/3.5/3.6/3.7 using
 wheals/zip
 - Python Mobile Game in batch/streaming in py2/3.5/3.6/3.7 using
 wheals/zip on direct and dataflow runner

 Mark

 On Thu, Oct 3, 2019 at 6:57 PM Ahmet Altay  wrote:

> I see most of the release validations have been completed and marked
> in the spreadsheet. Thank you all for doing that. If you have not
> validated/voted yet please take a look at the release candidate.
>
> On Thu, Oct 3, 2019 at 7:59 AM Thomas Weise  wrote:
>
>> I think there is a different reason why the release manager should
>> probably merge/approve all PRs that go into the release branch while the
>> release is in progress:
>>
>> If/when the need arises for another RC, then only those changes
>> should be included that are deemed blockers or explicitly agreed. 
>> Otherwise
>> the release can potentially be delayed by modifications that invalidate
>> prior verification or introduce new instability.
>>
>
> I agree with this reasoning. It expresses my concern in a more clear
> way.
>
>
>>
>> Thomas
>>
>>
>> On Thu, Oct 3, 2019 at 3:12 AM Maximilian Michels 
>> wrote:
>>
>>>  > For the next time, may I suggest asking release manager to do the
>>>  > merging to the release branch. We do not know whether there will
>>> be an
>>>  > RC2 or not. And if there will not be an RC2 release branch as of
>>> now
>>>  > does not directly correspond to what will be released.
>>>
>>> The ground truth for releases are the release tags, not the release
>>> branches. Downstream projects should not depend on the release
>>> branches.
>>> Release branches are merely important for the process of creating a
>>> release, but they lose validity after the RC has been created and
>>> released.
>>>
>>> On 02.10.19 11:45, Ahmet Altay wrote:
>>> > +1 (validated python quickstarts). Thank you Mark.
>>> >
>>> > On Wed, Oct 2, 2019 at 10:49 AM Maximilian Michels >> > > wrote:
>>> >
>>> > Thanks for preparing the release, Mark! I would like to address
>>> > https://issues.apache.org/jira/browse/BEAM-8303 in the
>>> release. I've
>>> > already merged the fix to the release-2.16.0 branch. If we do
>>> another
>>> > RC, we could include it. As a user is blocked on this, I would
>>> not vote
>>> > +1 for this RC, but I also do not want to block the release
>>> process.
>>> >
>>> >
>>> > Max, thank you for the clear communication for the importance and
>>> at the
>>> > same time non-blocking status of the issue.
>>> >
>>> > For the next time, may I suggest asking release manager to do the
>>> > merging to the release branch. We do not know whether there will
>>> be an
>>> > RC2 or not. And if there will not be an RC2 release branch as of
>>> now
>>> > does not directly correspond to what will be released.
>>> >
>>> >
>>> > On 01.10.19 09:18, Mark Liu wrote:
>>> >  > Hi everyone,
>>> >  >
>>> >  > Please review and vote on the release candidate #1 for the
>>> version
>>> >  > 2.16.0, as follows:
>>> >  > [ ] +1, Approve the release
>>> >  > [ ] -1, Do 

Re: [ANNOUNCE] New committer: Alan Myrvold

[Robin Qiu]

Congrats, Alan!

On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang 
wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang  wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang  wrote:
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada 
>>> wrote:
>>>
 Yooh! : D

 On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou  wrote:

> Congratulations, Alan!
>
> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay  wrote:
>
>> Hi,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Alan Myrvold
>>
>> Alan has been a long time Beam contributor. His contributions made
>> Beam more productive and friendlier [1] for all contributors with
>> significant improvements to Beam release process, automation, and
>> infrastructure.
>>
>> In consideration of Alan's contributions, the Beam PMC trusts him
>> with the responsibilities of a Beam committer [2].
>>
>> Thank you, Alan, for your contributions and looking forward to many
>> more!
>>
>> Ahmet, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>> [2] https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

[Robin Qiu]

Thank you Valentyn! Congratulations!

On Mon, Aug 26, 2019 at 2:28 PM Robert Bradshaw  wrote:

> Hi,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Valentyn Tymofieiev
>
> Valentyn has made numerous contributions to Beam over the last several
> years (including 100+ pull requests), most recently pushing through
> the effort to make Beam compatible with Python 3. He is also an active
> participant in design discussions on the list, participates in release
> candidate validation, and proactively helps keep our tests green.
>
> In consideration of Valentyn's contributions, the Beam PMC trusts him
> with the responsibilities of a Beam committer [1].
>
> Thank you, Valentyn, for your contributions and looking forward to many
> more!
>
> Robert, on behalf of the Apache Beam PMC
>
> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: [ANNOUNCE] Beam 2.15.0 Released!

[Robin Qiu]

Thank you Yifan!

On Fri, Aug 23, 2019 at 11:05 AM Rui Wang  wrote:

> Thank you Yifan!
>
> -Rui
>
> On Fri, Aug 23, 2019 at 9:21 AM Pablo Estrada  wrote:
>
>> Thanks Yifan!
>>
>> On Fri, Aug 23, 2019 at 8:54 AM Connell O'Callaghan 
>> wrote:
>>
>>>
>>> +1 thank you Yifan!!!
>>>
>>> On Fri, Aug 23, 2019 at 8:49 AM Ahmet Altay  wrote:
>>>
 Thank you Yifan!

 On Fri, Aug 23, 2019 at 8:00 AM Yifan Zou  wrote:

> The Apache Beam team is pleased to announce the release of version
> 2.15.0.
>
> Apache Beam is an open source unified programming model to define and
> execute data processing pipelines, including ETL, batch and stream
> (continuous) processing. See https://beam.apache.org
>
> You can download the release here:
>
> https://beam.apache.org/get-started/downloads/
>
> This release includes bug fixes, features, and improvements detailed on
> the Beam blog:
> https://beam.apache.org/blog/2019/08/22/beam-2.15.0.html
>
> Thanks to everyone who contributed to this release, and we hope you
> enjoy
> using Beam 2.15.0.
>
> Yifan Zou
>

Re: [ANNOUNCE] New committer: Rui Wang

[Robin Qiu]

Congratulations, Rui!

On Wed, Aug 7, 2019 at 5:03 AM Valentyn Tymofieiev 
wrote:

> Congrats, Rui!
>
> On Wed, Aug 7, 2019 at 1:00 PM Ismaël Mejía  wrote:
>
>> Congrats Rui!
>>
>> On Wed, Aug 7, 2019, 11:37 AM Gleb Kanterov  wrote:
>>
>>> Congratulations Rui! Well done!
>>>
>>> On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan 
>>> wrote:
>>>
 Well done Rui!!!

 On Tue, Aug 6, 2019 at 15:41 Chamikara Jayalath 
 wrote:

> Congrats Rui.
>
> On Tue, Aug 6, 2019 at 2:00 PM Melissa Pashniak 
> wrote:
>
>> Congrats Rui!
>>
>> On Tue, Aug 6, 2019 at 1:37 PM Yichi Zhang  wrote:
>>
>>> Congrats Rui!
>>>
>>> On Tue, Aug 6, 2019 at 1:29 PM Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> wrote:
>>>
 Congratulations, Rui! Thank you for your contributions to Beam!

 On Tue, Aug 6, 2019 at 10:35 AM sridhar inuog <
 sridharin...@gmail.com> wrote:

> Congratulations Rui!
>
> On Tue, Aug 6, 2019 at 12:09 PM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Congrats Rui!
>>
>> On Tue, Aug 6, 2019 at 9:50 AM Yifan Zou 
>> wrote:
>>
>>> Congratulations Rui!
>>>
>>> On Tue, Aug 6, 2019 at 9:47 AM Anton Kedin 
>>> wrote:
>>>
 Congrats!

 On Tue, Aug 6, 2019, 9:36 AM Ankur Goenka 
 wrote:

> Congratulations Rui!
> Well deserved 
>
> On Tue, Aug 6, 2019 at 9:35 AM Ahmet Altay 
> wrote:
>
>> Hi,
>>
>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Rui Wang.
>>
>> Rui has been an active contributor since May 2018. Rui has
>> been very active in Beam SQL [1] and continues to help out on 
>> user@
>> and StackOverflow. Rui is one of the top answerers for 
>> apache-beam tag [2].
>>
>> In consideration of Rui's contributions, the Beam PMC trusts
>> him with the responsibilities of a Beam committer [3].
>>
>> Thank you, Rui, for your contributions and looking forward to
>> many more!
>>
>> Ahmet, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://github.com/apache/beam/pulls?q=is%3Apr+author%3Aamaliujia
>> [2] https://stackoverflow.com/tags/apache-beam/topusers
>> [3] https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Re: [ANNOUNCE] New committer: Kyle Weaver

[Robin Qiu]

Congratulations, Kyle!

On Wed, Aug 7, 2019 at 5:04 AM Valentyn Tymofieiev 
wrote:

> Congrats, Kyle!
>
> On Wed, Aug 7, 2019 at 1:01 PM Ismaël Mejía  wrote:
>
>> Congrats Kyle, well deserved :clap: !
>>
>> On Wed, Aug 7, 2019, 11:22 AM Gleb Kanterov  wrote:
>>
>>> Congratulations!
>>>
>>> On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan 
>>> wrote:
>>>
 Well done congratulations Kyle!!!

 On Tue, Aug 6, 2019 at 21:58 Thomas Weise  wrote:

> Congrats!
>
> On Tue, Aug 6, 2019, 7:24 PM Reza Rokni  wrote:
>
>> Congratz!
>>
>> On Wed, 7 Aug 2019 at 06:40, Chamikara Jayalath 
>> wrote:
>>
>>> Congrats!!
>>>
>>> On Tue, Aug 6, 2019 at 3:33 PM Udi Meiri  wrote:
>>>
 Congrats Kyle!

 On Tue, Aug 6, 2019 at 2:00 PM Melissa Pashniak <
 meliss...@google.com> wrote:

> Congratulations Kyle!
>
> On Tue, Aug 6, 2019 at 1:36 PM Yichi Zhang 
> wrote:
>
>> Congrats Kyle!
>>
>> On Tue, Aug 6, 2019 at 1:29 PM Aizhamal Nurmamat kyzy <
>> aizha...@google.com> wrote:
>>
>>> Thank you, Kyle! And congratulations :)
>>>
>>> On Tue, Aug 6, 2019 at 10:09 AM Hannah Jiang <
>>> hannahji...@google.com> wrote:
>>>
 Congrats Kyle!

 On Tue, Aug 6, 2019 at 9:52 AM David Morávek <
 david.mora...@gmail.com> wrote:

> Congratulations Kyle!!
>
> Sent from my iPhone
>
> On 6 Aug 2019, at 18:47, Anton Kedin  wrote:
>
> Congrats!
>
> On Tue, Aug 6, 2019, 9:37 AM Ankur Goenka 
> wrote:
>
>> Congratulations Kyle!
>>
>> On Tue, Aug 6, 2019 at 9:35 AM Ahmet Altay 
>> wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Kyle Weaver.
>>>
>>> Kyle has been contributing to Beam for a while now. And in
>>> that time period Kyle got the portable spark runner feature 
>>> complete for
>>> batch processing. [1]
>>>
>>> In consideration of Kyle's contributions, the Beam PMC
>>> trusts him with the responsibilities of a Beam committer
>>>  [2].
>>>
>>> Thank you, Kyle, for your contributions and looking forward
>>> to many more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/c43678fc24c9a1dc9f48c51c51950aedcb9bc0fd3b633df16c3d595a@%3Cuser.beam.apache.org%3E
>>> [2] https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>
>>
>> --
>>
>> This email may be confidential and privileged. If you received this
>> communication by mistake, please don't forward it to anyone else, please
>> erase all copies and attachments, and please let me know that it has gone
>> to the wrong person.
>>
>> The above terms reflect a potential business arrangement, are
>> provided solely as a basis for further discussion, and are not intended 
>> to
>> be and do not constitute a legally binding obligation. No legally binding
>> obligations will be created, implied, or inferred until an agreement in
>> final form is executed in writing by all parties involved.
>>
>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Re: [ANNOUNCE] New committer: Jan Lukavský

[Robin Qiu]

Congrats!

On Wed, Jul 31, 2019 at 10:31 AM Aizhamal Nurmamat kyzy 
wrote:

> Congratulations, Jan! Thank you for your contributions!
>
> On Wed, Jul 31, 2019 at 10:04 AM Tanay Tummalapalli 
> wrote:
>
>> Congratulations!
>>
>> On Wed, Jul 31, 2019 at 10:05 PM Ahmet Altay  wrote:
>>
>>> Congratulations Jan! Thank you for your contributions!
>>>
>>> On Wed, Jul 31, 2019 at 2:30 AM Ankur Goenka  wrote:
>>>
 Congratulations Jan!

 On Wed, Jul 31, 2019, 1:23 AM David Morávek  wrote:

> Congratulations Jan, well deserved! ;)
>
> D.
>
> On Wed, Jul 31, 2019 at 10:17 AM Ryan Skraba  wrote:
>
>> Congratulations Jan!
>>
>> On Wed, Jul 31, 2019 at 10:10 AM Ismaël Mejía 
>> wrote:
>> >
>> > Hi,
>> >
>> > Please join me and the rest of the Beam PMC in welcoming a new
>> > committer: Jan Lukavský.
>> >
>> > Jan has been contributing to Beam for a while, he was part of the
>> team
>> > that contributed the Euphoria DSL extension, and he has done
>> > interesting improvements for the Spark and Direct runner. He has
>> also
>> > been active in the community discussions around the Beam model and
>> > other subjects.
>> >
>> > In consideration of Jan's contributions, the Beam PMC trusts him
>> with
>> > the responsibilities of a Beam committer [1].
>> >
>> > Thank you, Jan, for your contributions and looking forward to many
>> more!
>> >
>> > Ismaël, on behalf of the Apache Beam PMC
>> >
>> > [1] https://beam.apache.org/committer/committer
>>
>

Re: [POPOSAL] Integrate BigQuery-compatible HyperLogLog algorithm into Beam

[Robin Qiu]

Quick update: the PR implementing this feature has been sent out:
https://github.com/apache/beam/pull/9144. The design doc is also revamped
to reflect the design decisions we have made.

On Tue, Jun 25, 2019 at 2:05 PM Robin Qiu  wrote:

> Can you please add this to the design documents webpage.
>> https://beam.apache.org/contribute/design-documents/
>>
>
> Thanks for the reminder. Done! (https://github.com/apache/beam/pull/8947)
>
>
>> I am not sure if this feature should go into 'sdks/java/core' because
>> it seems a quite specific case, maybe it should go in the sketching
>> module so it can be easier to find,
>
>
> Adding it to a separate module under `extensions` sounds good to me.
>
>
>> or maybe in its own extension if
>> the 'mix' of dependencies may be an issue and then make this
>> dependency a requirement for the gcp module since I suppose the
>> ultimate goal is to integrate it there.
>>
>
> I guess we can shade dependencies of ZetaSketch if it creates a problem
> when integrated with Beam. But I would not relate it to a gcp module since
> I think it will be a useful feature regardless of whether users run it on
> GCP or not (although if run on GCP, it will get better integration with
> BigQuery).
>
> On Mon, Jun 24, 2019 at 1:55 PM Ismaël Mejía  wrote:
>
>> Thanks for bringing this Robin,
>>
>> Can you please add this to the design documents webpage.
>> https://beam.apache.org/contribute/design-documents/
>>
>> Let some comments in the doc, It is great that this is finally open
>> and even better that it becomes part of Beam.
>>
>> I am not sure if this feature should go into 'sdks/java/core' because
>> it seems a quite specific case, maybe it should go in the sketching
>> module so it can be easier to find, or maybe in its own extension if
>> the 'mix' of dependencies may be an issue and then make this
>> dependency a requirement for the gcp module since I suppose the
>> ultimate goal is to integrate it there.
>>
>> CC +arnaudfournier...@gmail.com original author of the sketching
>> library who may be interested on this.
>>
>>
>> On Mon, Jun 24, 2019 at 9:31 PM Rui Wang  wrote:
>> >
>> > Thanks Robin! It would also be interesting if we could offer HLL_COUNT
>> functions in BeamSQL based on your proposal!
>> >
>> >
>> > -Rui
>> >
>> > On Mon, Jun 24, 2019 at 10:47 AM Robin Qiu  wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I have written a doc proposing we integrate the HyperLogLog++
>> algorithm into Beam as a new combiner. The algorithm solves the
>> count-distinct problem, and the intermediate sketch (aggregator) format
>> will be compatible with sketches computed via the HLL_COUNT functions in
>> Google Cloud BigQuery (because they will be based on the same
>> implementation: ZetaSketch). The tracking JIRA issue is BEAM-7013.
>> >>
>> >> The API design proposed in the doc is subject to change and open to
>> comments. Please feel free to comment if you have any thoughts.
>> >>
>> >> Cheers,
>> >> Robin
>>
>

Re: [ANNOUNCE] New committer: Robert Burke

[Robin Qiu]

Congrats, Robert!!

On Tue, Jul 16, 2019 at 1:22 PM Alan Myrvold  wrote:

> Congrats, Robert!
>
> On Tue, Jul 16, 2019 at 11:46 AM Ismaël Mejía  wrote:
>
>> Congrats Robert!
>>
>>
>> On Tue, Jul 16, 2019 at 8:19 PM Yichi Zhang  wrote:
>> >
>> > Congratulations!
>> >
>> > On Tue, Jul 16, 2019 at 10:51 AM Holden Karau 
>> wrote:
>> >>
>> >> Congratulations! :)
>> >>
>> >> On Tue, Jul 16, 2019 at 10:50 AM Mikhail Gryzykhin 
>> wrote:
>> >>>
>> >>> Congratulations!
>> >>>
>> >>> On Tue, Jul 16, 2019 at 10:36 AM Ankur Goenka 
>> wrote:
>> 
>>  Congratulations Robert!
>> 
>>  Go GO!
>> 
>>  On Tue, Jul 16, 2019 at 10:34 AM Rui Wang  wrote:
>> >
>> > Congrats!
>> >
>> >
>> > -Rui
>> >
>> > On Tue, Jul 16, 2019 at 10:32 AM Udi Meiri 
>> wrote:
>> >>
>> >> Congrats Robert B.!
>> >>
>> >> On Tue, Jul 16, 2019 at 10:23 AM Ahmet Altay 
>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Robert Burke.
>> >>>
>> >>> Robert has been contributing to Beam and actively involved in the
>> community for over a year. He has been actively working on Go SDK, helping
>> users, and making it easier for others to contribute [1].
>> >>>
>> >>> In consideration of Robert's contributions, the Beam PMC trusts
>> him with the responsibilities of a Beam committer [2].
>> >>>
>> >>> Thank you, Robert, for your contributions and looking forward to
>> many more!
>> >>>
>> >>> Ahmet, on behalf of the Apache Beam PMC
>> >>>
>> >>> [1]
>> https://lists.apache.org/thread.html/8f729da2d3009059d7a8b2d8624446be161700dcfa953939dd3530c6@%3Cdev.beam.apache.org%3E
>> >>> [2]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>
>> >>
>> >>
>> >> --
>> >> Twitter: https://twitter.com/holdenkarau
>> >> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9
>> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: Python Utilities

[Robin Qiu]

Hi Shannon,

Thanks for sharing the repo! I took a quick look and I have a concern with
the naming of the transforms.

Currently, Beam Java already have "Select" and "Join" transforms. However,
they work on schemas, a feature that is not yet implemented in Beam Python.
(See
https://github.com/apache/beam/tree/77b295b1c2b0a206099b8f50c4d3180c248e252c/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms
)

To maintain consistency between SDKs, I think it is good to avoid having
two different transforms with the same name but different functions. So
maybe you can consider renaming the transforms or/and putting it in an
extension Python module, instead of the main ones?

Best,
Robin

On Mon, Jul 8, 2019 at 9:19 AM Shannon Duncan 
wrote:

> As a follow up. Here is the repo that contains the utilities for now.
> https://github.com/shadowcodex/apache-beam-utilities. Will put together a
> proper PR as code gets closer to production quality.
>
> - Shannon
>
> On Mon, Jul 8, 2019 at 9:20 AM Shannon Duncan 
> wrote:
>
>> Thanks Frederik,
>>
>> That's exactly where I was looking. I did get permission to open source
>> the utilities module. So I'm going to throw them up on my personal github
>> soon and share with the email group for a look over.
>>
>> I'm going to work on the utilities there because it's a quick dev
>> environment and then once they are ready for proper PR I'll begin working
>> them into the actual SDK for a PR.
>>
>> I also joined the slack #beam and #beam-python channels, I was unsure of
>> where most collaborators discussed items.
>>
>> - Shannon
>>
>> On Mon, Jul 8, 2019 at 9:09 AM Frederik Bode 
>> wrote:
>>
>>> Hi Shannon,
>>>
>>> This is probably a good starting point:
>>> https://github.com/apache/beam/blob/2d5e493abf39ee6fc89831bb0b7ec9fee592b9c5/sdks/python/apache_beam/transforms/combiners.py#L68
>>> .
>>>
>>> Frederik
>>>
>>> [image: https://ml6.eu]
>>> 
>>>
>>>
>>> * Frederik Bode*
>>>
>>> ML6 Ghent
>>> 
>>> +32 4 92 78 96 18
>>>
>>>
>>>  DISCLAIMER 
>>>
>>> This email and any files transmitted with it are confidential and
>>> intended solely for the use of the individual or entity to whom they are
>>> addressed. If you have received this email in error please notify the
>>> system manager. This message contains confidential information and is
>>> intended only for the individual named. If you are not the named addressee
>>> you should not disseminate, distribute or copy this e-mail. Please notify
>>> the sender immediately by e-mail if you have received this e-mail by
>>> mistake and delete this e-mail from your system. If you are not the
>>> intended recipient you are notified that disclosing, copying, distributing
>>> or taking any action in reliance on the contents of this information is
>>> strictly prohibited.
>>>
>>>
>>> On Mon, 8 Jul 2019 at 15:40, Shannon Duncan 
>>> wrote:
>>>
 I'm sure I could use some of the existing aggregations as a guide on
 how to make aggregations to fill the gap of missing ones. Such as creating
 Sum/Max/Min.

 GroupBy is really already handled with GroupByKey and CoGroupByKey
 unless you are thinking of a different type of GroupBy?

 - Shannon

 On Sun, Jul 7, 2019 at 10:47 PM Rui Wang  wrote:

> Maybe also adding Aggregation/GroupBy as utilities?
>
>
> -Rui
>
> On Sun, Jul 7, 2019 at 1:46 PM Shannon Duncan <
> joseph.dun...@liveramp.com> wrote:
>
>> Thanks Valentyn,
>>
>> I'll outline the utilities and accept any suggestions to add /
>> modify. These are really just shortcut PTransforms that I am working on 
>> to
>> simplify creating pipelines.
>>
>> Currently the utilities contain the following PTransforms:
>>
>> - Inner Join
>> - Left Outer Join
>> - Right Outer Join
>> - Full Outer Join
>> - PrepareKey (For selecting items in a dictionary to act as a key for
>> the joins)
>> - Select (very simple filter that returns only items you want from
>> the dictionary) (allows for defining a default nullValue)
>>
>> Currently these operations only work with dictionaries, but I'd be
>> interested to see how it would work for  tuples.
>>
>> I'm new to python so they may not be optimized or the best way, but
>> from my 

Re: [POPOSAL] Integrate BigQuery-compatible HyperLogLog algorithm into Beam

[Robin Qiu]

>
> Can you please add this to the design documents webpage.
> https://beam.apache.org/contribute/design-documents/
>

Thanks for the reminder. Done! (https://github.com/apache/beam/pull/8947)


> I am not sure if this feature should go into 'sdks/java/core' because
> it seems a quite specific case, maybe it should go in the sketching
> module so it can be easier to find,


Adding it to a separate module under `extensions` sounds good to me.


> or maybe in its own extension if
> the 'mix' of dependencies may be an issue and then make this
> dependency a requirement for the gcp module since I suppose the
> ultimate goal is to integrate it there.
>

I guess we can shade dependencies of ZetaSketch if it creates a problem
when integrated with Beam. But I would not relate it to a gcp module since
I think it will be a useful feature regardless of whether users run it on
GCP or not (although if run on GCP, it will get better integration with
BigQuery).

On Mon, Jun 24, 2019 at 1:55 PM Ismaël Mejía  wrote:

> Thanks for bringing this Robin,
>
> Can you please add this to the design documents webpage.
> https://beam.apache.org/contribute/design-documents/
>
> Let some comments in the doc, It is great that this is finally open
> and even better that it becomes part of Beam.
>
> I am not sure if this feature should go into 'sdks/java/core' because
> it seems a quite specific case, maybe it should go in the sketching
> module so it can be easier to find, or maybe in its own extension if
> the 'mix' of dependencies may be an issue and then make this
> dependency a requirement for the gcp module since I suppose the
> ultimate goal is to integrate it there.
>
> CC +arnaudfournier...@gmail.com original author of the sketching
> library who may be interested on this.
>
>
> On Mon, Jun 24, 2019 at 9:31 PM Rui Wang  wrote:
> >
> > Thanks Robin! It would also be interesting if we could offer HLL_COUNT
> functions in BeamSQL based on your proposal!
> >
> >
> > -Rui
> >
> > On Mon, Jun 24, 2019 at 10:47 AM Robin Qiu  wrote:
> >>
> >> Hi all,
> >>
> >> I have written a doc proposing we integrate the HyperLogLog++ algorithm
> into Beam as a new combiner. The algorithm solves the count-distinct
> problem, and the intermediate sketch (aggregator) format will be compatible
> with sketches computed via the HLL_COUNT functions in Google Cloud BigQuery
> (because they will be based on the same implementation: ZetaSketch). The
> tracking JIRA issue is BEAM-7013.
> >>
> >> The API design proposed in the doc is subject to change and open to
> comments. Please feel free to comment if you have any thoughts.
> >>
> >> Cheers,
> >> Robin
>

[POPOSAL] Integrate BigQuery-compatible HyperLogLog algorithm into Beam

[Robin Qiu]

Hi all,

I have written a doc

proposing we integrate the HyperLogLog++ algorithm

into Beam as a new combiner. The algorithm solves the count-distinct problem
, and the
intermediate sketch (aggregator) format will be compatible with sketches
computed via the HLL_COUNT functions

in Google Cloud BigQuery (because they will be based on the same
implementation: ZetaSketch ). The
tracking JIRA issue is BEAM-7013
.

The API design proposed in the doc is subject to change and open to
comments. Please feel free to comment if you have any thoughts.

Cheers,
Robin

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

[Robin Qiu]

Congrats, Mikhail!

On Fri, Jun 21, 2019 at 9:12 AM Alexey Romanenko 
wrote:

> Congrats, Mikhail!
>
> On 21 Jun 2019, at 18:01, Anton Kedin  wrote:
>
> Congrats!
>
> On Fri, Jun 21, 2019 at 3:55 AM Reza Rokni  wrote:
>
>> Congratulations!
>>
>> On Fri, 21 Jun 2019, 12:37 Robert Burke,  wrote:
>>
>>> Congrats
>>>
>>> On Fri, Jun 21, 2019, 12:29 PM Thomas Weise  wrote:
>>>
 Hi,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Mikhail Gryzykhin.

 Mikhail has been contributing to Beam and actively involved in the
 community for over a year. He developed the community build dashboard [1]
 and added substantial improvements to our build infrastructure. Mikhail's
 work also covers metrics, contributor documentation, development process
 improvements and other areas.

 In consideration of Mikhail's contributions, the Beam PMC trusts him
 with the responsibilities of a Beam committer [2].

 Thank you, Mikhail, for your contributions and looking forward to many
 more!

 Thomas, on behalf of the Apache Beam PMC

 [1] https://s.apache.org/beam-community-metrics
 [2]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer


>

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

[Robin Qiu]

Congratulations, Pablo!!

On Wed, May 15, 2019 at 10:43 AM Pablo Estrada  wrote:

> Thanks everyone for the encouragement, and thanks to the PMC for the
> recognition. I am honored and grateful. :)
> Best
> -P.
>
>
> *From: *Kenneth Knowles 
> *Date: *Tue, May 14, 2019, 10:25 PM
> *To: *dev
>
> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming Pablo Estrada to
>> join the PMC.
>>
>> Pablo first picked up BEAM-722 in October of 2016 and has been a steady
>> part of the Beam community since then. In addition to technical work on
>> Beam Python & Java & runners, I would highlight how Pablo grows Beam's
>> community by helping users, working on GSoC, giving talks at Beam Summits
>> and other OSS conferences including Flink Forward, and holding training
>> workshops. I cannot do justice to Pablo's contributions in a single
>> paragraph.
>>
>> Thanks Pablo, for being a part of Beam.
>>
>> Kenn
>>
>

Re: [ANNOUNCE] New committer announcement: Udi Meiri

[Robin Qiu]

Congratulations Udi!!!

*From: *Ruoyun Huang 
*Date: *Fri, May 3, 2019 at 2:39 PM
*To: * 

Congratulations Udi!
>
> On Fri, May 3, 2019 at 2:30 PM Ahmet Altay  wrote:
>
>> Congratulations, Udi!
>>
>> *From: *Kyle Weaver 
>> *Date: *Fri, May 3, 2019 at 2:11 PM
>> *To: * 
>>
>> Congratulations Udi! I look forward to sending you all my reviews for
>>> the next month (just kidding :)
>>>
>>> Kyle Weaver | Software Engineer | github.com/ibzib |
>>> kcwea...@google.com | +1650203
>>>
>>> On Fri, May 3, 2019 at 1:52 PM Charles Chen  wrote:
>>> >
>>> > Thank you Udi!
>>> >
>>> > On Fri, May 3, 2019, 1:51 PM Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> wrote:
>>> >>
>>> >> Congratulations, Udi! Thank you for all your contributions!!!
>>> >>
>>> >> From: Pablo Estrada 
>>> >> Date: Fri, May 3, 2019 at 1:45 PM
>>> >> To: dev
>>> >>
>>> >>> Thanks Udi and congrats!
>>> >>>
>>> >>> On Fri, May 3, 2019 at 1:44 PM Kenneth Knowles 
>>> wrote:
>>> 
>>>  Hi all,
>>> 
>>>  Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Udi Meiri.
>>> 
>>>  Udi has been contributing to Beam since late 2017, starting with
>>> HDFS support in the Python SDK and continuing with a ton of Python work. I
>>> also will highlight his work on community-building infrastructure,
>>> including documentation, experiments with ways to find reviewers for pull
>>> requests, gradle build work, analyzing and reducing build times.
>>> 
>>>  In consideration of Udi's contributions, the Beam PMC trusts Udi
>>> with the responsibilities of a Beam committer [1].
>>> 
>>>  Thank you, Udi, for your contributions.
>>> 
>>>  Kenn
>>> 
>>>  [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: Hello from Hannah Jiang

[Robin Qiu]

Welcome Hannah!

On Thu, Apr 25, 2019 at 1:26 PM Hannah Jiang  wrote:

> Thanks Kenneth!
>
> On Thu, Apr 25, 2019 at 1:24 PM Kenneth Knowles  wrote:
>
>> Welcome!
>>
>> On Thu, Apr 25, 2019 at 12:38 PM Matthias Baetens <
>> baetensmatth...@gmail.com> wrote:
>>
>>> Welcome to the community!
>>>
>>> On Thu, Apr 25, 2019, 18:55 Griselda Cuevas  wrote:
>>>
 Welcome Hannah! - Very excited to see you in the Beam community :)

 On Tue, 23 Apr 2019 at 12:59, Hannah Jiang 
 wrote:

> Hi everyone
>
> I joined Google recently and would work on Python portability part. I
> am happy to be part of the community. Looking forward to working with all
> of you together.
>
> I have a minor request, can admin please give me access to JIRA?
>
> Thanks,
> Hannah
>
>
>

Re: [ANNOUNCE] New committer announcement: Yifan Zou

[Robin Qiu]

Congratulations Yifan!

On Mon, Apr 22, 2019 at 10:17 AM Chamikara Jayalath 
wrote:

> Congrats Yifan!
>
> On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels 
> wrote:
>
>> Congrats! Great work.
>>
>> -Max
>>
>> On 22.04.19 19:00, Rui Wang wrote:
>> > Congratulations! Thanks for your contribution!!
>> >
>> > -Rui
>> >
>> > On Mon, Apr 22, 2019 at 9:57 AM Ruoyun Huang > > > wrote:
>> >
>> > Congratulations, Yifan!
>> >
>> > On Mon, Apr 22, 2019 at 9:48 AM Boyuan Zhang > > > wrote:
>> >
>> > Congratulations, Yifan~
>> >
>> > On Mon, Apr 22, 2019 at 9:29 AM Connell O'Callaghan
>> > mailto:conne...@google.com>> wrote:
>> >
>> > Well done Yifan!!!
>> >
>> > Thank you for sharing Kenn!!!
>> >
>> > On Mon, Apr 22, 2019 at 9:00 AM Ahmet Altay
>> > mailto:al...@google.com>> wrote:
>> >
>> > Congratulations, Yifan!
>> >
>> > On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson
>> > > > > wrote:
>> >
>> > Congratulations Yifan!
>> >
>> > On Mon, Apr 22, 2019 at 5:39 PM Cyrus Maden
>> > mailto:cma...@google.com>>
>> wrote:
>> >
>> > Congratulations Yifan!!
>> >
>> > On Mon, Apr 22, 2019 at 11:26 AM Kenneth Knowles
>> > mailto:k...@apache.org>>
>> wrote:
>> >
>> > Hi all,
>> >
>> > Please join me and the rest of the Beam PMC
>> > in welcoming a new committer: Yifan Zou.
>> >
>> > Yifan has been contributing to Beam since
>> > early 2018. He has proposed 70+ pull
>> > requests, adding dependency checking and
>> > improving test infrastructure. But something
>> > the numbers cannot show adequately is the
>> > huge effort Yifan has put into working with
>> > infra and keeping our Jenkins executors
>> healthy.
>> >
>> > In consideration of Yian's contributions,
>> > the Beam PMC trusts Yifan with the
>> > responsibilities of a Beamcommitter[1].
>> >
>> > Thank you, Yifan, for your contributions.
>> >
>> > Kenn
>> >
>> > [1]
>> >
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> > <
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >
>> >
>> >
>> >
>> > --
>> > 
>> > Ruoyun  Huang
>> >
>>
>

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

[Robin Qiu]

Congratulations, Boyuan!

On Wed, Apr 10, 2019 at 9:28 AM Connell O'Callaghan 
wrote:

> Well done and congratulations Boyuan!!!
>
> On Wed, Apr 10, 2019, 5:04 PM Lukasz Cwik  wrote:
>
>> Congratulations and well deserved.
>>
>> On Wed, Apr 10, 2019 at 9:00 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new committer:
>>> Boyuan Zhang.
>>>
>>> Boyuan has been contributing to Beam since early 2018. She has proposed
>>> 100+ pull requests across a wide range of topics: bug fixes, to integration
>>> tests, build improvements, metrics features, release automation. Two big
>>> picture things to highlight are building/releasing Beam Python wheels and
>>> managing the donation of the Beam Dataflow Java Worker, including help with
>>> I.P. clearance.
>>>
>>> In consideration of Boyuan's contributions, the Beam PMC trusts Boyuan
>>> with the responsibilities of a Beam committer [1].
>>>
>>> Thank you, Boyuan, for your contributions.
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/#an-apache-
>>> beam-committer
>>>
>>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Robin Qiu]

Congratulations, Mark!

On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:

> Congrats Mark!
>
> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>
>> Congratulations, Mark! 
>>
>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>>
>>> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
>>> wrote:
>>>
 Nice! Congratulations, Mark.

 On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
 ka.kucharc...@gmail.com> wrote:

> Congratulations, Mark! 
>
> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
> wrote:
>
>> Congratulations!
>>
>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>> wrote:
>>
>>> Congrats! :)
>>>
>>>
>>>
>>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> napisał(a):
>>>
 Congratulations, Mark!

 On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
 wrote:

> Yeaah  Mark! : ) Congrats : D
>
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
> wrote:
>
>> Congratulations Mark!
>>
>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done congratulations Mark!!!
>>>
>>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
>>> rob...@frantil.com> wrote:
>>>
 Congratulations Mark! 

 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Mark!
>
> Thanks for your contributions, in particular for your efforts
> to parallelize test execution for Python SDK and increase the 
> speed of
> Python precommit checks.
>
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
> k...@apache.org> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Mark Liu.
>>
>> Mark has been contributing to Beam since late 2016! He has
>> proposed 100+ pull requests. Mark was instrumental in expanding 
>> test and
>> infrastructure coverage, especially for Python. In
>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>> with the
>> responsibilities of a Beam committer [1].
>>
>> Thank you, Mark, for your contributions.
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/become-a-committer/
>> #an-apache-beam-committer
>>
> --

 *Aizhamal Nurmamat kyzy*

 Open Source Program Manager

 646-355-9740 Mobile

 601 North 34th Street, Seattle, WA 98103



>>
>> --
>> Cheers,
>> Gleb
>>
>

Re: Signing off

[Robin Qiu]

Thanks Scott and good luck in your next adventure!

Best,
Robin

On Fri, Feb 15, 2019 at 9:35 AM Udi Meiri  wrote:

> Good luck Scott!
>
> On Fri, Feb 15, 2019 at 9:32 AM Alex Amato  wrote:
>
>> Thanks's for your contributions Scott. We will miss you.
>>
>> On Fri, Feb 15, 2019 at 7:08 AM Etienne Chauchot 
>> wrote:
>>
>>> Thank you for your contributions Scott ! Your new project seems very
>>> fun. Enjoy !
>>>
>>> Etienne
>>>
>>> Le vendredi 15 février 2019 à 15:01 +0100, Ismaël Mejía a écrit :
>>>
>>> Your work and willingness to make Beam better will be missed.
>>>
>>> Good luck for the next phase!
>>>
>>>
>>> On Fri, Feb 15, 2019 at 1:39 PM Łukasz Gajowy  wrote:
>>>
>>>
>>> Good luck!
>>>
>>>
>>> pt., 15 lut 2019 o 11:24 Alexey Romanenko  
>>> napisał(a):
>>>
>>>
>>> Good luck, Scott, with your new adventure!
>>>
>>>
>>> On 15 Feb 2019, at 11:22, Maximilian Michels  wrote:
>>>
>>>
>>> Thank you for your contributions Scott. Best of luck!
>>>
>>>
>>> On 15.02.19 10:48, Michael Luckey wrote:
>>>
>>>
>>> Hi Scott,
>>>
>>> yes, thanks for all your time and all the best!
>>>
>>> michel
>>>
>>> On Fri, Feb 15, 2019 at 5:47 AM Kenneth Knowles >> > wrote:
>>>
>>>+1
>>>
>>>Thanks for the contributions to community & code, and enjoy the new
>>>
>>>chapter!
>>>
>>>Kenn
>>>
>>>On Thu, Feb 14, 2019 at 3:25 PM Thomas Weise >>
>>>> wrote:
>>>
>>>Hi Scott,
>>>
>>>Thank you for the many contributions to Beam and best of luck
>>>
>>>with the new endeavor!
>>>
>>>Thomas
>>>
>>>On Thu, Feb 14, 2019 at 10:37 AM Scott Wegner >>
>>>> wrote:
>>>
>>>I wanted to let you all know that I've decided to pursue a
>>>
>>>new adventure in my career, which will take me away from
>>>
>>>Apache Beam development.
>>>
>>>It's been a fun and fulfilling journey. Apache Beam has been
>>>
>>>my first significant experience working in open source. I'm
>>>
>>>inspired observing how the community has come together to
>>>
>>>deliver something great.
>>>
>>>Thanks for everything. If you're curious what's next: I'll
>>>
>>>be working on Federated Learning at Google:
>>>
>>>
>>> https://ai.googleblog.com/2017/04/federated-learning-collaborative.html
>>>
>>>Take care,
>>>
>>>Scott
>>>
>>>Got feedback? tinyurl.com/swegner-feedback
>>>
>>>
>>>
>>>
>>>
>>>

Re: a new contributor

[Robin Qiu]

Welcome, Heejong!

On Fri, Oct 19, 2018 at 11:55 AM Ahmet Altay  wrote:

> Welcome!
>
> On Fri, Oct 19, 2018 at 11:48 AM, Heejong Lee  wrote:
>
>> Hi,
>>
>> I just wanted to introduce myself as a new contributor. I'm a new member
>> of Apache Beam team at Google and will be working on IO modules. Happy to
>> meet you all!
>>
>> Thanks,
>> Heejong
>>
>
>

Re: [ANNOUNCE] New committers, October 2018

[Robin Qiu]

Congrats, Xinyu and Ankur!

On Fri, Oct 19, 2018 at 11:51 AM Daniel Oliveira 
wrote:

> Congratulations!
>
> On Fri, Oct 19, 2018 at 8:27 AM Thomas Weise  wrote:
>
>> Congrats!
>>
>>
>> On Fri, Oct 19, 2018 at 7:24 AM Ismaël Mejía  wrote:
>>
>>> Congratulations guys and welcome !
>>> On Fri, Oct 19, 2018 at 4:12 PM Jean-Baptiste Onofré 
>>> wrote:
>>> >
>>> > Congrats and welcome aboard !
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 19/10/2018 16:09, Kenneth Knowles wrote:
>>> > > Hi all,
>>> > >
>>> > > Hot on the tail of the summer announcement comes our pre-Hallowe'en
>>> > > celebration.
>>> > >
>>> > > Please join me and the rest of the Beam PMC in welcoming the
>>> following
>>> > > new committers:
>>> > >
>>> > >  - Xinyu Liu, author/maintainer of the Samza runner
>>> > >  - Ankur Goenka, major contributor to portability efforts
>>> > >
>>> > > And, as before, while I've noted some areas of contribution for each,
>>> > > most important is that they are a valued part of our Beam community
>>> that
>>> > > the PMC trusts with the responsibilities of a Beam committer [1].
>>> > >
>>> > > A big thanks to both for their contributions.
>>> > >
>>> > > Kenn
>>> > >
>>> > > [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbono...@apache.org
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>>
>>

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

[Robin Qiu]

Congratulations, Kenn!

On Wed, Sep 19, 2018 at 1:05 PM Lukasz Cwik  wrote:

> Congrats Kenn.
>
> On Wed, Sep 19, 2018 at 12:54 PM Davor Bonaci  wrote:
>
>> Hi everyone --
>> It is with great pleasure that I announce that at today's meeting of the
>> Foundation's Board of Directors, the Board has appointed Kenneth Knowles as
>> the second chair of the Apache Beam project.
>>
>> Kenn has served on the PMC since its inception, and is very active and
>> effective in growing the community. His exemplary posts have been cited in
>> other projects. I'm super happy to have Kenn accepted the nomination, and
>> I'm confident that he'll serve with distinction.
>>
>> As for myself, I'm not going anywhere. I'm still around and will be as
>> active as I have recently been. Thrilled to be able to pass the baton to
>> such a key member of this community and to have less administrative work to
>> do ;-).
>>
>> Please join me in welcoming Kenn to his new role, and I ask that you
>> support him as much as possible. As always, please let me know if you have
>> any questions.
>>
>> Davor
>>
>

Accessing attempted metrics from within a DoFn

[Robin Qiu]

Hi everyone,

I am writing a test [1] for the support of @RequiresStableInput annotation
in Java SDK [2]. In the test I need to have a ParDo make some side effects
and then fail. Currently, the only way to persist the side effect during
retries without depending on external states is using attempted metrics.
However, attempted metrics cannot be accessed from within a DoFn. So I am
wondering if we should make an improvement on this by, for example,
exposing a job handler to DoFn.

As Luke mentioned in another email thread, this improvement will also be
useful since:
* cancelling a pipeline from within the pipeline is useful
* starting a new job against the existing runner from in a pipeline is
useful
* accessing attempted metrics to test DoFn's with side effects is useful
for error handling testing

What do you think? All suggestions are appreciated.

Best,
Robin

[1] https://github.com/apache/beam/pull/6220
[2]
https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM/edit#

Re: Should we allow ValidatesRunner tests to have access to file systems?

[Robin Qiu]

Hi Alan and Luke,

Thanks for your comments! I agree on your suggestions, and now I have made
the test into an integration test. Please take a look at the PR when you
have time.

Luke's suggestion on using metrics sounds promising. I will open a separate
email thread to ask people's idea on making the improvement.

Best,
Robin

On Tue, Aug 28, 2018 at 5:42 PM Lukasz Cwik  wrote:

> I also agree about not having external dependencies in validates runner
> tests.
>
> One suggestion would have been to use attempted metrics but there is
> currently no way to get access to runner metrics from within a DoFn easily
> that is runner agnostic. This is likely a place for improvement since:
> * cancelling a pipeline from within the pipeline is useful
> * starting a new job against the existing runner from in a pipeline is
> useful
> * accessing attempted metrics to test DoFn's with side effects is useful
> for error handling testing
>
> On Mon, Aug 27, 2018 at 12:40 PM Alan Myrvold  wrote:
>
>> I think this should be an integration test if it requires more access
>> than the current ValidatesRunner tests.
>>
>> Although the ValidatesRunner and integration tests are similar, the
>> intent is that the validates runner tests are smaller and more like
>> component tests, and there have been discusions on fusing the validates
>> runner tests into a smaller set of pipelines.
>>
>> On Mon, Aug 27, 2018 at 11:27 AM Robin Qiu  wrote:
>>
>>> Hello everyone,
>>>
>>> I am writing a test [1] for the support of @RequiresStableInput
>>> annotation in Java SDK [2]. For the test to work, I need to have a ParDo
>>> make some side effect (e.g. writing to a file system). However,
>>> ValidatesRunner tests in Beam currently cannot depend on external states
>>> (cannot write to file systems). So I am wondering if it is a good idea to
>>> allow ValidatesRunner tests to have access to file systems. This way we can
>>> create more flexible ValidatesRunner tests.
>>>
>>> I could make this test a integration test to get access to file systems
>>> (e.g. like WordCountIT.java [3]). But functionally I think this test should
>>> be a ValidatesRunner test, because it is testing the support of some SDK
>>> features on runners.
>>>
>>> So what do you think? Any suggestions or concerns are appreciated.
>>>
>>> Best,
>>> Robin
>>>
>>> [1] https://github.com/apache/beam/pull/6220
>>> [2]
>>> https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM/edit#
>>> [3]
>>> https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
>>>
>>>

Re: An example of Integration test case

[Robin Qiu]

Hi Rakesh,

A python integration test example can be found here:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_it_test.py

Best,
Robin

On Tue, Aug 28, 2018 at 1:10 PM Rakesh Kumar  wrote:

> Hi,
>
> I am writing my streaming application using Python SDK. I also want to
> write an integration test cases. Do we have any good example of integration
> test that I can refer?
>
> Thank you,
> Rakesh
> --
> Rakesh Kumar
> Software Engineer
> 510-761-1364 |
>
> 
>

Should we allow ValidatesRunner tests to have access to file systems?

[Robin Qiu]

Hello everyone,

I am writing a test [1] for the support of @RequiresStableInput annotation
in Java SDK [2]. For the test to work, I need to have a ParDo make some
side effect (e.g. writing to a file system). However, ValidatesRunner tests
in Beam currently cannot depend on external states (cannot write to file
systems). So I am wondering if it is a good idea to allow ValidatesRunner
tests to have access to file systems. This way we can create more flexible
ValidatesRunner tests.

I could make this test a integration test to get access to file systems
(e.g. like WordCountIT.java [3]). But functionally I think this test should
be a ValidatesRunner test, because it is testing the support of some SDK
features on runners.

So what do you think? Any suggestions or concerns are appreciated.

Best,
Robin

[1] https://github.com/apache/beam/pull/6220
[2]
https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM/edit#
[3]
https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java

Re: How do we run pipeline using gradle?

[Robin Qiu]

Hey Huygaa,

If you have your Intellij set up, you can run it from there and edit
program arguments in "Run Configuration".

Best,
Robin

On Wed, Aug 15, 2018 at 10:50 AM Huygaa Batsaikhan 
wrote:

> When we run wordcount using maven, we pass "-P dataflow-runner" profile to
> set the runner. What is the equivalent of this in gradle? In other words,
> how can I run wordcount straight from my Beam repo code?
>

Re: Policy for Python ValidatesRunner vs IT tests?

[Robin Qiu]

Hi Pablo,

Beam Java has unit tests, ValidatesRunner tests, and Integration (E2E) test.

If I understand correctly, unit tests test the functionality of a component
and usually don't require a runner (is this true?); ValidatesRunner tests
(marked with @ValidatesRunner annotation) require runners; and integration
tests (named *IT.java) test an entire e2e pipeline.

Best,
Robin

On Tue, Aug 14, 2018 at 3:39 PM Pablo Estrada  wrote:

> Ah this is very helpful. I think this is missing in the testing guide.
> I'll make a short PR to mention it.
> Best
> -P.
>
> On Tue, Aug 14, 2018 at 3:37 PM Anton Kedin  wrote:
>
>> IT tests exist in java, similar to unit tests and not marked in a special
>> way, except they're called *IT.java instead of *Test.java. They're run from
>> corresponding tasks:
>>  -
>> https://github.com/apache/beam/blob/d6c5bf977fc688f289f1bb06e30f25b05bf987b2/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java#L33
>>
>>  -
>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/build.gradle#L85
>>
>> On Tue, Aug 14, 2018 at 3:27 PM Pablo Estrada  wrote:
>>
>>> Hello,
>>> In Python, we tag some test methods with @attr('ValidatesRunner') and
>>> @attr('IT'), which marks them to be run as pipeline tests.
>>>
>>> If I understand correctly:
>>> - ValidatesRunner tests are more like a component tests[1] as explained
>>> in Beam docs
>>> - IT tests are more like a E2E test[2] as explained in the docs. Is
>>> there an equivalent in Java?
>>> - Finally, there's ValidatesContainer tests. What are these for? What's
>>> the guidance for tagging our tests this way?
>>>
>>> Thanks!
>>> -P.
>>>
>>> [1] https://beam.apache.org/contribute/testing/#validatesrunner
>>> [2] https://beam.apache.org/contribute/testing/#e2e
>>> --
>>> Got feedback? go/pabloem-feedback
>>> 
>>>
>> --
> Got feedback? go/pabloem-feedback
> 
>

Re: Filtered Pre-commit triggering is BACK!

[Robin Qiu]

My PR with only changes in Java also triggered Python ("Run Python
PreCommit") and Go ("Run Go PreCommit") tests. Is that the expected
behavior?
https://github.com/apache/beam/pull/5787

On Wed, Jun 27, 2018 at 5:04 PM Ahmet Altay  wrote:

> Two of my PRs with python changes did not trigger any pre-commits. Could
> it be related to this change?
>
> https://github.com/apache/beam/pull/5768
> https://github.com/apache/beam/pull/5800
>
> Ahmet
>
>
>
> On Tue, Jun 26, 2018 at 2:30 PM, Andrew Pilloud 
> wrote:
>
>> Awesome! This will save so much time running tests.
>>
>> On Tue, Jun 26, 2018 at 2:29 PM Yifan Zou  wrote:
>>
>>> Thanks Scott! It's nice to have this feature.
>>>
>>> On Tue, Jun 26, 2018 at 2:24 PM Pablo Estrada 
>>> wrote:
>>>
 This is great. Reducing load on infrastructure should help Beam scale
 into a larger project : ) - Thanks Scott!

 On Tue, Jun 26, 2018 at 2:21 PM Scott Wegner  wrote:

> By popular demand [1], filtered pre-commit triggering is now
> re-enabled. Now when submitting pull request, only pre-commit tests for
> your affected files will run: if you change just the Go SDK, you'll no
> longer need to run Java pre-commits.
>
> Last time we introduced this change it regressed our ability to run
> other pre-commits via trigger phrase ("Run Java PreCommit"); this is now
> fixed by splitting the Jenkins jobs by trigger condition [2].
>
> Enjoy!
>
>
> [1]
> https://lists.apache.org/thread.html/f98af933ce1ffbf8ae85161dea032aab4577a600c749f46b1d85c226@%3Cdev.beam.apache.org%3E
> [2] https://github.com/apache/beam/pull/5757
>
 --
 Got feedback? go/pabloem-feedback
 

>>>
>

Invite to comment on the @RequiresStableInput design doc

[Robin Qiu]

Hello everyone,

I am Robin Qiu. I joined Google and started working on Beam Java SDK 2
months ago.

As my starting project, I am working on supporting the @RequiresStableInput
annotation in runners. Here is a short design doc. Please take a look and
feel free to comment.
https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM/edit?usp=sharing

You can also find the context of the problem in this email thread:
https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%3Cdev.beam.apache.org%3E
<https://www.google.com/url?q=https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%253Cdev.beam.apache.org%253E=D=1528933869705000=AFQjCNGr_TlLNieaRZMh6LOkOny_-IDeDQ>

Best,
Robin

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Robin Qiu]

Congrats to all!

On Fri, Jun 1, 2018 at 8:42 AM Henning Rohde  wrote:

> Congratulations!
>
> On Fri, Jun 1, 2018 at 7:03 AM Jesse Anderson 
> wrote:
>
>> Welcome!
>>
>> On Fri, Jun 1, 2018, 2:02 AM Etienne Chauchot 
>> wrote:
>>
>>> Congrats to all !
>>> Le jeudi 31 mai 2018 à 19:08 -0700, Davor Bonaci a écrit :
>>>
>>> Please join me and the rest of Beam PMC in welcoming the following
>>> contributors as our newest committers. They have significantly contributed
>>> to the project in different ways, and we look forward to many more
>>> contributions in the future.
>>>
>>> * Griselda Cuevas
>>> * Pablo Estrada
>>> * Jason Kuster
>>>
>>> (Apologizes for a delayed announcement, and the lack of the usual
>>> paragraph summarizing individual contributions.)
>>>
>>> Congratulations to all three! Welcome!
>>>
>>>

Re: I'm back and ready to help grow our community!

[Robin Qiu]

Congratulations! Welcome back!

On Thu, May 17, 2018 at 3:23 PM Reuven Lax  wrote:

> Congratulations! Good to see you back!
>
> Reuven
>
> On Thu, May 17, 2018 at 2:24 PM Griselda Cuevas  wrote:
>
>> Hi Everyone,
>>
>>
>> I was absent from the mailing list, slack channel and our Beam community
>> for the past six weeks, the reason was that I took a leave to focus on
>> finishing my Masters Degree, which I finally did on May 15th.
>>
>>
>> I graduated as a Masters of Engineering in Operations Research with a
>> concentration in Data Science from UC Berkeley. I'm glad to be part of this
>> community and I'd like to share this accomplishment with you so I'm adding
>> two pictures of that day :)
>>
>>
>> Given that I've seen so many new folks around, I'd like to use this
>> opportunity to re-introduce myself. I'm Gris Cuevas and I work at Google.
>> Now that I'm back, I'll continue to work on supporting our community in two
>> main streams: Contribution Experience & Events, Meetups, and Conferences.
>>
>>
>> It's good to be back and I look forward to collaborating with you.
>>
>>
>> Cheers,
>>
>> Gris
>>
>