Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[dev wearebold]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

Le mer. 20 nov. 2019 à 03:43, Kenneth Knowles  a écrit :

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Thomas Weise]

Congratulations!


On Wed, Nov 20, 2019, 7:56 PM Chamikara Jayalath 
wrote:

> Congrats!!
>
> On Wed, Nov 20, 2019 at 5:21 PM Daniel Oliveira 
> wrote:
>
>> Thank you everyone! I won't let you down. o7
>>
>> On Wed, Nov 20, 2019 at 2:12 PM Ruoyun Huang  wrote:
>>
>>> Congrats Daniel!
>>>
>>> On Wed, Nov 20, 2019 at 1:58 PM Robert Burke  wrote:
>>>
 Congrats Daniel! Much deserved.

 On Wed, Nov 20, 2019, 12:49 PM Udi Meiri  wrote:

> Congrats Daniel!
>
> On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver 
> wrote:
>
>> Congrats Dan! Keep up the good work :)
>>
>> On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden 
>> wrote:
>>
>>> Congratulations! This is great news.
>>>
>>> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:
>>>
 Congrats!


 -Rui

 On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congrats, Daniel!
>
> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Daniel Oliveira
>>
>> Daniel introduced himself to dev@ over two years ago and has
>> contributed in many ways since then. Daniel has contributed to 
>> general
>> project health, the portability framework, and all three languages: 
>> Java,
>> Python SDK, and Go. I would like to particularly highlight how he 
>> deleted
>> 12k lines of dead reference runner code [1].
>>
>> In consideration of Daniel's contributions, the Beam PMC trusts
>> him with the responsibilities of a Beam committer [2].
>>
>> Thank you, Daniel, for your contributions and looking forward to
>> many more!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1] https://github.com/apache/beam/pull/8380
>> [2]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>
>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Chamikara Jayalath]

Congrats!!

On Wed, Nov 20, 2019 at 5:21 PM Daniel Oliveira 
wrote:

> Thank you everyone! I won't let you down. o7
>
> On Wed, Nov 20, 2019 at 2:12 PM Ruoyun Huang  wrote:
>
>> Congrats Daniel!
>>
>> On Wed, Nov 20, 2019 at 1:58 PM Robert Burke  wrote:
>>
>>> Congrats Daniel! Much deserved.
>>>
>>> On Wed, Nov 20, 2019, 12:49 PM Udi Meiri  wrote:
>>>
 Congrats Daniel!

 On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver 
 wrote:

> Congrats Dan! Keep up the good work :)
>
> On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden 
> wrote:
>
>> Congratulations! This is great news.
>>
>> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:
>>
>>> Congrats!
>>>
>>>
>>> -Rui
>>>
>>> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congrats, Daniel!

 On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Daniel Oliveira
>
> Daniel introduced himself to dev@ over two years ago and has
> contributed in many ways since then. Daniel has contributed to general
> project health, the portability framework, and all three languages: 
> Java,
> Python SDK, and Go. I would like to particularly highlight how he 
> deleted
> 12k lines of dead reference runner code [1].
>
> In consideration of Daniel's contributions, the Beam PMC trusts
> him with the responsibilities of a Beam committer [2].
>
> Thank you, Daniel, for your contributions and looking forward to
> many more!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1] https://github.com/apache/beam/pull/8380
> [2]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

>>
>> --
>> 
>> Ruoyun  Huang
>>
>>

Re: [PROPOSAL] Preparing for Beam 2.18 release

[Tomo Suzuki]

Thank you for response.

On Wed, Nov 20, 2019 at 16:49 Chamikara Jayalath 
wrote:

>
>
> On Wed, Nov 20, 2019 at 1:04 PM Tomo Suzuki  wrote:
>
>> Hi Udi,
>>
>> (Question) I started learning how Beam dependencies are maintained
>> through releases. https://beam.apache.org/contribute/dependencies/ says
>>
>>
>> *Beam community has agreed on following policies regarding upgrading
>> dependencies.*
>>
>> ...
>>
>> *A significantly outdated dependency (identified manually or through the
>> automated Jenkins job) should result in a JIRA that is a blocker for the
>> next release. Release manager may choose to push the blocker to the
>> subsequent release or downgrade from a blocker.*
>>
>>
>> Is the statement above still valid? We have ~250 automatically created
>> tickets [1] for dependency upgrade.
>>
>
> I think it's up to the release manager as mentioned in the statement. We
> surely don't want to block releases by all these JIRAs but Beam community
> and/or release manager may decide to make some of these blockers if needed.
> I don't think the tool automatically makes the auto generated JIRAs
> release blockers.
>
> Thanks,
> Cham
>
>
>> [1]:
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20dependencies%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC
>>
>> Regards,
>> Tomo
>>
>>
>> On Wed, Nov 20, 2019 at 3:48 PM Udi Meiri  wrote:
>>
>>> Hi all,
>>>
>>> The next (2.18) release branch cut is scheduled for Dec 4, according to
>>> the calendar
>>> 
>>> .
>>> I would like to volunteer myself to do this release.
>>> The plan is to cut the branch on that date, and cherrypick release-blocking
>>> fixes afterwards if any.
>>>
>>> Any unresolved release blocking JIRA issues for 2.18 should have their
>>> "Fix Version/s" marked as "2.18.0".
>>>
>>> Any comments or objections?
>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
> --
Regards,
Tomo

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Daniel Oliveira]

Thank you everyone! I won't let you down. o7

On Wed, Nov 20, 2019 at 2:12 PM Ruoyun Huang  wrote:

> Congrats Daniel!
>
> On Wed, Nov 20, 2019 at 1:58 PM Robert Burke  wrote:
>
>> Congrats Daniel! Much deserved.
>>
>> On Wed, Nov 20, 2019, 12:49 PM Udi Meiri  wrote:
>>
>>> Congrats Daniel!
>>>
>>> On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver 
>>> wrote:
>>>
 Congrats Dan! Keep up the good work :)

 On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden  wrote:

> Congratulations! This is great news.
>
> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:
>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congrats, Daniel!
>>>
>>> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Daniel Oliveira

 Daniel introduced himself to dev@ over two years ago and has
 contributed in many ways since then. Daniel has contributed to general
 project health, the portability framework, and all three languages: 
 Java,
 Python SDK, and Go. I would like to particularly highlight how he 
 deleted
 12k lines of dead reference runner code [1].

 In consideration of Daniel's contributions, the Beam PMC trusts him
 with the responsibilities of a Beam committer [2].

 Thank you, Daniel, for your contributions and looking forward to
 many more!

 Kenn, on behalf of the Apache Beam PMC

 [1] https://github.com/apache/beam/pull/8380
 [2]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: GCP libraries up-to-date versions in Java

[Luke Cwik]

I took a look at the linkage checker and have opened up this PR[1] to allow
contributors to aid in performing dependency analysis within Apache Beam
during upgrades.

The current PR works by compiling and publishing all the Java artifacts to
your local maven repo and then runs the linkage checker against it with a
specified list of artifacts. For example by running:
./gradlew -Ppublishing
-PjavaLinkageArtifactIds=beam-sdks-java-core,beam-sdks-java-io-jdbc
:checkJavaLinkage

Produces:
Class javax.annotation.Nullable is not found;
  referenced by 1 class file
org.apache.beam.sdk.schemas.FieldValueTypeInformation
(beam-sdks-java-core-2.18.0-SNAPSHOT.jar)
Class org.brotli.dec.BrotliInputStream is not found;
  referenced by 1 class file

org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.brotli.BrotliCompressorInputStream
(beam-sdks-java-core-2.18.0-SNAPSHOT.jar)
Class com.github.luben.zstd.ZstdInputStream is not found;
  referenced by 1 class file

org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream
(beam-sdks-java-core-2.18.0-SNAPSHOT.jar)
... (lots more output) ...

I haven't tried running the linker analysis for all Apache Beam artifacts
yet but for anyone who is interested in doing dependency clean-up or
upgrades should be able to use the PR as is.

1: https://github.com/apache/beam/pull/10184

On Wed, Nov 20, 2019 at 12:16 PM Kenneth Knowles  wrote:

> On Wed, Nov 20, 2019 at 4:05 AM Elliotte Rusty Harold 
> wrote:
>
>> BOM or no BOM is an implementation detail.
>
>
> Agreed for the most part.
>
>
>> Using com.google.cloud:libraries-bom would make dependency management
>> simpler for developers, but the real issue is whether Beam can continue to
>> work with very old versions of the many libraries it depends on. Even if
>> this is acceptable for Beam, it's unlikely to be feasible for anyone who
>> needs to mix Beam code with other code.
>
>
> I believe every version of Beam's dependencies has been, and should
> continue to be, driven by what is best for Beam's users. That does mean
> making it easy for them to use the latest compatible version of their
> favorite libraries.
>
> There should be no self-incompatibility between Google minor version
>> releases. All the Google libraries in question follow semantic versioning.
>> E.g. Pubsub 1.43 would be fully API compatible with Pubsub 1.28, though not
>> the reverse. However there are likely to be important bug fixes in 1.43 and
>> definitely new features that 1.28 would not have. If there are any edge
>> cases where this is not true, that's a bug and if you file it against the
>> repo we'll try to fix it. We're also installing tooling to make this less
>> likely to happen by accident. However, right now any such problem is rare.
>>
>
> I'm glad we share the same ideals. If things were as good as you
> described, then we would have two good properties:
>
> 1. Users would always be able to force a newer minor version to trivially
> work around Beam's deps
> 2. Beam could always upgrade minor versions with no code change in Beam
> and no code change by users
>
> My experience is that this rarely works so simply. Generally, a user
> forces a new version of a library and it turns out that library or its
> dependencies has broken compatibility.
>
> Just reiterating that if semver really holds in these cases, then this
> proposal is fine with me. And if semver doesn't hold, I still think we
> should try to support the latest, but may also need to maintain a connector
> to support older versions that are still in wide use.
>
>
>> Looking at Beam's dependencies, the only case where there are major
>> version changes to address is Guava.
>>
>
> Beam has vendored Guava so it is mostly beside the point. Upgrading the
> vendored Guava does not interact with any of Beam's dependencies. See
> https://lists.apache.org/thread.html/c477d120a4c4626cbe675f8b03d84c6fe7938e36c8e2b55c492224cf@%3Cdev.beam.apache.org%3E
>
> Only KinesisIO and the ZetaSQL-to-Calcite translator actually have
> essential dependencies on Guava. In these cases, the version of Guava must
> necessarily be compatible with the Kinesis client and ZetaSQL,
> respectively. They may or may not be able to interop, and that is mostly
> out of our hands.
>
> The remaining issues are pre-1.0 libraries. OpenCensus is a particular
>> thorn in my side. Ideally these should not be used, at all. However if we
>> must, we should not expose them on the Beam API surface and we need to move
>> them forward quickly as they change.
>>
>
> This might deserve its own thread. This sounds like it should be
> well-hidden, vendored, or well-marked as "experimental".
>
> Kenn
>

Contributor permission for Beam Jira tickets

[Chuck Yang]

Hello Beam developers,

This is Chun from Cruise. I would like to make some contributions to
the Python SDK for Beam. Can someone add me as a contributor in the
Beam Jira? My username is cccyang.

Thanks!
Chun

-- 


*Confidentiality Note:* We care about protecting our proprietary 
information, confidential material, and trade secrets. This message may 
contain some or all of those things. Cruise will suffer material harm if 
anyone other than the intended recipient disseminates or takes any action 
based on this message. If you have received this message (including any 
attachments) in error, please delete it immediately and notify the sender 
promptly.

Re: [spark structured streaming runner] available on master

[Kai Jiang]

Cool, cool! Love to see Nexmark on Spark structured streaming runner
perfkit dashboard

On Wed, Nov 20, 2019 at 2:12 PM Pablo Estrada  wrote:

> Very cool! : ) Thanks to everyone involved moving this forward.
> Best
> -P.
>
> On Wed, Nov 20, 2019 at 8:50 AM Etienne Chauchot 
> wrote:
>
>> Forgot to say thanks everyone for their contribution to this especially
>> Alexey, Ryan and Ismael.
>>
>> Etienne
>> On 20/11/2019 17:12, Etienne Chauchot wrote:
>>
>> Hi all,
>>
>> I'm glad to announce that the new Spark runner based on Spark structured
>> streaming framework has been merged into master !
>>
>> It is not based on RDD/DStream API. See
>> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
>>
>> It is still experimental, its coverage of the Beam model is partial:
>>
>> - the runner passes 95% of the validates runner tests in batch mode.
>>
>> - It does not have support for streaming yet (waiting for the
>> multi-aggregations support in spark StructuredStreaming framework from the
>> Spark community)
>>
>> - Runner can execute Nexmark : perfkit dashboards yet to come
>>
>> - Some things are not wired up yet:
>>
>> - Beam Schemas not wired up
>>
>> - Optional features of the model not implemented:  state api, timer
>> api, splittable doFn api, …
>>
>> I will submit a PR to update the capability matrix in the coming days.
>>
>> Best
>>
>> Etienne
>>
>>
>>

Re: [spark structured streaming runner] available on master

[Pablo Estrada]

Very cool! : ) Thanks to everyone involved moving this forward.
Best
-P.

On Wed, Nov 20, 2019 at 8:50 AM Etienne Chauchot 
wrote:

> Forgot to say thanks everyone for their contribution to this especially
> Alexey, Ryan and Ismael.
>
> Etienne
> On 20/11/2019 17:12, Etienne Chauchot wrote:
>
> Hi all,
>
> I'm glad to announce that the new Spark runner based on Spark structured
> streaming framework has been merged into master !
>
> It is not based on RDD/DStream API. See
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
>
> It is still experimental, its coverage of the Beam model is partial:
>
> - the runner passes 95% of the validates runner tests in batch mode.
>
> - It does not have support for streaming yet (waiting for the
> multi-aggregations support in spark StructuredStreaming framework from the
> Spark community)
>
> - Runner can execute Nexmark : perfkit dashboards yet to come
>
> - Some things are not wired up yet:
>
> - Beam Schemas not wired up
>
> - Optional features of the model not implemented:  state api, timer
> api, splittable doFn api, …
>
> I will submit a PR to update the capability matrix in the coming days.
>
> Best
>
> Etienne
>
>
>

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Ruoyun Huang]

Congrats Daniel!

On Wed, Nov 20, 2019 at 1:58 PM Robert Burke  wrote:

> Congrats Daniel! Much deserved.
>
> On Wed, Nov 20, 2019, 12:49 PM Udi Meiri  wrote:
>
>> Congrats Daniel!
>>
>> On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver  wrote:
>>
>>> Congrats Dan! Keep up the good work :)
>>>
>>> On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden  wrote:
>>>
 Congratulations! This is great news.

 On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:

> Congrats!
>
>
> -Rui
>
> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congrats, Daniel!
>>
>> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
>> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Daniel Oliveira
>>>
>>> Daniel introduced himself to dev@ over two years ago and has
>>> contributed in many ways since then. Daniel has contributed to general
>>> project health, the portability framework, and all three languages: 
>>> Java,
>>> Python SDK, and Go. I would like to particularly highlight how he 
>>> deleted
>>> 12k lines of dead reference runner code [1].
>>>
>>> In consideration of Daniel's contributions, the Beam PMC trusts him
>>> with the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Daniel, for your contributions and looking forward to
>>> many more!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1] https://github.com/apache/beam/pull/8380
>>> [2]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

-- 

Ruoyun  Huang

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Robert Burke]

Congrats Daniel! Much deserved.

On Wed, Nov 20, 2019, 12:49 PM Udi Meiri  wrote:

> Congrats Daniel!
>
> On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver  wrote:
>
>> Congrats Dan! Keep up the good work :)
>>
>> On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden  wrote:
>>
>>> Congratulations! This is great news.
>>>
>>> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:
>>>
 Congrats!


 -Rui

 On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congrats, Daniel!
>
> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Daniel Oliveira
>>
>> Daniel introduced himself to dev@ over two years ago and has
>> contributed in many ways since then. Daniel has contributed to general
>> project health, the portability framework, and all three languages: Java,
>> Python SDK, and Go. I would like to particularly highlight how he deleted
>> 12k lines of dead reference runner code [1].
>>
>> In consideration of Daniel's contributions, the Beam PMC trusts him
>> with the responsibilities of a Beam committer [2].
>>
>> Thank you, Daniel, for your contributions and looking forward to many
>> more!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1] https://github.com/apache/beam/pull/8380
>> [2]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [PROPOSAL] Preparing for Beam 2.18 release

[Chamikara Jayalath]

On Wed, Nov 20, 2019 at 1:04 PM Tomo Suzuki  wrote:

> Hi Udi,
>
> (Question) I started learning how Beam dependencies are maintained through
> releases. https://beam.apache.org/contribute/dependencies/ says
>
>
> *Beam community has agreed on following policies regarding upgrading
> dependencies.*
>
> ...
>
> *A significantly outdated dependency (identified manually or through the
> automated Jenkins job) should result in a JIRA that is a blocker for the
> next release. Release manager may choose to push the blocker to the
> subsequent release or downgrade from a blocker.*
>
>
> Is the statement above still valid? We have ~250 automatically created
> tickets [1] for dependency upgrade.
>

I think it's up to the release manager as mentioned in the statement. We
surely don't want to block releases by all these JIRAs but Beam community
and/or release manager may decide to make some of these blockers if needed.
I don't think the tool automatically makes the auto generated JIRAs release
blockers.

Thanks,
Cham


> [1]:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20dependencies%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC
>
> Regards,
> Tomo
>
>
> On Wed, Nov 20, 2019 at 3:48 PM Udi Meiri  wrote:
>
>> Hi all,
>>
>> The next (2.18) release branch cut is scheduled for Dec 4, according to
>> the calendar
>> 
>> .
>> I would like to volunteer myself to do this release.
>> The plan is to cut the branch on that date, and cherrypick release-blocking
>> fixes afterwards if any.
>>
>> Any unresolved release blocking JIRA issues for 2.18 should have their
>> "Fix Version/s" marked as "2.18.0".
>>
>> Any comments or objections?
>>
>>
>
> --
> Regards,
> Tomo
>

Re: [PROPOSAL] Preparing for Beam 2.18 release

[Tomo Suzuki]

Hi Udi,

(Question) I started learning how Beam dependencies are maintained through
releases. https://beam.apache.org/contribute/dependencies/ says


*Beam community has agreed on following policies regarding upgrading
dependencies.*

...

*A significantly outdated dependency (identified manually or through the
automated Jenkins job) should result in a JIRA that is a blocker for the
next release. Release manager may choose to push the blocker to the
subsequent release or downgrade from a blocker.*


Is the statement above still valid? We have ~250 automatically created
tickets [1] for dependency upgrade.

[1]:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20dependencies%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC

Regards,
Tomo


On Wed, Nov 20, 2019 at 3:48 PM Udi Meiri  wrote:

> Hi all,
>
> The next (2.18) release branch cut is scheduled for Dec 4, according to
> the calendar
> 
> .
> I would like to volunteer myself to do this release.
> The plan is to cut the branch on that date, and cherrypick release-blocking
> fixes afterwards if any.
>
> Any unresolved release blocking JIRA issues for 2.18 should have their
> "Fix Version/s" marked as "2.18.0".
>
> Any comments or objections?
>
>

-- 
Regards,
Tomo

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Udi Meiri]

[ ] Beaver
[x] Hedgehog
[x] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[x] Robot dinosaur
[x] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 12:05 PM Eduardo Soldera <
eduardo.sold...@arquivei.com.br> wrote:

> [X ] Beaver
> [] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [X ] Robot dinosaur
> [ ] Firefly
> [] Cuttlefish
> [ ] Dumbo Octopus
> [] Angler fish
> [X] Capybara
>
> Em qua., 20 de nov. de 2019 às 17:03, Leonardo Miguel <
> leonardo.mig...@arquivei.com.br> escreveu:
>
>> [ ] Beaver
>> [X] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [X] Cuttlefish
>> [ ] Dumbo Octopus
>> [X] Angler fish
>> [X] Capybara
>>
>> Em ter., 19 de nov. de 2019 às 23:43, Kenneth Knowles 
>> escreveu:
>>
>>> Please cast your votes of approval [1] for animals you would support as
>>> Beam mascot. The animal with the most approval will be identified as the
>>> favorite.
>>>
>>> *** Vote for as many as you like, using this checklist as a template 
>>>
>>> [ ] Beaver
>>> [ ] Hedgehog
>>> [ ] Lemur
>>> [ ] Owl
>>> [ ] Salmon
>>> [ ] Trout
>>> [ ] Robot dinosaur
>>> [ ] Firefly
>>> [ ] Cuttlefish
>>> [ ] Dumbo Octopus
>>> [ ] Angler fish
>>>
>>> This vote will remain open for at least 72 hours.
>>>
>>> Kenn
>>>
>>> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
>>> https://www.electionscience.org/library/approval-voting/
>>>
>>
>>
>> --
>> []s
>>
>> Leonardo Alves Miguel
>> Data Engineer
>> (16) 3509-5515 | www.arquivei.com.br
>> 
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> 
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> 
>> 
>> 
>> 
>>
>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509- | www.arquivei.com.br
> 
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> 
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> 
> 
> 
> 
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Udi Meiri]

Congrats Daniel!

On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver  wrote:

> Congrats Dan! Keep up the good work :)
>
> On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden  wrote:
>
>> Congratulations! This is great news.
>>
>> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:
>>
>>> Congrats!
>>>
>>>
>>> -Rui
>>>
>>> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congrats, Daniel!

 On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Daniel Oliveira
>
> Daniel introduced himself to dev@ over two years ago and has
> contributed in many ways since then. Daniel has contributed to general
> project health, the portability framework, and all three languages: Java,
> Python SDK, and Go. I would like to particularly highlight how he deleted
> 12k lines of dead reference runner code [1].
>
> In consideration of Daniel's contributions, the Beam PMC trusts him
> with the responsibilities of a Beam committer [2].
>
> Thank you, Daniel, for your contributions and looking forward to many
> more!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1] https://github.com/apache/beam/pull/8380
> [2]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>



smime.p7s
Description: S/MIME Cryptographic Signature

[PROPOSAL] Preparing for Beam 2.18 release

[Udi Meiri]

Hi all,

The next (2.18) release branch cut is scheduled for Dec 4, according to the
calendar

.
I would like to volunteer myself to do this release.
The plan is to cut the branch on that date, and cherrypick release-blocking
fixes afterwards if any.

Any unresolved release blocking JIRA issues for 2.18 should have their "Fix
Version/s" marked as "2.18.0".

Any comments or objections?


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Kyle Weaver]

Congrats Dan! Keep up the good work :)

On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden  wrote:

> Congratulations! This is great news.
>
> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:
>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> Congrats, Daniel!
>>>
>>> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Daniel Oliveira

 Daniel introduced himself to dev@ over two years ago and has
 contributed in many ways since then. Daniel has contributed to general
 project health, the portability framework, and all three languages: Java,
 Python SDK, and Go. I would like to particularly highlight how he deleted
 12k lines of dead reference runner code [1].

 In consideration of Daniel's contributions, the Beam PMC trusts him
 with the responsibilities of a Beam committer [2].

 Thank you, Daniel, for your contributions and looking forward to many
 more!

 Kenn, on behalf of the Apache Beam PMC

 [1] https://github.com/apache/beam/pull/8380
 [2]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Cyrus Maden]

Congratulations! This is great news.

On Wed, Nov 20, 2019 at 3:24 PM Rui Wang  wrote:

> Congrats!
>
>
> -Rui
>
> On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev 
> wrote:
>
>> Congrats, Daniel!
>>
>> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Daniel Oliveira
>>>
>>> Daniel introduced himself to dev@ over two years ago and has
>>> contributed in many ways since then. Daniel has contributed to general
>>> project health, the portability framework, and all three languages: Java,
>>> Python SDK, and Go. I would like to particularly highlight how he deleted
>>> 12k lines of dead reference runner code [1].
>>>
>>> In consideration of Daniel's contributions, the Beam PMC trusts him with
>>> the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Daniel, for your contributions and looking forward to many
>>> more!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1] https://github.com/apache/beam/pull/8380
>>> [2]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Rui Wang]

Congrats!


-Rui

On Wed, Nov 20, 2019 at 11:48 AM Valentyn Tymofieiev 
wrote:

> Congrats, Daniel!
>
> On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new committer:
>> Daniel Oliveira
>>
>> Daniel introduced himself to dev@ over two years ago and has contributed
>> in many ways since then. Daniel has contributed to general project health,
>> the portability framework, and all three languages: Java, Python SDK, and
>> Go. I would like to particularly highlight how he deleted 12k lines of dead
>> reference runner code [1].
>>
>> In consideration of Daniel's contributions, the Beam PMC trusts him with
>> the responsibilities of a Beam committer [2].
>>
>> Thank you, Daniel, for your contributions and looking forward to many
>> more!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1] https://github.com/apache/beam/pull/8380
>> [2]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: GCP libraries up-to-date versions in Java

[Kenneth Knowles]

On Wed, Nov 20, 2019 at 4:05 AM Elliotte Rusty Harold 
wrote:

> BOM or no BOM is an implementation detail.


Agreed for the most part.


> Using com.google.cloud:libraries-bom would make dependency management
> simpler for developers, but the real issue is whether Beam can continue to
> work with very old versions of the many libraries it depends on. Even if
> this is acceptable for Beam, it's unlikely to be feasible for anyone who
> needs to mix Beam code with other code.


I believe every version of Beam's dependencies has been, and should
continue to be, driven by what is best for Beam's users. That does mean
making it easy for them to use the latest compatible version of their
favorite libraries.

There should be no self-incompatibility between Google minor version
> releases. All the Google libraries in question follow semantic versioning.
> E.g. Pubsub 1.43 would be fully API compatible with Pubsub 1.28, though not
> the reverse. However there are likely to be important bug fixes in 1.43 and
> definitely new features that 1.28 would not have. If there are any edge
> cases where this is not true, that's a bug and if you file it against the
> repo we'll try to fix it. We're also installing tooling to make this less
> likely to happen by accident. However, right now any such problem is rare.
>

I'm glad we share the same ideals. If things were as good as you described,
then we would have two good properties:

1. Users would always be able to force a newer minor version to trivially
work around Beam's deps
2. Beam could always upgrade minor versions with no code change in Beam and
no code change by users

My experience is that this rarely works so simply. Generally, a user forces
a new version of a library and it turns out that library or its
dependencies has broken compatibility.

Just reiterating that if semver really holds in these cases, then this
proposal is fine with me. And if semver doesn't hold, I still think we
should try to support the latest, but may also need to maintain a connector
to support older versions that are still in wide use.


> Looking at Beam's dependencies, the only case where there are major
> version changes to address is Guava.
>

Beam has vendored Guava so it is mostly beside the point. Upgrading the
vendored Guava does not interact with any of Beam's dependencies. See
https://lists.apache.org/thread.html/c477d120a4c4626cbe675f8b03d84c6fe7938e36c8e2b55c492224cf@%3Cdev.beam.apache.org%3E

Only KinesisIO and the ZetaSQL-to-Calcite translator actually have
essential dependencies on Guava. In these cases, the version of Guava must
necessarily be compatible with the Kinesis client and ZetaSQL,
respectively. They may or may not be able to interop, and that is mostly
out of our hands.

The remaining issues are pre-1.0 libraries. OpenCensus is a particular
> thorn in my side. Ideally these should not be used, at all. However if we
> must, we should not expose them on the Beam API surface and we need to move
> them forward quickly as they change.
>

This might deserve its own thread. This sounds like it should be
well-hidden, vendored, or well-marked as "experimental".

Kenn

Re: [discuss] Using a logger hierarchy in Python

[Valentyn Tymofieiev]

Based on my recent debugging experience for
https://issues.apache.org/jira/browse/BEAM-8651, I think it may be helpful
to include thread IDs, into the log entries, or have an option to easily
enable this. I imagine that having process ID may also be helpful in other
situations.

On Tue, Nov 19, 2019 at 11:17 AM Chad Dombrova  wrote:

> Pablo, it might be necessary to setup a root logging handler if one does
> not exist already.  I noticed that a LocalJobServicer that I was testing
> against stopped emitting tracebacks when I rebased onto the latest from
> master.  Setting up the root handler fixed it.  I'm still testing this, and
> I might be misinterpreting what I saw, but I wanted to get eyes on it in
> case I don't have time to get a definitive answer.
>
> -chad
>
>
>
> On Fri, Nov 15, 2019 at 4:30 PM Pablo Estrada  wrote:
>
>> Thanks all,
>> 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to
>> switch the variable name to _log via sed.
>> Best
>> -P.
>>
>> On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver  wrote:
>>
>>> +1 for per-module loggers (what Robert said).
>>>
>>> On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri  wrote:
>>>
 +1, but can we use something less verbose and shift key heavy than
 _LOGGER like log or _log?

 Also please dedupe with these existing bugs:
 https://issues.apache.org/jira/browse/BEAM-3523
 https://issues.apache.org/jira/browse/BEAM-1825

 On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise  wrote:

> Awesome, thanks Chad!
>
> On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova 
> wrote:
>
>> Hi Thomas,
>>
>>
>>> Will this include the ability for users to configure logging via
>>> pipeline options?
>>>
>>
>> We're working on a proposal to allow pluggable logging handlers that
>> can be configured via pipeline options.  For example, it would allow you 
>> to
>> add a new logging handler for StackDriver or Elasticsearch.  Will 
>> hopefully
>> have a document to share soon.
>>
>> -chad
>>
>>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Leonardo Miguel]

[ ] Beaver
[X] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[X] Cuttlefish
[ ] Dumbo Octopus
[X] Angler fish
[X] Capybara

Em ter., 19 de nov. de 2019 às 23:43, Kenneth Knowles 
escreveu:

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>


-- 
[]s

Leonardo Alves Miguel
Data Engineer
(16) 3509-5515 | www.arquivei.com.br

[image: Arquivei.com.br – Inteligência em Notas Fiscais]

[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Eduardo Soldera]

[X ] Beaver
[] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[X ] Robot dinosaur
[ ] Firefly
[] Cuttlefish
[ ] Dumbo Octopus
[] Angler fish
[X] Capybara

Em qua., 20 de nov. de 2019 às 17:03, Leonardo Miguel <
leonardo.mig...@arquivei.com.br> escreveu:

> [ ] Beaver
> [X] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [X] Cuttlefish
> [ ] Dumbo Octopus
> [X] Angler fish
> [X] Capybara
>
> Em ter., 19 de nov. de 2019 às 23:43, Kenneth Knowles 
> escreveu:
>
>> Please cast your votes of approval [1] for animals you would support as
>> Beam mascot. The animal with the most approval will be identified as the
>> favorite.
>>
>> *** Vote for as many as you like, using this checklist as a template 
>>
>> [ ] Beaver
>> [ ] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>> This vote will remain open for at least 72 hours.
>>
>> Kenn
>>
>> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
>> https://www.electionscience.org/library/approval-voting/
>>
>
>
> --
> []s
>
> Leonardo Alves Miguel
> Data Engineer
> (16) 3509-5515 | www.arquivei.com.br
> 
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> 
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> 
> 
> 
> 
>


-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509- | www.arquivei.com.br

[image: Arquivei.com.br – Inteligência em Notas Fiscais]

[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Robert Bradshaw]

On Tue, Nov 19, 2019 at 6:43 PM Kenneth Knowles  wrote:
>
> Please cast your votes of approval [1] for animals you would support as Beam 
> mascot. The animal with the most approval will be identified as the favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [X] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [X] Firefly
> [X] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and 
> https://www.electionscience.org/library/approval-voting/

Re: [ANNOUNCE] New committer: Daniel Oliveira

[Valentyn Tymofieiev]

Congrats, Daniel!

On Wed, Nov 20, 2019 at 11:47 AM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new committer:
> Daniel Oliveira
>
> Daniel introduced himself to dev@ over two years ago and has contributed
> in many ways since then. Daniel has contributed to general project health,
> the portability framework, and all three languages: Java, Python SDK, and
> Go. I would like to particularly highlight how he deleted 12k lines of dead
> reference runner code [1].
>
> In consideration of Daniel's contributions, the Beam PMC trusts him with
> the responsibilities of a Beam committer [2].
>
> Thank you, Daniel, for your contributions and looking forward to many more!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1] https://github.com/apache/beam/pull/8380
> [2]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

[ANNOUNCE] New committer: Daniel Oliveira

[Kenneth Knowles]

Hi all,

Please join me and the rest of the Beam PMC in welcoming a new committer:
Daniel Oliveira

Daniel introduced himself to dev@ over two years ago and has contributed in
many ways since then. Daniel has contributed to general project health, the
portability framework, and all three languages: Java, Python SDK, and Go. I
would like to particularly highlight how he deleted 12k lines of dead
reference runner code [1].

In consideration of Daniel's contributions, the Beam PMC trusts him with
the responsibilities of a Beam committer [2].

Thank you, Daniel, for your contributions and looking forward to many more!

Kenn, on behalf of the Apache Beam PMC

[1] https://github.com/apache/beam/pull/8380
[2]
https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

Re: GCP libraries up-to-date versions in Java

[Elliotte Rusty Harold]

On Wed, Nov 20, 2019 at 1:43 PM Luke Cwik  wrote:
>
> Minor note that Gradle 5 added support for BOMs[1].
>
> I think attempting to perform the upgrade (whether to use BOM or not) will be 
> a concerted effort every time to minimize the amount of breakage to users 
> while maximizing compatibility with the OSS ecosystem. Unfortunately I'm not 
> aware of any dependency analysis tooling that can perform some validation 
> stating that something is safe or not. If such a tool existed, it would make 
> it much easier for projects to perform upgrades and would also help users as 
> well.

Funny you should ask. We've been working on tooling like that such as
the linkage monitor and the Maven enforcer rule:

https://github.com/GoogleCloudPlatform/cloud-opensource-java

Mostly Maven based for the time being, but updates are possible.

At the end of the day, though, this is no substitute for extensive
unit and integration test suites. If we don't have those, then
dependencies are the least of our worries. If we do have those, we can
move forward  in reasonable confidence that minor version upgrades
won't break anything without causing a test to fail.

-- 
Elliotte Rusty Harold
elh...@ibiblio.org

Re: Portable runner bundle scheduling (Streaming/Python/Flink)

[Luke Cwik]

Dataflow has run into this issue as well. Dataflow has "work items" that
are converted into bundles that are executed on the SDK. Each work item
does a greedy assignment to the SDK worker with the fewest work items
assigned. As you surmised, we use SDF splitting in batch pipelines to
balance work. We would like to use splitting of SDFs in streaming pipelines
as well but Dataflow can't handle it as of right now.

As part of a few PRs, I have added basic SDF expansion to the shared runner
lib and slowly exposed the runner side hooks[2, 3] for SDK initiated
checkpointing and bundle finalization. There are still a few pieces left:
* exposing an API so the bundle can be split during execution
* adding the limited depth splitting logic that would add a basic form of
dynamic work rebalancing for all runners that decide to use it

1: https://github.com/apache/beam/pull/10045
2: https://github.com/apache/beam/pull/10065
3: https://github.com/apache/beam/pull/10074

On Wed, Nov 20, 2019 at 10:49 AM Thomas Weise  wrote:

> We found a problem with uneven utilization of SDK workers causing
> excessive latency with Streaming/Python/Flink. Remember that with Python,
> we need to execute multiple worker processes on a machine instead of
> relying on threads in a single worker, which requires the runner to make a
> decision to which worker to give a bundle for processing.
>
> The Flink runner has knobs to influence the number of records per bundle
> and the maximum duration for a bundle. But since the runner does not
> understand the cost of an individual record, it is possible that the
> duration of bundles fluctuates significantly due to the skew in processing
> time of individual records. And unless the bundle size is 1, multiple
> expensive records could be allocated to a single bundle before the cutoff
> time is reached. We notice this with a pipeline that executes models, but
> there are other use cases where the cost of individual records can vary
> significantly.
>
> Additionally, the Flink runner establishes the association between the
> subtask managing an executable stage and the SDK worker during
> initialization, lasting for the duration of the job. In other words,
> bundles for the same executable stage will always be sent to the same SDK
> worker. When the execution time skew is tied to specific keys (stateful
> processing), it further aggravates the issue.
>
> I started experimenting with the ability to schedule bundles on any
> available worker. Initially I'm trying a very basic approach, starting
> processing of a bundle only on a free environment (one that does not
> process any other bundle). This effectively removes the pipelining between
> subtask and SDK worker. Potentially waiting for an available environment is
> acceptable in this case, as the per bundle overhead is very small compared
> to the per record cost.
>
> However, even if this suffices for the use case I'm looking at, this is an
> area that will probably need more work going forward. Rather than the
> runner guessing how to schedule bundles, I think that the best long term
> solution would be SDF, where the user code can decide that something takes
> too long and defer remaining work (and the runner can redistribute it).
>
> Curious if anyone else has run into this issue yet and what other ideas
> there may be?
>
> Thanks,
> Thomas
>
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Austin Bennett]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[x ] Salmon
[x ] Trout
[x ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish



On Tue, Nov 19, 2019 at 6:43 PM Kenneth Knowles  wrote:

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Chris Collins]

On 2019/11/20 02:43:42, Kenneth Knowles  wrote: 
> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
> 
> *** Vote for as many as you like, using this checklist as a template 
> 
> [x] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [x] Salmon
> [x] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [x] Dumbo Octopus
> [ ] Angler fish
> 
> This vote will remain open for at least 72 hours.
> 
> Kenn
> 
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
> 

Portable runner bundle scheduling (Streaming/Python/Flink)

[Thomas Weise]

We found a problem with uneven utilization of SDK workers causing excessive
latency with Streaming/Python/Flink. Remember that with Python, we need to
execute multiple worker processes on a machine instead of relying on
threads in a single worker, which requires the runner to make a decision to
which worker to give a bundle for processing.

The Flink runner has knobs to influence the number of records per bundle
and the maximum duration for a bundle. But since the runner does not
understand the cost of an individual record, it is possible that the
duration of bundles fluctuates significantly due to the skew in processing
time of individual records. And unless the bundle size is 1, multiple
expensive records could be allocated to a single bundle before the cutoff
time is reached. We notice this with a pipeline that executes models, but
there are other use cases where the cost of individual records can vary
significantly.

Additionally, the Flink runner establishes the association between the
subtask managing an executable stage and the SDK worker during
initialization, lasting for the duration of the job. In other words,
bundles for the same executable stage will always be sent to the same SDK
worker. When the execution time skew is tied to specific keys (stateful
processing), it further aggravates the issue.

I started experimenting with the ability to schedule bundles on any
available worker. Initially I'm trying a very basic approach, starting
processing of a bundle only on a free environment (one that does not
process any other bundle). This effectively removes the pipelining between
subtask and SDK worker. Potentially waiting for an available environment is
acceptable in this case, as the per bundle overhead is very small compared
to the per record cost.

However, even if this suffices for the use case I'm looking at, this is an
area that will probably need more work going forward. Rather than the
runner guessing how to schedule bundles, I think that the best long term
solution would be SDF, where the user code can decide that something takes
too long and defer remaining work (and the runner can redistribute it).

Curious if anyone else has run into this issue yet and what other ideas
there may be?

Thanks,
Thomas

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Ruoyun Huang]

[ ] Beaver
[] Hedgehog
[ ] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[x] Firefly
[ ] Cuttlefish
[x] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 10:38 AM  wrote:

> [ ] Beaver
> [x] Hedgehog
> [ ] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [x] Cuttlefish
> [x] Dumbo Octopus
> [ ] Angler fish
>
>
>
> On 2019/11/20 02:43:42, Kenneth Knowles  wrote:
> > Please cast your votes of approval [1] for animals you would support as>
> > Beam mascot. The animal with the most approval will be identified as
> the>
> > favorite.>
> >
> > *** Vote for as many as you like, using this checklist as a template
> >
> >
> > [ ] Beaver>
> > [ ] Hedgehog>
> > [ ] Lemur>
> > [ ] Owl>
> > [ ] Salmon>
> > [ ] Trout>
> > [ ] Robot dinosaur>
> > [ ] Firefly>
> > [ ] Cuttlefish>
> > [ ] Dumbo Octopus>
> > [ ] Angler fish>
> >
> > This vote will remain open for at least 72 hours.>
> >
> > Kenn>
> >
> > [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and>
> > https://www.electionscience.org/library/approval-voting/>
> >
>
>
>

-- 

Ruoyun  Huang

Re: GCP libraries up-to-date versions in Java

[Luke Cwik]

Minor note that Gradle 5 added support for BOMs[1].

I think attempting to perform the upgrade (whether to use BOM or not) will
be a concerted effort every time to minimize the amount of breakage to
users while maximizing compatibility with the OSS ecosystem. Unfortunately
I'm not aware of any dependency analysis tooling that can perform some
validation stating that something is safe or not. If such a tool existed,
it would make it much easier for projects to perform upgrades and would
also help users as well.

1:
https://dzone.com/articles/gradle-goodness-use-bill-of-materials-bom-as-depen

On Wed, Nov 20, 2019 at 4:05 AM Elliotte Rusty Harold 
wrote:

> BOM or no BOM is an implementation detail. Using
> com.google.clou:libraries-bom would make dependency management simpler
> for developers, but the real issue is whether Beam can continue to
> work with very old versions of the many libraries it depends on. Even
> if this is acceptable for Beam, it's unlikely to be feasible for
> anyone who needs to mix Beam code with other code.
>
> There should be no self-incompatibility between Google minor version
> releases. All the Google libraries in question follow semantic
> versioning. E.g. Pubsub 1.43 would be fully API compatible with Pubsub
> 1.28, though not the reverse. However there are likely to be important
> bug fixes in 1.43 and definitely new features that 1.28 would not
> have. If there are any edge cases where this is not true, that's a bug
> and if you file it against the repo we'll try to fix it. We're also
> installing tooling to make this less likely to happen by accident.
> However, right now any such problem is rare.
>
> Behavior differences are another story. It is entirely possible that
> something like Pubsub 1.28 would simply no longer function due to
> changes at the backend. There's a deprecation cycle, announcements,
> and transition periods in all such cases; but a project like Beam
> can't stay on old versions forever. If they try, the backends will
> shift out from under them.
>
> Looking at Beam's dependencies, the only case where there are major
> version changes to address is Guava.
> This will take some work, but not an excessive amount. We should be
> able to move this up to 28.1-android with few code changes and no
> further API breaking changes in that library are planned for the
> future.
>
> The remaining issues are pre-1.0 libraries. OpenCensus is a particular
> thorn in my side. Ideally these should not be used, at all. However if
> we must, we should not expose them on the Beam API surface and we need
> to move them forward quickly as they change.
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[josh . stone]

[ ] Beaver
[x] Hedgehog
[ ] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[x] Cuttlefish
[x] Dumbo Octopus
[ ] Angler fish


On 2019/11/20 02:43:42, Kenneth Knowles http://apache.org/>> 
wrote: 
> Please cast your votes of approval [1] for animals you would support as> 
> Beam mascot. The animal with the most approval will be identified as the> 
> favorite.> 
> 
> *** Vote for as many as you like, using this checklist as a template > 
> 
> [ ] Beaver> 
> [ ] Hedgehog> 
> [ ] Lemur> 
> [ ] Owl> 
> [ ] Salmon> 
> [ ] Trout> 
> [ ] Robot dinosaur> 
> [ ] Firefly> 
> [ ] Cuttlefish> 
> [ ] Dumbo Octopus> 
> [ ] Angler fish> 
> 
> This vote will remain open for at least 72 hours.> 
> 
> Kenn> 
> 
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description 
>  and> 
> https://www.electionscience.org/library/approval-voting/ 
> > 
> 

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Pablo Estrada]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[x] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 10:27 AM Mia Gil Epner 
wrote:

> [x] Beaver
> [x] Hedgehog
> [x] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [x] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [x] Dumbo Octopus
> [ ] Angler fish
>
> On Tue, Nov 19, 2019 at 6:43 PM Kenneth Knowles  wrote:
>
>> Please cast your votes of approval [1] for animals you would support as
>> Beam mascot. The animal with the most approval will be identified as the
>> favorite.
>>
>> *** Vote for as many as you like, using this checklist as a template 
>>
>> [ ] Beaver
>> [ ] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>> This vote will remain open for at least 72 hours.
>>
>> Kenn
>>
>> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
>> https://www.electionscience.org/library/approval-voting/
>>
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Mia Gil Epner]

[x] Beaver
[x] Hedgehog
[x] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[x] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[x] Dumbo Octopus
[ ] Angler fish

On Tue, Nov 19, 2019 at 6:43 PM Kenneth Knowles  wrote:

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Luke Cwik]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[X] Cuttlefish
[ ] Dumbo Octopus
[X] Angler fish

On Wed, Nov 20, 2019 at 10:08 AM Eugene Kirpichov  wrote:

> [ ] Beaver
> [ ] Hedgehog
> [X] Lemur
> [X] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [X] Firefly
> [X] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> On Wed, Nov 20, 2019 at 9:47 AM Valentyn Tymofieiev 
> wrote:
>
>> [ ] Beaver
>> [X] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [X] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>> On Tue, Nov 19, 2019 at 6:44 PM Kenneth Knowles  wrote:
>>
>>> Please cast your votes of approval [1] for animals you would support as
>>> Beam mascot. The animal with the most approval will be identified as the
>>> favorite.
>>>
>>> *** Vote for as many as you like, using this checklist as a template 
>>>
>>> [ ] Beaver
>>> [ ] Hedgehog
>>> [ ] Lemur
>>> [ ] Owl
>>> [ ] Salmon
>>> [ ] Trout
>>> [ ] Robot dinosaur
>>> [ ] Firefly
>>> [ ] Cuttlefish
>>> [ ] Dumbo Octopus
>>> [ ] Angler fish
>>>
>>> This vote will remain open for at least 72 hours.
>>>
>>> Kenn
>>>
>>> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
>>> https://www.electionscience.org/library/approval-voting/
>>>
>>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Eugene Kirpichov]

[ ] Beaver
[ ] Hedgehog
[X] Lemur
[X] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[X] Firefly
[X] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 9:47 AM Valentyn Tymofieiev 
wrote:

> [ ] Beaver
> [X] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [X] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> On Tue, Nov 19, 2019 at 6:44 PM Kenneth Knowles  wrote:
>
>> Please cast your votes of approval [1] for animals you would support as
>> Beam mascot. The animal with the most approval will be identified as the
>> favorite.
>>
>> *** Vote for as many as you like, using this checklist as a template 
>>
>> [ ] Beaver
>> [ ] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>> This vote will remain open for at least 72 hours.
>>
>> Kenn
>>
>> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
>> https://www.electionscience.org/library/approval-voting/
>>
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Valentyn Tymofieiev]

[ ] Beaver
[X] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Tue, Nov 19, 2019 at 6:44 PM Kenneth Knowles  wrote:

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Dustin Rhodes]

[ x] Beaver
[ x] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ x] Firefly
[ x] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 9:28 AM Sam Rohde  wrote:

> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [x] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> On Wed, Nov 20, 2019 at 9:22 AM Alex Amato  wrote:
>
>> [ ] Beaver
>> [ ] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [X] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>>
>> On Wed, Nov 20, 2019 at 9:15 AM Kirill Kozlov 
>> wrote:
>>
>>> [ ] Beaver
>>> [ ] Hedgehog
>>> [X] Lemur
>>> [X] Owl
>>> [ ] Salmon
>>> [ ] Trout
>>> [ ] Robot dinosaur
>>> [ ] Firefly
>>> [ ] Cuttlefish
>>> [ ] Dumbo Octopus
>>> [X] Angler fish
>>>
>>>
>>> On Wed, Nov 20, 2019, 08:38 Cyrus Maden  wrote:
>>>
 Here's my vote, but I'm curious about the distinction between salmon
 and trout mascots :)

 [ ] Beaver
 [ ] Hedgehog
 [ X] Lemur
 [ ] Owl
 [ X] Salmon
 [ ] Trout
 [ ] Robot dinosaur
 [ X] Firefly
 [ ] Cuttlefish
 [ ] Dumbo Octopus
 [ X] Angler fish

 On Wed, Nov 20, 2019 at 11:24 AM Allan Wilson 
 wrote:

>
>
> On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:
>
> *** Vote for as many as you like, using this checklist as a
> template 
>
> [] Beaver
> [X] Hedgehog
> [X ] Lemur
> [ ] Owl
> [ ] Salmon
> [] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
>
>

Re: Library to Parse Thrift Files for ThriftIO

[Christopher Larsen]

Thanks Reuven and Ryan. Unfortunately the drift library doesn't include the
parser that was created in the original which is why we were thinking about
updating the older library which does include the parser.

As far as a better supported library we haven't found anything in pure Java
that includes a parser.

Best,
Chris

On Tue, Nov 19, 2019 at 7:21 AM Ryan Skraba  wrote:

> For info: https://github.com/airlift/drift has forked and maintained
> the code over the last few years.
>
> On Fri, Nov 15, 2019 at 7:23 PM Reuven Lax  wrote:
> >
> > At a quick glance, the license is Apache which is fine (though we'd have
> to check dependencies as well). I do notice that git repro is no longer
> maintained; is there a different one that is better supported?
> >
> > Reuven
> >
> > On Wed, Nov 13, 2019 at 7:04 AM Christopher Larsen <
> christopher.lar...@quantiphi.com> wrote:
> >>
> >> Hey everyone,
> >>
> >> In regards to the library that will be used to parse the .thrift files
> for ThriftIO we are thinking about using some of the code from the library
> found here and updating it to use Beam friendly packages. We would love to
> get the community's feedback on this approach.
> >>
> >> Best,
> >> Chris
> >>
> >> This message contains information that may be privileged or
> confidential and is the property of the Quantiphi Inc and/or its
> affiliates. It is intended only for the person to whom it is addressed. If
> you are not the intended recipient, any review, dissemination,
> distribution, copying, storage or other use of all or any portion of this
> message is strictly prohibited. If you received this message in error,
> please immediately notify the sender by reply e-mail and delete this
> message in its entirety
>

-- 
_This message contains information that may be privileged or confidential 
and is the property of the Quantiphi Inc and/or its affiliates_. It is 
intended only for the person to whom it is addressed. _If you are not the 
intended recipient, any review, dissemination, distribution, copying, 
storage or other use of all or any portion of this message is strictly 
prohibited. If you received this message in error, please immediately 
notify the sender by reply e-mail and delete this message in its 
*entirety*___

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Sam Rohde]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[x] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 9:22 AM Alex Amato  wrote:

> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [X] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
>
> On Wed, Nov 20, 2019 at 9:15 AM Kirill Kozlov 
> wrote:
>
>> [ ] Beaver
>> [ ] Hedgehog
>> [X] Lemur
>> [X] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [X] Angler fish
>>
>>
>> On Wed, Nov 20, 2019, 08:38 Cyrus Maden  wrote:
>>
>>> Here's my vote, but I'm curious about the distinction between salmon and
>>> trout mascots :)
>>>
>>> [ ] Beaver
>>> [ ] Hedgehog
>>> [ X] Lemur
>>> [ ] Owl
>>> [ X] Salmon
>>> [ ] Trout
>>> [ ] Robot dinosaur
>>> [ X] Firefly
>>> [ ] Cuttlefish
>>> [ ] Dumbo Octopus
>>> [ X] Angler fish
>>>
>>> On Wed, Nov 20, 2019 at 11:24 AM Allan Wilson 
>>> wrote:
>>>


 On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:

 *** Vote for as many as you like, using this checklist as a
 template 

 [] Beaver
 [X] Hedgehog
 [X ] Lemur
 [ ] Owl
 [ ] Salmon
 [] Trout
 [ ] Robot dinosaur
 [ ] Firefly
 [ ] Cuttlefish
 [ ] Dumbo Octopus
 [ ] Angler fish

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Alex Amato]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[X] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish


On Wed, Nov 20, 2019 at 9:15 AM Kirill Kozlov 
wrote:

> [ ] Beaver
> [ ] Hedgehog
> [X] Lemur
> [X] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [X] Angler fish
>
>
> On Wed, Nov 20, 2019, 08:38 Cyrus Maden  wrote:
>
>> Here's my vote, but I'm curious about the distinction between salmon and
>> trout mascots :)
>>
>> [ ] Beaver
>> [ ] Hedgehog
>> [ X] Lemur
>> [ ] Owl
>> [ X] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ X] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ X] Angler fish
>>
>> On Wed, Nov 20, 2019 at 11:24 AM Allan Wilson 
>> wrote:
>>
>>>
>>>
>>> On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:
>>>
>>> *** Vote for as many as you like, using this checklist as a template
>>> 
>>>
>>> [] Beaver
>>> [X] Hedgehog
>>> [X ] Lemur
>>> [ ] Owl
>>> [ ] Salmon
>>> [] Trout
>>> [ ] Robot dinosaur
>>> [ ] Firefly
>>> [ ] Cuttlefish
>>> [ ] Dumbo Octopus
>>> [ ] Angler fish
>>>
>>>
>>>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Kirill Kozlov]

[ ] Beaver
[ ] Hedgehog
[X] Lemur
[X] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[X] Angler fish


On Wed, Nov 20, 2019, 08:38 Cyrus Maden  wrote:

> Here's my vote, but I'm curious about the distinction between salmon and
> trout mascots :)
>
> [ ] Beaver
> [ ] Hedgehog
> [ X] Lemur
> [ ] Owl
> [ X] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ X] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ X] Angler fish
>
> On Wed, Nov 20, 2019 at 11:24 AM Allan Wilson 
> wrote:
>
>>
>>
>> On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:
>>
>> *** Vote for as many as you like, using this checklist as a template
>> 
>>
>> [] Beaver
>> [X] Hedgehog
>> [X ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>>
>>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Kyle Weaver]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[X] Robot dinosaur
[X] Firefly
[X] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish


On Wed, Nov 20, 2019 at 8:38 AM Cyrus Maden  wrote:

> Here's my vote, but I'm curious about the distinction between salmon and
> trout mascots :)
>
> [ ] Beaver
> [ ] Hedgehog
> [ X] Lemur
> [ ] Owl
> [ X] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ X] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ X] Angler fish
>
> On Wed, Nov 20, 2019 at 11:24 AM Allan Wilson 
> wrote:
>
>>
>>
>> On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:
>>
>> *** Vote for as many as you like, using this checklist as a template
>> 
>>
>> [] Beaver
>> [X] Hedgehog
>> [X ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>>
>>

Re: [spark structured streaming runner] available on master

[Etienne Chauchot]

Forgot to say thanks everyone for their contribution to this especially 
Alexey, Ryan and Ismael.


Etienne

On 20/11/2019 17:12, Etienne Chauchot wrote:

Hi all,

I'm glad to announce that the new Spark runner based on Spark 
structured streaming framework has been merged into master !


It is not based on RDD/DStream API. See 
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html


It is still experimental, its coverage of the Beam model is partial:

- the runner passes 95% of the validates runner tests in batch mode.

- It does not have support for streaming yet (waiting for the 
multi-aggregations support in spark StructuredStreaming framework from 
the Spark community)


- Runner can execute Nexmark : perfkit dashboards yet to come

- Some things are not wired up yet:

    - Beam Schemas not wired up

    - Optional features of the model not implemented:  state api, 
timer api, splittable doFn api, …


I will submit a PR to update the capability matrix in the coming days.

Best

Etienne

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Cyrus Maden]

Here's my vote, but I'm curious about the distinction between salmon and
trout mascots :)

[ ] Beaver
[ ] Hedgehog
[ X] Lemur
[ ] Owl
[ X] Salmon
[ ] Trout
[ ] Robot dinosaur
[ X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ X] Angler fish

On Wed, Nov 20, 2019 at 11:24 AM Allan Wilson  wrote:

>
>
> On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:
>
> *** Vote for as many as you like, using this checklist as a template
> 
>
> [] Beaver
> [X] Hedgehog
> [X ] Lemur
> [ ] Owl
> [ ] Salmon
> [] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
>
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Allan Wilson]

On 11/20/19, 8:44 AM, "Ryan Skraba"  wrote:

*** Vote for as many as you like, using this checklist as a template 

[] Beaver
[X] Hedgehog
[X ] Lemur
[ ] Owl
[ ] Salmon
[] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Jozef Vilcek]

On Wed, Nov 20, 2019 at 3:43 AM Kenneth Knowles  wrote:

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [X] Lemur
> [X] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [X] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>

[spark structured streaming runner] available on master

[Etienne Chauchot]

Hi all,

I'm glad to announce that the new Spark runner based on Spark structured 
streaming framework has been merged into master !


It is not based on RDD/DStream API. See 
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html


It is still experimental, its coverage of the Beam model is partial:

- the runner passes 95% of the validates runner tests in batch mode.

- It does not have support for streaming yet (waiting for the 
multi-aggregations support in spark StructuredStreaming framework from 
the Spark community)


- Runner can execute Nexmark : perfkit dashboards yet to come

- Some things are not wired up yet:

    - Beam Schemas not wired up

    - Optional features of the model not implemented:  state api, timer 
api, splittable doFn api, …


I will submit a PR to update the capability matrix in the coming days.

Best

Etienne

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Ryan Skraba]

*** Vote for as many as you like, using this checklist as a template 

[X] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[X] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Jan Lukavský]

[ ] Beaver
[x] Hedgehog
[ ] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[x] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On 11/20/19 3:01 PM, Robin Qiu wrote:


[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ X] Angler fish

On Wed, Nov 20, 2019 at 2:45 PM Alexey Romanenko 
mailto:aromanenko@gmail.com>> wrote:


[ ] Beaver
[x] Hedgehog
[ ] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[x] Firefly
[ ] Cuttlefish
[x] Dumbo Octopus
[ ] Angler fish


On 20 Nov 2019, at 14:38, Manu Zhang mailto:owenzhang1...@gmail.com>> wrote:

[] Beaver
[] Hedgehog
[x] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 8:38 PM Maximilian Michels
mailto:m...@apache.org>> wrote:

[ ] Beaver
[ ] Hedgehog
[x] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[x] Firefly
[x] Cuttlefish
[ ] Dumbo Octopus
[x] Angler fish

On 20.11.19 08:18, Alex Van Boxel wrote:
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ X] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ X] Angler fish
>
>   _/
> _/ Alex Van Boxel
>
>
> On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni mailto:r...@google.com>
> >> wrote:
>
>     [ ] Beaver
>     [ ] Hedgehog
>     [ ] Lemur
>     [ ] Owl
>     [X] Salmon
>     [ ] Trout
>     [ ] Robot dinosaur
>     [ ] Firefly
>     [ ] Cuttlefish
>     [X] Dumbo Octopus
>     [X] Angler fish
>
>     On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles
mailto:k...@apache.org>
>     >> wrote:
>
>         Please cast your votes of approval [1] for animals
you would
>         support as Beam mascot. The animal with the most
approval will
>         be identified as the favorite.
>
>         *** Vote for as many as you like, using this
checklist as a
>         template 
>
>         [ ] Beaver
>         [ ] Hedgehog
>         [ ] Lemur
>         [ ] Owl
>         [ ] Salmon
>         [ ] Trout
>         [ ] Robot dinosaur
>         [ ] Firefly
>         [ ] Cuttlefish
>         [ ] Dumbo Octopus
>         [ ] Angler fish
>
>         This vote will remain open for at least 72 hours.
>
>         Kenn
>
>         [1] See
> https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>
>
>
>     --
>
>     This email may be confidential and privileged. If you
received this
>     communication by mistake, please don't forward it to
anyone else,
>     please erase all copies and attachments, and please let
me know that
>     it has gone to the wrong person.
>
>     The above terms reflect a potential business
arrangement, are
>     provided solely as a basis for further discussion, and
are not
>     intended to be and do not constitute a legally binding
obligation.
>     No legally binding obligations will be created,
implied, or inferred
>     until an agreement in final form is executed in writing
by all
>     parties involved.
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Robin Qiu]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ X] Angler fish

On Wed, Nov 20, 2019 at 2:45 PM Alexey Romanenko 
wrote:

> [ ] Beaver
> [x] Hedgehog
> [ ] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [x] Firefly
> [ ] Cuttlefish
> [x] Dumbo Octopus
> [ ] Angler fish
>
> On 20 Nov 2019, at 14:38, Manu Zhang  wrote:
>
> [] Beaver
> [] Hedgehog
> [x] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> On Wed, Nov 20, 2019 at 8:38 PM Maximilian Michels  wrote:
>
>> [ ] Beaver
>> [ ] Hedgehog
>> [x] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [x] Firefly
>> [x] Cuttlefish
>> [ ] Dumbo Octopus
>> [x] Angler fish
>>
>> On 20.11.19 08:18, Alex Van Boxel wrote:
>> > [ ] Beaver
>> > [ ] Hedgehog
>> > [ ] Lemur
>> > [ ] Owl
>> > [ ] Salmon
>> > [ ] Trout
>> > [ ] Robot dinosaur
>> > [ X] Firefly
>> > [ ] Cuttlefish
>> > [ ] Dumbo Octopus
>> > [ X] Angler fish
>> >
>> >   _/
>> > _/ Alex Van Boxel
>> >
>> >
>> > On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni > > > wrote:
>> >
>> > [ ] Beaver
>> > [ ] Hedgehog
>> > [ ] Lemur
>> > [ ] Owl
>> > [X] Salmon
>> > [ ] Trout
>> > [ ] Robot dinosaur
>> > [ ] Firefly
>> > [ ] Cuttlefish
>> > [X] Dumbo Octopus
>> > [X] Angler fish
>> >
>> > On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles > > > wrote:
>> >
>> > Please cast your votes of approval [1] for animals you would
>> > support as Beam mascot. The animal with the most approval will
>> > be identified as the favorite.
>> >
>> > *** Vote for as many as you like, using this checklist as a
>> > template 
>> >
>> > [ ] Beaver
>> > [ ] Hedgehog
>> > [ ] Lemur
>> > [ ] Owl
>> > [ ] Salmon
>> > [ ] Trout
>> > [ ] Robot dinosaur
>> > [ ] Firefly
>> > [ ] Cuttlefish
>> > [ ] Dumbo Octopus
>> > [ ] Angler fish
>> >
>> > This vote will remain open for at least 72 hours.
>> >
>> > Kenn
>> >
>> > [1] See
>> > https://en.wikipedia.org/wiki/Approval_voting#Description and
>> > https://www.electionscience.org/library/approval-voting/
>> >
>> >
>> >
>> > --
>> >
>> > This email may be confidential and privileged. If you received this
>> > communication by mistake, please don't forward it to anyone else,
>> > please erase all copies and attachments, and please let me know that
>> > it has gone to the wrong person.
>> >
>> > The above terms reflect a potential business arrangement, are
>> > provided solely as a basis for further discussion, and are not
>> > intended to be and do not constitute a legally binding obligation.
>> > No legally binding obligations will be created, implied, or inferred
>> > until an agreement in final form is executed in writing by all
>> > parties involved.
>> >
>>
>
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Alexey Romanenko]

[ ] Beaver
[x] Hedgehog
[ ] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[x] Firefly
[ ] Cuttlefish
[x] Dumbo Octopus
[ ] Angler fish

> On 20 Nov 2019, at 14:38, Manu Zhang  wrote:
> 
> [] Beaver
> [] Hedgehog
> [x] Lemur
> [x] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
> 
> On Wed, Nov 20, 2019 at 8:38 PM Maximilian Michels  > wrote:
> [ ] Beaver
> [ ] Hedgehog
> [x] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [x] Firefly
> [x] Cuttlefish
> [ ] Dumbo Octopus
> [x] Angler fish
> 
> On 20.11.19 08:18, Alex Van Boxel wrote:
> > [ ] Beaver
> > [ ] Hedgehog
> > [ ] Lemur
> > [ ] Owl
> > [ ] Salmon
> > [ ] Trout
> > [ ] Robot dinosaur
> > [ X] Firefly
> > [ ] Cuttlefish
> > [ ] Dumbo Octopus
> > [ X] Angler fish
> > 
> >   _/
> > _/ Alex Van Boxel
> > 
> > 
> > On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni  >  
> > >> wrote:
> > 
> > [ ] Beaver
> > [ ] Hedgehog
> > [ ] Lemur
> > [ ] Owl
> > [X] Salmon
> > [ ] Trout
> > [ ] Robot dinosaur
> > [ ] Firefly
> > [ ] Cuttlefish
> > [X] Dumbo Octopus
> > [X] Angler fish
> > 
> > On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles  > 
> > >> wrote:
> > 
> > Please cast your votes of approval [1] for animals you would
> > support as Beam mascot. The animal with the most approval will
> > be identified as the favorite.
> > 
> > *** Vote for as many as you like, using this checklist as a
> > template 
> > 
> > [ ] Beaver
> > [ ] Hedgehog
> > [ ] Lemur
> > [ ] Owl
> > [ ] Salmon
> > [ ] Trout
> > [ ] Robot dinosaur
> > [ ] Firefly
> > [ ] Cuttlefish
> > [ ] Dumbo Octopus
> > [ ] Angler fish
> > 
> > This vote will remain open for at least 72 hours.
> > 
> > Kenn
> > 
> > [1] See
> > https://en.wikipedia.org/wiki/Approval_voting#Description 
> >  and
> > https://www.electionscience.org/library/approval-voting/ 
> > 
> > 
> > 
> > 
> > -- 
> > 
> > This email may be confidential and privileged. If you received this
> > communication by mistake, please don't forward it to anyone else,
> > please erase all copies and attachments, and please let me know that
> > it has gone to the wrong person.
> > 
> > The above terms reflect a potential business arrangement, are
> > provided solely as a basis for further discussion, and are not
> > intended to be and do not constitute a legally binding obligation.
> > No legally binding obligations will be created, implied, or inferred
> > until an agreement in final form is executed in writing by all
> > parties involved.
> > 

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Manu Zhang]

[] Beaver
[] Hedgehog
[x] Lemur
[x] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

On Wed, Nov 20, 2019 at 8:38 PM Maximilian Michels  wrote:

> [ ] Beaver
> [ ] Hedgehog
> [x] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [x] Firefly
> [x] Cuttlefish
> [ ] Dumbo Octopus
> [x] Angler fish
>
> On 20.11.19 08:18, Alex Van Boxel wrote:
> > [ ] Beaver
> > [ ] Hedgehog
> > [ ] Lemur
> > [ ] Owl
> > [ ] Salmon
> > [ ] Trout
> > [ ] Robot dinosaur
> > [ X] Firefly
> > [ ] Cuttlefish
> > [ ] Dumbo Octopus
> > [ X] Angler fish
> >
> >   _/
> > _/ Alex Van Boxel
> >
> >
> > On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni  > > wrote:
> >
> > [ ] Beaver
> > [ ] Hedgehog
> > [ ] Lemur
> > [ ] Owl
> > [X] Salmon
> > [ ] Trout
> > [ ] Robot dinosaur
> > [ ] Firefly
> > [ ] Cuttlefish
> > [X] Dumbo Octopus
> > [X] Angler fish
> >
> > On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles  > > wrote:
> >
> > Please cast your votes of approval [1] for animals you would
> > support as Beam mascot. The animal with the most approval will
> > be identified as the favorite.
> >
> > *** Vote for as many as you like, using this checklist as a
> > template 
> >
> > [ ] Beaver
> > [ ] Hedgehog
> > [ ] Lemur
> > [ ] Owl
> > [ ] Salmon
> > [ ] Trout
> > [ ] Robot dinosaur
> > [ ] Firefly
> > [ ] Cuttlefish
> > [ ] Dumbo Octopus
> > [ ] Angler fish
> >
> > This vote will remain open for at least 72 hours.
> >
> > Kenn
> >
> > [1] See
> > https://en.wikipedia.org/wiki/Approval_voting#Description and
> > https://www.electionscience.org/library/approval-voting/
> >
> >
> >
> > --
> >
> > This email may be confidential and privileged. If you received this
> > communication by mistake, please don't forward it to anyone else,
> > please erase all copies and attachments, and please let me know that
> > it has gone to the wrong person.
> >
> > The above terms reflect a potential business arrangement, are
> > provided solely as a basis for further discussion, and are not
> > intended to be and do not constitute a legally binding obligation.
> > No legally binding obligations will be created, implied, or inferred
> > until an agreement in final form is executed in writing by all
> > parties involved.
> >
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Maximilian Michels]

[ ] Beaver
[ ] Hedgehog
[x] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[x] Firefly
[x] Cuttlefish
[ ] Dumbo Octopus
[x] Angler fish

On 20.11.19 08:18, Alex Van Boxel wrote:

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ X] Angler fish

  _/
_/ Alex Van Boxel


On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni > wrote:


[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[X] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[X] Dumbo Octopus
[X] Angler fish

On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles mailto:k...@apache.org>> wrote:

Please cast your votes of approval [1] for animals you would
support as Beam mascot. The animal with the most approval will
be identified as the favorite.

*** Vote for as many as you like, using this checklist as a
template 

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

This vote will remain open for at least 72 hours.

Kenn

[1] See
https://en.wikipedia.org/wiki/Approval_voting#Description and
https://www.electionscience.org/library/approval-voting/



-- 


This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else,
please erase all copies and attachments, and please let me know that
it has gone to the wrong person.

The above terms reflect a potential business arrangement, are
provided solely as a basis for further discussion, and are not
intended to be and do not constitute a legally binding obligation.
No legally binding obligations will be created, implied, or inferred
until an agreement in final form is executed in writing by all
parties involved.

Re: GCP libraries up-to-date versions in Java

[Elliotte Rusty Harold]

BOM or no BOM is an implementation detail. Using
com.google.clou:libraries-bom would make dependency management simpler
for developers, but the real issue is whether Beam can continue to
work with very old versions of the many libraries it depends on. Even
if this is acceptable for Beam, it's unlikely to be feasible for
anyone who needs to mix Beam code with other code.

There should be no self-incompatibility between Google minor version
releases. All the Google libraries in question follow semantic
versioning. E.g. Pubsub 1.43 would be fully API compatible with Pubsub
1.28, though not the reverse. However there are likely to be important
bug fixes in 1.43 and definitely new features that 1.28 would not
have. If there are any edge cases where this is not true, that's a bug
and if you file it against the repo we'll try to fix it. We're also
installing tooling to make this less likely to happen by accident.
However, right now any such problem is rare.

Behavior differences are another story. It is entirely possible that
something like Pubsub 1.28 would simply no longer function due to
changes at the backend. There's a deprecation cycle, announcements,
and transition periods in all such cases; but a project like Beam
can't stay on old versions forever. If they try, the backends will
shift out from under them.

Looking at Beam's dependencies, the only case where there are major
version changes to address is Guava.
This will take some work, but not an excessive amount. We should be
able to move this up to 28.1-android with few code changes and no
further API breaking changes in that library are planned for the
future.

The remaining issues are pre-1.0 libraries. OpenCensus is a particular
thorn in my side. Ideally these should not be used, at all. However if
we must, we should not expose them on the Beam API surface and we need
to move them forward quickly as they change.

-- 
Elliotte Rusty Harold
elh...@ibiblio.org

Re: [Discuss] Beam mascot

[Maximilian Michels]

Thanks for following up with this Kenn.

-Max

On 16.11.19 00:05, Alex Van Boxel wrote:
I like the "Angler fish" as well... it certainly faster then a 
cuttlefish


  _/
_/ Alex Van Boxel


On Fri, Nov 15, 2019 at 9:57 PM Kirill Kozlov > wrote:


Angler fish? Found a few animated examples that may look interesting
[1, 2, 3, 4].
[1] https://www.pinterest.com/pin/121175046208927340/
[2] https://www.pinterest.com/pin/353180795779533157/
[3] https://www.pinterest.com/pin/121175046208927334/
[4]

https://graphicriver.net/item/cartoon-angler-fish/16828573?ref=gvector_id=1413785108_back=true

On Fri, Nov 15, 2019 at 11:39 AM Aizhamal Nurmamat kyzy
mailto:aizha...@apache.org>> wrote:

That last sketch of a cuttlefish with a hat is really really
good. I vote for that.

On Fri, Nov 15, 2019 at 10:33 AM Kenneth Knowles
mailto:k...@apache.org>> wrote:

PSA: cuttlefish tentacles look more like their face, not
their legs. Please find attached illustrations to that
effect. And also evidence that they are occasionally fancy.
Definitely *not* from a respectable designer or anyone who
could execution a professional logo.

I do think the PMC needs to shepherd this. I would suggest
starting with an approval vote [1] for animal (no plant
ideas?) and then an ASF-style vote to record the validated
result. From there, we can go through more standard design
process.

Kenn

[1] https://www.electionscience.org/library/approval-voting/

On Fri, Nov 15, 2019 at 6:54 AM Maximilian Michels
mailto:m...@apache.org>> wrote:

It's great we're having this discussion and we came up
with a lot of
great ideas through it. However, it is unclear how we
proceed from here.
Certainly, we can't let designers work with an
open-ended discussion on
the type of mascot.

Personally, I'm fine with _any_ kind of mascot, as long
as it is made by
a decent designer. The respectable designer I'm talking
to was very
generous to donate these sketches for free. I don't
think that this is
to be expected at all. Coming back to the original idea
of hiring
multiple designers, I don't see how we will pay those
designers to all
come up with a logo, unless we all donate money.

The sketches I've sent might not look like much because
there is still a
decent process involved in coming up with the final logo
which looks
good in different sizes, possibly with many iterations.
So far I've
tried to incorporate as many of the suggestions here,
but for the sake
of protecting the designer, I think I'll have to stop
doing that. It
simply won't work.

How to proceed from here? I think _any_ professionally
executed logo
will do, this is really more about the community
agreeing for the
greater good. Ultimately, I think the PMC might have to
decide on how to
proceed here.

Cheers,
Max

On 15.11.19 13:59, Hannah Jiang wrote:
 > I also vote for firefly.
 >
 > On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev
mailto:valen...@google.com>
 > >> wrote:
 >
 >     I like the firefly sketch a lot, it's my favorite
so far.
 >
 >     On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw
 >     mailto:rober...@google.com>
>> wrote:
 >
 >         #37 from the sketches was the cuttlefish,
which would put it at
 >         (with
 >         4 votes) the most popular so far. I do like
the firefly too.
 >
 >         On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas
mailto:g...@apache.org>
 >         >> wrote:
 >          >
 >          > Hi everyone, so exciting to see this convo
taking off!
 >          >