[NEED HELP] PMC only finalization items for release 2.30.0

Hi, I'm finishing 2.30.0 release and need help doing PMC only finalization items in the release guide ( https://beam.apache.org/contribute/release-guide/#10-finalize-the-release). Please let me know if any PMC members have some time to do these tasks :) Thanks!

Re: [VOTE] Release 2.30.0, release candidate #1

n Tue, Jun 8, 2021 at 12:45 PM Kenneth Knowles wrote: > +1 (binding) > > Verified wordcount with various configuration parameters and they all > worked. Particularly confirming that all the containers are chosen > correctly. > > Kenn > > On Tue, Jun 8, 2021 at 12:40 PM H

Re: [VOTE] Release 2.30.0, release candidate #1

right now. > Sure. Please let me know when you finish the validation. > > Kenn > > On Mon, Jun 7, 2021 at 4:28 PM Heejong Lee wrote: > >> FYI, we now have three binding votes and I will close the vote tomorrow >> morning. >> >> The RC build is validated f

Re: [VOTE] Release 2.30.0, release candidate #1

gt; >> > >> Thanks, > >> Cham > >> > >> On Thu, Jun 3, 2021 at 2:03 PM Tomo Suzuki wrote: > >>> > >>> +1 (non-binding) > >>> > >>> Thank you for the preparation. With the GCP dependencies of my > interest, th

[VOTE] Release 2.30.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.30.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

[NEED HELP] Populating the change list for 2.30.0 release

Hi Beam developers, I'm gathering the information for the changes in the 2.30.0 release. If you have any idea about important *new features* / *breaking changes* / *deprecation* / *known issues* for the 2.30.0 release, please note down them in CHANGES.md or just let me know. Thanks!

Re: Need maintainer permission of PyPI apache-beam package

It's done. Thanks Pablo! On Wed, May 19, 2021 at 11:21 AM Pablo Estrada wrote: > I've sent you an invite to be a project maintainer. Let me know if that > works. > Best > -P. > > On Tue, May 18, 2021 at 6:28 PM Heejong Lee wrote: > >> Hi, >> >>

Need maintainer permission of PyPI apache-beam package

Hi, I'm currently working on Beam 2.30.0 release and need help adding myself to the maintainer group of PyPI apache-beam package. My PyPI username is 'ihji'. Does anybody have permission for adding a new member to the apache-beam maintainer group? Thanks!

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.30.0 I will start building the RC release after we cherry-pick the last blocker. On Thu, Apr 29, 2021 at 12:48 AM Heejong Lee wrote: > We have 10 open issues for Fix Version 2.30.0: > https://issues.apache.org/jira/

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

AM Heejong Lee wrote: > FYI, I just cut the 2.30.0 release branch. From now on, late commits for > 2.30.0 need to be cherry-picked. If you have any late commits, please make > sure that their Jira issues have the correct Fix Version, 2.30.0. > > On Tue, Apr 27, 2021 at 7:52 AM

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

t; On Mon, Apr 26, 2021 at 2:33 PM Heejong Lee wrote: > >> >> >> On Mon, Apr 26, 2021 at 10:24 AM Robert Bradshaw >> wrote: >> >>> Confirming that the cut date is 4/28/2021 (in two days), right? >>> >> >> Yes, 2.30.0 branch is schedu

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

because each release represents a certain quantity of changes. But in this > case, the actual quantity of changes is affected by the re-cut, too. > > > > > > On Wed, Apr 21, 2021 at 4:12 PM Heejong Lee > wrote: > > >> > > >> Update on the 2.30.0

Re: [ANNOUNCE] New committer: Yichi Zhang

Congratulations :) On Wed, Apr 21, 2021 at 5:20 PM Tomo Suzuki wrote: > Congratulations! > > On Wed, Apr 21, 2021 at 7:48 PM Tyson Hamilton wrote: > >> Congrats! >> >> On Wed, Apr 21, 2021 at 4:37 PM Valentyn Tymofieiev >> wrote: >> >>> Well deserved and congrats, Yichi! >>> >>> On Wed, Apr 21

Re: [PROPOSAL] Preparing for Beam 2.30.0 release

> On Tue, Apr 20, 2021 at 4:55 PM Heejong Lee wrote: > >> Hi All, >> >> Beam 2.30.0 release is scheduled to be cut on April 21 according to the >> release calendar [1] >> >> I'd like to volunteer myself to be the release manager for this release

[PROPOSAL] Preparing for Beam 2.30.0 release

Hi All, Beam 2.30.0 release is scheduled to be cut on April 21 according to the release calendar [1] I'd like to volunteer myself to be the release manager for this release. I plan on cutting the release branch on the scheduled date. Any comments or objections ? Thanks, Heejong [1] https://cal

Re: [ANNOUNCE] New committer: Reza Ardeshir Rokni

Congratulations! On Thu, Sep 10, 2020 at 4:42 PM Robert Bradshaw wrote: > Thank you and welcome, Reza! > > On Thu, Sep 10, 2020 at 4:00 PM Ahmet Altay wrote: > >> Congratulations Reza! And thank you for your contributions! >> >> On Thu, Sep 10, 2020 at 3:59 PM Chamikara Jayalath >> wrote: >> >

Re: Re-running GitHub Actions jobs

On Thu, Sep 3, 2020 at 11:05 AM Brian Hulette wrote: > The new GitHub Actions workflows that run Java and Python tests against > different targets (macos, ubuntu, windows) are great! But just like our > Jenkins infra they flake occasionally. Should we be re-running all of these > jobs until we ge

Re: [VOTE] Release 2.23.0, release candidate #1

wse/BEAM-10397 before giving my >> vote. >> > > +Heejong Lee to comment on this. > > >> >> On Wed, Jul 15, 2020 at 10:51 AM Pablo Estrada >> wrote: >> > >> > +1 >> > I was able to run the python 3.8 quickstart from wheels on Dir

Re: XLang sub-graph representation within the SDKs pipeline types

On Wed, Jul 1, 2020 at 7:18 PM Robert Burke wrote: > From the Go SDK side, it was built that way nearly from the start. > Historically there was a direct SDK rep -> Dataflow rep conversion, but > that's been replaced with a SDK rep -> Beam Proto -> Dataflow rep > conversion. > > In particular, th

Re: Beam Jenkins Migration

This is awesome. Could non-committers also trigger the test now? On Wed, Jun 17, 2020 at 6:12 AM Damian Gadomski wrote: > Hello, > > Good news, we've just migrated to the new CI: https://ci-beam.apache.org. > As from now beam projects at builds.apache.org are disabled. > > If you experience any

Re: Python SDK ReadFromKafka: Timeout expired while fetching topic metadata

gt; wrote: > >> Seems like Java dependency is not being properly set up when running the >> cross-language Kafka step. I don't think this was available for Beam 2.21. >> Can you try with the latest Beam HEAD or Beam 2.22 when it's released ? >> +Heejong

Re: Jira components for cross-language transforms

If we use one meta component tag for all xlang related issues, I would prefer just "xlang". Then we could attach the "xlang" tag to not only language specific sdk tags but also other runner tags e.g. ['xlang', 'io-java-kafka'], ['xlang'', 'runner-dataflow']. On Thu, May 28, 2020 at 7:49 PM Robert

Re: No space left on device - beam-jenkins 1 and 7

Still seeing no space left on device errors on jenkins-7 (for example: https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2754/) On Fri, Mar 6, 2020 at 7:11 PM Alan Myrvold wrote: > Did a one time cleanup of tmp files owned by jenkins older than 3 days. > Agree that we need a longer

Re: Error logging from fn_api_runners

I think it should be either info or debug but not error. On Mon, Mar 2, 2020 at 2:35 PM Ning Kang wrote: > Hi, > > I just observed some error level loggings like these: > ``` > ERROR:apache_beam.runners.portability.fn_api_runner:created 1 workers > {'worker_5': > at 0x127fdaa58>} > ERROR:apache

Re: Cross-language pipelines status

On Tue, Feb 11, 2020 at 9:37 AM Alexey Romanenko wrote: > Hi all, > > I just wanted to ask for more details about the status of cross-language > pipelines (rather, transforms). I see some discussions about that here, but > I think it’s more around cross-language IOs. > > I’ll appreciate for any i

Re: Enabling a new Jenkins job

Fixed. Seed job was overridden by another scheduled seed job. Thanks, Udi! On Wed, Feb 5, 2020 at 2:04 PM Heejong Lee wrote: > I created a new Jenkins job in my PR[1] and the new job shows "This > project is currently disabled"[2]. Does anybody know how to enable the > new

Enabling a new Jenkins job

I created a new Jenkins job in my PR[1] and the new job shows "This project is currently disabled"[2]. Does anybody know how to enable the new job? [1]: https://github.com/apache/beam/pull/10758 [2]: https://builds.apache.org/job/beam_PostCommit_XVR_Spark/

Re: [ANNOUNCE] New committer: Hannah Jiang

Congratulations! :) On Tue, Jan 28, 2020 at 4:43 PM Yichi Zhang wrote: > Congrats Hannah! > > On Tue, Jan 28, 2020 at 3:57 PM Yifan Zou wrote: > >> Congratulations Hannah!! >> >> On Tue, Jan 28, 2020 at 3:55 PM Boyuan Zhang wrote: >> >>> Thanks for all your contributions! Congratulations~ >>>

Re: [DISCUSS][PROPOSAL] Improvements to the Apache Beam website

On Mon, Jan 27, 2020 at 11:19 AM Aizhamal Nurmamat kyzy wrote: > Hi Alexey, > > Answers are inline: > > Do we have any user demands for documentation translation into other >> languages? I’m asking this because, in my experience, it’s quite tough work >> to translate everything and it won’t be al

Re: Jenkins jobs not running for my PR 10438

+1: https://github.com/apache/beam/pull/10051 force-pushing again. retest this please. nothing works :( On Fri, Jan 3, 2020 at 12:55 AM Michał Walenia wrote: > Hi, > I'm also affected by this - I touched my PRs opened before the holiday > break and no jobs were triggered. Do we know what breaks

Re: External transform API in Java SDK

icket: https://issues.apache.org/jira/browse/BEAM-9048 On Mon, Dec 30, 2019 at 10:27 AM Luke Cwik wrote: > > > On Mon, Dec 23, 2019 at 12:20 PM Heejong Lee wrote: > >> >> >> On Fri, Dec 20, 2019 at 11:38 AM Luke Cwik wrote: >> >>> What do side inpu

Re: External transform API in Java SDK

DoFn.ProcessContext c) { out.output(x + c.sideInput(sideView)); } }) .withSideInputs(sideView)); > On Thu, Dec 19, 2019 at 4:39 PM Heejong Lee wrote: > >> I wanted to know if anybody has an

External transform API in Java SDK

I wanted to know if anybody has any comment on external transform API for Java SDK. `External.of()` can create external transform for Java SDK. Depending on input and output types, two additional methods are provided: `withMultiOutputs()` which specifies the type of PCollection and `withOutputType

Re: Artifact staging in cross-language pipelines

Hi, I wrote the draft on implementation plan[1]. The summary is on the first page. Any help would be appreciated! [1]: https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog/edit?usp=sharing On Thu, Dec 12, 2019 at 5:02 PM Heejong Lee wrote: > I'm brushing u

Re: Artifact staging in cross-language pipelines

v 26, 2019 at 3:54 AM Maximilian Michels wrote: > Hey Heejong, > > I don't think so. It would be great to push this forward. > > Thanks, > Max > > On 26.11.19 02:49, Heejong Lee wrote: > > Hi, > > > > Is anyone actively working on artifact staging e

Re: RFC: python static typing PR

Wow, this is a big step forward. As a long fan of strongly typed functional languages, I'm glad to see this change :) On Wed, Dec 11, 2019 at 9:44 AM Chad Dombrova wrote: > Hi all, > Robert has diligently reviewed the first batch of changes for this PR, and > all review notes are addressed and t

Re: Artifact staging in cross-language pipelines

Hi, Is anyone actively working on artifact staging extension for cross-language pipelines? I'm thinking I can contribute to it in coming Dec. If anyone has any progress on this and needs help, please let me know. Thanks, On Wed, Jun 12, 2019 at 2:42 AM Ismaël Mejía wrote: > Can you please add

Revamping the cross-language validate runner test suite

Hi, I'm working on revamping the cross-language validate runner test suite. Our current test suite for the cross-language transform is incomplete as it only has tests for Wordcount, DoFn, basic Count and basic Filter transforms. My plan is, in addition to our existing set of tests, to add all prim

Re: published containers overwrite locally built containers

; >> >> On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath >> wrote: >> >>> I think it makes sense to override published docker images with locally >>> built versions when testing HEAD. >>> >>> Thanks, >>> Cham >>> >>

***UNCHECKED*** Re: published containers overwrite locally built containers

) but it shouldn't be a problem in prod that the unique tag is assumed for each released version. On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath wrote: > I think it makes sense to override published docker images with locally > built versions when testing HEAD. > > Thanks

published containers overwrite locally built containers

Hi, happy halloween! I'm looking into failing cross language post commit tests: https://issues.apache.org/jira/browse/BEAM-8534 After a few runs, I've found that published SDK harness containers overwrite locally built containers when do

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

Congratulations! :) On Mon, Aug 26, 2019 at 2:44 PM Rui Wang wrote: > Congratulations! > > > -Rui > > On Mon, Aug 26, 2019 at 2:36 PM Hannah Jiang > wrote: > >> Congratulations Valentyn, well deserved! >> >> On Mon, Aug 26, 2019 at 2:34 PM Chamikara Jayalath >> wrote: >> >>> Congrats Valentyn!

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratulations! On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli wrote: > Congratulations! > > On Wed, Aug 7, 2019 at 11:27 PM Robin Qiu wrote: > >> Congratulations, Kyle! >> >> On Wed, Aug 7, 2019 at 5:04 AM Valentyn Tymofieiev >> wrote: >> >>> Congrats, Kyle! >>> >>> On Wed, Aug 7, 2019 a

Re: How to expose/use the External transform on Java SDK

I think it depends how we define "the core" part of the SDK. If we define the core as only the (abstract) data types which describe BEAM pipeline model then it would be more sensible to put external transform into a separate extension module (option 4). Otherwise, option 1 makes sense. On Wed, Jul

Re: [ANNOUNCE] New committer: Robert Burke

Congratulations! On Tue, Jul 16, 2019 at 1:34 PM Chamikara Jayalath wrote: > Congrats!! > > On Tue, Jul 16, 2019 at 1:31 PM Robin Qiu wrote: > >> Congrats, Robert!! >> >> On Tue, Jul 16, 2019 at 1:22 PM Alan Myrvold wrote: >> >>> Congrats, Robert! >>> >>> On Tue, Jul 16, 2019 at 11:46 AM Ismaë

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congratulations! On Wed, May 15, 2019 at 12:24 PM Niklas Hansson < niklas.sven.hans...@gmail.com> wrote: > Congratulations Pablo :) > > Den ons 15 maj 2019 kl 21:21 skrev Ruoyun Huang : > >> Congratulations, Pablo! >> >> *From: *Charles Chen >> *Date: *Wed, May 15, 2019 at 11:04 AM >> *To: *dev

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congratulations! On Fri, May 3, 2019 at 3:53 PM Reza Rokni wrote: > Congratulations ! > > *From: *Reuven Lax > *Date: *Sat, 4 May 2019, 06:42 > *To: *dev > > Thank you! >> >> On Fri, May 3, 2019 at 3:15 PM Ankur Goenka wrote: >> >>> Congratulations Udi! >>> >>> On Fri, May 3, 2019 at 3:00 PM C

Re: Artifact staging in cross-language pipelines

2019년 4월 23일 (화) 오전 2:07, Robert Bradshaw 님이 작성: > I've been out, so coming a bit late to the discussion, but here's my > thoughts. > > The expansion service absolutely needs to be able to provide the > dependencies for the transform(s) it expands. It seems the default, > foolproof way of doing th

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

/beam/sdk/coders/StringUtf8Coder.java#L50 > >>>>>>> [3] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/coders/coders.py#L321 > >>>>>>> > >>>>>>> > >>>>>&

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

; >> On Wed, Apr 3, 2019 at 6:21 PM Robert Burke >> wrote: >> >>> >> >>> String UTF8 was recently added as a "standard coder " URN in the >> protos, but I don't think that developed beyond Java, so adding it to >> Python would be reason

[DISCUSS] change the encoding scheme of Python StrUtf8Coder

Hi all, It looks like UTF-8 String Coder in Java and Python SDKs uses different encoding schemes. StringUtf8Coder in Java SDK puts the varint length of the input string before actual data bytes however StrUtf8Coder in Python SDK directly encodes the input string to bytes value. For the last few we

Re: How to use "PortableRunner" in Python SDK?

You can also try without --streaming option. There's a separate streaming wordcount example in the same directory. If you want to look into the output files, it would be easier to use external target like gs:// instead of local file. python -m apache_beam.examples.wordcount --input=/etc/profile -

[PROPOSAL] decrease the number of threads for BigQuery streaming insertAll

Hi, I want to suggest the change[1] of the thread pool type in BigQuery streaming insert for Java SDK (BEAM-6443). When we insert small data into BigQuery very fast by using BigQueryIO.write, it generates lots of rate limit exceeded errors in a log file. It's mainly because the number of threads t

Re: Add code quality checks to pre-commits.

gt;> notice major missing tests. >>> >>> We have never really used Sonarqube. It was turned on as a possibility >>> in the early days but never worked on past that point. Could be nice. I >>> suspect there's a lot to be gained by just finding very lo

Re: Add code quality checks to pre-commits.

I don't have any experience of using SonarQube but Coverity worked well for me. Looks like it already has beam repo: https://scan.coverity.com/projects/11881 On Thu, Jan 3, 2019 at 1:27 PM Reuven Lax wrote: > checkstyle and findbugs are already run as precommit checks, are they not? > > On Thu,

Re: [PROPOSAL] ParquetIO support for Python SDK

to base this on > byte sizes; will this be in v1 or will there be other parameter(s) > that we'll have to support going forward? > On Tue, Oct 30, 2018 at 10:42 PM Heejong Lee wrote: > > > > Thanks all for the valuable feedback on the document. Here's the summary

Re: [PROPOSAL] ParquetIO support for Python SDK

ct 24, 2018 at 4:45 PM Ahmet Altay wrote: > >> Thank you Heejong. Could you also share a summary of the design document >> (major points/decisions) in the mailing list? >> >> On Wed, Oct 24, 2018 at 4:08 PM, Heejong Lee wrote: >> >>> Hi, >>>

[PROPOSAL] ParquetIO support for Python SDK

Hi, I'm working on BEAM-: Parquet IO for Python SDK. Issue: https://issues.apache.org/jira/browse/BEAM- Design doc: https://docs.google.com/document/d/1-FT6zmjYhYFWXL8aDM5mNeiUnZdKnnB021zTo4S-0Wg WIP PR: https://github.com/apache/beam/pull/6763 Any feedback is appreciated. Thanks!

a new contributor

Hi, I just wanted to introduce myself as a new contributor. I'm a new member of Apache Beam team at Google and will be working on IO modules. Happy to meet you all! Thanks, Heejong