Re: New contributor to Beam documentation

Welcome! On Fri, Jan 31, 2020 at 12:03 PM Kyle Weaver wrote: > Hi Dave, > > Welcome to Beam! Looking forward to working with you :) > > Kyle > > On Fri, Jan 31, 2020 at 11:17 AM Dave Wrede wrote: > >> Hi all, >> >> I'm Dave and I recently started working on Dataflow documentation. In >> additio

[DISCUSSION] Improve release notes by adding a change list file

Hi all, We currently have two major ways to communicate changes in a release: - A blog post, to highlight major changes in the release. (Example for 2.17: [1]) - JIRA release notes pages listing all issues tagged for a specific release. (Example for 2.17 [2]). There are a few issues with this pro

Re: Release 2.19.0, release candidate #1

+1 -- Verified web page / blog post changes. Thank you for building this RC and pre-validating a large number of tests. On Wed, Jan 29, 2020 at 9:03 PM Boyuan Zhang wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version > 2.19.0, as follows: > [ ] +1, Approve

Re: [ANNOUNCE] Beam 2.18.0 Released

Thank you Udi! On Tue, Jan 28, 2020 at 2:13 PM kant kodali wrote: > Looks like > https://beam.apache.org/documentation/runners/capability-matrix/ needs to > be updated? since there seems to be support for spark structured streaming? > > On Tue, Jan 28, 2020 at 1:47 PM Connell O'Callaghan > wrot

Re: New contributor

Done. Added xbhuang to the list of contributors. On Mon, Jan 27, 2020 at 10:45 AM Xinbin Huang wrote: > Hi > > This is Xinbin Huang. Can someone add me as a contributor for Beam's Jira > issue tracker? I would like to create/assign tickets for my work. > > My Jira username is xbhuang . > > Thank

Re: New contributor

Welcome. What is your jira user name? On Fri, Jan 24, 2020 at 1:25 PM Xinbin Huang wrote: > Hi > > This is Xinbin Huang. Can someone add me as a contributor for Beam's Jira > issue tracker? I would like to create/assign tickets for my work. > > Thanks > Bin >

Re: Jenkins Master

/cc +Alan Myrvold +Yifan Zou +Dan Gazineu -- cc'ing folks who worked on Jenkins related tasks before. On Thu, Jan 23, 2020 at 10:10 AM Pablo Estrada wrote: > Removing private@, and adding dev@. > > ASF Infra has provisioned a new Jenkins master that we'll be able to use > with more independen

Re: [DISCUSS] Autoformat python code with Black

#x27;s style. I don't think it is feasible to >> reformat the >> >>>>>> > > entire Python SDK. >> >>>>>> > > >> >>>>>> > > Reformatted lines don't allow quick access to the Git history. >>

Re: [VOTE] Release 2.18.0, release candidate #1

Tue, Jan 21, 2020 at 11:16 AM Robert Bradshaw > wrote: > >> The source tarball seems to be missing the commit at >> >> https://github.com/apache/beam/commit/a61dfbf4570e3adb30e15315c116751faeda897e >> >> On Tue, Jan 21, 2020 at 9:49 AM Ahmet Altay wrote: &g

Re: [VOTE] Release 2.18.0, release candidate #1

All, could you help with validations and voting? On Wed, Jan 15, 2020 at 6:14 PM Ahmet Altay wrote: > +1, validated the same things, they still work. Thank you. > > On Wed, Jan 15, 2020 at 5:01 PM Udi Meiri wrote: > >> Dataflow containers have been updated. Test away. &g

Re: [VOTE] Release 2.18.0, release candidate #1

AGqyNUM-oLFo_ZXBpJw/edit#gid=1178617819 >> [10] https://hub.docker.com/u/apachebeam >> >> >> On Tue, Jan 14, 2020 at 6:34 PM Udi Meiri wrote: >> >>> Please don't do any Dataflow-based verifications yet, because we'll have >>> to redo them o

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

s >> returned. Users need to use tags to identify which images they should use. >> Since we release images with new tags for each version, it will overwhelm >> the users and give them an impression that the images are not organized >> well. It

Re: [DISCUSS] Integrate Google Cloud AI functionalities

On Wed, Jan 15, 2020 at 8:12 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Based on your feedback, I think it'd be fine to deal with the problem as > follows: > * for Python: put the transforms into `sdks/python/apache_beam/io/gcp/ai` > * for Java: create a `google-cloud-platform-ai

Re: [DISCUSS] Python static type checkers

On Mon, Jan 13, 2020 at 6:43 PM Chad Dombrova wrote: > > I agree with focusing one mypy for now, but I would propose soon after, >> or in parallel if it will be different folks, to work on pytype and enable >> it as a first class citizen similar to mypy. If there will be a large delta >> between

Re: Jenkins jobs not running for my PR 10438

Done. +Kenneth Knowles , any updates from INFRA on this? On Tue, Jan 14, 2020 at 6:43 PM Tomo Suzuki wrote: > It hit Dataflow quota error again. Can somebody run > Run Dataflow ValidatesRunner > for https://github.com/apache/beam/pull/10554 ? > > On Tue, Jan 14, 2020 at 12:14 PM Tomo Suzuki wr

Re: [VOTE] Release 2.18.0, release candidate #1

I verified python 2 quickstarts with batch and streaming pipelines, wheel files, and reviewed changes to the blog/website. Udi, could you send an updated version of the voting text with TODOs, template pieces removed? We can discuss changes to the template separately. My vote is +1 pending an upda

Re: [DISCUSS] Python static type checkers

> The most important gain would be compatibility with Google internal code. I would like to clarify this. This refers to users of Beam who by default are using pytype as part of the toolchain. Even though they are internal to a one single company and not vocal on Beam, they still represent a large

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

On Fri, Jan 10, 2020 at 3:33 PM Ahmet Altay wrote: > > > On Fri, Jan 10, 2020 at 3:32 PM Ankur Goenka wrote: > >> Also curious to know if apache provide any infra support fro projects >> under Apache umbrella and any quota limits they might have. >> > Maybe Ha

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

anization, it was quick and >> easy to add them. I imagine we'd have had to get approval from someone >> outside the project to do that under the apache org. But this won't need to >> happen very often, so it's probably not that big a deal. >> >> >>

[PROPOSAL] Transition released containers to the official ASF dockerhub organization

Hi all, I saw recent progress on the containers and wanted to bring this question to the attention of the dev list. Would it be possible to use the official ASF dockerhub organization for new Beam container releases? Concretely, starting from 2.19 could we release Beam containers to https://hub.d

Re: [ANNOUNCE] Beam 2.17.0 Released!

Thank you Mikhail! On Fri, Jan 10, 2020 at 12:40 PM Kyle Weaver wrote: > Hooray! Thanks to Mikhail and everyone else who contributed. > > On Fri, Jan 10, 2020 at 10:23 AM Maximilian Michels > wrote: > >> At last :) Thank you for making it happen Mikhail! Also thanks to >> everyone else who test

Re: [PROPOSAL] Preparing for Beam 2.19 release

Thank you Boyuan! On Wed, Jan 8, 2020 at 11:33 PM Kenneth Knowles wrote: > Comment: thank you for keeping on the schedule! > > And comment for owners of bugs: if you have a bug targeting this release > and it is not already in fixed or critical urgency, please remove the Fix > Version. Here is a

Re: [VOTE] Vendored Dependencies Release

+1 On Thu, Jan 9, 2020 at 2:04 PM Pablo Estrada wrote: > +1 > > verified sha1 and md5 hashes. > > On Thu, Jan 9, 2020 at 10:28 AM Luke Cwik wrote: > >> +1 >> >> I validated that no classes appeared outside of the >> org.apache.beam.vendor.grpc.v1p26p0 namespace and I also validated that the >>

Re: [VOTE] Release 2.17.0, release candidate #2

> On Fri, Dec 20, 2019 at 8:09 PM Ahmet Altay wrote: > >> +1, I validated the python2 quick starts using wheels. Thank you for >> pushing the release this far. >> >> On Thu, Dec 19, 2019 at 1:27 PM Kenneth Knowles wrote: >> >>> I verified the Java quicks

Re: [VOTE] Release 2.17.0, release candidate #2

t;> Jincheng >> >> [1] https://github.com/apache/beam/pull/10401 >> [2] https://github.com/apache/beam/pull/10402 >> >> >> Ahmet Altay 于2019年12月19日周四 上午3:31写道： >> >>> I validated python quickstarts with python 2. Wheels file are missing >&g

Re: [PROPOSAL] python precommit timeouts

This sounds reasonable. Would this be configurable per-test if needed? On Thu, Dec 19, 2019 at 5:52 PM Udi Meiri wrote: > Looking at this console log > , >

Re: [VOTE] Release 2.17.0, release candidate #2

I validated python quickstarts with python 2. Wheels file are missing but they work otherwise. Once the wheel files are added I will add my vote. On Wed, Dec 18, 2019 at 10:00 AM Luke Cwik wrote: > I verified the release and ran the quickstarts and found that release 2.16 > broke Apache Nemo run

Re: Root logger configuration

On Fri, Dec 13, 2019 at 1:08 PM Robert Bradshaw wrote: > The default behavior of Python logging is > > (1) Logging handlers may get added (usually in main or very close to it). > (2) Logging goes to those handlers, or a default handler if none were > added. > > With this proposal, we would have >

Re: RFC: python static typing PR

Thank you, Chad! This is awesome. On Wed, Dec 11, 2019 at 2:05 PM Robert Bradshaw wrote: > +1. Thanks! > > On Wed, Dec 11, 2019 at 1:44 PM Pablo Estrada wrote: > > > > Love it. I've merged it, since it was already approved by Robert - and > yes, we don't want to hit a merge conflict. > > > > On

Re: request for access: pypi and dockerhub

Udi, added 'udim' as a maintainer to pypi. Pablo, I updated you to be an owner. You should have privileges now. Ahmet On Tue, Dec 10, 2019 at 12:01 PM Pablo Estrada wrote: > I've added you as a maintainer in docker hub. > I don't have privileges to add you in pyp

Re: [PROPOSAL] Preparing for Beam 2.18 release

Thank you Udi for keeping the release cadence. +1 to cutting 2.18.0 branch on time. On Thu, Nov 21, 2019 at 10:07 AM Udi Meiri wrote: > Thanks Cham. Tomo, if there are any dependencies you believe are blockers > please mark them. > Also, only the sub-tasks >

Re: cython test instability

I tried to debug but did not make much progress. I cannot reproduce locally, however all python precommits and postcommits are failing. One guess is, setuptools released a new version that does not support eggs a few days ago, that might be the cause ( https://github.com/pypa/setuptools/blob/maste

Re: [Discuss] Beam Summit 2020 Dates & locations

eople availability >>> and influent the final number of participants in the end. >>> >>> > >>> > Cheers, >>> > Max >>> > >>> > On 07.11.19 22:50, Alex Van Boxel wrote: >>> >> For date wise, I'm won

Re: Cleaning up Approximate Algorithms in Beam

ial internal structure (and coders). This has been >>> discussed a lot, perhaps there are some ideas there we could borrow? >>> >>> And they will match search terms better, which is a major problem. >>>> >>> >>> I'm not following this--by

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congratulations, Daniel! On Sat, Nov 23, 2019 at 3:47 AM jincheng sun wrote: > > Congrats, Daniel! > Best, > Jincheng > > Alexey Romanenko 于2019年11月22日周五 下午5:47写道： > >> Congratulations, Daniel! >> >> On 22 Nov 2019, at 09:18, Jan Lukavský wrote: >> >> Congrats Daniel! >> On 11/21/19 10:11 AM,

Re: [ANNOUNCE] New committer: Brian Hulette

Congratulations, Brian! On Tue, Nov 19, 2019 at 11:04 PM Tanay Tummalapalli wrote: > Congratulations! > > On Wed, Nov 20, 2019 at 6:15 AM Aizhamal Nurmamat kyzy < > aizha...@apache.org> wrote: > >> Congratulations, Brian! >> >> On Mon, Nov 18, 2019 at 10:29 AM Łukasz Gajowy >> wrote: >> >>> Con

Re: Python Precommit duration pushing 2 hours

m removing the additional interactive test env + suite and add >> [interactive] dependencies as extra dependencies in tests_require: >> https://github.com/apache/beam/pull/10068 >> >> On Mon, Nov 11, 2019 at 2:15 PM Robert Bradshaw >> wrote: >> >>> On Fri

Re: Cleaning up Approximate Algorithms in Beam

Thank you for writing this summary. On Tue, Nov 12, 2019 at 6:35 PM Reza Rokni wrote: > Hi everyone; > > TL/DR : Discussion on Beam's various Approximate Distinct Count algorithms. > > Today there are several options for Approximate Algorithms in Apache Beam > 2.16 with HLLCount being the most r

Re: Cython unit test suites running without Cythonized sources

Thank you for spending time on this to clarify it for all of us! Much appreciated. On Sun, Nov 10, 2019 at 3:45 PM Chad Dombrova wrote: > Hi all, > > >> The sdist step creates a package that should be installed into each >> tox environment. If the tox environment has cython when this apache >> b

Re: Python Precommit duration pushing 2 hours

/ , so > perhaps we're not out of the woods yet (though in general things have > been a lot better). > > On Tue, Nov 5, 2019 at 10:52 AM Ahmet Altay wrote: > > > > GCP tests are already on separate locations. IO related tests are under > /sdks/python/apache_beam/io/gc

Re: Cython unit test suites running without Cythonized sources

x-installpkg > [3] > https://stackoverflow.com/questions/29870629/pip-install-test-dependencies-for-tox-from-setup-py > > > >> It's possible cython is present, but the build step is not >> invoked which would explain the skip for slow_coders_test. The correct >

Re: [Discuss] Beam Summit 2020 Dates & locations

I prefer bay are for NA summit. My reasoning is that there is a criticall mass of contributors and users in that location, probably more than alternative NA locations. I was not involved with planning recently and I do not know if there were people who could attend due to location previously. If th

Re: Cython unit test suites running without Cythonized sources

I believe tox is correctly installing cython and executes "python setup.py nosetests" which triggers cythonzation path inside setup.py. Some indications that cython is installed and used is the following log entries (from a recent precommit cron job [1]) - [ 1/12] Cythonizing apache_beam/coders/cod

Re: How to use a locally built worker image?

was updated (due to a bug fix >>>> for example). >>>> >>>> There's a PR for this [4]. Once we fix the default tag for Java/Go >>>> containers, the dev and release containers will be distinct, which makes it >>>> seldom important whether or not the image is `docker pull`

Re: Python Precommit duration pushing 2 hours

the core has reasonably stable >> abstractions that the connectors are built on, this should not change >> coverage much. >> >> Kenn >> >> On Mon, Nov 4, 2019 at 1:55 PM Ahmet Altay wrote: >> >>> PR for the proposed change: https://github.com/apach

Re: Python Precommit duration pushing 2 hours

PR for the proposed change: https://github.com/apache/beam/pull/9985 On Mon, Nov 4, 2019 at 1:35 PM Udi Meiri wrote: > +1 > > On Mon, Nov 4, 2019 at 12:09 PM Robert Bradshaw > wrote: > >> +1, this seems like a good step with a clear win. >> >> On Mon, Nov 4, 20

Re: Python Precommit duration pushing 2 hours

ell 9283 just added a well-tested feature. I don't think there's > anything to learn from that... I just wanted to circle back about it in > case others are curious about that spike. > >> >>>>>>>> > >> >>>>>>>> I

Re: RFC: python static typing PR

Thank you Chad, everyone else who helped with reviews so far. I think this is a positive change. I do share the worry of one more lint. I hope that we can hear from majority of beam python contributors. Good tooling and educational information would be helpful here. Ideally this will serve current

Re: [UPDATE] Preparing for Beam 2.17.0 release

On Mon, Oct 28, 2019 at 12:44 PM Gleb Kanterov wrote: > It looks like BigQueryIO DIRECT_READ is broken since 2.16.0, I've added a > ticket describing the problem and possible fix, see BEAM-8504 > [1]. > Should this be added to 2.16 blog post as a

Re: Is there good way to make Python SDK docs draft accessible?

Thank you for doing this. It should be possible to run tox as Udi suggested and create a PR for review purposes similar to the release process (see: https://beam.apache.org/contribute/release-guide/#build-the-pydoc-api-reference ) /cc +Valentyn Tymofieiev -- This is likely a required item before

Re: Multiple Outputs from Expand in Python

Is DoOutputsTuple what you are looking for? [1] You can look at this expand function using it [2]. [1] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pvalue.py#L204 [2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py#L1283 On Fri, Oct 25,

Re: Python Precommit duration pushing 2 hours

unlike multiple > precommit tests which run in parallel to each other. > > The precommit ITs we run are batch and streaming wordcount tests on Py2 > and one Py3 version, so it's not a lot of tests. > > On Thu, Oct 24, 2019 at 1:07 PM Ahmet Altay wrote: > >> +1 to s

Re: Python Precommit duration pushing 2 hours

+1 to separating ITs from precommit. Downside would be, when Chad tried to do something similar [1] it was noted that the total time to run all precommit tests would increase and also potentially increase the queue time. Another alternative, we could run a smaller set of IT tests in precommits and

Re: Python2 postcommit broken

Filed https://issues.apache.org/jira/browse/BEAM-8412 for this. On Wed, Oct 16, 2019 at 7:11 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Hello all, > > I've noticed that since last two days all Python2 post commit tests have > failed[1]. Logs show it's because of an exception in

Re: [PROPOSAL] Preparing for Beam 2.17.0 release

+1. Thank you. On Tue, Oct 15, 2019 at 8:35 PM Jean-Baptiste Onofré wrote: > It sounds good to me. > > +1 > > Regards > JB > > On 15/10/2019 20:37, Mikhail Gryzykhin wrote: > > Hi all, > > > > Beam 2.17 release branch cut is scheduled on Oct 23 according to > > the release calendar [1]. I would

Re: Introduction - BeamSQL and Extend the Nexmark Benchmarking Suite

Welcome! For the Nexmark, you can take a look at this thread [1] with links to relevant resources. Let us know if you have any questions. Ahmet [1] https://lists.apache.org/thread.html/b8b9f93ca6337b0b81c1900f2bb817cc1a1174889c0fc94c44ca9845@%3Cdev.beam.apache.org%3E On Sun, Oct 13, 2019 at 9:5

Re: Add myself to jira to update Avro dependency

Welcome to the project. I added your username to JIRA. On Fri, Oct 11, 2019 at 5:08 PM Jordanna wrote: > This is Jordanna Chord from Google. I'd like to update Apache Beam to > 1.9.1 of the Avro library. Can someone add me as a contributor for Beam's > Jira issue tracker so that I can assign/cre

Re: Beam Python fails to run on macOS 10.15?

Kamil, would it be possible for you to test with 0.15? I suggest moving the discussion to BEAM-8368 to have a single place for the discussion. On Thu, Oct 10, 2019 at 12:35 PM Brian Hulette wrote: > As noted in BEAM-8368 it seems like pinning to 0.13.0 is a workaround for > now. It would be usef

Re: Introduction to the mailing list

a8d7e78e13265dd4a20243e0c10d87 > > > On Sat, Oct 5, 2019 at 8:31 AM Manuela Chamda Tchakoute > wrote: > > > > Hello, > > > > Thank you for reaching out. I will take a look at all the links sent and > get back to you in case of any queries. > > > > Regards,

Re: Introduction to the mailing list

Welcome Manuela. You can look at (https://issues.apache.org/jira/browse/BEAM-2855) as the starting point. It has links to 2 previously merged PRs that can serve as starting points for writing new nexmark queries. There are nexmark queries described in (https://cwiki.apache.org/confluence/display/B

Re: [VOTE] Release 2.16.0, release candidate #1

t will be released. >> >> The ground truth for releases are the release tags, not the release >> branches. Downstream projects should not depend on the release branches. >> Release branches are merely important for the process of creating a >> release, but they l

Re: How to use a locally built worker image?

I do not believe this is a blocker for Beam 2.16. I agree that it would be good to fix this. On Wed, Oct 2, 2019 at 3:15 PM Hannah Jiang wrote: > Hi Thomas > > Thanks for bring this up. > > Now Python uses sdk version as a default tag, while Java and Go use latest > as a default tag. I agree usi

Re: [VOTE] Release 2.16.0, release candidate #1

+1 (validated python quickstarts). Thank you Mark. On Wed, Oct 2, 2019 at 10:49 AM Maximilian Michels wrote: > Thanks for preparing the release, Mark! I would like to address > https://issues.apache.org/jira/browse/BEAM-8303 in the release. I've > already merged the fix to the release-2.16.0 bra

Re: Dockerhub push denied for py3.6 and py3.7 image

On Wed, Oct 2, 2019 at 10:14 AM Robert Bradshaw wrote: > Please add robertwb0 as well. > Done. > > On Wed, Oct 2, 2019 at 9:09 AM Ahmet Altay wrote: > > > > > > > > On Tue, Oct 1, 2019 at 8:44 PM Pablo Estrada wrote: > >> > >> When sh

Re: Dockerhub push denied for py3.6 and py3.7 image

Thank you! On Wed, Oct 2, 2019 at 9:41 AM Pablo Estrada wrote: > I've added you as an owner. > Best > -P. > > On Wed, Oct 2, 2019 at 9:09 AM Ahmet Altay wrote: > >> >> >> On Tue, Oct 1, 2019 at 8:44 PM Pablo Estrada wrote: >> >>> When

Re: Dockerhub push denied for py3.6 and py3.7 image

ther have a PMC-managed account, or allow more PMC members to > have admin privileges - for redundancy. > I remember we agreed to add any interested PMC members for redundancy. Could you add my username ('aaltaybeam') to the list please? Ahmet > Best > -P. > >

Re: Dockerhub push denied for py3.6 and py3.7 image

Who are the admins on dockerhub currently? Is there a page that shows a list? The next person doing the release will probably run into similar issues. For example, pypi page for beam [1] shows lists of maintainers. [1] https://pypi.org/project/apache-beam/ Thank you, Ahmet On Tue, Oct 1, 2019 at

Re: Reading from BigQuery on portable runners in Python SDK

+Chamikara Jayalath and +Pablo Estrada might have ideas related to this. On Tue, Oct 1, 2019 at 2:39 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > If anyone is interested, here is a link to my code: > https://github.com/kamilwu/beam/tree/bounded-source-for-bq > > On Tue, Oct 1, 2

Re: Introduction + Support in Comms for Beam!

Welcome! On Tue, Oct 1, 2019 at 3:26 AM Jesse Anderson wrote: > Excellent and welcome! > > [image: Big Data Institute] Jesse Anderson > Managing Director > Big Data Institute > (775) 393 9122 | je...@bigdatainstitute.io > bigdatainstitute.io > > > On Tue, Oct 1

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev wrote: > Hi everyone, > > Please vote whether to sign a pledge on behalf of Apache Beam to sunset > Beam Python 2 offering (in new releases) in 2020 on > http://python3stament.org as follows: > > [ ] +1: Sign a pledge to discontinue support

[ANNOUNCE] New committer: Alan Myrvold

Hi, Please join me and the rest of the Beam PMC in welcoming a new committer: Alan Myrvold Alan has been a long time Beam contributor. His contributions made Beam more productive and friendlier [1] for all contributors with significant improvements to Beam release process, automation, and infrast

Re: Plan for dropping python 2 support

Thanks a lot for sharing your thoughts, I completely agree that we need to minimize the burden on our users as much as possible. Especially in this case when we are offering a robust python 3 solution just now. However I do share the same concerns related to dependencies and tool chains, It will be

Re: Next LTS?

I agree with retiring 2.7 as the LTS family. Based on my experience with users 2.7 does not have a particularly high adoption and as pointed out has known critical issues. Declaring another LTS pending demand sounds reasonable but how are we going to gauge this demand? +Yifan Zou +Alan Myrvold o

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

aver | Software Engineer | github.com/ibzib > > <http://github.com/ibzib> | kcwea...@google.com kcwea...@google.com> > > > > > > On Tue, Sep 17, 2019 at 3:38 PM Ahmet Altay > <mailto:al...@google.com>> wrote: > > > > > > > >

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

he root > of the problem. > I might be wrong, IIUC, it will temporarily resolve the reported issues. Is this not accurate? > > -Max > > On 17.09.19 14:20, Ahmet Altay wrote: > > Could you make that change and see if it would have addressed the issue > > here? > >

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

stance of PortableRunner. > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > > > On Tue, Sep 17, 2019 at 2:14 PM Ahmet Altay wrote: > >> Is not this flag set automatically for the portable runner here [1] ? >> >&g

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

Is not this flag set automatically for the portable runner here [1] ? [1] https://github.com/apache/beam/blob/f0aa877b8703eed4143957b4cd212aa026238a6e/sdks/python/apache_beam/pipeline.py#L160 On Tue, Sep 17, 2019 at 2:07 PM Robert Bradshaw wrote: > On Tue, Sep 17, 2019 at 1:43 PM Thomas Weise

Re: portableWordCountBatch and portableWordCountStreaming failing in Python PreCommit

On Mon, Sep 16, 2019 at 2:07 PM Kyle Weaver wrote: > The original issue ("GetJobMetrics is unimplemented") is still probably > hiding under the docker issue, so I filed > https://issues.apache.org/jira/browse/BEAM-8245 for it > Thank you. Who should be assigned to this issue? If metrics are not

Re: portableWordCountBatch and portableWordCountStreaming failing in Python PreCommit

To clarify are they errors or warnings? There is a plan to stop supporting python 2 by ~end of the year. +Valentyn Tymofieiev shared details about it earlier on the dev@ list. On Mon, Sep 16, 2019 at 11:34 AM Ning Kang wrote: > Hi! I've been seeing some errors during "Python PreCommit". > I'm s

Re: MQTT to Python SDK

A framework for python sdk to use a native unbounded connector does not exist yet. You might be able to use the same connector from Java using cross language transforms. /cc +Chamikara Jayalath On Mon, Sep 16, 2019 at 11:00 AM Lucas Magalhães < lucas.magalh...@paralelocs.com.br> wrote: > Hello

Re: Possible Python SDK performance regression

duced. On Fri, Sep 6, 2019 at 7:11 PM Valentyn Tymofieiev wrote: > Sounds like these regressions need to be investigated ahead of 2.16.0 > release. > > On Fri, Sep 6, 2019 at 6:44 PM Thomas Weise wrote: > >> >> >> On Fri, Sep 6, 2019 at 6:23 PM Ahmet Altay wrote: &

Re: [discuss] How we support our users on Slack / Mailing list / StackOverflow

ose that > have benchmarks to hit. I am willing to dig into Slack's API if desired, > to surface/forward messages as useful. Not sure how all that would look, > open to figure it out. > > > > On Fri, Sep 6, 2019 at 4:47 PM Ahmet Altay wrote: > >> Both Stack

Re: Possible Python SDK performance regression

/apache-beam-testing.appspot.com/explore?dashboard=5691127080419328 >> [2] >> https://drive.google.com/file/d/1ERlnN8bA2fKCUPBHTnid1l__81qpQe2W/view >> [3] >> https://github.com/apache/beam/commit/2d5e493abf39ee6fc89831bb0b7ec9fee592b9c5 >> >> >> >>

Re: [discuss] How we support our users on Slack / Mailing list / StackOverflow

Both StackOverflow and mailing lists have better answer rates for python questions. Suggesting either one of them makes sense. I also find StackOverflow easier to use but that is a personal preference. The original problem is that lack of support within Slack. Both mailing list and stackoverflow a

Re: Interactive Beam - support for caching and introspection of PCollections

(I believe you wanted to add +David Yan ) I am happy to see there are multiple related efforts. Both are introducing concepts. I would hope that beyond conflicts, we are not creating duplication and building a coherent experience. Could you reference to the discussions where this was agreed upon?

Re: Possible Python SDK performance regression

streaming test, that would be > great. We would use what we have internally as starting point. > > On Thu, Sep 5, 2019 at 5:00 PM Ahmet Altay wrote: > >> >> >> On Thu, Sep 5, 2019 at 4:15 PM Thomas Weise wrote: >> >>> The workload is quite differe

Re: Possible Python SDK performance regression

probably won't be practical to do a bisect due to the high cost of >>> each iteration with our fork/deploy setup. >>> >>> Perhaps it is time to setup something with the synthetic source that >>> works just with Beam as dependency. >>> >> I

Re: Possible Python SDK performance regression

There are a few in this dashboard [1], but not very useful in this case because they do not go back more than a month and not very comprehensive. I do not see a jump there. Thomas, would it be possible to bisect to find what commit caused the regression? +Pablo Estrada do we have any python on fl

Re: Beam at Google Summer of Code 2019

Thank you Tanay for all your contributions during summer and looking forward to more of it :) On Wed, Sep 4, 2019 at 10:38 AM Tanay Tummalapalli wrote: > Hi everyone, > > I've completed Google Summer of Code '19[1]. > I had fun working on Beam for the past 3 months and learning about Beam > inte

Re: [VOTE] Vendored Dependencies Release

+1 On Tue, Sep 3, 2019 at 10:52 AM Andrew Pilloud wrote: > +1 > > Inspected the jar it looked reasonable. > > Andrew > > On Tue, Sep 3, 2019 at 9:06 AM Rui Wang wrote: > >> Friendly ping. >> >> >> -Rui >> >> On Thu, Aug 29, 2019 at 9:50 AM Rui Wang wrote: >> >>> Thanks Kai and Andrew. Now prga

Re: [PROPOSAL] Preparing for Beam 2.16.0 release

+1, thank you for keeping the cadence. On Thu, Aug 29, 2019 at 7:42 PM Alan Myrvold wrote: > +1 Thanks for keeping to the schedule, Mark. > > On Thu, Aug 29, 2019 at 6:21 PM jincheng sun > wrote: > >> Hi Mark, >> >> +1 and thank you for keeping the cadence! >> >> BTW I have mark the Fix Version

Re: Improve container support

ent/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing> >>>>> for >>>>> easy commenting and reviewing. From this moment, all changes will be >>>>> updated to the Google Docs and I will sync to wiki after finalize all >>>>> plans. >

Re: Improve container support

On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw wrote: > On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang > wrote: > > > > Hi team > > > > I am working on improving docker container support for Beam. We would > like to publish prebuilt containers for each release version and daily > snapshot. Curren

Re: [ANNOUNCE] Beam 2.15.0 Released!

Thank you Yifan! On Fri, Aug 23, 2019 at 8:00 AM Yifan Zou wrote: > The Apache Beam team is pleased to announce the release of version 2.15.0. > > Apache Beam is an open source unified programming model to define and > execute data processing pipelines, including ETL, batch and stream > (continu

Re: Java 11 compatibility question

On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold wrote: > > > On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía wrote: > >> a per case approach (the exception could be portable runners not based on >> Java). >> >> Of course other definitions of being Java 11 compatible are interesting >> but pro

Re: [PROPOSAL] An initial Schema API in Python

On Tue, Aug 20, 2019 at 3:48 PM Brian Hulette wrote: > > > On Tue, Aug 20, 2019 at 1:41 PM Robert Bradshaw > wrote: > >> On Mon, Aug 19, 2019 at 5:44 PM Ahmet Altay wrote: >> > >> > >> > >> > On Mon, Aug 19, 2019 at 9:56 AM Brian Hulett

Re: [VOTE] Release 2.15.0, release candidate #2

mmits for different versions so that flakes do not stack up. Even if a suite is >90% reliable, if we stack up with 4 version, the reliability will get much lower. > >> On Mon, Aug 19, 2019 at 5:27 PM Ahmet Altay wrote: >> >>> Thank you. Unless there are any object

Re: (mini-doc) Beam (Flink) portable job templates

On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise wrote: > There is a PR open for this: https://github.com/apache/beam/pull/9331 > > (it wasn't tagged with the JIRA and therefore not linked) > > I think it is worthwhile to explore how we could further detangle the > client side Python and Java depend

Re: Need advice: PubsubIO external transform PR

Adding a few folks who might help +Chamikara Jayalath +Lukasz Cwik +Maximilian Michels On Sat, Aug 17, 2019 at 11:27 AM Chad Dombrova wrote: > Hi all, > I've got a PR[1] for adding > external transform support to PubsubIO so that it will work with p

Re: [PROPOSAL] An initial Schema API in Python

On Mon, Aug 19, 2019 at 9:56 AM Brian Hulette wrote: > > > On Fri, Aug 16, 2019 at 5:17 PM Chad Dombrova wrote: > >> >> Agreed on float since it seems to trivially map to a double, but I’m >>> torn on int still. While I do want int type hints to work, it doesn’t seem >>> appropriate to map it to

<    1   2   3   4   5   6   7   8   9   10   >