Re: error with DirectRunner

> >> Thanks Udi. I agree, since it works fine removing either the side input >> or the last flatten and combine operation. >> >> On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri wrote: >> >>> This looks like a FnApiRunner bug. >>> When I override u

Re: error with DirectRunner

+Robert Bradshaw I would be happy to debug and fix this, but I'd need more guidance on where to look. On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri wrote: > Created https://issues.apache.org/jira/browse/BEAM-5927 > > On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik wrote: > >>

BEAM-6018: memory leak in thread pool instantiation

HI, I've identified a memory leak when GcsUtil.java instantiates a ThreadPoolExecutor (https://issues.apache.org/jira/browse/BEAM-6018). The code uses the getExitingExecutorService

Re: BEAM-6018: memory leak in thread pool instantiation

e been caching the executor within > the GcsUtil as a static instance instead of creating one each time. Could > have been missed during code review / slow code changes over time. GcsUtil > is not well "loved". > > On Thu, Nov 8, 2018 at 11:00 AM Udi Meiri wrote: >

Re: [VOTE] Mark 2.7.0 branch as a long term support (LTS) branch

+1 On Fri, Nov 9, 2018 at 8:31 AM Maximilian Michels wrote: > +1 > > On 09.11.18 09:38, Robert Bradshaw wrote: > > +1 approve. > > On Fri, Nov 9, 2018 at 2:47 AM Ahmet Altay wrote: > >> > >> Hi all, > >> > >> Please review the following statement: > >> > >> "2.7.0 branch will be marked as the l

Re: BEAM-6018: memory leak in thread pool instantiation

e/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java#L133-L138 On Thu, Nov 8, 2018 at 11:41 PM Dan Halperin wrote: > >> On Thu, Nov 8, 2018 at 2:12 PM Udi Meiri wrote: >> >>> Both options risk delaying worker shutdown if the executor's shutdown()

Re: Spotless and lint precommit

+1 and parallelize the 3 lint tasks On Tue, Nov 13, 2018 at 10:43 AM Thomas Weise wrote: > +1 > > > On Tue, Nov 13, 2018 at 9:06 AM Ruoyun Huang wrote: > >> +1 >> >> On Tue, Nov 13, 2018 at 8:29 AM Maximilian Michels >> wrote: >> >>> +1 >>> >>> On 13.11.18 14:22, Robert Bradshaw wrote: >>> > I

Please do not merge Python PRs

It seems that Gradle is not getting the correct exit status from test runs. Possible culprit: https://github.com/apache/beam/pull/6903 smime.p7s Description: S/MIME Cryptographic Signature

Re: Please do not merge Python PRs

Recreated locally: https://gradle.com/s/psqgcywnc3h2m On Wed, Nov 14, 2018 at 5:16 PM Udi Meiri wrote: > It seems that Gradle is not getting the correct exit status from test runs. > Possible culprit: https://github.com/apache/beam/pull/6903 > smime.p7s Description: S/MIME Cryp

Re: Please do not merge Python PRs

https://github.com/apache/beam/pull/7048 is the rollback PR On Wed, Nov 14, 2018 at 5:28 PM Ahmet Altay wrote: > Thank you Udi. Could you send a rollback PR? > > I believe this is https://issues.apache.org/jira/browse/BEAM-6048 > > On Wed, Nov 14, 2018 at 5:16 PM, Udi Meiri

Re: Please do not merge Python PRs

All clear, Python tests are reporting errors correctly again. On Wed, Nov 14, 2018 at 5:57 PM Udi Meiri wrote: > https://github.com/apache/beam/pull/7048 is the rollback PR > > On Wed, Nov 14, 2018 at 5:28 PM Ahmet Altay wrote: > >> Thank you Udi. Could you send a rollback P

Re: Need help regarding memory leak issue

If you're working with Dataflow, it supports this flag: https://github.com/apache/beam/blob/75e9f645c7bec940b87b93f416823b020e4c5f69/sdks/python/apache_beam/options/pipeline_options.py#L602 which uses guppy for heap profiling. On Fri, Nov 16, 2018 at 3:08 PM Ruoyun Huang wrote: > Even tough the

Re: Python Static Typing: Next Steps

Off-topic: Python lint via pre-commit should be much faster. (I wrote my own modified-file-only lint in the past) On Mon, Mar 2, 2020 at 2:08 PM Kyle Weaver wrote: > > Python lint takes 4-5mins to complete. I think if the mypy analysis is > really on the order of 10s, the additional time won't m

Re: Install Jenkins AnsiColor plugin

> seem to be some ansi plugin already installed but I couldn’t get it to > work. > > -chad > > > On Sun, Mar 8, 2020 at 10:52 AM Ismaël Mejía wrote: > >> Did this ever happen? If not what is blocking it? >> >> >> >> On Tue, Oct 22, 2019 at

Re: Hello Beam Community!

Welcome! On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang wrote: > Welcome! > > On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay wrote: > >> Welcome Brittany! >> >> On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann >> wrote: >> >>> Hello Beam Community! >>> >>> My name is Brittany Hermann and I recently

Re: Are docker image tags shared within a jenkins worker?

The Python HDFS IT uses the jenkins BUILD_TAG to create unique names: PROJECT_NAME=$(echo hdfs_IT-${BUILD_TAG:-non-jenkins}) The BUILD_TAG is unique and easily traced back to the Jenkins job that made it. It might need some sanitizing though if it contains any invalid characters. On Tue, Mar 24,

Re: [BEAM-9322] Python SDK discussion on correct output tag names

On Thu, Mar 26, 2020 at 10:13 AM Luke Cwik wrote: > The issue seems to be that a PCollection can have a "tag" associated with > it and PTransform expansion can return an arbitrary nested dictionary/tuple > yet we need to figure out what the user wanted as the local name for the > PCollection from

Re: Implementing type hints on multi-output PTransforms

Hi Joshua, I've been working on type hints recently. Your issue is similar to this: https://issues.apache.org/jira/browse/BEAM-8782 (exactly the same if tags are passed to with_outputs() in the example). There's also this related bug about type inference: https://issues.apache.org/jira/browse/BEAM-

Re: [DISCUSS] Let's establish a guideline for using Python type annotations in Beam codebase

I agree with Robert to only put Any where it really can be any type. I'm not sure how much typing we should add. At minimum: external APIs and wherever mypy complains. Ideally I would like to have annotations everywhere, because this reduces uncertainty when modifying existing code. You are increa

Re: Implementing type hints on multi-output PTransforms

PSAL] ..." to this mailing list describing what you want to change should be enough. A document could also work; I'm not aware of one that touches on this. Your PR would need to have an associated JIRA (feel free to take over any of the ones I've mentioned). > Best, > Joshua >

Re: Website publish jobs fail recently

Hey, I was looking at this today but could not figure it out. The machines we run the publish jobs probably vary from our regular apache-beam-testing Jenkins ones. I tried researching all the reasons why this might be happening but came up empty. On Tue, Apr 14, 2020 at 10:19 AM Kyle Weaver wrote

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

If this process is used in releases we would benefit from running it regularly to ensure it isn't broken and thus delay releases (and add work for the release manager). Does it make sense to put it in postcommit? On Wed, Apr 15, 2020 at 2:30 PM Kyle Weaver wrote: > Looks like the same error as t

Re: Website publish jobs fail recently

enneth Knowles wrote: > >> Indeed, publish jobs have write access to things, which normal builds do >> not. >> >> I suggest reaching out to bui...@apache.org >> >> Kenn >> >> On Tue, Apr 14, 2020 at 5:58 PM Udi Meiri wrote: >> >>> He

Re: Website publish jobs fail recently

ost 2.20 release. >> >> On Tue, Apr 14, 2020 at 9:01 PM Kenneth Knowles wrote: >> >>> Indeed, publish jobs have write access to things, which normal builds do >>> not. >>> >>> I suggest reaching out to bui...@apache.org >>> >>

Re: [ANNOUNCE] Beam 2.20.0 Released

You'll need to add --tags On Fri, Apr 24, 2020 at 11:53 AM Jan Lukavský wrote: > Hm, that is strange: > > ~/git/apache/beam$ git status > On branch master > Your branch is up to date with 'origin/master'. > ~/git/apache/beam$ git pull > Already up to date. > ~/git/apache/beam$ git tag | grep v2.

Re: Jira PR links not being generated?

We had such a file for a short while but it was removed: https://github.com/apache/beam/pull/10645 I don't believe it contained any PR link settings though +Pablo Estrada On Mon, Apr 27, 2020 at 1:56 PM Kyle Weaver wrote: > I went ahead and filed https://issues.apache.org/jira/browse/BEAM-9833

Re: JIRA Committer Permissions

Should this step be added to our new committer guide? On Fri, Apr 24, 2020 at 6:21 PM Luke Cwik wrote: > I noticed that several committers only had contributor level permissions > and I went and updated your account permissions for the Beam project to be > committer level. Feel free to let me kn

Re: Jira PR links not being generated?

id not need this file last week, and now we do this week. > https://github.com/apache/beam/pull/11541 > > On Mon, Apr 27, 2020 at 6:24 PM Udi Meiri wrote: > >> We had such a file for a short while but it was removed: >> https://github.com/apache/beam/pull/10645 >>

Re: Jenkins jobs not running for my PR 10438

Alexey, what you're doing should be working (commits should trigger tests, as should "retest this please" and other phrases). https://issues.apache.org/jira/browse/INFRA-19836 tracks this issue On Tue, Apr 28, 2020 at 10:04 AM Alexey Romanenko wrote: > Does anyone know the “golden rule” how to

Re: How to submit PRs for dependant changes?

(a) or (c) should work. (c) is preferred if you want faster reviews. For multiple JIRAs, I've seen both [BEAM-123,BEAM-456] and [BEAM-123][BEAM-456] formats. One of them works but I'm not sure which. :D You can always manually add a PR to a JIRA. On Sun, Apr 26, 2020 at 2:49 PM Reuven Lax wrot

Jenkins test triggers

Hi, Has anyone noticed any changes today (since ~6h ago) to how tests are triggered on PRs? Are they triggering always, some of the time, not at all? Are phrase comment triggers working? Thanks. (background: https://issues.apache.org/jira/browse/INFRA-19836) smime.p7s Description: S/MIME Crypto

Re: Python 3.7 docker container fails to build

I checked node 8 and it had over 40GB space available. Does your job require more than that? Long term, I'm thinking we could clean up workspaces for successful jobs. This should free up additional space (I guess at least 100GB). https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin

Re: Python 3.7 docker container fails to build

e log above: > > > > --info tag was passed to docker image build commands with PythonDocker > > Precommit to capture more logs. Without the tag, errors from DockerFile > > step are not printed out to the console. > > > > On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri

Re: Beam 2.21 release update

Probably not the issue, but double checking: are you running "pip install -r sdks/python/build-requirements.txt" first? On Wed, May 6, 2020 at 7:22 PM Thomas Weise wrote: > I'm working on rebasing our fork to 2.21.0 and run into a problem > installing grpcio-tools that leads to *ModuleNotFoundEr

Re: Beam 2.21 release update

Is there a reason why this cannot happen automatically when >> running python3.6 setup.py sdist bdist_wheel ? >> > > I _believe_ this should happen automatically here: > https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365. > Maybe there is a problem the

Re: Beam 2.21 release update

tutils/cmd.py", line 326, in run_command > self.distribution.run_command(command) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "setup.py", line 235, in run > gen_protos.gen

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations Robin! On Tue, May 19, 2020, 10:15 Valentyn Tymofieiev wrote: > Congratulations, Robin! > > On Tue, May 19, 2020 at 9:10 AM Yichi Zhang wrote: > >> Congrats Robin! >> >> On Tue, May 19, 2020 at 8:56 AM Kamil Wasilewski < >> kamil.wasilew...@polidea.com> wrote: >> >>> Congrats! >

Re: [ANNOUNCE] Beam 2.21.0 Released

Woohoo! On Thu, May 28, 2020 at 4:16 AM Kyle Weaver wrote: > The Apache Beam team is pleased to announce the release of version 2.21.0. > > Apache Beam is an open source unified programming model to define and > execute data processing pipelines, including ETL, batch and stream > (continuous) pr

latest release on Github

Hi, Trying out the new Github layout (feature preview) I've noticed that the latest release is featured more prominently and is at v2.16.0 for some reason. https://github.com/apache/beam/releases/tag/v2.16.0 Any idea how to fix this, and what instructions should be added to the release guide? sm

Re: latest release on Github

ating Github releases [1], short of that however we can just use the >> UI. >> >> [1] >> https://lists.apache.org/thread.html/re367d262333078501c47856e9b8a0fc3fd7db60c2d2ebb181275481a%40%3Cdev.beam.apache.org%3E >> >> >> On Mon, Jun 1, 2020 at 3:07 PM Udi Meir

Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

That's great! Can someone please make custom emojis for our Slack from the expression sheet images? :) ( https://github.com/apache/beam/blob/master/website/www/site/static/images/mascot/model_sheet.png ) A smiling firefly and one with heart eyes would be awesome. On Wed, Jun 3, 2020 at 11:04 AM A

Kafka IO performance tests leaving behind disks on GCP

Hi, I opened a bug on what seems to be leftover GKE disk images: https://issues.apache.org/jira/browse/BEAM-10145 Can anyone familiar with these take a look? Thanks! smime.p7s Description: S/MIME Cryptographic Signature

python precommit error - google-auth depenedency?

Hi, I'm trying to understand these "pip check" failures: ERROR: google-auth 1.16.1 has requirement rsa<4.1,>=3.1.4, but you'll have rsa 4.1 which is incompatible https://builds.apache.org/job/beam_PreCommit_Python_Cron/2860/console However, when I do pip install dist/apache-beam-2.23.0.dev0.tar

Re: python precommit error - google-auth depenedency?

You could try installing and calling pipdeptree [1] from a Jenkins job, > and see if it helps. > > [1] https://pypi.org/project/pipdeptree/ > On Wed, Jun 10, 2020 at 11:00 AM Udi Meiri wrote: > >> Hi, >> I'm trying to understand these "pip check" failure

Re: python precommit error - google-auth depenedency?

Seems like manually installing rsa==4.0 satisfies deps, but pip doesn't do transitive deps well. Would it be right to put a direct dependency on rsa<4.1,>=3.1.4 in setup.py? On Wed, Jun 10, 2020 at 11:48 AM Udi Meiri wrote: > Thanks, that helped in an unexpected way. :) > I

Re: python precommit error - google-auth depenedency?

On Wed, Jun 10, 2020 at 1:59 PM Ahmet Altay wrote: > > > On Wed, Jun 10, 2020 at 1:29 PM Kenneth Knowles wrote: > >> You may be interested in following https://github.com/pypa/pip/issues/988 if >> you are not already. >> >> Kenn >> >> O

Re: python precommit error - google-auth depenedency?

4.1 drops Python 2 support, so I'm not sure if we're ready for that yet. On Wed, Jun 10, 2020 at 2:20 PM Ahmet Altay wrote: > Looks like there is an attempt to fix this: > https://github.com/googleapis/google-auth-library-python/pull/524 > > On Wed, Jun 10, 2020 at 2:

Re: python precommit error - google-auth depenedency?

Yes you're right, Py2 envs are still using 4.0. On Wed, Jun 10, 2020 at 3:03 PM Ahmet Altay wrote: > > > On Wed, Jun 10, 2020 at 2:25 PM Udi Meiri wrote: > >> 4.1 drops Python 2 support, so I'm not sure if we're ready for that yet. >> > &g

Re: python precommit error - google-auth depenedency?

>>>> >>>> Could we add it as a test dependency? Or if that is not possible, add >>>> it but remove it before next release? >>>> >>>> It seems like there is a release PR on google-auth ( >>>> https://github.com/googleapis/google-auth-li

Re: Beam Jenkins Migration

This is great! Looking forward to it. Would any metrics need to be migrated over to the new Jenkins? http://metrics.beam.apache.org/ On Fri, Jun 12, 2020 at 9:57 AM Tyson Hamilton wrote: > Very exciting! Thanks for the advanced notice Damian. > > On Fri, Jun 12, 2020 at 7:58 AM Damian Gadomsk

Re: DISCUSS: FnAPI proto stabiliization

I'm not very familiar with this effort. Were there ITs / POCs created for these changes? (to surface any obvious bugs) Are these changes usable in DirectRunner? On Fri, Jun 12, 2020 at 8:50 AM Luke Cwik wrote: > A few months back there was a discussion[1] about performing work to > stabilize th

Re: Python2.7 Beam End-of-Life Date

+1 On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay wrote: > As a concrete proposal, could we commit to removing python 2 support by > 2.24? In other words, mark the next release 2.23 as the last python 2 > compatible Beam version. > > On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev > wrote: > >

Re: Automation for Jira

Interesting: you could consider the JIRA as active as long as the linked PRs are open. On Mon, Jun 15, 2020 at 2:28 PM Luke Cwik wrote: > One thing I noticed is that links being added to issues automatically > (e.g. a PR is opened that tags something) doesn't reset the activity > counter so thin

Re: [ANNOUNCE] New PMC Member: Alexey Romanenko

Congratulations! On Tue, Jun 16, 2020 at 9:33 AM rahul patwari wrote: > Congrats Alexey! > > On Tue, Jun 16, 2020 at 9:27 PM Ismaël Mejía wrote: > >> Please join me and the rest of Beam PMC in welcoming Alexey Romanenko as >> our >> newest PMC member. >> >> Alexey has significantly contributed

Re: Commands to detect style issues quickly before sending PR

Another tip for Python: Also "pip install pre-commit" will run yapf and pylint on changed files during "git commit". On Fri, Jun 26, 2020 at 4:03 PM Valentyn Tymofieiev wrote: > See also: > https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-Formatting > > > Task :sdks:pytho

Re: [Proposal] Adding Python Coverage Reports To CI/CD

de whether their >> tests need revision. >> > >> > You can view/comment on the proposal here, or the full text of the >> proposal at the end of this email. Let me know what you think, or if there >> are any suggestions for improvements. Thanks! >&

Re: [Proposal] Supporting Python Annotation Typehints on PTransform

ink, or if you have any suggestions for > improvements. Thanks! > > Best, > Saavan > > --- > > Supporting Python Annotations on PTransforms > > https://issues.apache.org/jira/browse/BEAM-10258 > > Author: Saavan Nanavati (saa...@google.com ) > > Revi

Re: Monitoring performance for releases

On Thu, Jul 9, 2020 at 12:48 PM Maximilian Michels wrote: > Not yet, I just learned about the migration to a new frontend, including > a new backend (InfluxDB instead of BigQuery). > > > - Are the metrics available on metrics.beam.apache.org? > > Is http://metrics.beam.apache.org online? I was n

[PROPOSAL] Make PBegin and PDone public in the Python SDK

Details: One item of interest that came up during the implementation of BEAM-10258 [1] is how to treat PTransforms that act like sources or sinks. These transforms have either no input or output PCollections, respectively. Internally, we use PBegin and PDone to denote this. (ex: [2]) IIUC, PBegin

Re: [PROPOSAL] Make PBegin and PDone public in the Python SDK

think there's any use for it in the Python SDK (a >> transform can simply not return any value (which is equivalent to >> returning None) if it has no outputs. >> >> On Mon, Jul 13, 2020 at 8:17 PM Udi Meiri wrote: >> > >> > Details: >> > One

Re: No space left on device - beam-jenkins 1 and 7

p and running the tests without write permission to /tmp, etc) > > Kenn > > On Mon, Jul 20, 2020 at 11:39 AM Ahmet Altay wrote: > >> Related to workspace directory growth, +Udi Meiri filed >> a relevant issue previously ( >> https://issues.apache.org/jira/browse/BE

Jenkins trigger phrase "run seed job" not working?

HI, I'm trying to test a groovy change but I can't seem to trigger the seed job. It worked yesterday so I'm not sure what changed. https://github.com/apache/beam/pull/12326 smime.p7s Description: S/MIME Cryptographic Signature

Re: Jenkins trigger phrase "run seed job" not working?

it might be related to >> this change: https://github.com/apache/beam/pull/12319. >> >> /cc +Tyson Hamilton >> >> On Wed, Jul 22, 2020 at 1:17 PM Udi Meiri wrote: >> >>> HI, >>> I'm trying to test a groovy change but I can't seem to trigger the seed >>> job. It worked yesterday so I'm not sure what changed. >>> >>> https://github.com/apache/beam/pull/12326 >>> >>> smime.p7s Description: S/MIME Cryptographic Signature

Re: Jenkins trigger phrase "run seed job" not working?

nd configure the jobs with the apache > committers present on the ghprb whitelist. > > > Hope that I didn't miss anything ;) It isn't that easy to investigate that > kind of issues with my poor privileges ;) > > > Regards, > > Damian > > > On Thu, Ju

Re: No space left on device - beam-jenkins 1 and 7

gt;>>>>> 242M2020-07-19 12:14./beam-pipeline-tempUTXqlM >>>>>> 242M2020-07-19 12:11./beam-pipeline-tempx3Yno3 >>>>>> 242M2020-07-19 12:05./beam-pipeline-tempyCrMYq >>>>>> 236M2020-07-19 12:14./beam-pipeline-tem

Re: No space left on device - beam-jenkins 1 and 7

In the "jenkins" user home directory. On Fri, Jul 24, 2020, 11:19 Tyson Hamilton wrote: > Everything I've been looking at is in the /tmp dir. Where are the > workspaces, or what are the named? > > > > > On Fri, Jul 24, 2020 at 11:03 AM Udi Meiri wrote: >

Re: No space left on device - beam-jenkins 1 and 7

space >>> directory. When I run a du on that, it takes really long. I'll let it keep >>> running for a while to see if it ever returns a result but so far this >>> seems suspect. >>> >>> >>> >>> >>> >>> On Fr

Re: No space left on device - beam-jenkins 1 and 7

test jobs need to try harder at cleaning up their leftover images. (assuming they're already doing it?) On Fri, Jul 24, 2020 at 12:31 PM Udi Meiri wrote: > The additional slots (@3 directories) take up even more space now than > before. > > I'm testing out https://github.com/

Re: Jenkins trigger phrase "run seed job" not working?

as the github usernames. My current idea is to read this >> from LDAP as a part of the seed job and configure the jobs with the apache >> committers present on the ghprb whitelist. >> >> >> Hope that I didn't miss anything ;) It isn't that easy to inves

Re: [VOTE] Make Apache Beam 2.24.0 the final release supporting Python 2.

+1 On Fri, Jul 24, 2020 at 10:47 AM Maximilian Michels wrote: > +1 > > On 24.07.20 18:54, Pablo Estrada wrote: > > +1 - thank you Valentyn! > > -P. > > > > On Thu, Jul 23, 2020 at 1:29 PM Chamikara Jayalath > > wrote: > > > > +1 > > > > On Thu, Jul 23, 2020

Re: No space left on device - beam-jenkins 1 and 7

apache.org/job/beam_Inventory_apache-beam-jenkins-14/501/console > > > > On Fri, Jul 24, 2020 at 1:07 PM Tyson Hamilton > wrote: > > > > > Yes, these are on the same volume in the /var/lib/docker directory. I'm > > > unsure if they clean up leftover

Re: Git commit history: "fixup" commits

https://github.com/marketplace/actions/gs-commit-message-checker On Tue, Aug 4, 2020 at 10:25 AM Robert Bradshaw wrote: > +1, thanks for the reminder. > > This should be really easy to automate, using > https://developer.github.com/webhooks/event-payloads/#pull_request to > give a warning when t

Re: Broken links in code velocity dashboard

There's this old bug about it: https://issues.apache.org/jira/browse/BEAM-8389 On Fri, Aug 7, 2020 at 4:45 AM Damian Gadomski wrote: > Unfortunately, I'm not aware of any recent changes. > > On Thu, Aug 6, 2020 at 10:00 PM Ahmet Altay wrote: > >> Damian, or anyone else, do you know if there wer

Re: Memory Issue When Running Beam On Flink

HI David, I'm not familiar with Flink, but assuming there aren't any memory management issues in the runner or SDK, try reducing proceesing_time_duration (30 minutes currently) to 60 seconds and see how long it takes for memory usage to reach the limit. Could you also say how long it currently take

JIRA - can't set resolution?

Example: https://issues.apache.org/jira/browse/BEAM-10751 When I click "resolve issue" the status changes to "resolved" but resolution is still "unresolved" and I can't change it. I believe there was a change a while back to JIRA? smime.p7s Description: S/MIME Cryptographic Signature

Re: JIRA - can't set resolution?

jira/browse/INFRA-20563 >> >> On Tue, Aug 25, 2020 at 10:34 AM Udi Meiri wrote: >> >>> Example: https://issues.apache.org/jira/browse/BEAM-10751 >>> When I click "resolve issue" the status changes to "resolved" but >>>

JIRA components

Hi, I was going to create a Python types component in our JIRA. (sdk-py-types) Any objections? smime.p7s Description: S/MIME Cryptographic Signature

Re: JIRA components

Thu, Sep 17, 2020 at 7:17 PM Kenneth Knowles wrote: > >> What does it mean? >> >> Kenn >> >> On Thu, Sep 17, 2020 at 6:18 PM Udi Meiri wrote: >> >>> Hi, I was going to create a Python types component in our JIRA. >>> (sdk-py-types) >>> Any objections? >>> >> smime.p7s Description: S/MIME Cryptographic Signature

Re: Shutting down Perfkit Explorer

, but the > datastore related stuff remains functional. > > On Tue, Sep 22, 2020 at 10:37 AM Udi Meiri wrote: > >> Is it possible to create a simple "hello world" application instead? >> >> On Tue, Sep 22, 2020 at 10:35 AM Udi Meiri wrote: >> >>>

Re: [PROPOSAL] Preparing for Beam 2.26.0 release

+1 sg! On Tue, Oct 27, 2020 at 10:02 AM Robert Burke wrote: > Hello everyone! > > The next Beam release (2.26.0) is scheduled to be cut on November 4th > according to the release calendar [1]. > > I'd like to volunteer myself to handle this release. I plan on cutting the > branch on November 5th

Re: Jenkins trigger phrase "run seed job" not working?

Yeah, I've had that issue recently: https://github.com/apache/beam/pull/13213 (seed job did not trigger) I've personally never seen where this is configured, but it sounds like it. What happens if you edit a file in the PR? It should make you a co-author. On Mon, Nov 9, 2020 at 3:36 AM Kamil Wasi

Re: [ANNOUNCE] New PMC Member: Chamikara Jayalath

Congrats Cham! On Thu, Jan 21, 2021 at 4:25 PM Griselda Cuevas wrote: > Congratulations Cham!!! Well deserved :) > > On Thu, 21 Jan 2021 at 15:23, Connell O'Callaghan > wrote: > >> Well done Cham!!! Thank you for all your contributions to date!!! >> >> >> On Thu, Jan 21, 2021 at 3:18 PM Rui Wan

Re: BEAM-3713: Moving from nose to pytest

se/pytest which tests to run. So if I wanted to run post-commit tests on direct runner I would use this pytest flag: "-m 'it_postcommit and not no_direct'". On Wed, Mar 24, 2021 at 5:41 PM Ahmet Altay wrote: > All PRs look either merged or closed. > > +Udi Meiri might

Codecov Bash Uploader Security Notice

I got this email: https://about.codecov.io/security-update/ This is where we use codecov: https://github.com/apache/beam/blob/39923d8f843ecfd3d89443dccc359c14aea8f26f/sdks/python/tox.ini#L105 I'm not sure if this runs the "bash uploader", but we do set a CODECOV_TOKEN environment variable. smim

Re: Codecov Bash Uploader Security Notice

>From the notice: "We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders." On Thu, Apr 15, 2021 at 11:35 AM Udi Meiri wrote: >

Re: BEAM-3713: Moving from nose to pytest

r('ValidatesRunner') -> @pytest.mark.it_validatesrunner On Thu, Mar 25, 2021 at 10:45 AM Udi Meiri wrote: > Hi Benjamin, > > AFAIK nose is only used for integration tests (unit tests were converted > to pytest a while back). > These ITs should all be running periodi

Re: One Pager - Test Command Line Discoverability in Beam

My first place to go would be here: https://cwiki.apache.org/confluence/display/BEAM/Java+Tips (although it doesn't document your use-case) You are right that finding the correct gradle task or jenkins job is not straightforward. On Tue, May 25, 2021 at 12:48 PM Alex Amato wrote: > Friendly pi

Re: [PROPOSAL] Preparing for Beam 2.33.0 Release

t; See my comment below. > > On 26 Aug 2021, at 03:44, Ahmet Altay wrote: > > On Wed, Aug 25, 2021 at 1:50 PM Udi Meiri wrote: > >> I've found 2 performance regressions in IO. Could someone triage these? >> > > >> https://issues.apache.org/jira/brows

[VOTE] Release 2.33.0, release candidate 1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.33.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

[RESULT] [VOTE] Release 2.33.0, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 8 approving votes, 4 of which are binding: * Ahmet Altay * Alexey Romanenko * Robert Bradshaw * Chamikara Jayalath There are no disapproving votes. Thanks everyone!

Re: [RESULT] [VOTE] Release 2.33.0, release candidate #1

I spoke too soon. We will be doing an rc2 On Mon, Sep 27, 2021 at 1:29 PM Udi Meiri wrote: > I'm happy to announce that we have unanimously approved this release. > > There are 8 approving votes, 4 of which are binding: > * Ahmet Altay > * Alexey Romanenko > * Robert

[VOTE] Release 2.33.0, release candidate 2

Hi everyone, Please review and vote on the release candidate #2 for the version 2.33.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

[RESULT] [VOTE] Release 2.33.0, release candidate #2

I'm happy to announce that we have unanimously approved this release. There are 8 approving votes, 5 of which are binding: * Ahmet Altay * Alexey Romanenko * Robert Bradshaw * Kenneth Knowles * Chamikara Jayalath There are no disapproving votes. Thanks everyone!

[ANNOUNCE] Beam 2.33.0 Release

The Apache Beam team is pleased to announce the release of version 2.33.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release her

Proposal: Structured Logging

Hi all, I have written a proposal for Structured Logging in Beam: https://s.apache.org/beam-structured-logging Please LMK what you think. Any comments welcome here or in the doc. - Udi

Re: Proposal: Structured Logging

The document has been updated. Please take another look and let me know if there are any objections Thanks! On Wed, Mar 8, 2023 at 1:59 PM Udi Meiri wrote: > Hi all, > I have written a proposal for Structured Logging in Beam: > https://s.apache.org/beam-structured-logging > >

Re: Structured Logging in python

Hi, I believe this wasn't implemented for Python (only Java). You can try adding structured data (extra keyword) under the key "custom_data" and that might work. On 2024/04/11 17:49:43 Valentyn Tymofieiev wrote: > Thanks for reaching out. There was a proposal a while back: > https://s.apache.org

<    1   2   3   4