[PROPOSAL] Using Bazel and Docker for Python SDK development and tests

Hi, In light of increasing Python pre-commit times due to the added Python 3 tests, I thought it might be time to re-evaluate the tools used for Python tests and development, and propose an alternative. Currently, we use nosetests, tox, and virtualenv for testing. The proposal is to use Bazel, wh

Python Datastore client upgrade plan

Hi, Sadly upgrading googledatastore -> google-cloud-datastore is non-trivial ( https://issues.apache.org/jira/browse/BEAM-4543). I wrote a doc to summarize the plan: https://docs.google.com/document/d/1sL9p7NE5Z0p-5SB5uwpxWrddj_UCESKSrsvDTWNKqb4/edit?usp=sharing Contents pasted below: Beam Python

Re: [PROPOSAL] Using Bazel and Docker for Python SDK development and tests

On Wed, Oct 17, 2018 at 1:38 AM Robert Bradshaw wrote: > On Tue, Oct 16, 2018 at 12:48 AM Udi Meiri wrote: > >> Hi, >> >> In light of increasing Python pre-commit times due to the added Python 3 >> tests, >> I thought it might be time to re-evaluate th

Re: [PROPOSAL] Using Bazel and Docker for Python SDK development and tests

be able to work for Python as well. >> > > +1, that's a great idea. Together with --parallel--safe-build should be > sufficient. > > We could separately look into whether it's worth adding annotations > (positive or negative) to mark tests which have low value to be

Re: New Edit button on beam.apache.org pages

Love it! On Wed, Oct 24, 2018 at 2:26 PM Ahmet Altay wrote: > Really cool! Thank you! > > On Wed, Oct 24, 2018 at 2:24 PM, Alan Myrvold wrote: > >> To make small documentation changes easier, there is now an Edit button >> at the top right of the pages on https://beam.apache.org. This button >>

Re: error with DirectRunner

This looks like a FnApiRunner bug. When I override use_fnapi_runner = False in direct_runner.py the pipeline works. It seems like either the side-input to _copy_number or the Flatten operation is the culprit. On Mon, Oct 29, 2018 at 2:37 PM Allie Chen wrote: > Hi, > > I have a project that star

Re: error with DirectRunner

> >> Thanks Udi. I agree, since it works fine removing either the side input >> or the last flatten and combine operation. >> >> On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri wrote: >> >>> This looks like a FnApiRunner bug. >>> When I override u

Re: error with DirectRunner

+Robert Bradshaw I would be happy to debug and fix this, but I'd need more guidance on where to look. On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri wrote: > Created https://issues.apache.org/jira/browse/BEAM-5927 > > On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik wrote: > >>

BEAM-6018: memory leak in thread pool instantiation

HI, I've identified a memory leak when GcsUtil.java instantiates a ThreadPoolExecutor (https://issues.apache.org/jira/browse/BEAM-6018). The code uses the getExitingExecutorService

Re: BEAM-6018: memory leak in thread pool instantiation

e been caching the executor within > the GcsUtil as a static instance instead of creating one each time. Could > have been missed during code review / slow code changes over time. GcsUtil > is not well "loved". > > On Thu, Nov 8, 2018 at 11:00 AM Udi Meiri wrote: >

Re: [VOTE] Mark 2.7.0 branch as a long term support (LTS) branch

+1 On Fri, Nov 9, 2018 at 8:31 AM Maximilian Michels wrote: > +1 > > On 09.11.18 09:38, Robert Bradshaw wrote: > > +1 approve. > > On Fri, Nov 9, 2018 at 2:47 AM Ahmet Altay wrote: > >> > >> Hi all, > >> > >> Please review the following statement: > >> > >> "2.7.0 branch will be marked as the l

Re: BEAM-6018: memory leak in thread pool instantiation

e/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java#L133-L138 On Thu, Nov 8, 2018 at 11:41 PM Dan Halperin wrote: > >> On Thu, Nov 8, 2018 at 2:12 PM Udi Meiri wrote: >> >>> Both options risk delaying worker shutdown if the executor's shutdown()

Re: Spotless and lint precommit

+1 and parallelize the 3 lint tasks On Tue, Nov 13, 2018 at 10:43 AM Thomas Weise wrote: > +1 > > > On Tue, Nov 13, 2018 at 9:06 AM Ruoyun Huang wrote: > >> +1 >> >> On Tue, Nov 13, 2018 at 8:29 AM Maximilian Michels >> wrote: >> >>> +1 >>> >>> On 13.11.18 14:22, Robert Bradshaw wrote: >>> > I

Please do not merge Python PRs

It seems that Gradle is not getting the correct exit status from test runs. Possible culprit: https://github.com/apache/beam/pull/6903 smime.p7s Description: S/MIME Cryptographic Signature

Re: Please do not merge Python PRs

Recreated locally: https://gradle.com/s/psqgcywnc3h2m On Wed, Nov 14, 2018 at 5:16 PM Udi Meiri wrote: > It seems that Gradle is not getting the correct exit status from test runs. > Possible culprit: https://github.com/apache/beam/pull/6903 > smime.p7s Description: S/MIME Cryp

Re: Please do not merge Python PRs

https://github.com/apache/beam/pull/7048 is the rollback PR On Wed, Nov 14, 2018 at 5:28 PM Ahmet Altay wrote: > Thank you Udi. Could you send a rollback PR? > > I believe this is https://issues.apache.org/jira/browse/BEAM-6048 > > On Wed, Nov 14, 2018 at 5:16 PM, Udi Meiri

Re: Please do not merge Python PRs

All clear, Python tests are reporting errors correctly again. On Wed, Nov 14, 2018 at 5:57 PM Udi Meiri wrote: > https://github.com/apache/beam/pull/7048 is the rollback PR > > On Wed, Nov 14, 2018 at 5:28 PM Ahmet Altay wrote: > >> Thank you Udi. Could you send a rollback P

Re: Need help regarding memory leak issue

If you're working with Dataflow, it supports this flag: https://github.com/apache/beam/blob/75e9f645c7bec940b87b93f416823b020e4c5f69/sdks/python/apache_beam/options/pipeline_options.py#L602 which uses guppy for heap profiling. On Fri, Nov 16, 2018 at 3:08 PM Ruoyun Huang wrote: > Even tough the

Re: [VOTE] Release 2.9.0, release candidate #1

I see a regression for query 7 spark runner batch mode on about 2018-11-13. [image: image.png] On Thu, Dec 6, 2018 at 2:46 AM Chamikara Jayalath wrote: > Hi everyone, > > Please rev

Re: [VOTE] Release 2.9.0, release candidate #1

For DirectRunner there are regressions in query 7 sql direct runner batch mode <https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424&widget=732741424&container=411089194> (2x) and streaming mode (5x). On Thu, Dec 6, 2018 at 5:59 PM Udi Meiri wrote: > I

Java performance tests dashboard

Hi Lukasz, I was looking for statistics on I/O performance for writes of many files (~10k) on GCS. I found this dashboard and I have some questions. 1. The tests that are "local filesystem" seem to be running on Dataflow

Re: Issue with publishing maven artefacts locally

On Wed, Dec 12, 2018 at 11:00 AM Scott Wegner wrote: > Thanks for pointing this out Alexy. This seems like we unintentionally > broke something in PR#7197 [1] > > +Garrett Jones , who authored the change. > Garrett can you help investigate? > > I went to check to see if we have any existing Jenki

excessive java precommit logging

Hi all, I'd like to reduce precommit log sizes on Jenkins. For example: https://builds.apache.org/job/beam_PreCommit_Java_Commit/3181/consoleFull is 79M, which makes Chrome sluggish to use on it (tab is constantly using a whole cpu core). I know this might be controversial, but I'd like to propose

Re: excessive java precommit logging

to be much easier on Chrome >> 3. Download the consoleText output locally and use your favorite log >> viewer that can scale to large files. >> >> [1] https://gradle.com/s/ckhjrjdexpuzm >> >> On Wed, Dec 19, 2018 at 10:42 AM Udi Meiri wrote: >> >>>

Re: Add code quality checks to pre-commits.

+1 for adding more code quality signals. Could we add them in an advisory-only mode at first? (a warning and not an error) I'm curious how the "technical debt" metric is determined. I'm not familiar with SonarQube. What languages does it support? On Thu, Jan 3, 2019 at 10:19 AM Mikhail Gryzykhin

Re: workspace cleanups needed on jenkins master

On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía wrote: > Bringing this subject for awareness to dev@ > We are sadly part of this top. > Does somebody know what this data is? And if we can clean it periodically? > Can somebody with more sysadmin super powers take a look and act on this. > >

Re: workspace cleanups needed on jenkins master

old Can you confirm? > > --Mikhail > > Have feedback <http://go/migryz-feedback>? > > > On Thu, Jan 3, 2019 at 1:59 PM Udi Meiri wrote: > >> >> >> On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía wrote: >> >>> Bringing this subject for aw

Re: excessive java precommit logging

1) curl /consoleText | less >> > >> > This is when I just want to quickly look for something. >> > >> > 2) curl /consoleText > log.txt >> > less log.txt >> > >> > Here we store the log to a file first, then use 'less' or 

Re: Enforce javadoc comments in public methods?

+1 On Mon, Jan 7, 2019 at 4:49 PM Daniel Oliveira wrote: > +1 > > I like this idea, especially with the line number requirement. The exact > number of lines is debatable, but you could go as low as 10 lines and that > would exclude any trivial setters and getters. Even better might be if it's >

TestDirectRunner for Java?

Hi, I want to use DirectRunner for a new IT I'm writing, since it's testing I/O code that's runner agnostic. The problem is that DirectRunner doesn't have a TestDataflowRunner analog, so features like OnSuccessMatcher aren't available. Any objections to adding a TestDirectRunner class? smime.p7s

Re: TestDirectRunner for Java?

t; the use case was when that was introduced, versus waiting for termination > and running the assertion more directly. Can you enlighten me? > > Kenn > > On Tue, Jan 15, 2019 at 4:15 PM Udi Meiri wrote: > >> Hi, >> I want to use DirectRunner for a new IT I'm writ

Adding KMS support to generic filesystem interface

Hi, I'd like to add support for creating files using a cloud Key Management System. A KMS allows you to audit, create, rotate, and disable encryption keys. Both AWS and GCP have such a service.. I wanted to show the community what I've been working on and see if there are any comments or objection

Re: Adding KMS support to generic filesystem interface

until next > monday. I hope waiting a bit is ok with you if you want some feedback > from me. > > I am curious if you considered or are aware of this issue: > BEAM-3821 Support a pluggable key management system (KMS) > https://issues.apache.org/jira/browse/BEAM-3821 > >

Re: gradle clean causes long-running python installs

setup.py has requirements like setuptools, which are installed in the virtual environment. So even running the clean command requires the virtualenv to be set up. A possible fix could be to skip :beam-sdks-python:cleanPython if setupVirtualenv has not been run. (perhaps by checking for the existen

Re: gradle clean causes long-running python installs

s, etc? > > Kenn > > On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri wrote: > >> setup.py has requirements like setuptools, which are installed in the >> virtual environment. >> So even running the clean command requires the virtualenv to be set up. >> >> A p

Confluence wiki edit access request

username: udim Thanks! smime.p7s Description: S/MIME Cryptographic Signature

Re: Cross-language pipelines

Also debugability: collecting logs from each of these systems. On Tue, Jan 22, 2019 at 10:53 AM Chamikara Jayalath wrote: > Thanks Robert. > > On Tue, Jan 22, 2019 at 4:39 AM Robert Bradshaw > wrote: > >> Now that we have the FnAPI, I started playing around with support for >> cross-language pi

Re: Confluence wiki edit access request

bump On Fri, Jan 18, 2019 at 1:57 PM Udi Meiri wrote: > username: udim > > Thanks! > smime.p7s Description: S/MIME Cryptographic Signature

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

Some options: - You could wait to assert until after p.waitForFinish(). - You could PAssert using SerializableMatcher and allow any lastModifiedTime. On Tue, Jan 22, 2019 at 3:56 PM Alex Amato wrote: > +Jeff, Eugene, > > Hi Jeff and Eugene, > > I've noticed that Jeff's PR >

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

gt; looking at for reference? >>> >>>- I guess to this you need to implement the SerializableMatcher >>>interface and use the matcher as an option in the pipeline options. >>> >>> >>> On Tue, Jan 22, 2019 at 4:28 PM Udi Meiri wrote: &

Re: Confluence wiki edit access request

Thank you! On Wed, Jan 23, 2019 at 12:49 PM Ismaël Mejía wrote: > Done. > > On Tue, Jan 22, 2019 at 8:53 PM Udi Meiri wrote: > > > > bump > > > > On Fri, Jan 18, 2019 at 1:57 PM Udi Meiri wrote: > >> > >> username: udim > >>

Re: Findbugs -> Spotbugs ?

+1 for spotbugs On Thu, Jan 31, 2019 at 5:09 AM Gleb Kanterov wrote: > Agree, spotbugs brings static checks that aren't covered in error-prone, > it's a good addition. There are few conflicts between error-prone and > spotbugs, for instance, the approach to enum switch exhaustiveness, but it > c

Re: Jenkins slowness

Precommits times for Python and Java have been slowly climbing: http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1&from=1546974237153&to=1549566237153&theme=light On Thu, Feb 7, 2019 at 10:54 AM Udi Meiri wrote: > If anyone has done any investigation/is working

Jenkins slowness

If anyone has done any investigation/is working on this please share. I'm investigating Jenkins slowness. I've noticed it happening since yesterday: precommits taking 3 hours to start, phrase commands similarly taking as much time to register. My current theory is that we have a job that's are ta

Re: Jenkins slowness

There is also excessive python test logging tracked here: https://issues.apache.org/jira/browse/BEAM-6603 On Thu, Feb 7, 2019, 12:23 Udi Meiri I suggest disabling Jacoco and re-enabling the build cache until we can > migrate to Gradle 5. I imagine the migration to v5 is not a simple cha

Re: Jenkins slowness

eems to be lots of work done now which >> previously was cacheable. >> >> Not sure, whether this is the culprit- or part of it -, but I d suggest >> to upgrade to gradle 5 pretty fast. >> >> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri wrote: >> >>> If an

Re: Jenkins slowness

d only be a merge of an pending + one additional PR upgrading the > versions. > > On 7. Feb 2019, at 21:23, Udi Meiri wrote: > > I suggest disabling Jacoco and re-enabling the build cache until we can > migrate to Gradle 5. I imagine the migration to v5 is not a simple change. &

Re: Signing off

Good luck Scott! On Fri, Feb 15, 2019 at 9:32 AM Alex Amato wrote: > Thanks's for your contributions Scott. We will miss you. > > On Fri, Feb 15, 2019 at 7:08 AM Etienne Chauchot > wrote: > >> Thank you for your contributions Scott ! Your new project seems very fun. >> Enjoy ! >> >> Etienne >>

Re: gradle clean causes long-running python installs

t; >>>>> Can we just limit clean to do cleanup?! >>>>> >>>>> >>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw >>>>> wrote: >>>>> >>>>>> Just some background, grpcio-tools is what's use

Re: gradle clean causes long-running python installs

ld? In case of the latter, clean >> isn't called anyway, right? >> >> If the former, its probably worth to put more effort into python setup >> anyway, e.g. declaring proper input/outputs... If done correctly (if thats >> possible at all) clean will work out of th

Re: What quick command to catch common issues before pushing a python PR?

Talking about Python: I only know of "./gradlew lint", which include style and some py3 compliance checking. There is no auto-fix like spotlessApply AFAIK. As a side-note, I really dislike our python line continuation indent rule, since pycharm can't be configured to adhere to it and I find myself

precommit tests: please ignore Python_pytest

Hi all, I'm testing running Python tests using pytest, and I've added a temporary Jenkins jobs that seems to be triggering for PRs, even though I set triggerPathPatterns to an empty list. (file ) Please igno

Re: precommit tests: please ignore Python_pytest

figured that >> it might not be fully-functional yet. >> >> It would be nice if there was a way to enable those hooks only for a >> testing PR, such that they do not interfere with other PRs. >> >> Perhaps somebody has an idea how to do that? >> >> -

Re: Added a Jira beginner's guide to the wiki.

My favorite way to navigate JIRA is using a Chrome search engine. You configure it like this: [image: Screenshot from 2019-02-27 17-11-26.png] (URL is: https://issues.apache.org/jira/secure/QuickSearch.jspa?searchString=%s) And search by writing in the location bar: "j BEAM-1234" will take you to

Re: Virtualenv setup issues on new machine

I think gradle is complaining that the path can't be found. Is there more information if you run it with --info? On Thu, Feb 28, 2019, 14:35 Ankur Goenka wrote: > Hi Beamers, > > I am trying build python sdk from a fresh git checkout on a new linux > machine but the setupVirtualEnv task is faili

Re: Virtualenv setup issues on new machine

Weird. Is that a known bug? On Thu, Feb 28, 2019 at 3:19 PM Ankur Goenka wrote: > The issue seems to be with "." in the virtualenv path. > virtualenv works after moving from > "/usr/local/google/home/goenka/.local/bin" to "/usr/bin" > > On Thu, Feb

Re: [BEAM-6761] Pydoc is giving cryptic error messages, blocking my PR :(

I think it's referring to the big comment at the top of the sdks/python/apache_beam/testing/metric_result_matchers.py. The line numbers are relative to the beginning of the block. On Fri, Mar 1, 2019 at 2:21 PM Alex Amato wrote: > BEAM-6761

Re: [BEAM-6761] Pydoc is giving cryptic error messages, blocking my PR :(

42 ) ] errors = metric_result_matchers.verify_all(all_metrics, matchers) self.assertFalse(errors, errors) """ https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html On Fri, Mar 1, 2019 at 3:31 PM Udi Meiri wrote: > I think it's referring to the big c

Re: Python Datastore client upgrade plan

ll, this is a good plan to get us out of a tough situation with an >>> old dependency. >>> >>> On Tue, Oct 16, 2018 at 6:59 PM, Udi Meiri wrote: >>> >>>> Hi, >>>> Sadly upgrading googledatastore -> google-cloud-datastore is >&g

Re: Selectively running tests?

Do you have an example PR Pablo? On Mon, Mar 18, 2019, 18:23 Alan Myrvold wrote: > The includedRegions was set up as part of > https://issues.apache.org/jira/browse/BEAM-4445 and there are additional > paths added from > https://github.com/apache/beam/blob/6bb4b2332b11bd8295ac6965be8426b9c38fa4

Re: Selectively running tests?

are more extensive for Java worker > changes. I noticed that my change runs Go, Python and Java precommits: > https://github.com/apache/beam/pull/8080 > > On Tue, Mar 19, 2019 at 10:27 AM Udi Meiri wrote: > >> Do you have an example PR Pablo? >> >> >>

Codecov Bash Uploader Security Notice

I got this email: https://about.codecov.io/security-update/ This is where we use codecov: https://github.com/apache/beam/blob/39923d8f843ecfd3d89443dccc359c14aea8f26f/sdks/python/tox.ini#L105 I'm not sure if this runs the "bash uploader", but we do set a CODECOV_TOKEN environment variable. smim

Re: Codecov Bash Uploader Security Notice

>From the notice: "We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders." On Thu, Apr 15, 2021 at 11:35 AM Udi Meiri wrote: >

Re: BEAM-3713: Moving from nose to pytest

r('ValidatesRunner') -> @pytest.mark.it_validatesrunner On Thu, Mar 25, 2021 at 10:45 AM Udi Meiri wrote: > Hi Benjamin, > > AFAIK nose is only used for integration tests (unit tests were converted > to pytest a while back). > These ITs should all be running periodi

Re: One Pager - Test Command Line Discoverability in Beam

My first place to go would be here: https://cwiki.apache.org/confluence/display/BEAM/Java+Tips (although it doesn't document your use-case) You are right that finding the correct gradle task or jenkins job is not straightforward. On Tue, May 25, 2021 at 12:48 PM Alex Amato wrote: > Friendly pi

Re: [PROPOSAL] Preparing for Beam 2.33.0 Release

t; See my comment below. > > On 26 Aug 2021, at 03:44, Ahmet Altay wrote: > > On Wed, Aug 25, 2021 at 1:50 PM Udi Meiri wrote: > >> I've found 2 performance regressions in IO. Could someone triage these? >> > > >> https://issues.apache.org/jira/brows

[VOTE] Release 2.33.0, release candidate 1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.33.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

[RESULT] [VOTE] Release 2.33.0, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 8 approving votes, 4 of which are binding: * Ahmet Altay * Alexey Romanenko * Robert Bradshaw * Chamikara Jayalath There are no disapproving votes. Thanks everyone!

Re: [RESULT] [VOTE] Release 2.33.0, release candidate #1

I spoke too soon. We will be doing an rc2 On Mon, Sep 27, 2021 at 1:29 PM Udi Meiri wrote: > I'm happy to announce that we have unanimously approved this release. > > There are 8 approving votes, 4 of which are binding: > * Ahmet Altay > * Alexey Romanenko > * Robert

[VOTE] Release 2.33.0, release candidate 2

Hi everyone, Please review and vote on the release candidate #2 for the version 2.33.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

[RESULT] [VOTE] Release 2.33.0, release candidate #2

I'm happy to announce that we have unanimously approved this release. There are 8 approving votes, 5 of which are binding: * Ahmet Altay * Alexey Romanenko * Robert Bradshaw * Kenneth Knowles * Chamikara Jayalath There are no disapproving votes. Thanks everyone!

[ANNOUNCE] Beam 2.33.0 Release

The Apache Beam team is pleased to announce the release of version 2.33.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release her

Re: NOTICE: New Python PreCommit jobs

I have a WiP PR to convert Beam to use pytest, but it's been stalled. The nice thing about pytest-xdist is that it runs tests in a multi-process, single-thread-per-process fashion, so one test isn't affected by another changing some global setting. The not-so-nice thing is that xdist adds some glob

Re: NOTICE: New Python PreCommit jobs

On Fri, Oct 4, 2019 at 10:35 AM Chad Dombrova wrote: > > I have a WiP PR to convert Beam to use pytest, but it's been stalled. >> > > What would it take to get it back on track? > Besides needing to convert ITs (removing save_main_session), which can be split out to a later PR, there's verifying

Re: Install Jenkins AnsiColor plugin

Seems to be already installed: https://issues.apache.org/jira/browse/INFRA-16944 Do we just need to enable it somehow? This might work: https://jenkinsci.github.io/job-dsl-plugin/#method/javaposse.jobdsl.dsl.helpers.wrapper.WrapperContext.colorizeOutput BTW, our Jenkins is maintained by ASF's Infr

Re: Install Jenkins AnsiColor plugin

r changes related to this. > > > > On Tue, Oct 22, 2019 at 10:45 AM Yifan Zou wrote: > >> Thanks, Udi! The ansicolor plugin was applied to ASF Jenkins universally. >> You might need to explicitly enable the coloroutput in your jenkins dsl. >> >> On Tue,

Re: Install Jenkins AnsiColor plugin

run seed job > > Does anyone have any concerns with me trying this out now? > > -chad > > On Tue, Oct 22, 2019 at 11:42 AM Udi Meiri wrote: > >> Also note that changing the job DSL doesn't take effect until the "seed" >> job runs. (use the "run s

Re: Is there good way to make Python SDK docs draft accessible?

I believe that generating pydoc for the website is still a manual process (unlike the rest of the website?). The reviewer will need to manually generate the docs (checkout the PR, run tox -e docs). On Mon, Oct 28, 2019 at 10:55 AM Yoshiki Obata wrote: > Hi all. > > I'm working on enabling to gen

Re: Python Precommit duration pushing 2 hours

> > > > On Tue, Oct 29, 2019, 6:46 AM Kenneth Knowles wrote: > >> > >> Noting for the benefit of the thread archive in case someone goes > digging and wonders if this affects other SDKs: the Java SDK memoizes > DoFnSignatures and generated DoFnInvoker classes.

Re: RFC: python static typing PR

Thank you for this monumental work Chad. I've added my review comments, though I haven't had time to look at all the files. Do you believe that a future mypy plugin could replace pipeline type checks in Beam, or are there limits to what it can do? On Wed, Oct 30, 2019 at 10:25 AM Chad Dombrova wr

Re: Python Precommit duration pushing 2 hours

without cython > or GCP dependencies. This could reduce the precommit time by ~30 minutes. > > > > What do you think? > > > > Ahmet > > > > > > On Tue, Oct 29, 2019 at 11:15 AM Robert Bradshaw > wrote: > >> > >> https://github.com/apache

Cython unit test suites running without Cythonized sources

I opened this bug today after commenting on Chad's type hints PR. https://issues.apache.org/jira/browse/BEAM-8572?filter=-1 I am 95% sure that our Precommit tests are using tarballs that are built without Cython (including the Cython

Re: [Discuss] Beam mascot

Dumbo octopus anyone? https://youtu.be/DmqikqvLLLw?t=263 On Mon, Nov 11, 2019 at 2:06 PM Luke Cwik wrote: > The real answer, what cool schwag can we get based upon the mascot. > > On Mon, Nov 11, 2019 at 2:04 PM Kenneth Knowles wrote: > >> I'm with Luke on cuttlefish. We can have color changin

Re: [Discuss] Beam Summit 2020 Dates & locations

+1 for better organization. I would have gone to ApacheCon LV had I known there was going to be a Beam summit there. On Tue, Nov 12, 2019 at 9:31 AM Alexey Romanenko wrote: > On 8 Nov 2019, at 11:32, Maximilian Michels wrote: > > > > The dates sounds good to me. I agree that the bay area has an

Re: [ANNOUNCE] New committer: Brian Hulette

Congrats Brian! On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang wrote: > Congrats Brian! > > On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu wrote: > >> Congrats, Brian! >> >> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira >> wrote: >> >>> Congratulations Brian! It's well deserved. >>> >>> On Fri,

Re: [discuss] Using a logger hierarchy in Python

+1, but can we use something less verbose and shift key heavy than _LOGGER like log or _log? Also please dedupe with these existing bugs: https://issues.apache.org/jira/browse/BEAM-3523 https://issues.apache.org/jira/browse/BEAM-1825 On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: > Awesome

[PROPOSAL] Preparing for Beam 2.18 release

Hi all, The next (2.18) release branch cut is scheduled for Dec 4, according to the calendar . I would like to volunteer myself to do this release. The plan is to cut the branch on that date, an

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congrats Daniel! On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver wrote: > Congrats Dan! Keep up the good work :) > > On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden wrote: > >> Congratulations! This is great news. >> >> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang wrote: >> >>> Congrats! >>> >>> >>> -Ru

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [x] Hedgehog [x] Lemur [x] Owl [ ] Salmon [ ] Trout [x] Robot dinosaur [x] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish On Wed, Nov 20, 2019 at 12:05 PM Eduardo Soldera < eduardo.sold...@arquivei.com.br> wrote: > [X ] Beaver > [] Hedgehog > [ ] Lemur > [ ] Owl > [ ] Salmon

Re: [PROPOSAL] Preparing for Beam 2.18 release

community >> and/or release manager may decide to make some of these blockers if needed. >> I don't think the tool automatically makes the auto generated JIRAs >> release blockers. >> >> Thanks, >> Cham >> >> >>> [1]: >>> https:

Re: Python tests and setup

I've been looking at using pyenv in the past few days. I've also had a broken 3.6.8 install on my machine for a while now, but using pyenv I managed to install 3.6.9 with little difficulty: pyenv install 3.6.9 PYENV_VERSION=3.6.9 ./gradlew :sdks:python:test-suites:tox:py36:setupVirtualenv (You can

Re: cython test instability

I managed to recreate one of the issues with this command: ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do echo "python setup.py -q nosetests --tests apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh ; done This reliably gives me: OSError: [Errno 17] File exists:

Re: cython test instability

Correction: the error is not gone after removing the line. I get instead: error: [Errno 17] File exists: '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg' On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri wrote: > I managed to recreate one of the issues with

Re: cython test instability

Basically, I believe what's happening is that a new Gradle task was added that uses setup.py but doesn't have the same dependency on some main setup.py task that all others depend on (list sdist). On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri wrote: > Correction: the error is n

Re: cython test instability

ose that only have source > distributions (i.e. no wheels). But why are the eggs being installed in > sdks/python/.eggs instead of into the virtualenv created by setupVirtualenv > gradle task or by tox? > > > On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri wrote: > >> Basica

Re: cython test instability

equries=['pytest-runner'] was present -- it's in a gradle traceback > now rather than the console log. I've also seen different packages listed > as the culprit (simplegeneric, timeloop). > > -chad > > > > On Tue, Nov 26, 2019 at 4:47 PM Udi Meiri wrote: &

Re: cython test instability

y test", but the alternative is a simple "setup.py && pip install". On Tue, Nov 26, 2019 at 5:14 PM Chad Dombrova wrote: > Sorry wrong link: https://github.com/apache/beam/pull/9915 > > > > On Tue, Nov 26, 2019 at 5:12 PM Udi Meiri wrote: > >> I looke

Python staging file weirdness

I was investigating a Dataflow postcommit test failure (endpoints_pb2 missing), and saw this in the staging directory: $ gsutil ls gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1126202146-314738.1574799706.314882 gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkin

Python interactive runner: test dependencies removed

As part of a move to stop using the deprecated (and racey) setup.py keywords setup_requires and test_require, interactive runner dependencies have been removed from tests in https://github.com/apache/beam/pull/10227 If this breaks any tests, please let me know. smime.p7s Description: S/MIME Cryp

[RELEASE] Tracking 2.18

Following the release calendar, I plan on cutting the 2.18 release branch today. There are currently 8 release blockers . smime.p7s Description: S/MIME Cryptographic Signature

<    1   2   3   4   >