Re: [DISCUSS] Dependency management for Python SDK Container

2019-08-06 Thread Ahmet Altay
On Mon, Aug 5, 2019 at 9:49 PM Valentyn Tymofieiev wrote: > On Tue, Aug 6, 2019 at 2:29 AM Ahmet Altay wrote: > >> >> >> On Mon, Aug 5, 2019 at 1:43 AM Valentyn Tymofieiev >> wrote: >> >>> - The purpose of install_requires in setup.py

Re: [DISCUSS] Dependency management for Python SDK Container

2019-08-05 Thread Ahmet Altay
sure that files are changed at the same time. > Valentyn > > [1] > https://packaging.python.org/discussions/install-requires-vs-requirements/ > [2] https://pip.pypa.io/en/stable/user_guide/#requirements-files > > On Sat, Aug 3, 2019 at 2:47 AM Ahmet Altay wrote: > >> >&g

Re: beam_PreCommit_Java_Commit is broken

2019-08-05 Thread Ahmet Altay
This is tracked here: https://issues.apache.org/jira/browse/BEAM-7892 /cc +Valentyn Tymofieiev +Udi Meiri On Mon, Aug 5, 2019 at 9:18 AM Alexey Romanenko wrote: > According to "git bisect”, it seems like this was introduced with commit > “149153b525236327badb138b09235ff735045adf” in PR >

Re: Support ZetaSQL as a new SQL dialect in BeamSQL

2019-08-02 Thread Ahmet Altay
Thank you Rui for the heads up. A question to the community, does the size of the change require any process besides the usual PR reviews? On Fri, Aug 2, 2019 at 10:23 AM Rui Wang wrote: > Hi community, > > I have been working on supporting ZetaSQL[1] as a SQL dialect in BeamSQL. > ZetaSQL is

Re: [BEAM-7819] -python - parsing message_id from PubSub message to the PubSubMessage attributes property

2019-08-02 Thread Ahmet Altay
directrunner this is not the case with the > dataflow runner; but I'm not receiving any errors in the debug stream on my > console, only a series of info around the worker configuration. > > This may well be me doing stuff wrong, so apologies if I'm being thick > here! Have I missed som

Re: [PROPOSAL] An initial Schema API in Python

2019-08-02 Thread Ahmet Altay
typing to indicate that a function accepts a type of int, would it be compatible with numpy types? > >> [1] >> https://docs.google.com/a/google.com/document/d/1uu9pJktzT_O3DxGd1-Q2op4nRk4HekIZbzi-0oTAips/edit?disco=CtLItNA >> [2] https://docs.python.org/3/library/stdtyp

Re: [DISCUSS] Dependency management for Python SDK Container

2019-08-02 Thread Ahmet Altay
On Fri, Aug 2, 2019 at 4:34 PM Brian Hulette wrote: > Thanks for the reply, I added some responses inline. > > On Fri, Aug 2, 2019 at 2:42 PM Ahmet Altay wrote: > > > > There is a value in explicitly pinning the dependencies to be used in > the containers: > &

Re: [PROPOSAL] An initial Schema API in Python

2019-08-02 Thread Ahmet Altay
Thank you Brian. I did not spend enough time yet to review. Some early questions, I apologize if I missed an earlier discussion. - Do we need to support python 2? If supporting python 2 will complicate things, we could make this a python3 only feature. - Why are we mapping to numpy types? Design

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Ahmet Altay
>>>> > >>> >>>> > >>> I installed from source, and ran unit tests for Python in 2.7, >>>> 3.5, 3.6. >>>> > >>> >>>> > >>> Also ran a number of integration tests on Py 3.5 on Dataflow

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Ahmet Altay
Congratulations Jan! Thank you for your contributions! On Wed, Jul 31, 2019 at 2:30 AM Ankur Goenka wrote: > Congratulations Jan! > > On Wed, Jul 31, 2019, 1:23 AM David Morávek wrote: > >> Congratulations Jan, well deserved! ;) >> >> D. >> >> On Wed, Jul 31, 2019 at 10:17 AM Ryan Skraba

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-29 Thread Ahmet Altay
+1, validated python 2 quickstarts. On Fri, Jul 26, 2019 at 5:46 PM Ahmet Altay wrote: > To confirm, I manuall validated leader board on python. It is working. > > On Fri, Jul 26, 2019 at 5:23 PM Yifan Zou wrote: > >> AFAIK, there should not be any special prerequisite

Re: [BEAM-7819] -python - parsing message_id from PubSub message to the PubSubMessage attributes property

2019-07-29 Thread Ahmet Altay
Hi Matthew, This looks like a reasonable approach. There is a difference how direct runner reads from pubsub compared to other runners. As you convert to a PR, please pay attention to the difference and add tests for both cases. On Mon, Jul 29, 2019 at 8:35 AM Matthew Darwin <

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-26 Thread Ahmet Altay
To confirm, I manuall validated leader board on python. It is working. On Fri, Jul 26, 2019 at 5:23 PM Yifan Zou wrote: > AFAIK, there should not be any special prerequisites for this. Things the > script does including: > 1. download the python rc in zip > 2. start virtualenv and install the

Re: contributor permission for Beam Jira tickets

2019-07-24 Thread Ahmet Altay
Welcome. I added ningk@ as a JIRA contributor. On Wed, Jul 24, 2019 at 4:19 PM Ning Kang wrote: > Bump this thread as a friendly ping! > > On Tue, Jul 16, 2019 at 5:08 PM Ning Kang wrote: > >> Hi, >> >> This is Ning Kang from Google. I'm working on the interactive beam >>

Re: An update on Python postcommit tests

2019-07-19 Thread Ahmet Altay
Thank you! Hopefully this will improve the state of testing. On Fri, Jul 19, 2019 at 2:53 PM Valentyn Tymofieiev wrote: > We have split Python 2, Python 3.5 - 3.7 postcommit test suites into > individual Jenkins jobs that can be triggered with a phrase of their own. > > "Run Python PostCommit"

Re: [PROPOSAL] Preparing for Beam 2.15.0 release

2019-07-17 Thread Ahmet Altay
+1 Thank you for keeping the cadence! On Wed, Jul 17, 2019 at 2:00 PM Rui Wang wrote: > +1. Thanks Yifan to take it over! > > > Rui > > On Wed, Jul 17, 2019 at 1:56 PM Alan Myrvold wrote: > >> +1 Thanks for keeping the release cadence going. I like to see regular >> releases happening. >> >>

[ANNOUNCE] New committer: Robert Burke

2019-07-16 Thread Ahmet Altay
Hi, Please join me and the rest of the Beam PMC in welcoming a new committer: Robert Burke. Robert has been contributing to Beam and actively involved in the community for over a year. He has been actively working on Go SDK, helping users, and making it easier for others to contribute [1]. In

Re: [2.14.0] Release Progress Update

2019-07-15 Thread Ahmet Altay
Anton, any updates on this release? Do you need help? On Fri, Jun 28, 2019 at 11:42 AM Anton Kedin wrote: > I have been running validation builds (had some hickups with that), > everything looks mostly good, except failures in `:beam-test-tools` and > `:io:aws`. Now I will start cherry-picking

Re: Blogpost Beam Summit 2019

2019-06-25 Thread Ahmet Altay
Thank you for writing and sharing this. I enjoyed reading it :) I think it is worth sharing it as a tweet [1] as well. [1] s.apache.org/beam-tweets On Tue, Jun 25, 2019 at 10:16 AM Valentyn Tymofieiev wrote: > Hi Juta, > > Thanks for sharing! You can also consider sending it to user mailing

Re: python integration tests flake detection

2019-06-25 Thread Ahmet Altay
There is a nose plugin [1] for outputting test results in xunit format. Would that work? [1] https://nose.readthedocs.io/en/latest/plugins/xunit.html On Tue, Jun 25, 2019 at 10:04 AM Udi Meiri wrote: > The current state of Python post-commit tests is pretty flaky. > I was wondering if we had

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

2019-06-21 Thread Ahmet Altay
Congratulations! On Fri, Jun 21, 2019 at 12:58 PM Ankur Goenka wrote: > Congrats Mikhail! > > On Fri, Jun 21, 2019 at 11:55 AM Tanay Tummalapalli > wrote: > >> Congratulations! >> >> On Fri, Jun 21, 2019 at 10:35 PM Rui Wang wrote: >> >>> Congrats! >>> >>> >>> -Rui >>> >>> On Fri, Jun 21,

Re: Discussion/Proposal: support Sort Merge Bucket joins in Beam

2019-06-21 Thread Ahmet Altay
Thank you Claire, this looks promising. Explicitly adding a few folks that might have feedback: +Ismaël Mejía +Robert Bradshaw +Lukasz Cwik +Chamikara Jayalath On Mon, Jun 17, 2019 at 2:12 PM Claire McGinty wrote: > Hey dev@! > > Myself and a few other Spotify data engineers have put

Re: [VOTE] Release vendored artifacts upgrading Guava usage to 26.0-jre, release candidate #1

2019-06-20 Thread Ahmet Altay
d.html/9efb2aeab102e41367bf6b1f274d3ee5990024afd934392a339c4d00@%3Cdev.beam.apache.org%3E > > On Thu, Jun 20, 2019 at 11:20 AM Ahmet Altay wrote: > >> What is the best way to validate this? >> >> On Thu, Jun 20, 2019 at 9:51 AM Lukasz Cwik wrote: >> >>> Hi everyone, &g

Re: [VOTE] Release vendored artifacts upgrading Guava usage to 26.0-jre, release candidate #1

2019-06-20 Thread Ahmet Altay
What is the best way to validate this? On Thu, Jun 20, 2019 at 9:51 AM Lukasz Cwik wrote: > Hi everyone, > > Please review the release of the following artifacts that we vendor: > beam-vendor-guava-26_0-jre > beam-vendor-grpc-1_21_0 > > Please vote as follows: > [ ] +1, Approve the release > [

Re: Plan for dropping python 2 support

2019-06-18 Thread Ahmet Altay
. >> >> The cost of maintaining Python 2.7 support is higher than 0. Some issues >> that come to mind: >> - Maintaining Py2.7 / Py 3+ compatibility of Beam codebase makes it >> difficult to use Python 3 syntax in Beam which may be necessary to support >> and test

Re: Python dependency compatibility badges

2019-06-12 Thread Ahmet Altay
Looks like a nice improvement to me. To make it very explicit, it seems to focus on compatibility issues with google managed libraries even though the reports identify general old dependencies as well. On Wed, Jun 12, 2019 at 6:19 AM Ismaël Mejía wrote: > +1 looks nice and can motivate people

Re: JIRA Permission Request

2019-06-11 Thread Ahmet Altay
Hi Charith, I added you as a contributor.

Re: Beam's Conda package

2019-06-11 Thread Ahmet Altay
maintainers to the package. In the meantime, I will try to keep the conda package up-to-date. On Fri, May 10, 2019 at 1:18 PM Ahmet Altay wrote: > https://github.com/sodre seems to be the person behind it. Does anybody > know who is that person? > > *From: *Charles Chen > *Date: *Fr

Re: Plan for dropping python 2 support

2019-06-07 Thread Ahmet Altay
(very) unlikely beyond 2020. > > On Wed, Jun 5, 2019 at 6:34 PM Ahmet Altay wrote: > > > > I agree with the sentiment on this thread. Our priority needs to be > offering good python 3 support that we can comfortably recommend users to > switch. Progress on that so far has been

Re: [PROPOSAL] Preparing for Beam 2.14.0 release

2019-06-06 Thread Ahmet Altay
+1, thank you for keeping the cadence. On Thu, Jun 6, 2019 at 9:04 AM Anton Kedin wrote: > Hello Beam community! > > Beam 2.14 release branch cut date is June 19 according to the release > calendar [1]. I would like to volunteer myself to do this release. The plan > is to cut the branch on that

Re: Plan for dropping python 2 support

2019-06-05 Thread Ahmet Altay
I agree with the sentiment on this thread. Our priority needs to be offering good python 3 support that we can comfortably recommend users to switch. Progress on that so far has been promising and I do anticipate that we will reach there in the near future. My proposal would be, once we reach to

Re: [VOTE] Release 2.13.0, release candidate #2

2019-06-05 Thread Ahmet Altay
20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.13.0 > > On Wed, Jun 5, 2019 at 4:23 PM Chamikara Jayalath > wrote: > > > > > > > > On Tue, Jun 4, 2019 at 5:02 PM Ahmet Altay wrote: > >> > &g

Re: [VOTE] Release 2.13.0, release candidate #2

2019-06-04 Thread Ahmet Altay
ad requested to include following backport PR before the RC: >>>> https://github.com/apache/beam/pull/8714 - It's not blocking but >>>> would be nice if someone can merge it for any future release from this >>>> branch. >>>> >>>> Thanks, >

Re: Jira tracker permission

2019-06-04 Thread Ahmet Altay
Welcome! On Mon, Jun 3, 2019 at 10:31 PM Pablo Estrada wrote: > I've added you as contributor - welcome > -P. > > On Mon, Jun 3, 2019, 9:16 PM Yichi Zhang wrote: > >> Hi, beam-dev, >> >> This is Yichi Zhang from Google, I just started looking into beam >> projects and will be actively working

Re: [DISCUSS] Cookbooks for users with knowledge in other frameworks

2019-06-03 Thread Ahmet Altay
s, would be most useful I believe. > > > > On Sat, Jun 1, 2019 at 4:31 PM Reza Rokni wrote: > >> For layer 1, what about working through this link as a starting point : >> https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations >> ? >> > +1

Re: BQ IT tests fail on TestDataflowRunner - Python SDK

2019-06-03 Thread Ahmet Altay
Do you have any other changes? Are you trying from head with a clean virtual environment? If you can share a link to dataflow job (in the apache-beam-testing GCP project), we can try to look at additional logs as well. On Mon, Jun 3, 2019 at 1:42 PM Tanay Tummalapalli wrote: > Hi everyone, > >

Re: [DISCUSS] Cookbooks for users with knowledge in other frameworks

2019-05-31 Thread Ahmet Altay
ettle on a version that is agreeable. (Examples have the same issue, they are probably opinionated today based on the author but it works out.) > > > On Thu, 30 May 2019 at 08:56, Ahmet Altay wrote: > >> Hi all, >> >> Inspired by the user asking about a Spark feature

Re: [VOTE] Release 2.13.0, release candidate #2

2019-05-31 Thread Ahmet Altay
+1 I validated python 2 quickstarts. On Fri, May 31, 2019 at 10:22 AM Lukasz Cwik wrote: > I did the Java local quickstart for all the runners in the release > validation sheet and gearpump failed for me due to a missing dependency. > Even after I fixed up the dependency, the pipeline then got

[DISCUSS] Cookbooks for users with knowledge in other frameworks

2019-05-29 Thread Ahmet Altay
Hi all, Inspired by the user asking about a Spark feature in Beam [1] in the release thread, I searched the user@ list and noticed a few instances of people asking for question like "I can do X in Spark, how can I do that in Beam?" Would it make sense to add documentation to explain how certain

Re: [DISCUSS] Autoformat python code with Black

2019-05-29 Thread Ahmet Altay
file=sys.stderr) > > > > error: cannot format > > > > > /home/ismael/upstream/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py: > > > > Cannot parse: 160:34: print(traceback_string, file=sys.stderr) > > > > error: cannot format

Re: [VOTE] Release 2.13.0, release candidate #1

2019-05-29 Thread Ahmet Altay
We have a quite a bit of cherry pick requests. Are they all for major/blocking issues? Have we uncovered issues in release validation that is normally missing in our daily tests? On Wed, May 29, 2019 at 10:20 AM Thomas Weise wrote: > Added: https://github.com/apache/beam/pull/8714 > > > On Tue,

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-28 Thread Ahmet Altay
an 16, 2019 at 6:11 PM Ankur Goenka < > goe...@google.com> wrote: > >>> > >>>>>> > >>> > >>>>>> grc.io seems to be a good option. Given that we don't need > the hosting server name in the image name makes it easily c

Re: [DISCUSS] Autoformat python code with Black

2019-05-28 Thread Ahmet Altay
I am in the same boat with Robert, I am in favor of autoformatters but I am not familiar with this one. My concerns are: - The product is clearly marked as beta with a big warning. - It looks like mostly a single person project. For the same reason I also strongly prefer not using a fork for a

Re: DISCUSS: Sorted MapState API

2019-05-23 Thread Ahmet Altay
On Thu, May 23, 2019 at 1:38 PM Lukasz Cwik wrote: > > > On Thu, May 23, 2019 at 11:37 AM Rui Wang wrote: > >> A few obvious problems with this code: >>> 1. Removing the elements already processed from the bag requires >>> clearing and rewriting the entire bag. This is O(n^2) in the number of

Re: Proposal: Add permanent url to community metrics dashboard

2019-05-21 Thread Ahmet Altay
: > @Ahmet Altay > Thank you for the comment. > > Point on search engines is really good. If that happens we can look into > configuring robots.txt to notify search engines to ignore whole domain. > The link is a redirect to static IP. So it is still confusing. > > Havin

Re: Proposal: Add permanent url to community metrics dashboard

2019-05-20 Thread Ahmet Altay
Hi Mikhail, Thank you for your work on this. I have some comments: - There is already a short link (https://s.apache.org/beam-community-metrics). Would a link from contributing to beam page (if there is not one already) sufficient> People can bookmark the short link if they need to quickly

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

2019-05-15 Thread Ahmet Altay
Congratulations! On Tue, May 14, 2019 at 9:11 PM Robert Burke wrote: > Woohoo! Well deserved. > > On Tue, May 14, 2019, 8:34 PM Reuven Lax wrote: > >> Congratulations! >> >> *From: *Mikhail Gryzykhin >> *Date: *Tue, May 14, 2019 at 8:32 PM >> *To: * >> >> Congratulations Pablo! >>> >>> On

Re: [VOTE] Remove deprecated Java Reference Runner code from repository.

2019-05-14 Thread Ahmet Altay
+1 *From: *Lukasz Cwik *Date: *Tue, May 14, 2019 at 2:20 PM *To: *dev +1 > > *From: *Daniel Oliveira > *Date: *Tue, May 14, 2019 at 2:19 PM > *To: *dev > > Hello everyone, >> >> I'm calling for a vote on removing the deprecated Java Reference Runner >> code. The PR for the change has already

Re: Intro

2019-05-14 Thread Ahmet Altay
Welcome! Added you as a contributor to JIRA. *From: *Damien Desfontaines *Date: *Tue, May 14, 2019 at 1:24 PM *To: * Hi folks, > > I'm Damien from the Anonymization team at Google. I might contribute a > couple of PRs on the Go SDK. Can someone give me permission to assign Jira > tickets to

Re: Beam's Conda package

2019-05-10 Thread Ahmet Altay
ree/c96274713fcc5970c967c20e84859e73d0efa0d0 > > *From: *Lukasz Cwik > *Date: *Fri, May 10, 2019 at 1:02 PM > *To: *dev > > I'm not aware of who set up conda as well. There seem to have been ~4500 >> downloads of the package so that is a good amount of users. >> >> On Fri, May 10, 2019 at 11:

Beam's Conda package

2019-05-10 Thread Ahmet Altay
Hi all, There a conda package for apache-beam [1]. As far as I know, we do not release this package. Does anyone know who owns this? It was last updated to use 2.9.0, at least it would be good to add a newer version there. We also don't test in that environment so I am not sure how well it works

Re: Plans for Python type hints

2019-05-09 Thread Ahmet Altay
Thank you Udi. This is very helpful. I added a few comments. Overall proposed changes make sense to me. *From: *Udi Meiri *Date: *Wed, May 8, 2019 at 9:29 AM *To: * Hi, > I've written a document, with input from robertwb@, detailing the > direction forward I want to take type hints in Python

Re: [discuss] Reducing log verbosity for Python failures?

2019-05-08 Thread Ahmet Altay
], is that these are > logged in debug mode, but when the test fails, the test runner dumps them > all to stdout. > Best > -P. > > [1] > https://i1.wp.com/gifrific.com/wp-content/uploads/2015/02/Chris-Farley-Oh-God-Theyre-Gonna-Know-Im-Dumb-Conan-Interview.gif?ssl=1 > > *

Re: [discuss] Reducing log verbosity for Python failures?

2019-05-08 Thread Ahmet Altay
+1 It is hard to debug with lots logs messages. And if anybody is using them for development we can make those logs debug logs and hide them by default. *From: *Robert Bradshaw *Date: *Wed, May 8, 2019 at 3:01 PM *To: *dev +1 to making them significantly more compact in most cases. > > From:

Re: [DISCUSS] Backwards compatibility of @Experimental features

2019-05-08 Thread Ahmet Altay
uld >> become not experimental) I think we will hardly find an agreement so I >> think this should be treated in a per case basis by the maintainers, but if >> you want to follow up on that discussion we can open another thread for >> this. >> >> >> >> On Sa

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-05-07 Thread Ahmet Altay
*To: * Thanks Ahmet for the time you put into this. AFAIU Roberts proposal > resolves my concerns. > > On Mon, May 6, 2019 at 8:01 PM Ahmet Altay wrote: > >> Thank you Max. Michael, does the current state of the proposal address >> your concerns? >> >> *From: *Maxim

Re: Better naming for runner specific options

2019-05-06 Thread Ahmet Altay
;> > not namespaced. All PipelineOptions exist in a global namespace >>>>> so >>>>> > they need to be careful to be very precise. >>>>> > >>>>> > It is a good point that even though they may be multiple uses for >

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-05-06 Thread Ahmet Altay
with rest of their other dependencies and test that combination with ease. I agree it is not necessary for us or any other party that just wants to test beam packages and its direct dependencies in isolation. > > Thanks, > Max > > On 02.05.19 19:43, Ahmet Altay wrote: > > &

Re: Access to JIRA

2019-05-06 Thread Ahmet Altay
Welcome Frederik! *From: *Ismaël Mejía *Date: *Sat, May 4, 2019 at 2:31 PM *To: * It should work now, enjoy! > On Sat, May 4, 2019 at 7:58 PM Frederik Bode wrote: > >> >> Hello everyone! >> >> I'm working on porting beam to python 3. Could I please get access >> to JIRA? My JIRA username is

Re: Contributing to Beam

2019-05-06 Thread Ahmet Altay
Welcome Shehzaad! *From: *Ismaël Mejía *Date: *Sat, May 4, 2019 at 2:30 PM *To: * Done, welcome! > > On Sat, May 4, 2019 at 3:43 AM Shehzaad Nakhoda > wrote: > >> Hello >> >> I’m hoping to work with Rueven Lax (Google) on some enhancements and >> existing issues. >> >> I would appreciate the

Re: [ANNOUNCE] New committer announcement: Udi Meiri

2019-05-03 Thread Ahmet Altay
Congratulations, Udi! *From: *Kyle Weaver *Date: *Fri, May 3, 2019 at 2:11 PM *To: * Congratulations Udi! I look forward to sending you all my reviews for > the next month (just kidding :) > > Kyle Weaver | Software Engineer | github.com/ibzib | > kcwea...@google.com | +1650203 > > On Fri,

Re: Better naming for runner specific options

2019-05-03 Thread Ahmet Altay
I agree, that is a good point. *From: *Lukasz Cwik *Date: *Fri, May 3, 2019 at 9:37 AM *To: *dev The concept of a machine type isn't necessarily limited to Dataflow. If it > made sense for a runner, they could use AWS/Azure machine types as well. > > On Fri, May 3, 2019 at 9:32 AM Ah

Re: Better naming for runner specific options

2019-05-03 Thread Ahmet Altay
This idea was discussed in a PR a few months ago, and JIRA was filed as a follow up [1]. IMO, it makes sense to use a namespace prefix. The primary issue here is that, such a change will very likely be a backward incompatible change and would be hard to do before the next major version. [1]

Re: [DISCUSS][SQL] Providing support for DISTINCT aggregations

2019-05-02 Thread Ahmet Altay
java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java#L178 >> [2] >> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java#L48 >> >> >> On Thu,

Re: [DISCUSS][SQL] Providing support for DISTINCT aggregations

2019-05-02 Thread Ahmet Altay
>From my limited understanding, would not the stateful combinefn option require observing the whole input before being able combine and the risk of blowing memory is actually very high except for trivial inputs? On Thu, May 2, 2019 at 11:50 AM Brian Hulette wrote: > Hi everyone, > Currently

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-05-02 Thread Ahmet Altay
these artifacts as the actual release. >>>> >>>> Now if we just try to add (2c) Push these artifacts to Pypi, it will >>>> be treated (by pypi's tooling, anyone who downloads the tarball, ...) >>>> as an actual release. You also can't re-push a tarball with th

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-05-02 Thread Ahmet Altay
ts (the idea being that named releases should > >> never change). So we'd need to change step (1) to update the version > >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc > >> suffix), rebuild, resign before publishing. > >> > >&

Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

2019-05-02 Thread Ahmet Altay
Congratulations! On Thu, May 2, 2019 at 9:54 AM Yifan Zou wrote: > Congratulations! Well deserved! > > On Thu, May 2, 2019 at 9:37 AM Rui Wang wrote: > >> Congratulations! >> >> >> -Rui >> >> On Thu, May 2, 2019 at 8:23 AM Michael Luckey >> wrote: >> >>> Congrats! Well deserved! >>> >>> On

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-04-30 Thread Ahmet Altay
30, 2019 at 3:59 PM Robert Bradshaw wrote: > On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay wrote: > > > > This conversation get quite Python centric. Is there a similar need for > Java? > > I think Java is already covered. Go is a different story (but the even > versio

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-04-29 Thread Ahmet Altay
binaries. And that is compatible with RC section of http://www.apache.org/legal/release-policy.html#release-types Ahmet [1] https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay wrote

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

2019-04-26 Thread Ahmet Altay
It is possible to download dependencies with pip to a local directory and install from there [1]. As a side benefit this is supposed to speed up the installation process. Since we setup virtualenv multiple times, this could actually help us in a single run. And if we can keep this cache across

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-04-26 Thread Ahmet Altay
to avoid increased >> > toil on the release manager. >> > >> > One potential hitch I see is that current release process updates the >> > versions to x.y.z (no RC or other pre-release indicator in the version >> > number) whereas pypi (and other systems) ty

Re: [PROPOSAL] Prepare for LTS bugfix release 2.7.1

2019-04-26 Thread Ahmet Altay
I agree with both keeping 2.7.x going until a new LTS is declared and declaring LTS spost-release after some use. 2.12 might actually be a good candidate, with multiple RCs/validations it presumably is well tested. We can consider that after it gets some real world use. On Fri, Apr 26, 2019 at

Re: [PROPOSAL] Prepare for LTS bugfix release 2.7.1

2019-04-25 Thread Ahmet Altay
+1 great idea. Could you share a link to the list of issues you are planning to patch? For clarification, how long we plan to support 2.7.x line? On Thu, Apr 25, 2019 at 5:42 PM Kenneth Knowles wrote: > Hi all, > > Since the release of 2.7.0 we have identified some serious bugs: > > - There

Re: contributor permission for Beam Jira

2019-04-24 Thread Ahmet Altay
Done. Thank you for contributing. On Wed, Apr 24, 2019 at 10:18 AM Yoshiki Obata wrote: > Hello everyone > > This is Yoshiki Obata. > I created ticket BEAM-7137 and plan to fix it. > I'm glad someone would add me as contributer to Jira. > > my Jira username is yoshiki.obata > > Best regards, >

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-04-24 Thread Ahmet Altay
gt;> artifact process as possible. >> >> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay wrote: >> >>> To clarify my proposal, I am proposing publishing to the production pypi >>> repository with an rc tag in the version. And in turn allow users to depend >>> on bea

Re: [BEAM-3344] Please review & [BEAM-352] Assigne me.

2019-04-24 Thread Ahmet Altay
Hi, I assigned you BEAM-352 and restarted the failing tests on your PR. There is also a reviewer assigned to your PR. Ahmet On Wed, Apr 24, 2019 at 11:26 AM Madhusudhan Reddy Vennapusa < sudhan...@gmail.com> wrote: > HI Team, > > I worked on [BEAM-3344] and raised a pull request, though Java

Re: [Discuss] Publishing pre-release artifacts to repositories

2019-04-24 Thread Ahmet Altay
> I also don't know who owns beam package in test pypi repo. Does anybody know? > > In short: +1, and I would suggest using the test PyPi repo to avoid > publishing to the main PyPi repo. > Best > -P. > > [1] https://test.pypi.org/ > [2] https://test.pypi.org/project/apa

[Discuss] Publishing pre-release artifacts to repositories

2019-04-24 Thread Ahmet Altay
Hi all, What do you think about the idea of publishing pre-release artifacts as part of the RC emails? For Python this would translate into publishing the same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a similar thing with Maven

Re: [VOTE] Release 2.12.0, release candidate #4

2019-04-23 Thread Ahmet Altay
e with you on point 2. All of these issues were in RC1 and >> could have been fixed for RC2. >> >> Andrew >> >> On Tue, Apr 23, 2019 at 2:58 PM Ahmet Altay wrote: >> >>> Thank you Andrew. I will suggest two improvements to the release process: >>

Re: Hello from Hannah Jiang

2019-04-23 Thread Ahmet Altay
Welcome! I could not find your user name in JIRA. Have you registered? You need to register first then we can add you as a contributor to Beam/ On Tue, Apr 23, 2019 at 1:31 PM Hannah Jiang wrote: > Thanks Aizhamal. > Here is my user name: jiangxuehua1...@gmail.com > > Thanks, > Hannah > > > >

Re: [VOTE] Release 2.12.0, release candidate #4

2019-04-22 Thread Ahmet Altay
+1 (binding) Verified the python 2 wheel files with quick start examples. On Mon, Apr 22, 2019 at 11:26 AM Ahmet Altay wrote: > I built the wheel files. They are in the usual place along with other > python artifacts. I will test them a bit and update here. Could someone > else p

Re: [VOTE] Release 2.12.0, release candidate #4

2019-04-22 Thread Ahmet Altay
I built the wheel files. They are in the usual place along with other python artifacts. I will test them a bit and update here. Could someone else please try the wheel files as well? Andrew, could you sign and hash the wheel files? On Mon, Apr 22, 2019 at 10:11 AM Ahmet Altay wrote: >

Re: [VOTE] Release 2.12.0, release candidate #4

2019-04-22 Thread Ahmet Altay
I verified - signatures and hashes. - python streaming quickstart guide I would like to verify the wheel files before voting. Please let us know when they are ready. Also, if you need help with building wheel files I can help/build. Ahmet On Mon, Apr 22, 2019 at 3:33 AM Maximilian Michels

Re: [ANNOUNCE] New committer announcement: Yifan Zou

2019-04-22 Thread Ahmet Altay
Congratulations, Yifan! On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson wrote: > Congratulations Yifan! > > On Mon, Apr 22, 2019 at 5:39 PM Cyrus Maden wrote: > >> Congratulations Yifan!! >> >> On Mon, Apr 22, 2019 at 11:26 AM Kenneth Knowles wrote: >> >>> Hi all, >>> >>> Please join me and the

Re: Possible bug in accumulating triggers Python DirectRunner?

2019-04-19 Thread Ahmet Altay
>>>> >>> (actual is:) >>> >>>> + ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10'] >>>> -- >>> >>> >>> >>> >>> On Fri

Re: Possible bug in accumulating triggers Python DirectRunner?

2019-04-19 Thread Ahmet Altay
On Fri, Apr 19, 2019 at 1:58 PM Pablo Estrada wrote: > Hello all, > I've been slowly learning a bit about life in streaming, with state, > timers, triggers, etc. > > The other day, I tried out a trigger pipeline that did not have the > behavior that I was expecting, and I am looking for feedback

Re: Projects Can Apply Individually for Google Season of Docs

2019-04-17 Thread Ahmet Altay
gt;- > >+pabl...@apache.org , please complete the org >application form [4] >- > >@Ahmet Altay , please complete alternative > administrator form [5] >- > >@pabl...@apache.org , @Ahmet Altay > , and all other contributors th

Re: New contributor to Beam

2019-04-17 Thread Ahmet Altay
Welcome! On Wed, Apr 17, 2019 at 12:26 PM Rose Nguyen wrote: > Welcome, Cyrus!! > > On Wed, Apr 17, 2019 at 11:58 AM Niklas Hansson < > niklas.sven.hans...@gmail.com> wrote: > >> Welcome :) >> >> Den ons 17 apr. 2019 kl 20:33 skrev Aizhamal Nurmamat kyzy < >> aizha...@google.com>: >> >>>

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-15 Thread Ahmet Altay
ion that Brian >> put forth. I like how the change is being applied to the Beam Java SDK >> harness and not just Dataflow so all portable runner users get this as well. >> >> On Wed, Apr 10, 2019 at 9:03 PM Kenneth Knowles wrote: >> >>> >>> >>>

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Ahmet Altay
On Wed, Apr 10, 2019 at 7:59 PM Kenneth Knowles wrote: > TL;DR I like the simple approach better than the ServiceLoader solution > when a particular DoFn depends on the result. The ServiceLoader solution > fits when it is somewhat independent of a particular DoFn (I'm not sure the > use

Re: JDK11 support?

2019-04-09 Thread Ahmet Altay
Most recently +Pablo Estrada and +Michał Walenia were working on Java 11 related JIRAs. They may have more context. On Tue, Apr 9, 2019 at 1:22 PM Maximilian Michels wrote: > Hi Yi, > > That's a great question. Beam is still on Java 8 at the moment. There is > a JIRA issue for making Beam

Re: PostCommit tests currently failing

2019-04-09 Thread Ahmet Altay
+Lukasz Cwik +Mikhail Gryzykhin (Adding authors for #8203 and #8204 explicitly.) On Tue, Apr 9, 2019 at 5:05 AM Michael Luckey wrote: > Hi, > > looks as if > - beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow > - beam_PostCommit_Java_Nexmark_Dataflow > -

Re: [VOTE] Release 2.12.0, release candidate #2

2019-04-09 Thread Ahmet Altay
-1 unfortunately. Reason is https://issues.apache.org/jira/browse/BEAM-7038. 2.12 includes changes to coders that are not compatible with previous versions breaking some Dataflow use cases. On Mon, Apr 8, 2019 at 3:54 PM Andrew Pilloud wrote: > Hi everyone, > > Please review and vote on the

Re: Implementation an S3 file system for python SDK - Updated

2019-04-08 Thread Ahmet Altay
+dev +Pablo Estrada +Chamikara Jayalath +Udi Meiri Thank you Pasan. I quickly looked at the proposal and it looks good. Added a few folks who could offer additional feedback. On Mon, Apr 8, 2019 at 12:13 AM Pasan Kamburugamuwa < pasankamburugamu...@gmail.com> wrote: > Hi, > > I have updated

Re: [DISCUSS] Backwards compatibility of @Experimental features

2019-04-05 Thread Ahmet Altay
that case it is pretty easy to fork >>> and make a new and better IO. >>> >>> And as an extreme view, I would prefer if we add a deadline for >>> experimental features, then our default action is to remove them, not >>> declare them stable. If noone is trying to

Re: Projects Can Apply Individually for Google Season of Docs

2019-04-05 Thread Ahmet Altay
Thank you Aizhamal for volunteering. I am happy to help as an administrator. cc: +Rose Nguyen +Melissa Pashniak in case they will be interested in mentorship and/or administration. On Fri, Apr 5, 2019 at 9:16 AM Thomas Weise wrote: > This is great. Beam documentation needs work in several

Re: Implementation an S3 file system for python SDK

2019-04-03 Thread Ahmet Altay
+Pablo Estrada On Wed, Apr 3, 2019 at 8:46 AM Lukasz Cwik wrote: > +dev > > On Wed, Apr 3, 2019 at 2:03 AM Pasan Kamburugamuwa < > pasankamburugamu...@gmail.com> wrote: > >> Hi , >> >> I have completed a proposal to implementation an S3 file system for >> python SDK for the google summer of

Re: Contibutor permissions for Beam Jira tickets

2019-04-01 Thread Ahmet Altay
Welcome to the project! On Mon, Apr 1, 2019 at 6:23 AM Ismaël Mejía wrote: > You have now the Contributor role, and I assigned the ticket you asked for. > Enjoy! > > Ismaël > > On Mon, Apr 1, 2019 at 12:35 PM Madhusudhan Reddy Vennapusa > wrote: > > > > Hi, > > > > This is Madhu, I am

Re: Unexpected TestStream behavior when testing stateful DoFn

2019-03-28 Thread Ahmet Altay
I agree it looks like a bug. Filed https://issues.apache.org/jira/browse/BEAM-6934 On Thu, Mar 28, 2019 at 5:18 PM Reuven Lax wrote: > This looks like a bug to me. > > On Thu, Mar 28, 2019 at 2:52 PM Amar Pai wrote: > >> Hi, >> >> I'm running into some unexpected behavior when trying to unit

<    1   2   3   4   5   6   7   8   9   10   >