Re: GSOC - Summer of Code, on Beam?

2019-01-16 Thread Ismaël Mejía
I put together a (currently empty) saved search for Beam's GSoC issues: > https://issues.apache.org/jira/issues/?filter=12345337 > > Kenn > > On Wed, Dec 12, 2018 at 1:10 AM Ismaël Mejía wrote: >> >> Oh I had not seen that the announce was officia

Re: [PROPOSAL] Prepare Beam 2.10.0 release

2019-01-16 Thread Ismaël Mejía
Ok since there were not many issues I did the 'update' for the misplaced issues to version 2.10. We are good to go. New resolved issues in master musg go now into 2.11.0 On Wed, Jan 16, 2019 at 10:38 AM Ismaël Mejía wrote: > > This means that the tickets resolved and marked for 2.11

Re: [PROPOSAL] Prepare Beam 2.10.0 release

2019-01-16 Thread Ismaël Mejía
9 at 10:31 AM Kenneth Knowles wrote: >> >> I'm on it. >> >> On Tue, Jan 15, 2019 at 8:10 AM Ismaël Mejía wrote: >>> >>> There is also another issue, after the 2.10.0 branch cut some >>> identifier in the build was not changed and the Apache Beam S

Re: [PROPOSAL] Prepare Beam 2.10.0 release

2019-01-15 Thread Ismaël Mejía
> by avoiding all the other changes on master. The timing of the cut is always > > going to include older and newer changes anyhow. > > > > Kenn > > > > On Wed, Jan 2, 2019 at 1:08 PM Ismaël Mejía > <mailto:ieme...@gmail.com>> wrote: > > > >

Re: Merge of vendored Guava (Some PRs need a rebase)

2019-01-14 Thread Ismaël Mejía
people from using the > unvendored Guava? If not, such a rule could be useful because it's almost > inevitable that the unvedored Guava will slip back in. > > Cheers, > Max > > On 14.01.19 05:55, Ismaël Mejía wrote: > > We merged today the PR [1] that changes most of the code

Re: Beam Dependency Check Report (2019-01-14)

2019-01-14 Thread Ismaël Mejía
I don't know if there is an issue with this report but I have the impression that once a dependency update is done, it stops checking it and if a new version of the library is published it is not reported. Also I don't see some libraries updates, What is the current status, are some libs disabled

Merge of vendored Guava (Some PRs need a rebase)

2019-01-14 Thread Ismaël Mejía
We merged today the PR [1] that changes most of the code to use our new guava vendored dependency. In practice it means that most of the imports of the classes were changed from `com.google.common.` to `org.apache.beam.vendor.guava.v20_0.com.google.common.` This is a great improvement to fix a

Re: Dev contact for KafkaIO

2019-01-10 Thread Ismaël Mejía
Hello, Thanks a lot Raghu for all the nice work you did for KafkaIO and the project in general. I hope we can bother you a bit (only when needed) with advanced Kafka related issues. Best wishes for your new adventure. Ismaël On Thu, Jan 10, 2019 at 11:58 AM Alexey Romanenko wrote: > > Raghu, >

Re: Enforce javadoc comments in public methods?

2019-01-08 Thread Ismaël Mejía
-0 Same comments than Robert I am particularly worried on how this affect contributors in particular casual ones. Even if the intended idea is good I am also worried that people just write poor comments to get rid of the annoyance. Have you already estimated how hard is the current codebase

Re: Beam snapshots broken

2019-01-02 Thread Ismaël Mejía
holidays now and ready to pick this up. > > On Thu, Dec 27, 2018 at 10:56 AM Ismaël Mejía wrote: >> >> Thanks Andrew for the info. >> I think things can wait a bit considering the time of the year, I just >> wanted to raise awareness about the issue. >> I suppos

Re: [PROPOSAL] Prepare Beam 2.10.0 release

2019-01-02 Thread Ismaël Mejía
Can you please create 2.11 tag in JIRA so we can move the JIRAs that are not blocking. I have quite a bunch of pending code reviews that hoped to get into this one but well now probably they shall wait. (My excuses for the people who may be impacted, I had not checked that the date was in the

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2019-01-02 Thread Ismaël Mejía
>> > > Kenn >>> > > >>> > > [1] https://issues.apache.org/jira/browse/BEAM-6249 >>> > > >>> > > On Thu, Dec 20, 2018 at 12:02 PM Scott Wegner >>> > > >> > >

Re: Adding ":beam-runners-direct-java:needsRunnerTests" to "Run Java PreCommit"

2018-12-28 Thread Ismaël Mejía
Big +1 I am in awe that they are not active anymore, they used to be in the good old maven days, when IDE integration worked and most JIRAs were not about build or test issues. Sigh. On Fri, Dec 28, 2018 at 3:24 PM Reuven Lax wrote: > > Kenn and I both noticed that some needsRunner tests time

Fwd: workspace cleanups needed on jenkins master

2018-12-27 Thread Ismaël Mejía
Bringing this subject for awareness to dev@ We are sadly part of this top. Does somebody know what this data is? And if we can clean it periodically? Can somebody with more sysadmin super powers take a look and act on this. -- Forwarded message - From: Chris Lambertus Date: Thu,

Re: Beam snapshots broken

2018-12-27 Thread Ismaël Mejía
>> >> I can reproduce this on my machine, and reverting >> https://github.com/apache/beam/pull/7324 fixed the problem. There is a >> separate thread in dev@ about releasing vendored gRPC v0.2, I'm wondering if >> it will this issue. >> >> On Thu

Re: Beam snapshots broken

2018-12-27 Thread Ismaël Mejía
>> Beam9 is offline right now. But, the job also failed on beam4 and 13 with >> "Could not determine the dependencies of task ':beam-sdks-python:test.". >> Seems like the task dependency did not setup properly. >> >> >> >> On Wed, Dec 12, 2018 at

Re: Contributor permission for Beam Jira tickets

2018-12-24 Thread Ismaël Mejía
Done you can now self-assign JIRAs. Please take a look at the Contribution guide [1] and the PTransform guide [2] in case you have questions [1] https://beam.apache.org/contribute/ [2] https://beam.apache.org/contribute/ptransform-style-guide/ And also don't hesitate to ask here or in slack.

Re: (fyi) PR#7324 updates how projects consume vendored guava

2018-12-20 Thread Ismaël Mejía
I may be misreading or probably missed part of the previous discussion, but why don't we have the vendoring artifacts in a different repo? This will do the release really independent and avoid the artificial staging step, no? Thomas what do you mean with: > It is unfortunate that we have to

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Ismaël Mejía
Does this imply that we need a subsequent full release afterwards? I am assuming this new release is related to the reported issues with the dataflow worker or is this something different? On Thu, Dec 20, 2018 at 2:51 AM Kenneth Knowles wrote: > > +1 > > - sigs good > - `jar tf` looks good > >

Re: Issue with publishing maven artefacts locally

2018-12-12 Thread Ismaël Mejía
Thanks Garrett for the quick fix. I just tested and it is working now. I found a second issue (not related to Garrett's PR, it was the reason why I detected that local artifacts were not updated in our jenkins (in the other thread). To validate that our daily snapshots don't break existing code

Re: Beam snapshots broken

2018-12-12 Thread Ismaël Mejía
; > On Tue, Dec 11, 2018 at 9:10 AM Ismaël Mejía wrote: >> >> It seems that Beam snapshots are broken since Dec. 2 >> https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_Gradle_NightlySnapshot/ >> >> It seems "The :beam-website:startDockerContainer task failed." >> Can somebody please take a look.

Re: GSOC - Summer of Code, on Beam?

2018-12-12 Thread Ismaël Mejía
/timeline On Tue, Dec 11, 2018 at 6:14 PM Ismaël Mejía wrote: > > You have to register the concrete proposal, no need to register the > project since the organization (ASF) is already part of GSoC. > On Tue, Dec 11, 2018 at 12:42 PM Maximilian Michels wrote: > > > > I think t

Re: GSOC - Summer of Code, on Beam?

2018-12-11 Thread Ismaël Mejía
e project to be able to receive applications from students? > > On 07.12.18 16:44, Ismaël Mejía wrote: > > Last year we had two proposals. Kenneth proposal around SQL was > > accepted and if I remember correctly a success. > > > > For students interested you should do the co

Beam snapshots broken

2018-12-11 Thread Ismaël Mejía
It seems that Beam snapshots are broken since Dec. 2 https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_Gradle_NightlySnapshot/ It seems "The :beam-website:startDockerContainer task failed." Can somebody please take a look.

Re: OOO

2018-12-10 Thread Ismaël Mejía
Thanks for the community awareness, enjoy the time with the baby and see you soon. On Fri, Dec 7, 2018 at 9:20 PM Lukasz Cwik wrote: > > I'll be away for the next three months taking care of my little one[1] and am > excited to see what happens within Apache Beam when I return. > > I have been

Re: [VOTE] Release 2.9.0, release candidate #1

2018-12-07 Thread Ismaël Mejía
Looking at the dates on the Spark runner git log there was a PR merged to change Spark translation from classes to URNs. I cannot see how this can impact performance. Looking at the other queries in the dashboards, there seems to be a great variability in the executions of the Spark runner to the

Re: Contributor permission for Beam Jira tickets

2018-11-16 Thread Ismaël Mejía
Hello Fabien, welcome to Beam, you have now permission to self assign JIRA tickets. Enjoy! On Fri, Nov 16, 2018 at 1:59 PM Fabien Rousseau wrote: > > Hi, I'm using BEAM and Cassandra. I'd like to be granted contributor > permission for Beam JIRA tickets. My user id is frousseau. > > Thanks. >

Re: [DISCUSS] SplittableDoFn Java SDK User Facing API

2018-11-15 Thread Ismaël Mejía
Some late comments, and my pre excuses if some questions look silly, but the last documents were a lot of info that I have not yet fully digested. I have some questions about the ‘new’ Backlog concept following a quick look at the PR https://github.com/apache/beam/pull/6969/files 1. Is the

Re: MongoIO - streaming

2018-11-06 Thread Ismaël Mejía
Hello, There is not yet support for an Unbounded source for Mongo (streams) in Beam yet. You can track (or contribute to) the progress of the support on this JIRA: https://issues.apache.org/jira/browse/BEAM-2639 On Tue, Nov 6, 2018 at 9:02 AM Chaim Turkel wrote: > > Hi, > I am looking for a

Re: [DISCUSS] More precision supported by DATETIME field in Schema

2018-11-06 Thread Ismaël Mejía
+1 to more precision even to the nano level, probably via Reuven's proposal of a different internal representation. On Tue, Nov 6, 2018 at 9:19 AM Robert Bradshaw wrote: > > +1 to offering more granular timestamps in general. I think it will be > odd if setting the element timestamp from a row

Re: Stackoverflow Questions

2018-11-06 Thread Ismaël Mejía
Is this somehow bidirectional? or should we then go and answer in SO? Asking this because in part the interest of user@ is also to learn from other people problems/solutions, so having the answers there would be ideal. On Mon, Nov 5, 2018 at 9:57 PM Kenneth Knowles wrote: > > +user@ > > I think

Re: Evolving a Coder for an added field

2018-11-05 Thread Ismaël Mejía
For some extra context this change touches more than FileIO, in reality this will affect updates in any file-based pipelines because the metadata on each file will have now an extra field for the lastModifiedDate. The PR looks perfect, only issue is the backwards compatibility Coder question.

Re: Never get spotless errors with this one weird trick

2018-11-02 Thread Ismaël Mejía
Nice idea, thanks for sharing and thanks Scott for separating this in the build. On Thu, Nov 1, 2018 at 11:51 PM Alan Myrvold wrote: > > Thanks for the trick. I added it to > https://cwiki.apache.org/confluence/display/BEAM/Java+Tips > > On Thu, Nov 1, 2018 at 2:26 PM Ankur Goenka wrote: >> >>

Re: [ANNOUNCE] New committer announcement, Euphoria edition

2018-11-02 Thread Ismaël Mejía
Congratulations, and thanks for all the hard work on making Euphoria Beam ready ! On Fri, Nov 2, 2018 at 12:06 AM Scott Wegner wrote: > > Congrats David! > > On Thu, Nov 1, 2018 at 2:21 PM Thomas Weise wrote: >> >> Congrats! >> >> >> On Thu, Nov 1, 2018 at 12:05 PM Łukasz Gajowy >> wrote: >>>

Re: Data Preprocessing in Beam

2018-11-01 Thread Ismaël Mejía
this and contribute with this preprocessing > extension to beam. > > Cheers. > > On 10/31/2018 02:13 PM, Ismaël Mejía wrote: > > Hello, > > > > I mentored Arnaud to contribute the sketching extension into Beam and > > from a quick look at Alex paper + imple

Re: PCollectionViews$SimplePCollectionView.hashCode allocates memory

2018-10-31 Thread Ismaël Mejía
Vojta you are right, your implementation seems like a good improvement. Can you please create a JIRA and eventually if you are interested do a PR to contribute a fix for it. Regards, Ismaël On Wed, Oct 31, 2018 at 2:18 PM Vojtech Janota wrote: > > Hi, > > I'm currently profiling memory

Re: Data Preprocessing in Beam

2018-10-31 Thread Ismaël Mejía
Hello, I mentored Arnaud to contribute the sketching extension into Beam and from a quick look at Alex paper + implementation, I think this should be an independent extension. Sketching is a collection of transforms that rely on probabilistic data structures to give approximate results and

Re: bigquery issue

2018-10-31 Thread Ismaël Mejía
Hello, If you think it is a bug (or issue) you can report it at Apache's JIRA https://issues.apache.org/jira/projects/BEAM/issues If it is more of a use related question probably it is better to do it in the user@ mailing list. Notice that reporting issues is a way of contributing to Beam, for

Re: [VOTE] Release 2.8.0, release candidate #1

2018-10-29 Thread Ismaël Mejía
uestion about the initial LTS discussion. We have decided to not make > 2.8.0 a LTS release. Should we wait until next release to patch this issue? > What is the cost/benefit of maintaining this branch? > >> >> >> Kenn >> >> On Mon, Oct 29, 2018 at 8:40 A

Re: [VOTE] Release 2.8.0, release candidate #1

2018-10-29 Thread Ismaël Mejía
published to users this has to be > 2.8.1. Ahmet - we are to this point, right? > > Kenn > > On Mon, Oct 29, 2018 at 8:40 AM Ismaël Mejía wrote: >> >> First thanks Etienne and Kenn for noting the performance issue. I >> reviewed the discussed PR.It introduced a new ‘@E

Re: [ANNOUNCE] New committers, October 2018

2018-10-19 Thread Ismaël Mejía
Congratulations guys and welcome ! On Fri, Oct 19, 2018 at 4:12 PM Jean-Baptiste Onofré wrote: > > Congrats and welcome aboard ! > > Regards > JB > > On 19/10/2018 16:09, Kenneth Knowles wrote: > > Hi all, > > > > Hot on the tail of the summer announcement comes our pre-Hallowe'en > >

Re: Java > 8 support

2018-10-18 Thread Ismaël Mejía
gt;> Maven. We can follow a similar pattern where Java11 (or any other version) >>> have their own projects and as the base Java version moves up, we can merge >>> those modules back in to simplify the set of modules being developed. >>> >>> On Wed, O

Re: [ANNOUNCE] New committers & PMC members, Summer 2018 edition

2018-10-18 Thread Ismaël Mejía
Congrats everyone! On Wed, Oct 17, 2018 at 12:12 PM Tim Robertson wrote: >> >> Great to see the community growing! > > > Indeed - congratulations to everyone and from my side, thank you for the > recognition. I'm looking forward to working more closely with you all. > > > > On Wed, Oct 17, 2018

Re: Java > 8 support

2018-10-17 Thread Ismaël Mejía
work do you estimate is necessary to >>> support Java 11 with some of the existing sources? I understand that we >>> have many, many sources, but perhaps some of the more popular ones (e.g. >>> TextIO)? >>> >>> Thanks! >>> -P. >>> &

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

2018-10-15 Thread Ismaël Mejía
+1 On Mon, Oct 15, 2018 at 8:14 PM Mikhail Gryzykhin wrote: > > +1 also suggested that in another thread. > > --Mikhail > > Have feedback? > > > On Mon, Oct 15, 2018 at 11:10 AM Rui Wang wrote: >> >> +1 >> >> Agree that people might be only interested in JIRA activities but not in the >> build.

Re: Java > 8 support

2018-10-10 Thread Ismaël Mejía
Just wanted to clarify, there is already a JIRA for ongoing work on Java 11 support. https://issues.apache.org/jira/browse/BEAM-2530 I led the initial work on supporting what at the time was Java 9/10, so far the biggest blockers were around the ApiSurface tests (not at all compatible with these

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

2018-10-10 Thread Ismaël Mejía
The simplest thing we can do is just to pin all the deps of the LTS and not move them in any maintenance release if not a strong reason to do so. The next subject is to make maintainers aware of which release will be the LTS in advance so they decide what to do with the dependencies versions. In

Re: [DISCUSS] Gradle for the build ?

2018-10-10 Thread Ismaël Mejía
JB mentioned some factual points, I think most of the community embraced the move to gradle with the best hopes about the promised improvements, but with the perspective of time, it is clear that we have not delivered on many of them. I can easily understand the growing bitterness about the

Re: [PROPOSAL] Prepare Beam 2.8.0 release

2018-10-04 Thread Ismaël Mejía
Hello, Thanks Ahmet for volunteering to do the release, and proposing this as an LTS. I have still some questions on our LTS policies (which may have consequences on the discussed release): What are the expected implications of upgrades in the LTS, e.g. If a connector let’s say Kafka is

Re: Purpose of GcpApiSurfaceTest in google-cloud-platform SDK

2018-10-03 Thread Ismaël Mejía
Just adding a bit to what Kenn said, this is the JIRA that covers the classpath issue (which seem not to affect you at least) https://issues.apache.org/jira/browse/BEAM-3748 Note also that the classpath based resolution is not future compatible, I saw it broke when working on the migration to Java

Re: SplittableDoFn

2018-10-03 Thread Ismaël Mejía
> 500, then the Runner could provide some value > > like 212.2 which the SDK would then round to 212. > > If the backlog that the SDK was reporting 57 > > pubsub me

Re: Does anyone have a strong intelliJ setup?

2018-10-01 Thread Ismaël Mejía
Hello Alex, I understand your pain and thanks for bringing this subject, I also have found many issues in the process to the point of believing recently that it is undeterministic. Last time I followed the process ~3 weeks ago. I had to clean up all caches (both remove the intelliJ temp files and

Modular IO presentation at Apachecon

2018-09-26 Thread Ismaël Mejía
Hello, today Eugene and me did a talk about about modular APIs for IO at ApacheCon. This talk introduces some common patterns that we have found while creating IO connectors and also presents recent ideas like dynamic destinations, sequential writes among others using FileIO as a use case. In

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

2018-09-19 Thread Ismaël Mejía
Congratulations and welcome Kenn as new chair! Thanks Davor for your hard work too. On Wed, Sep 19, 2018 at 11:14 PM Rui Wang wrote: > Congrats! > > -Rui > > On Wed, Sep 19, 2018 at 2:12 PM Chamikara Jayalath > wrote: > >> Congrats! >> >> On Wed, Sep 19, 2018 at 2:05 PM Ahmet Altay wrote: >>

Re: Add ruoyun into contributor list?

2018-09-19 Thread Ismaël Mejía
Done On Wed, Sep 19, 2018 at 10:17 PM ruo...@google.com wrote: > > Hi, Folks, > > Can some one add me as contributor in the Beam issue tracker? > > account name in JIRA: ruoyun > > Thanks!

Re: Tracking and resolving release blocking bugs

2018-09-19 Thread Ismaël Mejía
We have done this so far by letting the JIRA issues 'Open' with the 'Fix version' corresponding to the upcoming release and we track the progress between the branch cut and the first release candidate with the assigned parties, the process has been the de-facto standard since long time ago and has

Re: SplittableDoFn

2018-09-17 Thread Ismaël Mejía
Thanks a lot Luke for bringing this back to the mailing list and Ryan for taking the notes. I would like to know if there was some discussion, or if you guys have given some thought to the required changes in the SDK (API) part. What will be the equivalent of `splitAtFraction` and what should IO

Re: Spotless broken on master

2018-09-12 Thread Ismaël Mejía
Thanks Thomas, I will use that one now. On Wed, Sep 12, 2018 at 12:35 PM Thomas Weise wrote: > > The equivalent target to run should be "check" - see > https://beam.apache.org/contribute/ > > > On Wed, Sep 12, 2018 at 6:01 AM Ismaël Mejía wrote: >> >

Re: Spotless broken on master

2018-09-12 Thread Ismaël Mejía
but without the fear of breaking stuff. On Tue, Sep 11, 2018 at 11:55 PM Andrew Pilloud wrote: > > I don't think spotless is included in the default test target. Jenkins runs a > more expanded ':javaPreCommit' gradle target. > > Andrew > > On Tue, Sep 11, 2018 at 2:32 P

Re: Spotless broken on master

2018-09-11 Thread Ismaël Mejía
Mmm this is weird, I tested this locally and passed without issue, I am wondering how could this happen. Thanks anyway for the quick fix.

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

2018-09-06 Thread Ismaël Mejía
Dal Tran this is really nice, this is a much awaited contribution, welcome! On Thu, Sep 6, 2018 at 9:18 PM Ahmet Altay wrote: > > Welcome! Thank you! > > On Thu, Sep 6, 2018 at 11:40 AM, Chamikara Jayalath > wrote: >> >> Welcome Dat Tran and thanks for the contribution. >> >> On Thu, Sep 6,

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

2018-09-05 Thread Ismaël Mejía
BIG +1, the previous work on having docker build images [1] had a similar goal (to have a reproducible build environment). But this is even better because we will guarantee the exact same environment in Jenkins as well as any further improvements. It is important to document the setup process as

Re: [BEAM-960] Backoff in the DirectRunner if no work is available

2018-09-05 Thread Ismaël Mejía
Hello Vojtech, Thanks a lot for your work on this fix and for pinging us for review. Improving all runners is important and CPU improvements in direct runner definitely are welcome because we all benefit of this (for testing). Review is in process so you should have news soon. On Thu, Aug 30,

Re: builds.apache.org refused connections since last night

2018-09-04 Thread Ismaël Mejía
The issue with 'beam-site' seems related to the jenkins outage. Can somebody with more knowledge/access take a look at this? On Sat, Sep 1, 2018 at 2:36 AM Udi Meiri wrote: > > My mistake. I thought you needed to appear as "member" on apache/beam-site as > well. > > On Fri, Aug 31, 2018 at 5:06

Re: Beam Schemas: current status

2018-08-30 Thread Ismaël Mejía
Thanks Reuven for the excellent summary and thanks to all the guys who worked in the Schema/SQL improvements. This is great for usability. I really like the idea of making user experience simpler, e.g. by automatically inferring Coders. Some questions: - Any plans to add similar improvements for

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

2018-08-28 Thread Ismaël Mejía
I think we should refine the strategy on dependencies discussed recently. Sorry to come late with this (I did not follow closely the previous discussion), but the current approach is clearly not in line with the industry reality (at least not for IO connectors + Hadoop + Spark/Flink use). A

Re: [Request] Beam Contributor Role

2018-08-23 Thread Ismaël Mejía
Hi! You can self assign JIRAs now, I also assigned the one you are working on. Good luck with your work ! Ismaël On Thu, Aug 23, 2018 at 10:05 AM Pascal Gula wrote: > Dear PMC members, > I am currently working on a Python version of mongoDB IO connector and > already opened a Jira ticket for

Re: [DISCUSS] Performance of write() in file based IO

2018-08-22 Thread Ismaël Mejía
Spark runner uses the Spark broadcast mechanism to materialize the side input PCollections in the workers, not sure exactly if this is efficient assigned in an optimal way but seems logical at least. Just wondering if we shouldn't better first tackle the fact that if the pipeline does not have

Re: Bootstrapping Beam's Job Server

2018-08-21 Thread Ismaël Mejía
It is also worth to mention that apart of the testing/development use case there is also the case of supporting people running in Hadoop distributions. There are two extra reasons to want a process based version: (1) Some Hadoop distributions run in machines with really old kernels where docker

Re: [ANNOUNCE] Apache Beam 2.6.0 released!

2018-08-09 Thread Ismaël Mejía
Two really interesting features in 2.6.0 not mentioned in the announcement email: - Bounded SplittableDoFn support is available now in all runners (SDF is the new IO connector API). - HBaseIO was updated to be the first IO supporting Bounded SDF (using readAll). On Fri, Aug 10, 2018 at

Re: [VOTE] Community Examples Repository

2018-08-09 Thread Ismaël Mejía
3 for all the reasons discussed above. I think there are better ways to improve the status quo without the extra maintenance of having a new repo for this. On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay wrote: > If we go forward with (3), could we actually update our documentation on > how we will

Re: SQS source

2018-07-31 Thread Ismaël Mejía
up the review the >> 2.6.0 branch has already been cut, and we try and only cherry pick >> important bugfixes. Hopefully the next release will be soon, and it's also >> possible to use the nightly Beam releases in the interim. >> >> Reuven >> >> On Tue, Jul

Re: SQS source

2018-07-31 Thread Ismaël Mejía
m/pull/5995 >> >> And will be starting my work here: >> https://github.com/JohnRudolfLewis/beam/tree/Add-SqsIO >> >> >> On Thu, Jul 19, 2018 at 1:43 AM, Jean-Baptiste Onofré >> wrote: >>> >>> Agree with Ismaël. >>> >>>

Re: Beam Docs Contributor

2018-07-30 Thread Ismaël Mejía
Welcome ! Great to see someone new working in this important area for the project. On Mon, Jul 30, 2018 at 5:57 AM Kai Jiang wrote: > Welcome Rose! > ᐧ > > On Sun, Jul 29, 2018 at 8:53 PM Rui Wang wrote: > >> Welcome! >> >> -Rui >> >> On Sun, Jul 29, 2018 at 7:07 PM Griselda Cuevas wrote: >>

Re: Vendoring / Shading Protobuf and gRPC

2018-07-19 Thread Ismaël Mejía
-protobuf from intellij. >>>>> Adding >>>>> :beam-model-fn-execution/build/libs/beam-model-fn-execution-2.6.0-SNAPSHOT.jar >>>>> and >>>>> :beam-model-job-management/build/libs/beam-model-job-management-2.6.0-SNAPSHOT.jar >>>>> to the

Re: SQS source

2018-07-19 Thread Ismaël Mejía
Thanks for your interest John, it would be a really nice contribution to add SQS support. Some context on the kinesis stuff: The reason why kinesis is still in a separate module is more related to a licensing problem. Kinesis uses some native libraries that are published under a not 100% apache

Re: [BEAM-4814] Add client configuration to aws options #5983

2018-07-18 Thread Ismaël Mejía
Hello, Since I have been changing options related parts of S3 I can take a look, so I will take the R: I will also probably ping Jacob Marble (the author of S3FileSystem for additional comments), thanks for bringing this contribution. On Wed, Jul 18, 2018 at 10:43 PM John Rudolf Lewis wrote: >

Re: Live coding & reviewing adventures

2018-07-18 Thread Ismaël Mejía
Have you thought about creating some sort of index page for your past live streams? At least for the non-review ones it can provide great value given that searching videos is not the easiest thing to do. On Wed, Jul 18, 2018 at 12:51 AM Holden Karau wrote: > > Sure! I’ll respond with a pip freeze

Re: CODEOWNERS for apache/beam repo

2018-07-17 Thread Ismaël Mejía
Is there a way to put inactive people as not reviewers for the blame case? I think it can be useful considering that a good amount of our committers are not active at the moment and auto-assigning reviews to them seem like a waste of energy/time. On Tue, Jul 17, 2018 at 1:58 AM Eugene Kirpichov

Re: Performance issue in Beam 2.4 onwards

2018-07-17 Thread Ismaël Mejía
Given that the 2.6.0 cut is supposed to be today (or next days), what is the status on this, has it been identified / reverted ? or is there any other plan ? On Tue, Jul 10, 2018 at 2:50 PM Reuven Lax wrote: > > If we added something slow to the core library in order to better test >

Re: An update on Eugene

2018-07-16 Thread Ismaël Mejía
I am sad to read this, but at the same happy for you and your future adventures Eugene. Thanks a lot for all the work you have done in this project, all the work on SDF, the improvements on composability and the File-based IOs, and of course for your reviews that really helped improve the quality

Re: Vendoring / Shading Protobuf and gRPC

2018-07-12 Thread Ismaël Mejía
java/io/google-cloud-platform. I know that this will be >>> impossible for some connectors without backwards incompatible changes since >>> they exposed protobuf on their API surface. I know that Chamikara was >>> looking to shade this away in the sdks/java/io

Re: [ANNOUNCEMENT] Nexmark included to the CI

2018-07-12 Thread Ismaël Mejía
That’s great to see in action, great work Etienne! Is there any document on how to integrate ‘stuff’ into the dashboards? I think this is worth having for people willing to do so like Kai or Andrew. Are there any docs on this? or maybe Lukasz Gajowy know ? On Thu, Jul 12, 2018 at 9:04 PM Andrew

Re: Help! Beam SQL needs more committer support

2018-07-03 Thread Ismaël Mejía
It has been long time since the last time I checked at the SQL work but in case you guys still need more people to review you can count on me too. On Mon, Jul 2, 2018 at 7:23 PM Rui Wang wrote: > Haha! I don't even know who I should @ now because there are so many > helpful hands! > > -Rui > >

Re: Using user developped source in streamline python

2018-06-27 Thread Ismaël Mejía
It seems like a nice opportunity also to contribute them to the project in case you are able to work on them. Don't hesitate to contact us or ask for help if needed. On Wed, Jun 27, 2018 at 4:45 PM Ahmet Altay wrote: > Hi Sébastien, > > Currently there is no work in progress for including the

Re: [DISCUSS] Automation for Java code formatting

2018-06-27 Thread Ismaël Mejía
++1 - This solves the big mismatch of autoformatting using eclipse's google java style and the google-java-format plugin that makes a common source of unneeded diffs part of the reviews. - The spotless plugin makes this trivial to do and uniform. We need to update the instructions on formatting

Re: Going on leave for a bit

2018-06-26 Thread Ismaël Mejía
Enjoy your family time. Best wishes, Ismael On Tue, Jun 26, 2018 at 12:13 PM Pei HE wrote: > (A late) Congrats for the newborn! > -- > Pei > > On Tue, Jun 26, 2018 at 1:42 PM, Kenneth Knowles wrote: > > Hi friends, > > > > I think I did not mention on dev@ at the time, but my child #2

Re: scala/scio

2018-06-25 Thread Ismaël Mejía
​Agree with Kenn, granularity would be key for per-module Beam releases. We can easily think today of releasing separately IOs and extensions because they depend only on the public API of the SDK which has not changed much since 2.0.0. So scio will probably fit this category. A different story

Re: [PROPOSAL] Merge samza-runner to master

2018-06-25 Thread Ismaël Mejía
+1 It is important to have new runners merged (even if not 100% complete) so they benefit of the fixes going on, and that they can easily (and incrementally) start to track the new portability features as they develop. What is next then ? Who triggers the green button so this happens? On

Re: Auto closing stale PRs label

2018-06-25 Thread Ismaël Mejía
> been watching it and didn't see any that seemed wrong. > > Kenn > > > On Mon, Jun 25, 2018 at 12:52 AM Ismaël Mejía wrote: > >> I saw some PRs auto closed recently and I was wondering if we could >> adjust the label that is added to the autoclosed PRs, currentl

Auto closing stale PRs label

2018-06-25 Thread Ismaël Mejía
I saw some PRs auto closed recently and I was wondering if we could adjust the label that is added to the autoclosed PRs, currently it is 'wontfix' but this label sends a fake (and negative) message. Can we parametrize the bot to put something closer to the intention like 'autoclosed'? Who can

Re: [FYI] New Apache Beam Swag Store!

2018-06-13 Thread Ismaël Mejía
Great ! Thanks Gris and Matthias for putting this in place. Hope to get that hoodie soon. As a suggestion, more colors too, and eventually a t-shirt just with the big B logo. On Mon, Jun 11, 2018 at 6:50 PM Mikhail Gryzykhin wrote: > > That's nice! > > More colors are appreciated :) > > --Mikhail

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-12 Thread Ismaël Mejía
Yes, ok I was not aware it was already being addressed, nice. On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay wrote: > > Ismaël, > > I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing the > issue by changing the findbugs back to "com.github.stephenc.findbugs". Is > this what you

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-12 Thread Ismaël Mejía
There is another issue highlighted by Scott Wegner in a non-related to the vote PR discussion. https://github.com/apache/beam/pull/5540 It seems that in the migration to gradle we changed the findbugs-annotations library from com.github.stephenc.findbugs to

Re: [Proposal] Automation For Beam Dependency Check

2018-06-04 Thread Ismaël Mejía
Is there a way to add to that weekly report the new dependencies that were introduced in the week before, or that have changed? We are not addressing another important problem: Leaking of dependencies. I am not aware of the gradle equivalent of the maven dependency plugin that helps to determine

Re: [SQL] Unsupported features

2018-06-04 Thread Ismaël Mejía
This is super interesting, great work Kai! Just for curiosity, How are you validating this? It would be really interesting to have this also as part of some kind of IT for the future. On Fri, Jun 1, 2018 at 7:43 PM Kai Jiang wrote: > Sounds a good idea! I will file the major problems later

Re: [Proposal] Apache Beam Swag Store

2018-06-01 Thread Ismaël Mejía
Any news on this ? On Tue, Nov 7, 2017 at 1:46 AM Matthias Baetens wrote: > > +1 > Awesome stuff, thanks for the effort Gris! > > On Mon, Nov 6, 2017 at 5:31 AM, Griselda Cuevas > wrote: > > > Excellent - we're getting approval from the trademark@ folks. I'll update > > the thread when I have

Re: Documenting Github PR jenkins trigger phrases

2018-06-01 Thread Ismaël Mejía
Any progress on this ? Is there a JIRA to track it ? On Fri, May 11, 2018 at 6:02 PM Jean-Baptiste Onofré wrote: > > Agree, I thought there's already a PR about that. > > Regards > JB > > On 11/05/2018 16:11, Alexey Romanenko wrote: > > +1 to add such reference guide of Jenkins commands to

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

2018-06-01 Thread Ismaël Mejía
Congratulations! On Fri, Jun 1, 2018 at 8:26 AM Pei HE wrote: > > Congrats! > > On Fri, Jun 1, 2018 at 2:12 PM, Charles Chen wrote: > > Congratulations everyone! > > > > > > On Thu, May 31, 2018, 10:14 PM Pablo Estrada wrote: > >> > >> Thanks to the PMC! Very humbled and excited to keep taking

Re: Reducing Committer Load for Code Reviews

2018-05-31 Thread Ismaël Mejía
If I understood correctly what is proposed is: - Committers to be able to have their PRs reviewed by non-committers and be able to self-merge. - For non-committers nothing changes. This enables a committer (wearing contributor head) to merge their own changes without committer approval, so we

<    1   2   3   4   5   6   7   8   >