Re: Going on leave for a bit

2018-06-26 Thread Ismaël Mejía
Enjoy your family time. Best wishes, Ismael On Tue, Jun 26, 2018 at 12:13 PM Pei HE wrote: > (A late) Congrats for the newborn! > -- > Pei > > On Tue, Jun 26, 2018 at 1:42 PM, Kenneth Knowles wrote: > > Hi friends, > > > > I think I did not mention on dev@ at the time, but my child #2 arrived

Re: scala/scio

2018-06-25 Thread Ismaël Mejía
​Agree with Kenn, granularity would be key for per-module Beam releases. We can easily think today of releasing separately IOs and extensions because they depend only on the public API of the SDK which has not changed much since 2.0.0. So scio will probably fit this category. A different story wo

Re: [PROPOSAL] Merge samza-runner to master

2018-06-25 Thread Ismaël Mejía
+1 It is important to have new runners merged (even if not 100% complete) so they benefit of the fixes going on, and that they can easily (and incrementally) start to track the new portability features as they develop. What is next then ? Who triggers the green button so this happens? On Sat,

Re: Auto closing stale PRs label

2018-06-25 Thread Ismaël Mejía
; been watching it and didn't see any that seemed wrong. > > Kenn > > > On Mon, Jun 25, 2018 at 12:52 AM Ismaël Mejía wrote: > >> I saw some PRs auto closed recently and I was wondering if we could >> adjust the label that is added to the autoclosed PRs, currentl

Auto closing stale PRs label

2018-06-25 Thread Ismaël Mejía
I saw some PRs auto closed recently and I was wondering if we could adjust the label that is added to the autoclosed PRs, currently it is 'wontfix' but this label sends a fake (and negative) message. Can we parametrize the bot to put something closer to the intention like 'autoclosed'? Who can ta

Re: [FYI] New Apache Beam Swag Store!

2018-06-13 Thread Ismaël Mejía
Great ! Thanks Gris and Matthias for putting this in place. Hope to get that hoodie soon. As a suggestion, more colors too, and eventually a t-shirt just with the big B logo. On Mon, Jun 11, 2018 at 6:50 PM Mikhail Gryzykhin wrote: > > That's nice! > > More colors are appreciated :) > > --Mikhail

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-12 Thread Ismaël Mejía
Yes, ok I was not aware it was already being addressed, nice. On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay wrote: > > Ismaël, > > I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing the > issue by changing the findbugs back to "com.github.stephenc.findbugs". Is > this what you a

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-12 Thread Ismaël Mejía
There is another issue highlighted by Scott Wegner in a non-related to the vote PR discussion. https://github.com/apache/beam/pull/5540 It seems that in the migration to gradle we changed the findbugs-annotations library from com.github.stephenc.findbugs to com.google.code.findbugs:findbugs-annota

Re: [Proposal] Automation For Beam Dependency Check

2018-06-04 Thread Ismaël Mejía
Is there a way to add to that weekly report the new dependencies that were introduced in the week before, or that have changed? We are not addressing another important problem: Leaking of dependencies. I am not aware of the gradle equivalent of the maven dependency plugin that helps to determine m

Re: [SQL] Unsupported features

2018-06-04 Thread Ismaël Mejía
This is super interesting, great work Kai! Just for curiosity, How are you validating this? It would be really interesting to have this also as part of some kind of IT for the future. On Fri, Jun 1, 2018 at 7:43 PM Kai Jiang wrote: > Sounds a good idea! I will file the major problems later and

Re: [Proposal] Apache Beam Swag Store

2018-06-01 Thread Ismaël Mejía
Any news on this ? On Tue, Nov 7, 2017 at 1:46 AM Matthias Baetens wrote: > > +1 > Awesome stuff, thanks for the effort Gris! > > On Mon, Nov 6, 2017 at 5:31 AM, Griselda Cuevas > wrote: > > > Excellent - we're getting approval from the trademark@ folks. I'll update > > the thread when I have new

Re: Documenting Github PR jenkins trigger phrases

2018-06-01 Thread Ismaël Mejía
Any progress on this ? Is there a JIRA to track it ? On Fri, May 11, 2018 at 6:02 PM Jean-Baptiste Onofré wrote: > > Agree, I thought there's already a PR about that. > > Regards > JB > > On 11/05/2018 16:11, Alexey Romanenko wrote: > > +1 to add such reference guide of Jenkins commands to Testing

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

2018-06-01 Thread Ismaël Mejía
Congratulations! On Fri, Jun 1, 2018 at 8:26 AM Pei HE wrote: > > Congrats! > > On Fri, Jun 1, 2018 at 2:12 PM, Charles Chen wrote: > > Congratulations everyone! > > > > > > On Thu, May 31, 2018, 10:14 PM Pablo Estrada wrote: > >> > >> Thanks to the PMC! Very humbled and excited to keep taking

Re: Reducing Committer Load for Code Reviews

2018-05-31 Thread Ismaël Mejía
If I understood correctly what is proposed is: - Committers to be able to have their PRs reviewed by non-committers and be able to self-merge. - For non-committers nothing changes. This enables a committer (wearing contributor head) to merge their own changes without committer approval, so we sho

Re: parquet/beam

2018-05-31 Thread Ismaël Mejía
If I understand correctly Arrow allows a common multi language in-memory data representation, so basically it is a columnar data format that you can use to transfer data betweeen libraries in python (pandas, numpy, etc), Java and other languages. This avoids the round-trip to disk to do so. So we s

Re: Reducing Committer Load for Code Reviews

2018-05-31 Thread Ismaël Mejía
+1 This will help reviews go faster. And in the IO reviews makes extra sense, because a common need is to ping external people who are not committers but experts in the respective data stores. Of course this puts more trust in the committers but makes sense. On Thu, May 31, 2018 at 3:46 PM Kennet

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

2018-05-31 Thread Ismaël Mejía
Great ! Matthias I think it makes sense to have these guidelines in the beam site better than a google doc. Can you please submit a PR for this? On Thu, May 31, 2018 at 8:03 AM Matthias Baetens wrote: > > Hey Eugene, hi all! > > Happy to say your talk is now on the Beam YouTube channel and can b

Re: Launching a Portable Pipeline

2018-05-23 Thread Ismaël Mejía
Interesting document, two questions: 1. Why JobService is runner specific? Couldn't at least a good part of it be reused given that the runner specific parts are mostly in the translation? or I am missing other reasons? 2. What about authentication and authorisation for production runners ? Once

Re: Closing (automatically?) inactive pull requests

2018-05-23 Thread Ismaël Mejía
Given that reaching consensus in both communities seems like a harder task than just deciding our policy. in the Beam side Why don't we just go ahead and vote around this + build the tool, and if the Flink guys are interested they can take it, no? in the future we can share that code. On Wed, May

Re: Proposal: keeping precommit times fast

2018-05-23 Thread Ismaël Mejía
I second Robert idea of ‘inteligently’ running only the affected tests, probably there is no need to run Java for a go fix (and eventually if any issue it can be catched in postcommit), same for a dev who just fixed something in KafkaIO and has to wait for other IO tests to pass. I suppose that lan

Re: The full list of proposals / prototype documents

2018-05-23 Thread Ismaël Mejía
+1 and thanks for volunteering for this Alexey. We really need to make this more accesible. On Wed, May 23, 2018 at 6:00 PM Alexey Romanenko wrote: > Joseph, Eugene - thank you very much for the links! > All, regarding one common entry point for all design documents. Could we just have a dedicat

Re: [VOTE] Go SDK

2018-05-22 Thread Ismaël Mejía
+1 (binding) Go SDK brings new language support for a community not well supported in the Big Data world the Go developers, so this is a great. Also the fact that this is the first SDK integrated with the portability work makes it an interesting project to learn lessons from for future languages.

Re: I'm back and ready to help grow our community!

2018-05-22 Thread Ismaël Mejía
I missed somehow this email thread. Congratulations Gris and welcome back! On Fri, May 18, 2018 at 5:34 AM Jesse Anderson wrote: > Congrats! > On Thu, May 17, 2018, 6:44 PM Robert Burke wrote: >> Congrats & welcome back! >> On Thu, May 17, 2018, 5:44 PM Huygaa Batsaikhan wrote: >>> Welcome

What is the Impulse and why do we need it?

2018-05-18 Thread Ismaël Mejía
I have seen multiple mentions of 'Impulse' in JIRAs and some on other discussions, but have not seen any document or concrete explanation on what's Impulse and why we need it. This seems like an internal implementation detail but it is probably a good idea to explain it somewhere (my excuses if thi

What is the future of Reshuffle?

2018-05-18 Thread Ismaël Mejía
I saw in a recent thread that the use of the Reshuffle transform was recommended to solve an user issue: https://lists.apache.org/thread.html/87ef575ac67948868648e0a8110be242f811bfff8fdaa7f9b758b933@%3Cdev.beam.apache.org%3E I can see why it may fix the reported issue. I am just curious about the

Re: [DISCUSS] Remove findbugs from sdks/java

2018-05-18 Thread Ismaël Mejía
ng has already started on Spark [4] (thanks!) >>> [1] https://github.com/apache/beam/pull/5319 >>> [2] https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20%3D%20Open%20AND%20labels%20%3D%20errorprone >>> [3] https://issues.apache.org/jira/browse/BEA

Re: [DISCUSS] Remove findbugs from sdks/java

2018-05-17 Thread Ismaël Mejía
+0.7 also. Findbugs support for more recent versions of Java is lacking and the maintenance seems frozen in time. As a possible plan b can we identify the missing important validations to identify how much we lose and if it is considerable, maybe we can create a minimal configuration for those, an

Re: Wait.on() - "Do this, then that" transform

2018-05-17 Thread Ismaël Mejía
This sounds super interesting and useful ! Eugene can you please elaborate on this phrase 'has to return a result that can be waited on'. It is not clear for me what this means and I would like to understand this to evaluate what other IOs could potentially support this. On Thu, May 17, 2018 at

Re: Can I get the contributor role?

2018-05-11 Thread Ismaël Mejía
Done and welcome! On Fri, May 11, 2018 at 9:38 AM Carlos Alonso wrote: > Hi everyone!! > I'm working on https://issues.apache.org/jira/browse/BEAM-4257 and I'd like to get the task assigned. > Can a PMC for the project add me as a contributor and assign me the ticket? > Thanks!

Re: Pubsub to Beam SQL

2018-05-10 Thread Ismaël Mejía
Hi, Jumping a bit late to this discussion. This sounds super nice. But I could not access the document. How hard would it be to do this for other 'unbounded' sources, e.g. Kafka ? On Sat, May 5, 2018 at 2:56 AM Andrew Pilloud wrote: > I don't think we should jump to adding a extension, but TBLPRO

Re: Graal instead of docker?

2018-05-05 Thread Ismaël Mejía
Graal would not be a viable solution for the reasons Henning and Andrew mentioned, or put in other words, when users choose a programming language they don’t choose only a ‘friendly’ syntax or programming model, they choose also the ecosystem that comes with it, and the libraries that make their li

Re: Apache Beam Jenkins Machines Upgrade

2018-04-27 Thread Ismaël Mejía
This is fantastic. I saw the reduction of the build times in Jenkins today, they reduced dramatically. Excellent work guys congratulations! On Fri, Apr 27, 2018 at 3:00 PM Etienne Chauchot wrote: > Awesome ! Very good news ! Thanks for your work guys ! > Etienne > Le vendredi 27 avril 2018 à 00

Re: Maven build broken

2018-04-26 Thread Ismaël Mejía
Thanks Thomas for the fix, Agree 100% maven should still work until the decision to drop it off is taken. On Thu, Apr 26, 2018 at 5:51 PM Thomas Weise wrote: > Here is a PR to fix it: > https://github.com/apache/beam/pull/5232 > I think that until switch to gradle is complete, at least "mvn ve

Re: Beam SQL Improvements

2018-04-26 Thread Ismaël Mejía
Hello Anton, Thanks for the descriptive email and the really useful work. Any plans to tackle PCollections of GenericRecord/IndexedRecords? it seems Avro is a natural fit for this approach too. Regards, Ismaël On Wed, Apr 25, 2018 at 9:04 PM, Anton Kedin wrote: > Hi, > > I want to highlight a c

Re: Request to become contributor and have BEAM-4096 assigned

2018-04-23 Thread Ismaël Mejía
Done, assigned the ticket. You now have the contributor role so you can self-assign other tickets in the future. Welcome! On Mon, Apr 23, 2018 at 5:34 AM, Jean-Baptiste Onofré wrote: > Welcome aboard ! > > Pretty sure a PMC member will do the change in Jira quickly (unfortunately > I don't have

Re: [Go SDK] Proposal: Set up a Vanity Import Path

2018-04-18 Thread Ismaël Mejía
t's good to know! > I had heard of that specific rule, but I didn't realized it pertained to > filing of a JIRA issue (when related to a proposal) as well. > Thank you. > > On Wed, 18 Apr 2018 at 13:08 Ismaël Mejía wrote: >> >> +1 Nice idea and proposal. >

Re: [Go SDK] Proposal: Set up a Vanity Import Path

2018-04-18 Thread Ismaël Mejía
+1 Nice idea and proposal. This was not a vote thread but for the future it is a good idea to let a bigger time window before reaching consensus. Notice that a formal vote lets at least 72h for participants to voice their opinion before concluding something. https://www.apache.org/foundation/voti

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-18 Thread Ismaël Mejía
Most criticla pending issues to test before a gradle based release is that those generated poms are equal to the maven ones. Which means that they don't add by mistake compile dependencies where they were not. It is also critical to validate that all the meta-info is generated as it was done with

Re: Add a (temporary) Portable Flink branch to the ASF repo?

2018-04-18 Thread Ismaël Mejía
+1 to bring this to ASF repo ASAP. Also +1 to Aljoscha's idea if we can guarantee no interference. An extra branch has the issue of rebasing that will be 'automatically' solved if the development continues in master. Also other it will be easier to more contributors to jump in. On Wed, Apr 18, 20

Re: Beam7 Outage

2018-04-12 Thread Ismaël Mejía
beam5 has been failing in the last week. Almost all builds there break. On Thu, Apr 12, 2018, 6:41 PM Jason Kuster wrote: > Hi all, > > The Jenkins Beam7 executor gave up the ghost some time in the last couple > of days. I've been on the line with Infra yesterday and today getting it > fixed, an

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-11 Thread Ismaël Mejía
+1 to delay 2 weeks as Ahmet proposes. We can cut the branch in two weeks in a more stable shape doing it right now is not a good idea. We can justify the delay on the impact of transitioning the build system. On Wed, Apr 11, 2018 at 10:35 PM, Ahmet Altay wrote: > +1 to delaying 2 weeks. > > I t

Re: Development of new HTTP source

2018-04-11 Thread Ismaël Mejía
There is also Romain's filesystem PR that wraps vfs as a Beam filesystem . That would at least in theory allow to read from HTTP and FTP, no? Of course this is different from RestIO because it won't have the full HTTP verbs semantics but if the goal is just to read from (GET), maybe we should try

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-04-10 Thread Ismaël Mejía
+1000 to Romain's point on dependencies, we have to obsessively pay attention to the consistency of the dependencies, this is critical for users and we cannot radically change the produced artifacts or we risk of breaking their applications.. On Tue, Apr 10, 2018 at 6:56 AM, Romain Manni-Bucau w

Re: DirectRunner in test - await completion of workers threads?

2018-04-10 Thread Ismaël Mejía
It seems there is still an issue with teardown not being called in failed tasks, just created BEAM-4040 to track it. On Thu, Apr 5, 2018 at 4:45 PM, Tim Robertson wrote: > Will do - I'll report the result on https://github.com/apache/beam/pull/4905 > > On Thu, Apr 5, 2018 at 11

Re: Gradle Tips and tricks

2018-04-06 Thread Ismaël Mejía
7;s what I mentioned this morning on Slack. >> >> I think it would be great to have a summary of the current state on the >> dev >> mailing list. >> >> At the end of the day, the contribution guide should be updated (we have a >> Jira >> about that afair).

Gradle Tips and tricks

2018-04-06 Thread Ismaël Mejía
After some discussion on slack it is clear that we need to document some of the gradle replacements of our common maven commands. We started a shared doc yesterday to share some of those and other gradle tips and tricks. I invite everyone who can help to add their favorite gradle 'incantations' the

Re: DirectRunner in test - await completion of workers threads?

2018-04-05 Thread Ismaël Mejía
For info, Romain's PR was merged today, can you confirm if this fixes the issue Tim. On Sun, Apr 1, 2018 at 9:21 PM, Tim Robertson wrote: > Thanks all. > > I went with what I outlined above, which you can see in this test. > https://github.com/timrobertson100/beam/blob/BEAM-3848/sdks/java/io/solr

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

2018-04-05 Thread Ismaël Mejía
Nice, I created a short link so people can refer to it easily in future discussions, website, etc. https://s.apache.org/beam-fn-api-metrics Thanks for sharing. On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw wrote: > Thanks for the nice writeup. I added some comments. > > On Wed, Apr 4, 2018

Re: [PROPOSAL] Scripting extension based on Java JSR-223

2018-03-23 Thread Ismaël Mejía
are welcome so please to the other people in the community, feel free to add those here for discussion then we can bootstrap a better design document. On Fri, Mar 23, 2018 at 8:32 PM, Thomas Weise wrote: > +1, nice! > > On Fri, Mar 23, 2018 at 4:03 AM, Ismaël Mejía wrote: >> >

[PROPOSAL] Scripting extension based on Java JSR-223

2018-03-23 Thread Ismaël Mejía
This is a really simple proposal to add an extension with transforms that package the Java Scripting API )JSR-223) [1] to allow users to specialize some transforms via a scripting language. This work was initially created by Romain [2] and I just took it with his authorization and refined it to mak

Re: Gradle status

2018-03-22 Thread Ismaël Mejía
I don't think that removing all maven descriptors was the expected path, no ? Or even a good idea at this moment. I understood that what we were going to do was to replace incrementally the CI until we cover the whole maven functionality and then remove it, from looking at the JIRA ticket https://

Re: [VOTE] Release 2.4.0, release candidate #3

2018-03-20 Thread Ismaël Mejía
+1 (binding) - Validated hashs - mvn clean verify -Prelease OK - Run nexmark on direct/flink/spark (it works save the regression already tracked on RC2). Thanks Robert for being managing the release. ps. One 'future' question after seeing the wheel artifacts (that seem to be python version / OS

Re: Configuring file-based transforms with different options

2018-03-09 Thread Ismaël Mejía
>> >>> Long story short with hints you should be able to say "use that >>> specialize config here". >>> >>> Now, personally, I'd like to see a way to specialize config per >>> transform. With an hint an easy way is to use a pr

Re: The Go SDK got accidentally merged - options to deal with the pain

2018-03-09 Thread Ismaël Mejía
+1 to let it evolve in master (+Davor points), having ongoing work on master makes sense given the state of advance + the hope that this won't add any issue for the other modules. On Thu, Mar 8, 2018 at 7:30 PM, Robert Bradshaw wrote: > +1 to both of these points. SGA should have probably already

Configuring file-based transforms with different options

2018-03-08 Thread Ismaël Mejía
I was trying to create a really simple pipeline that read from a bucket in a filesystem (s3) and writes to a different bucket in the same filesystem. S3Options options = PipelineOptionsFactory.fromArgs(args).create().as(S3Options.class); Pipeline pipeline = Pipeline.create(options); pi

Re: [VOTE] Release 2.4.0, release candidate #2

2018-03-08 Thread Ismaël Mejía
I confirm that the new release fixes both problems reported previously: - python package name - nexmark query 10 mutability issue with the direct runner. One extra regression is that the the fix produced a way longer execution time on the query. Not sure if a blocker but worth tracking. Query 10

Re: [YouTube channel] Add video: Apache Beam meetup London 2: use case in finance + IO in Beam and Splittable DoFns

2018-03-08 Thread Ismaël Mejía
+1 I was wondering if we can also add a playlist that links to presentations Beamers have done in different conferences, e.g. some of the public available talks from the past by Frances/Tyler/others are worth to be included so they can be easily found. (of course not sure if we need approval from

Re: [VOTE] Release 2.4.0, release candidate #1

2018-03-07 Thread Ismaël Mejía
Hello, So far I found two issues: 1. The python zip file includes the name apache-beam-2.4.0.dev0 instead of apache-beam-2.4.0 not sure if this is a big issue but when I installed and tested it via pip I saw the .dev0 suffix still. 2. Direct runner has a regression on Nexmark’s query 10. I filled

Re: Beam 2.4.0

2018-03-01 Thread Ismaël Mejía
The average time just in the vote process for Beam since we are out of the incubator is 17.5 days with an average of 75 days between versions. Version Vote Period No. days 2.3.030/01-16/02 17 days (83 days since last) 2.2.027/10-25/11 29 days (101 days since last) 2.1.011/07-16

Re: Fwd: Google Summer of Code 2018 is coming

2018-02-26 Thread Ismaël Mejía
Does this mean that we can still propose subjects? I thought the deadline was over, when is it the deadline? On Mon, Feb 26, 2018 at 5:17 PM, Kenneth Knowles wrote: > Hi Kai, > > Thanks for writing this! I would love to be a mentor for GSoC if it falls > within my expertise. > > As a reminder to

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

2018-02-21 Thread Ismaël Mejía
For info, this issue is tracked and the fix is in progress: https://issues.apache.org/jira/browse/BEAM-3720 On Tue, Feb 20, 2018 at 4:43 PM, Jean-Baptiste Onofré wrote: > I would wait a feedback from the original author of the commit. > > Regards > JB > Le 20 févr. 2018, à 16:42, Alexey Romanen

Re: @TearDown guarantees

2018-02-21 Thread Ismaël Mejía
cleaning up in-process resources (open connections, etc.), >>>> it's not the right answer for cleaning up persistent resources. If you rely >>>> on TearDown to delete VMs or delete files, there will be cases in which >>>> those files of VMs are not

Re: @TearDown guarantees

2018-02-19 Thread Ismaël Mejía
I also had a different understanding of the lifecycle of a DoFn. My understanding of the use case for every method in the DoFn was clear and perfectly aligned with Thomas explanation, but what I understood was that in a general terms ‘@Setup was where I got resources/prepare connections and @Teard

Re: [PROPOSAL] Add a blog post for every new release

2018-02-16 Thread Ismaël Mejía
's >>> not >>> systematic. >>> >>> So, I agree: it's a good idea and I would give some highlights about what >>> we are >>> doing and where we are heading. However, I don't think we have to "enforce" >>> su

Re: PipelineOptions fromSystemProps?

2018-02-14 Thread Ismaël Mejía
+1 to Romain idea with Kenn's approach, we should probably reserve the 'beam.' namespace too because we are already using it for system properties in the spark runner, following along the convention of 'spark.*' in Apache Spark. Any ideas on how to handle this for the env vraiables case ? maybe th

Re: [VOTE] Release 2.3.0, release candidate #3

2018-02-14 Thread Ismaël Mejía
+1 (binding) Validated SHAs + tag vs source.zip file. Run mvn clean install -Prelease OK Validated that the 3 regressions reported for RC1 were fixed. Run Nexmark on Direct/Flink runner on local mode, no regressions now. Installed python version on virtualenv and run local wordcount with success.

Re: Add Errorprone to build process?

2018-02-13 Thread Ismaël Mejía
> On Tue, Feb 13, 2018, 12:23 PM Ismaël Mejía wrote: >> >> Kenn submitted a PR for this yesterday. So I assume is taken. >> https://github.com/apache/beam/pull/4667 >> >> On Tue, Feb 13, 2018 at 8:49 PM, Eugene Kirpichov >> wrote: >> > Filed https

Re: Add Errorprone to build process?

2018-02-13 Thread Ismaël Mejía
Kenn submitted a PR for this yesterday. So I assume is taken. https://github.com/apache/beam/pull/4667 On Tue, Feb 13, 2018 at 8:49 PM, Eugene Kirpichov wrote: > Filed https://issues.apache.org/jira/browse/BEAM-3697 > > This seems to be a good task for a new contributor, easy and potentially > wi

Re: Proposal: build Python wheel distributions for Apache Beam releases

2018-02-12 Thread Ismaël Mejía
+1 for wheels, they are the standard binary distribution format so it makes sense. Also wheels support packaging python 2 and 3 on universal packages so they are future proof. On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw wrote: > +1, is it too late to try to release these as part of the 2.3

Re: [VOTE] Release 2.3.0, release candidate #3

2018-02-12 Thread Ismaël Mejía
> Romain Manni-Bucau > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > 2018-02-12 17:32 GMT+01:00 Reuven Lax : >> >> If it's possible to just publish the module separately, that seems better >> than RC4 >> >> On Mon, Feb 12, 2018 at 8:28

Re: [VOTE] Release 2.3.0, release candidate #3

2018-02-12 Thread Ismaël Mejía
Sorry the skip was an error while merging this module with the tests during the move to Java 1.8. I will create a JIRA + PR, but wonder if we can somehow just publish this artifact to avoid creating a new RC+vote.. On Mon, Feb 12, 2018 at 5:22 PM, Jean-Baptiste Onofré wrote: > But that's it: depl

Re: [VOTE] Release 2.3.0, release candidate #2

2018-02-09 Thread Ismaël Mejía
> Do we have a release validation spreadsheet for >>> > this one? >>> > > >>> > > On Thu, Feb 8, 2018 at 9:30 AM Ahmet Altay >>> > mailto:al...@google.com> >>> >

Re: A 15x speed-up in local Python DirectRunner execution

2018-02-08 Thread Ismaël Mejía
Sounds impressive, and with the extra portability stuff, great ! Worth the switch just for he user experience improvement. On Thu, Feb 8, 2018 at 5:52 PM, Robert Bradshaw wrote: > This is going to be a great improvement for our users! I'll take a > look at the pull request. > > On Wed, Feb 7, 201

Re: [VOTE] Release 2.3.0, release candidate #2

2018-02-08 Thread Ismaël Mejía
+1 (binding) Validated SHAs + tag vs source.zip file. Run mvn clean install -Prelease OK Validated that the 3 regressions reported for RC1 were fixed. Run Nexmark on Direct/Flink runner on local mode, no regressions now. Installed python version on virtualenv and run wordcount with success. On Th

Re: [PROPOSAL] Switch from Guava futures vs Java 8 futures

2018-02-01 Thread Ismaël Mejía
+1 On Thu, Feb 1, 2018 at 9:53 PM, Romain Manni-Bucau wrote: > +1 indeed > > Le 1 févr. 2018 21:34, "Eugene Kirpichov" a écrit : >> >> Reducing dependency on Guava in favor of something Java-standard sounds >> great, +1. >> >> On Thu, Feb 1, 2018 at 11:53 AM Reuven Lax wrote: >>> >>> Unless the

Re: [VOTE] Release 2.3.0, release candidate #1

2018-02-01 Thread Ismaël Mejía
Hi, I started to test the release with Nexmark and found three issues (from minor to more important): 1. Small issues to run Nexmark with the release (BEAM-3531 fixed, BEAM-3592 in PR): BEAM-3531 Nexmark failed with NPE with DEFAULT suite BEAM-3592 Spark-runner profile is broken on Nexmark after

Re: FOSDEM mini office hour?

2018-02-01 Thread Ismaël Mejía
Excellent idea, I will be presenting at FOSDEM too. We can meet at the time you propose, so we can discuss with more people interested on Beam. We need just to define a meeting place. I will be also around the ASF booth on saturday if any other person wants to talk about Beam or ASF related stuff

Re: [DISCUSS] [Java] Private shaded dependency uber jars

2018-02-01 Thread Ismaël Mejía
* you only get the shaded dependencies of the >>> >> things that >>> >> are required >>> >> * its one less dependency for users to manage >>> >> Risks >>> >>

Re: [VOTE] Release 2.3.0, release candidate #1

2018-01-31 Thread Ismaël Mejía
What is the common procedure in cases like this ? Because it doesn't seems that it needs a re-vote, just an extra day or two for validation, any ideas JB ? On Wed, Jan 31, 2018 at 10:41 PM, Alan Myrvold wrote: > Yes, it is a dataflow step. Happy to test this again when they are > available. > > O

Re: Should we have a predictable test run order?

2018-01-31 Thread Ismaël Mejía
Is the conclusion of this thread is that we should then make the test execution random, remember that currently it uses the default order that is filesystem-based as Dan mentioned and that produces some minor inconsistencies between mac/linux. It is going to be interesting to see how much extra fl

Re: [DISCUSS] State of the project: Culture and governance

2018-01-31 Thread Ismaël Mejía
r option to >> explore. >> >> 3. You got me point: it's subjective. The link to Flink doesn't bring >> anything >> more: "There is no strict protocol for becoming a committer". So, no >> problem to >> add such section, but I don'

Re: [DISCUSSION] Add hint/option on PCollection

2018-01-31 Thread Ismaël Mejía
This is a subject we have already discussed in the past. It was part on the discussion on ‘data locality’ for the runners on top of HDFS. In that moment the argument for ‘hints’ was that a transform could send hints to the runners so they properly allocate the readers improving its execution. This

Re: [DISCUSS] State of the project: Culture and governance

2018-01-29 Thread Ismaël Mejía
>>> disagree >>> about publishing the criteria to earn committership, and even more for >>> PMC. As >>> already said, a contribution can have many forms, so, criteria in term of >>> number >>> can be inaccurate. >>> >>> As these subje

[PROPOSAL] Add a blog post for every new release

2018-01-29 Thread Ismaël Mejía
This is a fork of a recent message I sent as part of the preparations for the next release. [tl;dr] I would like to propose that we create a new blog post for every new release and that this becomes part of the release guide. I think that even if we do shorter releases we need to make this part o

Re: [HEADS UP] Preparing Beam 2.3.0

2018-01-25 Thread Ismaël Mejía
I saw some recent reports on issues with CassandraIO that are not blockers (not data loss) but IMO deserve to be included because basically the issues imply that users cannot read from Cassandra in parallel, and they were reported by production users. Probably a good idea to finish these before the

Re: [Proposal] Apache Beam Event's Calendar

2018-01-25 Thread Ismaël Mejía
+1 I think it makes sense to separate it the calendar in two, one for the CFPs more interesting for dev@ and one for confirmed events where there will be presentations on Beam that concerns more the users (user@). It also probably makes sense to include this one in the website too. On Thu, Jan

Re: Gradle / Mvn diff

2018-01-23 Thread Ismaël Mejía
Big +1 to Kenneth's idea on a cookbook for common tasks, it is easy to feel frustrated with gradle when we cannot achieve easily some of the tasks we need to, like overwriting a configuration property. The case that Romain mentioned above is a common one, e.g. I used to test kafka with a different

[DISCUSS] State of the project: Culture and governance

2018-01-23 Thread Ismaël Mejía
This is a sub-thread of the state of the project one initiated by Davor. Since this subject can be part of the community issues I would like to focus on the state of the project for its contributors so we don’t mix the discussion with the end-user thread. I hope other members of the community brin

Re: Eclipse support

2018-01-23 Thread Ismaël Mejía
; Thanks, > Ron > > > > Sent via the Samsung Galaxy S7 active, an AT&T 4G LTE smartphone > > Original message > From: Ismaël Mejía > Date: 1/22/18 1:29 AM (GMT-08:00) > To: dev@beam.apache.org > Subject: Re: Eclipse support > > Hello, > >

Re: Eclipse support

2018-01-22 Thread Ismaël Mejía
Hello, Thanks for bringing this info, I tried to compile with the eclipse compiler and I can confirm that it does not wok, Eclipse's JDT is more annoying about generics so it could be related to this. Filled https://issues.apache.org/jira/browse/BEAM-3508 to track it. Feel free to contribute a fi

Re: Strata Conference this March 6-8

2018-01-16 Thread Ismaël Mejía
Maybe a good idea to try to organize a Beam meetup in london in the same dates in case some of the people around can jump in and talk too. On Wed, Jan 17, 2018 at 2:51 AM, Ron Gonzalez wrote: > Works for me... > > On Tuesday, January 16, 2018, 5:45:33 PM PST, Holden Karau > wrote: > > > How woul

Re: Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5661

2018-01-16 Thread Ismaël Mejía
We just merged a fix for this, it should be OK now. Sorry for the inconvience and thanks for the fix JB. On Tue, Jan 16, 2018 at 5:07 AM, Kenneth Knowles wrote: > In case anyone is following, this is > https://issues.apache.org/jira/browse/BEAM-3438 > > Kenn > > On Mon, Jan 15, 2018 at 5:08 PM, A

Re: [DISCUSS] State of the project

2018-01-16 Thread Ismaël Mejía
Thanks Davor for opening this discussion and HUGE +1 to do this every year or in cycles. I will fork this thread into a new one for the Culture / Project management part issues as suggested. About the diversity of users across runners subject I think that this requires more attention to unificatio

Re: [DISCUSS] State of the project: Community growth

2018-01-16 Thread Ismaël Mejía
Some ideas I have around the issues mentioned before: * Annual User Survey One important thing we should do is some sort of annual survey of users to have some feedback on the state of the Beam from the users point of view, we can take for example as a template the survey done by the Rust communi

Re: Switching to Java 8

2018-01-08 Thread Ismaël Mejía
h was not supporting j8 in > current (beam dependency) version. > > > Romain Manni-Bucau > @rmannibucau | Blog | Old Blog | Github | LinkedIn > > 2018-01-08 11:33 GMT+01:00 Ismaël Mejía : >> >> Excellent news ! Probably a good idea to fill JIRAs to all of those. I >>

Re: Switching to Java 8

2018-01-08 Thread Ismaël Mejía
Excellent news ! Probably a good idea to fill JIRAs to all of those. I would add: - Remove the references in the website to Java 7 - Remove Java 7 and any related task from the CI - Update the docker dev build images (I will take this one since reproducible build is my pet project) - Upgrade the I

Re: Happy new year

2018-01-02 Thread Ismaël Mejía
Happy new year everyone ! Agree 100% with Davor. 2017 was a good year for the project and it is worth to thank everyone who helped to make the project better. Now is the time to work to have a great project in 2018 too ! Best wishes to all (and kudos to JB too for the top 5) ! Ismaël On Mon,

Re: remove avro from core?

2017-12-19 Thread Ismaël Mejía
There are two important points raised here that we are missing in the discussion: 1. We don’t have any kind of security validation for Beam and its dependencies. This is important and it would be really nice if we could get this improved and some sort of automation on this. However I doubt there i

Re: Euphoria Java 8 DSL - proposal

2017-12-18 Thread Ismaël Mejía
Hi, It is great to see that you guys have achieved a maturity point to propose this. Congratulations for your work and the idea to contribute it into Beam. I remember from a previous discussion with Jan about the model mismatch between Euphoria and Beam, because of some design decisions of both p

Re: Apache Ignite as a distributed processing back-ends

2017-12-13 Thread Ismaël Mejía
I personally think that having the code as part of Apache Beam is better: Advantages: 1. Refactorings + new functionalities for free, notice that the runner APIs are internal and still evolving due to new ideas like the portability API. 2. All the Beam infrastructure of the project for ‘free’ (CI,

<    2   3   4   5   6   7   8   >