Re: Next LTS?

2020-03-25 Thread Ismaël Mejía
gt;>>>> wrote: > >>> >>>>> > > >>> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of > >>> >>>>> > interest. > >>> >>>>> > > >

Re: FlinkRequiresStableInput test is very flaky

2020-03-24 Thread Ismaël Mejía
Just saw that Luke created https://issues.apache.org/jira/browse/BEAM-9578 so good to go on that one. On Tue, Mar 24, 2020 at 10:12 AM Ismaël Mejía wrote: > > Oh that looks like a considerable bug and could explain the not so > good performance of the Portable VR tests (if this is

Re: [PROPOSAL] Snowflake Java Connector for Apache Beam

2020-03-24 Thread Ismaël Mejía
that the FileIO part is generic enough so we can use S3 too because users can be using Snowflake in AWS too. On Tue, Mar 24, 2020 at 10:10 AM Ismaël Mejía wrote: > Great ! > It seems this pattern (COPY + parallel file read) is becoming a standard > for > 'data warehouses' we are usi

Re: FlinkRequiresStableInput test is very flaky

2020-03-24 Thread Ismaël Mejía
Oh that looks like a considerable bug and could explain the not so good performance of the Portable VR tests (if this is happening for every call as you mention). Looks like more worth of a JIRA even than the FlinkRequiresStableInputTest. On Mon, Mar 23, 2020 at 7:42 PM Reuven Lax wrote: > >

Re: [PROPOSAL] Snowflake Java Connector for Apache Beam

2020-03-24 Thread Ismaël Mejía
Great ! It seems this pattern (COPY + parallel file read) is becoming a standard for 'data warehouses' we are using something similar too in the AWS Redshift PR (WIP) for details: https://github.com/apache/beam/pull/10206 Maybe worth for all of us to check and se eif we can converge the

Re: Question on SerializableCoder

2020-03-23 Thread Ismaël Mejía
The link to the previous covnersation (discussion happened in private@ and I suppose we can bring some relevant bits here if needed) https://lists.apache.org/thread.html/2e1c00999e992e15b08938866bfe7bd3c3d3b3d4d7aa2f8f6eb4600d%40%3Cprivate.beam.apache.org%3E I remember Robert had some points

Re: Proposal: Beam to use GCP Libraries BOM

2020-03-20 Thread Ismaël Mejía
gt;>>>>> >>>>>> How would the Apache Beam BOM and GCP BOM work together? >>>>>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas >>>>>> wrote: >>>>>>> >>>>>>> Big +1, this is a step

Re: Contributor permission for Beam Jira tickets

2020-03-16 Thread Ismaël Mejía
Done, welcome! On Mon, Mar 16, 2020 at 5:26 PM Yaron Neuman wrote: > > Hi, > > This is Yaron Neuman. > I would like to contribute issues related to java-sdk and dataflow runner. > Can someone add me as a contributor for Beam's Jira issue > tracker? I would like to create/assign tickets for my

Re: Autolink references for Apache Beam

2020-03-15 Thread Ismaël Mejía
Autolink references are working now. On Fri, Mar 13, 2020 at 9:31 PM Ismaël Mejía wrote: > > Thanks Kyle for catching that, I pinged INFRA again hopefully it will > be working soon. > > On Fri, Mar 13, 2020 at 8:49 PM Kyle Weaver wrote: > > > > Sounds like a r

Re: Contributor permission for Beam Jira tickets

2020-03-15 Thread Ismaël Mejía
Done, welcome! On Sun, Mar 15, 2020 at 6:30 AM Anup Ratnakar Raikar wrote: > > Hi All, > > My JIRA username is anupraikar. Kindly add me to the issue tracker. > > Thanks and regards, > Anup > > On Sat, Mar 14, 2020 at 3:32 AM Anup Ratnakar Raikar > wrote: >> >> Hi Everyone, >> >> This is Anup

Re: Autolink references for Apache Beam

2020-03-13 Thread Ismaël Mejía
ra/browse/9475 > > On Fri, Mar 13, 2020 at 12:42 PM Ismaël Mejía wrote: >> >> Github released some months ago a new feature to configure Autolink >> references >> for Apache Beam a new github feature to have automatic links to issues in >> external systems e.g.

Autolink references for Apache Beam

2020-03-13 Thread Ismaël Mejía
Github released some months ago a new feature to configure Autolink references for Apache Beam a new github feature to have automatic links to issues in external systems e.g. JIRA for our case. https://github.blog/2019-10-14-introducing-autolink-references/ For info I filled INFRA-19967 to

Re: Contributing Twister2 runner to Apache Beam

2020-03-09 Thread Ismaël Mejía
Good points Kenn. I think we mostly agree on what has been discussed in this thread the pros/cons of having runners on our repository, but this is probably not the best moment in time to change any policy in that aspect. So if nobody objects I think we can proceed. I am OOO this week so with less

Re: Java SplittableDoFn Watermark API

2020-03-09 Thread Ismaël Mejía
his use case of short lived pipelines with not big data is not the average Beam use case, but still the numbers don’t look good to sell. On Wed, Mar 4, 2020 at 10:52 PM Luke Cwik wrote: > > > > On Wed, Mar 4, 2020 at 7:36 AM Ismaël Mejía wrote: >> >> > I think we should mov

Re: [DISCUSS] @Experimental annotations - processes and alternatives

2020-03-08 Thread Ismaël Mejía
Kenn can you adjust the script to match only source code files: `--include \*.java --include \*.py --include \*.go` otherwise it produces a lot of extra false positives due to html files and cache files. Also can we extract the full annotation as a column so we can filter/group for the full kind

Re: Install Jenkins AnsiColor plugin

2020-03-08 Thread Ismaël Mejía
Did this ever happen? If not what is blocking it? On Tue, Oct 22, 2019 at 10:13 PM Udi Meiri wrote: > > Your proposal will only affect the seed job (which doesn't do color outputs > AFAIK). > I think you want to add colorizeOutput() here: >

Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288 RC #3

2020-03-05 Thread Ismaël Mejía
+1 (binding) Verified signatures Verified that there are no conscrypt classes or binaries in jar Verified pom.xml has runtime dependency on conscrypt On Thu, Mar 5, 2020 at 9:14 PM Jean-Baptiste Onofre wrote: > > +1 (binding) > > Regards > JB > > Le 5 mars 2020 à 19:55, Luke Cwik a écrit : > >

Re: [VOTE] Upgrade gradle to 6.2

2020-03-05 Thread Ismaël Mejía
/gradle-6-whats-new/tree/master/demos/hello-gradle-6 On Tue, Feb 25, 2020 at 7:24 PM Luke Cwik wrote: > > +1 > > On Tue, Feb 25, 2020 at 12:49 AM Gleb Kanterov wrote: >> >> +1 (non-binding) >> >> On Tue, Feb 25, 2020 at 9:38 AM Ismaël Mejía wrote: >>> &

Re: [Discuss] Propose Calcite Vendor Release (1.22.0)

2020-03-05 Thread Ismaël Mejía
The calcite vote already passed so this is good to go, thanks for volunteering Rui. https://lists.apache.org/thread.html/r4962a4a2bacf481f2ee1064806b78829d96385c2e4a3c0ecb24a55a2%40%3Cdev.calcite.apache.org%3E On Thu, Mar 5, 2020 at 8:10 AM Kai Jiang wrote: > > Thanks, Rui! Big +1 for calcite

Re: Proposal: Beam to use GCP Libraries BOM

2020-03-05 Thread Ismaël Mejía
+1 Sounds like a good improvement for users and maintainers ! On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel wrote: > > +1, I can remember the countless hours that we fought with Google > dependencies. > > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath wrote: >> >> +1 for this. >> >> This will

Re: Java SplittableDoFn Watermark API

2020-03-04 Thread Ismaël Mejía
r Java users) and (2) that the portability abstractions are mature. We are getting there, but not yet there. On Tue, Mar 3, 2020 at 6:57 PM Robert Bradshaw wrote: > > On Tue, Mar 3, 2020 at 9:11 AM Ismaël Mejía wrote: > > > > > the unification of bounded/unbounded within Spli

Re: Java SplittableDoFn Watermark API

2020-03-04 Thread Ismaël Mejía
nner (even if in a naive way) and unlock IO authors to develop IOs that could be tested easily, it looks like worth to have a default override for the Unbounded case. On Tue, Mar 3, 2020 at 7:03 PM Luke Cwik wrote: > > > > On Tue, Mar 3, 2020 at 9:11 AM Ismaël Mejía wrote: >> &g

Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288

2020-03-03 Thread Ismaël Mejía
-1 It seems we still are adding the conscrypt native libraries as part in META-INF $ jar tvf beam-vendor-grpc-1_26_0-0.3.jar | grep conscrypt 2218176 Mon Sep 17 10:36:24 CEST 2018 META-INF/native/libconscrypt_openjdk_jni-linux-x86_64.so 1720832 Mon Sep 17 10:36:24 CEST 2018

Re: Java SplittableDoFn Watermark API

2020-03-03 Thread Ismaël Mejía
dk/transforms/Watch.java#L757 > > 2: > > https://github.com/apache/beam/blob/a16725593b84b84b37bc67cd202d1ac8b724c6f4/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L275 > > 3: > > https://github.com/apache/beam/blob/fb42666a4e1aec0413f161c742d8f010ef9fe9f2/sdks/j

Re: Python Static Typing: Next Steps

2020-03-03 Thread Ismaël Mejía
+1 to do it by default. Great to see the typing work arrive to this maturity milestone. We can also refer to some of the mypy typing docs for newbies on the subject. On Tue, Mar 3, 2020 at 10:15 AM Kamil Wasilewski wrote: > > +1 for enabling mypy as a precommit job > > This however could be a

Re: KafkaIO: Configurable timeout for setupInitialOffset()

2020-03-02 Thread Ismaël Mejía
Done, also assigned the issue you mentioned in the previous email to you. On Mon, Mar 2, 2020 at 12:56 PM Jozef Vilcek wrote: > Recently I had a problem with Beam pipeline unable to start due to > unhealthy broker in the list of configured bootstrap servers. I have > created a JIRA for it and

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

2020-02-29 Thread Ismaël Mejía
I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 5 of which are binding: * Luke Cwik * Pablo Estrada * Robert Bradshaw * Ismaël Mejía * Kenneth Knowles There are no disapproving votes. Thanks everyone! On Fri, Feb 28, 2020 at 6:39 PM Kenneth

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

2020-02-28 Thread Ismaël Mejía
Congratulations Kamil! On Fri, Feb 28, 2020 at 7:09 PM Yichi Zhang wrote: > Congrats, Kamil! > > On Fri, Feb 28, 2020 at 9:53 AM Valentyn Tymofieiev > wrote: > >> Congratulations, Kamil! >> >> On Fri, Feb 28, 2020 at 9:34 AM Pablo Estrada wrote: >> >>> Hi everyone, >>> >>> Please join me and

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

2020-02-28 Thread Ismaël Mejía
30 AM Luke Cwik wrote: > >> > >> +1 (binding) > >> > >> Verified signatures and contents of jar to not contain module-info.class > >> > >> On Wed, Feb 26, 2020 at 10:45 AM Kai Jiang wrote: > >>> > >>> +1 (non-binding) > &

Re: Java SplittableDoFn Watermark API

2020-02-28 Thread Ismaël Mejía
transform Polling patterns? https://github.com/apache/beam/blob/650e6cd9c707472c34055382a1356cf22d14ee5e/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java#L178 On Fri, Feb 28, 2020 at 10:47 AM Ismaël Mejía wrote: > Really interesting! Implementing correctly the watermark

Re: Java SplittableDoFn Watermark API

2020-02-28 Thread Ismaël Mejía
Really interesting! Implementing correctly the watermark has been a common struggle for IO authors, to the point that some IOs still have issues around that. So +1 for this, in particular if we can get to reuse common patterns. I was not aware of Boyuan's work around this, really nice. One aspect

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

2020-02-28 Thread Ismaël Mejía
One interesting variable that has not being mentioned is what versions of python 3 are available to users via their distribution channels (the linux distributions they use to develop/run the pipelines). - RHEL 8 users have python 3.6 available - RHEL 7 users have python 3.6 available - Debian

gcr.io images clean up

2020-02-27 Thread Ismaël Mejía
We ended up looking at the gcr.io images due to some issue with tests today (due to the prefix change in the docker images (beam_) with Michał and we noticed that there are literally hundreds of them, do we have clean up jobs for those?

Re: Docker images are migrated to Apache org.

2020-02-27 Thread Ismaël Mejía
at 2:30 PM Ismaël Mejía wrote: > Great news, Thanks Hannah for all your attention to this issue, let's not > forget to thank the INFRA guys for their help. > > On Tue, Feb 25, 2020 at 2:03 AM Hannah Jiang > wrote: > >> Thanks everyone for reporting issues and potential prob

Re: Docker images are migrated to Apache org.

2020-02-27 Thread Ismaël Mejía
Great news, Thanks Hannah for all your attention to this issue, let's not forget to thank the INFRA guys for their help. On Tue, Feb 25, 2020 at 2:03 AM Hannah Jiang wrote: > Thanks everyone for reporting issues and potential problems etc. > Please feel free to let me know if you have any

Re: Can any contributor trigger PreCommit tests?

2020-02-26 Thread Ismaël Mejía
done On Wed, Feb 26, 2020 at 10:43 PM Wenbing Bai wrote: > Hi Liu, > > Here is my PR https://github.com/apache/beam/pull/10901. Pablo is helping > on this PR. > > Thank you for the quick response. > > Wenbing > > On Tue, Feb 25, 2020 at 1:16 PM Liu Wang wrote: > >> How can I become a

Re: [DISCUSSION] Use github actions for python wheels ?

2020-02-26 Thread Ismaël Mejía
+1 I have been migrating multiple projects into github actions recently and even if for some tasks it is not as mature and polished as travis it has proven to be way more reliable. On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay wrote: > I created https://issues.apache.org/jira/browse/BEAM-9388 to

[VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

2020-02-26 Thread Ismaël Mejía
Please review the release of the following artifacts that we vendor: * beam-vendor-bytebuddy-1_10_8 Hi everyone, Please review and vote on the release candidate #2 for the version 0.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments)

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8

2020-02-25 Thread Ismaël Mejía
Thanks Luke there is indeed an extra module-info.class in the vendored jar. This vote is cancelled. A new vote will be opened once the fix is merged. PTAL https://github.com/apache/beam/pull/10970 On Tue, Feb 25, 2020 at 6:39 PM Luke Cwik wrote: > -1 > > The jar contains

[VOTE] Vendored Dependencies Release Byte Buddy 1.10.8

2020-02-25 Thread Ismaël Mejía
Please review the release of the following artifacts that we vendor: * beam-vendor-bytebuddy-1_10_8 Hi everyone, Please review and vote on the release candidate #1 for the version 0.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments)

Re: [PROPOSAL] Vendored bytebuddy dependency release

2020-02-25 Thread Ismaël Mejía
It seems we have a clear consensus so I will proceed with the vote. Thanks! On Tue, Feb 25, 2020 at 2:19 AM Kenneth Knowles wrote: > +1 thanks! > > On Mon, Feb 24, 2020 at 1:13 PM Pablo Estrada wrote: > >> +1 >> >> On Sun, Feb 23, 2020 at 8:31 PM Luke Cwik wrote: >> >>> +1 >>> >>> On Sun,

Re: [ANNOUNCE] New committer: Chad Dombrova

2020-02-25 Thread Ismaël Mejía
Congratulations, so well deserved for the lots of amazing work and new perspectives you have broght into the project !!! On Tue, Feb 25, 2020 at 8:24 AM Austin Bennett wrote: > Hooray! > > On Mon, Feb 24, 2020, 11:21 PM Alex Van Boxel wrote: > >> Congrats! >> >> _/ >> _/ Alex Van Boxel >> >>

Re: [VOTE] Upgrade gradle to 6.2

2020-02-25 Thread Ismaël Mejía
+1 great to have our build updated, please share if there are new interesting features/plugin advantages we can benefit from too. On Tue, Feb 25, 2020 at 8:24 AM Jean-Baptiste Onofré wrote: > Hi Alex > > I also have couple of contacts at Gradle. Let me know if needed. > > Regards > JB > > Le

Re: [ANNOUNCE] New committer: Jincheng Sun

2020-02-24 Thread Ismaël Mejía
Congrats Jincheng! On Mon, Feb 24, 2020 at 1:39 PM Gleb Kanterov wrote: > Congratulations! > > On Mon, Feb 24, 2020 at 1:18 PM Hequn Cheng wrote: > >> Congratulations Jincheng, well deserved! >> >> Best, >> Hequn >> >> On Mon, Feb 24, 2020 at 7:21 PM Reza Rokni wrote: >> >>> Congrats! >>> >>>

[PROPOSAL] Vendored bytebuddy dependency release

2020-02-22 Thread Ismaël Mejía
The version of bytebuddy Beam is vendoring (1.9.3) is more than 16 months old. I would like to propose that we upgrade it [1] to the most recent version (1.10.8) [2] so we can benefit of the latest improvements for Java 11 and upgraded ASM. If everyone agrees I would like to volunteer as the

Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.2 for BEAM-9252

2020-02-22 Thread Ismaël Mejía
+1 (binding) Verified signatures, produced jar from code and checked that artifact does not contain both Main.class and module-info.classes Thanks for taking care of this Tomo and Luke. On Sat, Feb 22, 2020 at 10:27 AM jincheng sun wrote: > +1(non-binding) > > Best, > Jincheng > > > Kai

Re: [PROPOSAL] Preparing for Beam 2.20.0 release

2020-02-20 Thread Ismaël Mejía
-priorities/ On Thu, Feb 20, 2020 at 3:42 AM Ahmet Altay wrote: > Curions, was there a resolution on BEAM-9252? Would it be a release > blocker? > > On Fri, Feb 14, 2020 at 12:42 AM Ismaël Mejía wrote: > >> Thanks Rui for volunteering and for keeping the release pace! >>

Re: Custom 2.20 failing on Dataflow: what am I doing wrong?

2020-02-19 Thread Ismaël Mejía
Alex/Gleb can someone of you please add the detailed instructions that worked for you in some section of cwiki. I have the impression that this will benefit us all at some point. Thanks, On Tue, Feb 18, 2020 at 9:46 AM Alex Van Boxel wrote: > Thanks everyone. This really helped a lot. I used

Re: thrift command missing on Jenkins?

2020-02-18 Thread Ismaël Mejía
For info the fix was merged into master, kudos to Christopher Larsen for the quick fix. On Tue, Feb 18, 2020 at 9:43 PM Ismaël Mejía wrote: > For info we decided to took the approach 2 Chris will open a PR to do so > and we will get it fixed ASAP. > Thanks Alex for reporting

Re: thrift command missing on Jenkins?

2020-02-18 Thread Ismaël Mejía
For info we decided to took the approach 2 Chris will open a PR to do so and we will get it fixed ASAP. Thanks Alex for reporting and sorry for the inconvenience. On Tue, Feb 18, 2020 at 9:36 PM Ismaël Mejía wrote: > Mmm that one could be on me, I tested it locally and merged the PR to

Re: thrift command missing on Jenkins?

2020-02-18 Thread Ismaël Mejía
Mmm that one could be on me, I tested it locally and merged the PR today, it worked flawlessly in my machine, but now I realize that I have thrift installed locally because it is a requirement when you hack on parquet-mr code. So two workarounds possible: 1. We install Thrift 0.12 in all the

Re: [DISCUSS] Drop support for Flink 1.7

2020-02-18 Thread Ismaël Mejía
hat the version support should remain based on user > feedback and independent of the Flink release policy. If there are Beam > users that require an older Flink version and it is difficult to upgrade, > then we should consider that. > > > On Tue, Feb 18, 2020 at 8:57 AM Ismaël Mejía

Re: [DISCUSS] Drop support for Flink 1.7

2020-02-18 Thread Ismaël Mejía
Just to be sure, you mean Flink 1.11.0-SNAPSHOT ONLY in the next branch dependency? We shou,d not have any SNAPSHOT dependency from other projects in Beam. On Tue, Feb 18, 2020 at 5:34 PM wrote: > Hi Jincheng, > I think there should be a "process" for this. I would propose to: > a) create new

Re: [ANNOUNCE] New committer: Alex Van Boxel

2020-02-18 Thread Ismaël Mejía
Congrats Alex! Well done! On Tue, Feb 18, 2020 at 5:10 PM Gleb Kanterov wrote: > Congratulations! > > On Tue, Feb 18, 2020 at 5:02 PM Brian Hulette wrote: > >> Congratulations Alex! Well deserved! >> >> On Tue, Feb 18, 2020 at 7:49 AM Pablo Estrada wrote: >> >>> Hi everyone, >>> >>> Please

Re: [REQUEST] Workshop at Data Day in Mexico City - March 26th

2020-02-17 Thread Ismaël Mejía
Happy to figure out ways to help/share >> the things am putting together. No need to reinvent the wheel. >> >> Cheers, >> Austin >> >> >> >> On Fri, Feb 14, 2020, 3:57 PM Ismaël Mejía wrote: >> >>> Hello, >>> >>> I am int

Re: Contributor permission for Beam Jira tickets

2020-02-16 Thread Ismaël Mejía
done On Sun, Feb 16, 2020 at 9:02 AM Yu Watanabe wrote: > Hi, > > This is Yu Watanabe from Creationline. > I would like to contribute issues related to elasticsearch and kafka. > Can someone add me as a contributor for Beam's Jira issue > tracker? I would like to create/assign tickets for my

Re: [REQUEST] Workshop at Data Day in Mexico City - March 26th

2020-02-14 Thread Ismaël Mejía
Hello, I am interested and I can make it in spanish if it is easier for the audience. Regards, Ismaël On Sat, Feb 15, 2020 at 12:42 AM María Cruz wrote: > Hello everyone, > > There is an opportunity to host an Apache Beam workshop at the next Data > Day in Mexico City *[1]*. The event will

Re: jira search in chrome omnibox

2020-02-14 Thread Ismaël Mejía
irefox. > https://cwiki.apache.org/confluence/display/BEAM/Jira+Tips > > On Fri, Feb 14, 2020 at 8:20 AM Ismaël Mejía wrote: > >> For Firefox users: >> >> You can replicate the same behavior but it requires a bit more work: >> >> Firefox uses a

Re: Dynamic timers now supported!

2020-02-14 Thread Ismaël Mejía
gt; Also Ismael, what makes this a batch mode test? When I look at the >> failing stack trace, the failure is in FlinkStreamingPipelineTranslator. >> >> Reuven >> >> On Fri, Feb 14, 2020 at 8:52 AM Ismaël Mejía wrote: >> >>> Apparently the fix of Dynami

Re: Dynamic timers now supported!

2020-02-14 Thread Ismaël Mejía
on the exception: https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/6608/ On Wed, Feb 12, 2020 at 10:37 AM Ismaël Mejía wrote: > Great to know you get it working on Dataflow easily Reuven. As a new > feature it > looks great! > > Agree with Kenn maybe worth to op

Re: jira search in chrome omnibox

2020-02-14 Thread Ismaël Mejía
For Firefox users: You can replicate the same behavior but it requires a bit more work: Firefox uses a format called OpenSearch so you have to generate an installable XML via this page. https://ready.to/search/en/# the search name: Beam Issues the front search term:

Beam Postcommit SQL broken RESOURCE_EXHAUSTED pubsub topics

2020-02-14 Thread Ismaël Mejía
I noticed today that the Beam Postcommit SQL is broken: https://builds.apache.org/job/beam_PostCommit_SQL/4019/ Digging deeper the log message says: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Your project has exceeded a limit: (type="topics-per-project", current=1, maximum=1).

Re: [PROPOSAL] Preparing for Beam 2.20.0 release

2020-02-14 Thread Ismaël Mejía
Thanks Rui for volunteering and for keeping the release pace! Since we are discussing the next release, I would like to highlight that nobody apparently is working on this blocker issue: BEAM-9252 Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT

Re: daily dataflow job failing today

2020-02-13 Thread Ismaël Mejía
PTAL? https://github.com/apache/beam/pull/10851 On Thu, Feb 13, 2020 at 9:39 AM Ismaël Mejía wrote: > > > I can argue for not pinning and bounding with major version ranges. This > gives flexibility to users to mix other third party libraries that share > common dependencies

Re: Python2.7 Beam End-of-Life Date

2020-02-13 Thread Ismaël Mejía
ing out python2 by default which definitely help in this direction. Thanks for updating the roadmap Ahmet On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay wrote: > > > On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía wrote: > >> I am with Chad on this, we should probably exten

Re: daily dataflow job failing today

2020-02-13 Thread Ismaël Mejía
13, 2020 at 2:26 AM Ahmet Altay wrote: > > > On Wed, Feb 12, 2020 at 12:54 PM Ismaël Mejía wrote: > >> Independently of the bug in the dependency release the fact that the Beam >> Python >> SDK does not have pinned fixed dependency numbers is error-prone. We may &

Re: Labels on PR

2020-02-12 Thread Ismaël Mejía
it matches the description `core-constructIOn-java` and also `extensIOns` making the point of having general categories not really useful. On Wed, Feb 12, 2020 at 3:01 PM Ismaël Mejía wrote: > The prefix is just extra characters makes readibility worse, notice > that the full category (e.g. ios/r

Re: daily dataflow job failing today

2020-02-12 Thread Ismaël Mejía
Independently of the bug in the dependency release the fact that the Beam Python SDK does not have pinned fixed dependency numbers is error-prone. We may continue to have this kind of problems until we fix this (with other dependencies too). In the Java SDK we do not accept such type of dynamic

Re: Labels on PR

2020-02-12 Thread Ismaël Mejía
wrote: > Ismael, I saw that you removed the prefix. I still want to have some > grouping between the subtypes, so I color coded them. > > I also added all the labels in the file. We now have 62 labels. > > _/ > _/ Alex Van Boxel > > > On Wed, Feb 12, 2020 a

Re: Labels on PR

2020-02-12 Thread Ismaël Mejía
Forgot to mention, older PRs will look not classified, up to you guys if you want to do manually. All new PRs will be automatically labeled. On Wed, Feb 12, 2020 at 12:02 PM Ismaël Mejía wrote: > For info Alex's PR to suport autolabeler was merged today and INFRA > enabled the plugi

Re: Labels on PR

2020-02-12 Thread Ismaël Mejía
> I do think per-runner makes sense, rather than a general "runners." > IOs might make sense as well. Not sure about all the extensions-* I'd > leave those out for now. > > On Tue, Feb 11, 2020 at 5:56 AM Ismaël Mejía wrote: > > > > > So I propose going si

Re: Dynamic timers now supported!

2020-02-12 Thread Ismaël Mejía
gt;>> >>>> Thanks for finding this. Hopefully the bug is easy .to fix. The tests >>>> indeed never ran on any runner except for the DirectRunner, which is >>>> something I should've noticed in the code review. >>>> >>>> Reuven >>

Re: Python2.7 Beam End-of-Life Date

2020-02-12 Thread Ismaël Mejía
I am with Chad on this, we should probably extend it a bit more, even if it makes us struggle a bit at least we have some workarounds as Robert suggests, and as Chad said there are still many people playing the python 3 catchup game, so worth to support those users. But maybe it is worth to

Re: Labels on PR

2020-02-11 Thread Ismaël Mejía
ul during the life-cycle > of a PR. > > _/ > _/ Alex Van Boxel > > > On Tue, Feb 11, 2020 at 9:46 AM Ismaël Mejía wrote: > >> Let some comments too, let's keep the discussion on refinements in the PR. >> >> On Tue, Feb 11, 2020 at 9:13 AM jincheng sun

Re: Jenkins Jobs Trigger Request

2020-02-11 Thread Ismaël Mejía
done On Tue, Feb 11, 2020 at 10:58 AM Shoaib Zafar wrote: > Hi everyone, > > I appreciate if someone could trigger the jobs for this PR: > https://github.com/apache/beam/pull/10712 > > Thanks. > > *Shoaib Zafar* > Software Engineering Lead > Mobile: +92 333 274 6242 > Skype: live:shoaibzafar_1

Re: Labels on PR

2020-02-11 Thread Ismaël Mejía
>>> Cool! >>>> >>>> On Mon, Feb 10, 2020 at 9:27 AM Robert Burke >>>> wrote: >>>> >>>>> +1 to autolabeling >>>>> >>>>> On Mon, Feb 10, 2020, 9:21 AM Luke Cwik wrote: >>>>> >>>

Re: Labels on PR

2020-02-10 Thread Ismaël Mejía
+1 You don't need to write your own action, there is already one autolabeler action [1]. INFRA can easily configure it for Beam (as they did for Avro [2]) if we request it. The plugin is quite easy to configure and works like a charm [3]. [1] https://github.com/probot/autolabeler [1]

Re: Jenkins jobs not running for my PR 10438

2020-02-07 Thread Ismaël Mejía
done On Fri, Feb 7, 2020 at 4:05 PM Tomo Suzuki wrote: > Hi Beam committers, > > I appreciate if you can run precommit checks for > https://github.com/apache/beam/pull/10769 > with the following 6 commands: > > Run Java PostCommit > Run Java HadoopFormatIO Performance Test > Run BigQueryIO

Re: Jenkins outage

2020-02-07 Thread Ismaël Mejía
/console On Fri, Feb 7, 2020 at 3:14 PM Ismaël Mejía wrote: > I am getting "java.lang.IllegalStateException: java.io.IOException: No > space left on device" on apache-beam-jenkins-7 > Can somebody please clean the space. Thanks. > https://builds.apache.org/job/beam_PostCommit

Re: Unable to run ParDoTests from CLI

2020-02-07 Thread Ismaël Mejía
Use ./gradlew :runners:direct-java:needsRunner --tests "*ParDoTest\$TimerTests" For ValidatesRunner for example: /gradlew :runners:direct-java:validatesRunner --tests "*ParDoTest\$TimerFamily*" Credit to Brian who helped me because I was struggling with the same issue last week. On Fri, Feb

Re: Jenkins outage

2020-02-07 Thread Ismaël Mejía
I am getting "java.lang.IllegalStateException: java.io.IOException: No space left on device" on apache-beam-jenkins-7 Can somebody please clean the space. Thanks. https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark_PR/26/ On Fri, Feb 7, 2020 at 2:19 PM Ismaël Mejía wrote:

Re: Jenkins outage

2020-02-07 Thread Ismaël Mejía
Thanks for taking care of this issue with INFRA Michał. Everything back to normal! On Fri, Feb 7, 2020 at 11:34 AM Michał Walenia wrote: > Everything looks fine now, the jobs are triggering correctly again > > On Fri, Feb 7, 2020 at 10:06 AM Michał Walenia > wrote: > >> Hi there, >> it seems

Re: [DISCUSS] Autoformat python code with Black

2020-02-06 Thread Ismaël Mejía
PM Michał Walenia > wrote: > >> Hi, >> the PR is merged, all checks were green :) >> Enjoy prettier Python! >> >> On Thu, Feb 6, 2020 at 11:11 AM Ismaël Mejía wrote: >> >>> Agree no need for vote for this because the consensus is clear and the >>

Re: SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long

2020-02-06 Thread Ismaël Mejía
ostcommit should succeed. >> >> On Thu, Feb 6, 2020 at 11:57 AM Ismaël Mejía wrote: >> > >> > This one is still broken, maybe there are two different data sources, >> one for the '_PR' version and the normal one, can you please confirm? >> > https://buil

Re: SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long

2020-02-06 Thread Ismaël Mejía
This one is still broken, maybe there are two different data sources, one for the '_PR' version and the normal one, can you please confirm? https://builds.apache.org/job/beam_PostCommit_SQL/ On Thu, Feb 6, 2020 at 5:44 PM Brian Hulette wrote: > Sorry for the delay. I had some issues updating

Re: [DISCUSS] Autoformat python code with Black

2020-02-06 Thread Ismaël Mejía
s route we > >> > should add a pre-commit, plus instructions on how to run the tool > >> > (similar to spotless). > >> > > >> > On Mon, Jan 27, 2020 at 10:00 AM Udi Meiri wrote: > >> > > > >> > > I've done a pass on the PR on co

Re: Dynamic timers now supported!

2020-02-03 Thread Ismaël Mejía
to be updated >> > to include the timer family id in the map keys. >> > >> > Surprisingly, the new TimerMap tests seem to pass on Spark >> > ValidatesRunner, even though the Spark runner wasn't updated! I wonder >> > if this means that the Spark runn

Re: Jenkins jobs not running for my PR 10438

2020-01-31 Thread Ismaël Mejía
done On Fri, Jan 31, 2020 at 1:51 PM Rehman Murad Ali < rehman.murad...@venturedive.com> wrote: > Hi, > > I appreciate if someone could trigger the jobs for this PR: > > https://github.com/apache/beam/pull/10627 > > *Thanks* > > *Rehman Murad Ali* > Software Engineer > Mobile: +92 3452076766 >

Re: Release 2.19.0, release candidate #1

2020-01-31 Thread Ismaël Mejía
+1 (binding) - Validated signatures - Run Python wordcount on Direct runner (from wheels) - Run Python wordcount on Flink runner with job-server image (via wheels) - Run Python wordcount on Spark runner with job-server from source (via wheels) - Validate no regressions on Nexmark for Spark

Re: [ANNOUNCE] New committer: Hannah Jiang

2020-01-29 Thread Ismaël Mejía
Congrats Hannah! On Wed, Jan 29, 2020 at 9:35 AM Katarzyna Kucharczyk < ka.kucharc...@gmail.com> wrote: > Congratulations, Hannah! :) > > On Wed, Jan 29, 2020 at 6:04 AM jincheng sun > wrote: > >> Congrats Hannah! >> >> Ankur Goenka 于2020年1月29日 周三11:35写道: >> >>> Congrats Hannah! >>> >>> On Tue,

Re: Jenkins jobs not running for my PR 10438

2020-01-28 Thread Ismaël Mejía
gineer > Mobile: +92 3452076766 > Skype: rehman.muradali > > > On Mon, Jan 27, 2020 at 6:21 PM Rehman Murad Ali < > rehman.murad...@venturedive.com> wrote: > >> Thanks, Ismaël. >> >> *Rehman Murad Ali* >> Software Engineer >> Mobile: +92 3

Re: New contributor

2020-01-28 Thread Ismaël Mejía
You were added to JIRA. Enjoy! On Tue, Jan 28, 2020 at 12:04 PM Ludovic Boutros wrote: > Dear all, > > please, could you add me as a contributor in Jira ? My user name is > lboutros. > > Thank you, > > Ludovic. >

Re: [ANNOUNCE] New committer: Michał Walenia

2020-01-28 Thread Ismaël Mejía
Congratulations Michał, well deserved! On Tue, Jan 28, 2020 at 8:54 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Congrats, Michał! > > On Tue, Jan 28, 2020 at 3:03 AM Udi Meiri wrote: > >> Congratulations Michał! >> >> On Mon, Jan 27, 2020 at 3:49 PM Chamikara Jayalath >>

Re: Jenkins jobs not running for my PR 10438

2020-01-27 Thread Ismaël Mejía
done On Mon, Jan 27, 2020 at 2:13 PM Rehman Murad Ali < rehman.murad...@venturedive.com> wrote: > Hi, > > I appreciate if somebody can run jobs for this PR: > > https://github.com/apache/beam/pull/10627 > > *Thanks & Regards* > > > > *Rehman Murad Ali* > Software Engineer > Mobile: +92

Re: New contributor

2020-01-25 Thread Ismaël Mejía
Done, Welcome to the project! On Sat, Jan 25, 2020 at 11:28 AM Jeffrey Sorensen wrote: > > Can you add me as a contributor - thanks > > Jira profile username is Sorensen > github sorensenjs >

Re: Dynamic timers now supported!

2020-01-24 Thread Ismaël Mejía
This looks great, thanks for the contribution Rehman! I have some questions (note I have not looked at the code at all). - Is this working for both portable and non portable runners? - What do other runners need to implement to support this (e.g. Spark)? Maybe worth to add this to the website

Re: [DISCUSS] Autoformat python code with Black

2020-01-24 Thread Ismaël Mejía
t; >>>>>>>>> > >> >>>>>>>>> > autopep8 is more along the lines of spotless in Java -- it >> only corrects code that breaks the project's style rules. The big problem >> with Beam's current style is that it is so esoteric that autopep8 can't >

Re: New contributor

2020-01-23 Thread Ismaël Mejía
Done. Welcome to Beam! On Thu, Jan 23, 2020 at 1:58 PM Nicholas Got wrote: > Hi, > > This is Nicholas Got. Can someone add me as a contributor for Beam's Jira > issue tracker? I would like to create/assign tickets for my work. ASF Jira > username: ngot > > Regards, > Nicholas Got >

Re: [DISCUSS] Autoformat python code with Black

2020-01-23 Thread Ismaël Mejía
> *biggest* >>>>>>>> gripe about the current style. PyCharm doesn't have enough control >>>>>>>> either. So, if we can choose a style that can be expressed by flake8 >>>>>>>> or >>>>>>>> pycodestyl

Re: [VOTE] Release 2.18.0, release candidate #1

2020-01-22 Thread Ismaël Mejía
+1 (binding) - Validated signatures - Run Python wordcount on Direct runner (from wheels) - Run Python wordcount on Flink runner with job-server image (via wheels) - Run Python wordcount on Spark runner with job-server from source (via wheels) - Validate no regressions on Nexmark for Spark

<    1   2   3   4   5   6   7   8   >