Re: jackson to parse options?

2018-01-09 Thread Romain Manni-Bucau
ukasz Cwik" <lc...@google.com> a écrit : > Romain, how has Jackson been a classpath breaker? > > > On Tue, Jan 9, 2018 at 1:20 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> Hmm, beam already owns the cli parsing - that is what I meant - i

Re: jackson to parse options?

2018-01-09 Thread Romain Manni-Bucau
Hmm, beam already owns the cli parsing - that is what I meant - it only misses the arg delimiter (ie quoting) and adding it is easy no? Le 9 janv. 2018 21:19, "Robert Bradshaw" <rober...@google.com> a écrit : > On Tue, Jan 9, 2018 at 11:48 AM, Romain Manni-Bucau > <rma

Re: jackson to parse options?

2018-01-09 Thread Romain Manni-Bucau
on usage for string -> > string[] conversion. > > On Mon, Jan 8, 2018 at 10:06 PM, Romain Manni-Bucau <rmannibu...@gmail.com > > wrote: >> >> Lukasz, the use case is to znsure the config used can still map the CLI >> and that options dont start to abuse json so it is

Re: jackson to parse options?

2018-01-08 Thread Romain Manni-Bucau
thing (convert maps to json without needing the person to write it by hand) like groovy does. Since we are migrating away from Maven it doesn't seem worthwhile to spend time on to make it easier to write the args in the Maven poms. Is there another use case that is being missed? On Mon, Jan 8,

Re: jackson to parse options?

2018-01-08 Thread Romain Manni-Bucau
It isn't > >> quite as nice as standard arg parse libraries, but it isn't too bad. It > >> would be great to improve, though. > >> > >> Jackson is for machine-to-machine communication or other situations > where > >> command line parsing doesn't wor

Re: [DISCUSS] Towards Beam 2.3.0

2018-01-08 Thread Romain Manni-Bucau
Hi JB, I'd like https://github.com/apache/beam/pull/4235 to be integrated if possible Also the JUnit 5 PR brings some light changes which can be worth the "3" digit upgrade so if anyone has some time to review it can be a good candidate too. Thanks for driving it Romain M

Re: Switching to Java 8

2018-01-08 Thread Romain Manni-Bucau
+1000 also requires to upgrade @Auto* processor which was not supporting j8 in current (beam dependency) version. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com&g

Re: JUnit 5 TestPipeline

2018-01-08 Thread Romain Manni-Bucau
+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > PAssert & TestPipeline are the first targets for sure. We have some extend > of TestPipeline (for instance in the Spark runner for streaming part). But > it should be transparent. > > Regards > JB > > On 01/06/2018 09:09 AM,

Re: jackson to parse options?

2018-01-07 Thread Romain Manni-Bucau
needed or bring much being in the core - like avro as mentionned in another thread. Can need a sanitizing round. Short term it was really a "why is it that complicated" ;). Regards JB On 01/07/2018 11:38 AM, Romain Manni-Bucau wrote: > Hi guys, > > not sure i fully get

jackson to parse options?

2018-01-07 Thread Romain Manni-Bucau
/src/main/java/org/talend/sdk/component/server/lang/StringPropertiesTokenizer.java Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

Re: JUnit 5 TestPipeline

2018-01-06 Thread Romain Manni-Bucau
Unit > 4 and 5 since I believe we have a lot of testing utilities in our test jars > which some users do rely on. > > On Fri, Jan 5, 2018 at 9:08 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> FYI, started to code a bit in https://github.com/rmannibu >>

Re: JUnit 5 TestPipeline

2018-01-05 Thread Romain Manni-Bucau
FYI, started to code a bit in https://github.com/rmannibucau/incubator-beam/tree/fb/BEAM-3415_junit-5-integration The surefire setup is not yet correct but it gives some idea of where it goes if you want to have a look. Will try to finish it next week Have a good week-end! Romain Manni-Bucau

Re: JUnit 5 TestPipeline

2018-01-05 Thread Romain Manni-Bucau
created https://issues.apache.org/jira/browse/BEAM-3415 for reference Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

JUnit 5 TestPipeline

2018-01-05 Thread Romain Manni-Bucau
Hi guys, Any plan to work on a JUnit 5 test pipeline? I can surely give some help if needed but would be nice to support it soon. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpres

remove avro from core?

2017-12-19 Thread Romain Manni-Bucau
ion and remove it as a hard requirement of the core to bring more consistency and modularity to beam. Wdyt? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https:

Re: [INFO] Spark runner updated to Spark 2.2.1

2017-12-18 Thread Romain Manni-Bucau
Congrats, was waited for a long time! Very impatient to see the announcement of the 2.3! Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://g

Re: Euphoria Java 8 DSL - proposal

2017-12-17 Thread Romain Manni-Bucau
Hi guys A DSL would be very welcomed, in particular if fluent. Open question: did you study to implement Stream API (surely extending it to have a BeamStream and a few more features like sides etc)? Would be very natural and integrable easily anywhere and avoid a new API discovery. Hazelcast

Re: [DISCUSS] Migrating Apache Beam PreCommits/PostCommits to Gradle Criteria

2017-12-15 Thread Romain Manni-Bucau
If it is the ~same Im not sure i see the point to move but please dont keep gradle and mvn in master - more than jenkins ;). It makes it hard to follow the project and do prs. Le 15 déc. 2017 20:42, "Lukasz Cwik" a écrit : > I'm proposing that the criteria to remove a

Re: Apache Ignite as a distributed processing back-ends

2017-12-12 Thread Romain Manni-Bucau
; >> As for Ignite community, we would prefer to hold the integration in your >> repo. >> >> — >> Denis >> >> On Dec 7, 2017, at 9:54 PM, Romain Manni-Bucau <rmannibu...@gmail.com> >> wrote: >> >> Hi >> >> This sounds awesome to ha

Re: Apache Ignite as a distributed processing back-ends

2017-12-07 Thread Romain Manni-Bucau
Hi This sounds awesome to have an Ignite runner which could compete with hazelcast-jet. The entry point would be https://beam.apache.org/contribute/runner-guide/ IMHO. Being on Ignite cluster also opens a lot of doors - reusing the filesystem or distributed structures. Very exiting. Le 8 déc.

Re: How to cope with Maven transient network issues?

2017-12-05 Thread Romain Manni-Bucau
FYI https://github.com/apache/maven-wagon/pull/37 Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-12-05 6:54 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > > > Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpic...@google.com>

Re: How to cope with Maven transient network issues?

2017-12-04 Thread Romain Manni-Bucau
Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpic...@google.com> a écrit : On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 4 déc. 2017 21:45, "Eugene Kirpichov" <kirpic...@google.com> a écrit : > &g

Re: How to cope with Maven transient network issues?

2017-12-04 Thread Romain Manni-Bucau
s on asf repo we must work with infra to fix it rather than working around the clients. If on a repo we dont control we can try to get rid of it maybe? On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <k...@google

Re: How to cope with Maven transient network issues?

2017-12-04 Thread Romain Manni-Bucau
2017-12-04 18:58 GMT+01:00 Kenneth Knowles <k...@google.com>: > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: >> >> Need to check but if plugin dependencies were not tuned it should be the >> default with a retry

Re: How to cope with Maven transient network issues?

2017-12-02 Thread Romain Manni-Bucau
dle to use that all the time? > > On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> If you use wagon http - and not lightweigh - it should have retries by >> default. >> >> Le 1 déc. 2017 22:31, "Valentyn Tymofi

Re: How to cope with Maven transient network issues?

2017-12-01 Thread Romain Manni-Bucau
If you use wagon http - and not lightweigh - it should have retries by default. Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" a écrit : > Has this ever been brought up in Maven dev community? Perhaps they have > some suggestions. It sounds like a reasonable feature request. >

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
2017-11-30 20:36 GMT+01:00 Kenneth Knowles <k...@google.com>: > On Thu, Nov 30, 2017 at 11:28 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: >> >> This is my short term concern yes. Note that the opposite is not sane >> neither (too big) cause it forces

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
handling/idempotence but this is more complicated so can be worth another thread once this one is fixed which can end up on being specific each time :(. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-30 20:21 GMT+01:00 Eugene Kirpichov <kirpic...@google.com>

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
Le 30 nov. 2017 19:23, "Kenneth Knowles" <k...@google.com> a écrit : On Thu, Nov 30, 2017 at 10:03 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hmm, > > ESIO: https://github.com/apache/beam/blob/master/sdks/java/io/elas > ticsearch/src/main/java/org/a

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
rom the whole user API and make it all @Internal, no? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-30 18:58 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Agree, but maybe we can inform the runner if wanted no ? > > Honestly, from my side, I'm fin

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
@Ben: would all IO be rewritten to use that and the bundle concept dropped from the API to avoid any ambiguity and misleading usage like in current IOs? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-30 18:43 GMT+01:00 Ben Chambers <bchamb...@google.com>:

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
gt; On Thu, Nov 30, 2017 at 6:17 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: >> >> It sounds reasonable to me. >> >> And agree for Spark, I would like to merge Spark 2 update first. >> >> Regards >> JB >> >> On 11/30/2017 03:

Re: makes bundle concept usable?

2017-11-30 Thread Romain Manni-Bucau
Guys, what about moving getMaxBundleSize from flink options to pipeline options. I think all runners can support it right? Spark code needs the merge of the v2 before being able to be implemented probably but I don't see any blocker. wdyt? Romain Manni-Bucau @rmannibucau | Blog | Old Blog

Re: On voting and process

2017-11-29 Thread Romain Manni-Bucau
Hi guys I think everyone has a good will and the issue is more coming from the fact that now 2 build systems have to be maintained - and are not symmetrically maintained :(. Will be fixee soon so no need to discuss it much IMHO. Next time a major change will be done the impacts once merged

Re: [discuss] java profile

2017-11-29 Thread Romain Manni-Bucau
for incremental setup and is boring as well as ut slows down the dev workflow for cli driven dev. On Tue, Nov 28, 2017 at 10:12 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 29 nov. 2017 01:29, "Robert Bradshaw" <rober...@google.com> a écrit : >

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

2017-11-28 Thread Romain Manni-Bucau
ap the sql usage in multiple UDF would make it powerful and ready to use. Le 29 nov. 2017 07:01, "Romain Manni-Bucau" <rmannibu...@gmail.com> a écrit : > My user wishes - whatever version, it is just a number after all ;): > > - make coder usage simpler and consistent (PCo

Re: [DISCUSS] Thinking about Beam 3.x roadmap and release schedule

2017-11-28 Thread Romain Manni-Bucau
My user wishes - whatever version, it is just a number after all ;): - make coder usage simpler and consistent (PCollection TypeDescriptor and Coder are duplicated in term of API) - have a beam api (split from the sdk and internals and impl) - have SDF supported by runners - have a SDFRunner

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
se > >>> >> of incorrectly specified inputs/outputs for certain tasks. > >>> >> > >>> >> The data is available here > >>> >> > >>> >> > >>> >> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_ > NJqEQakUgnNIQ7Qbjzu8

Re: [VOTE] Use Gradle for Apache Beam developmental processes

2017-11-28 Thread Romain Manni-Bucau
-1 (non binding) gradle discourages contributions which is a big pitfall for an asf project and maven/gradle comparison is unfair due to the threading setup of maven (hardcoded thread count and no parallelize builder tusage). Le 28 nov. 2017 19:38, "Jason Kuster" a écrit

Re: [discuss] java profile

2017-11-28 Thread Romain Manni-Bucau
, a process, and some outputs is enough to know > when the process needs to be rerun. > > On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau <rmannibu...@gmail.com > > wrote: > >> Hmm, no. >> >> Incremental build is never correctly implemented cause there is just

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
have missed them) like the signing of artifacts etc. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-28 18:14 GMT+01:00 Kenneth Knowles <k...@google.com>: > Yea, I think voting is the next step. Luke - I think you are obviously the > right person to set

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
and aligned with the config machine. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-28 9:53 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > Just to answer a previous question: > > An ASF github search gives me these stats: > > - m

Re: gradle dirty files blocking maven build

2017-11-28 Thread Romain Manni-Bucau
/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd/cmd/etcd [etc...] There is really something fishy in the build which is "breaking" the filesystem and making any indexing tool (like an IDE) broken. Romain M

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
Le 27 nov. 2017 23:07, "Eugene Kirpichov" <kirpic...@google.com.invalid> a écrit : On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > > Romain, as menti

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
: > Incremental builds aren't correctly setup right now so your likely to see > Python/Go rebuild even if there were no changes. See > https://issues.apache.org/jira/browse/BEAM-3253 > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> &g

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
the frontend-maven-plugin if needed for other languages. That said it can be an interesting other thread since people consuming these languages will probably want their mainstream build tool and a "standard" repository layout rather than a java one. But this is harder to measure. > > On

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
that was the goal: validate there was no side effect of the changes on the whole project. Now the "not java" part of the build will not be impacted by java changed so this is the part i want to skip since it takes a lot of time and I have guarantees it is safe to skip them. Romain M

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
gradle build --no-daemon (with gradle 4.2) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-27 20:21 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: > What is the gradle command you are using to build just the Python SDK? > > On Mon, Nov 27, 2

Re: [discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
Hmm, issue is the same with gradle (locally python build takes 15mn alone which is as much as the java build and it is not parallelized I think) pl is not as smooth since it means doing it on each command whereas the proposal is automatically activated through settings.xml Romain Manni-Bucau

[discuss] java profile

2017-11-27 Thread Romain Manni-Bucau
he doesn't modify at all - other languages. Wdyt? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
hacks). Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-27 19:46 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > I have collected data by running several builds against master using Gradle > and Maven without using Gradle's support for increm

Re: gradle dirty files blocking maven build

2017-11-25 Thread Romain Manni-Bucau
re you running ? I don't find any ".gogradle" >>> files. >>> >>> Thanks, >>> Manu >>> >>> On Fri, Nov 24, 2017 at 5:09 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>> Let me try on my local

Re: gradle dirty files blocking maven build

2017-11-25 Thread Romain Manni-Bucau
i, Nov 24, 2017 at 5:09 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Let me try on my local copy. > > > > Thanks for the report. > > > > Regards > > JB > > > > On 11/24/2017 10:04 AM, Romain Manni-Bucau wrote: > > >

gradle dirty files blocking maven build

2017-11-24 Thread Romain Manni-Bucau
the build very slow in the best case and just locked in the worse one. Just in case you observe it, "find . -name '.gogradle' | xargs rm -Rf" solves it. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: [VOTE] Release 2.2.0, release candidate #4

2017-11-20 Thread Romain Manni-Bucau
Tested on some custom transforms and integrations, +1 (non-binding), thanks for the perseverance Reuven! Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-20 9:14 GMT+01:00 Reuven Lax <re...@google.com.invalid>: > FYI these generated files have been rem

Re: makes bundle concept usable?

2017-11-18 Thread Romain Manni-Bucau
early a subset of the whole bundle and avoid to reprocess it if it fails later. So to summarize I see 2 outcomes: 1. impl SDF in all runners 2. make the bundle size upper bounded - through a pipeline option - in all runners, not sure this one is doable everywhere since I mainly checked spark

Re: makes bundle concept usable?

2017-11-18 Thread Romain Manni-Bucau
cal PCollection being written to this IO, give the code of a hypothetical DoFn implementing the write using your API, and explain what you'd expect to happen at runtime. I'm going to re-engage in this thread after such an example is given. On Sat, Nov 18, 2017, 5:00 AM Romain Manni-Bucau <rmannibu...@g

Re: makes bundle concept usable?

2017-11-18 Thread Romain Manni-Bucau
to keep the existing API? Le 17 nov. 2017 19:20, "Raghu Angadi" <rang...@google.com.invalid> a écrit : On Fri, Nov 17, 2017 at 1:02 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Yep, just take ES IO, if a part of a bundle fails you are in an > unmanaged state

Re: makes bundle concept usable?

2017-11-17 Thread Romain Manni-Bucau
ot;retry" it seems it covers the needs. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 9:23 GMT+01:00 Eugene Kirpichov <kirpic...@google.com.invalid>: > I must admit I'm still failing to understand the problem, so let's step > back even further. >

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
custom state persistence. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 7:44 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Thanks for the explanation. Agree, we might talk about different things > using the same wording. > > I'm also

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
hich can work I have to admit if SDF are implemented by runners. Is there a roadmap/status on that? Last time I checked SDF was a great API without support :(. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 7:25 GMT+01:00 Eugene Kirpichov <kirpic...@google.com.i

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
entry cost and usage of beam way further. In my mind it is exactly what jbatch/spring-batch uses but adapted to beam (stream in particular) case. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 6:49 GMT+01:00 Reuven Lax <re...@google.com.invalid>: >

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
shouldCheckpoint(); IV. @Checkpointer Serializable getCheckpoint(); in the dofn per element Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-17 6:06 GMT+01:00 Raghu Angadi <rang...@google.com.invalid>: > How would you define it (rough API is fine)?. Wit

Re: makes bundle concept usable?

2017-11-16 Thread Romain Manni-Bucau
2017-11-16 12:18 GMT+01:00 Reuven Lax <re...@google.com.invalid>: > On Wed, Nov 15, 2017 at 9:16 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> @Reuven: it looks like a good workaround >> @Ken: thks a lot for the link! >> >> @all: >>

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
point algorithm. Tend to think the first is easier. > 3. Agree to have a core PTransform for that. > > Regards > JB > > > On 11/15/2017 02:16 PM, Romain Manni-Bucau wrote: >> >> @Reuven: it looks like a good workaround >> @Ken: thks a lot for the link! >> &

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
get this kind of "composite" pattern in the beam core? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15 14:12 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: > In case the connection is not clear to folks on this thread, I

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
batches? Beam doesnt everywhere so I guess it is not important - at least for my cases this statement is true. > > Reuven > > On Wed, Nov 15, 2017 at 5:58 PM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> Overall goal is to ensure each 100 elements max, a &quo

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
boolean shouldCheckpoint(); Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15 11:04 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > And the control is given to the DoFn developer via annotations, right ? > > So, bundle would be "hid

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
ossible to use today. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15 10:27 GMT+01:00 Reuven Lax <re...@google.com.invalid>: > It's in the dev list archives, not sure if there's a doc yet. > > I'm not quite sure I understand what you mean by a "flus

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
Hmm, I didn't find the doc - if you have the link not far it would be appreciated - but "before" sounds not enough, it should be "after" in case there was a "flush" no? Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-15

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
on the transform (or dofn) it would be used to control when the checkpoint is done. Thinking out loud it sounds close to jbatch checkpoint algorithm (https://docs.oracle.com/javaee/7/api/javax/batch/api/chunk/CheckpointAlgorithm.html) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
d but can we get something more reliable for the user? Maybe we need a @BeforeBatch or something like that. > > Regards > JB > > > On 11/15/2017 09:38 AM, Romain Manni-Bucau wrote: >> >> Hi guys, >> >> The subject is a bit provocative but the topic is real and coming

Re: makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
Hi Reuven, how does it help since you will still "send to the transform" the data before the commit point of beam and therefore not be able to reprocess the same data in case of a failure between your eager flush and next commit point? Romain Manni-Bucau @rmannibucau | Blog | Old Blo

makes bundle concept usable?

2017-11-15 Thread Romain Manni-Bucau
espected and you can process multiple times the same records. Any plan to make this API reliable and controllable from a beam point of view (at least in a max manner)? Thanks, Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn

Re: [VOTE] Release 2.2.0, release candidate #3

2017-11-10 Thread Romain Manni-Bucau
Both issues are particular cases. Can the 2.2.0 be out and a 2.2.1 done quickly after? Would be very appreciated to have the 2.2.0 fixes to not depend on snapshots anymore due to some blockers found in the core of previous releases. Le 10 nov. 2017 21:23, "Chamikara Jayalath"

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
2017 00:10, "Lukasz Cwik" <lc...@google.com.invalid> a écrit : > I wouldn't mind merging this change in so I could setup those Gradle > Jenkins precommits. > > As per our contribution guidelines, any committer willing to sign off on > the PR? > > On Thu, Nov 9

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
ime criteria. Do you share it as well or prefer time over other criteria - which leads to other conclusions and options indeed and can make us not understanding each other? On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > I will try next week yes but

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
vement > > over Maven or that you did not and you experienced build times that were > > equivalent to Maven? > > > > On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> > > wrote: > > > >> 2017-11-09 18:38 GMT+01:

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
2017-11-09 18:38 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> (this is another topic so we can maybe open another thread) issue is >> not much about python

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
it (except time ;)) but it is always a bit annoying to git clone then not be able to build. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-09 18:07 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > Hmm, I have had good luck when following the Python quick st

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
The 1.3.5 file is when i installed the python dependencies manually to make the build passing (the pip command never passed on my computer and therefore the build always has been broken until i installed it manually - independently from the build tool). Romain Manni-Bucau @rmannibucau | Blog

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
a "?" can probably be exclude as well if created by the build at some point. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Thanks for the update. I was swamped on some meeting

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Romain Manni-Bucau
gradle branch doesnt build for me (some rat issues) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Great ! > > What explain these difference ? I'm curious especially for the clean b

Re: [VOTE] Release 2.2.0, release candidate #2

2017-11-03 Thread Romain Manni-Bucau
https://repository.apache.org/content/repositories/orgapachebeam-1022/org/apache/beam/beam-parent/2.2.0/beam-parent-2.2.0-source-release.zip doesn't look "corrupted" so it looks ok to only update the dist area and reuse the same staging repo Romain Manni-Bucau @rmannibucau | Blog

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-03 Thread Romain Manni-Bucau
ch modules, jenkins/travis have this ability since they support scripting - document how to setup a "fastBuild" profile in its settings.xml which bypasses checkstyle, enforcer plugin, findbugs, etc... for fast development iterations Romain Manni-Bucau @rmannibucau | Blog | Old Blog

Re: spark-submit forces jackson 2.4.4

2017-10-02 Thread Romain Manni-Bucau
> > > Jacob > > On Mon, Oct 2, 2017 at 11:17 AM, Jacob Marble <jmar...@kochava.com> wrote: > > > There is a lot of chatter about AWS and Jackson on their forums, etc. I > > have been using the AWS SDK and Jackson 2.8.9 for a couple of weeks > without &

Re: spark-submit forces jackson 2.4.4

2017-10-02 Thread Romain Manni-Bucau
Hi Jacob, isn't aws API only supporting jackson 2.6 and not 2.8? Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc

graph generator?

2017-05-25 Thread Romain Manni-Bucau
/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://blog-rmannibucau.rhcloud.com> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.

Re: Make runner implementation smoother?

2017-05-24 Thread Romain Manni-Bucau
) can provide #apache-beam on freenode Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://blog-rmannibucau.rhcloud.com> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.

<    1   2   3   4   5