k" a écrit :
> https://beam.apache.org/blog/2016/05/27/where-is-my-
> pcollection-dot-map.html
>
> On 11. Mar 2018, at 22:21, Romain Manni-Bucau
> wrote:
>
>
>
> Le 12 mars 2018 00:16, "Reuven Lax" a écrit :
>
> I think it would be interesting to s
Just to illustrate what I was looking for @beam: this kind of page
https://tomcat.apache.org/tomcat-80-eol.html but maybe not that fine
grained.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Ol
sounds good, let's wait this day (so tomorrow with the timezone issue) to
see if there is any other proposals and if not start a vote tmr.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmann
Works for me.
What's the procedure to add it on the website (and where can we add it)?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.c
am to users (don't forget big data is not used by "small" projects
but mainly by companies only for obvious reasons ;)).
If we think in terms of active branches, what does it mean?
2 max active branches at a time?
>
> My €0.01
>
> Regards
> JB
> Le 13 mars 201
uot;).length()
(my username being rmannibucau):
@Benchmark
public void stringLength(final Blackhole blackhole) {
blackhole.consume(System.getProperty("user.name").length());
}
the execution is slower than completionfuture in beam case:
Comparison.stringLength thrpt 5
e what I proposed is wrong - it is likely -
but what is your alternative to try to make beam 1. better percieved, 2.
usable by companies (integrable in a software plan)?
>
>
>
> On Tue, Mar 13, 2018 at 1:23 AM Romain Manni-Bucau
> wrote:
>
>> Up?
>>
>> What
the (core dev) drawback but the advantage for the communities and
companies using beam is that they know they can rely on it and plan
migrations as needed to never be on a no more maintained version.
wdyt?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
Le 12 mars 2018 23:05, "Chamikara Jayalath" a écrit :
On Mon, Mar 12, 2018 at 2:36 PM Romain Manni-Bucau
wrote:
>
>
> Le 12 mars 2018 22:22, "Chamikara Jayalath" a
> écrit :
>
>
>
> On Mon, Mar 12, 2018 at 12:42 PM Romain Manni-Bucau
> wrot
Le 12 mars 2018 22:22, "Chamikara Jayalath" a écrit :
On Mon, Mar 12, 2018 at 12:42 PM Romain Manni-Bucau
wrote:
>
>
> Le 12 mars 2018 18:56, "Chamikara Jayalath" a
> écrit :
>
> Agree. We need file-system abstractions in all languages since (1) us
hese filesystems is the same in all languages because it would be annoying
> to provide the same configuration (in slightly different ways) for Java,
> Python, Go, ...
>
> On Fri, Mar 9, 2018 at 2:01 PM, Romain Manni-Bucau
> wrote:
>
>>
>>
>> Le 9 mars 2018 21:35,
; Do you have data that supports this?
>
> Note that in reality for something like passing an element between DoFns,
> the constant in o(1) actually matters. Decreasing SDK harness overhead is a
> good thing though.
>
> On Mon, Mar 12, 2018 at 10:14 AM, Romain Manni-Bucau <
>
a user defines
> these filesystems is the same in all languages because it would be annoying
> to provide the same configuration (in slightly different ways) for Java,
> Python, Go, ...
>
> On Fri, Mar 9, 2018 at 2:01 PM, Romain Manni-Bucau
> wrote:
>
>>
>>
>
ve
you the best of both worlds.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rma
since start/advance bufferizes current in most (all?) impl. So it is likely
a bit different.
To be concrete iterator often handle hasNext implicitly in next() if needed
whereas beam IOs never do it.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |
I agree Thomas but I kind of read it as "yes we can drop that constraint".
If not we should also check we are used in a thread safe context etc which
will likely never hit the user sdk API so why doing that case a particular
case? Am I missing something?
Romain Manni-Bucau
@rmannibu
Hi guys,
why reader#getCurrent* can throw NoSuchElementException,
my understanding is that the runner will guarantee that start or advance
was called and returned true when calling getCurrent so this is a case
which shouldn't happen, no?
Romain Manni-Bucau
@rmannibucau <https://twi
trivial to impl generally I guess.
However being close of this api can help a lot so +1 to try to have a java
dsl on top of current api. Would also be neat to integrate it with
completionstage :).
Reuven
On Sun, Mar 11, 2018 at 2:29 PM Romain Manni-Bucau
wrote:
>
>
> Le 11 mars 2018 21:18, &
Le 12 mars 2018 00:19, "Reuven Lax" a écrit :
On Sun, Mar 11, 2018 at 7:40 AM Romain Manni-Bucau
wrote:
> makes me think beam should maybe do 2 internals changes before moving
> forward on (s)df API changes:
>
> 1. define a beam singleton per JVM (classloader hier
eter and the parameterproviders can use 1. to handle
> their own state. First impl of the parameterprovider SPI would be a) state
> b) timer c) reactive handlers and potentially user parameter providers
> (service like which can be singleton in the scope of a "JVM" thanks
citizen.
Also curious if we could impl reduce with pipeline result = get an output
of a batch from the runner (client) jvm. I see how to do it for longs -
with metrics - but not for collect().
Regards
JB
On 11/03/2018 19:46, Romain Manni-Bucau wrote:
> Hi guys,
>
> don't know
us reasons ;)).
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Bo
parameterprovider SPI would be a) state
b) timer c) reactive handlers and potentially user parameter providers
(service like which can be singleton in the scope of a "JVM" thanks to 1).
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.m
ext should go away completely. At that point it has
> little use except for a way to send output downstream.
>
> On Sun, Mar 11, 2018, 6:07 AM Romain Manni-Bucau
> wrote:
>
>> Hmm, thinking out loud but completionstage should/could be extended to
>> replace processcontext s
programming is not
>>> explored much outside of Javascript (and maybe Haskell) and requires
>>> greater discipline in programming in a functional manner - if you are
>>> mutating stuff in your callback you are going to have bugs, and then when
>>> you add concurre
n poc what i have in mind next month to share more (it is not
only about reactive programming but mainly java 8 enhancements).
Kenn
On Sat, Mar 10, 2018 at 9:31 AM Romain Manni-Bucau
wrote:
>
>
> 2018-03-10 17:30 GMT+01:00 Reuven Lax :
>
>> Have you considered drafting in det
efore) and then propose to impl it ~natively and move it as main API
for another major.
>
>
> On Sat, Mar 10, 2018 at 7:23 AM Romain Manni-Bucau
> wrote:
>
>>
>>
>> 2018-03-10 16:19 GMT+01:00 Reuven Lax :
>>
>>> This is another version (ma
ssElement
> and populate add needed.
>
This is right however in my head it was a single way movemenent to enforce
the design to be reactive and not fake a reactive API with a sync and not
reactive impl which is what would be done today with both support I fear.
>
> On Sat, Mar 10, 2018,
ve completed). I think that's basically what Kenn
> was suggesting, unless I'm missing something.
>
>
> On Fri, Mar 9, 2018 at 11:07 PM Romain Manni-Bucau
> wrote:
>
>> Yes, callback based. Beam today is synchronous and until bundles+combines
>> are react
active. Are you referring to a specific way of writing the code?
On Fri, Mar 9, 2018 at 1:59 PM Reuven Lax wrote:
> What do you mean by reactive?
>
> On Fri, Mar 9, 2018, 6:58 PM Romain Manni-Bucau
> wrote:
>
>> @Kenn: why not preferring to make beam reactive? Would alow to
Romain,
>
> we already discussed about that.
>
> I stated the following while ago (quick short summary):
>
> "No problem to use Google Doc but a summary/discussion has to always be
> sent on the mailing list"
>
> Regards
> JB
>
> On 09/03/2018 23:14,
Hi guys
A lot of design docs are done through gdoc. I see how awesome it is to work
but @asf it has a few issues - and more bothering, it quickly makes the
tracking blurry.
Out of my head here are a few issues I see/hit:
- it doesnt happen on the list (even if a mail is sent the content is not
a
/github.com/aws/aws-sdk-java/issues/1107 it seems that this is
>>>>> possible via a setting on the client, so that the specified region is used
>>>>> as the default but if the bucket is in a different region things still
>>>>> work.
>>>>>
@Kenn: why not preferring to make beam reactive? Would alow to scale way
more without having to hardly synchronize multithreading. Elegant and
efficient :). Beam 3?
Le 9 mars 2018 22:49, "Kenneth Knowles" a écrit :
> I will start with the "exciting futuristic" answer, which is that we
> envisio
WS infer it automatically? Per https://github.com/aws/aws
>>>> -sdk-java/issues/1107 it seems that this is possible via a setting on
>>>> the client, so that the specified region is used as the default but if the
>>>> bucket is in a different region things still wo
ally.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik wrote:
>>>
>>>> Based upon the criteria that was discussed on the mailing list[1], I
>>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>&g
The "hint" would probably to use hints :) - indees this joke refers to the
hint thread.
Long story short with hints you should be able to say "use that specialize
config here".
Now, personally, I'd like to see a way to specialize config per transform.
With an hint an easy way is to use a prefix:
t is a valid concern, but
> we have mvn postcommit so the coverage is not lost. The benefits of quicker
> reviews and less Jenkins congestion seem worth it to me.
>
> Kenn
>
> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
> wrote:
>
>> @Luskasz: not sure Im the bes
imental. I
guess that should apply to RestrictionTracker too, but I wouldn't be too
worried about that, since it only makes sense to use in the context of
those methods.
On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau
wrote:
> Hmm, does sdf api misses some @Experimental then?
&
1
>>> 2.4.0 30.8 3244.3 1
>>>
>>> Query 10 - Streaming/Unbounded
>>> Version Runtime(sec) Events(/sec)Results
>>> 2.3.0 6.315873.0 1
>>> 2.4.0 1
@Luskasz: not sure Im the best to host it since I know more gradle
internals that user interface/ecosystem but happy to help. Will also
require a "sudo" merger for this day to merge fixes asap - guess we can
bypass reviews or have a fast cycle plan for this day to avoid it to be a
week?
Le 8 mars
the other parts work
(spark, flink, direct runner, java core) on my projects
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc
On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles
>>>>>> wrote:
>>>>>>
>>>>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and
g
3. We fix all build issues if so (let say in a week)
4. Pr can nees updates but no more mvn merge
April is gradle month :)
Wdyt?
On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau
wrote:
>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik" a écrit :
>
> Thanks for bringin
Open question: did you think to a way to run the portable api on top of any
runner to implement it once? Since runners have primitive it should be
doable and avoid a per runner codebase, no? Other benefit: no direct
portable api code in runners, yeah :). (Im thinking to a runner decorator
or orches
t the end we fail to deliver something
consistent. So whatever tool is selected I'm tempted to say drop other
build files and jenkins hooks references.
What about doing it after 2.4 vote?
On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau
wrote:
> Up,
>
> We discussed to have a stro
Le 7 mars 2018 17:25, "Robert Bradshaw" a écrit :
On Wed, Mar 7, 2018 at 1:01 AM Romain Manni-Bucau
wrote:
> -1 (non binding). As mentionned in the thread about this release, until
> BEAM-3409 is fixed the pipeline API is not really reliable and requires
> workarounds to
ncies+plugins
are added in pom.xml AFAIK, so it seems the adoption is very slow to not
say rejected.
What do we do about that? When do we drop the double build maintenance -
whatever is picked?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibuca
.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<htt
Just to share a bit more than a ticket here is a bootstrap impl
https://github.com/apache/beam/pull/4803
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github &
Hi guys,
tried to reapply the waitUntilFinish fix - without breaking the compilation
this time ;) - in https://github.com/apache/beam/pull/4790, anyone able to
help to review and move that forward?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
2018-03-02 18:12 GMT+01:00 Robert Bradshaw :
> On Fri, Mar 2, 2018 at 8:45 AM Romain Manni-Bucau
> wrote:
>
> > Hi guys,
>
> > I didn't find a page about beam release support. With the fast minor
> release rrythm which is targetted by beam (see other threads on th
created https://issues.apache.org/jira/browse/BEAM-3786 to track the
discussion (without putting too much details in the ticket for now)
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibuca
@Reuven: this was what I had in mind yes.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.
ystem correctly.
Sounds like something unrelated to vfs right? That said it is not too late
to use a fallback mecanism when parsing the path?
1: https://issues.apache.org/jira/browse/BEAM-2283
On Mon, Mar 5, 2018 at 1:45 PM, Romain Manni-Bucau
wrote:
>
>
> Le 5 mars 2018 22:26, &q
it ecosystem friendly and not
create yet another one.
On Mon, Mar 5, 2018 at 1:05 PM Romain Manni-Bucau
wrote:
> Not backing vfs by a filesystem sounds saner so VfsIO is probably the way
> to go. It would be a FileIO concurrent and hopefully replacement on the
> mid/long term.
was proposed.
>>
>>
>>> On Mon, Mar 5, 2018 at 11:37 AM Robert Bradshaw
>>> wrote:
>>>
>>>>
>>>> On Mon, Mar 5, 2018 at 11:23 AM Romain Manni-Bucau <
>>>> rmannibu...@gmail.com> wrote:
>>>>
>>>>&
m
>
>
>
> On Mon, Mar 5, 2018 at 10:56 AM Reuven Lax wrote:
>
>> Are the filesystem classes marked experimental? If so, precise
>> compatibility is less of a concern. However vfs does need to have better fs
>> support first.
>>
>> Also what about other lan
better way to go through vfs to have more pluggability than a custom system
rarely extended in the ecosystem, no?
>
> On Mon, Mar 5, 2018, 3:35 PM Romain Manni-Bucau
> wrote:
>
>> I'd say to beam 2.x and to beam 3 to move all IO/extension from the core
>> to actual IO/e
Interesting view Thomas - and it makes a lot of sense. Would you rather see
2 modules? embedded-runner+portable-runner+direct-runner (with inheritance
in between)? Would work for me.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.met
I'd say to beam 2.x and to beam 3 to move all IO/extension from the core to
actual IO/extension modules. Sounds compatible this way - in the sense we
can have it eagerly without breaking anything.
wdyt?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |
Yes (FileIO being the visible part of the FileSystems iceberg ;)).
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuc
the volumes require more.
If we don't go that way we should think about what is the reference
implementation and maybe just drop some usages of the direct runner and
enhance another runner supporting embedded runs to support all the beam API
(for instance flink runner).
Does it make it cleare
appear?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<
responsible for buffering these inputs. That way there's no need for a
>> special DoFnRunner at all here.
>>
>>
>> On Fri, Mar 2, 2018 at 1:01 PM Romain Manni-Bucau
>> wrote:
>>
>>> Hi guys,
>>>
>>> what's the rational behind
Mar 2, 2018 at 12:53 PM, Romain Manni-Bucau
wrote:
> Hi guys,
>
> wonder if you discussed or thought to break down what is called today the
> direct runner in an embedded runner which would be modular an extensible.
>
> What I have in mind is the following:
>
> 1. have a s
sure why).
Personally I'd be to drop completely the interface but aliasing the first
one with a default method to bridge the process methods sounds good as well
and allows to reduce the forked code between both branches.
wdyt?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau
ore usable for light/small
but production oriented environments for users, would also start to align
the work done for the portability (thinking to recent python enhancements
in runners) without loosing the strong validation done in tests or preprod
envs.
Was it already mentionned/thought? If not,
re is a maven
plugin, not sure about gradle but it is clearly not a technical blocker).
It would allow to enforce the policy at build time and avoid surprises.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
&
Seems you are right Charles, not sure why it happens but found another
place with the exact same content but with writable inputs. Thanks a lot!
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Ol
not
github which is mainstream? Is it a "test" thing?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <
Thinking out loud: why not trying a 2.3.1 with small fixes only and the 2.4
after 6 weeks starting from the 2.3.0 real release date.
Le 28 févr. 2018 04:24, "Jean-Baptiste Onofré" a écrit :
> OK, maybe I wasn't clear: for me the cycle is ~ 6 weeks once a release is
> out,
> not when it started.
n you explain "unusable?" We have hundreds of direct-runner tests that
> appear to still be running just fine. Is this a new regression, or old
> behavior?
>
>
> On Tue, Feb 27, 2018 at 1:30 PM Romain Manni-Bucau
> wrote:
>
>> Updated 3409 as blocker since
Updated 3409 as blocker since it makes direct runner - and therefore any
test - unusable at all (leaks+unexpected wait times + retry strategy
required+no real alternative even for dofn now dofntester is deprecated).
Le 27 févr. 2018 21:45, "Chamikara Jayalath" a
écrit :
> Looks like previous URL
done
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> |
ay at
>>>> least users
>>>> would have a motivation to implement the teardown method, otherwise it
>>>> doesn’t
>>>> make any sense to have it (API wise).
>>>>
>>>> On Mon, Feb 19, 2018 at 11:30 PM, Eugene Kirpichov <
>>>
into
>> > an upcoming release, which is a feedback loop that results in our
>> releases
>> > taking months instead of weeks. We should however try to get pending
>> PRs
>> > wrapped up.
>> >
>> > On Tue, Feb 20, 2018 at 2:15 PM Romain Manni-Bucau
#x27;s already in the
> branch. That doesn't mean it's not a good call to action to try and
> get long-pending PRs or similar wrapped up.
>
> On Tue, Feb 20, 2018 at 2:10 PM, Romain Manni-Bucau
> wrote:
> > There are a lot of long pending PR, would be good to merge th
There are a lot of long pending PR, would be good to merge them before 2.4.
Some are bringing tests for the 2.3 release which can be critical to
include.
Maybe we should list the pr and jira we want it before picking a date?
Le 20 févr. 2018 22:02, "Konstantinos Katsiapis" a
écrit :
> +1 since
+1
if nobody beats me at it I can have a look next week probably - but happy
to let somebody else have fun too ;)
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com&g
Hi guys,
is it intended beam shades itself?
$ jar tf
~/.m2/repository/org/apache/beam/beam-runners-direct-java/2.3.0/beam-runners-direct-java-2.3.0.jar
| grep '/DoFnRunner.class'
org/apache/beam/runners/direct/repackaged/runners/core/DoFnRunner.class
Romain Manni-Bucau
@rmannibu
nds
PTransform, PCollection> {
private Coder coder;
private DoFn fn;
@Override
public PCollection expand(final PCollection input) {
return input.apply(ParDo.of(fn)).setCoder(coder);
}
}
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |
.
wdyt?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> |
+1, thks Eugene
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in
now and how
to implement the need in beam.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.
Le 19 févr. 2018 22:56, "Reuven Lax" a écrit :
On Mon, Feb 19, 2018 at 1:51 PM, Romain Manni-Bucau
wrote:
>
>
> Le 19 févr. 2018 21:28, "Reuven Lax" a écrit :
>
> How do you call teardown? There are cases in which the Java code gets no
> indication tha
s
are bugs, kill -9 to shutdown is a bug too. Other cases let call shutdown
with a hook worse case.
On Mon, Feb 19, 2018, 12:24 PM Romain Manni-Bucau
wrote:
> Restarting doesnt mean you dont call teardown. Except a bug there is no
> reason - technically - it happens, no reason.
>
>
of rigor required to stop going in circles.
On Mon, Feb 19, 2018 at 12:14 PM Reuven Lax wrote:
> Workers restarting is not a bug, it's standard often expected.
>
> On Mon, Feb 19, 2018, 12:03 PM Romain Manni-Bucau
> wrote:
>
>> Nothing, as mentionned it is a bug so rec
Restarting doesnt mean you dont call teardown. Except a bug there is no
reason - technically - it happens, no reason.
Le 19 févr. 2018 21:14, "Reuven Lax" a écrit :
> Workers restarting is not a bug, it's standard often expected.
>
> On Mon, Feb 19, 2018, 12:03 PM Rom
ld
> Teardown be called on?
>
> On Mon, Feb 19, 2018, 10:20 AM Romain Manni-Bucau
> wrote:
>
>> This is what i want and not 99 teardowns for 100 setups until
>> there is an unexpected crash (= a bug).
>>
>> Le 19 févr. 2018 18:57, "Reuven Lax"
This is what i want and not 99 teardowns for 100 setups until there
is an unexpected crash (= a bug).
Le 19 févr. 2018 18:57, "Reuven Lax" a écrit :
>
>
> On Mon, Feb 19, 2018 at 7:11 AM, Romain Manni-Bucau > wrote:
>
>>
>>
>> 2018-02-19 15:
2018-02-19 15:57 GMT+01:00 Reuven Lax :
>
>
> On Mon, Feb 19, 2018 at 12:35 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> @Reuven: in practise it is created by pool of 256 but leads to the same
>> pattern, the teardown is just a "if (iCreatedT
n information in the JVM. In
all case you come back to the init()/destroy() lifecycle even if you fake
to get connections with bundles.
Just to make it obvious: SDF mentions are just cause SDF imply all the
current issues with the loose definition of the bean lifecycles at an
exponential level, n
far enough that it would no longer need the resources, they would
> get cleaned up. This can be done at pipeline shutdown, or incrementally
> during a streaming pipeline, etc.
>
> Would something like this be a better fit for your use case? If not, why
> is handling teardown within a s
ent basic transforms.
Le 18 févr. 2018 20:07, "Reuven Lax" a écrit :
>
>
> On Sun, Feb 18, 2018 at 10:50 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> Le 18 févr. 2018 19:28, "Ben Chambers" a écrit :
>>
>>
put any unified practise on top of
beam, it also makes beam very hard to integrate or to use to build higher
level libraries or softwares.
This is why i tried to not start the workaround discussions and just stay
at API level.
-- Ben
On Sun, Feb 18, 2018 at 9:56 AM Romain Manni-Bucau
wrote
ving a fn responsible of the
cleanup? Kind of making the teardown a processelement? This is a nice
workaround but it is not always possible as mentionned. Ismael even has a
nice case where this just fails and teardown would work - was with AWS, not
a bigquery bug, but same design.
>
> On Sun
ppen but it is unexpected and means something
happent" then it is fine to have a manual - or auto if fancy - recovery
procedure. This is where it makes all the difference and impacts the
developpers, ops (all users basically).
>
> On Sun, Feb 18, 2018 at 9:16 AM Romain Manni-Bucau
&g
ing hardware simply failed
> (which happens quite often at scale), and in many other conditions.
>
> "Best effort" is the commonly used term to describe such behavior. Please
> feel free to file bugs for cases where you observed a runner not call
> Teardown in a situation wh
2018-02-18 18:00 GMT+01:00 Eugene Kirpichov :
>
>
> On Sun, Feb 18, 2018, 2:06 AM Romain Manni-Bucau
> wrote:
>
>>
>>
>> Le 18 févr. 2018 00:23, "Kenneth Knowles" a écrit :
>>
>> On Sat, Feb 17, 2018 at 3:09 PM, Romain Manni-Bucau &l
201 - 300 of 540 matches
Mail list logo
