Hi all,
I came by this ticket https://issues.apache.org/jira/browse/BEAM-2456. I
know that the metrics subject has already been discussed a lot, but I
would like to revive the discussion.
The aim in this ticket is to avoid relying on the runner to provide the
metrics because they don't have
Hi all,
Etienne forgot to mention that we started a PoC about that.
What I started is to wrap the Pipeline creation to include a thread that polls
periodically the metrics in the pipeline result (it's what I proposed when I
compared with Karaf Decanter some time ago).
Then, this thread marshal
I think discussing a runner agnostic way of configuring how metrics are
extracted is a great idea -- thanks for bringing it up Etienne!
Using a thread that polls the pipeline result relies on the program that
created and submitted the pipeline continuing to run (eg., no machine
faults, network pro
Yeah, I think that something in the runner makes sense.
The only drawback is that it would require some enforcement on the runners and
change on all runners.
If it could be part of the Runner API, that would help I think.
The idea of the thread poller was more in PoC way. Only the polling/send
Thanks Ben for your comments!
Indeed, there is an issue about failover regarding the polling thread.
To that extent, pushing metrics to a sink would be better. To make this
push runner agnostic, doing the code in the runner-common part of beam
would be good. Maybe in the runner API like JB sug
Hi Etienne,
yeah, I think it makes sense to update the PoC.
I like the package/class name you are proposing.
Thanks !
Regards
JB
On 11/29/2017 10:30 AM, Etienne Chauchot wrote:
Thanks Ben for your comments!
Indeed, there is an issue about failover regarding the polling thread. To that
exten
Hi all,
I sketched a little doc [1] about this subject. It tries to sum up the
differences between the runners towards metrics extraction and propose
some possible designs to have a runner agnostic extraction of the metrics.
It is a 2 pages long doc, can you please comment it, and correct it
Hi,
thanks for the doc. I left some comments.
Regards
JB
On 12/11/2017 05:33 PM, Etienne Chauchot wrote:
Hi all,
I sketched a little doc [1] about this subject. It tries to sum up the
differences between the runners towards metrics extraction and propose some
possible designs to have a runn
Hi all,
Just to let you know that I have just submitted the PR [1]:
This PR adds a MetricsPusher discussed in this [2] document in scenario 3.b.
It merges and pushes beam metrics at a configurable (via
pipelineOptions) frequency to a configurable sink. By default the sink
is a DummySink also u
Hi guys,
As part of the work bellow I need the help of Google Dataflow engine
maintainers:
AFAIK Dataflow being a cloud hosted engine, the related runner is very
different from the others. It just submits a job
to the cloud hosted engine. So, no access to metrics container etc... from the
runner
Hi Etienne,
as we might want to keep the runners consistent on such feature, I think it
makes sense to have this in the dataflow runner.
Especially, if it's not used by end-users, there's no impact in the runner.
So, +1 to add MetricsPusher in dataflow runner.
My $0.01
Regards
JB
On 03/26/201
Thanks for keeping this discussion going, Etienne. I can help investigate
what it would take to add support for Dataflow runner. I've filed BEAM-3926
to track [1].
Is there a @ValidatesRunner integration test [2] that can be used to verify
when the functionality has been correctly implemented for
Hi JB,
I guess you mean add it on the engine, not on the runner, as dataflow runner is
more a client
Le lundi 26 mars 2018 à 17:36 +0200, Jean-Baptiste Onofré a écrit :
> Hi Etienne,
>
> as we might want to keep the runners consistent on such feature, I think it
> makes sense to have this in the
Hi Scott,
Thanks for the help and thanks for the ticket creation. I'll add the equivalent
tickets for flink and spark but the
integration is already done in the PR you reviewed
(https://github.com/apache/beam/pull/4548)
Maybe I'll split the PR into different runner coverage (I had already grouped
14 matches
Mail list logo