Hi Xintong,
In terms of code, I think it's not complicated. It's all about we need a
public discussion for the new metric name.
And we don't want to block the release for the rarely used metric.
Best,
Jark
On Fri, 14 Oct 2022 at 10:07, Xintong Song wrote:
> @Qingsheng,
>
> I'm overall +1 to yo
@Qingsheng,
I'm overall +1 to your proposal, with only one question: How complicated is
it to come up with a metric for the internal traffic?
I'm asking because, as the new feature is already out for 1.15 & 1.16, it
would be nice if the corresponding new metrics can also be available in
these ver
Hi devs and users,
It looks like we are getting an initial consensus in the discussion so I
started a voting thread [1] just now. Looking forward to your feedback!
[1] https://lists.apache.org/thread/ozlf82mkm6ndx2n1vdgq532h156p4lt6
Best,
Qingsheng
On Thu, Oct 13, 2022 at 10:41 PM Jing Ge wro
Hi Qingsheng,
Thanks for the clarification. +1, I like the idea. Pointing both numXXXOut
and numXXXSend to the same external data transfer metric does not really
break the new SinkV2 design, since there was no requirement to monitor the
internal traffic. So, I think both developer and user can liv
Hi Jing,
Thanks for the reply!
Let me rephrase my proposal: we’d like to use numXXXOut registered on
SinkWriterOperator to reflect the traffic to the external system for
compatibility with old versions before 1.15, and make numXXXSend have the
same value as numXXXOut for compatibility within 1.15
Hi Qingsheng,
Just want to make sure we are on the same page. Are you suggesting
switching the naming between "numXXXSend" and "numXXXOut" or reverting all
the changes we did with FLINK-26126 and FLINK-26492?
For the naming switch, please pay attention that the behaviour has been
changed since we
As a supplement, considering it could be a big reconstruction
redefining internal and external traffic and touching metric names in
almost all operators, this requires a lot of discussions and we might
do it finally in Flink 2.0. I think compatibility is a bigger blocker
in front of us, as the outp
Thanks Chesnay for the reply. +1 for making a unified and clearer
metric definition distinguishing internal and external data transfers.
As you described, having IO in operators is quite common such as
dimension tables in Table/SQL API. This definitely deserves a FLIP and
an overall design.
Howeve
Currently I think that would be a mistake.
Ultimately what we have here is the culmination of us never really
considering how the numRecordsOut metric should behave for operators
that emit data to other operators _and_ external systems. This goes
beyond sinks.
This even applies to numRecordsIn
Thanks for the details Chesnay!
By “alias” I mean to respect the original definition made in FLIP-33 for
numRecordsOut, which is the number of records written to the external system,
and keep numRecordsSend as the same value as numRecordsOut for compatibility.
I think keeping numRecordsOut for
+1 for reverting these changes in Flink 1.16, so I will cancel 1.16.0-rc1.
+1 for `numXXXSend` as the alias of `numXXXOut` in 1.15.3.
Best,
Xingbo
Chesnay Schepler 于2022年10月10日周一 19:13写道：
> > I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut
>
> But that's not possible. If it
> I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut
But that's not possible. If it were that simple there would have never
been a need to introduce another metric in the first place.
It's a rather fundamental issue with how the new sinks work, in that
they emit data to the
On 10/10/2022 11:24, Martijn Visser wrote:
Sidenote: metric names are not mentioned in the FLIP process as a
public API. Might make sense to have a separate follow-up to add that
to the list (I do think we should list them there).
That's a general issue we have. There's a lot of things we _ us
Thanks everyone for joining the discussion!
> Do you have any idea what has happened in the process here?
The discussion in this PR [1] shows some details and could be helpful to
understand the original motivation of the renaming. We do have a test case for
guarding metrics but unfortunaly the
+1 for reverting these changes in Flink 1.16.
For 1.15.3, can we make these metrics available via both names (numXXXOut
and numXXXSend)? In this way we don't break it for those who already
migrated to 1.15 and numXXXSend. That means we still need to change
SinkWriterOperator to use another metric
Hi Qingsheng,
Do you have any idea what has happened in the process here? Do we know why
they were changed? I was under the impression that these metric names were
newly introduced due to the new interfaces and because it still depends on
each connector implementing these.
Sidenote: metric names
Thanks Qingsheng for starting this thread.
+1 on reverting sink metric name and releasing 1.15.3 to fix this inconsistent
behavior.
Best,
Leonard
> 2022年10月10日 下午3:06，Jark Wu 写道：
>
> Thanks for discovering this problem, Qingsheng!
>
> I'm also +1 for reverting the breaking changes.
>
Thanks for discovering this problem, Qingsheng!
I'm also +1 for reverting the breaking changes.
IIUC, currently, the behavior of "numXXXOut" metrics of the new and old
sink is inconsistent.
We have to break one of them to have consistent behavior. Sink V2 is an
evolving API which is just introduc
Thanks for driving, Qingsheng.
+1 for reverting sink metric name.
We often forget that metric is also one of the important APIs.
+1 for releasing 1.15.3 to fix this.
Best,
Jingsong
On Sun, Oct 9, 2022 at 11:35 PM Becket Qin wrote:
>
> Thanks for raising the discussion, Qingsheng,
>
> +1 on re
Thanks for raising the discussion, Qingsheng,
+1 on reverting the breaking changes.
In addition, we might want to release a 1.15.3 to fix this and update the
previous release docs with this known issue, so that users can upgrade to
1.15.3 when they hit it. It would also be good to add some backwa
20 matches
Mail list logo
