Re: MetricRegistryTestUtils java class (flink-runtime/metrics) not found in source code version 1.14.3

[Martijn Visser]

Hi Prasanna,

Just a quick note that the Github links are all pointing to release
candidate 1 for 1.14.3. The released version is in
https://github.com/apache/flink/blob/release-1.14.3/flink-runtime/src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTestUtils.java

Best regards,

Martijn

On Tue, 1 Feb 2022 at 07:35, Prasanna kumar 
wrote:

> NVM was able to find it in a different place
>
>
> https://github.com/apache/flink/blob/release-1.14.3-rc1/flink-runtime/src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTestUtils.java
>
> On Tue, Feb 1, 2022 at 11:58 AM Prasanna kumar <
> prasannakumarram...@gmail.com> wrote:
>
>> Hi,
>>
>> Team, We are writing our own prometheus reporter to make sure that we are
>> capturing data in histograms rather than summaries.
>>
>> We were able to do it successfully in version 1.12.7.
>>
>> But while upgrading to version 1.14.3 , we find
>> that MetricRegistryTestUtils is not available in the src code given by
>> flink in github.
>>
>> PrometheusReporterTest.java throws error that this file is unavailable
>>
>> [image: Screen Shot 2022-02-01 at 11.50.09 AM.png]
>>
>> [image: Screen Shot 2022-02-01 at 11.53.17 AM.png]
>>
>> The below code is in 1.12.7 but method
>> defaultMetricRegistryConfiguration  is deprecated in the latest version.
>>
>> [image: Screen Shot 2022-02-01 at 11.51.46 AM.png]
>>
>> Looking at the Github link
>> https://github.com/apache/flink/tree/release-1.14.3-rc1/flink-runtime/src/main/java/org/apache/flink/runtime/metrics
>> also shows that the MetricRegistryTestUtils is not available. It's not
>> available in
>> https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics
>> master branch as well.
>>
>> [image: Screen Shot 2022-02-01 at 11.55.19 AM.png]
>>
>> Could the community please add the Class file in GITHUB.
>>
>> Thanks,
>> Prasanna.
>>
>

MetricRegistryTestUtils java class (flink-runtime/metrics) not found in source code version 1.14.3

[Prasanna kumar]

Hi,

Team, We are writing our own prometheus reporter to make sure that we are
capturing data in histograms rather than summaries.

We were able to do it successfully in version 1.12.7.

But while upgrading to version 1.14.3 , we find
that MetricRegistryTestUtils is not available in the src code given by
flink in github.

PrometheusReporterTest.java throws error that this file is unavailable

[image: Screen Shot 2022-02-01 at 11.50.09 AM.png]

[image: Screen Shot 2022-02-01 at 11.53.17 AM.png]

The below code is in 1.12.7 but method defaultMetricRegistryConfiguration  is
deprecated in the latest version.

[image: Screen Shot 2022-02-01 at 11.51.46 AM.png]

Looking at the Github link
https://github.com/apache/flink/tree/release-1.14.3-rc1/flink-runtime/src/main/java/org/apache/flink/runtime/metrics
also shows that the MetricRegistryTestUtils is not available. It's not
available in
https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics
master branch as well.

[image: Screen Shot 2022-02-01 at 11.55.19 AM.png]

Could the community please add the Class file in GITHUB.

Thanks,
Prasanna.

Re: MetricRegistryTestUtils java class (flink-runtime/metrics) not found in source code version 1.14.3

[Prasanna kumar]

NVM was able to find it in a different place

https://github.com/apache/flink/blob/release-1.14.3-rc1/flink-runtime/src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTestUtils.java

On Tue, Feb 1, 2022 at 11:58 AM Prasanna kumar <
prasannakumarram...@gmail.com> wrote:

> Hi,
>
> Team, We are writing our own prometheus reporter to make sure that we are
> capturing data in histograms rather than summaries.
>
> We were able to do it successfully in version 1.12.7.
>
> But while upgrading to version 1.14.3 , we find
> that MetricRegistryTestUtils is not available in the src code given by
> flink in github.
>
> PrometheusReporterTest.java throws error that this file is unavailable
>
> [image: Screen Shot 2022-02-01 at 11.50.09 AM.png]
>
> [image: Screen Shot 2022-02-01 at 11.53.17 AM.png]
>
> The below code is in 1.12.7 but method defaultMetricRegistryConfiguration  is
> deprecated in the latest version.
>
> [image: Screen Shot 2022-02-01 at 11.51.46 AM.png]
>
> Looking at the Github link
> https://github.com/apache/flink/tree/release-1.14.3-rc1/flink-runtime/src/main/java/org/apache/flink/runtime/metrics
> also shows that the MetricRegistryTestUtils is not available. It's not
> available in
> https://github.com/apache/flink/tree/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics
> master branch as well.
>
> [image: Screen Shot 2022-02-01 at 11.55.19 AM.png]
>
> Could the community please add the Class file in GITHUB.
>
> Thanks,
> Prasanna.
>

[jira] [Created] (FLINK-25901) Investigate why hostname might be resolved to different ips on azure pipeline

[Yun Gao (Jira)]

Yun Gao created FLINK-25901:
---

 Summary: Investigate why hostname might be resolved to different 
ips on azure pipeline
 Key: FLINK-25901
 URL: https://issues.apache.org/jira/browse/FLINK-25901
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines
Affects Versions: 1.15.0
Reporter: Yun Gao


When debugging failures of some e2e tests we found that the hostname might be 
resolved to different IP addresses. This issues would need to continue 
investigating. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25900) Create view example does not assign alias to functions resulting in generated names like EXPR$5

[Mans Singh (Jira)]

Mans Singh created FLINK-25900:
--

 Summary: Create view example does not assign alias to functions 
resulting in generated names like EXPR$5
 Key: FLINK-25900
 URL: https://issues.apache.org/jira/browse/FLINK-25900
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table SQL / API
Affects Versions: 1.14.3
Reporter: Mans Singh
 Fix For: 1.15.0


The create view example query:
{noformat}
Flink SQL> CREATE VIEW MyView1 AS SELECT LOCALTIME, LOCALTIMESTAMP, 
CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_ROW_TIMESTAMP(), NOW(), 
PROCTIME();
{noformat}
produces generated column names for CURRENT_ROW_TIMESTAMP() (EXPR$5), NOW() 
(EXPR$6), and PROCTIME() (EXPR$7) since it does not assign aliases, as shown 
below:

 

 
{code:java}
Flink SQL> CREATE VIEW MyView1 AS SELECT LOCALTIME, LOCALTIMESTAMP, 
CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_ROW_TIMESTAMP(), NOW(), 
PROCTIME();
> 
Flink SQL> describe MyView1;
+---+-+---+-++---+
|              name |                        type |  null | key | extras | 
watermark |
+---+-+---+-++---+
|         LOCALTIME |                     TIME(0) | FALSE |     |        |      
     |
|    LOCALTIMESTAMP |                TIMESTAMP(3) | FALSE |     |        |      
     |
|      CURRENT_DATE |                        DATE | FALSE |     |        |      
     |
|      CURRENT_TIME |                     TIME(0) | FALSE |     |        |      
     |
| CURRENT_TIMESTAMP |            TIMESTAMP_LTZ(3) | FALSE |     |        |      
     |
|            EXPR$5 |            TIMESTAMP_LTZ(3) | FALSE |     |        |      
     |
|            EXPR$6 |            TIMESTAMP_LTZ(3) | FALSE |     |        |      
     |
|            EXPR$7 | TIMESTAMP_LTZ(3) *PROCTIME* | FALSE |     |        |      
     |
+---+-+---+-++---+
8 rows in set
 
{code}
 

The documentation shows aliased names 
[Timezone|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/timezone/#decide-time-functions-return-value]

 

 
{code:java}
++-+---+-++---+
|   name |type |  null | key | extras | 
watermark |
++-+---+-++---+
|  LOCALTIME | TIME(0) | false | || 
  |
| LOCALTIMESTAMP |TIMESTAMP(3) | false | || 
  |
|   CURRENT_DATE |DATE | false | || 
  |
|   CURRENT_TIME | TIME(0) | false | || 
  |
|  CURRENT_TIMESTAMP |TIMESTAMP_LTZ(3) | false | || 
  |
|CURRENT_ROW_TIMESTAMP() |TIMESTAMP_LTZ(3) | false | || 
  |
|  NOW() |TIMESTAMP_LTZ(3) | false | || 
  |
| PROCTIME() | TIMESTAMP_LTZ(3) *PROCTIME* | false | || 
  |
++-+---+-++---+
 {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator

[Thomas Weise]

HI Xintong,

Thanks for the feedback and please see responses below -->

On Fri, Jan 28, 2022 at 12:21 AM Xintong Song  wrote:

> Thanks Thomas for drafting this FLIP, and everyone for the discussion.
>
> I also have a few questions and comments.
>
> ## Job Submission
> Deploying a Flink session cluster via kubectl & CR and then submitting jobs
> to the cluster via Flink cli / REST is probably the approach that requires
> the least effort. However, I'd like to point out 2 weaknesses.
> 1. A lot of users use Flink in perjob/application modes. For these users,
> having to run the job in two steps (deploy the cluster, and submit the job)
> is not that convenient.
> 2. One of our motivations is being able to manage Flink applications'
> lifecycles with kubectl. Submitting jobs from cli sounds not aligned with
> this motivation.
> I think it's probably worth it to support submitting jobs via kubectl & CR
> in the first version, both together with deploying the cluster like in
> perjob/application mode and after deploying the cluster like in session
> mode.
>

The intention is to support application management through operator and CR,
which means there won't be any 2 step submission process, which as you
allude to would defeat the purpose of this project. The CR example shows
the application part. Please note that the bare cluster support is an
*additional* feature for scenarios that require external job management. Is
there anything on the FLIP page that creates a different impression?


>
> ## Versioning
> Which Flink versions does the operator plan to support?
> 1. Native K8s deployment was firstly introduced in Flink 1.10
> 2. Native K8s HA was introduced in Flink 1.12
> 3. The Pod template support was introduced in Flink 1.13
> 4. There was some changes to the Flink docker image entrypoint script in,
> IIRC, Flink 1.13
>

Great, thanks for providing this. It is important for the compatibility
going forward also. We are targeting Flink 1.14.x upwards. Before the
operator is ready there will be another Flink release. Let's see if anyone
is interested in earlier versions?


>
> ## Compatibility
> What kind of API compatibility we can commit to? It's probably fine to have
> alpha / beta version APIs that allow incompatible future changes for the
> first version. But eventually we would need to guarantee backwards
> compatibility, so that an early version CR can work with a new version
> operator.
>

Another great point and please let me include that on the FLIP page. ;-)

I think we should allow incompatible changes for the first one or two
versions, similar to how other major features have evolved recently, such
as FLIP-27.

Would be great to get broader feedback on this one.

Cheers,
Thomas



>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Jan 28, 2022 at 1:18 PM Thomas Weise  wrote:
>
> > Thanks for the feedback!
> >
> > >
> > > # 1 Flink Native vs Standalone integration
> > > Maybe we should make this more clear in the FLIP but we agreed to do
> the
> > > first version of the operator based on the native integration.
> > > While this clearly does not cover all use-cases and requirements, it
> > seems
> > > this would lead to a much smaller initial effort and a nicer first
> > version.
> > >
> >
> > I'm also leaning towards the native integration, as long as it reduces
> the
> > MVP effort. Ultimately the operator will need to also support the
> > standalone mode. I would like to gain more confidence that native
> > integration reduces the effort. While it cuts the effort to handle the TM
> > pod creation, some mapping code from the CR to the native integration
> > client and config needs to be created. As mentioned in the FLIP, native
> > integration requires the Flink job manager to have access to the k8s API
> to
> > create pods, which in some scenarios may be seen as unfavorable.
> >
> >  > > > # Pod Template
> > > > > Is the pod template in CR same with what Flink has already
> > > supported[4]?
> > > > > Then I am afraid not the arbitrary field(e.g. cpu/memory resources)
> > > could
> > > > > take effect.
> >
> > Yes, pod template would look almost identical. There are a few settings
> > that the operator will control (and that may need to be blacklisted), but
> > in general we would not want to place restrictions. I think a mechanism
> > where a pod template is merged from multiple layers would also be
> > interesting to make this more flexible.
> >
> > Cheers,
> > Thomas
> >
>

Re: [DISCUSS] Deprecate/remove Twitter connector

[Andrew Otto]

Any SSE/EventSource Java Client should work.  I have not personally used
one.  From a quick search, maybe
https://github.com/launchdarkly/okhttp-eventsource or something like it?



On Mon, Jan 31, 2022 at 11:45 AM Francesco Guardiani <
france...@ververica.com> wrote:

> > Shameless plug:  Maybe the Wikipedia EventStreams
>  SSE API
>  would make for a great
> connector example in Flink?
>
> Sounds like a great idea! Do you have a ready to use Java Client for that?
>
> On Mon, Jan 31, 2022 at 3:47 PM Jing Ge  wrote:
>
>> Thanks @Martijn for driving this! +1 for deprecating and removing it. All
>> the concerns mentioned previously are valid. It is good to know that the
>> upcoming connector template/archetype will help the user for the kickoff.
>> Beyond that, speaking of using a real connector as a sample, since Flink is
>> heading towards the unified batch and stream processing, IMHO, it would be
>> nice to pick up a feasible connector for this trend to let the user get a
>> sample close to the use cases.
>>
>> Best regards
>> Jing
>>
>> On Mon, Jan 31, 2022 at 3:07 PM Andrew Otto  wrote:
>>
>>> Shameless plug:  Maybe the Wikipedia EventStreams
>>>  SSE
>>> API  would make for a great
>>> connector example in Flink?
>>>
>>> :D
>>>
>>> On Mon, Jan 31, 2022 at 5:41 AM Martijn Visser 
>>> wrote:
>>>
 Hi all,

 Thanks for your feedback. It's not about having this connector in the
 main repo, that has been voted on already. This is strictly about the
 connector itself, since it's not maintained and most probably also can't be
 used due to changes in Twitter's API that aren't reflected in our connector
 implementation. Therefore I propose to remove it.

 Fully agree on the template part, what's good to know is that a
 connector template/archetype is part of the goals for the external
 connector repository.

 Best regards,

 Martijn

 On Mon, 31 Jan 2022 at 11:32, Francesco Guardiani <
 france...@ververica.com> wrote:

> Hi,
>
> I agree with the concern about having this connector in the main repo.
> But I think in general it doesn't harm to have a sample connector to show
> how to develop a custom connector, and I think that the Twitter connector
> can be a good candidate for such a template. It needs rework for sure, as
> it has evident issues, notably it doesn't work with table.
>
> So i understand if we wanna remove what we have right now, but I think
> we should have some replacement for a "connector template", which is both
> ready to use and easy to hack to build your own connector starting from 
> it.
> Twitter API is a good example for such a template, as it's both "related"
> to the known common use cases of Flink and because is quite simple to get
> started with.
>
> FG
>
> On Sun, Jan 30, 2022 at 12:31 PM David Anderson 
> wrote:
>
>> I agree.
>>
>> The Twitter connector is used in a few (unofficial) tutorials, so if
>> we remove it that will make it more difficult for those tutorials to be
>> maintained. On the other hand, if I recall correctly, that connector uses
>> V1 of the Twitter API, which has been deprecated, so it's really not very
>> useful even for that purpose.
>>
>> David
>>
>>
>>
>> On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to discuss deprecating Flinks' Twitter connector [1].
>>> This was one of the first connectors that was added to Flink, which 
>>> could
>>> be used to access the tweets from Twitter. Given the evolution of Flink
>>> over Twitter, I don't think that:
>>>
>>> * Users are still using this connector at all
>>> * That the code for this connector should be in the main Flink
>>> codebase.
>>>
>>> Given the circumstances, I would propose to deprecate and remove
>>> this connector. I'm looking forward to your thoughts. If you agree, 
>>> please
>>> also let me know if you think we should first deprecate it in Flink 1.15
>>> and remove it in a version after that, or if you think we can remove it
>>> directly.
>>>
>>> Best regards,
>>>
>>> Martijn Visser
>>> https://twitter.com/MartijnVisser82
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/twitter/
>>>
>>>

Re: [DISCUSS] Deprecate/remove Twitter connector

[Andrew Otto]

https://golb.hplar.ch/2018/02/Access-Server-Sent-Events-from-Java.html
looks like a nice tutorial.

On Mon, Jan 31, 2022 at 12:27 PM Andrew Otto  wrote:

> Any SSE/EventSource Java Client should work.  I have not personally used
> one.  From a quick search, maybe
> https://github.com/launchdarkly/okhttp-eventsource or something like it?
>
>
>
> On Mon, Jan 31, 2022 at 11:45 AM Francesco Guardiani <
> france...@ververica.com> wrote:
>
>> > Shameless plug:  Maybe the Wikipedia EventStreams
>>  SSE API
>>  would make for a great
>> connector example in Flink?
>>
>> Sounds like a great idea! Do you have a ready to use Java Client for
>> that?
>>
>> On Mon, Jan 31, 2022 at 3:47 PM Jing Ge  wrote:
>>
>>> Thanks @Martijn for driving this! +1 for deprecating and removing it.
>>> All the concerns mentioned previously are valid. It is good to know that
>>> the upcoming connector template/archetype will help the user for the
>>> kickoff. Beyond that, speaking of using a real connector as a sample, since
>>> Flink is heading towards the unified batch and stream processing, IMHO, it
>>> would be nice to pick up a feasible connector for this trend to let the
>>> user get a sample close to the use cases.
>>>
>>> Best regards
>>> Jing
>>>
>>> On Mon, Jan 31, 2022 at 3:07 PM Andrew Otto  wrote:
>>>
 Shameless plug:  Maybe the Wikipedia EventStreams
  SSE
 API  would make for a
 great connector example in Flink?

 :D

 On Mon, Jan 31, 2022 at 5:41 AM Martijn Visser 
 wrote:

> Hi all,
>
> Thanks for your feedback. It's not about having this connector in the
> main repo, that has been voted on already. This is strictly about the
> connector itself, since it's not maintained and most probably also can't 
> be
> used due to changes in Twitter's API that aren't reflected in our 
> connector
> implementation. Therefore I propose to remove it.
>
> Fully agree on the template part, what's good to know is that a
> connector template/archetype is part of the goals for the external
> connector repository.
>
> Best regards,
>
> Martijn
>
> On Mon, 31 Jan 2022 at 11:32, Francesco Guardiani <
> france...@ververica.com> wrote:
>
>> Hi,
>>
>> I agree with the concern about having this connector in the main
>> repo. But I think in general it doesn't harm to have a sample connector 
>> to
>> show how to develop a custom connector, and I think that the Twitter
>> connector can be a good candidate for such a template. It needs rework 
>> for
>> sure, as it has evident issues, notably it doesn't work with table.
>>
>> So i understand if we wanna remove what we have right now, but I
>> think we should have some replacement for a "connector template", which 
>> is
>> both ready to use and easy to hack to build your own connector starting
>> from it. Twitter API is a good example for such a template, as it's both
>> "related" to the known common use cases of Flink and because is quite
>> simple to get started with.
>>
>> FG
>>
>> On Sun, Jan 30, 2022 at 12:31 PM David Anderson <
>> da...@alpinegizmo.com> wrote:
>>
>>> I agree.
>>>
>>> The Twitter connector is used in a few (unofficial) tutorials, so if
>>> we remove it that will make it more difficult for those tutorials to be
>>> maintained. On the other hand, if I recall correctly, that connector 
>>> uses
>>> V1 of the Twitter API, which has been deprecated, so it's really not 
>>> very
>>> useful even for that purpose.
>>>
>>> David
>>>
>>>
>>>
>>> On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser <
>>> mart...@ververica.com> wrote:
>>>
 Hi everyone,

 I would like to discuss deprecating Flinks' Twitter connector [1].
 This was one of the first connectors that was added to Flink, which 
 could
 be used to access the tweets from Twitter. Given the evolution of Flink
 over Twitter, I don't think that:

 * Users are still using this connector at all
 * That the code for this connector should be in the main Flink
 codebase.

 Given the circumstances, I would propose to deprecate and remove
 this connector. I'm looking forward to your thoughts. If you agree, 
 please
 also let me know if you think we should first deprecate it in Flink 
 1.15
 and remove it in a version after that, or if you think we can remove it
 directly.

 Best regards,

 Martijn Visser
 https://twitter.com/MartijnVisser82

 [1]
>

[jira] [Created] (FLINK-25899) Add Java connected components example to flink-statefun-playground

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-25899:
-

 Summary: Add Java connected components example to 
flink-statefun-playground
 Key: FLINK-25899
 URL: https://issues.apache.org/jira/browse/FLINK-25899
 Project: Flink
  Issue Type: Improvement
  Components: Build System / Stateful Functions
Affects Versions: statefun-3.1.1
Reporter: Till Rohrmann


In order to show what we can do with Stateful Functions I suggest to add an 
example for how to calculate the connected components from a stream of vertices.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Deprecate/remove Twitter connector

[Francesco Guardiani]

> Shameless plug:  Maybe the Wikipedia EventStreams
 SSE API
 would make for a great
connector example in Flink?

Sounds like a great idea! Do you have a ready to use Java Client for that?

On Mon, Jan 31, 2022 at 3:47 PM Jing Ge  wrote:

> Thanks @Martijn for driving this! +1 for deprecating and removing it. All
> the concerns mentioned previously are valid. It is good to know that the
> upcoming connector template/archetype will help the user for the kickoff.
> Beyond that, speaking of using a real connector as a sample, since Flink is
> heading towards the unified batch and stream processing, IMHO, it would be
> nice to pick up a feasible connector for this trend to let the user get a
> sample close to the use cases.
>
> Best regards
> Jing
>
> On Mon, Jan 31, 2022 at 3:07 PM Andrew Otto  wrote:
>
>> Shameless plug:  Maybe the Wikipedia EventStreams
>>  SSE API
>>  would make for a great
>> connector example in Flink?
>>
>> :D
>>
>> On Mon, Jan 31, 2022 at 5:41 AM Martijn Visser 
>> wrote:
>>
>>> Hi all,
>>>
>>> Thanks for your feedback. It's not about having this connector in the
>>> main repo, that has been voted on already. This is strictly about the
>>> connector itself, since it's not maintained and most probably also can't be
>>> used due to changes in Twitter's API that aren't reflected in our connector
>>> implementation. Therefore I propose to remove it.
>>>
>>> Fully agree on the template part, what's good to know is that a
>>> connector template/archetype is part of the goals for the external
>>> connector repository.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Mon, 31 Jan 2022 at 11:32, Francesco Guardiani <
>>> france...@ververica.com> wrote:
>>>
 Hi,

 I agree with the concern about having this connector in the main repo.
 But I think in general it doesn't harm to have a sample connector to show
 how to develop a custom connector, and I think that the Twitter connector
 can be a good candidate for such a template. It needs rework for sure, as
 it has evident issues, notably it doesn't work with table.

 So i understand if we wanna remove what we have right now, but I think
 we should have some replacement for a "connector template", which is both
 ready to use and easy to hack to build your own connector starting from it.
 Twitter API is a good example for such a template, as it's both "related"
 to the known common use cases of Flink and because is quite simple to get
 started with.

 FG

 On Sun, Jan 30, 2022 at 12:31 PM David Anderson 
 wrote:

> I agree.
>
> The Twitter connector is used in a few (unofficial) tutorials, so if
> we remove it that will make it more difficult for those tutorials to be
> maintained. On the other hand, if I recall correctly, that connector uses
> V1 of the Twitter API, which has been deprecated, so it's really not very
> useful even for that purpose.
>
> David
>
>
>
> On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser 
> wrote:
>
>> Hi everyone,
>>
>> I would like to discuss deprecating Flinks' Twitter connector [1].
>> This was one of the first connectors that was added to Flink, which could
>> be used to access the tweets from Twitter. Given the evolution of Flink
>> over Twitter, I don't think that:
>>
>> * Users are still using this connector at all
>> * That the code for this connector should be in the main Flink
>> codebase.
>>
>> Given the circumstances, I would propose to deprecate and remove this
>> connector. I'm looking forward to your thoughts. If you agree, please 
>> also
>> let me know if you think we should first deprecate it in Flink 1.15 and
>> remove it in a version after that, or if you think we can remove it
>> directly.
>>
>> Best regards,
>>
>> Martijn Visser
>> https://twitter.com/MartijnVisser82
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/twitter/
>>
>>

[jira] [Created] (FLINK-25898) Add README.md to flink-statefun/statefun-sdk-js

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-25898:
-

 Summary: Add README.md to flink-statefun/statefun-sdk-js
 Key: FLINK-25898
 URL: https://issues.apache.org/jira/browse/FLINK-25898
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Affects Versions: statefun-3.2.0
Reporter: Till Rohrmann


We should add a {{README.md}} to {{flink-statefun/statefun-sdk-js}}. This would 
then also be displayed on 
[npmjs.com|https://www.npmjs.com/package/apache-flink-statefun].

Unfortunately, in order to publish the {{README.md}} we have to release a new 
version of the npm package.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25897) Update project configuration gradle doc to 7.x version

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-25897:
---

 Summary: Update project configuration gradle doc to 7.x version
 Key: FLINK-25897
 URL: https://issues.apache.org/jira/browse/FLINK-25897
 Project: Flink
  Issue Type: Bug
Reporter: Francesco Guardiani
Assignee: Mario Rincon-Nigro


Update the gradle build script and its doc page to 7.x



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25896) Buffer debloating issues with high parallelism

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-25896:
--

 Summary: Buffer debloating issues with high parallelism
 Key: FLINK-25896
 URL: https://issues.apache.org/jira/browse/FLINK-25896
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Affects Versions: 1.14.3, 1.15.0
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25895) Add ExecNodeGraph ser/de

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-25895:
---

 Summary: Add ExecNodeGraph ser/de
 Key: FLINK-25895
 URL: https://issues.apache.org/jira/browse/FLINK-25895
 Project: Flink
  Issue Type: Sub-task
Reporter: Francesco Guardiani
Assignee: Francesco Guardiani


Right now we have an intermediate model called {{JsonPlanGraph}} to ser/de the 
ExecNodeGraph. This model is fine, but we should do the conversion through 
jackson rather than manually ser/de it with {{ExecNodeGraphJsonPlanGenerator}}.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Deprecate/remove Twitter connector

[Jing Ge]

Thanks @Martijn for driving this! +1 for deprecating and removing it. All
the concerns mentioned previously are valid. It is good to know that the
upcoming connector template/archetype will help the user for the kickoff.
Beyond that, speaking of using a real connector as a sample, since Flink is
heading towards the unified batch and stream processing, IMHO, it would be
nice to pick up a feasible connector for this trend to let the user get a
sample close to the use cases.

Best regards
Jing

On Mon, Jan 31, 2022 at 3:07 PM Andrew Otto  wrote:

> Shameless plug:  Maybe the Wikipedia EventStreams
>  SSE API
>  would make for a great
> connector example in Flink?
>
> :D
>
> On Mon, Jan 31, 2022 at 5:41 AM Martijn Visser 
> wrote:
>
>> Hi all,
>>
>> Thanks for your feedback. It's not about having this connector in the
>> main repo, that has been voted on already. This is strictly about the
>> connector itself, since it's not maintained and most probably also can't be
>> used due to changes in Twitter's API that aren't reflected in our connector
>> implementation. Therefore I propose to remove it.
>>
>> Fully agree on the template part, what's good to know is that a connector
>> template/archetype is part of the goals for the external
>> connector repository.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Mon, 31 Jan 2022 at 11:32, Francesco Guardiani <
>> france...@ververica.com> wrote:
>>
>>> Hi,
>>>
>>> I agree with the concern about having this connector in the main repo.
>>> But I think in general it doesn't harm to have a sample connector to show
>>> how to develop a custom connector, and I think that the Twitter connector
>>> can be a good candidate for such a template. It needs rework for sure, as
>>> it has evident issues, notably it doesn't work with table.
>>>
>>> So i understand if we wanna remove what we have right now, but I think
>>> we should have some replacement for a "connector template", which is both
>>> ready to use and easy to hack to build your own connector starting from it.
>>> Twitter API is a good example for such a template, as it's both "related"
>>> to the known common use cases of Flink and because is quite simple to get
>>> started with.
>>>
>>> FG
>>>
>>> On Sun, Jan 30, 2022 at 12:31 PM David Anderson 
>>> wrote:
>>>
 I agree.

 The Twitter connector is used in a few (unofficial) tutorials, so if we
 remove it that will make it more difficult for those tutorials to be
 maintained. On the other hand, if I recall correctly, that connector uses
 V1 of the Twitter API, which has been deprecated, so it's really not very
 useful even for that purpose.

 David



 On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser 
 wrote:

> Hi everyone,
>
> I would like to discuss deprecating Flinks' Twitter connector [1].
> This was one of the first connectors that was added to Flink, which could
> be used to access the tweets from Twitter. Given the evolution of Flink
> over Twitter, I don't think that:
>
> * Users are still using this connector at all
> * That the code for this connector should be in the main Flink
> codebase.
>
> Given the circumstances, I would propose to deprecate and remove this
> connector. I'm looking forward to your thoughts. If you agree, please also
> let me know if you think we should first deprecate it in Flink 1.15 and
> remove it in a version after that, or if you think we can remove it
> directly.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/twitter/
>
>

Re: [DISCUSS] Deprecate/remove Twitter connector

[Andrew Otto]

Shameless plug:  Maybe the Wikipedia EventStreams
 SSE API
 would make for a great
connector example in Flink?

:D

On Mon, Jan 31, 2022 at 5:41 AM Martijn Visser 
wrote:

> Hi all,
>
> Thanks for your feedback. It's not about having this connector in the main
> repo, that has been voted on already. This is strictly about the connector
> itself, since it's not maintained and most probably also can't be used due
> to changes in Twitter's API that aren't reflected in our connector
> implementation. Therefore I propose to remove it.
>
> Fully agree on the template part, what's good to know is that a connector
> template/archetype is part of the goals for the external
> connector repository.
>
> Best regards,
>
> Martijn
>
> On Mon, 31 Jan 2022 at 11:32, Francesco Guardiani 
> wrote:
>
>> Hi,
>>
>> I agree with the concern about having this connector in the main repo.
>> But I think in general it doesn't harm to have a sample connector to show
>> how to develop a custom connector, and I think that the Twitter connector
>> can be a good candidate for such a template. It needs rework for sure, as
>> it has evident issues, notably it doesn't work with table.
>>
>> So i understand if we wanna remove what we have right now, but I think we
>> should have some replacement for a "connector template", which is both
>> ready to use and easy to hack to build your own connector starting from it.
>> Twitter API is a good example for such a template, as it's both "related"
>> to the known common use cases of Flink and because is quite simple to get
>> started with.
>>
>> FG
>>
>> On Sun, Jan 30, 2022 at 12:31 PM David Anderson 
>> wrote:
>>
>>> I agree.
>>>
>>> The Twitter connector is used in a few (unofficial) tutorials, so if we
>>> remove it that will make it more difficult for those tutorials to be
>>> maintained. On the other hand, if I recall correctly, that connector uses
>>> V1 of the Twitter API, which has been deprecated, so it's really not very
>>> useful even for that purpose.
>>>
>>> David
>>>
>>>
>>>
>>> On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser 
>>> wrote:
>>>
 Hi everyone,

 I would like to discuss deprecating Flinks' Twitter connector [1]. This
 was one of the first connectors that was added to Flink, which could be
 used to access the tweets from Twitter. Given the evolution of Flink over
 Twitter, I don't think that:

 * Users are still using this connector at all
 * That the code for this connector should be in the main Flink
 codebase.

 Given the circumstances, I would propose to deprecate and remove this
 connector. I'm looking forward to your thoughts. If you agree, please also
 let me know if you think we should first deprecate it in Flink 1.15 and
 remove it in a version after that, or if you think we can remove it
 directly.

 Best regards,

 Martijn Visser
 https://twitter.com/MartijnVisser82

 [1]
 https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/twitter/

[jira] [Created] (FLINK-25894) Explicitly configure japicmp oldVersion for flink-streaming-java

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-25894:


 Summary: Explicitly configure japicmp oldVersion for 
flink-streaming-java
 Key: FLINK-25894
 URL: https://issues.apache.org/jira/browse/FLINK-25894
 Project: Flink
  Issue Type: Technical Debt
  Components: Build System
Affects Versions: 1.15.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0


Since flink-streaming-java lost its scala suffix japicmp can't find the 
artifact to compare it against, as the 1.14 artifact still has the suffix.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Looking for maintainers for Cassandra connector or decide to remove connector

[Marco Zühlke]

Hi,

just discovered the voting thread and wanted to raise my hand to in an attempt 
to maintain the Apache Cassandra connector. At  work Kafka -> Flink -> 
Cassandra is a very common usage pattern for me.

>From other ongoing threads I got that the connectors would be removed from 
>main repository anyways and in the future maintained in the 
>https://github.com/apache/flink-connectors repo, right ?

>From the listed topic I think that the adopting the new ASync API combined 
>with probably support for Cassandra 4.0 are the most pressing issue. 

Best reagards,
Marco

On 2022/01/21 08:22:42 Martijn Visser wrote:
> Hi everyone,
> 
> We're looking for community members, who would like to maintain Flink's
> Cassandra connector [1] going forward. The connector currently is only
> available as a Sink for DataStream users and the original contributors are
> unable to work on further improvements.
> 
> An overview of some of the things that are missing on the Cassandra
> connector:
> 
> * Not using the Unified Sink API [2] or ASync API [3]
> * Can't be used in the Table API / SQL
> * Not available as a Source
> * Not available for Lookups
> * Not using the latest supported versions for Cassandra
> 
> If you would like to take on this responsibility or can join this effort in
> a supporting role, please reach out!
> 
> If we can't find maintainers for this connector, I'll open up a vote to
> deprecate this connector and remove it.
> 
> I'm looking forward to your thoughts.
> 
> Best regards,
> 
> Martijn Visser
> https://twitter.com/MartijnVisser82
> 
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/cassandra/
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> [3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> 

[RESULT][VOTE] Apache Flink Stateful Functions 3.2.0, release candidate #1

[Till Rohrmann]

Hi everyone,

I am pleased to announce that we have unanimously approved this release:

There are 5 approving votes, 3 of which are binding:
- Mingmin Xu (non-binding)
- Igal Shilman (binding)
- Seth Wiesman (non-binding)
- Tzu-Li Tai (binding)
- Till Rohrmann (binding)

There are no disapproving votes.

Thanks everyone!

Cheers,
Till

Re: [VOTE] Apache Flink Stateful Functions 3.2.0, release candidate #1

[Till Rohrmann]

Thanks a lot for the votes everyone! We have reached the required number of
binding votes :-) I will close the vote now and post the result as a
separate email.

Cheers,
Till

On Sat, Jan 29, 2022 at 5:29 PM Till Rohrmann  wrote:

> +1 (binding)
>
> - verified checksums and signatures
> - built from sources and ran e2e tests
> - Reviewed announcement PR
> - Ran the JS greeter example from flink-statefun-playground
>
> Cheers,
> Till
>
> On Fri, Jan 28, 2022 at 9:32 PM Tzu-Li (Gordon) Tai 
> wrote:
>
>> +1 (binding)
>>
>> - verified checksums and signature
>> - built from source with e2e tests
>> - Ran Js greeter example
>> - Checked new updated Dockerfile to be released
>>
>> Thanks,
>> Gordon
>>
>> On Fri, Jan 28, 2022 at 8:58 AM Seth Wiesman  wrote:
>>
>> > +1 (non-binding)
>> >
>> > - verified checksums and signatures
>> > - built from source and ran e2e tests
>> > - checked dependencies / licenses
>> > - deployed greeter with golang sdk
>> > - reviewed blog post
>> >
>> > Cheers,
>> >
>> > Seth
>> >
>> >
>> > On Fri, Jan 28, 2022 at 4:22 AM Igal Shilman  wrote:
>> >
>> > > +1 (binding)
>> > >
>> > > - verified checksum and signatures
>> > > - build from source code
>> > > - run e2e tests
>> > > - verified that there are no binary artifacts in the source release
>> > > - built the python sdk from sources, and run the python greeter
>> example
>> > > - built from source the js sdk and run the javascript greeter example.
>> > >
>> > >
>> > > Thanks,
>> > > Igal.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Jan 27, 2022 at 8:20 PM Mingmin Xu 
>> wrote:
>> > >
>> > > > +1 (non-binding)
>> > > >
>> > > > - verified checksum and signatures
>> > > > - build from source code
>> > > > - version checked
>> > > > - test docker PR
>> > > > - test flink-statefun-playground/greeter with 3.2.0
>> > > >
>> > > > Misc, do we want to upgrade flink-statefun-playground together?
>> > Currently
>> > > > the README information is a little behind.
>> > > >
>> > > > B.R.
>> > > > Mingmin
>> > > >
>> > > >
>> > > > On Wed, Jan 26, 2022 at 4:55 AM Till Rohrmann > >
>> > > > wrote:
>> > > >
>> > > > > Hi everyone,
>> > > > >
>> > > > > a quick update on the vote:
>> > > > >
>> > > > > The correct link for the artifacts at the Apache Nexus repository
>> is
>> > > > >
>> > >
>> https://repository.apache.org/content/repositories/orgapacheflink-1485/.
>> > > > >
>> > > > > Moreover, there is now also a tag for the GoLang SDK:
>> > > > >
>> > >
>> https://github.com/apache/flink-statefun/tree/statefun-sdk-go/v3.2.0-rc1
>> > > > .
>> > > > >
>> > > > > Cheers,
>> > > > > Till
>> > > > >
>> > > > > On Tue, Jan 25, 2022 at 10:49 PM Till Rohrmann <
>> trohrm...@apache.org
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi everyone,
>> > > > > >
>> > > > > > Please review and vote on the release candidate #1 for the
>> version
>> > > > 3.2.0
>> > > > > > of Apache Flink Stateful Functions, as follows:
>> > > > > >
>> > > > > > [ ] +1, Approve the release
>> > > > > > [ ] -1, Do not approve the release (please provide specific
>> > comments)
>> > > > > >
>> > > > > > **Release Overview**
>> > > > > >
>> > > > > > As an overview, the release consists of the following:
>> > > > > > a) Stateful Functions canonical source distribution, to be
>> deployed
>> > > to
>> > > > > the
>> > > > > > release repository at dist.apache.org
>> > > > > > b) Stateful Functions Python SDK distributions to be deployed to
>> > PyPI
>> > > > > > c) Maven artifacts to be deployed to the Maven Central
>> Repository
>> > > > > > d) New Dockerfiles for the release
>> > > > > > e) GoLang SDK (contained in the repository)
>> > > > > > f) JavaScript SDK (contained in the repository; will be
>> uploaded to
>> > > npm
>> > > > > > after the release)
>> > > > > >
>> > > > > > **Staging Areas to Review**
>> > > > > >
>> > > > > > The staging areas containing the above mentioned artifacts are
>> as
>> > > > > follows,
>> > > > > > for your review:
>> > > > > > * All artifacts for a) and b) can be found in the corresponding
>> dev
>> > > > > > repository at dist.apache.org [2]
>> > > > > > * All artifacts for c) can be found at the Apache Nexus
>> Repository
>> > > [3]
>> > > > > >
>> > > > > > All artifacts are signed with the key
>> > > > > > B9499FA69EFF5DEEEBC3C1F5BA7E4187C6F73D82 [4]
>> > > > > >
>> > > > > > Other links for your review:
>> > > > > > * JIRA release notes [5]
>> > > > > > * source code tag "release-3.2.0-rc1" [6]
>> > > > > > * PR for the new Dockerfiles [7]
>> > > > > > * PR to update the website Downloads page to include Stateful
>> > > Functions
>> > > > > > links [8]
>> > > > > > * GoLang SDK [9]
>> > > > > > * JavaScript SDK [10]
>> > > > > >
>> > > > > > **Vote Duration**
>> > > > > >
>> > > > > > The voting time will run for at least 72 hours.
>> > > > > > It is adopted by majority approval, with at least 3 PMC
>> affirmative
>> > > > > votes.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Till
>> > > > > >
>> > > > > > [1]
>> > > > > 

Re: [Consulting] Should a Flink class loader be a singleton?

[Till Rohrmann]

Hi Zhiping,

I am pulling in Timo who knows the sql client probably best. He might be
able to help you. In general, this sounds like a bug to me. Maybe you can
provide us with the full stack trace and maybe even the debug logs of the
failing run.

Cheers,
Till

On Sun, Jan 30, 2022 at 4:21 AM Zhiping Wu  wrote:

> Hi there,
>
> I'm trying to consult whether a flink class loader(ParentFirstClassLoader
> or ChildFirstClassLoader) must be a singleton?
>
> I met a problem with the below error message by using sql-client to submit
> sql jobs to a standalone session model cluster with Flink 1.13.3.
> I also tried ParentFirstClassLoader & ChildFirstClassLoader, both of them
> will meet this problem.
>
> Caused by: java.lang.ClassCastException:
> class org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to
> class org.apache.hudi.common.fs.HoodieWrapperFileSystem
> (org.apache.hudi.common.fs.HoodieWrapperFileSystem
> is in unnamed module of loader
> org.apache.flink.runtime.execution.librarycache.
> FlinkUserCodeClassLoaders$ParentFirstClassLoader @*384d2142*;
> org.apache.hudi.common.fs.HoodieWrapperFileSystem
> is in unnamed module of loader
> org.apache.flink.runtime.execution.librarycache.
> FlinkUserCodeClassLoaders$ParentFirstClassLoader @*35e21dd1*)
>
> From this documentation, it shows I might have two versions of the class
> which are different in application code and flink libraries.
> But in my case, Flink should don't contains HoodieWrapperFileSystem class,
> and I didn't add the hudi-flink-bundle jar to flink lib folder, instead, i
> used sql-client.sh embedded -j /huid-flink-bundle-xxx.jar.
>
> So I'm wondering if the root reason is that they are loaded by two
> classloader instances? And it seems that FlinkUserCodeClassLoaders is a
> singleton, but ParentFirstClassLoader and ChildFirstClassLoader aren't from
> flink source code.
>

JDBC: Support connection pooling

[Dario Heinisch]

Hey there,

Hope everyone is well!

Correct me if I am wrong but it seems like Flink does not support 
connection pooling for JDBC sinks.
So for every sink we open a connection, if we have N sinks we have N 
open connections.
There are multiple JDBC connection pooling libraries which can be used 
to support having N JDBC sinks

but only M open connections to the DB where as M < N.

Given the amount of different JDBC connection pooling options, I would 
propose to update the JDBCSink 
(https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcSink.java) 
with another method which would take some sort of an interface of the 
likes of:


```
public interface ConnectionProvider {

   Connection getConnection();

}

```

to build the SinkFunction.
This would allow the user to use whatever connection pooling library 
he/she wants and he/she would only have to implement the 
`ConnectionProvider` interface.


There is already the `JdbcConnectionProvider` 
(https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/internal/connection/JdbcConnectionProvider.java) 
interface but it is marked as internal.


Let me know what you think; I can turn this into a jira ticket and try 
my best if wanted.


Best regards,

Dario

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

[Gabor Somogyi]

Thanks for the confirmation, now it works!

G


On Mon, Jan 31, 2022 at 12:25 PM Chesnay Schepler 
wrote:

> You should have permissions now. Note that I saw 2 accounts matching
> your name, and I picked gaborgsomogyi.
>
> On 31/01/2022 11:28, Gabor Somogyi wrote:
> > Not sure if the mentioned write right already given or not but I still
> > don't see any edit button.
> >
> > G
> >
> >
> > On Fri, Jan 28, 2022 at 5:08 PM Gabor Somogyi  >
> > wrote:
> >
> >> Hi Robert,
> >>
> >> That would be awesome.
> >>
> >> My cwiki username: gaborgsomogyi
> >>
> >> G
> >>
> >>
> >> On Fri, Jan 28, 2022 at 5:06 PM Robert Metzger 
> >> wrote:
> >>
> >>> Hey Gabor,
> >>>
> >>> let me know your cwiki username, and I can give you write permissions.
> >>>
> >>>
> >>> On Fri, Jan 28, 2022 at 4:05 PM Gabor Somogyi <
> gabor.g.somo...@gmail.com>
> >>> wrote:
> >>>
>  Thanks for making the design better! No further thing to discuss from
> my
>  side.
> 
>  Started to reflect the agreement in the FLIP doc.
>  Since I don't have access to the wiki I need to ask Marci to do that
> >>> which
>  may take some time.
> 
>  G
> 
> 
>  On Fri, Jan 28, 2022 at 3:52 PM David Morávek 
> wrote:
> 
> > Hi,
> >
> > AFAIU an under registration TM is not added to the registered TMs map
>  until
> >> RegistrationResponse ..
> >>
> > I think you're right, with a careful design around threading
> >>> (delegating
> > update broadcasts to the main thread) + synchronous initial update
> >>> (that
> > would be nice to avoid) this should be doable.
> >
> > Not sure what you mean "we can't register the TM without providing it
>  with
> >> token" but in unsecure configuration registration must happen w/o
>  tokens.
> > Exactly as you describe it, this was meant only for the "kerberized /
> > secured" cluster case, in other cases we wouldn't enforce a non-null
>  token
> > in the response
> >
> > I think this is a good idea in general.
> > +1
> >
> > If you don't have any more thoughts on the RPC / lifecycle part, can
> >>> you
> > please reflect it into the FLIP?
> >
> > D.
> >
> > On Fri, Jan 28, 2022 at 3:16 PM Gabor Somogyi <
> >>> gabor.g.somo...@gmail.com
> > wrote:
> >
> >>> - Make sure DTs issued by single DTMs are monotonically increasing
>  (can
> >> be
> >> sorted on TM side)
> >>
> >> AFAIU an under registration TM is not added to the registered TMs
> >>> map
> > until
> >> RegistrationResponse
> >> is processed which would contain the initial tokens. If that's true
>  then
> >> how is it possible to have race with
> >> DTM update which is working on the registered TMs list?
> >> To be more specific "taskExecutors" is the registered map of TMs to
>  which
> >> DTM can send updated tokens
> >> but this doesn't contain the under registration TM while
> >> RegistrationResponse is not processed, right?
> >>
> >> Of course if DTM can update while RegistrationResponse is processed
>  then
> >> somehow sorting would be
> >> required and that case I would agree.
> >>
> >> - Scope DT updates by the RM ID and ensure that TM only accepts
> >>> update
> > from
> >> the current leader
> >>
> >> I've planned this initially the mentioned way so agreed.
> >>
> >> - Return initial token with the RegistrationResponse, which should
> >>> make
> > the
> >> RPC contract bit clearer (ensure that we can't register the TM
> >>> without
> >> providing it with token)
> >>
> >> I think this is a good idea in general. Not sure what you mean "we
>  can't
> >> register the TM without
> >> providing it with token" but in unsecure configuration registration
>  must
> >> happen w/o tokens.
> >> All in all the newly added tokens field must be somehow optional.
> >>
> >> G
> >>
> >>
> >> On Fri, Jan 28, 2022 at 2:22 PM David Morávek 
> >>> wrote:
> >>> We had a long discussion with Chesnay about the possible edge
> >>> cases
>  and
> >> it
> >>> basically boils down to the following two scenarios:
> >>>
> >>> 1) There is a possible race condition between TM registration (the
> > first
> >> DT
> >>> update) and token refresh if they happen simultaneously. Than the
> >>> registration might beat the refreshed token. This could be easily
> >> addressed
> >>> if DTs could be sorted (eg. by the expiration time) on the TM
> >>> side.
>  In
> >>> other words, if there are multiple updates at the same time we
> >>> need
>  to
> >> make
> >>> sure that we have a deterministic way of choosing the latest one.
> >>>
> >>> One idea by Chesnay that popped up during this discussion was
> >>> whether
> > we
> >>> could simply return the initial token with the
> >>> Re

Re: [VOTE] Deprecate Cassandra connector

[Ada Wong]

+1

Martijn Visser  于2022年1月31日周一 18:48写道：
>
> Hi everyone,
>
> I would like to open up a vote to deprecate Cassandra in Flink 1.15 and
> remove it in the next version. I've previously mentioned that we were
> looking for maintainers for Cassandra, but so far none have come forward
> [1].
>
> The vote will last for at least 72 hours, and will be accepted by
> a consensus of active committers.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://lists.apache.org/thread/1qokt5tp8dcp58dmshbwjc43ssbm1vvk

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

[Chesnay Schepler]

You should have permissions now. Note that I saw 2 accounts matching 
your name, and I picked gaborgsomogyi.


On 31/01/2022 11:28, Gabor Somogyi wrote:

Not sure if the mentioned write right already given or not but I still
don't see any edit button.

G


On Fri, Jan 28, 2022 at 5:08 PM Gabor Somogyi 
wrote:


Hi Robert,

That would be awesome.

My cwiki username: gaborgsomogyi

G


On Fri, Jan 28, 2022 at 5:06 PM Robert Metzger 
wrote:


Hey Gabor,

let me know your cwiki username, and I can give you write permissions.


On Fri, Jan 28, 2022 at 4:05 PM Gabor Somogyi 
wrote:


Thanks for making the design better! No further thing to discuss from my
side.

Started to reflect the agreement in the FLIP doc.
Since I don't have access to the wiki I need to ask Marci to do that

which

may take some time.

G


On Fri, Jan 28, 2022 at 3:52 PM David Morávek  wrote:


Hi,

AFAIU an under registration TM is not added to the registered TMs map

until

RegistrationResponse ..


I think you're right, with a careful design around threading

(delegating

update broadcasts to the main thread) + synchronous initial update

(that

would be nice to avoid) this should be doable.

Not sure what you mean "we can't register the TM without providing it

with

token" but in unsecure configuration registration must happen w/o

tokens.

Exactly as you describe it, this was meant only for the "kerberized /
secured" cluster case, in other cases we wouldn't enforce a non-null

token

in the response

I think this is a good idea in general.
+1

If you don't have any more thoughts on the RPC / lifecycle part, can

you

please reflect it into the FLIP?

D.

On Fri, Jan 28, 2022 at 3:16 PM Gabor Somogyi <

gabor.g.somo...@gmail.com

wrote:


- Make sure DTs issued by single DTMs are monotonically increasing

(can

be
sorted on TM side)

AFAIU an under registration TM is not added to the registered TMs

map

until

RegistrationResponse
is processed which would contain the initial tokens. If that's true

then

how is it possible to have race with
DTM update which is working on the registered TMs list?
To be more specific "taskExecutors" is the registered map of TMs to

which

DTM can send updated tokens
but this doesn't contain the under registration TM while
RegistrationResponse is not processed, right?

Of course if DTM can update while RegistrationResponse is processed

then

somehow sorting would be
required and that case I would agree.

- Scope DT updates by the RM ID and ensure that TM only accepts

update

from

the current leader

I've planned this initially the mentioned way so agreed.

- Return initial token with the RegistrationResponse, which should

make

the

RPC contract bit clearer (ensure that we can't register the TM

without

providing it with token)

I think this is a good idea in general. Not sure what you mean "we

can't

register the TM without
providing it with token" but in unsecure configuration registration

must

happen w/o tokens.
All in all the newly added tokens field must be somehow optional.

G


On Fri, Jan 28, 2022 at 2:22 PM David Morávek 

wrote:

We had a long discussion with Chesnay about the possible edge

cases

and

it

basically boils down to the following two scenarios:

1) There is a possible race condition between TM registration (the

first

DT

update) and token refresh if they happen simultaneously. Than the
registration might beat the refreshed token. This could be easily

addressed

if DTs could be sorted (eg. by the expiration time) on the TM

side.

In

other words, if there are multiple updates at the same time we

need

to

make

sure that we have a deterministic way of choosing the latest one.

One idea by Chesnay that popped up during this discussion was

whether

we

could simply return the initial token with the

RegistrationResponse

to

avoid making an extra call during the TM registration.

2) When the RM leadership changes (eg. because zookeeper session

times

out)

there might be a race condition where the old RM is shutting down

and

updates the tokens, that it might again beat the registration

token

of

the

new RM. This could be avoided if we scope the token by

_ResourceManagerId_

and only accept updates for the current leader (basically we'd

have

an

extra parameter to the _updateDelegationToken_ method).

-

DTM is way simpler then for example slot management, which could

receive

updates from the JobMaster that RM might not know about.

So if you want to go in the path you're describing it should be

doable

and

we'd propose following to cover all cases:

- Make sure DTs issued by single DTMs are monotonically increasing

(can

be

sorted on TM side)
- Scope DT updates by the RM ID and ensure that TM only accepts

update

from

the current leader
- Return initial token with the RegistrationResponse, which should

make

the

RPC contract bit clearer (ensure that we can't register the TM

without

providing it with token)

Any thoughts?


On Fri, Jan 28, 2022 at 1

[VOTE] Remove Twitter connector

[Martijn Visser]

Hi everyone,

I would like to open up a vote to remove the Twitter connector in Flink
1.15. This was brought up previously for a discussion [1].

The vote will last for at least 72 hours, and will be accepted by
a consensus of active committers.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1] https://lists.apache.org/thread/7sdvp4hj93rh0cz8r50800stzrpgkdm2

[VOTE] Deprecate Cassandra connector

[Martijn Visser]

Hi everyone,

I would like to open up a vote to deprecate Cassandra in Flink 1.15 and
remove it in the next version. I've previously mentioned that we were
looking for maintainers for Cassandra, but so far none have come forward
[1].

The vote will last for at least 72 hours, and will be accepted by
a consensus of active committers.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1] https://lists.apache.org/thread/1qokt5tp8dcp58dmshbwjc43ssbm1vvk

[VOTE] Deprecate NiFi connector

[Martijn Visser]

Hi everyone,

I would like to open up a vote to deprecate NiFi in Flink 1.15 and remove
it in the next version. I've previously mentioned that we were looking for
maintainers for NiFi, but so far none have come forward [1].

The vote will last for at least 72 hours, and will be accepted by a
consensus of active committers.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1] https://lists.apache.org/thread/1vs3wmk66vsq6l4npjsfzltft4tz5tkq

Re: [DISCUSS] Deprecate/remove Twitter connector

[Martijn Visser]

Hi all,

Thanks for your feedback. It's not about having this connector in the main
repo, that has been voted on already. This is strictly about the connector
itself, since it's not maintained and most probably also can't be used due
to changes in Twitter's API that aren't reflected in our connector
implementation. Therefore I propose to remove it.

Fully agree on the template part, what's good to know is that a connector
template/archetype is part of the goals for the external
connector repository.

Best regards,

Martijn

On Mon, 31 Jan 2022 at 11:32, Francesco Guardiani 
wrote:

> Hi,
>
> I agree with the concern about having this connector in the main repo. But
> I think in general it doesn't harm to have a sample connector to show how
> to develop a custom connector, and I think that the Twitter connector can
> be a good candidate for such a template. It needs rework for sure, as it
> has evident issues, notably it doesn't work with table.
>
> So i understand if we wanna remove what we have right now, but I think we
> should have some replacement for a "connector template", which is both
> ready to use and easy to hack to build your own connector starting from it.
> Twitter API is a good example for such a template, as it's both "related"
> to the known common use cases of Flink and because is quite simple to get
> started with.
>
> FG
>
> On Sun, Jan 30, 2022 at 12:31 PM David Anderson 
> wrote:
>
>> I agree.
>>
>> The Twitter connector is used in a few (unofficial) tutorials, so if we
>> remove it that will make it more difficult for those tutorials to be
>> maintained. On the other hand, if I recall correctly, that connector uses
>> V1 of the Twitter API, which has been deprecated, so it's really not very
>> useful even for that purpose.
>>
>> David
>>
>>
>>
>> On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to discuss deprecating Flinks' Twitter connector [1]. This
>>> was one of the first connectors that was added to Flink, which could be
>>> used to access the tweets from Twitter. Given the evolution of Flink over
>>> Twitter, I don't think that:
>>>
>>> * Users are still using this connector at all
>>> * That the code for this connector should be in the main Flink codebase.
>>>
>>> Given the circumstances, I would propose to deprecate and remove this
>>> connector. I'm looking forward to your thoughts. If you agree, please also
>>> let me know if you think we should first deprecate it in Flink 1.15 and
>>> remove it in a version after that, or if you think we can remove it
>>> directly.
>>>
>>> Best regards,
>>>
>>> Martijn Visser
>>> https://twitter.com/MartijnVisser82
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/twitter/
>>>
>>>

Re: [DISCUSS] Deprecate/remove Twitter connector

[Francesco Guardiani]

Hi,

I agree with the concern about having this connector in the main repo. But
I think in general it doesn't harm to have a sample connector to show how
to develop a custom connector, and I think that the Twitter connector can
be a good candidate for such a template. It needs rework for sure, as it
has evident issues, notably it doesn't work with table.

So i understand if we wanna remove what we have right now, but I think we
should have some replacement for a "connector template", which is both
ready to use and easy to hack to build your own connector starting from it.
Twitter API is a good example for such a template, as it's both "related"
to the known common use cases of Flink and because is quite simple to get
started with.

FG

On Sun, Jan 30, 2022 at 12:31 PM David Anderson 
wrote:

> I agree.
>
> The Twitter connector is used in a few (unofficial) tutorials, so if we
> remove it that will make it more difficult for those tutorials to be
> maintained. On the other hand, if I recall correctly, that connector uses
> V1 of the Twitter API, which has been deprecated, so it's really not very
> useful even for that purpose.
>
> David
>
>
>
> On Fri, Jan 21, 2022 at 9:34 AM Martijn Visser 
> wrote:
>
>> Hi everyone,
>>
>> I would like to discuss deprecating Flinks' Twitter connector [1]. This
>> was one of the first connectors that was added to Flink, which could be
>> used to access the tweets from Twitter. Given the evolution of Flink over
>> Twitter, I don't think that:
>>
>> * Users are still using this connector at all
>> * That the code for this connector should be in the main Flink codebase.
>>
>> Given the circumstances, I would propose to deprecate and remove this
>> connector. I'm looking forward to your thoughts. If you agree, please also
>> let me know if you think we should first deprecate it in Flink 1.15 and
>> remove it in a version after that, or if you think we can remove it
>> directly.
>>
>> Best regards,
>>
>> Martijn Visser
>> https://twitter.com/MartijnVisser82
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/twitter/
>>
>>

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

[Gabor Somogyi]

Not sure if the mentioned write right already given or not but I still
don't see any edit button.

G


On Fri, Jan 28, 2022 at 5:08 PM Gabor Somogyi 
wrote:

> Hi Robert,
>
> That would be awesome.
>
> My cwiki username: gaborgsomogyi
>
> G
>
>
> On Fri, Jan 28, 2022 at 5:06 PM Robert Metzger 
> wrote:
>
>> Hey Gabor,
>>
>> let me know your cwiki username, and I can give you write permissions.
>>
>>
>> On Fri, Jan 28, 2022 at 4:05 PM Gabor Somogyi 
>> wrote:
>>
>> > Thanks for making the design better! No further thing to discuss from my
>> > side.
>> >
>> > Started to reflect the agreement in the FLIP doc.
>> > Since I don't have access to the wiki I need to ask Marci to do that
>> which
>> > may take some time.
>> >
>> > G
>> >
>> >
>> > On Fri, Jan 28, 2022 at 3:52 PM David Morávek  wrote:
>> >
>> > > Hi,
>> > >
>> > > AFAIU an under registration TM is not added to the registered TMs map
>> > until
>> > > > RegistrationResponse ..
>> > > >
>> > >
>> > > I think you're right, with a careful design around threading
>> (delegating
>> > > update broadcasts to the main thread) + synchronous initial update
>> (that
>> > > would be nice to avoid) this should be doable.
>> > >
>> > > Not sure what you mean "we can't register the TM without providing it
>> > with
>> > > > token" but in unsecure configuration registration must happen w/o
>> > tokens.
>> > > >
>> > >
>> > > Exactly as you describe it, this was meant only for the "kerberized /
>> > > secured" cluster case, in other cases we wouldn't enforce a non-null
>> > token
>> > > in the response
>> > >
>> > > I think this is a good idea in general.
>> > > >
>> > >
>> > > +1
>> > >
>> > > If you don't have any more thoughts on the RPC / lifecycle part, can
>> you
>> > > please reflect it into the FLIP?
>> > >
>> > > D.
>> > >
>> > > On Fri, Jan 28, 2022 at 3:16 PM Gabor Somogyi <
>> gabor.g.somo...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > > - Make sure DTs issued by single DTMs are monotonically increasing
>> > (can
>> > > > be
>> > > > sorted on TM side)
>> > > >
>> > > > AFAIU an under registration TM is not added to the registered TMs
>> map
>> > > until
>> > > > RegistrationResponse
>> > > > is processed which would contain the initial tokens. If that's true
>> > then
>> > > > how is it possible to have race with
>> > > > DTM update which is working on the registered TMs list?
>> > > > To be more specific "taskExecutors" is the registered map of TMs to
>> > which
>> > > > DTM can send updated tokens
>> > > > but this doesn't contain the under registration TM while
>> > > > RegistrationResponse is not processed, right?
>> > > >
>> > > > Of course if DTM can update while RegistrationResponse is processed
>> > then
>> > > > somehow sorting would be
>> > > > required and that case I would agree.
>> > > >
>> > > > - Scope DT updates by the RM ID and ensure that TM only accepts
>> update
>> > > from
>> > > > the current leader
>> > > >
>> > > > I've planned this initially the mentioned way so agreed.
>> > > >
>> > > > - Return initial token with the RegistrationResponse, which should
>> make
>> > > the
>> > > > RPC contract bit clearer (ensure that we can't register the TM
>> without
>> > > > providing it with token)
>> > > >
>> > > > I think this is a good idea in general. Not sure what you mean "we
>> > can't
>> > > > register the TM without
>> > > > providing it with token" but in unsecure configuration registration
>> > must
>> > > > happen w/o tokens.
>> > > > All in all the newly added tokens field must be somehow optional.
>> > > >
>> > > > G
>> > > >
>> > > >
>> > > > On Fri, Jan 28, 2022 at 2:22 PM David Morávek 
>> wrote:
>> > > >
>> > > > > We had a long discussion with Chesnay about the possible edge
>> cases
>> > and
>> > > > it
>> > > > > basically boils down to the following two scenarios:
>> > > > >
>> > > > > 1) There is a possible race condition between TM registration (the
>> > > first
>> > > > DT
>> > > > > update) and token refresh if they happen simultaneously. Than the
>> > > > > registration might beat the refreshed token. This could be easily
>> > > > addressed
>> > > > > if DTs could be sorted (eg. by the expiration time) on the TM
>> side.
>> > In
>> > > > > other words, if there are multiple updates at the same time we
>> need
>> > to
>> > > > make
>> > > > > sure that we have a deterministic way of choosing the latest one.
>> > > > >
>> > > > > One idea by Chesnay that popped up during this discussion was
>> whether
>> > > we
>> > > > > could simply return the initial token with the
>> RegistrationResponse
>> > to
>> > > > > avoid making an extra call during the TM registration.
>> > > > >
>> > > > > 2) When the RM leadership changes (eg. because zookeeper session
>> > times
>> > > > out)
>> > > > > there might be a race condition where the old RM is shutting down
>> and
>> > > > > updates the tokens, that it might again beat the registration
>> token
>> > of
>> > > > the
>> > > > > new RM. This could be avoided if we scope the

[jira] [Created] (FLINK-25893) ResourceManagerServiceImpl's lifecycle can lead to exceptions

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-25893:
-

 Summary: ResourceManagerServiceImpl's lifecycle can lead to 
exceptions
 Key: FLINK-25893
 URL: https://issues.apache.org/jira/browse/FLINK-25893
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.14.3, 1.15.0
Reporter: Till Rohrmann


The {{ResourceManagerServiceImpl}} lifecycle can lead to exceptions when 
calling {{ResourceManagerServiceImpl.deregisterApplication}}. The problem 
arises when the {{DispatcherResourceManagerComponent}} is shutdown before the 
{{ResourceManagerServiceImpl}} gains leadership or while it is starting the 
{{ResourceManager}}.

One problem is that {{deregisterApplication}} returns an exceptionally 
completed future if there is no leading {{ResourceManager}}.

Another problem is that if there is a leading {{ResourceManager}}, then it can 
still be the case that it has not been started yet. If this is the case, then 
[ResourceManagerGateway.deregisterApplication|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManagerServiceImpl.java#L143]
 will be discarded. The reason for this behaviour is that we create a 
{{ResourceManager}} in one {{Runnable}} and only start it in another. Due to 
this there can be the {{deregisterApplication}} call that gets the {{lock}} in 
between.

I'd suggest to correct the lifecycle and contract of the 
{{ResourceManagerServiceImpl.deregisterApplication}}.

Please note that due to this problem, the error reporting of this method has 
been suppressed. See FLINK-25885 for more details.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25892) Develop ArchUnit test for connectors

[Jing Ge (Jira)]

Jing Ge created FLINK-25892:
---

 Summary: Develop ArchUnit test for connectors
 Key: FLINK-25892
 URL: https://issues.apache.org/jira/browse/FLINK-25892
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Jing Ge
Assignee: Jing Ge


ArchUnit test should be developed for connector submodules after the ArchUnit 
infra for test code has been built.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25891) NoClassDefFoundError AsyncSSLPrivateKeyMethod in benchmark

[Anton Kalashnikov (Jira)]

Anton Kalashnikov created FLINK-25891:
-

 Summary: NoClassDefFoundError AsyncSSLPrivateKeyMethod in benchmark
 Key: FLINK-25891
 URL: https://issues.apache.org/jira/browse/FLINK-25891
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Affects Versions: 1.15.0
Reporter: Anton Kalashnikov






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25890) Implement scroll tracking on Flink documentation

[Martijn Visser (Jira)]

Martijn Visser created FLINK-25890:
--

 Summary: Implement scroll tracking on Flink documentation
 Key: FLINK-25890
 URL: https://issues.apache.org/jira/browse/FLINK-25890
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25889) Implement scroll tracking on Flink project website

[Martijn Visser (Jira)]

Martijn Visser created FLINK-25889:
--

 Summary: Implement scroll tracking on Flink project website
 Key: FLINK-25889
 URL: https://issues.apache.org/jira/browse/FLINK-25889
 Project: Flink
  Issue Type: Sub-task
  Components: Project Website
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] FLIP-211: Kerberos delegation token framework

[Márton Balassi]

+1 (binding)

Given [1] I consider the issue David raised resolved. Thanks David and
please confirm here.

[1]  https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj

On Mon, Jan 24, 2022 at 11:07 AM David Morávek 
wrote:

> Hi Gabor,
>
> Thanks for driving this. This is headed in a right direction, but I feel
> that the FLIP still might need bit more work.
>
> -1 (non-binding) until the discussion thread is resolved [1].
>
> [1] https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj
>
> Best,
> D.
>
>
>
> On Mon, Jan 24, 2022 at 10:47 AM Gyula Fóra  wrote:
>
> > Hi Gabor,
> >
> > +1 (binding) from me
> >
> > This is a great effort and significant improvement to the Kerberos
> security
> > story .
> >
> > Cheers
> > Gyula
> >
> > On Fri, 21 Jan 2022 at 15:58, Gabor Somogyi 
> > wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start the vote for FLIP-211 [1], which was discussed
> and
> > > reached a consensus in the discussion thread [2].
> > >
> > > The vote will be open for at least 72h, unless there is an objection or
> > not
> > > enough votes.
> > >
> > > BR,
> > > G
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework
> > >
> > > [2] https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj
> > >
> >
>

[jira] [Created] (FLINK-25888) Capture time that the job spends on deploying tasks

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-25888:


 Summary: Capture time that the job spends on deploying tasks
 Key: FLINK-25888
 URL: https://issues.apache.org/jira/browse/FLINK-25888
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Coordination, Runtime / Metrics
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0


FLINK-23976 added standardized metrics for capturing how much time we spend in 
each {{JobStatus}}. However, certain states in practice consist of several 
stages; for example the RUNNING state also includes the deployment of tasks.

To get a better picture on where time is spent I propose to add new metrics 
that capture the deployingTime based on the execution states. This will 
additionally get us closer to a proper uptime metric, which ideally will be 
runningTime - various stage time metrics.

A job is considered to be deploying,
* for batch jobs, if no task is running and at least one task is being deployed
* for streaming jobs, if at least one task is being deployed

The semantics are different for batch/streaming jobs because they differ in 
terms of how they make progress. For a streaming job all tasks need to be 
deployed for checkpointing to make work. For batch jobs any deployed task 
immediately starts progressing the job.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25887) FLIP-193: Snapshots ownership follow ups

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-25887:


 Summary: FLIP-193: Snapshots ownership follow ups
 Key: FLINK-25887
 URL: https://issues.apache.org/jira/browse/FLINK-25887
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Dawid Wysakowicz
 Fix For: 1.16.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25886) Hard-coded ZK version in FlinkContainersBuilder#buildZookeeperContainer

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-25886:


 Summary: Hard-coded ZK version in 
FlinkContainersBuilder#buildZookeeperContainer
 Key: FLINK-25886
 URL: https://issues.apache.org/jira/browse/FLINK-25886
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Affects Versions: 1.15.0
Reporter: Chesnay Schepler


The Zookeeper version is hard-coded in 
FlinkContainersBuilder#buildZookeeperContainer, when it ideally should be tied 
to the {{zookeeper.version}} maven property.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)