Sorry I didn't want to offend anybody if it was perceived like this. I can
see that me joining very late into the discussion w/o constructive ideas
was not nice. My motivation for asking for the reasoning behind the current
design proposal is primarily the lack of Kerberos knowledge. Moreover, it
h
Hi Team!
Let's all calm down a little and not let our emotions affect the discussion
too much.
There has been a lot of effort spent from all involved parties so this is
quite understandable :)
Even though not everyone said this explicitly, it seems that everyone more
or less agrees that a feature
First of, at no point have we questioned the use-case and importance of
this feature, and the fact that David, Till and me spent time looking at
the FLIP, asking questions, and discussing different aspects of it
should make this obvious.
I'd appreciate it if you didn't dismiss our replies that
> And even
if we do it like this, there is no guarantee that it works because there
can be other applications bombing the KDC with requests.
1. The main issue to solve here is that workloads using delegation tokens
are stopping after 7 days with default configuration.
2. This is not new design, it
Hi Till!
The delegation token framework solves a few production problems, KDC
scalability is just one and probably not the most important.
As Gabor has explained some of which are:
- Solves the problem for token renewal for long running jobs which would
currently time out and die
- Improves sec
I don't have a good alternative solution but it sounds to me a bit as if we
are trying to solve Kerberos' scalability problems within Flink. And even
if we do it like this, there is no guarantee that it works because there
can be other applications bombing the KDC with requests. From a
maintainabil
Oh and the most important reason I've forgotten.
Without the feature in the FLIP all secure workloads with delegation tokens
are going to stop when tokens are reaching it's max lifetime 🙂
This is around 7 days with default config...
On Thu, Feb 3, 2022 at 5:30 PM Gabor Somogyi
wrote:
> That's no
That's not the single purpose of the feature but in some environments it
caused problems.
The main intention is not to deploy keytab to all the nodes because the
attack surface is bigger + reduce the KDC load.
I've already described the situation previously in this thread so copying
it here.
-
What I don't understand is how this could overload the KDC. Aren't
tokens valid for a relatively long time period?
For new deployments where many TMs are started at once I could imagine
it temporarily, but shouldn't the accesses to the KDC eventually
naturally spread out?
The FLIP mentions s
Dawid Wysakowicz created FLINK-25952:
Summary: Savepoint on S3 are not relocatable even if entropy
injection is not enabled
Key: FLINK-25952
URL: https://issues.apache.org/jira/browse/FLINK-25952
+1
On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser
wrote:
> Hi everyone,
>
> I would like to open up a vote to remove the Twitter connector in Flink
> 1.15. This was brought up previously for a discussion [1].
>
> The vote will last for at least 72 hours, and will be accepted by
> a consensus of
> I would prefer not choosing the first option
Then the second option may play only.
> I am not a Kerberos expert but is it really so that every application that
wants to use Kerberos needs to implement the token propagation itself? This
somehow feels as if there is something missing.
OK, so fir
+1 (binding)
On Thu, Feb 3, 2022 at 8:39 AM Fabian Paul wrote:
> This connector is really a relict of the past.
>
> +1 (binding)
>
> Best,
> Fabian
>
> On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser
> wrote:
> >
> > Hi everyone,
> >
> > I would like to open up a vote to remove the Twitter conn
+1 (binding)
On Thu, Feb 3, 2022 at 8:44 AM Fabian Paul wrote:
> Thanks for driving the deprecation efforts.
>
> +1 (binding)
>
> Best,
> Fabian
>
> On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser
> wrote:
> >
> > Hi everyone,
> >
> > I would like to open up a vote to deprecate NiFi in Flink 1.
Great news. Thanks for driving this effort Martijn and helping with it
Chesnay and Konstantin :-)
Cheers,
Till
On Thu, Feb 3, 2022 at 3:13 PM Martijn Visser wrote:
> Hi everyone,
>
> A short update from my end: with the help of Chesnay and Konstantin both
> the Project Website, all Flink docume
I would prefer not choosing the first option
> Make the TM accept tasks only after registration(not sure if it's
possible or makes sense at all)
because it effectively means that we change how Flink's component lifecycle
works for distributing Kerberos tokens. It also effectively means that a TM
Hi everyone,
+1. If or when someone starts an effort of migrating the new APIs in the
future, the code will still be accessible even if it's already deleted in
the latest versions.
Thanks,
Konstantin
On Mon, Jan 31, 2022 at 11:46 AM Martijn Visser
wrote:
> Hi everyone,
>
> I would like to ope
This connector is really a relict of the past.
+1 (binding)
Best,
Fabian
On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser wrote:
>
> Hi everyone,
>
> I would like to open up a vote to remove the Twitter connector in Flink
> 1.15. This was brought up previously for a discussion [1].
>
> The vote
+1 (binding)
Best
Ingo
On 31.01.22 11:46, Martijn Visser wrote:
Hi everyone,
I would like to open up a vote to remove the Twitter connector in Flink
1.15. This was brought up previously for a discussion [1].
The vote will last for at least 72 hours, and will be accepted by
a consensus of act
Thanks for driving the deprecation efforts.
+1 (binding)
Best,
Fabian
On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser wrote:
>
> Hi everyone,
>
> I would like to open up a vote to deprecate NiFi in Flink 1.15 and remove
> it in the next version. I've previously mentioned that we were looking fo
> Isn't this something the underlying resource management system could do
or which every process could do on its own?
I was looking for such feature but not found.
Maybe we can solve the propagation easier but then I'm waiting on better
suggestion.
If anybody has better/more simple idea then plea
Martijn Visser created FLINK-25951:
--
Summary: Implement Matomo on JavaDocs
Key: FLINK-25951
URL: https://issues.apache.org/jira/browse/FLINK-25951
Project: Flink
Issue Type: Sub-task
Hi everyone,
A short update from my end: with the help of Chesnay and Konstantin both
the Project Website, all Flink documentation (back to 1.0) and the Statefun
websites no longer send data to Google Analytics, only to Matomo. The
privacy policy has also been adjusted and it includes an opt-out.
Hi everyone,
Sorry for joining this discussion late. I also did not read all responses
in this thread so my question might already be answered: Why does Flink
need to be involved in the propagation of the tokens? Why do we need
explicit RPC calls in the Flink domain? Isn't this something the under
Matthias Pohl created FLINK-25950:
-
Summary: Delete retry mechanism from ZooKeeperUtils.deleteZNode
Key: FLINK-25950
URL: https://issues.apache.org/jira/browse/FLINK-25950
Project: Flink
Issu
Here's an example for the TM to run workloads without being connected to
the RM, while potentially having a valid token:
1. TM registers at RM
2. JobMaster requests slot from RM -> TM gets notified
3. JM fails over
4. TM re-offers the slot to the failed over JobMaster
5. TM reconnects to RM at s
> but it can happen that the JobMaster+TM collaborate to run stuff without
the TM being registered at the RM
Honestly I'm not educated enough within Flink to give an example to such
scenario.
Until now I thought JM defines tasks to be done and TM just blindly
connects to external systems and does
> Just to learn something new. I think local recovery is clear to me
which is not touching external systems like Kafka or so (correct me if
I'm wrong). Is it possible that such case the user code just starts to
run blindly w/o JM coordination and connects to external systems to do
data processi
> Any error in loading the provider (be it by accident or explicit checks)
then is a setup error and we can fail the cluster.
Fail fast is a good direction in my view. In Spark I wanted to go to this
direction but there were other opinions so there if a provider is not
loaded then the workload goe
Ahmed Hamdy created FLINK-25949:
---
Summary: [FLIP-171] Kinesis Firehose sink builder falls back to
wrong http protocol.
Key: FLINK-25949
URL: https://issues.apache.org/jira/browse/FLINK-25949
Project: Fl
1)
The manager certainly shouldn't check for specific implementations.
The problem with classpath-based checks is it can easily happen that the
provider can't be loaded in the first place (e.g., if you don't use
reflection, which you currently kinda force), and in that case Flink
can't tell whe
Zichen Liu created FLINK-25948:
--
Summary: KDS / KDF Sink should call .close() to clean up resources
Key: FLINK-25948
URL: https://issues.apache.org/jira/browse/FLINK-25948
Project: Flink
Issue T
Thanks for the quick response!
Appreciate your invested time...
G
On Thu, Feb 3, 2022 at 11:12 AM Chesnay Schepler wrote:
> Thanks for answering the questions!
>
> 1) Does the HBase provider require HBase to be on the classpath?
>
To be instantiated no, to obtain a token yes.
> If so, th
Chesnay Schepler created FLINK-25947:
Summary: License checker doesn't flink-table-planner
Key: FLINK-25947
URL: https://issues.apache.org/jira/browse/FLINK-25947
Project: Flink
Issue Typ
Chesnay Schepler created FLINK-25946:
Summary: table-planner-loader jar NOTICE should list Scala
Key: FLINK-25946
URL: https://issues.apache.org/jira/browse/FLINK-25946
Project: Flink
Iss
Zichen Liu created FLINK-25945:
--
Summary: Intermittent Failures on KDF AZP 2
Key: FLINK-25945
URL: https://issues.apache.org/jira/browse/FLINK-25945
Project: Flink
Issue Type: New Feature
Zichen Liu created FLINK-25944:
--
Summary: Intermittent Failures on KDF AZP
Key: FLINK-25944
URL: https://issues.apache.org/jira/browse/FLINK-25944
Project: Flink
Issue Type: New Feature
Thanks for answering the questions!
1) Does the HBase provider require HBase to be on the classpath?
If so, then could it even be loaded if Hbase is on the classpath?
If not, then you're assuming the classpath of the JM/TM to be the
same, which isn't necessarily true (in general; and als
Zichen Liu created FLINK-25943:
--
Summary: New Kinesis, Firehose to provide a state serializer
Key: FLINK-25943
URL: https://issues.apache.org/jira/browse/FLINK-25943
Project: Flink
Issue Type: N
Francesco Guardiani created FLINK-25942:
---
Summary: Use jackson jdk8/time modules for Duration ser/de
Key: FLINK-25942
URL: https://issues.apache.org/jira/browse/FLINK-25942
Project: Flink
Till Rohrmann created FLINK-25941:
-
Summary:
KafkaSinkITCase.testAbortTransactionsAfterScaleInBeforeFirstCheckpoint fails on
AZP
Key: FLINK-25941
URL: https://issues.apache.org/jira/browse/FLINK-25941
Till Rohrmann created FLINK-25940:
-
Summary:
pyflink/datastream/tests/test_data_stream.py::StreamingModeDataStreamTests::test_keyed_process_function_with_state
failed on AZP
Key: FLINK-25940
URL: https://issues.a
Till Rohrmann created FLINK-25939:
-
Summary: PyFlink YARN per-job on Docker test fails on AZP because
it could not acquire all required slots
Key: FLINK-25939
URL: https://issues.apache.org/jira/browse/FLINK-25939
Till Rohrmann created FLINK-25938:
-
Summary:
SynchronousCheckpointITCase.taskDispatcherThreadPoolAllowsForSynchronousCheckpoints
fails on AZP
Key: FLINK-25938
URL: https://issues.apache.org/jira/browse/FLINK-2593
Please see my answers inline. Hope provided satisfying answers to all
questions.
G
On Thu, Feb 3, 2022 at 9:17 AM Chesnay Schepler wrote:
> I have a few question that I'd appreciate if you could answer them.
>
>1. How does the Provider know whether it is required or not?
>
> All registered
Till Rohrmann created FLINK-25937:
-
Summary: SQL Client end-to-end test e2e fails on AZP
Key: FLINK-25937
URL: https://issues.apache.org/jira/browse/FLINK-25937
Project: Flink
Issue Type: Bug
Hi Till,
Thanks.
Please tell me if you want that we do the review of
https://github.com/apache/flink/pull/18610 before Friday 3h pm so that I
book some time or if we do that when I get back.
Best
Etienne
Le 02/02/2022 à 16:48, Till Rohrmann a écrit :
Thanks for letting us know Etienne. Ha
I have a few question that I'd appreciate if you could answer them.
1. How does the Provider know whether it is required or not?
2. How does the configuration of Providers work (how do they get access
to a configuration)?
3. How does a user select providers? (Is it purely based on the
provi
48 matches
Mail list logo
