Re: Re: [VOTE] FLIP-415: Introduce a new join operator to support minibatch

2024-01-18 Thread liu ron
+1(binding)

Best,
Ron

Xuyang  于2024年1月19日周五 13:58写道:

> +1 (non-binding)--
>
> Best!
> Xuyang
>
>
>
>
>
> 在 2024-01-19 13:28:52,"Lincoln Lee"  写道:
> >+1 (binding)
> >
> >Best,
> >Lincoln Lee
> >
> >
> >Benchao Li  于2024年1月19日周五 13:15写道:
> >
> >> +1 (binding)
> >>
> >> shuai xu  于2024年1月19日周五 12:58写道:
> >>
> >>> Dear Flink Developers,
> >>>
> >>> Thank you for providing feedback on FLIP-415: Introduce a new join
> >>> operator to support minibatch[1]. I'd like to start a vote on this
> FLIP.
> >>> Here is the discussion thread[2].
> >>>
> >>> After the discussion, this FLIP will not introduce any new Option. The
> >>> minibatch join will default to compacting the changelog. As for the
> option
> >>> to control compaction within the minibatch that was mentioned in the
> >>> discussion, it could be discussed in a future FLIP.
> >>>
> >>> The vote will be open for at least 72 hours unless there is an
> objection
> >>> or
> >>> insufficient votes.
> >>>
> >>> Best,
> >>> Xu Shuai
> >>>
> >>> [1]
> >>> FLIP-415: Introduce a new join operator to support minibatch - Apache
> >>> Flink - Apache Software Foundation
> >>> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> >
> >>> cwiki.apache.org
> >>> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> >
> >>> [image: favicon.ico]
> >>> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> >
> >>> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
> >
> >>> [2]
> >>> lists.apache.org
> >>> 
> >>> [image: favicon.ico]
> >>> 
> >>> 
> >>>
> >>>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
> >>
>


Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

2024-01-18 Thread Zakelly Lan
Hi all,

After rethinking, I‘ve come to agree that the Flink 2.0 would be a fittable
version for this kind of change. I'd like to reopen this discussion for
making this in 2.0. I suggest modifying the exception directly in Flink 2.0
rather than introducing a new set of APIs without any additional
adjustments. Having two sets of APIs with only exceptions changed could
lead to confusion and unnecessary maintenance burden. WDYT? @Jing Ge @David
Radley @Yuan Mei


Best,
Zakelly

On Mon, Oct 16, 2023 at 7:20 PM Jing Ge  wrote:

> Hi Zakelly,
>
> Thanks for the clarification. But I have to agree with Yuan. It is a
> breaking change as you mentioned that users who catch those exceptions will
> be forced to change their code. Changing it during the minor releases
> violates the backward compatibility promise of Flink public(Evolving) API.
> I would suggest explicitly writing down the breaking change in the FLIP and
> let the community decide. For me, it might not be an optimized solution to
> introduce breaking changes between minor releases, but I personally won't
> block your effort.
>
> Best regards,
> Jing
>
> On Fri, Oct 13, 2023 at 9:37 PM Zakelly Lan  wrote:
>
> > Hi Jing,
> >
> > For jobs that catch exceptions, they will need to modify their code.
> > If they still wish to catch the exception, they should catch the
> > `Throwable` instead.
> >
> > As I explained, this use case is not common because:
> > 1. The user-defined processElement function, where the state APIs are
> > called, already declares that it throws `Exception`, so there is no
> > need for user code to catch these exceptions. The compiler or IDE will
> > not prompt them to do so.
> > 2. These exceptions are fatal, and users can do very little or nothing
> > to prevent a job failure. Also in this case, Flink already provides
> > error logging for users.
> >
> > And for minor releases, APIs annotated with @PublicEvolving are
> > allowed to have signature changes[1]. So I think the callers will
> > expect this when migrating to a new version.
> >
> >
> > Hi David,
> >
> > It is true that recompilation is necessary when migrating to the next
> > minor release, as Flink is a rapidly evolving project. API
> > compatibility is essential, but it should not hinder development.
> > That's why the API compatibility guarantees of Flink[1] were created.
> > If we expect stable binaries/sources across dot versions, then we
> > should prohibit any breaking changes, not just this one. Therefore, we
> > can discuss a new rule for this. However, for now, this proposal
> > aligns with the current rules.
> >
> > Regarding Flink 2.0, if my estimate is correct, it will be released at
> > least one year later. The API will undergo a significant redesign, and
> > user code may need to be essentially rewritten. The main focus of
> > Flink 2.0 will be on API design, not on reorganizing exceptions as
> > proposed here. After Flink 2.0 is released, many existing users may
> > continue using Flink 1.x for a long time. This work aims to provide a
> > cleaner API for Flink 1.x before we stop updating it.
> >
> >
> > Best,
> > Zakelly
> >
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
> >
> >
> > On Fri, Oct 13, 2023 at 11:16 PM David Radley 
> > wrote:
> > >
> > > Hi,
> > > It seems that migrating to the next dot version and finding you need to
> > recompile, would be frustrating and couse unexpected work, as I suspect a
> > jar file using the old API will give an exception around not finding the
> > method – I am not sure how this would surface in typical applications at
> > runtime.
> > >
> > > Technically because this is tagged as @PublicEvolving then we can make
> > this breaking change.
> > >
> > > So there will be migration issues if people are using this API, have we
> > an idea on how many of our users are using it?
> > >
> > > If we use @PublicEvolving then maybe we should have stable binaries
> that
> > only include public APIs and then another bleeding edge package
> containing
> > @PublicEvolving content, so users can choose.
> > >
> > > Organisations I have worked with would not tend to want to or expect to
> > have to recompile their applications on a dot version – as this would
> > normally mean a lot more testing for them.
> > >
> > > On balance, as I am risk averse, I would suggest delaying this to v2 as
> > Jing has proposed. This is a cleaner API, is there a demand for this in a
> > dot version? If the community think this is too risk averse, then we
> could
> > go with 1.19.
> > > WDYT?
> > >
> > > Kind regards, David.
> > >
> > >
> > >
> > > From: Jing Ge 
> > > Date: Friday, 13 October 2023 at 14:30
> > > To: dev@flink.apache.org 
> > > Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the
> > exceptions thrown in state interfaces
> > > HI Zakelly,
> > >
> > > What about the jobs that catch those exceptions? Will these downstream
> > > callers that expect this e

[jira] [Created] (FLINK-34168) Refactor all callers that using the public Xxx getXxx(String key) and public void setXxx(String key, Xxx value)

2024-01-18 Thread Rui Fan (Jira)
Rui Fan created FLINK-34168:
---

 Summary: Refactor all callers that using the public Xxx 
getXxx(String key) and public void setXxx(String key, Xxx value) 
 Key: FLINK-34168
 URL: https://issues.apache.org/jira/browse/FLINK-34168
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration
Reporter: Rui Fan
Assignee: Xuannan Su


Refactor all callers that using the public Xxx getXxx(String key) and public 
void setXxx(String key, Xxx value)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


退订

2024-01-18 Thread dev
退订



[jira] [Created] (FLINK-34167) add dependence to fit jdk21

2024-01-18 Thread blackpighe (Jira)
blackpighe created FLINK-34167:
--

 Summary: add dependence to fit jdk21
 Key: FLINK-34167
 URL: https://issues.apache.org/jira/browse/FLINK-34167
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC
Affects Versions: jdbc-3.1.1
Reporter: blackpighe


add dependence to fit jdk21

When pipelining jdk 21+flink 1.19, an error occurred with the message
{code:java}
javax.activation.UnsupportedDataTypeException {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34166) KeyedLookupJoinWrapper incorrectly process delete message for inner join when previous lookup result is empty

2024-01-18 Thread lincoln lee (Jira)
lincoln lee created FLINK-34166:
---

 Summary: KeyedLookupJoinWrapper incorrectly process delete message 
for inner join when previous lookup result is empty
 Key: FLINK-34166
 URL: https://issues.apache.org/jira/browse/FLINK-34166
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.18.1, 1.17.2
Reporter: lincoln lee
Assignee: lincoln lee
 Fix For: 1.19.0, 1.18.2


KeyedLookupJoinWrapper(when 'table.optimizer.non-deterministic-update.strategy
' is set to 'TRY_RESOLVE' and the lookup join exists NDU problemns) incorrectly 
process delete message for inner join when previous lookup result is empty

The intermediate delete result 
{code}
expectedOutput.add(deleteRecord(3, "c", null, null));
{code}
in current case 
KeyedLookupJoinHarnessTest#testTemporalInnerJoinWithFilterLookupKeyContainsPk 
is incorrect:
{code}
@Test
public void testTemporalInnerJoinWithFilterLookupKeyContainsPk() throws 
Exception {
OneInputStreamOperatorTestHarness testHarness =
createHarness(JoinType.INNER_JOIN, FilterOnTable.WITH_FILTER, 
true);

testHarness.open();

testHarness.processElement(insertRecord(1, "a"));
testHarness.processElement(insertRecord(2, "b"));
testHarness.processElement(insertRecord(3, "c"));
testHarness.processElement(insertRecord(4, "d"));
testHarness.processElement(insertRecord(5, "e"));
testHarness.processElement(updateBeforeRecord(3, "c"));
testHarness.processElement(updateAfterRecord(3, "c2"));
testHarness.processElement(deleteRecord(3, "c2"));
testHarness.processElement(insertRecord(3, "c3"));

List expectedOutput = new ArrayList<>();
expectedOutput.add(insertRecord(1, "a", 1, "Julian"));
expectedOutput.add(insertRecord(4, "d", 4, "Fabian"));
expectedOutput.add(deleteRecord(3, "c", null, null));
expectedOutput.add(insertRecord(3, "c2", 6, "Jark-2"));
expectedOutput.add(deleteRecord(3, "c2", 6, "Jark-2"));
expectedOutput.add(insertRecord(3, "c3", 9, "Jark-3"));

assertor.assertOutputEquals("output wrong.", expectedOutput, 
testHarness.getOutput());
testHarness.close();
}
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Confluence access

2024-01-18 Thread Сергей Парышев

Hi devs! Can I get access to confluence? I want suggest a FLIP.

[jira] [Created] (FLINK-34165) It seems that Apache download link has been changed

2024-01-18 Thread Jing Ge (Jira)
Jing Ge created FLINK-34165:
---

 Summary: It seems that Apache download link has been changed
 Key: FLINK-34165
 URL: https://issues.apache.org/jira/browse/FLINK-34165
 Project: Flink
  Issue Type: Bug
  Components: flink-docker
Affects Versions: 1.18.1, 1.17.2, 1.16.3, 1.15.4
Reporter: Jing Ge
 Attachments: image-2024-01-19-07-55-07-775.png

The link 
[https://www.apache.org/dist/flink/flink-1.17.2/flink-1.17.2-bin-scala_2.12.tgz.asc][1]
 worked previously now redirect to a list page which leads to a wrong 
flink.tgz.asc with HTML instead of expected signature.

!image-2024-01-19-07-55-07-775.png!

The link should be replace with 
https://downloads.apache.org/flink/flink-1.17.2/flink-1.17.2-bin-scala_2.12.tgz.asc

 

[1] 
https://github.com/apache/flink-docker/blob/627987997ca7ec86bcc3d80b26df58aa595b91af/1.17/scala_2.12-java11-ubuntu/Dockerfile#L48C19-L48C101



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34164) [Benchmark] Compilation error since Jan. 16th

2024-01-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34164:
---

 Summary: [Benchmark] Compilation error since Jan. 16th
 Key: FLINK-34164
 URL: https://issues.apache.org/jira/browse/FLINK-34164
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Reporter: Zakelly Lan
Assignee: Junrui Li


An error occured during the benchmark compile:
{code:java}
13:17:40 [ERROR] 
/mnt/jenkins/workspace/flink-main-benchmarks/flink-benchmarks/warning:[options] 
bootstrap class path not set in conjunction with -source 8
13:17:40 
/mnt/jenkins/workspace/flink-main-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/benchmark/StreamGraphUtils.java:38:19:
 error: cannot find symbol {code}
It seems related with the FLINK-33980



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: Re: [VOTE] FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources

2024-01-18 Thread Jark Wu
+1 (binding)

Best,
Jark

On Tue, 16 Jan 2024 at 18:01, Xuyang  wrote:

> +1 (non-binding)
>
>
> --
>
> Best!
> Xuyang
>
>
>
>
>
> 在 2024-01-16 17:52:38,"Leonard Xu"  写道:
> >+1 (binding)
> >
> >Best,
> >Leonard
> >
> >> 2024年1月16日 下午5:40,Hang Ruan  写道:
> >>
> >> +1 (non-binding)
> >>
> >> Best,
> >> Hang
> >>
> >> Jiabao Sun  于2024年1月9日周二 19:39写道:
> >>
> >>> Hi Devs,
> >>>
> >>> I'd like to start a vote on FLIP-377: Support fine-grained
> configuration
> >>> to control filter push down for Table/SQL Sources[1]
> >>> which has been discussed in this thread[2].
> >>>
> >>> The vote will be open for at least 72 hours unless there is an
> objection
> >>> or not enough votes.
> >>>
> >>> [1]
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768
> >>> [2] https://lists.apache.org/thread/nvxx8sp9jm009yywm075hoffr632tm7j
> >>>
> >>> Best,
> >>> Jiabao
>


Re:Re: [VOTE] FLIP-389: Annotate SingleThreadFetcherManager as PublicEvolving

2024-01-18 Thread Xuyang
+1 (non-binding)


--

Best!
Xuyang





在 2024-01-19 10:16:23,"Qingsheng Ren"  写道:
>+1 (binding)
>
>Thanks for the work, Hongshun!
>
>Best,
>Qingsheng
>
>On Tue, Jan 16, 2024 at 11:21 AM Leonard Xu  wrote:
>
>> Thanks Hongshun for driving this !
>>
>> +1(binding)
>>
>> Best,
>> Leonard
>>
>> > 2024年1月3日 下午8:04,Hongshun Wang  写道:
>> >
>> > Dear Flink Developers,
>> >
>> > Thank you for providing feedback on FLIP-389: Annotate
>> > SingleThreadFetcherManager as PublicEvolving[1] on the discussion
>> > thread[2]. The goal of the FLIP is as follows:
>> >
>> >   - To expose the SplitFetcherManager / SingleThreadFetcheManager as
>> >   Public, allowing connector developers to easily create their own
>> threading
>> >   models in the SourceReaderBase by implementing addSplits(),
>> removeSplits(),
>> >   maybeShutdownFinishedFetchers() and other functions.
>> >   - To hide the element queue from the connector developers and simplify
>> >   the SourceReaderBase to consist of only SplitFetcherManager and
>> >   RecordEmitter as major components.
>> >
>> >
>> > Any additional questions regarding this FLIP? Looking forward to hearing
>> > from you.
>> >
>> >
>> > Thanks,
>> > Hongshun Wang
>> >
>> >
>> > [1]
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465498
>> >
>> > [2] https://lists.apache.org/thread/b8f509878ptwl3kmmgg95tv8sb1j5987
>>
>>


Re:Re: [VOTE] FLIP-415: Introduce a new join operator to support minibatch

2024-01-18 Thread Xuyang
+1 (non-binding)--

Best!
Xuyang





在 2024-01-19 13:28:52,"Lincoln Lee"  写道:
>+1 (binding)
>
>Best,
>Lincoln Lee
>
>
>Benchao Li  于2024年1月19日周五 13:15写道:
>
>> +1 (binding)
>>
>> shuai xu  于2024年1月19日周五 12:58写道:
>>
>>> Dear Flink Developers,
>>>
>>> Thank you for providing feedback on FLIP-415: Introduce a new join
>>> operator to support minibatch[1]. I'd like to start a vote on this FLIP.
>>> Here is the discussion thread[2].
>>>
>>> After the discussion, this FLIP will not introduce any new Option. The
>>> minibatch join will default to compacting the changelog. As for the option
>>> to control compaction within the minibatch that was mentioned in the
>>> discussion, it could be discussed in a future FLIP.
>>>
>>> The vote will be open for at least 72 hours unless there is an objection
>>> or
>>> insufficient votes.
>>>
>>> Best,
>>> Xu Shuai
>>>
>>> [1]
>>> FLIP-415: Introduce a new join operator to support minibatch - Apache
>>> Flink - Apache Software Foundation
>>> 
>>> cwiki.apache.org
>>> 
>>> [image: favicon.ico]
>>> 
>>> 
>>> [2]
>>> lists.apache.org
>>> 
>>> [image: favicon.ico]
>>> 
>>> 
>>>
>>>
>>
>> --
>>
>> Best,
>> Benchao Li
>>


Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Yun Tang
+1 (binding)

Best
Yun Tang

From: Rui Fan <1996fan...@gmail.com>
Sent: Friday, January 19, 2024 13:10
To: dev@flink.apache.org 
Subject: Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

+1 (binding)

Best,
Rui

On Fri, Jan 19, 2024 at 12:15 PM Yanfei Lei  wrote:

> +1 (binding)
>
> Best,
> Yanfei
>
> Hangxiang Yu  于2024年1月19日周五 12:13写道:
> >
> > +1 (binding)
> >
> > On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan 
> wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to start a vote on the FLIP-416: Deprecate and remove the
> > > RestoreMode#LEGACY [1]. The discussion thread is here [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/ookkEQ
> > > [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
>


Re: [VOTE] FLIP-415: Introduce a new join operator to support minibatch

2024-01-18 Thread Lincoln Lee
+1 (binding)

Best,
Lincoln Lee


Benchao Li  于2024年1月19日周五 13:15写道:

> +1 (binding)
>
> shuai xu  于2024年1月19日周五 12:58写道:
>
>> Dear Flink Developers,
>>
>> Thank you for providing feedback on FLIP-415: Introduce a new join
>> operator to support minibatch[1]. I'd like to start a vote on this FLIP.
>> Here is the discussion thread[2].
>>
>> After the discussion, this FLIP will not introduce any new Option. The
>> minibatch join will default to compacting the changelog. As for the option
>> to control compaction within the minibatch that was mentioned in the
>> discussion, it could be discussed in a future FLIP.
>>
>> The vote will be open for at least 72 hours unless there is an objection
>> or
>> insufficient votes.
>>
>> Best,
>> Xu Shuai
>>
>> [1]
>> FLIP-415: Introduce a new join operator to support minibatch - Apache
>> Flink - Apache Software Foundation
>> 
>> cwiki.apache.org
>> 
>> [image: favicon.ico]
>> 
>> 
>> [2]
>> lists.apache.org
>> 
>> [image: favicon.ico]
>> 
>> 
>>
>>
>
> --
>
> Best,
> Benchao Li
>


Re: [VOTE] FLIP-415: Introduce a new join operator to support minibatch

2024-01-18 Thread Benchao Li
+1 (binding)

shuai xu  于2024年1月19日周五 12:58写道:

> Dear Flink Developers,
>
> Thank you for providing feedback on FLIP-415: Introduce a new join
> operator to support minibatch[1]. I'd like to start a vote on this FLIP.
> Here is the discussion thread[2].
>
> After the discussion, this FLIP will not introduce any new Option. The
> minibatch join will default to compacting the changelog. As for the option
> to control compaction within the minibatch that was mentioned in the
> discussion, it could be discussed in a future FLIP.
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> Best,
> Xu Shuai
>
> [1]
> FLIP-415: Introduce a new join operator to support minibatch - Apache
> Flink - Apache Software Foundation
> 
> cwiki.apache.org
> 
> [image: favicon.ico]
> 
> 
> [2]
> lists.apache.org
> 
> [image: favicon.ico]
> 
> 
>
>

-- 

Best,
Benchao Li


Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Rui Fan
+1 (binding)

Best,
Rui

On Fri, Jan 19, 2024 at 12:15 PM Yanfei Lei  wrote:

> +1 (binding)
>
> Best,
> Yanfei
>
> Hangxiang Yu  于2024年1月19日周五 12:13写道:
> >
> > +1 (binding)
> >
> > On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan 
> wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to start a vote on the FLIP-416: Deprecate and remove the
> > > RestoreMode#LEGACY [1]. The discussion thread is here [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/ookkEQ
> > > [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
> > >
> > >
> > > Best,
> > > Zakelly
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
>


[VOTE] FLIP-415: Introduce a new join operator to support minibatch

2024-01-18 Thread shuai xu
Dear Flink Developers,

Thank you for providing feedback on FLIP-415: Introduce a new join operator to 
support minibatch[1]. I'd like to start a vote on this FLIP. Here is the 
discussion thread[2]. 

After the discussion, this FLIP will not introduce any new Option. The 
minibatch join will default to compacting the changelog. As for the option to 
control compaction within the minibatch that was mentioned in the discussion, 
it could be discussed in a future FLIP.

The vote will be open for at least 72 hours unless there is an objection or
insufficient votes.

Best,
Xu Shuai

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
[2]
https://lists.apache.org/thread/pd9gzslq20dtzzfphxqvwhc43hrzo2y1


Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Yanfei Lei
+1 (binding)

Best,
Yanfei

Hangxiang Yu  于2024年1月19日周五 12:13写道:
>
> +1 (binding)
>
> On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan  wrote:
>
> > Hi everyone,
> >
> > I'd like to start a vote on the FLIP-416: Deprecate and remove the
> > RestoreMode#LEGACY [1]. The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection or
> > insufficient votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/ookkEQ
> > [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
> >
> >
> > Best,
> > Zakelly
> >
>
>
> --
> Best,
> Hangxiang.


Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Hangxiang Yu
+1 (binding)

On Fri, Jan 19, 2024 at 12:10 PM Zakelly Lan  wrote:

> Hi everyone,
>
> I'd like to start a vote on the FLIP-416: Deprecate and remove the
> RestoreMode#LEGACY [1]. The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/ookkEQ
> [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
>
>
> Best,
> Zakelly
>


-- 
Best,
Hangxiang.


Re: [VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Yuan Mei
+1 binding

Best
Yuan

On Fri, Jan 19, 2024 at 12:09 PM Zakelly Lan  wrote:

> Hi everyone,
>
> I'd like to start a vote on the FLIP-416: Deprecate and remove the
> RestoreMode#LEGACY [1]. The discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/ookkEQ
> [2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o
>
>
> Best,
> Zakelly
>


[VOTE] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-18 Thread Zakelly Lan
Hi everyone,

I'd like to start a vote on the FLIP-416: Deprecate and remove the
RestoreMode#LEGACY [1]. The discussion thread is here [2].

The vote will be open for at least 72 hours unless there is an objection or
insufficient votes.

[1] https://cwiki.apache.org/confluence/x/ookkEQ
[2] https://lists.apache.org/thread/ho77fx13lw4ds52t0fs1xqz2vtn50n2o


Best,
Zakelly


退订

2024-01-18 Thread 李乐
退订

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

2024-01-18 Thread Hongshun Wang
Hi Devs,
  Thanks for all your advice, it helps a lot. I have already revised
the document[1] and started a vote[2].

Thanks,

Hongshun


[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465498

[2] https://lists.apache.org/thread/t1zff21z440pvv48jyhm8pgtqsyplchn

On Fri, Jan 12, 2024 at 1:00 AM Becket Qin  wrote:

> Hi Qingsheng,
>
> Thanks for the comment. I think the initial idea is to hide the queue
> completely from the users, i.e. make FutureCompletingBlockingQueue class
> internal. If it is OK to expose the class to the users, then just returning
> the queue sounds reasonable to me.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Jan 10, 2024 at 10:39 PM Hongshun Wang 
> wrote:
>
>> Hi Qingsheng,
>>
>>
>> I agree with you that it would be clearer to have a new interface that
>> extracts the SplitFetcher creation and management logic from the current
>> SplitFetcherManager. However, extensive modifications to the interface may
>> influence a lot and cause compatibility issues. Perhaps we can consider
>> doing it later, rather than in this FLIP.
>>
>>
>> Adding a new internal method, SplitFetcherManager#getQueue(), to
>> SourceReaderBase seems to be a better option than exposing methods like
>> poll and notifyAvailable on SplitFetcherManager.
>>
>>
>> I have taken this valuable suggestion and updated the FLIP accordingly.
>>
>>
>> Thanks,
>>
>> Hongshun
>>
>> On Thu, Jan 11, 2024 at 2:09 PM Qingsheng Ren  wrote:
>>
>>> Hi Hongshun and Becket,
>>>
>>> Sorry for being late in the discussion! I went through the entire FLIP
>>> but I still have some concerns about the new SplitFetcherManager.
>>>
>>> First of all I agree that we should hide the elementQueue from connector
>>> developers. This could simplify the interface exposed to developers so that
>>> they can focus on the interaction with external systems.
>>>
>>> However in the current FLIP, SplitFetcherManager exposes 4 more methods,
>>> poll / getAvailabilityFuture / notifyAvailable / noAvailableElement, which
>>> are tightly coupled with the implementation of the elementQueue. The naming
>>> of these methods look weird to me, like what does it mean to "poll from a
>>> SplitFetcherManager" / "notify a SplitFetcherManager available"? To clarify
>>> these methods we have to explain to developers that "well we hide a queue
>>> inside SplitFetcherMamager and the poll method is actually polling from the
>>> queue". I'm afraid these methods will implicitly expose the concept and the
>>> implementation of the queue to developers.
>>>
>>> I think a cleaner solution would be having a new interface that extracts
>>> SplitFetcher creating and managing logic from the current
>>> SplitFetcherManager, but having too many concepts might make the entire
>>> Source API even harder to understand. To make a compromise, I'm considering
>>> only exposing constructors of SplitFetcherManager as public APIs, and
>>> adding a new internal method SplitFetcherManager#getQueue() for
>>> SourceReaderBase (well it's a bit hacky I admit but I think exposing
>>> methods like poll and notifyAvailable on SplitFetcherManager is even
>>> worth). WDTY?
>>>
>>> Thanks,
>>> Qingsheng
>>>
>>> On Thu, Dec 21, 2023 at 8:36 AM Becket Qin  wrote:
>>>
 Hi Hongshun,

 I think the proposal in the FLIP is basically fine. A few minor
 comments:

 1. In FLIPs, we define all the user-sensible changes as public
 interfaces.
 The public interface section should list all of them. So, the code
 blocks
 currently in the proposed changes section should be put into the public
 interface section instead.

 2. It would be good to put all the changes of one class together. For
 example, for SplitFetcherManager, we can say:
 - Change SplitFetcherManager from Internal to PublicEvolving.
 - Deprecate the old constructor exposing the
 FutureCompletingBlockingQueue, and add new constructors as replacements
 which creates the FutureCompletingBlockingQueue instance internally.
 - Add a few new methods to expose the functionality of the internal
 FutureCompletingBlockingQueue via the SplitFetcherManager.
And then follows the code block containing all the changes above.
 Ideally, the changes should come with something like "// <-- New",
 so
 that it is. easier to be found.

 3. In the proposed changes section, it would be good to add some more
 detailed explanation of the idea behind the public interface changes. So
 even people new to Flink can understand better how exactly the interface
 changes will help fulfill the motivation. For example, regarding the
 constructor signature change, we can say the following. We can mention a
 few things in this section:
 - By exposing the SplitFetcherManager / SingleThreadFetcheManager,
 by
 implementing addSplits() and removeSplits(), connector developers can
 easily create their own thre

Re: [VOTE] FLIP-389: Annotate SingleThreadFetcherManager as PublicEvolving

2024-01-18 Thread Qingsheng Ren
+1 (binding)

Thanks for the work, Hongshun!

Best,
Qingsheng

On Tue, Jan 16, 2024 at 11:21 AM Leonard Xu  wrote:

> Thanks Hongshun for driving this !
>
> +1(binding)
>
> Best,
> Leonard
>
> > 2024年1月3日 下午8:04,Hongshun Wang  写道:
> >
> > Dear Flink Developers,
> >
> > Thank you for providing feedback on FLIP-389: Annotate
> > SingleThreadFetcherManager as PublicEvolving[1] on the discussion
> > thread[2]. The goal of the FLIP is as follows:
> >
> >   - To expose the SplitFetcherManager / SingleThreadFetcheManager as
> >   Public, allowing connector developers to easily create their own
> threading
> >   models in the SourceReaderBase by implementing addSplits(),
> removeSplits(),
> >   maybeShutdownFinishedFetchers() and other functions.
> >   - To hide the element queue from the connector developers and simplify
> >   the SourceReaderBase to consist of only SplitFetcherManager and
> >   RecordEmitter as major components.
> >
> >
> > Any additional questions regarding this FLIP? Looking forward to hearing
> > from you.
> >
> >
> > Thanks,
> > Hongshun Wang
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465498
> >
> > [2] https://lists.apache.org/thread/b8f509878ptwl3kmmgg95tv8sb1j5987
>
>


[jira] [Created] (FLINK-34163) Migrate SimplifyJoinConditionRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34163:
---

 Summary: Migrate SimplifyJoinConditionRule
 Key: FLINK-34163
 URL: https://issues.apache.org/jira/browse/FLINK-34163
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34162) Migrate LogicalUnnestRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34162:
---

 Summary: Migrate LogicalUnnestRule
 Key: FLINK-34162
 URL: https://issues.apache.org/jira/browse/FLINK-34162
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34161) Migrate RewriteMinusAllRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34161:
---

 Summary: Migrate RewriteMinusAllRule
 Key: FLINK-34161
 URL: https://issues.apache.org/jira/browse/FLINK-34161
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34160) Migrate FlinkCalcMergeRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34160:
---

 Summary: Migrate FlinkCalcMergeRule
 Key: FLINK-34160
 URL: https://issues.apache.org/jira/browse/FLINK-34160
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34159) Migrate ConstantRankNumberColumnRemoveRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34159:
---

 Summary: Migrate ConstantRankNumberColumnRemoveRule
 Key: FLINK-34159
 URL: https://issues.apache.org/jira/browse/FLINK-34159
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34158) Migrate WindowAggregateReduceFunctionsRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34158:
---

 Summary: Migrate WindowAggregateReduceFunctionsRule
 Key: FLINK-34158
 URL: https://issues.apache.org/jira/browse/FLINK-34158
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34157) Migrate FlinkLimit0RemoveRule

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34157:
---

 Summary: Migrate FlinkLimit0RemoveRule
 Key: FLINK-34157
 URL: https://issues.apache.org/jira/browse/FLINK-34157
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-18 Thread Mason Chen
Hi Lijie,

That's also a possibility but I would prefer to keep it consistent with how
the existing metric APIs are used.

For example, in the current metric APIs [1], there is no way to figure out
the vertexid and subtaskindex without getting the job graph from
`/jobs//plan` and correspondingly there are no APIs to return a map
of all metrics for every vertex and to return a map of all metrics for
every subtask. Essentially, the plan API is already required to use the
finer-grained metric apis.

In addition, keeping the design similar lends itself better for the
implementation. The metric handler utilities [2] assume a
ComponentMetricStore is returned rather than a Map.

I've updated the FLIP doc (
https://cwiki.apache.org/confluence/display/FLINK/FLIP-417%3A+Expose+JobManagerOperatorMetrics+via+REST+API)
with our discussion so far.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#rest-api-integration
[2]
https://github.com/apache/flink/blob/a41229b24d82e8c561350c42d8a98dfb865c3f69/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/metrics/AbstractMetricsHandler.java#L109

Best,
Mason

On Wed, Jan 17, 2024 at 8:28 AM Lijie Wang  wrote:

> Hi Mason,
>
> Thanks for driving the discussion. +1 for the proposal.
>
> How about we return all operator metrics in a vertex? (the response is a
> map of operatorId/operatorName -> operator-metrics). Correspondingly, the
> url may be changed to /jobs//vertices//operator-metrics
>
> In this way, users can skip the step of obtaining the operator id.
>
> Best,
> Lijie
>
> Hang Ruan  于2024年1月17日周三 10:31写道:
>
> > Hi, Mason.
> >
> > The field `operatorName` in JobManagerOperatorQueryScopeInfo refers to
> the
> > fields in OperatorQueryScopeInfo and chooses the operatorName instead of
> > OperatorID.
> > It is fine by my side to change from opertorName to operatorID in this
> > FLIP.
> >
> > Best,
> > Hang
> >
> > Mason Chen  于2024年1月17日周三 09:39写道:
> >
> > > Hi Xuyang and Hang,
> > >
> > > Thanks for your support and feedback! See my responses below:
> > >
> > > 1. IIRC, in a sense, operator ID and vertex ID are the same thing. The
> > > > operator ID can
> > > > be converted from the vertex ID[1]. Therefore, it is somewhat strange
> > to
> > > > have both vertex
> > > > ID and operator ID in a single URL.
> > > >
> > > I think Hang explained it perfectly. Essentially, a vertix may contain
> > one
> > > or more operators so the operator ID is required to distinguish this
> > case.
> > >
> > > 2. If I misunderstood the semantics of operator IDs here, then what is
> > the
> > > > relationship
> > > > between vertex ID and operator ID, and do we need a url like
> > > > `/jobs//vertices//operators/`
> > > > to list all operator ids under this vertices?
> > > >
> > > Good question, we definitely need expose operator IDs through the REST
> > API
> > > to make this usable. I'm looking at how users would currently discover
> > the
> > > vertex id to query. From the supported REST APIs [1], you can currently
> > > obtain it from
> > >
> > > 1. `/jobs/`
> > > 2. `/jobs//plan`
> > >
> > > From the response of both these APIs, they include the vertex ids (the
> > > vertices AND nodes fields), but not the operator ids. We would need to
> > add
> > > the logic to the plan generation [2]. The response is a little
> confusing
> > > because there is a field in the vertices called "operator name". I
> > propose
> > > to add a new field called "operators" to the vertex response object,
> > which
> > > would be a list of objects with the structure
> > >
> > > Operator
> > > {
> > >   "id": "THE-FLINK-GENERATED-ID"
> > > }.
> > >
> > > The JobManagerOperatorQueryScopeInfo has three fields: jobID, vertexID
> > and
> > > > operatorName. So we should use the operator name in the API.
> > > > If you think we should use the operator id, there need be more
> changes
> > > > about it.
> > > >
> > > I think we should use operator id since it uniquely identifies an
> > > operator--on the contrary, the operator name does not (it may be empty
> or
> > > repeated between operators by the user). I actually had a question on
> > that
> > > since you implemented the metric group. What's the reason we use
> operator
> > > name currently? Could it also use operator id so we can match against
> the
> > > id?
> > >
> > > [1]
> > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/
> > > [2]
> > >
> > >
> >
> https://github.com/apache/flink/blob/416cb7aaa02c176e01485ff11ab4269f76b5e9e2/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/jsonplan/JsonPlanGenerator.java#L53
> > >
> > > Best,
> > > Mason
> > >
> > >
> > > On Thu, Jan 11, 2024 at 10:54 PM Hang Ruan 
> > wrote:
> > >
> > > > Hi, Mason.
> > > >
> > > > Thanks for driving this FLIP.
> > > >
> > > > The JobManagerOperatorQueryScopeInfo has three fields: jobID,
> vertexID
> > > and
> > > > operatorName. So we should use the operator name in the API.
> > > > If you think we should 

[jira] [Created] (FLINK-34156) Move Flink Calcite rules from Scala to Java

2024-01-18 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-34156:
---

 Summary: Move Flink Calcite rules from Scala to Java
 Key: FLINK-34156
 URL: https://issues.apache.org/jira/browse/FLINK-34156
 Project: Flink
  Issue Type: Technical Debt
  Components: Table SQL / Planner
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin
 Fix For: 2.0.0


This is an umbrella task for migration of Calcite rules from Scala to Java 
mentioned at https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

The reason is that since 1.28.0 ( CALCITE-4787 - Move core to use Immutables 
instead of ImmutableBeans ) Calcite started to use Immutables 
(https://immutables.github.io/) and since 1.29.0 removed ImmutableBeans ( 
CALCITE-4839 - Remove remnants of ImmutableBeans post 1.28 release ). All rule 
configuration related api which is not Immutables based is marked as 
deprecated. Since Immutables implies code generation while java compilation it 
is seems impossible to use for rules in Scala code.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[DISCUSS] Mark org.apache.flink.util.Preconditions @Public

2024-01-18 Thread Alexander Fedulov
Hi everyone,

pretty much every connector utilizes checks from the
org.apache.flink.util.Preconditions class, which is currently marked
as @Internal. This causes CONNECTOR_CLASSES_ONLY_DEPEND_ON_PUBLIC_API
architecture tests to fail and results in the violation exceptions being
filled with the lines like this:
https://github.com/apache/flink-connector-kafka/blob/8d531898d7df775535084e0413442c86a4c497ed/flink-connector-kafka/archunit-violations/984f05c0-ec82-405e-9bcc-d202dbe7202e#L22-L27

I'd like to propose to mark the Preconditions class as @Public of
@PublicEvolving. There are also some other similar candidates, like
ClosureCleaner, but the Preconditions is by far the most widespread.
Alternatively we could add an exception to the ArchRule definition.

What do you think?

Best,
Alex


[jira] [Created] (FLINK-34155) Recurring SqlExecutionException

2024-01-18 Thread Jeyhun Karimov (Jira)
Jeyhun Karimov created FLINK-34155:
--

 Summary: Recurring SqlExecutionException
 Key: FLINK-34155
 URL: https://issues.apache.org/jira/browse/FLINK-34155
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.8.0
Reporter: Jeyhun Karimov
 Attachments: disk-full.log

When analyzing very big maven log file in our CI system, I found out that there 
is a recurring {{{}SqlException (subset of the log file is attached){}}}:

 
{{org.apache.flink.table.gateway.service.utils.SqlExecutionException: Only 
'INSERT/CREATE TABLE AS' statement is allowed in Statement Set or use 'END' 
statement to submit Statement Set.}}
 
 

which leads to:

 
{{06:31:41,155 [flink-rest-server-netty-worker-thread-22] ERROR 
org.apache.flink.table.gateway.rest.handler.statement.FetchResultsHandler [] - 
Unhandled exception.}}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34154) Bump org.apache.zookeeper:zookeeper from 3.5.9 to 3.7.2 for Kafka connector

2024-01-18 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34154:
--

 Summary: Bump org.apache.zookeeper:zookeeper from 3.5.9 to 3.7.2 
for Kafka connector
 Key: FLINK-34154
 URL: https://issues.apache.org/jira/browse/FLINK-34154
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Kafka
Reporter: Martijn Visser
Assignee: Martijn Visser


The Flink Kafka connector still uses Zookeeper but only for tests. Version 
3.5.9 has a CVE, we should bump this to avoid getting falsely flagged for this 
vulnerability 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34153) Set ALWAYS ChainingStrategy in TemporalSort

2024-01-18 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-34153:


 Summary: Set ALWAYS ChainingStrategy in TemporalSort
 Key: FLINK-34153
 URL: https://issues.apache.org/jira/browse/FLINK-34153
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.19.0


Similarly to FLINK-27992 we should ALWAYS chaining strategy in TemporalSort 
operator



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


FLink runtime meeting some error,Hoping to give some solutions

2024-01-18 Thread 15868861416


| |
博星
|
|


|
The specific error is as shown above.



[jira] [Created] (FLINK-34152) Tune memory of autoscaled jobs

2024-01-18 Thread Maximilian Michels (Jira)
Maximilian Michels created FLINK-34152:
--

 Summary: Tune memory of autoscaled jobs
 Key: FLINK-34152
 URL: https://issues.apache.org/jira/browse/FLINK-34152
 Project: Flink
  Issue Type: New Feature
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.8.0


The current autoscaling algorithm adjusts the parallelism of the job task 
vertices according to the processing needs. By adjusting the parallelism, we 
systematically scale the amount of CPU for a task. At the same time, we also 
indirectly change the amount of memory tasks have at their dispense. However, 
there are some problems with this.
 # Memory is overprovisioned: On scale up we may add more memory then we 
actually need. Even on scale down, the memory / cpu ratio can still be off and 
too much memory is used.
 # Memory is underprovisioned: For stateful jobs, we risk running into 
OutOfMemoryErrors on scale down. Even before running out of memory, too little 
memory can have a negative impact on the effectiveness of the scaling.

We lack the capability to tune memory proportionally to the processing needs. 
In the same way that we measure CPU usage and size the tasks accordingly, we 
need to evaluate memory usage and adjust the heap memory size.

A tuning algorithm could look like this:
h2. 1. Establish a memory baseline

We observe the average heap memory usage at task managers.
h2. 2. Calculate memory usage per record

The memory requirements per record can be estimated by calculating this ratio:
{noformat}
memory_per_rec = sum(heap_usage) / sum(records_processed)
{noformat}
This ratio is surprisingly constant based off empirical data.
h2. 3. Scale memory proportionally to the per-record memory needs
{noformat}
memory_per_tm = expected_records_per_sec * memory_per_rec / num_task_managers 
{noformat}
A minimum memory limit needs to be added to avoid scaling down memory too much. 
The max memory per TM should be equal to the initially defined user-specified 
limit from the ResourceSpec. 
{noformat}
memory_per_tm = max(min_limit, memory_per_tm)
memory_per_tm = min(max_limit, memory_per_tm) {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34151) Integrate Karpenter resource limits into cluster capacity check

2024-01-18 Thread Maximilian Michels (Jira)
Maximilian Michels created FLINK-34151:
--

 Summary: Integrate Karpenter resource limits into cluster capacity 
check
 Key: FLINK-34151
 URL: https://issues.apache.org/jira/browse/FLINK-34151
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.8.0


FLINK-33771 added cluster capacity checking for Flink Autoscaling decisions. 
The checks respect the scaling limits of the Kubernetes Cluster Autoscaler. 

We should also support Karpenter-based resource checks, as Karpenter is the 
preferred method of expanding the cluster size in some environments.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34150) file sink e2e tests with local setup are not executed if s3 credentials are not provided

2024-01-18 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34150:
-

 Summary: file sink e2e tests with local setup are not executed if 
s3 credentials are not provided
 Key: FLINK-34150
 URL: https://issues.apache.org/jira/browse/FLINK-34150
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.18.1, 1.17.2, 1.19.0
Reporter: Matthias Pohl


While looking into test parity between Azure Pipelines and GitHub Actions, I 
noticed that the only OpenSSL-based test is 
{{flink-end-to-end-tests/test-scripts/test_file_sink.sh}} which comes with a 
local and a s3 setting.

S3 requires S3 credential and the bucket information to be available through 
environment variables. That's handled in 
[flink-end-to-end-tests/test-scripts/common_s3.sh#L25|https://github.com/apache/flink/blob/8a9a08bf408aae8a33438a38614199efeb8f1c63/flink-end-to-end-tests/test-scripts/common_s3.sh#L25].
 The problem is that this shell script is also source'd when running the test 
with local setup (see 
[flink-end-to-end-tests/test-scripts/test_file_sink.sh#L27|https://github.com/apache/flink/blob/a6bea224ed012e5594ee755526f54ae7f3b0d22f/flink-end-to-end-tests/test-scripts/test_file_sink.sh#L27]).

This means that also the local test is only running in the main repository 
which is not necessary.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[RESULT][VOTE] Accept Flink CDC into Apache Flink

2024-01-18 Thread Leonard Xu
Hi all,

Thank you for your enthusiastic participation!

I am happy to announce that Flink CDC project contribution initiated by 
Ververica has been officially accepted[1] by the Apache Flink community.

Result:
 - 32 PMC members voted, that is 2/3 of the total PMC members(48).
 - 31 non-PMC members voted.
 - 32 x +1 (binding)
 - 31 x +1 (non-binding)
 - 0 x -1
 - 0 x +/-0

Binding +1 votes:
- Qingsheng Ren
- Yuan Mei
- Jingsong Li
- Xintong Song
- Jark Wu
- Yu Li
- Guowei Ma
- Robert Metzger
- Leonard Xu
- Márton Balassi
- Danny Cranmer
- Dian Fu
- Xingbo Huang
- Martijn Visser
- Maximilian Michels
- Piotr Nowojski
- Dawid Wysakowicz
- Matthias Pohl
- Yang Wang
- Zhu Zhu
- godfrey he
- jincheng sun
- Hequn Cheng
- Kurt Yang
- Zhijiang
- Becket Qin
- Dong Lin
- Yun Gao  
- Gyula Fóra  
- Ufuk Celebi 
- Timo Walther  
- Fabian Hueske  

Non-binding +1 votes:
- Zili Chen
- Feng Wang
- Benchao Li
- Jing Ge
- gongzhongqiang
- Sergey Nuyanzin
- Lincoln Lee
- Samrat Deb
- Hang Ruan
- Rui Fan
- Yangze Guo
- Yuxin Tan
- Feng Jin
- xiangyu feng
- Jiabao Sun
- Lijie Wang
- Jane Chan
- Peter Huang
- Venkata Sanath Muppalla
- Sharath
- ConradJam
- liu ron
- Xuyang
- Xuannan Su
- Hangxiang Yu
- Austin Bennett
- Rodrigo Meneses
- Yun Tang
- Congxian Qiu
- Gunnar Morling
- Swapnal Varma

Next, I will create an issue on jira to complete the project donation. Some 
subtasks I can think of so far are as follows:
Migrate repo from ververica to apche
Migrate doc website from ververica to flink 
Add NOTICE files for Flink CDC project
Update copyright and license file
Remove unwanted bundle dependencies
Setup CI for Flink CDC project
Setup release infrastructure for Flink CDC project

Best,
Leonard

Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-18 Thread Leonard Xu
Thank you everyone for voting! 

The voting period has passed and we have enough binding votes!

Thus I will close the voting and summarize the result in another email.


Best,
Leonard


> 2024年1月17日 下午5:54,Swapnal Varma  写道:
> 
> +1 (non-binding)
> 
> Best,
> Swapnal
> 
> On Wed, Jan 17, 2024 at 1:57 AM Gunnar Morling
>  wrote:
> 
>> +1 (non-binding)
>> 
>> Best,
>> 
>> --Gunnar
>> 
>> 
>> Am Mo., 15. Jan. 2024 um 18:24 Uhr schrieb Fabian Hueske <
>> fhue...@apache.org
>>> :
>> 
>>> +1 (binding)
>>> 
>>> Cheers, Fabian
>>> 
>>> 
>>> Timo Walther  schrieb am Mo., 15. Jan. 2024, 16:00:
>>> 
 +1 (binding)
 
 Cheers,
 Timo
 
 
 On 09.01.24 10:58, xiangyu feng wrote:
> +1 (non-binding)
> 
> Regards,
> Xiangyu Feng
> 
> Danny Cranmer  于2024年1月9日周二 17:50写道:
> 
>> +1 (binding)
>> 
>> Thanks,
>> Danny
>> 
>> On Tue, Jan 9, 2024 at 9:31 AM Feng Jin 
>>> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Best,
>>> Feng Jin
>>> 
>>> On Tue, Jan 9, 2024 at 5:29 PM Yuxin Tan 
 wrote:
>>> 
 +1 (non-binding)
 
 Best,
 Yuxin
 
 
 Márton Balassi  于2024年1月9日周二 17:25写道:
 
> +1 (binding)
> 
> On Tue, Jan 9, 2024 at 10:15 AM Leonard Xu 
>> wrote:
> 
>> +1(binding)
>> 
>> Best,
>> Leonard
>> 
>>> 2024年1月9日 下午5:08,Yangze Guo  写道:
>>> 
>>> +1 (non-binding)
>>> 
>>> Best,
>>> Yangze Guo
>>> 
>>> On Tue, Jan 9, 2024 at 5:06 PM Robert Metzger <
>> rmetz...@apache.org
 
>> wrote:
 
 +1 (binding)
 
 
 On Tue, Jan 9, 2024 at 9:54 AM Guowei Ma <
>> guowei@gmail.com>
> wrote:
 
> +1 (binding)
> Best,
> Guowei
> 
> 
> On Tue, Jan 9, 2024 at 4:49 PM Rui Fan <1996fan...@gmail.com
>>> 
 wrote:
> 
>> +1 (non-binding)
>> 
>> Best,
>> Rui
>> 
>> On Tue, Jan 9, 2024 at 4:41 PM Hang Ruan <
>>> ruanhang1...@gmail.com>
>> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Best,
>>> Hang
>>> 
>>> gongzhongqiang  于2024年1月9日周二
>>> 16:25写道:
>>> 
 +1 non-binding
 
 Best,
 Zhongqiang
 
 Leonard Xu  于2024年1月9日周二 15:05写道:
 
> Hello all,
> 
> This is the official vote whether to accept the Flink CDC
>>> code
 contribution
> to Apache Flink.
> 
> The current Flink CDC code, documentation, and website
>> can
>> be
> found here:
> code: https://github.com/ververica/flink-cdc-connectors
>> <
> https://github.com/ververica/flink-cdc-connectors>
> docs: https://ververica.github.io/flink-cdc-connectors/
>> <
> https://ververica.github.io/flink-cdc-connectors/>
> 
> This vote should capture whether the Apache Flink
>> community
>>> is
>>> interested
> in accepting, maintaining, and evolving Flink CDC.
> 
> Regarding my original proposal[1] in the dev mailing
>> list,
>> I
> firmly
 believe
> that this initiative aligns perfectly with Flink. For the
>>> Flink
 community,
> it represents an opportunity to bolster Flink's
>> competitive
 edge
> in
> streaming
> data integration, fostering the robust growth and
>> prosperity
>>> of
> the
 Apache
> Flink
> ecosystem. For the Flink CDC project, becoming a
>> sub-project
>>> of
>> Apache
> Flink
> means becoming an integral part of a neutral open-source
> community,
> capable of
> attracting a more diverse pool of contributors.
> 
> All Flink CDC maintainers are dedicated to continuously
> contributing
>> to
> achieve
> seamless integration with Flink. Additionally, PMC
>> members
>>> like
> Jark,
> Qingsheng,
> and I are willing to infacilitate the expansion of
>>> contributors
> and
> committers to
> effectively maintain this new sub-project.
> 
> This is a "Adoption

[jira] [Created] (FLINK-34149) Flink Kafka connector can't compile against 1.19-SNAPSHOT

2024-01-18 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34149:
--

 Summary: Flink Kafka connector can't compile against 1.19-SNAPSHOT
 Key: FLINK-34149
 URL: https://issues.apache.org/jira/browse/FLINK-34149
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Runtime / Checkpointing
Affects Versions: 1.19.0
Reporter: Martijn Visser


The Flink Kafka connector for {{main}} fails for 1.19-SNAPSHOT, see 
https://github.com/apache/flink-connector-kafka/actions/runs/7569481434/job/20612876543#step:14:134

{code:java}
Error:  COMPILATION ERROR : 
[INFO] -
Error:  
/home/runner/work/flink-connector-kafka/flink-connector-kafka/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/StoppableKafkaEnumContextProxy.java:[65,8]
 
org.apache.flink.connector.kafka.dynamic.source.enumerator.StoppableKafkaEnumContextProxy
 is not abstract and does not override abstract method 
setIsProcessingBacklog(boolean) in 
org.apache.flink.api.connector.source.SplitEnumeratorContext
{code}

This interface seems to be added as part of 
https://issues.apache.org/jira/browse/FLINK-32514 / 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
The FLIP indicates that the changes should be backward compatible, but that 
appears to have not been the case



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Re: Flink autoscaler scaling report

2024-01-18 Thread Yu Chen
Hi Yang,

You can run `StandaloneAutoscalerEntrypoint`, and the scale report will print 
in log (info level) by LoggingEventHandler[2].

[1] 
flink-kubernetes-operator/flink-autoscaler-standalone/src/main/java/org/apache/flink/autoscaler/standalone/StandaloneAutoscalerEntrypoint.java
 at main · apache/flink-kubernetes-operator (github.com) 

[2] 
https://github.com/apache/flink-kubernetes-operator/blob/48df9d35ed55ae8bb513d9153e9f6f668da9e1c3/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/event/LoggingEventHandler.java#L43C18-L43C18

Best,
Yu Chen


> 2024年1月18日 18:20,Yang LI  写道:
> 
> Hello dear flink community,
> 
> I noticed that there's a scaling report feature (specifically, the strings
> defined in AutoscalerEventHandler) in the Flink operator autoscaler.
> However, I'm unable to find this information in the Flink operator logs.
> Could anyone guide me on how to access or visualize this scaling report?
> 
> Thanks,
> Yang



[jira] [Created] (FLINK-34148) Potential regression (Jan. 13): stringWrite with Java8

2024-01-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34148:
---

 Summary: Potential regression (Jan. 13): stringWrite with Java8
 Key: FLINK-34148
 URL: https://issues.apache.org/jira/browse/FLINK-34148
 Project: Flink
  Issue Type: Improvement
  Components: API / Type Serialization System
Reporter: Zakelly Lan


Significant drop of performance in stringWrite with Java8 from commit 
[881062f352|https://github.com/apache/flink/commit/881062f352f8bf8c21ab7cbea95e111fd82fdf20]
 to 
[5d9d8748b6|https://github.com/apache/flink/commit/5d9d8748b64ff1a75964a5cd2857ab5061312b51]
 . It only involves strings not so long (128 or 4).

stringWrite.128.ascii(Java8) baseline=1089.107756 current_value=754.52452
stringWrite.128.chinese(Java8) baseline=504.244575 current_value=295.358989
stringWrite.128.russian(Java8) baseline=655.582639 current_value=421.030188
stringWrite.4.chinese(Java8) baseline=9598.791964 current_value=6627.929927
stringWrite.4.russian(Java8) baseline=11070.666415 current_value=8289.95767



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


Flink autoscaler scaling report

2024-01-18 Thread Yang LI
Hello dear flink community,

I noticed that there's a scaling report feature (specifically, the strings
defined in AutoscalerEventHandler) in the Flink operator autoscaler.
However, I'm unable to find this information in the Flink operator logs.
Could anyone guide me on how to access or visualize this scaling report?

Thanks,
Yang


[jira] [Created] (FLINK-34147) TimestampData to/from LocalDateTime is ambiguous

2024-01-18 Thread Rui Li (Jira)
Rui Li created FLINK-34147:
--

 Summary: TimestampData to/from LocalDateTime is ambiguous
 Key: FLINK-34147
 URL: https://issues.apache.org/jira/browse/FLINK-34147
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Reporter: Rui Li


It seems TimestampData is essentially an {{Instant}}. Therefore an implicit 
time zone is used in the {{fromLocalDateTime}} and {{toLocalDateTime}} methods. 
However neither the method name nor the API doc indicates which time zone is 
used. So from caller's perspective, the results of these two methods are 
ambiguous.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34146) JDBC lookup joins fail with RDB column names containing colons

2024-01-18 Thread david radley (Jira)
david radley created FLINK-34146:


 Summary: JDBC lookup joins fail with RDB column names containing 
colons
 Key: FLINK-34146
 URL: https://issues.apache.org/jira/browse/FLINK-34146
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC, Table SQL / JDBC
Affects Versions: 1.18.1
Reporter: david radley


[https://github.com/apache/flink-connector-jdbc/pull/79] adds filter support 
for lookup joins. This was implemented using FieldNamedPreparedStatements in 
line with the way that the join key was implemented.   The 
[FieldNamedPreparedStatementImpl 
logic|https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L221]
 explicitly tests for the colon key and can incorrectly pickup column names.  
So JDBC lookup joins fail with RDB column names containing colons when used in 
filters and lookup keys.

It looks like we have used the approach from 
[https://stackoverflow.com/questions/2309970/named-parameters-in-jdbc]. It says 
{{Please note that the above simple example does not handle using named 
parameter twice. Nor does it handle using the : sign inside quotes.}} It looks 
like we could play with some Regex Patterns to see if we can get one that works 
well for us.

 

A junit that shows the issue can be added to
FieldNamedPreparedStatementImplTest
 
...
private final String[] fieldNames2 =
new String[] \{"id?:", "name:?", "email", "ts", "field1", "field_2", 
"__field_3__"};
private final String[] keyFields2 = new String[] \{"id?:", "__field_3__"};
...
@Test
void testSelectStatementWithWeirdCharacters() {
String selectStmt = dialect.getSelectFromStatement(tableName, fieldNames2, 
keyFields2);
assertThat(selectStmt)
.isEqualTo(
"SELECT `id?:`, `name:?`, `email`, `ts`, `field1`, `field_2`, `__field_3__` 
FROM `tbl` "
+ "WHERE `id?:` = :id?: AND `__field_3__` = :__field_3__");
NamedStatementMatcher.parsedSql(
"SELECT `id?:`, `name:?`, `email`, `ts`, `field1`, `field_2`, `__field_3__` 
FROM `tbl` "
+ "WHERE `id?:` = ? AND `__field_3__` = ?")
.parameter("id", singletonList(1))
.parameter("__field_3__", singletonList(2))
.matches(selectStmt);
}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34145) File source connector support dynamic source parallelism inference in batch jobs

2024-01-18 Thread xingbe (Jira)
xingbe created FLINK-34145:
--

 Summary: File source connector support dynamic source parallelism 
inference in batch jobs
 Key: FLINK-34145
 URL: https://issues.apache.org/jira/browse/FLINK-34145
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Affects Versions: 1.19.0
Reporter: xingbe


[FLIP-379|https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs]
 has introduced support for dynamic source parallelism inference in batch jobs, 
and we plan to give priority to enabling this feature for the file source 
connector.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34144) Update the documentation and configuration description about dynamic source parallelism inference

2024-01-18 Thread xingbe (Jira)
xingbe created FLINK-34144:
--

 Summary: Update the documentation and configuration description 
about dynamic source parallelism inference
 Key: FLINK-34144
 URL: https://issues.apache.org/jira/browse/FLINK-34144
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.19.0
Reporter: xingbe


[FLIP-379|https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs#FLIP379:Dynamicsourceparallelisminferenceforbatchjobs-IntroduceDynamicParallelismInferenceinterfaceforSource]
 introduces the new feature of dynamic source parallelism inference, and we 
plan to update the documentation and configuration items accordingly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34143) Modify the effective strategy of `execution.batch.adaptive.auto-parallelism.default-source-parallelism`

2024-01-18 Thread xingbe (Jira)
xingbe created FLINK-34143:
--

 Summary: Modify the effective strategy of 
`execution.batch.adaptive.auto-parallelism.default-source-parallelism`
 Key: FLINK-34143
 URL: https://issues.apache.org/jira/browse/FLINK-34143
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.19.0
Reporter: xingbe


Currently, if users do not set the 
`{{{}execution.batch.adaptive.auto-parallelism.default-source-parallelism`{}}} 
configuration option, the AdaptiveBatchScheduler defaults to a parallelism of 1 
for source vertices. In 
[FLIP-379|https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs#FLIP379:Dynamicsourceparallelisminferenceforbatchjobs-IntroduceDynamicParallelismInferenceinterfaceforSource],
 the value of 
`{{{}execution.batch.adaptive.auto-parallelism.default-source-parallelism`{}}} 
will act as the upper bound for inferring dynamic source parallelism, and 
continuing with the current policy is no longer appropriate.

We plan to change the effectiveness strategy of 
`{{{}execution.batch.adaptive.auto-parallelism.default-source-parallelism`{}}}; 
when the user does not set this config option, we will use the value of 
`{{{}execution.batch.adaptive.auto-parallelism.max-parallelism`{}}} as the 
upper bound for source parallelism inference. If 
{{`execution.batch.adaptive.auto-parallelism.max-parallelism`}} is also not 
configured, the value of `{{{}parallelism.default`{}}} will be used as a 
fallback.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (FLINK-34142) TaskManager WorkingDirectory is not removed during shutdown

2024-01-18 Thread Prabhu Joseph (Jira)
Prabhu Joseph created FLINK-34142:
-

 Summary: TaskManager WorkingDirectory is not removed during 
shutdown 
 Key: FLINK-34142
 URL: https://issues.apache.org/jira/browse/FLINK-34142
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.17.1, 1.16.0
Reporter: Prabhu Joseph


TaskManager WorkingDirectory is not removed during shutdown. 

*Repro*

 
{code:java}
1. Execute a Flink batch job within a Flink on YARN Session

flink-yarn-session -d

flink run -d /usr/lib/flink/examples/batch/WordCount.jar --input 
s3://prabhuflinks3/INPUT --output s3://prabhuflinks3/OUT

{code}
The batch job completes successfully, but the taskmanager working directory is 
not being removed.
{code:java}
[root@ip-1-2-3-4 container_1705470896818_0017_01_02]# ls -R -lrt 
/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02
/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02:
total 0
drwxr-xr-x 2 yarn yarn  6 Jan 18 08:34 tmp
drwxr-xr-x 4 yarn yarn 66 Jan 18 08:34 blobStorage
drwxr-xr-x 2 yarn yarn  6 Jan 18 08:34 slotAllocationSnapshots
drwxr-xr-x 2 yarn yarn  6 Jan 18 08:34 localState

/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02/tmp:
total 0

/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02/blobStorage:
total 0
drwxr-xr-x 2 yarn yarn 94 Jan 18 08:34 job_d11f7085314ef1fb04c4e12fe292185a
drwxr-xr-x 2 yarn yarn  6 Jan 18 08:34 incoming

/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02/blobStorage/job_d11f7085314ef1fb04c4e12fe292185a:
total 12
-rw-r--r-- 1 yarn yarn 10323 Jan 18 08:34 
blob_p-cdd441a64b3ea6eed0058df02c6c10fd208c94a8-86d84864273dad1e8084d8ef0f5aad52

/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02/blobStorage/incoming:
total 0

/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02/slotAllocationSnapshots:
total 0

/mnt2/yarn/usercache/hadoop/appcache/application_1705470896818_0017/tm_container_1705470896818_0017_01_02/localState:
total 0


{code}
*Analysis*

1. The TaskManagerRunner removes the working directory only when its 'close' 
method is called, which never happens.
{code:java}
public void close() throws Exception {
try {
closeAsync().get();
} catch (ExecutionException e) {

ExceptionUtils.rethrowException(ExceptionUtils.stripExecutionException(e));
}
}

public CompletableFuture closeAsync() {
return closeAsync(Result.SUCCESS);
}
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)