[jira] [Created] (FLINK-33157) Support AWS SDK V2 for Flink's S3 FileSystem

[Maomao Min (Jira)]

Maomao Min created FLINK-33157:
--

 Summary: Support AWS SDK V2 for Flink's S3 FileSystem
 Key: FLINK-33157
 URL: https://issues.apache.org/jira/browse/FLINK-33157
 Project: Flink
  Issue Type: Improvement
  Components: FileSystems
Reporter: Maomao Min


Currently, Flink's S3 FileSystem only supports AWS SDK V1. It's recommended by 
AWS to use AWS SDK V2 for improvements and longer maintenance support. To 
complete this work, the following modules need to be updated:
 * flink-s3-fs-base
 * flink-s3-fs-hadoop
 * flink-s3-fs-presto

The Hadoop has already added support for AWS SDK V2, but this is not the case 
for Presto. To unblock component flink-s3-fs-presto for AWS SDK V2 support, we 
might have the following options:
 # Flink can have a copy of Presto's S3 FileSystem with AWS SDK V2 support 
added.
 # Add AWS SDK V2 support in Presto's S3 FileSystem in Presto and use new 
Presto version with this feature in Flink.
 # Use Trino's S3FileSystem, which has already supported AWS SDK V2, instead of 
Presto's S3FileSystem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Xuannan Su]

Hi all,

We would like to revive the discussion and provide a quick update on
the recent work of the FLIP. We have implemented a POC[1], run cases
in the flink-benchmarks[2] against the POC, and verified that many of
the operators in the benchmark will enable object-reuse without code
changes, while the global object-reuse is disabled.

Please let me know if you have any further comments on the FLIP. If
there are no more comments, we will open the voting in 3 days.

Best regards,
Xuannan

[1] https://github.com/apache/flink/pull/22897
[2] https://github.com/apache/flink-benchmarks


On Fri, Jul 7, 2023 at 9:18 AM Dong Lin  wrote:
>
> Hi Jing,
>
> Thank you for the suggestion. Yes, we can extend it to support null if in
> the future we find any use-case for this flexibility.
>
> Best,
> Dong
>
> On Thu, Jul 6, 2023 at 7:55 PM Jing Ge  wrote:
>
> > Hi Dong,
> >
> > one scenario I could imagine is that users could enable global object
> > reuse features but force deep copy for some user defined specific functions
> > because of any limitations. But that is only my gut feeling. And agree, we
> > could keep the solution simple for now as FLIP described and upgrade to 3VL
> > once there are such real requirements that are rising.
> >
> > Best regards,
> > Jing
> >
> > On Thu, Jul 6, 2023 at 12:30 PM Dong Lin  wrote:
> >
> >> Hi Jing,
> >>
> >> Thank you for the detailed explanation. Please see my reply inline.
> >>
> >> On Thu, Jul 6, 2023 at 3:17 AM Jing Ge  wrote:
> >>
> >>> Hi Xuannan, Hi Dong,
> >>>
> >>> Thanks for your clarification.
> >>>
> >>> @Xuannan
> >>>
> >>> A Jira ticket has been created for the doc update:
> >>> https://issues.apache.org/jira/browse/FLINK-32546
> >>>
> >>> @Dong
> >>>
> >>> I don't have a concrete example. I just thought about it from a
> >>> conceptual or pattern's perspective. Since we have 1. coarse-grained 
> >>> global
> >>> switch(CGS as abbreviation), i.e. the pipeline.object-reuse and 2.
> >>> fine-grained local switch(FGS as abbreviation), i.e. the
> >>> objectReuseCompliant variable for specific operators/functions, there will
> >>> be the following patterns with appropriate combinations:
> >>>
> >>> pattern 1: coarse-grained switch only. Local object reuse will be
> >>> controlled by the coarse-grained switch:
> >>> 1.1 cgs == true -> local object reused enabled
> >>> 1.2 cgs == true  -> local object reused enabled
> >>> 1.3 cgs == false -> local object reused disabled, i.e. deep copy enabled
> >>> 1.4 cgs == false -> local object reused disabled, i.e. deep copy enabled
> >>>
> >>> afaiu, this is the starting point. I wrote 4 on purpose to make the
> >>> regression check easier. We can consider it as the combinations with
> >>> cgs(true/false) and fgs(true/false) while fgs is ignored.
> >>>
> >>> Now we introduce fine-grained switch. There will be two patterns:
> >>>
> >>> pattern 2: fine-grained switch over coarse-grained switch.
> >>> Coarse-grained switch will be ignored when the local fine-grained switch
> >>> has different value:
> >>> 2.1 cgs == true and fgs == true -> local object reused enabled
> >>> 2.2 cgs == true and fgs == false -> local object reused disabled, i.e.
> >>> deep copy enabled
> >>> 2.3 cgs == false and fgs == true -> local object reused enabled
> >>> 2.4 cgs == false and fgs == false -> local object reused disabled, i.e.
> >>> deep copy enabled
> >>>
> >>> cgs is actually ignored.
> >>>
> >>> Current FLIP is using a slightly different pattern:
> >>>
> >>> pattern 3: fine-grained switch over coarse-grained switch only when
> >>> coarse-grained switch is off, i..e cgs OR fgs:
> >>> 3.1 cgs == true and fgs == true -> local object reused enabled
> >>> 3.2 cgs == true and fgs == false -> local object reused enabled
> >>> 3.3 cgs == false and fgs == true -> local object reused enabled
> >>> 3.4 cgs == false and fgs == false -> local object reused disabled, i.e.
> >>> deep copy enabled
> >>>
> >>> All of those patterns are rational and each has different focus. It
> >>> depends on the real requirement to choose one of them.
> >>>
> >>> As we can see, if fgs is using 2VL, there is a regression between
> >>> pattern 1 and pattern 2. You are absolutely right in this case. That's why
> >>> I suggested 3VL, i.e. fgs will have triple values: true, false,
> >>> unknown(e.g. null)
> >>>
> >>> pattern 4: 3VL fgs with the null as init value (again, there are just
> >>> two combination, I made it 4 on purpose):
> >>> 4.1 cgs == true and fgs == null -> local object reused enabled
> >>> 4.2 cgs == true and fgs == null -> local object reused enabled
> >>> 4.3 cgs == false and fgs == null -> local object reused disabled, i.e.
> >>> deep copy enabled
> >>> 4.4 cgs == false and fgs == null -> local object reused disabled, i.e.
> >>> deep copy enabled
> >>>
> >>> Since the default value of fgs is null, pattern 4 is backward compatible
> >>> with pattern 1, which means no regression.
> >>>
> >>> Now we will set value to fgs and follow the pattern 2:
> >>> 4.5 cgs == tr

[jira] [Created] (FLINK-33158) Cryptic exception when there is a StreamExecSort in JsonPlan

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-33158:


 Summary: Cryptic exception when there is a StreamExecSort in 
JsonPlan
 Key: FLINK-33158
 URL: https://issues.apache.org/jira/browse/FLINK-33158
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.17.1, 1.16.2
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.18.0


{code}
CREATE TABLE MyTable (
   a bigint,
   b int not null,
   c varchar,
   d timestamp(3)
with (
   'connector' = 'values',
   'bounded' = 'false'
)

insert into MySink SELECT a, a from MyTable order by b
{code}

fails with:

{code}
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException:
 For input string: "null" (through reference chain: 
org.apache.flink.table.planner.plan.nodes.exec.serde.JsonPlanGraph["nodes"]->java.util.ArrayList[2])
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33159) Use the variables for java and Maven version checks in the pom file

[david radley (Jira)]

david radley created FLINK-33159:


 Summary: Use the variables for java and Maven version checks in 
the pom file 
 Key: FLINK-33159
 URL: https://issues.apache.org/jira/browse/FLINK-33159
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: david radley
 Fix For: 1.7.3, 1.8.4


Use the variables for java and Maven version checks in the pom file.

In the pom file I notice line

__
                   _1.8.0_
 __

This should use the variable like this:

  \{_}{_}${target.java.version}{_}{_}

I also see 2 instanced of _1.8_

The second instance should refer to the variable

also the Maven versions should use variables.

 

I will provide a pr for this.

 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Close orphaned/stale PRs

[Venkatakrishnan Sowrirajan]

Hi Martijn,

Agree with your point that closing a PR without any review feedback even
after 'X' days is discouraging to a new contributor. I understand that this
is a capacity problem. Capacity problem cannot be solved by this proposal
and it is beyond the scope of this proposal.

Regarding your earlier question,
> What's the added value of
closing these PRs

   - Having lots of inactive PRs lingering around shows the project is less
   active. I am not saying this is the only way to determine how active a
   project is, but this is one of the key factors.
   - A large number of PRs open can be discouraging for (new) contributors
   but on the other hand I agree closing an inactive PR without any reviews
   can also drive contributors away.

Having said all of that, I agree closing PRs that don't have any reviews to
start with should be avoided from the final proposal.

> I'm +1 for (automatically) closing up PRs after X days which:
a) Don't have a CI that has passed
b) Don't follow the code contribution guide (like commit naming conventions)
c) Have changes requested but aren't being followed-up by the contributor

In general, I'm largely +1 on your above proposal except for the
implementation feasibility.

Also, I have picked a few other popular projects that have implemented the
Github's actions stale rule to see if we can borrow some ideas. Below
projects are listed in the order of the most invasive (for lack of a better
word) to the least invasive actions taken wrt PR without any updates for a
long period of time.

1. Trino

TL;DR - No updates in the PR for the last 21 days, tag other maintainers
for review. If there are no updates for 21 days after that, close the PR
with this message - "*Closing this pull request, as it has been stale for
six weeks. Feel free to re-open at any time.*"
Trino's stale PR Github action rule (stale.yaml)


2. Apache Spark

TL;DR - No updates in the PR in the last 100 days, closing the PR with this
message - "*We're closing this PR because it hasn't been updated in a
while. This isn't a judgement on the merit of the PR in any way. It's
just a way of keeping the PR queue manageable. If you'd like to revive this
PR, please reopen it and ask a committer to remove the Stale tag!*"
Spark's discussion in their mailing list
 on
closing stale PRs. Spark's stale PR github action rule (stale.yaml
).

3. Python

TL;DR - No updates in the PR for the last 30 days, then tag the PR as
stale. Note: Python project *doesn't* close the stale PRs.

Python discussion

in the mailing list to close stale PRs. Python's stale PR github action
rule (stale.yaml
)

Few others Apache Beam

(closes
inactive PRs after 60+ days), Apache Airflow

(closes
inactive PRs after 50 days)

Let me know what you think. Looking forward to hearing from others in the
community and their experiences.

[1] Github Action - Close Stale Issues -
https://github.com/marketplace/actions/close-stale-issues

Regards
Venkata krishnan


On Thu, Sep 21, 2023 at 6:03 AM Martijn Visser 
wrote:

> Hi all,
>
> I really believe that the problem of the number of open PRs is just
> that there aren't enough reviewers/resources available to review them.
>
> > Stale PRs can clutter the repository, and closing them helps keep it
> organized and ensures that only relevant and up-to-date PRs are present.
>
> Sure, but what's the indicator that the PR is stale? The fact that
> there has been no reviewer yet to review it, doesn't mean that the PR
> is stale. For me, a stale PR is a PR that has been reviewed, changes
> have been requested and the contributor isn't participating in the
> discussion anymore. But that's a different story compared to closing
> PRs where there has been no review done at all.
>
> > It mainly helps the project maintainers/reviewers to focus on only the
> actively updated trimmed list of PRs that are ready for review.
>
> I disagree that closing PRs helps with this. If you want to help
> maintainers/reviewers, we should have a situation where it's obvious
> that a PR is really ready (meaning, CI has passed, PR contents/commit
> message etc are following the code contribution guidelines).
>
> > It helps Flink users who are waiting on a PR that enhances an existing
> feature or fixes an issue a clear indication on whether the PR will be
> continually worked on and eventually get a closure or not and therefore
> will be closed.
>
> Having other PRs being closed doesn't incre

[jira] [Created] (FLINK-33160) Print the remote address when an exception occurs in the PartitionRequestQueue

[dizhou cao (Jira)]

dizhou cao created FLINK-33160:
--

 Summary: Print the remote address when an exception occurs in the 
PartitionRequestQueue
 Key: FLINK-33160
 URL: https://issues.apache.org/jira/browse/FLINK-33160
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Reporter: dizhou cao


Add the information of the remote address in the exception handling of the 
PartitionRequestQueue, so that network issues can be located through the 
network quintuple.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33161) [benchmark] Java17 profile for benchmarks

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-33161:
---

 Summary: [benchmark] Java17 profile for benchmarks
 Key: FLINK-33161
 URL: https://issues.apache.org/jira/browse/FLINK-33161
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Reporter: Zakelly Lan
Assignee: Zakelly Lan
 Fix For: 1.18.1


Flink has supported java 17, however, the benchmark does not support running 
with java 17 out of the box. The runner complains:
Caused by: java.lang.RuntimeException: 
java.lang.reflect.InaccessibleObjectException: Unable to make field private 
final java.lang.Object[] java.util.Arrays$ArrayList.a accessible: module 
java.base does not "opens java.util" to unnamed module @14899482
01:39:16at 
com.twitter.chill.java.ArraysAsListSerializer.(ArraysAsListSerializer.java:69)
01:39:16at 
org.apache.flink.api.java.typeutils.runtime.kryo.FlinkChillPackageRegistrar.registerSerializers(FlinkChillPackageRegistrar.java:67)
01:39:16at 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.getKryoInstance(KryoSerializer.java:513)
01:39:16at 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:522)
01:39:16at 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:348)
01:39:16at 
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:165)
01:39:16at 
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:43)
01:39:16at 
org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:54)
01:39:16at 
org.apache.flink.runtime.io.network.api.writer.RecordWriter.serializeRecord(RecordWriter.java:151)
01:39:16at 
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:107)
01:39:16at 
org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:55)
01:39:16at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:134)
01:39:16at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collectAndCheckIfChained(RecordWriterOutput.java:114)
01:39:16at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:95)
01:39:16at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:48)
01:39:16at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:59)
01:39:16at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:31)
01:39:16at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:425)
01:39:16at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:520)
01:39:16at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:110)
01:39:16at 
org.apache.flink.benchmark.ContinuousFileReaderOperatorBenchmark$MockSourceFunction.run(ContinuousFileReaderOperatorBenchmark.java:101)
01:39:16at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:114)
01:39:16at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:71)
01:39:16at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:338)
01:39:16  Caused by: java.lang.reflect.InaccessibleObjectException: Unable to 
make field private final java.lang.Object[] java.util.Arrays$ArrayList.a 
accessible: module java.base does not "opens java.util" to unnamed module 
@14899482
01:39:16at 
java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
01:39:16at 
java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
01:39:16at 
java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
01:39:16at 
java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
01:39:16at 
com.twitter.chill.java.ArraysAsListSerializer.(ArraysAsListSerializer.java:67)
01:39:16... 23 more
 

To resolve this, we should add a target profile for java 17 in pom.xml of 
benchmark project and provide some cli arg like "--add-opens 
java.base/java.util=ALL-UNNAMED" for the execution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33162) seperate the executor in DefaultDispatcherResourceManagerComponentFactory for MetricFetcher and webMonitorEndpoint

[xiaogang zhou (Jira)]

xiaogang zhou created FLINK-33162:
-

 Summary: seperate the executor in 
DefaultDispatcherResourceManagerComponentFactory for MetricFetcher and 
webMonitorEndpoint
 Key: FLINK-33162
 URL: https://issues.apache.org/jira/browse/FLINK-33162
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / REST
Affects Versions: 1.16.0
Reporter: xiaogang zhou
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33163) Support Java 21 (LTS)

[Jira]

Maciej Bryński created FLINK-33163:
--

 Summary: Support Java 21 (LTS)
 Key: FLINK-33163
 URL: https://issues.apache.org/jira/browse/FLINK-33163
 Project: Flink
  Issue Type: Bug
Reporter: Maciej Bryński


Based on https://issues.apache.org/jira/browse/FLINK-15736 we should have 
similar ticket for Java 21 LTS.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)