[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836099#comment-17836099 ] Shilun Fan commented on FLINK-35040: [~fanrui] During the compilation process, we found that commons-compress requires a higher version of commons-io, otherwise there will be a class not found exception. > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836005#comment-17836005 ] Shilun Fan edited comment on FLINK-35040 at 4/11/24 5:21 AM: - [~fanrui] [~Weijie Guo] How is serializerHeavyString tested? I haven't seen any testing scripts here. Could you provide some hints? I found the project at [https://github.com/apache/flink-benchmarks]. I will take a look at the relevant tests as soon as possible. was (Author: slfan1989): [~fanrui] [~Weijie Guo] How is serializerHeavyString tested? I haven't seen any testing scripts here. Could you provide some hints? > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836004#comment-17836004 ] Shilun Fan commented on FLINK-35040: [~fanrui] I received your message, and I apologize for any inconvenience caused. The reason for upgrading commons-compress to version 1.26.0 is indeed due to the known CVE issues in commons-compress 1.24.0. We can refer to the following link: [https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.24.0] Direct vulnerabilities: CVE-2024-26308 CVE-2024-25710 Addressing the CVE issue is indeed necessary to prevent vulnerabilities in our system. I suggest we consider upgrading to version 1.26.1 to address this problem. Reverting to 1.24.0 might only serve as a temporary solution. > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836005#comment-17836005 ] Shilun Fan commented on FLINK-35040: [~fanrui] [~Weijie Guo] How is serializerHeavyString tested? I haven't seen any testing scripts here. Could you provide some hints? > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34955) Upgrade commons-compress to 1.26.0
[ https://issues.apache.org/jira/browse/FLINK-34955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834682#comment-17834682 ] Shilun Fan edited comment on FLINK-34955 at 4/7/24 2:17 PM: [~gongzhongqiang] Of course, if upgrading is possible, it would be a positive step forward. I think we should give it a try. I see that you have created the relevant JIRA ticket, so you can go ahead and attempt it. Hopefully, it will be successful. (However, my preference would be to stick with version 1.26 for now, and consider upgrading to 1.26.1 in the future. If other components of Flink need upgrading, I think it would be best to upgrade them to version 1.26 as well. Removing dependencies, in my opinion, is not a good option.) was (Author: slfan1989): [~gongzhongqiang] Of course, if upgrading is possible, it would be a positive step forward. I think we should give it a try. I see that you have created the relevant JIRA ticket, so you can go ahead and attempt it. Hopefully, it will be successful. > Upgrade commons-compress to 1.26.0 > -- > > Key: FLINK-34955 > URL: https://issues.apache.org/jira/browse/FLINK-34955 > Project: Flink > Issue Type: Improvement >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > commons-compress 1.24.0 has CVE issues, try to upgrade to 1.26.0, we can > refer to the maven link > https://mvnrepository.com/artifact/org.apache.commons/commons-compress -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34955) Upgrade commons-compress to 1.26.0
[ https://issues.apache.org/jira/browse/FLINK-34955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834682#comment-17834682 ] Shilun Fan commented on FLINK-34955: [~gongzhongqiang] Of course, if upgrading is possible, it would be a positive step forward. I think we should give it a try. I see that you have created the relevant JIRA ticket, so you can go ahead and attempt it. Hopefully, it will be successful. > Upgrade commons-compress to 1.26.0 > -- > > Key: FLINK-34955 > URL: https://issues.apache.org/jira/browse/FLINK-34955 > Project: Flink > Issue Type: Improvement >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > commons-compress 1.24.0 has CVE issues, try to upgrade to 1.26.0, we can > refer to the maven link > https://mvnrepository.com/artifact/org.apache.commons/commons-compress -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34955) Upgrade commons-compress to 1.26.0
[ https://issues.apache.org/jira/browse/FLINK-34955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834681#comment-17834681 ] Shilun Fan commented on FLINK-34955: [~gongzhongqiang] From my personal perspective, I believe upgrading to version 1.26.0 should be sufficient as this version has already fixed the CVE issue. As for upgrading to 1.26.1, I think we can consider it after some time. Removing commons-codec might prove to be challenging because Flink has dependencies on Hadoop and HBase (both of which directly depend on commons-codec). If we remove commons-codec, it may result in the Hadoop and HBase modules being unable to compile successfully. > Upgrade commons-compress to 1.26.0 > -- > > Key: FLINK-34955 > URL: https://issues.apache.org/jira/browse/FLINK-34955 > Project: Flink > Issue Type: Improvement >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > commons-compress 1.24.0 has CVE issues, try to upgrade to 1.26.0, we can > refer to the maven link > https://mvnrepository.com/artifact/org.apache.commons/commons-compress -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35003) Update zookeeper to 3.8.4 to address CVE-2024-23944
Shilun Fan created FLINK-35003: -- Summary: Update zookeeper to 3.8.4 to address CVE-2024-23944 Key: FLINK-35003 URL: https://issues.apache.org/jira/browse/FLINK-35003 Project: Flink Issue Type: Improvement Components: BuildSystem / Shaded Reporter: Shilun Fan Update zookeeper to 3.8.4 to address CVE-2024-23944 https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.8.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34955) Upgrade commons-compress to 1.26.0
Shilun Fan created FLINK-34955: -- Summary: Upgrade commons-compress to 1.26.0 Key: FLINK-34955 URL: https://issues.apache.org/jira/browse/FLINK-34955 Project: Flink Issue Type: Improvement Reporter: Shilun Fan commons-compress 1.24.0 has CVE issues, try to upgrade to 1.26.0, we can refer to the maven link https://mvnrepository.com/artifact/org.apache.commons/commons-compress -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31761) Fix Some Typo And Improve lambda expressions.
[ https://issues.apache.org/jira/browse/FLINK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711623#comment-17711623 ] Shilun Fan commented on FLINK-31761: [~martijnvisser] Thanks for your message! > Fix Some Typo And Improve lambda expressions. > - > > Key: FLINK-31761 > URL: https://issues.apache.org/jira/browse/FLINK-31761 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.17.0 >Reporter: Shilun Fan >Priority: Major > Labels: pull-request-available > > When reading the code, I found 2 typo errors and improved the lambada > expression. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31761) Fix Some Typo And Improve lambda expressions.
Shilun Fan created FLINK-31761: -- Summary: Fix Some Typo And Improve lambda expressions. Key: FLINK-31761 URL: https://issues.apache.org/jira/browse/FLINK-31761 Project: Flink Issue Type: Improvement Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.17.0 Reporter: Shilun Fan When reading the code, I found 2 typo errors and improved the lambada expression. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31510) Use getMemorySize instead of getMemory
[ https://issues.apache.org/jira/browse/FLINK-31510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702427#comment-17702427 ] Shilun Fan commented on FLINK-31510: [~lemonjing] we use int32 for memory now, if a cluster has 10k nodes, each node has 210G memory, we will get a negative total cluster memory. > Use getMemorySize instead of getMemory > -- > > Key: FLINK-31510 > URL: https://issues.apache.org/jira/browse/FLINK-31510 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.18.0 >Reporter: Shilun Fan >Priority: Major > Labels: pull-request-available > > In YARN-4844, use getMemorySize instead of getMemory, because using int to > represent memory may exceed the bounds in some cases and produce negative > numbers. > This change was merged in HADOOP-2.8.0, we should use getMemorySize instead > of getMemory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31510) Use getMemorySize instead of getMemory
Shilun Fan created FLINK-31510: -- Summary: Use getMemorySize instead of getMemory Key: FLINK-31510 URL: https://issues.apache.org/jira/browse/FLINK-31510 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.18.0 Reporter: Shilun Fan In YARN-4844, use getMemorySize instead of getMemory, because using int to represent memory may exceed the bounds in some cases and produce negative numbers. This change was merged in HADOOP-2.8.0, we should use getMemorySize instead of getMemory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31480) Fix Typo in YarnClusterDescriptor
[ https://issues.apache.org/jira/browse/FLINK-31480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated FLINK-31480: --- Issue Type: Improvement (was: Bug) > Fix Typo in YarnClusterDescriptor > - > > Key: FLINK-31480 > URL: https://issues.apache.org/jira/browse/FLINK-31480 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.17.1 >Reporter: Shilun Fan >Priority: Major > Labels: pull-request-available > > There is a typo in the comment for YarnClusterDescriptor, this jira will fix > it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31480) Fix Typo in YarnClusterDescriptor
Shilun Fan created FLINK-31480: -- Summary: Fix Typo in YarnClusterDescriptor Key: FLINK-31480 URL: https://issues.apache.org/jira/browse/FLINK-31480 Project: Flink Issue Type: Bug Components: Deployment / YARN Affects Versions: 1.17.1 Reporter: Shilun Fan There is a typo in the comment for YarnClusterDescriptor, this jira will fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31402) Remove ConverterUtils Deprecated Method
[ https://issues.apache.org/jira/browse/FLINK-31402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated FLINK-31402: --- Description: Many methods in ConverterUtils have been Deprecated, we can replace the Deprecated methods and remove this class(ConverterUtils). (was: Many methods in ConverterUtils are obsolete, we can replace the obsolete methods and remove this class(ConverterUtils).) > Remove ConverterUtils Deprecated Method > --- > > Key: FLINK-31402 > URL: https://issues.apache.org/jira/browse/FLINK-31402 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.17.0 >Reporter: Shilun Fan >Priority: Major > Labels: pull-request-available > > Many methods in ConverterUtils have been Deprecated, we can replace the > Deprecated methods and remove this class(ConverterUtils). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31402) Remove ConverterUtils Deprecated Method
Shilun Fan created FLINK-31402: -- Summary: Remove ConverterUtils Deprecated Method Key: FLINK-31402 URL: https://issues.apache.org/jira/browse/FLINK-31402 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.17.0 Reporter: Shilun Fan Many methods in ConverterUtils are obsolete, we can replace the obsolete methods and remove this class(ConverterUtils). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31230) Improve YarnClusterDescriptor memory unit display
Shilun Fan created FLINK-31230: -- Summary: Improve YarnClusterDescriptor memory unit display Key: FLINK-31230 URL: https://issues.apache.org/jira/browse/FLINK-31230 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.17.1 Reporter: Shilun Fan When we use the yarn-session.sh -q command, the basic information of the NM will be printed, and the memory unit is MB. If we have 128GB of memory, it will display 131072 MB. At the same time, if we have multiple NMs, finally totalMemeory doesn't show units. This jira will use hadoop's StringUtil to format unit output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31222) Remove usage of deprecated ConverterUtils.toApplicationId
Shilun Fan created FLINK-31222: -- Summary: Remove usage of deprecated ConverterUtils.toApplicationId Key: FLINK-31222 URL: https://issues.apache.org/jira/browse/FLINK-31222 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.17.1 Reporter: Shilun Fan When reading the code, I found that we use ConverterUtils.toApplicationId to convert applicationId, this method is deprecated, we should use ApplicationId.fromString -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31221) Fix Typo in YarnConfigOptions
Shilun Fan created FLINK-31221:
--
Summary: Fix Typo in YarnConfigOptions
Key: FLINK-31221
URL: https://issues.apache.org/jira/browse/FLINK-31221
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Affects Versions: 1.17.1
Reporter: Shilun Fan
I found a typo in YarnConfigOptions, I will fix it.
willl -> will
{code:java}
public static final ConfigOption LOCALIZED_KEYTAB_PATH =
.
"Local (on NodeManager) path where kerberos keytab file
will be"
+ " localized to. If "
+ SHIP_LOCAL_KEYTAB.key()
+ " set to "
+ "true, Flink willl ship the keytab file as a
YARN local "
+ "resource. In this case, the path is relative
to the local "
+ "resource directory. If set to false, Flink"
+ " will try to directly locate the keytab from
the path itself.");
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30243) Remove duplicate commas
[ https://issues.apache.org/jira/browse/FLINK-30243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641132#comment-17641132 ] Shilun Fan commented on FLINK-30243: [~martijnvisser] Thanks for the documentation! > Remove duplicate commas > --- > > Key: FLINK-30243 > URL: https://issues.apache.org/jira/browse/FLINK-30243 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.16.0 >Reporter: Shilun Fan >Priority: Minor > Labels: pull-request-available > > When reading the code, I found that there are duplicate commas in the > comments, I submit pr for improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30243) Remove duplicate commas
Shilun Fan created FLINK-30243: -- Summary: Remove duplicate commas Key: FLINK-30243 URL: https://issues.apache.org/jira/browse/FLINK-30243 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Affects Versions: 1.16.0 Reporter: Shilun Fan When reading the code, I found that there are duplicate commas in the comments, I submit pr for improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
