Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]

[via GitHub]

1996fanrui commented on code in PR #23635:
URL: https://github.com/apache/flink/pull/23635#discussion_r1381212518


##
flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java:
##
@@ -708,6 +708,20 @@ public class TaskManagerOptions {
 "Time we wait for the timers in milliseconds to 
finish all pending timer threads"
 + " when the stream task is cancelled.");
 
+public static final ConfigOption 
TASK_MANAGER_LOAD_BALANCE_MODE =

Review Comment:
   Thanks @RocMarshal  for the analysis!
   
   How about creating a JIRA to add `taskmanager.load-balance.mode` and 
deprecate `cluster.evenly-spread-out-slots` first?
   - It don't introduce any feature, just using the new option and deprecate 
old option.
   - It should be the first JIRA, and `TaskManagerLoadBalanceMode` just have 2 
enums: `None` and `Slots`.
   - In the second JIRA(maybe this PR), you can start your feature: adding the 
`Tasks`, and supports it.
   
   I think this process is clearer and easier than the process you proposed, 
and it don't effect users side.
   
   --
   
   Besides how to implement this, I have a question about compatibility of 
`taskmanager.load-balance.mode` and `cluster.evenly-spread-out-slots`.
   
   As I understand :
   - `taskmanager.load-balance.mode : Slots`  == 
`cluster.evenly-spread-out-slots: true`
   - `taskmanager.load-balance.mode : None`  == 
`cluster.evenly-spread-out-slots: false` (By default.)
   
   After this FLIP is merged, if a job with `cluster.evenly-spread-out-slots: 
true`, and didn't set the `taskmanager.load-balance.mode`. Should we change the 
default value from `None` to `Slots`?
   
   - If yes, we can support old users directly.
   - If no, users must set `taskmanager.load-balance.mode : Slots` manually if 
they want to the `evenly-spread-out-slots` feature.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Yuxin Tan (Jira)]

[ https://issues.apache.org/jira/browse/FLINK-33445 ]


Yuxin Tan deleted comment on FLINK-33445:
---

was (Author: tanyuxin):
[~liyubin117] Thanks for helping, assigned to you.

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Assignee: Yubin Li
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
> adding an introduction about how to migrate DataSet API to DataStream has 
> been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33304] Introduce the DeduplicatedMutator to resolve the mutation write conflict problem [flink-connector-hbase]

[via GitHub]

Tan-JiaLiang commented on code in PR #30:
URL: 
https://github.com/apache/flink-connector-hbase/pull/30#discussion_r1381181977


##
flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java:
##
@@ -256,4 +263,39 @@ public void 
onException(RetriesExhaustedWithDetailsException exception, Buffered
 // if the failure handler decides to throw an exception
 failureThrowable.compareAndSet(null, exception);
 }
+
+/**
+ * Thread-safe class, grouped mutations by rows and keep the latest 
mutation. For more info, see
+ * https://issues.apache.org/jira/browse/HBASE-8626";>HBASE-8626.
+ */
+private static class DeduplicatedMutator {

Review Comment:
   Sorry for breaking our previous agreement. The reason for using 
`synchronized` is that `HBaseSinkFunction` has a timer task to periodically 
call the `flush` API, which makes the `DeduplicatedMutator#mutations` (the map 
for deduplication) has a potentially thread-safe problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-32650][protobuf]Added the ability to split flink-protobuf code… [flink]

[via GitHub]

libenchao commented on PR #23162:
URL: https://github.com/apache/flink/pull/23162#issuecomment-1791884216

   > @libenchao Sorry to bother you, do you have time to do the last step of 
reveiw and commit recently?
   
   Thanks for the patience, I'll review this next week, let's move forward to 
get it in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-33417) Update netty version to 4.1.83 for flink-shaded

[Yuxin Tan (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782399#comment-17782399
 ] 

Yuxin Tan commented on FLINK-33417:
---

[~mapohl] Actually, as I don't possess an in-depth understanding of Netty, I 
haven't determined the internal cause of the error or how this can resolve the 
issue. But it indeed works.

> Update netty version to 4.1.83 for flink-shaded
> ---
>
> Key: FLINK-33417
> URL: https://issues.apache.org/jira/browse/FLINK-33417
> Project: Flink
>  Issue Type: Bug
>  Components: BuildSystem / Shaded
>Affects Versions: shaded-16.1
>Reporter: Yuxin Tan
>Assignee: Yuxin Tan
>Priority: Major
>  Labels: pull-request-available
> Attachments: 09DD8FAA-7701-4AA0-85F9-4ABA6C5117DF.png, 
> BE207321-9677-4721-9415-BD3312C29824.png
>
>
> In our ARM environment, we encounter a compile error when using Flink 1.17.
>  (The BE20xxx pic is the error when using 4.1.82. The 09DDxx pic is the pic 
> of compiling successfully after using 4.1.83.)
> Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82. However, 
> flink-shaded 16.1 fails to compile in the ARM environment. As a result, we 
> are unable to compile Flink 1.17 due to this issue.
> We have tested compiling flink-shaded using netty 4.1.83 or a later version 
> in ARM env, and it can compile successfully.
> Taking into consideration the previous discussions regarding compatibility 
> and the dependency of external connectors on this version, I propose 
> addressing the bug by only updating flink-shaded's netty to a minor version 
> (e.g., 4.1.83) rather than backporting FLINK-32032. 
> To implement the update, maybe a new release of flink-shaded 16.2 needs to be 
> released.
> The discussion details is at 
> https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33446) SubQueryAntiJoinTest#testMultiNotExistsWithCorrelatedOnWhere_NestedCorrelation doesn't produce the correct plan

[Shengkai Fang (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shengkai Fang updated FLINK-33446:
--
Description: 
Although this test doesn't throw an exception, the final plan produces 3 
columns rather than 2 after optimization.

{code:java}
LogicalProject(inputs=[0..1], exprs=[[$4]])
+- LogicalFilter(condition=[IS NULL($5)])
   +- LogicalJoin(condition=[AND(=($0, $2), =($1, $3))], joinType=[left])
  :- LogicalProject(exprs=[[+(2, $0), +(3, $1)]])
  :  +- LogicalTableScan(table=[[default_catalog, default_database, l, 
source: [TestTableSource(a, b, c)]]])
  +- LogicalProject(inputs=[0..2], exprs=[[true]])
 +- LogicalAggregate(group=[{0, 1, 2}])
+- LogicalProject(inputs=[0..2])
   +- LogicalFilter(condition=[IS NULL($3)])
  +- LogicalJoin(condition=[true], joinType=[left])
 :- LogicalFilter(condition=[IS NOT NULL($0)])
 :  +- LogicalProject(exprs=[[+($0, 1)]])
 : +- LogicalTableScan(table=[[default_catalog, 
default_database, r, source: [TestTableSource(d, e, f)]]])
 +- LogicalProject(inputs=[0..1], exprs=[[true]])
+- LogicalAggregate(group=[{0, 1}])
   +- LogicalProject(exprs=[[$3, $0]])
  +- LogicalFilter(condition=[AND(=($1, $0), 
=(CAST($2):BIGINT, $3))])
 +- LogicalProject(exprs=[[+($0, 4), +($0, 5), 
+($0, 6), CAST(+($0, 6)):BIGINT]])
+- 
LogicalTableScan(table=[[default_catalog, default_database, t, source: 
[TestTableSource(i, j, k)]]])

{code}

After digging, I think it's the SubQueryRemoveRule doesn't generate the 
Correlate but generates the Join node, which causes the failure of the 
decorrelation. For a quick fix, I think we should throw an exception to notify 
users it's not a supported feature in the Flink. 

There might exist 2 ways to fix this issue:
1. Expand subquery when converting SQL to rel.  After experimenting with 
calcite, I found that the Sql2RelConverter generates the correct plan.

{code:java}
LogicalProject(inputs=[0..1])
+- LogicalFilter(condition=[IS NULL($2)])
   +- LogicalCorrelate(correlation=[$cor7], joinType=[left], 
requiredColumns=[{0, 1}])
  :- LogicalProject(exprs=[[+(2, $0), +(3, $1)]])
  :  +- LogicalTableScan(table=[[default_catalog, default_database, l, 
source: [TestTableSource(a, b, c)]]])
  +- LogicalAggregate(group=[{}], agg#0=[MIN($0)])
 +- LogicalProject(exprs=[[true]])
+- LogicalFilter(condition=[AND(=($0, $cor7.d2), IS NULL($1))])
   +- LogicalCorrelate(correlation=[$cor4], joinType=[left], 
requiredColumns=[{0}])
  :- LogicalProject(inputs=[0])
  :  +- LogicalTableScan(table=[[default_catalog, 
default_database, r, source: [TestTableSource(d1, e, f)]]])
  +- LogicalAggregate(group=[{}], agg#0=[MIN($0)])
 +- LogicalProject(exprs=[[true]])
+- LogicalFilter(condition=[AND(=($0, $cor4.d1), =($1, 
$cor4.d1), =(CAST($2):BIGINT, $cor7.d3))])
   +- LogicalProject(exprs=[[+($0, 4), +($0, 5), +($0, 
6)]])
  +- LogicalTableScan(table=[[default_catalog, 
default_database, t, source: [TestTableSource(i, j, k)]]])
{code}

You can find the new plan uses a correlate node rather than a join node.

2. CALCITE-4686 might fix this problem by removing the nested correlation node.







  was:
Although this test doesn't throw an exception, you can find the final produce 3 
columns rather than 2 columns after optimization.

{code:java}
LogicalProject(inputs=[0..1], exprs=[[$4]])
+- LogicalFilter(condition=[IS NULL($5)])
   +- LogicalJoin(condition=[AND(=($0, $2), =($1, $3))], joinType=[left])
  :- LogicalProject(exprs=[[+(2, $0), +(3, $1)]])
  :  +- LogicalTableScan(table=[[default_catalog, default_database, l, 
source: [TestTableSource(a, b, c)]]])
  +- LogicalProject(inputs=[0..2], exprs=[[true]])
 +- LogicalAggregate(group=[{0, 1, 2}])
+- LogicalProject(inputs=[0..2])
   +- LogicalFilter(condition=[IS NULL($3)])
  +- LogicalJoin(condition=[true], joinType=[left])
 :- LogicalFilter(condition=[IS NOT NULL($0)])
 :  +- LogicalProject(exprs=[[+($0, 1)]])
 : +- LogicalTableScan(table=[[default_catalog, 
default_database, r, source: [TestTableSource(d, e, f)]]])
 +- LogicalProject(inputs=[0..1], exprs=[[true]])
+- LogicalAggregate(group=[{0, 1}])
   +- LogicalProject(exprs=[[$3, $0]])
  +- LogicalFilter(condition=[AND(=($1, $0), 
=(CAST($2):BIGINT, $3))])
 +- Lo

[jira] [Created] (FLINK-33446) SubQueryAntiJoinTest#testMultiNotExistsWithCorrelatedOnWhere_NestedCorrelation doesn't produce the correct plan

[Shengkai Fang (Jira)]

Shengkai Fang created FLINK-33446:
-

 Summary: 
SubQueryAntiJoinTest#testMultiNotExistsWithCorrelatedOnWhere_NestedCorrelation 
doesn't produce the correct plan
 Key: FLINK-33446
 URL: https://issues.apache.org/jira/browse/FLINK-33446
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.17.2, 1.19.0, 1.18.1
Reporter: Shengkai Fang


Although this test doesn't throw an exception, you can find the final produce 3 
columns rather than 2 columns after optimization.

{code:java}
LogicalProject(inputs=[0..1], exprs=[[$4]])
+- LogicalFilter(condition=[IS NULL($5)])
   +- LogicalJoin(condition=[AND(=($0, $2), =($1, $3))], joinType=[left])
  :- LogicalProject(exprs=[[+(2, $0), +(3, $1)]])
  :  +- LogicalTableScan(table=[[default_catalog, default_database, l, 
source: [TestTableSource(a, b, c)]]])
  +- LogicalProject(inputs=[0..2], exprs=[[true]])
 +- LogicalAggregate(group=[{0, 1, 2}])
+- LogicalProject(inputs=[0..2])
   +- LogicalFilter(condition=[IS NULL($3)])
  +- LogicalJoin(condition=[true], joinType=[left])
 :- LogicalFilter(condition=[IS NOT NULL($0)])
 :  +- LogicalProject(exprs=[[+($0, 1)]])
 : +- LogicalTableScan(table=[[default_catalog, 
default_database, r, source: [TestTableSource(d, e, f)]]])
 +- LogicalProject(inputs=[0..1], exprs=[[true]])
+- LogicalAggregate(group=[{0, 1}])
   +- LogicalProject(exprs=[[$3, $0]])
  +- LogicalFilter(condition=[AND(=($1, $0), 
=(CAST($2):BIGINT, $3))])
 +- LogicalProject(exprs=[[+($0, 4), +($0, 5), 
+($0, 6), CAST(+($0, 6)):BIGINT]])
+- 
LogicalTableScan(table=[[default_catalog, default_database, t, source: 
[TestTableSource(i, j, k)]]])

{code}

After digging, I think it's the SubQueryRemoveRule doesn't generate the 
Correlate but generates the Join node, which causes the failure of the 
decorrelation. For a quick fix, I think we should throw an exception to notify 
users it's not a supported feature in the Flink. 

There might exist 2 ways to fix this issue:
1. Expand subquery when converting SQL to rel.  After experimenting with 
calcite, I find the Sql2RelConverter generates the correct plan.

{code:java}
LogicalProject(inputs=[0..1])
+- LogicalFilter(condition=[IS NULL($2)])
   +- LogicalCorrelate(correlation=[$cor7], joinType=[left], 
requiredColumns=[{0, 1}])
  :- LogicalProject(exprs=[[+(2, $0), +(3, $1)]])
  :  +- LogicalTableScan(table=[[default_catalog, default_database, l, 
source: [TestTableSource(a, b, c)]]])
  +- LogicalAggregate(group=[{}], agg#0=[MIN($0)])
 +- LogicalProject(exprs=[[true]])
+- LogicalFilter(condition=[AND(=($0, $cor7.d2), IS NULL($1))])
   +- LogicalCorrelate(correlation=[$cor4], joinType=[left], 
requiredColumns=[{0}])
  :- LogicalProject(inputs=[0])
  :  +- LogicalTableScan(table=[[default_catalog, 
default_database, r, source: [TestTableSource(d1, e, f)]]])
  +- LogicalAggregate(group=[{}], agg#0=[MIN($0)])
 +- LogicalProject(exprs=[[true]])
+- LogicalFilter(condition=[AND(=($0, $cor4.d1), =($1, 
$cor4.d1), =(CAST($2):BIGINT, $cor7.d3))])
   +- LogicalProject(exprs=[[+($0, 4), +($0, 5), +($0, 
6)]])
  +- LogicalTableScan(table=[[default_catalog, 
default_database, t, source: [TestTableSource(i, j, k)]]])
{code}

You can find the new plan uses a correlate node rather than join node.

2. CALCITE-4686 might fix this problem by removing the nested correlation node.









--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782381#comment-17782381
 ] 

zhengzhili edited comment on FLINK-33423 at 11/3/23 3:54 AM:
-

This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. 

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

Then change it to call it in org.apache.flink.yarn.utils 
#getYarnAndHadoopConfiguration

!image-2023-11-03-11-05-01-182.png|width=732,height=134!

This way, the original call to Utils#getHadoopConfiguration won't be affected 
anywhere else

 


was (Author: JIRAUSER302860):
This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. 

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

Then change it to call it in org.apache.flink.yarn.utils 
#getYarnAndHadoopConfiguration

!image-2023-11-03-11-05-01-182.png|width=732,height=134!

This way no other places where Utils#getHadoopConfiguration was originally 
called will be affected

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, image-2023-11-03-10-35-07-868.png, 
> image-2023-11-03-11-05-01-182.png, 微信图片_20231101151644.png, 
> 微信图片_20231101152359.png, 微信图片_20231101152404.png, 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Pa

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782381#comment-17782381
 ] 

zhengzhili edited comment on FLINK-33423 at 11/3/23 3:53 AM:
-

This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. 

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

Then change it to call it in org.apache.flink.yarn.utils 
#getYarnAndHadoopConfiguration

!image-2023-11-03-11-05-01-182.png|width=732,height=134!

This way no other places where Utils#getHadoopConfiguration was originally 
called will be affected

 


was (Author: JIRAUSER302860):
This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. 

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

Then change it to call it in org.apache.flink.yarn.utils 
#getYarnAndHadoopConfiguration

!image-2023-11-03-11-05-01-182.png|width=732,height=134!

 

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, image-2023-11-03-10-35-07-868.png, 
> image-2023-11-03-11-05-01-182.png, 微信图片_20231101151644.png, 
> 微信图片_20231101152359.png, 微信图片_20231101152404.png, 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopCon

Re: [PR] [FLINK-32650][protobuf]Added the ability to split flink-protobuf code… [flink]

[via GitHub]

ljw-hit commented on PR #23162:
URL: https://github.com/apache/flink/pull/23162#issuecomment-1791846985

   @libenchao Sorry to bother you, do you have time to do the last step of 
reveiw and commit recently?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-32738][formats] PROTOBUF format supports projection push down [flink]

[via GitHub]

ljw-hit commented on PR #23323:
URL: https://github.com/apache/flink/pull/23323#issuecomment-1791845703

   @zhougit86  Sorry, I'm a little busy recently. You can take a look at this 
[PR](https://github.com/apache/flink/pull/23162). There are some modifications 
to the logic of codegen and the code will be split. You may need to rebase this 
for compatibility.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782381#comment-17782381
 ] 

zhengzhili edited comment on FLINK-33423 at 11/3/23 3:05 AM:
-

This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. 

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

Then change it to call it in org.apache.flink.yarn.utils 
#getYarnAndHadoopConfiguration

!image-2023-11-03-11-05-01-182.png|width=732,height=134!

 

 


was (Author: JIRAUSER302860):
This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. Then change it to 
call it in org.apache.flink.yarn.utils #getYarnAndHadoopConfiguration

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, image-2023-11-03-10-35-07-868.png, 
> image-2023-11-03-11-05-01-182.png, 微信图片_20231101151644.png, 
> 微信图片_20231101152359.png, 微信图片_20231101152404.png, 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This

[jira] [Commented] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Yuxin Tan (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782385#comment-17782385
 ] 

Yuxin Tan commented on FLINK-33445:
---

[~liyubin117] Thanks for helping, assigned to you.

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Assignee: Yubin Li
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
> adding an introduction about how to migrate DataSet API to DataStream has 
> been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Wencong Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782384#comment-17782384
 ] 

Wencong Liu commented on FLINK-33445:
-

Thanks [~liyubin117] ! Assigned to you. Please go ahead.

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Assignee: Yubin Li
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
> adding an introduction about how to migrate DataSet API to DataStream has 
> been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Yuxin Tan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuxin Tan reassigned FLINK-33445:
-

Assignee: Yubin Li

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Assignee: Yubin Li
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
> adding an introduction about how to migrate DataSet API to DataStream has 
> been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782381#comment-17782381
 ] 

zhengzhili edited comment on FLINK-33423 at 11/3/23 2:37 AM:
-

This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. Then change it to 
call it in org.apache.flink.yarn.utils #getYarnAndHadoopConfiguration

!image-2023-11-03-10-35-07-868.png|width=713,height=284!


was (Author: JIRAUSER302860):
This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. Then call this 
method in org.apache.flink.yarn.Utils#getYarnAndHadoopConfiguration instead

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, image-2023-11-03-10-35-07-868.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782381#comment-17782381
 ] 

zhengzhili commented on FLINK-33423:


This is probably change train of thought, I can keep the original method, 
adding new method HadoopUtils#getYarnAndHadoopConfiguration. Then call this 
method in org.apache.flink.yarn.Utils#getYarnAndHadoopConfiguration instead

!image-2023-11-03-10-35-07-868.png|width=713,height=284!

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, image-2023-11-03-10-35-07-868.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhengzhili updated FLINK-33423:
---
Attachment: image-2023-11-03-10-35-07-868.png

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, image-2023-11-03-10-35-07-868.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Yubin Li (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782380#comment-17782380
 ] 

Yubin Li commented on FLINK-33445:
--

[~Wencong Liu] Could you please assign this to me? thanks!

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
> adding an introduction about how to migrate DataSet API to DataStream has 
> been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Wencong Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wencong Liu updated FLINK-33445:

Description: 
The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
adding an introduction about how to migrate DataSet API to DataStream has been 
merged into master branch. Here is the 
[LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
 in the Flink website.

According to the [contribution 
guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
 we should add an identical markdown file in {{content.zh/}} and translate it 
to Chinese. Any community volunteers are welcomed to take this task.

 

  was:
The FLIINK-33041 about adding an introduction about how to migrate DataSet API 
to DataStream has been merged into master branch. Here is the 
[LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
 in the Flink website.

According to the [contribution 
guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
 we should add an identical markdown file in {{content.zh/}} and translate it 
to Chinese. Any community volunteers are welcomed to take this task.

 


> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
> adding an introduction about how to migrate DataSet API to DataStream has 
> been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Wencong Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wencong Liu updated FLINK-33445:

Component/s: chinese-translation
 (was: Documentation)

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Priority: Major
> Fix For: 1.19.0
>
>
> The FLIINK-33041 about adding an introduction about how to migrate DataSet 
> API to DataStream has been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Wencong Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wencong Liu updated FLINK-33445:

Labels: starter  (was: )

> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Priority: Major
>  Labels: starter
> Fix For: 1.19.0
>
>
> The FLIINK-33041 about adding an introduction about how to migrate DataSet 
> API to DataStream has been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Wencong Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wencong Liu updated FLINK-33445:

Description: 
The FLIINK-33041 about adding an introduction about how to migrate DataSet API 
to DataStream has been merged into master branch. Here is the 
[LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
 in the Flink website.

According to the [contribution 
guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
 we should add an identical markdown file in {{content.zh/}} and translate it 
to Chinese. Any community volunteers are welcomed to take this task.

 

  was:
The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
adding an introduction about how to migrate DataSet API to DataStream has been 
merged into master branch. Here is the link in the Flink website: [How to 
Migrate from DataSet to DataStream | Apache 
Flink|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]

According to the [contribution 
guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
 we should add an identical markdown file in {{content.zh/}} and translate it 
to Chinese. Any community volunteers are welcomed to take this task.

 


> Translate DataSet migration guideline to Chinese
> 
>
> Key: FLINK-33445
> URL: https://issues.apache.org/jira/browse/FLINK-33445
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.19.0
>Reporter: Wencong Liu
>Priority: Major
> Fix For: 1.19.0
>
>
> The FLIINK-33041 about adding an introduction about how to migrate DataSet 
> API to DataStream has been merged into master branch. Here is the 
> [LINK|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]
>  in the Flink website.
> According to the [contribution 
> guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
>  we should add an identical markdown file in {{content.zh/}} and translate it 
> to Chinese. Any community volunteers are welcomed to take this task.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33445) Translate DataSet migration guideline to Chinese

[Wencong Liu (Jira)]

Wencong Liu created FLINK-33445:
---

 Summary: Translate DataSet migration guideline to Chinese
 Key: FLINK-33445
 URL: https://issues.apache.org/jira/browse/FLINK-33445
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.19.0
Reporter: Wencong Liu
 Fix For: 1.19.0


The [FLIINK-33041|https://issues.apache.org/jira/browse/FLINK-33041] about 
adding an introduction about how to migrate DataSet API to DataStream has been 
merged into master branch. Here is the link in the Flink website: [How to 
Migrate from DataSet to DataStream | Apache 
Flink|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/dataset_migration/]

According to the [contribution 
guidelines|https://flink.apache.org/how-to-contribute/contribute-documentation/#chinese-documentation-translation],
 we should add an identical markdown file in {{content.zh/}} and translate it 
to Chinese. Any community volunteers are welcomed to take this task.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782371#comment-17782371
 ] 

zhengzhili edited comment on FLINK-33423 at 11/3/23 1:30 AM:
-

We can indeed do this by adding flink.yarn., but core and hdfs can load 
the configuration by default, but yarn does not.Isn't that weird  
!image-2023-11-03-09-27-59-792.png|width=659,height=142!


was (Author: JIRAUSER302860):
We can do this by adding flink.yarn., but core and hdfs can load the 
configuration by default, but yarn does not.Isn't that weird 
!image-2023-11-03-09-27-59-792.png|width=659,height=142!

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, 微信图片_20231101151644.png, 
> 微信图片_20231101152359.png, 微信图片_20231101152404.png, 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782371#comment-17782371
 ] 

zhengzhili edited comment on FLINK-33423 at 11/3/23 1:28 AM:
-

We can do this by adding flink.yarn., but core and hdfs can load the 
configuration by default, but yarn does not.Isn't that weird 
!image-2023-11-03-09-27-59-792.png|width=659,height=142!


was (Author: JIRAUSER302860):
We can do this by adding flink.yarn., but core and hdfs can load the 
configuration by default, but yarn does not.Isn't that weird

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> image-2023-11-03-09-27-59-792.png, 微信图片_20231101151644.png, 
> 微信图片_20231101152359.png, 微信图片_20231101152404.png, 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782371#comment-17782371
 ] 

zhengzhili commented on FLINK-33423:


We can do this by adding flink.yarn., but core and hdfs can load the 
configuration by default, but yarn does not.Isn't that weird

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33444) Fixing the stability of "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33444:

Description: 
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable_  will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 

  was:
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 


> Fixing the stability of 
> "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"
> ---
>
> Key: FLINK-33444
> URL: https://issues.apache.org/jira/browse/FLINK-33444
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing a solution to fix potential flakiness in the following test:
> {code:java}
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305

[jira] [Updated] (FLINK-33444) Fixing the stability of "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33444:

Description: 
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_ variable  will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 

  was:
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable_  will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 


> Fixing the stability of 
> "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"
> ---
>
> Key: FLINK-33444
> URL: https://issues.apache.org/jira/browse/FLINK-33444
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing a solution to fix potential flakiness in the following test:
> {code:java}
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:3

[jira] [Updated] (FLINK-33444) Fixing the stability of "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33444:

Description: 
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_ variable will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 

  was:
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_ variable  will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 


> Fixing the stability of 
> "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"
> ---
>
> Key: FLINK-33444
> URL: https://issues.apache.org/jira/browse/FLINK-33444
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing a solution to fix potential flakiness in the following test:
> {code:java}
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:30

[jira] [Updated] (FLINK-33444) Fixing the stability of "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33444:

Description: 
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable will be consistent. There are 
currently two versions of query output that can be stored in _results_ :

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 

  was:
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable will be consistent. There are 
currently two versions of query output that can be stored in _results_:

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 


> Fixing the stability of 
> "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"
> ---
>
> Key: FLINK-33444
> URL: https://issues.apache.org/jira/browse/FLINK-33444
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing a solution to fix potential flakiness in the following test:
> {code:java}
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
>

[jira] [Created] (FLINK-33444) Fixing the stability of "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"

[Krishna Anandan Ganesan (Jira)]

Krishna Anandan Ganesan created FLINK-33444:
---

 Summary: Fixing the stability of 
"org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"
 Key: FLINK-33444
 URL: https://issues.apache.org/jira/browse/FLINK-33444
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Krishna Anandan Ganesan


We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable will be consistent. There are 
currently two versions of query output that can be stored in _results_:

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}.
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}.

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33444) Fixing the stability of "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33444:

Description: 
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable will be consistent. There are 
currently two versions of query output that can be stored in _results_:

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 

  was:
We are proposing a solution to fix potential flakiness in the following test:
{code:java}
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
{code}

 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 
expected: "[+I[{1=a, 2=b}], +I[{3=c}]]"
 but was: "[+I[{2=b, 1=a}], +I[{3=c}]]"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped
{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _results_variable will be consistent. There are 
currently two versions of query output that can be stored in _results_:

# The actual order that is expected where the output of the map attribute is 
{1=a,2=b}.
# The other order which is shown in the error extract above where the ordering 
of the map attribute changes to {2=b,1=a}.

*POTENTIAL FIX:*
* The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
* By asserting whether the contents in _results_ are in one of the two orders, 
we can ascertain that the expected attributes with their contents are received 
as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

 


> Fixing the stability of 
> "org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase.testUDTF"
> ---
>
> Key: FLINK-33444
> URL: https://issues.apache.org/jira/browse/FLINK-33444
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing a solution to fix potential flakiness in the following test:
> {code:java}
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF {code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase#testUDTF
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   TableEnvHiveConnectorITCase.testUDTF:270->lambda$testUDTF$5:305 

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

jnh5y commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380958351


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/typeutils/TypeCoercion.scala:
##
@@ -53,85 +53,4 @@ object TypeCoercion {
 
   case _ => false
 }
-
-  /**
-   * All the supported cast types in flink-table.
-   *
-   * Note: No distinction between explicit and implicit conversions Note: This 
is a subset of
-   * SqlTypeAssignmentRule Note: This may lose type during the cast.
-   */
-  def canCast(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (_, VARCHAR | CHAR) => true
-
-  case (VARCHAR | CHAR, _) if isNumeric(to) => true
-  case (VARCHAR | CHAR, BOOLEAN) => true
-  case (VARCHAR | CHAR, DECIMAL) => true
-  case (VARCHAR | CHAR, DATE) => true
-  case (VARCHAR | CHAR, TIME_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITH_LOCAL_TIME_ZONE) => true
-
-  case (BOOLEAN, _) if isNumeric(to) => true
-  case (BOOLEAN, DECIMAL) => true
-  case (_, BOOLEAN) if isNumeric(from) => true
-  case (DECIMAL, BOOLEAN) => true
-
-  case (_, _) if isNumeric(from) && isNumeric(to) => true
-  case (_, DECIMAL) if isNumeric(from) => true
-  case (DECIMAL, _) if isNumeric(to) => true
-  case (DECIMAL, DECIMAL) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (TINYINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (SMALLINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (DOUBLE, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (FLOAT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-
-  case (DATE, TIME_WITHOUT_TIME_ZONE) => false
-  case (TIME_WITHOUT_TIME_ZONE, DATE) => false
-  case (_, _) if isTimePoint(from) && isTimePoint(to) => true
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, TINYINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, SMALLINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, DOUBLE) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, FLOAT) => true
-
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case _ => false
-}
-
-  /** All the supported reinterpret types in flink-table. */
-  def canReinterpret(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case (DATE, BIGINT) => true
-  case (TIME_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTERVAL_YEAR_MONTH, BIGINT) => true
-
-  case _ => false

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33443:

Description: 
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 * The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 * The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1: "a", 2: "b"}.
 # The other order is the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2: "b", 1: "a"}.

*POTENTIAL FIX:*
 * The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
 * By asserting on whether the contents in _result_ are in one of the two 
orders, we can ascertain that the expected attributes with their contents are 
received as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

  was:
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1: "a", 2: "b"}.
 # The other order is the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2: "b", 1: "a"}.

*POTENTIAL FIX:*
 * The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
 * By asserting on whether the contents in _result_ are in one of the two 
orders, we can ascertain that the expected attributes with their contents are 
received as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.


> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   HiveRunnerITCase.testWriteComplexType:166 
> expected: "[1,2,3]{1:"a",2:"b"}   {"f1":3,"f2":"c"}"
>  but was: "[1,2,3]{2:"b",1:"a"}   {"f1":3,"f2":"c"

[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33443:

Description: 
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1: "a", 2: "b"}.
 # The other order is the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2: "b", 1: "a"}.

*POTENTIAL FIX:*
 * The fix that I can suggest/have ready to raise a PR for is introducing 
another assertion on the second variant of the query output.
 * By asserting on whether the contents in _result_ are in one of the two 
orders, we can ascertain that the expected attributes with their contents are 
received as expected should the order in which they are received, not matter.

Please share your thoughts on this finding and let me know if any other 
potential fix is possible for this test.

  was:
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1: "a", 2: "b"}.
 # The other order is the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2: "b", 1: "a"}.

*POTENTIAL FIX:*
 * The fix that I can suggest/have ready to raise a PR for is by introducing 
another assertion on the second variant of the query output.
 *  


> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  # The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  # The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   HiveRunnerITCase.testWriteComplexType:166 
> expected: "[1,2,3]{1:"a",2:"b"}   {"f1":3,"f2":"c"}"
>  but was: "[1,2,3]{2:"b",1:"a"}   {"f1":3,"f2":"c"}"
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of 
> elements received in the _result_ variable will be consistent. There are 
> currently two versions of query output that can be stored in _result._
>  # The act

[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33443:

Description: 
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1: "a", 2: "b"}.
 # The other order is the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2: "b", 1: "a"}.

*POTENTIAL FIX:*
 * The fix that I can suggest/have ready to raise a PR for is by introducing 
another assertion on the second variant of the query output.
 *  

  was:
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1:"a", 2:"b"}.
 # The other order being the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2:"b", 1:"a"}.

*POTENTIAL FIX:*


> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  # The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  # The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   HiveRunnerITCase.testWriteComplexType:166 
> expected: "[1,2,3]{1:"a",2:"b"}   {"f1":3,"f2":"c"}"
>  but was: "[1,2,3]{2:"b",1:"a"}   {"f1":3,"f2":"c"}"
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of 
> elements received in the _result_ variable will be consistent. There are 
> currently two versions of query output that can be stored in _result._
>  # The actual order that is expected where the output of the map attribute is 
> \{1: "a", 2: "b"}.
>  # The other order is the one shown in the error extract above where the 
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
>  * The fix that I can suggest/have ready to raise a PR for is by introducing 
> another assertion on the second variant of the query output.
>  *  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33443:

Description: 
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}
 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the _result_ variable will be consistent. There are 
currently two versions of query output that can be stored in _result._
 # The actual order that is expected where the output of the map attribute is 
\{1:"a", 2:"b"}.
 # The other order being the one shown in the error extract above where the 
ordering of the map attribute from the table is \{2:"b", 1:"a"}.

*POTENTIAL FIX:*

  was:
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}

 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the "result" variable will be consistent. There are 
currently two versions of query output that can be stored into result. 


> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  # The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  # The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   HiveRunnerITCase.testWriteComplexType:166 
> expected: "[1,2,3]{1:"a",2:"b"}   {"f1":3,"f2":"c"}"
>  but was: "[1,2,3]{2:"b",1:"a"}   {"f1":3,"f2":"c"}"
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of 
> elements received in the _result_ variable will be consistent. There are 
> currently two versions of query output that can be stored in _result._
>  # The actual order that is expected where the output of the map attribute is 
> \{1:"a", 2:"b"}.
>  # The other order being the one shown in the error extract above where the 
> ordering of the map attribute from the table is \{2:"b", 1:"a"}.
> *POTENTIAL FIX:*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33443:

Description: 
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}

 # The following error will be encountered:

{code:java}
[ERROR] Failures: 
[ERROR]   HiveRunnerITCase.testWriteComplexType:166 
expected: "[1,2,3]  {1:"a",2:"b"}   {"f1":3,"f2":"c"}"
 but was: "[1,2,3]  {2:"b",1:"a"}   {"f1":3,"f2":"c"}"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}

*ROOT CAUSE ANALYSIS:*
The test is currently flaky because of the assumption that the order of 
elements received in the "result" variable will be consistent. There are 
currently two versions of query output that can be stored into result. 

  was:
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
The test is currently flaky because of the assumption that the order of 
elements received in "result" will be consistent.

*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}

 # The following error will be encountered.


> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  # The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  # The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   HiveRunnerITCase.testWriteComplexType:166 
> expected: "[1,2,3]{1:"a",2:"b"}   {"f1":3,"f2":"c"}"
>  but was: "[1,2,3]{2:"b",1:"a"}   {"f1":3,"f2":"c"}"
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of 
> elements received in the "result" variable will be consistent. There are 
> currently two versions of query output that can be stored into result. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Anandan Ganesan updated FLINK-33443:

Description: 
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
The test is currently flaky because of the assumption that the order of 
elements received in "result" will be consistent.

*STEPS TO REPRODUCE THE ISSUE:*
 # The following command can be run to execute the test with the 
[NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:

{code:java}
mvn -pl flink-connectors/flink-connector-hive 
edu.illinois:nondex-maven-plugin:2.1.1:nondex 
-Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
{code}

 # The following error will be encountered.

  was:
We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}


> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> The test is currently flaky because of the assumption that the order of 
> elements received in "result" will be consistent.
> *STEPS TO REPRODUCE THE ISSUE:*
>  # The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  # The following error will be encountered.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33443) Make the test "testWriteComplexType" stable

[Krishna Anandan Ganesan (Jira)]

Krishna Anandan Ganesan created FLINK-33443:
---

 Summary: Make the test "testWriteComplexType" stable
 Key: FLINK-33443
 URL: https://issues.apache.org/jira/browse/FLINK-33443
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Krishna Anandan Ganesan


We are proposing to make the following test stable:
{code:java}
org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33363) docker bases images can't run the java compiler

[Henning Schmiedehausen (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782352#comment-17782352
 ] 

Henning Schmiedehausen commented on FLINK-33363:


So, the comment actually brought me on the right track (and here is how to fix 
this well without resorting to changing the configuration):

I had the "org.apache.flink:flink-table-runtime" as a dependency for my job. 
And when I packaged it up for deployment, the classes from that jar ended up in 
my job jar. This caused the classloader to choke.

removing this dependency (or dropping it to provided scope) fixed the problem 
for me.


> docker bases images can't run the java compiler
> ---
>
> Key: FLINK-33363
> URL: https://issues.apache.org/jira/browse/FLINK-33363
> Project: Flink
>  Issue Type: Bug
>  Components: flink-docker, Table SQL / Runtime
>Affects Versions: 1.17.1
>Reporter: Henning Schmiedehausen
>Priority: Blocker
>
> I have set up a small cluster (Job Manager + 2 Task managers) using docker 
> compose. When submitting a flink job that needs Calcite planning, it crashes 
> with
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: Unable to instantiate java compiler
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) 
> ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown 
> Source) [?:?]
> at java.lang.Thread.run(Unknown Source) [?:?]
> Caused by: java.lang.IllegalStateException: Unable to instantiate java 
> compiler
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:163)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:141)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:73)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:197)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
>  ~[?:?]
> at 
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
>  ~[?:?]
> at 
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565) ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428) ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)

[jira] [Commented] (FLINK-33363) docker bases images can't run the java compiler

[Henning Schmiedehausen (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782348#comment-17782348
 ] 

Henning Schmiedehausen commented on FLINK-33363:


Can't do anything with the slack link, it seems that the workspace does not 
allow public signup and I don't have an account (and my apache.org mail address 
does not work).

> docker bases images can't run the java compiler
> ---
>
> Key: FLINK-33363
> URL: https://issues.apache.org/jira/browse/FLINK-33363
> Project: Flink
>  Issue Type: Bug
>  Components: flink-docker, Table SQL / Runtime
>Affects Versions: 1.17.1
>Reporter: Henning Schmiedehausen
>Priority: Blocker
>
> I have set up a small cluster (Job Manager + 2 Task managers) using docker 
> compose. When submitting a flink job that needs Calcite planning, it crashes 
> with
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: Unable to instantiate java compiler
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) 
> ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown 
> Source) [?:?]
> at java.lang.Thread.run(Unknown Source) [?:?]
> Caused by: java.lang.IllegalStateException: Unable to instantiate java 
> compiler
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:163)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:141)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:73)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:197)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
>  ~[?:?]
> at 
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
>  ~[?:?]
> at 
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565) ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428) ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208) 
> ~[?:?]
> at 
> org.apache.calcite.plan

[jira] [Commented] (FLINK-33363) docker bases images can't run the java compiler

[Henning Schmiedehausen (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782346#comment-17782346
 ] 

Henning Schmiedehausen commented on FLINK-33363:


Will try. I will still keep using the official flink images ("docker pull 
flink:latest")  so this may need resolution on your side. 

> docker bases images can't run the java compiler
> ---
>
> Key: FLINK-33363
> URL: https://issues.apache.org/jira/browse/FLINK-33363
> Project: Flink
>  Issue Type: Bug
>  Components: flink-docker, Table SQL / Runtime
>Affects Versions: 1.17.1
>Reporter: Henning Schmiedehausen
>Priority: Blocker
>
> I have set up a small cluster (Job Manager + 2 Task managers) using docker 
> compose. When submitting a flink job that needs Calcite planning, it crashes 
> with
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: Unable to instantiate java compiler
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) 
> ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at 
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108)
>  ~[flink-dist-1.17.1.jar:1.17.1]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown 
> Source) [?:?]
> at java.lang.Thread.run(Unknown Source) [?:?]
> Caused by: java.lang.IllegalStateException: Unable to instantiate java 
> compiler
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:163)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:141)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:73)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
>  ~[?:?]
> at 
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:197)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
>  ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
>  ~[?:?]
> at 
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
>  ~[?:?]
> at 
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565) ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:428) ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:251)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:130)
>  ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:208) 
> ~[?:?]
> at 
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.j

[jira] [Comment Edited] (FLINK-33430) Fix Test Failures

[dserfe (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782344#comment-17782344
 ] 

dserfe edited comment on FLINK-33430 at 11/2/23 10:49 PM:
--

Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening. It's fine not to add this plugin 
because it will have some overheads, but I believe it would be good for the 
builds if the test failures could be fixed.

2) Root cause of test failures:
[testSerializerTree|#L35] calls method getSerializerTree(Line 43) to return a 
string, and then assert it equals to an expected one. In line 291 of 
[getSerializerTree|#L270], it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
[documentation|[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]]).
 When iterating the array, the elements are inserted to a string and returned. 
As a result, testSerializerTree asserts a non-deterministic string equals a 
deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!


was (Author: JIRAUSER302797):
Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening. It's fine not to add this plugin 
because it will have some overheads, but I believe it would be good for the 
builds if the test failures could be fixed.

2) Root cause of test failures:
[testSerializerTree|#L35] calls method getSerializerTree(Line 43) to return a 
string, and then assert it equals to an expected one. In line 291 of 
[getSerializerTree|#L270]], it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
[documentation|[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]]).
 When iterating the array, the elements are inserted to a string and returned. 
As a result, testSerializerTree asserts a non-deterministic string equals a 
deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!

> Fix Test Failures
> -
>
> Key: FLINK-33430
> URL: https://issues.apache.org/jira/browse/FLINK-33430
> Project: Flink
>  Issue Type: Bug
>Reporter: dserfe
>Priority: Minor
>  Labels: pull-request-available
>
> This issue is to fix test failures caused by flakiness.
> Steps to reproduce test failures:
> 1)Run the following commands
> {code:bash}
> mvn -pl flink-tests edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.test.misc.GenericTypeInfoTest#testSerializerTree
> {code}
>  
> 2) Then we found the following test failures:
> {code:bash}
> [ERROR] GenericTypeInfoTest.testSerializerTree:53 Expected: "GenericTypeInfo 
> (PojoWithCollectionGeneric)\n pojos:java.util.List\n key:int\n 
> sqlDate:java.sql.Date\n bigInt:java.math.BigInteger\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n scalaBigInt:scala.math.BigInt\n bigInteger:java.math.BigInteger\n 
> mixed:java.util.List\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  group:java.lang.String\n date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n" but: was "GenericTypeInfo (PojoWithCollectionGeneric)\n key:int\n 
> bigDecimalKeepItNul

[jira] [Comment Edited] (FLINK-33430) Fix Test Failures

[dserfe (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782344#comment-17782344
 ] 

dserfe edited comment on FLINK-33430 at 11/2/23 10:49 PM:
--

Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening. It's fine not to add this plugin 
because it will have some overheads, but I believe it would be good for the 
builds if the test failures could be fixed.

2) Root cause of test failures:
[testSerializerTree|#L35] calls method getSerializerTree(Line 43) to return a 
string, and then assert it equals to an expected one. In line 291 of 
[getSerializerTree|#L270]], it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
[documentation|[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]]).
 When iterating the array, the elements are inserted to a string and returned. 
As a result, testSerializerTree asserts a non-deterministic string equals a 
deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!


was (Author: JIRAUSER302797):
Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening. It's fine not to add this plugin 
because it will have some overheads, but I believe it would be good for the 
builds if the test failures could be fixed.

2) Root cause of test failures:
[testSerializerTree|#L35]] calls method getSerializerTree(Line 43) to return a 
string, and then assert it equals to an expected one. In line 291 of 
[getSerializerTree|#L270]], it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
[documentation|[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]]).
 When iterating the array, the elements are inserted to a string and returned. 
As a result, testSerializerTree asserts a non-deterministic string equals a 
deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!

> Fix Test Failures
> -
>
> Key: FLINK-33430
> URL: https://issues.apache.org/jira/browse/FLINK-33430
> Project: Flink
>  Issue Type: Bug
>Reporter: dserfe
>Priority: Minor
>  Labels: pull-request-available
>
> This issue is to fix test failures caused by flakiness.
> Steps to reproduce test failures:
> 1)Run the following commands
> {code:bash}
> mvn -pl flink-tests edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.test.misc.GenericTypeInfoTest#testSerializerTree
> {code}
>  
> 2) Then we found the following test failures:
> {code:bash}
> [ERROR] GenericTypeInfoTest.testSerializerTree:53 Expected: "GenericTypeInfo 
> (PojoWithCollectionGeneric)\n pojos:java.util.List\n key:int\n 
> sqlDate:java.sql.Date\n bigInt:java.math.BigInteger\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n scalaBigInt:scala.math.BigInt\n bigInteger:java.math.BigInteger\n 
> mixed:java.util.List\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  group:java.lang.String\n date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n" but: was "GenericTypeInfo (PojoWithCollectionGeneric)\n key:int\n 
> bigDecimalKeepItN

[jira] [Comment Edited] (FLINK-33430) Fix Test Failures

[dserfe (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782344#comment-17782344
 ] 

dserfe edited comment on FLINK-33430 at 11/2/23 10:47 PM:
--

Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening. It's fine not to add this plugin 
because it will have some overheads, but I believe it would be good for the 
builds if the test failures could be fixed.

2) Root cause of test failures:
[testSerializerTree|#L35]] calls method getSerializerTree(Line 43) to return a 
string, and then assert it equals to an expected one. In line 291 of 
[getSerializerTree|#L270]], it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
[documentation|[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]]).
 When iterating the array, the elements are inserted to a string and returned. 
As a result, testSerializerTree asserts a non-deterministic string equals a 
deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!


was (Author: JIRAUSER302797):
Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening.

2) Root cause of test failures:
[testSerializerTree|#L35]] calls method
getSerializerTree(Line 43) to return a string, and then assert it equals to an 
expected one. In line 291 of [getSerializerTree|#L270]], it calls another 
method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
documentation 
[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]). When 
iterating the array, the elements are inserted to a string and returned. As a 
result, testSerializerTree asserts a non-deterministic string equals
a deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!

> Fix Test Failures
> -
>
> Key: FLINK-33430
> URL: https://issues.apache.org/jira/browse/FLINK-33430
> Project: Flink
>  Issue Type: Bug
>Reporter: dserfe
>Priority: Minor
>  Labels: pull-request-available
>
> This issue is to fix test failures caused by flakiness.
> Steps to reproduce test failures:
> 1)Run the following commands
> {code:bash}
> mvn -pl flink-tests edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.test.misc.GenericTypeInfoTest#testSerializerTree
> {code}
>  
> 2) Then we found the following test failures:
> {code:bash}
> [ERROR] GenericTypeInfoTest.testSerializerTree:53 Expected: "GenericTypeInfo 
> (PojoWithCollectionGeneric)\n pojos:java.util.List\n key:int\n 
> sqlDate:java.sql.Date\n bigInt:java.math.BigInteger\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n scalaBigInt:scala.math.BigInt\n bigInteger:java.math.BigInteger\n 
> mixed:java.util.List\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  group:java.lang.String\n date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n" but: was "GenericTypeInfo (PojoWithCollectionGeneric)\n key:int\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n bigInt:java.math.BigInteger\n sqlDate:java.sql.Date\n 
> pojos:java.util.List\n scal

[jira] [Comment Edited] (FLINK-33430) Fix Test Failures

[dserfe (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782344#comment-17782344
 ] 

dserfe edited comment on FLINK-33430 at 11/2/23 10:44 PM:
--

Hi Matthias, thanks for your comments!

To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening.

2) Root cause of test failures:
[testSerializerTree|#L35]] calls method
getSerializerTree(Line 43) to return a string, and then assert it equals to an 
expected one. In line 291 of [getSerializerTree|#L270]], it calls another 
method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
documentation 
[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]). When 
iterating the array, the elements are inserted to a string and returned. As a 
result, testSerializerTree asserts a non-deterministic string equals
a deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!


was (Author: JIRAUSER302797):
Hi Matthias, thanks for your comments!
To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening.

2) Root cause of test failures:
[testSerializerTree|[https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/misc/GenericTypeInfoTest.java#L35]]
 calls method
getSerializerTree(Line 43) to return a string, and then assert it equals to an 
expected one. In line 291 of 
[getSerializerTree|[https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L270]],
 it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
documentation 
[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]). When 
iterating the array, the elements are inserted to a string and returned. As a 
result, testSerializerTree asserts a non-deterministic string equals
a deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!

> Fix Test Failures
> -
>
> Key: FLINK-33430
> URL: https://issues.apache.org/jira/browse/FLINK-33430
> Project: Flink
>  Issue Type: Bug
>Reporter: dserfe
>Priority: Minor
>  Labels: pull-request-available
>
> This issue is to fix test failures caused by flakiness.
> Steps to reproduce test failures:
> 1)Run the following commands
> {code:bash}
> mvn -pl flink-tests edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.test.misc.GenericTypeInfoTest#testSerializerTree
> {code}
>  
> 2) Then we found the following test failures:
> {code:bash}
> [ERROR] GenericTypeInfoTest.testSerializerTree:53 Expected: "GenericTypeInfo 
> (PojoWithCollectionGeneric)\n pojos:java.util.List\n key:int\n 
> sqlDate:java.sql.Date\n bigInt:java.math.BigInteger\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n scalaBigInt:scala.math.BigInt\n bigInteger:java.math.BigInteger\n 
> mixed:java.util.List\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  group:java.lang.String\n date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n" but: was "GenericTypeInfo (PojoWithCollectionGeneric)\n key:int\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInt

[jira] [Commented] (FLINK-33430) Fix Test Failures

[dserfe (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782344#comment-17782344
 ] 

dserfe commented on FLINK-33430:


Hi Matthias, thanks for your comments!
To clarify:
1) This PR is to fix a flaky test which can lead to potential test failures, 
the test failures may happen even when running the test suite (maybe multiple 
times) without NonDex. 
> it verifies that the code behaves in the same way among different JVM 
> implementations.  Is this correct?
Yes, NonDex is designed to detect such potential issues, so the issues can be 
fixed to avoid test failures happening.

2) Root cause of test failures:
[testSerializerTree|[https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/misc/GenericTypeInfoTest.java#L35]]
 calls method
getSerializerTree(Line 43) to return a string, and then assert it equals to an 
expected one. In line 291 of 
[getSerializerTree|[https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L270]],
 it calls another method 
getGenericTypeTree([https://github.com/dserfe/flink/blob/a4ad86fb083f90503938a9c3d816cdda9dc22427/flink-java/src/main/java/org/apache/flink/api/java/Utils.java#L301])
  that leverages API getDeclaredFields to return an array containing the names 
of Field objects reflecting all the declared fields. However, the elements in 
the array returned are not sorted and are not in any particular order (from the 
documentation 
[https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html]). When 
iterating the array, the elements are inserted to a string and returned. As a 
result, testSerializerTree asserts a non-deterministic string equals
a deterministic one, so the test will fail during some runs.

If you don't like the proposed fix, please let me know and we can discuss other 
fixes!

> Fix Test Failures
> -
>
> Key: FLINK-33430
> URL: https://issues.apache.org/jira/browse/FLINK-33430
> Project: Flink
>  Issue Type: Bug
>Reporter: dserfe
>Priority: Minor
>  Labels: pull-request-available
>
> This issue is to fix test failures caused by flakiness.
> Steps to reproduce test failures:
> 1)Run the following commands
> {code:bash}
> mvn -pl flink-tests edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.test.misc.GenericTypeInfoTest#testSerializerTree
> {code}
>  
> 2) Then we found the following test failures:
> {code:bash}
> [ERROR] GenericTypeInfoTest.testSerializerTree:53 Expected: "GenericTypeInfo 
> (PojoWithCollectionGeneric)\n pojos:java.util.List\n key:int\n 
> sqlDate:java.sql.Date\n bigInt:java.math.BigInteger\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n scalaBigInt:scala.math.BigInt\n bigInteger:java.math.BigInteger\n 
> mixed:java.util.List\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  group:java.lang.String\n date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n" but: was "GenericTypeInfo (PojoWithCollectionGeneric)\n key:int\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n bigInt:java.math.BigInteger\n sqlDate:java.sql.Date\n 
> pojos:java.util.List\n scalaBigInt:scala.math.BigInt\n 
> bigInteger:java.math.BigInteger\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n group:java.lang.String\n mixed:java.util.List\n"   {code}
> The root cause of the test failure is due to assuming a consistent order of 
> {{{}serTree{}}}, but the elements of it are actually not consistent, to fix 
> the test failure:
> _change the assertion to compare two lists, which can ignore the orders of 
> elements_



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33309] Add `-Djava.security.manager=allow` [flink]

[via GitHub]

snuyanzin commented on code in PR #23547:
URL: https://github.com/apache/flink/pull/23547#discussion_r1380841887


##
flink-dist/src/main/flink-bin/bin/config.sh:
##
@@ -334,6 +334,12 @@ if [ -z "${FLINK_ENV_JAVA_OPTS}" ]; then
 
 # Remove leading and ending double quotes (if present) of value
 FLINK_ENV_JAVA_OPTS="-XX:+IgnoreUnrecognizedVMOptions $( echo 
"${FLINK_ENV_JAVA_OPTS}" | sed -e 's/^"//'  -e 's/"$//' )"
+
+JAVA_SPEC_VERSION=`"${JAVA_RUN}" -XshowSettings:properties 2>&1 | grep 
"java.specification.version" | cut -d "=" -f 2 |  tr -d '[:space:]'`
+if [[ $(echo "$JAVA_SPEC_VERSION > 17" | bc )  == "1"  ]]; then

Review Comment:
   yes, I should have been checking that failure
the issue was with jdk 8 for which it returns `1.8`

finally i replaced `bc` with usage of substring from latest dot till end



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33442] Copy local state collection preventively [flink]

[via GitHub]

flinkbot commented on PR #23654:
URL: https://github.com/apache/flink/pull/23654#issuecomment-1791591391

   
   ## CI report:
   
   * 4dc9e7f281af5045294418f26c5d874df8729a19 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-33442) UnsupportedOperationException thrown from RocksDBIncrementalRestoreOperation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-33442:
---
Labels: pull-request-available  (was: )

> UnsupportedOperationException thrown from RocksDBIncrementalRestoreOperation
> 
>
> Key: FLINK-33442
> URL: https://issues.apache.org/jira/browse/FLINK-33442
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Affects Versions: 1.17.1
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.17.2
>
>
> When using the new rescaling API, it's possible to get
> {code:java}
> 2023-10-31 18:25:05,179 ERROR 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] - 
> Caught unexpected exception.
> java.lang.UnsupportedOperationException: null
>   at java.util.Collections$1.remove(Collections.java:4714) ~[?:?]
>   at java.util.AbstractCollection.remove(AbstractCollection.java:299) 
> ~[?:?]
>   at 
> org.apache.flink.runtime.checkpoint.StateObjectCollection.remove(StateObjectCollection.java:105)
>  ~[flink-runtime-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithRescaling(RocksDBIncrementalRestoreOperation.java:294)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:167)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:327)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:338)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:355)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:166)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:256)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:735)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:710)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:676)
>  ~[flink-dist-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
>  [flink-runtime-1.17.1-143.jar:1.17.1-143]
>   at 
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) 
> [flink-runtime-1.17.1-143.jar:1.17.1-143]
>   at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) 
> [flink-runtime-1.17.1-143.jar:1.17.1-143]
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) 
> [flink-runtime-1.17.1-143.jar:1.17.1-143]
>   at java.lang.Thread.run(Thread.java:829) [?:?]
> 2023-10-31 18:25:05,182 WARN  
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] - 
> Exception while restoring keyed state backend for 
> KeyedProcess

[jira] [Updated] (FLINK-33442) UnsupportedOperationException thrown from RocksDBIncrementalRestoreOperation

[Roman Khachatryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Khachatryan updated FLINK-33442:
--
Description: 
When using the new rescaling API, it's possible to get
{code:java}
2023-10-31 18:25:05,179 ERROR 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] - 
Caught unexpected exception.
java.lang.UnsupportedOperationException: null
at java.util.Collections$1.remove(Collections.java:4714) ~[?:?]
at java.util.AbstractCollection.remove(AbstractCollection.java:299) 
~[?:?]
at 
org.apache.flink.runtime.checkpoint.StateObjectCollection.remove(StateObjectCollection.java:105)
 ~[flink-runtime-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithRescaling(RocksDBIncrementalRestoreOperation.java:294)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:167)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:327)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:338)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:355)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:166)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:256)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:735)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:710)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:676)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
 [flink-runtime-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) 
[flink-runtime-1.17.1-143.jar:1.17.1-143]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) 
[flink-runtime-1.17.1-143.jar:1.17.1-143]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) 
[flink-runtime-1.17.1-143.jar:1.17.1-143]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-10-31 18:25:05,182 WARN  
org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] - 
Exception while restoring keyed state backend for 
KeyedProcessOperator_353a6b34b8b7f1c1d0fb4616d911049c_(1/2) from alternative 
(1/2), will retry while more alternatives are available.
org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected 
exception.
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:407)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99)
 ~[fli

[jira] [Created] (FLINK-33442) UnsupportedOperationException thrown from RocksDBIncrementalRestoreOperation

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-33442:
-

 Summary: UnsupportedOperationException thrown from 
RocksDBIncrementalRestoreOperation
 Key: FLINK-33442
 URL: https://issues.apache.org/jira/browse/FLINK-33442
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.17.1
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.17.2


When using the new rescaling API, it's possible to get
{code:java}
2023-10-31 18:25:05,179 ERROR 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] - 
Caught unexpected exception.
java.lang.UnsupportedOperationException: null
at java.util.Collections$1.remove(Collections.java:4714) ~[?:?]
at java.util.AbstractCollection.remove(AbstractCollection.java:299) 
~[?:?]
at 
org.apache.flink.runtime.checkpoint.StateObjectCollection.remove(StateObjectCollection.java:105)
 ~[flink-runtime-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithRescaling(RocksDBIncrementalRestoreOperation.java:294)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:167)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:327)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:338)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:355)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:166)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:256)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:735)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:710)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:676)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
 [flink-runtime-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) 
[flink-runtime-1.17.1-143.jar:1.17.1-143]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) 
[flink-runtime-1.17.1-143.jar:1.17.1-143]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) 
[flink-runtime-1.17.1-143.jar:1.17.1-143]
at java.lang.Thread.run(Thread.java:829) [?:?]
2023-10-31 18:25:05,182 WARN  
org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] - 
Exception while restoring keyed state backend for 
KeyedProcessOperator_353a6b34b8b7f1c1d0fb4616d911049c_(1/2) from alternative 
(1/2), will retry while more alternatives are available.
org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected 
exception.
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:407)
 ~[flink-dist-1.17.1-143.jar:1.17.1-143]
at 
org.apache.flink.

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 8:14 PM:
-

 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs to link 
to  resoucemanager. For example, the YarnClusterDescriptor#isReadyForDeployment 
method requires the configuration of yarn (yarn.resourcemanager.xxx) to link to 
RM. If yarn is not configured, errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

Can you assign this task to me? I will improve it

 


was (Author: JIRAUSER302860):
 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

Can you assign this task to me? I will improve it

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) |

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 8:12 PM:
-

 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

Can you assign this task to me? I will improve it

 


was (Author: JIRAUSER302860):
 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

Can you assign this task to me? I'll perfect it

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
>

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 8:09 PM:
-

 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

Can you assign this task to me? I'll perfect it

 


was (Author: JIRAUSER302860):
 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。。

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 8:08 PM:
-

 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

 


was (Author: JIRAUSER302860):
 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Had

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 8:06 PM:
-

 [~martijnvisser]   [~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

 


was (Author: JIRAUSER302860):
[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

 

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopC

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 8:00 PM:
-

[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.See the screenshot below：

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

yarn-site can be loaded in a special way, but it will be inconvenient. Just 
like the configuration of hdfs-site, it is better to load the configuration 
directory in advance. Then specify special configurations.

 


was (Author: JIRAUSER302860):
[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.

 

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ bo

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 7:52 PM:
-

[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.

 

!image-2023-11-03-03-45-05-640.png|width=1038,height=147!


was (Author: JIRAUSER302860):
[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.

 

!image-2023-11-03-03-45-05-640.png!

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possible

[jira] [Comment Edited] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili edited comment on FLINK-33423 at 11/2/23 7:52 PM:
-

[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? – Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.

 

!image-2023-11-03-03-45-05-640.png!


was (Author: JIRAUSER302860):
[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? -- Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, image-2023-11-03-03-45-05-640.png, 
> 微信图片_20231101151644.png, 微信图片_20231101152359.png, 微信图片_20231101152404.png, 
> 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " 

[jira] [Commented] (FLINK-33423) Resolve the problem that YarnClusterClientFactory cannot load yarn configurations

[zhengzhili (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782310#comment-17782310
 ] 

zhengzhili commented on FLINK-33423:


[~ferenc-csaky] I also think that this method (Hadooputills 
#getHadoopConfiguration) cannot load yarn-site, but new methods can be added to 
load it.
I am wondering why is this necessary though? -- Because the client needs a 
linked resoucemanager. For example, the 
YarnClusterDescriptor#isReadyForDeployment method requires the configuration of 
yarn (yarn.resourcemanager.xxx) to link to RM. If yarn is not configured, 
errors may occur.

> Resolve the problem that YarnClusterClientFactory cannot load yarn 
> configurations
> -
>
> Key: FLINK-33423
> URL: https://issues.apache.org/jira/browse/FLINK-33423
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Attachments: flinktest.png, 微信图片_20231101151644.png, 
> 微信图片_20231101152359.png, 微信图片_20231101152404.png, 微信截图_20231101152725.png
>
>
> YarnClusterClientFactory. getClusterDescriptor method   Unable to load the 
> configuration for yarn .  The reason is that it is called 
> HadoopUtils.getHadoopConfiguration and this method only loading HDFS 
> configuration.
> The call chain looks like this: 
> YarnClusterClientFactory#getClusterDescriptor-->Utils#getYarnAndHadoopConfiguration-->
>   HadoopUtils.getHadoopConfiguration --> Hadooputills#addHadoopConfIfFound 
> However, the HadoopUtils#addHadoopConfIfFound method does not load yarn 
> configuration information
> First,YarnClusterClientFactory#getClusterDescriptor This method call 
> Utils.getYarnAndHadoopConfiguration method
> {quote}private YarnClusterDescriptor getClusterDescriptor(Configuration 
> configuration)  
> Unknown macro: \{ final YarnClient yarnClient = 
> YarnClient.createYarnClient(); final YarnConfiguration yarnConfiguration = 
> Utils.getYarnAndHadoopConfiguration(configuration); 
> yarnClient.init(yarnConfiguration); yarnClient.start(); return new 
> YarnClusterDescriptor( configuration, yarnConfiguration, yarnClient, 
> YarnClientYarnClusterInformationRetriever.create(yarnClient), false); }
> {quote}
> It then calls Utils# getYarnAndHadoopConfiguration method, in the call 
> HadoopUtils# getHadoopConfiguration methods will only loading the Hadoop 
> configuration unable to load the configuration for Yarn.
> {quote}    public static YarnConfiguration getYarnAndHadoopConfiguration(
>             org.apache.flink.configuration.Configuration flinkConfig) 
> Unknown macro: \{         final YarnConfiguration yarnConfig = 
> getYarnConfiguration(flinkConfig);         
> yarnConfig.addResource(HadoopUtils.getHadoopConfiguration(flinkConfig));      
>    return yarnConfig;     }
> {quote}
> Then in HadoopUtils. GetHadoopConfiguration methods this Approach in the 3 
> will through HadoopUtils# addHadoopConfIfFound method to load the 
> configuration file
> {quote}    public static Configuration getHadoopConfiguration(
>     
>     。
>     
> // Approach 3: HADOOP_CONF_DIR environment variable
>         String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
>         if (hadoopConfDir != null) {
>             LOG.debug("Searching Hadoop configuration files in 
> HADOOP_CONF_DIR: {}", hadoopConfDir);
>             foundHadoopConfiguration =
>                     addHadoopConfIfFound(result, hadoopConfDir) || 
> foundHadoopConfiguration;
>         }
>         。
> }
> {quote}
>  
> Finally, it calls the Hadooputills#addHadoopConfIfFound  method, which loads 
> only the core-site and hdfs-site configuration but not the yarn-site 
> configuration
> {quote}private static boolean addHadoopConfIfFound(
> Configuration configuration, String possibleHadoopConfPath)
> Unknown macro: \{ boolean foundHadoopConfiguration = false; if (new 
> File(possibleHadoopConfPath).exists()) Unknown macro}
> if (new File(possibleHadoopConfPath + "/hdfs-site.xml").exists())
> Unknown macro: \{ configuration.addResource( new 
> org.apache.hadoop.fs.Path(possibleHadoopConfPath + "/hdfs-site.xml")); 
> LOG.debug( "Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop 
> configuration"); foundHadoopConfiguration = true; }
> }
> return foundHadoopConfiguration;
> }
> {quote}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33441] Implement restore tests for ExecUnion node [flink]

[via GitHub]

flinkbot commented on PR #23653:
URL: https://github.com/apache/flink/pull/23653#issuecomment-1791443918

   
   ## CI report:
   
   * 2689903dc838e6376ea80b4700cf32b8b4933f00 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-33441) Implement restore tests for ExecUnion node

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-33441:
---
Labels: pull-request-available  (was: )

> Implement restore tests for ExecUnion node
> --
>
> Key: FLINK-33441
> URL: https://issues.apache.org/jira/browse/FLINK-33441
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Bonnie Varghese
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[PR] [FLINK-33441] Implement restore tests for ExecUnion node [flink]

[via GitHub]

bvarghese1 opened a new pull request, #23653:
URL: https://github.com/apache/flink/pull/23653

   
   
   ## What is the purpose of the change
   
   Add restore tests for ExecUnion node
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
 - *Added restore tests for ExecUnion node which verifies the generated 
compiled plan with the saved compiled plan*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-33441) Implement union restore tests for ExecUnion node

[Bonnie Varghese (Jira)]

Bonnie Varghese created FLINK-33441:
---

 Summary: Implement union restore tests for ExecUnion node
 Key: FLINK-33441
 URL: https://issues.apache.org/jira/browse/FLINK-33441
 Project: Flink
  Issue Type: Sub-task
Reporter: Bonnie Varghese






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33441) Implement restore tests for ExecUnion node

[Bonnie Varghese (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bonnie Varghese updated FLINK-33441:

Summary: Implement restore tests for ExecUnion node  (was: Implement union 
restore tests for ExecUnion node)

> Implement restore tests for ExecUnion node
> --
>
> Key: FLINK-33441
> URL: https://issues.apache.org/jira/browse/FLINK-33441
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Bonnie Varghese
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

jnh5y commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380394903


##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/expressions/validation/ArrayTypeValidationTest.scala:
##
@@ -31,12 +31,6 @@ class ArrayTypeValidationTest extends ArrayTypeTestBase {
   .isThrownBy(() => testSqlApi("ARRAY['string', 12]", "FAIL"))
   }
 
-  @Test
-  def testObviousInvalidIndexTableApi(): Unit = {
-assertThatExceptionOfType(classOf[ValidationException])
-  .isThrownBy(() => testTableApi('f2.at(0), "FAIL"))

Review Comment:
   Is this tested somewhere else now?  (Not suggesting that it must be.)



##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/expressions/validation/ArrayTypeValidationTest.scala:
##
@@ -31,12 +31,6 @@ class ArrayTypeValidationTest extends ArrayTypeTestBase {
   .isThrownBy(() => testSqlApi("ARRAY['string', 12]", "FAIL"))
   }
 
-  @Test
-  def testObviousInvalidIndexTableApi(): Unit = {
-assertThatExceptionOfType(classOf[ValidationException])
-  .isThrownBy(() => testTableApi('f2.at(0), "FAIL"))

Review Comment:
   Is this tested somewhere else now?  (Not suggesting that it must be.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

dawidwys commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380400786


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/typeutils/TypeCoercion.scala:
##
@@ -53,85 +53,4 @@ object TypeCoercion {
 
   case _ => false
 }
-
-  /**
-   * All the supported cast types in flink-table.
-   *
-   * Note: No distinction between explicit and implicit conversions Note: This 
is a subset of
-   * SqlTypeAssignmentRule Note: This may lose type during the cast.
-   */
-  def canCast(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (_, VARCHAR | CHAR) => true
-
-  case (VARCHAR | CHAR, _) if isNumeric(to) => true
-  case (VARCHAR | CHAR, BOOLEAN) => true
-  case (VARCHAR | CHAR, DECIMAL) => true
-  case (VARCHAR | CHAR, DATE) => true
-  case (VARCHAR | CHAR, TIME_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITH_LOCAL_TIME_ZONE) => true
-
-  case (BOOLEAN, _) if isNumeric(to) => true
-  case (BOOLEAN, DECIMAL) => true
-  case (_, BOOLEAN) if isNumeric(from) => true
-  case (DECIMAL, BOOLEAN) => true
-
-  case (_, _) if isNumeric(from) && isNumeric(to) => true
-  case (_, DECIMAL) if isNumeric(from) => true
-  case (DECIMAL, _) if isNumeric(to) => true
-  case (DECIMAL, DECIMAL) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (TINYINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (SMALLINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (DOUBLE, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (FLOAT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-
-  case (DATE, TIME_WITHOUT_TIME_ZONE) => false
-  case (TIME_WITHOUT_TIME_ZONE, DATE) => false
-  case (_, _) if isTimePoint(from) && isTimePoint(to) => true
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, TINYINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, SMALLINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, DOUBLE) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, FLOAT) => true
-
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case _ => false
-}
-
-  /** All the supported reinterpret types in flink-table. */
-  def canReinterpret(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case (DATE, BIGINT) => true
-  case (TIME_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTERVAL_YEAR_MONTH, BIGINT) => true
-
-  case _ => false

Review Comment:
   `canCast` method I removed because it was not used
   
   Now, answering you question. The way I understand it. `REINTERPRET_CAST` is 
an internal function, that requires knowledge of the internal data formats. It 
says to change the `Logical` type without touching the underlying data format. 
It can be used if both logical types use the same physical format.
   
   `CAST` can actually modify data. E.g. you can cast an INT to STRING which 
changes the underlying format etc.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-28050][connectors] Migrate StreamExecutionEnvironment#fromElements() implementation to FLIP-27 Source API [flink]

[via GitHub]

zentol commented on code in PR #23553:
URL: https://github.com/apache/flink/pull/23553#discussion_r1380507456


##
flink-connectors/flink-connector-datagen/src/main/java/org/apache/flink/connector/datagen/functions/FromElementsGeneratorFunction.java:
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.datagen.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.java.typeutils.OutputTypeConfigurable;
+import org.apache.flink.connector.datagen.source.GeneratorFunction;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Objects;
+
+/**
+ * A stream generator function that returns a sequence of elements.
+ *
+ * This generator function serializes the elements using Flink's type 
information. That way, any
+ * object transport using Java serialization will not be affected by the 
serializability of the
+ * elements.
+ *
+ * NOTE: This source has a parallelism of 1.
+ *
+ * @param  The type of elements returned by this function.
+ */
+@Internal
+public class FromElementsGeneratorFunction
+implements GeneratorFunction, OutputTypeConfigurable {
+
+private static final long serialVersionUID = 1L;
+
+private static final Logger LOG = 
LoggerFactory.getLogger(FromElementsGeneratorFunction.class);
+
+/** The (de)serializer to be used for the data elements. */
+private @Nullable TypeSerializer serializer;
+
+/** The actual data elements, in serialized form. */
+private byte[] elementsSerialized;
+
+/** The number of elements emitted already. */
+private int numElementsEmitted;
+
+private final transient Iterable elements;
+private transient DataInputView input;
+
+public FromElementsGeneratorFunction(TypeSerializer serializer, 
OUT... elements)
+throws IOException {
+this(serializer, Arrays.asList(elements));
+}
+
+public FromElementsGeneratorFunction(TypeSerializer serializer, 
Iterable elements)
+throws IOException {
+this.serializer = Preconditions.checkNotNull(serializer);
+this.elements = elements;
+serializeElements();
+}
+
+@SafeVarargs
+public FromElementsGeneratorFunction(OUT... elements) {
+this(Arrays.asList(elements));
+}
+
+public FromElementsGeneratorFunction(Iterable elements) {
+this.serializer = null;
+this.elements = elements;
+checkIterable(elements, Object.class);
+}
+
+@VisibleForTesting
+@Nullable
+public TypeSerializer getSerializer() {
+return serializer;
+}
+
+private void serializeElements() throws IOException {
+Preconditions.checkState(serializer != null, "serializer not set");
+LOG.info("Serializing elements using  " + serializer);
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+DataOutputViewStreamWrapper wrapper = new 
DataOutputViewStreamWrapper(baos);
+
+try {
+for (OUT element : elements) {
+serializer.serialize(element, wrapper);
+}
+} catch (Exception e) {
+throw new IOException("Serializing the source elements failed: " + 
e.getMessage(), e);
+}
+this.elementsSerialized = baos.toByteArray();
+}
+
+@Override
+public void open(SourceReaderContext readerContext) throws Exception {
+ByteArrayInputStre

Re: [PR] [FLINK-33309] Add `-Djava.security.manager=allow` [flink]

[via GitHub]

zentol commented on code in PR #23547:
URL: https://github.com/apache/flink/pull/23547#discussion_r1380568055


##
flink-dist/src/main/flink-bin/bin/config.sh:
##
@@ -334,6 +334,12 @@ if [ -z "${FLINK_ENV_JAVA_OPTS}" ]; then
 
 # Remove leading and ending double quotes (if present) of value
 FLINK_ENV_JAVA_OPTS="-XX:+IgnoreUnrecognizedVMOptions $( echo 
"${FLINK_ENV_JAVA_OPTS}" | sed -e 's/^"//'  -e 's/"$//' )"
+
+JAVA_SPEC_VERSION=`"${JAVA_RUN}" -XshowSettings:properties 2>&1 | grep 
"java.specification.version" | cut -d "=" -f 2 |  tr -d '[:space:]'`
+if [[ $(echo "$JAVA_SPEC_VERSION > 17" | bc )  == "1"  ]]; then

Review Comment:
   bc isn't available in our CI image. any reason this cant use bash arithmetic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

dawidwys commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380484716


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/typeutils/TypeCoercion.scala:
##
@@ -53,85 +53,4 @@ object TypeCoercion {
 
   case _ => false
 }
-
-  /**
-   * All the supported cast types in flink-table.
-   *
-   * Note: No distinction between explicit and implicit conversions Note: This 
is a subset of
-   * SqlTypeAssignmentRule Note: This may lose type during the cast.
-   */
-  def canCast(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (_, VARCHAR | CHAR) => true
-
-  case (VARCHAR | CHAR, _) if isNumeric(to) => true
-  case (VARCHAR | CHAR, BOOLEAN) => true
-  case (VARCHAR | CHAR, DECIMAL) => true
-  case (VARCHAR | CHAR, DATE) => true
-  case (VARCHAR | CHAR, TIME_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITH_LOCAL_TIME_ZONE) => true
-
-  case (BOOLEAN, _) if isNumeric(to) => true
-  case (BOOLEAN, DECIMAL) => true
-  case (_, BOOLEAN) if isNumeric(from) => true
-  case (DECIMAL, BOOLEAN) => true
-
-  case (_, _) if isNumeric(from) && isNumeric(to) => true
-  case (_, DECIMAL) if isNumeric(from) => true
-  case (DECIMAL, _) if isNumeric(to) => true
-  case (DECIMAL, DECIMAL) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (TINYINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (SMALLINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (DOUBLE, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (FLOAT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-
-  case (DATE, TIME_WITHOUT_TIME_ZONE) => false
-  case (TIME_WITHOUT_TIME_ZONE, DATE) => false
-  case (_, _) if isTimePoint(from) && isTimePoint(to) => true
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, TINYINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, SMALLINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, DOUBLE) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, FLOAT) => true
-
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case _ => false
-}
-
-  /** All the supported reinterpret types in flink-table. */
-  def canReinterpret(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case (DATE, BIGINT) => true
-  case (TIME_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTERVAL_YEAR_MONTH, BIGINT) => true
-
-  case _ => false

Review Comment:
   No
   * `tinyint` = `byte`
   * `smallint` = `short`
   
   No other types have that physical format.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-33427) Mark new relocated autoscaler configs IGNORE in the operator

[Jira]

 [ 
https://issues.apache.org/jira/browse/FLINK-33427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi closed FLINK-33427.
--
Resolution: Fixed

[{{81a2d99}}|https://github.com/apache/flink-kubernetes-operator/commit/81a2d993dffb6b193a582fcc0f08b28e5bb1073d]
 in main

> Mark new relocated autoscaler configs IGNORE in the operator
> 
>
> Key: FLINK-33427
> URL: https://issues.apache.org/jira/browse/FLINK-33427
> Project: Flink
>  Issue Type: Bug
>  Components: Kubernetes Operator
>Reporter: Gyula Fora
>Assignee: Gyula Fora
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: kubernetes-operator-1.7.0
>
>
> The operator currently only ignores "kubrernetes.operator" prefixed configs 
> to not trigger upgrades. Autoscaler configs should also fall in this category.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33309] Add `-Djava.security.manager=allow` [flink]

[via GitHub]

zentol commented on code in PR #23547:
URL: https://github.com/apache/flink/pull/23547#discussion_r1380565691


##
flink-core/pom.xml:
##
@@ -34,7 +34,14 @@ under the License.
jar
 

-

Re: [PR] [Flink 31966] Flink Kubernetes operator lacks TLS support [flink-kubernetes-operator]

[via GitHub]

mbalassi commented on PR #689:
URL: 
https://github.com/apache/flink-kubernetes-operator/pull/689#issuecomment-1791276327

   Hi @tagarr! Thanks for looking into this. Many Flink committers are 
traveling this week and next week due to Flink Forward Seattle happening. I 
will ask @gaborgsomogyi to review this for you when he is back from vacation 
next week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33427] Handle new autoscaler config keys like operator configs [flink-kubernetes-operator]

[via GitHub]

mbalassi merged PR #700:
URL: https://github.com/apache/flink-kubernetes-operator/pull/700


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-28050][connectors] Migrate StreamExecutionEnvironment#fromElements() implementation to FLIP-27 Source API [flink]

[via GitHub]

zentol commented on code in PR #23553:
URL: https://github.com/apache/flink/pull/23553#discussion_r1380547823


##
flink-connectors/flink-connector-datagen/src/main/java/org/apache/flink/connector/datagen/functions/FromElementsGeneratorFunction.java:
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.datagen.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.java.typeutils.OutputTypeConfigurable;
+import org.apache.flink.connector.datagen.source.GeneratorFunction;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Objects;
+
+/**
+ * A stream generator function that returns a sequence of elements.
+ *
+ * This generator function serializes the elements using Flink's type 
information. That way, any
+ * object transport using Java serialization will not be affected by the 
serializability of the
+ * elements.
+ *
+ * NOTE: This source has a parallelism of 1.
+ *
+ * @param  The type of elements returned by this function.
+ */
+@Internal
+public class FromElementsGeneratorFunction
+implements GeneratorFunction, OutputTypeConfigurable {
+
+private static final long serialVersionUID = 1L;
+
+private static final Logger LOG = 
LoggerFactory.getLogger(FromElementsGeneratorFunction.class);
+
+/** The (de)serializer to be used for the data elements. */
+private @Nullable TypeSerializer serializer;
+
+/** The actual data elements, in serialized form. */
+private byte[] elementsSerialized;
+
+/** The number of elements emitted already. */
+private int numElementsEmitted;
+
+private final transient Iterable elements;
+private transient DataInputView input;
+
+public FromElementsGeneratorFunction(TypeSerializer serializer, 
OUT... elements)
+throws IOException {
+this(serializer, Arrays.asList(elements));
+}
+
+public FromElementsGeneratorFunction(TypeSerializer serializer, 
Iterable elements)
+throws IOException {
+this.serializer = Preconditions.checkNotNull(serializer);
+this.elements = elements;
+serializeElements();
+}
+
+@SafeVarargs
+public FromElementsGeneratorFunction(OUT... elements) {
+this(Arrays.asList(elements));
+}
+
+public FromElementsGeneratorFunction(Iterable elements) {
+this.serializer = null;
+this.elements = elements;
+checkIterable(elements, Object.class);
+}
+
+@VisibleForTesting
+@Nullable
+public TypeSerializer getSerializer() {
+return serializer;
+}
+
+private void serializeElements() throws IOException {
+Preconditions.checkState(serializer != null, "serializer not set");
+LOG.info("Serializing elements using  " + serializer);
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+DataOutputViewStreamWrapper wrapper = new 
DataOutputViewStreamWrapper(baos);
+
+try {
+for (OUT element : elements) {
+serializer.serialize(element, wrapper);
+}
+} catch (Exception e) {
+throw new IOException("Serializing the source elements failed: " + 
e.getMessage(), e);
+}
+this.elementsSerialized = baos.toByteArray();
+}
+
+@Override
+public void open(SourceReaderContext readerContext) throws Exception {
+ByteArrayInputStre

Re: [PR] [FLINK-33309] Add `-Djava.security.manager=allow` [flink]

[via GitHub]

zentol commented on code in PR #23547:
URL: https://github.com/apache/flink/pull/23547#discussion_r1380568055


##
flink-dist/src/main/flink-bin/bin/config.sh:
##
@@ -334,6 +334,12 @@ if [ -z "${FLINK_ENV_JAVA_OPTS}" ]; then
 
 # Remove leading and ending double quotes (if present) of value
 FLINK_ENV_JAVA_OPTS="-XX:+IgnoreUnrecognizedVMOptions $( echo 
"${FLINK_ENV_JAVA_OPTS}" | sed -e 's/^"//'  -e 's/"$//' )"
+
+JAVA_SPEC_VERSION=`"${JAVA_RUN}" -XshowSettings:properties 2>&1 | grep 
"java.specification.version" | cut -d "=" -f 2 |  tr -d '[:space:]'`
+if [[ $(echo "$JAVA_SPEC_VERSION > 17" | bc )  == "1"  ]]; then

Review Comment:
   bc isn't available in our CI image. any reason this cant use plain bash 
arithmetic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33419] Port PROCTIME/ROWTIME functions to the new inference stack [flink]

[via GitHub]

jnh5y commented on code in PR #23634:
URL: https://github.com/apache/flink/pull/23634#discussion_r1380561361


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/fieldExpression.scala:
##
@@ -103,71 +103,6 @@ case class TableReference(name: String, tableOperation: 
QueryOperation)
   override def toString: String = s"$name"
 }
 
-abstract class TimeAttribute(val expression: PlannerExpression)
-  extends UnaryExpression
-  with WindowProperty {
-
-  override private[flink] def child: PlannerExpression = expression
-}
-
-case class RowtimeAttribute(expr: PlannerExpression) extends 
TimeAttribute(expr) {
-
-  override private[flink] def validateInput(): ValidationResult = {
-child match {
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isProctimeIndicatorType(tpe) =>
-ValidationFailure("A proctime window cannot provide a rowtime 
attribute.")
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isRowtimeIndicatorType(tpe) =>
-// rowtime window
-ValidationSuccess
-  case WindowReference(_, Some(tpe)) if tpe == Types.LONG || tpe == 
Types.SQL_TIMESTAMP =>
-// batch time window
-ValidationSuccess
-  case WindowReference(_, _) =>
-ValidationFailure("Reference to a rowtime or proctime window 
required.")
-  case any =>
-ValidationFailure(
-  s"The '.rowtime' expression can only be used for table definitions 
and windows, " +
-s"while [$any] was found.")
-}
-  }
-
-  override def resultType: TypeInformation[_] = {
-child match {
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isRowtimeIndicatorType(tpe) =>
-// rowtime window
-TimeIndicatorTypeInfo.ROWTIME_INDICATOR
-  case WindowReference(_, Some(tpe)) if tpe == Types.LONG || tpe == 
Types.SQL_TIMESTAMP =>
-// batch time window
-Types.SQL_TIMESTAMP

Review Comment:
   Does this test need to return an SQL_TIMESTAMP?
   
   ```
   TestSpec.forStrategy(
   "ROWTIME type strategy on long in batch 
mode",
   
SpecificInputTypeStrategies.windowTimeIndicator(
   TimestampKind.ROWTIME))
   .calledWithArgumentTypes(DataTypes.BIGINT())
   .expectArgumentTypes(DataTypes.BIGINT()));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-30481][FLIP-277] GlueCatalog Implementation [flink-connector-aws]

[via GitHub]

Samrat002 commented on PR #47:
URL: 
https://github.com/apache/flink-connector-aws/pull/47#issuecomment-1791223447

   @PrabhuJoseph please review the PR whenever time 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33419] Port PROCTIME/ROWTIME functions to the new inference stack [flink]

[via GitHub]

jnh5y commented on code in PR #23634:
URL: https://github.com/apache/flink/pull/23634#discussion_r1380561361


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/fieldExpression.scala:
##
@@ -103,71 +103,6 @@ case class TableReference(name: String, tableOperation: 
QueryOperation)
   override def toString: String = s"$name"
 }
 
-abstract class TimeAttribute(val expression: PlannerExpression)
-  extends UnaryExpression
-  with WindowProperty {
-
-  override private[flink] def child: PlannerExpression = expression
-}
-
-case class RowtimeAttribute(expr: PlannerExpression) extends 
TimeAttribute(expr) {
-
-  override private[flink] def validateInput(): ValidationResult = {
-child match {
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isProctimeIndicatorType(tpe) =>
-ValidationFailure("A proctime window cannot provide a rowtime 
attribute.")
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isRowtimeIndicatorType(tpe) =>
-// rowtime window
-ValidationSuccess
-  case WindowReference(_, Some(tpe)) if tpe == Types.LONG || tpe == 
Types.SQL_TIMESTAMP =>
-// batch time window
-ValidationSuccess
-  case WindowReference(_, _) =>
-ValidationFailure("Reference to a rowtime or proctime window 
required.")
-  case any =>
-ValidationFailure(
-  s"The '.rowtime' expression can only be used for table definitions 
and windows, " +
-s"while [$any] was found.")
-}
-  }
-
-  override def resultType: TypeInformation[_] = {
-child match {
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isRowtimeIndicatorType(tpe) =>
-// rowtime window
-TimeIndicatorTypeInfo.ROWTIME_INDICATOR
-  case WindowReference(_, Some(tpe)) if tpe == Types.LONG || tpe == 
Types.SQL_TIMESTAMP =>
-// batch time window
-Types.SQL_TIMESTAMP

Review Comment:
   Does this test need to return an SQL_TIMESTAMP:
   
   ```
   TestSpec.forStrategy(
   "ROWTIME type strategy on long in batch 
mode",
   
SpecificInputTypeStrategies.windowTimeIndicator(
   TimestampKind.ROWTIME))
   .calledWithArgumentTypes(DataTypes.BIGINT())
   .expectArgumentTypes(DataTypes.BIGINT()));
   ```



##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/fieldExpression.scala:
##
@@ -103,71 +103,6 @@ case class TableReference(name: String, tableOperation: 
QueryOperation)
   override def toString: String = s"$name"
 }
 
-abstract class TimeAttribute(val expression: PlannerExpression)
-  extends UnaryExpression
-  with WindowProperty {
-
-  override private[flink] def child: PlannerExpression = expression
-}
-
-case class RowtimeAttribute(expr: PlannerExpression) extends 
TimeAttribute(expr) {
-
-  override private[flink] def validateInput(): ValidationResult = {
-child match {
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isProctimeIndicatorType(tpe) =>
-ValidationFailure("A proctime window cannot provide a rowtime 
attribute.")
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isRowtimeIndicatorType(tpe) =>
-// rowtime window
-ValidationSuccess
-  case WindowReference(_, Some(tpe)) if tpe == Types.LONG || tpe == 
Types.SQL_TIMESTAMP =>
-// batch time window
-ValidationSuccess
-  case WindowReference(_, _) =>
-ValidationFailure("Reference to a rowtime or proctime window 
required.")
-  case any =>
-ValidationFailure(
-  s"The '.rowtime' expression can only be used for table definitions 
and windows, " +
-s"while [$any] was found.")
-}
-  }
-
-  override def resultType: TypeInformation[_] = {
-child match {
-  case WindowReference(_, Some(tpe: TypeInformation[_])) if 
isRowtimeIndicatorType(tpe) =>
-// rowtime window
-TimeIndicatorTypeInfo.ROWTIME_INDICATOR
-  case WindowReference(_, Some(tpe)) if tpe == Types.LONG || tpe == 
Types.SQL_TIMESTAMP =>
-// batch time window
-Types.SQL_TIMESTAMP

Review Comment:
   Does this test need to return an SQL_TIMESTAMP:
   
   ```
   TestSpec.forStrategy(
   "ROWTIME type strategy on long in batch 
mode",
   
SpecificInputTypeStrategies.windowTimeIndicator(
   TimestampKind.ROWTIME))
   .calledWithArgumentTypes(DataTypes.BIGINT())
   .expectArgumentTypes(DataTypes.BIGINT()));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th

Re: [PR] [FLINK-33419] Port PROCTIME/ROWTIME functions to the new inference stack [flink]

[via GitHub]

jnh5y commented on code in PR #23634:
URL: https://github.com/apache/flink/pull/23634#discussion_r1380550619


##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/WindowTimeIndictorInputTypeStrategy.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference.strategies;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.inference.ArgumentCount;
+import org.apache.flink.table.types.inference.CallContext;
+import org.apache.flink.table.types.inference.ConstantArgumentCount;
+import org.apache.flink.table.types.inference.InputTypeStrategy;
+import org.apache.flink.table.types.inference.Signature;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.table.types.logical.TimestampKind;
+import org.apache.flink.table.types.logical.utils.LogicalTypeChecks;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * An {@link InputTypeStrategy} for {@link BuiltInFunctionDefinitions#ROWTIME} 
and {@link
+ * BuiltInFunctionDefinitions#PROCTIME}.
+ */
+@Internal
+public final class WindowTimeIndictorInputTypeStrategy implements 
InputTypeStrategy {
+private final TimestampKind timestampKind;
+
+public WindowTimeIndictorInputTypeStrategy(TimestampKind timestampKind) {
+this.timestampKind = timestampKind;
+}
+
+@Override
+public ArgumentCount getArgumentCount() {
+return ConstantArgumentCount.of(1);
+}
+
+@Override
+public Optional> inferInputTypes(
+CallContext callContext, boolean throwOnFailure) {
+final LogicalType type = 
callContext.getArgumentDataTypes().get(0).getLogicalType();
+
+if (timestampKind == TimestampKind.PROCTIME && 
!LogicalTypeChecks.isTimeAttribute(type)) {
+return callContext.fail(
+throwOnFailure, "Reference to a rowtime or proctime window 
required.");
+}
+
+if (timestampKind == TimestampKind.ROWTIME && 
LogicalTypeChecks.isProctimeAttribute(type)) {
+return callContext.fail(
+throwOnFailure, "A proctime window cannot provide a 
rowtime attribute.");
+}
+
+if (!LogicalTypeChecks.canBeTimeAttributeType(type) && 
!type.is(LogicalTypeRoot.BIGINT)) {
+return callContext.fail(
+throwOnFailure, "Reference to a rowtime or proctime window 
required.");

Review Comment:
   Do we have a test case which covers this branch?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33419] Port PROCTIME/ROWTIME functions to the new inference stack [flink]

[via GitHub]

jnh5y commented on code in PR #23634:
URL: https://github.com/apache/flink/pull/23634#discussion_r1380547338


##
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/InputTypeStrategiesTest.java:
##
@@ -636,7 +639,45 @@ ANY, explicit(DataTypes.INT())
 .expectSignature("f(, )")
 .expectArgumentTypes(
 
DataTypes.ARRAY(DataTypes.INT().notNull()).notNull(),
-DataTypes.INT()));
+DataTypes.INT()),
+TestSpec.forStrategy(
+"PROCTIME type strategy",
+
SpecificInputTypeStrategies.windowTimeIndicator(
+TimestampKind.PROCTIME))
+
.calledWithArgumentTypes(timeIndicatorType(TimestampKind.PROCTIME))
+.expectSignature("f()")
+
.expectArgumentTypes(timeIndicatorType(TimestampKind.PROCTIME)),
+TestSpec.forStrategy(
+"PROCTIME type strategy on non time indicator",
+
SpecificInputTypeStrategies.windowTimeIndicator(
+TimestampKind.PROCTIME))
+.calledWithArgumentTypes(DataTypes.BIGINT())
+.expectErrorMessage("Reference to a rowtime or 
proctime window required."),
+TestSpec.forStrategy(
+"ROWTIME type strategy",
+
SpecificInputTypeStrategies.windowTimeIndicator(
+TimestampKind.ROWTIME))
+
.calledWithArgumentTypes(timeIndicatorType(TimestampKind.ROWTIME))
+.expectSignature("f()")
+
.expectArgumentTypes(timeIndicatorType(TimestampKind.ROWTIME)),
+TestSpec.forStrategy(
+"ROWTIME type strategy on proctime indicator",
+
SpecificInputTypeStrategies.windowTimeIndicator(
+TimestampKind.ROWTIME))
+
.calledWithArgumentTypes(timeIndicatorType(TimestampKind.PROCTIME))
+.expectErrorMessage(
+"A proctime window cannot provide a rowtime 
attribute."),

Review Comment:
   Can/should we add a test to cover the other direction?  E.g. using the 
proctime strategy on a rowtime indicator?  (Such a test passes.  I'll admit 
that I don't quite understand how to mix and match the two.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-28050][connectors] Migrate StreamExecutionEnvironment#fromElements() implementation to FLIP-27 Source API [flink]

[via GitHub]

zentol commented on code in PR #23553:
URL: https://github.com/apache/flink/pull/23553#discussion_r1380524335


##
flink-formats/flink-avro/src/test/java/org/apache/flink/table/runtime/batch/AvroTypesITCase.java:
##
@@ -240,7 +240,7 @@ public void testAvroToAvro() {
 }
 
 private DataStream testData(StreamExecutionEnvironment env) {
-return env.fromElements(USER_1, USER_2, USER_3);
+return env.fromElements(USER_1, USER_2, USER_3).setParallelism(1);

Review Comment:
   huh. These changes does make one wonder whether we shouldn't set the 
parallelism to 1 by default, but allow the user to override it 🤔 



##
flink-formats/flink-avro/src/test/java/org/apache/flink/table/runtime/batch/AvroTypesITCase.java:
##
@@ -240,7 +240,7 @@ public void testAvroToAvro() {
 }
 
 private DataStream testData(StreamExecutionEnvironment env) {
-return env.fromElements(USER_1, USER_2, USER_3);
+return env.fromElements(USER_1, USER_2, USER_3).setParallelism(1);

Review Comment:
   huh. These changes does make one wonder whether we shouldn't set the 
parallelism to 1 by default, but allow the user to override it 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-28050][connectors] Migrate StreamExecutionEnvironment#fromElements() implementation to FLIP-27 Source API [flink]

[via GitHub]

zentol commented on PR #23553:
URL: https://github.com/apache/flink/pull/23553#issuecomment-1791157635

   > What is the issue with adding the new method?
   
   By default you may not add new methods to classes because it can break 
downstream classes. Personally I'd say it's fine to do it here though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

dawidwys commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380488848


##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/SpecificTypeStrategies.java:
##
@@ -59,12 +59,9 @@ public final class SpecificTypeStrategies {
 public static final TypeStrategy ARRAY = new ArrayTypeStrategy();
 
 /** Type strategy specific for array element. */
-public static final TypeStrategy ARRAY_ELEMENT =
-callContext ->
-Optional.of(
-((CollectionDataType) 
callContext.getArgumentDataTypes().get(0))
-.getElementDataType());
+public static final TypeStrategy ARRAY_ELEMENT = new 
ArrayElementTypeStrategy();
 
+public static final TypeStrategy ITEM_AT = new ItemAtTypeStrategy();

Review Comment:
   One is `InputTypeStrategy` the other `TypeStrategy`. One is for input the 
other for output.
   
   The difference between `SpecificInputTypeStrategy` and `InputTypeStrategies` 
is that in the first we have strategies used only by a single function. The 
other reusable strategies.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-33411) Implement type inference for window properties functions

[Dawid Wysakowicz (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Wysakowicz reassigned FLINK-33411:


Assignee: Zhenqiu Huang  (was: Peter Huang)

> Implement type inference for window properties functions
> 
>
> Key: FLINK-33411
> URL: https://issues.apache.org/jira/browse/FLINK-33411
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Reporter: Dawid Wysakowicz
>Assignee: Zhenqiu Huang
>Priority: Major
> Fix For: 1.19.0
>
>
> https://github.com/apache/flink/blob/91d81c427aa6312841ca868d54e8ce6ea721cd60/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/windowProperties.scala
> Functions:
> * WINDOW_START
> * WINDOW_END



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

dawidwys commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380498075


##
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/TypeStrategiesTestBase.java:
##
@@ -154,6 +160,11 @@ public TestSpec expectErrorMessage(String 
expectedErrorMessage) {
 return this;
 }
 
+public TestSpec compareConversionClass() {

Review Comment:
   Yes, I added this to the test framework in case anyone would like to 
actually compare based on the old behaviour. Not sure, what could be the use 
case, but wanted to leave that possibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

dawidwys commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380485817


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/typeutils/TypeCoercion.scala:
##
@@ -53,85 +53,4 @@ object TypeCoercion {
 
   case _ => false
 }
-
-  /**
-   * All the supported cast types in flink-table.
-   *
-   * Note: No distinction between explicit and implicit conversions Note: This 
is a subset of
-   * SqlTypeAssignmentRule Note: This may lose type during the cast.
-   */
-  def canCast(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (_, VARCHAR | CHAR) => true
-
-  case (VARCHAR | CHAR, _) if isNumeric(to) => true
-  case (VARCHAR | CHAR, BOOLEAN) => true
-  case (VARCHAR | CHAR, DECIMAL) => true
-  case (VARCHAR | CHAR, DATE) => true
-  case (VARCHAR | CHAR, TIME_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITH_LOCAL_TIME_ZONE) => true
-
-  case (BOOLEAN, _) if isNumeric(to) => true
-  case (BOOLEAN, DECIMAL) => true
-  case (_, BOOLEAN) if isNumeric(from) => true
-  case (DECIMAL, BOOLEAN) => true
-
-  case (_, _) if isNumeric(from) && isNumeric(to) => true
-  case (_, DECIMAL) if isNumeric(from) => true
-  case (DECIMAL, _) if isNumeric(to) => true
-  case (DECIMAL, DECIMAL) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (TINYINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (SMALLINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (DOUBLE, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (FLOAT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-
-  case (DATE, TIME_WITHOUT_TIME_ZONE) => false
-  case (TIME_WITHOUT_TIME_ZONE, DATE) => false
-  case (_, _) if isTimePoint(from) && isTimePoint(to) => true
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, TINYINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, SMALLINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, DOUBLE) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, FLOAT) => true
-
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case _ => false
-}
-
-  /** All the supported reinterpret types in flink-table. */
-  def canReinterpret(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case (DATE, BIGINT) => true
-  case (TIME_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTERVAL_YEAR_MONTH, BIGINT) => true
-
-  case _ => false

Review Comment:
   btw, `canCast` was/should be replaced with `LogicalTypeCasts`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

dawidwys commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380495095


##
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/ItemAtIndexArgumentTypeStrategyTest.java:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import 
org.apache.flink.table.types.inference.strategies.ItemAtIndexArgumentTypeStrategy;
+import org.apache.flink.table.types.utils.DataTypeFactoryMock;
+
+import java.util.stream.Stream;
+
+/** Tests for {@link ItemAtIndexArgumentTypeStrategy}. */
+class ItemAtIndexArgumentTypeStrategyTest extends InputTypeStrategiesTestBase {
+
+private static final InputTypeStrategy ITEM_AT_INPUT_STRATEGY =
+BuiltInFunctionDefinitions.AT
+.getTypeInference(new DataTypeFactoryMock())
+.getInputTypeStrategy();
+
+@Override
+protected Stream testData() {
+
+return Stream.of(
+TestSpec.forStrategy("Validate integer index for an array", 
ITEM_AT_INPUT_STRATEGY)
+.calledWithArgumentTypes(
+DataTypes.ARRAY(DataTypes.STRING().notNull()),
+DataTypes.SMALLINT().notNull())
+.expectSignature(
+"f([ | ], [ | 
])")
+.expectArgumentTypes(
+DataTypes.ARRAY(DataTypes.STRING().notNull()),
+DataTypes.SMALLINT().notNull()),
+TestSpec.forStrategy(
+"Validate not an integer index for an array",
+ITEM_AT_INPUT_STRATEGY)
+.calledWithArgumentTypes(
+DataTypes.ARRAY(DataTypes.STRING().notNull()), 
DataTypes.STRING())
+.expectErrorMessage(
+"Array can be indexed only using an INTEGER 
NUMERIC type."),
+TestSpec.forStrategy("Validate correct map key", 
ITEM_AT_INPUT_STRATEGY)
+.calledWithArgumentTypes(
+DataTypes.MAP(DataTypes.BIGINT(), 
DataTypes.STRING().notNull()),
+DataTypes.SMALLINT())
+.expectSignature(
+"f([ | ], [ | 
])")
+.expectArgumentTypes(
+DataTypes.MAP(DataTypes.BIGINT(), 
DataTypes.STRING().notNull()),
+DataTypes.BIGINT()),
+TestSpec.forStrategy("Validate incorrect map key", 
ITEM_AT_INPUT_STRATEGY)

Review Comment:
   `"Validate correct map key"` has that already.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

dawidwys commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380488848


##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/SpecificTypeStrategies.java:
##
@@ -59,12 +59,9 @@ public final class SpecificTypeStrategies {
 public static final TypeStrategy ARRAY = new ArrayTypeStrategy();
 
 /** Type strategy specific for array element. */
-public static final TypeStrategy ARRAY_ELEMENT =
-callContext ->
-Optional.of(
-((CollectionDataType) 
callContext.getArgumentDataTypes().get(0))
-.getElementDataType());
+public static final TypeStrategy ARRAY_ELEMENT = new 
ArrayElementTypeStrategy();
 
+public static final TypeStrategy ITEM_AT = new ItemAtTypeStrategy();

Review Comment:
   One is `InputTypeStrategy` the other `TypeStrategy`. One is for input the 
other for output.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

jnh5y commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380448415


##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java:
##
@@ -288,6 +288,50 @@ public static boolean supportsExplicitCast(LogicalType 
sourceType, LogicalType t
 return supportsCasting(sourceType, targetType, true);
 }
 
+/**
+ * Returns whether the source type can be reinterpreted as the target type.
+ *
+ * Reinterpret casts correspond to the SQL reinterpret_cast and 
represent the logic behind a
+ * {@code REINTERPRET_CAST(sourceType AS targetType)} operation.
+ */
+public static boolean supportsReinterpretCast(LogicalType sourceType, 
LogicalType targetType) {
+if (sourceType.getTypeRoot() == targetType.getTypeRoot()) {
+return true;
+}
+
+switch (targetType.getTypeRoot()) {
+case INTEGER:
+switch (sourceType.getTypeRoot()) {
+case DATE:
+case TIME_WITHOUT_TIME_ZONE:
+case INTERVAL_YEAR_MONTH:
+return true;
+default:
+return false;
+}
+case BIGINT:
+switch (sourceType.getTypeRoot()) {
+case DATE:
+case TIME_WITHOUT_TIME_ZONE:
+case TIMESTAMP_WITHOUT_TIME_ZONE:
+case INTERVAL_DAY_TIME:
+case INTERVAL_YEAR_MONTH:
+return true;
+default:
+return false;
+}
+case DATE:
+case TIME_WITHOUT_TIME_ZONE:
+case INTERVAL_YEAR_MONTH:
+return sourceType.getTypeRoot() == INTEGER;

Review Comment:
   >The switch is on the target type.
   Doh, I was reading too quickly.  I'll say that's slightly surprising.  It 
feels more natural to match on the first type first.  (IMO.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

jnh5y commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380400086


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/collection.scala:
##
@@ -1,105 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.flink.table.planner.expressions
-
-import org.apache.flink.api.common.typeinfo.{BasicArrayTypeInfo, 
BasicTypeInfo, PrimitiveArrayTypeInfo, TypeInformation}
-import org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO
-import org.apache.flink.api.java.typeutils.{MapTypeInfo, ObjectArrayTypeInfo, 
RowTypeInfo}
-import org.apache.flink.table.planner.typeutils.TypeInfoCheckUtils.{isArray, 
isMap}
-import org.apache.flink.table.planner.validate.{ValidationFailure, 
ValidationResult, ValidationSuccess}
-
-case class ArrayElement(array: PlannerExpression) extends PlannerExpression {
-
-  override private[flink] def children: Seq[PlannerExpression] = Seq(array)
-
-  override def toString = s"($array).element()"
-
-  override private[flink] def resultType = array.resultType match {
-case oati: ObjectArrayTypeInfo[_, _] => oati.getComponentInfo
-case bati: BasicArrayTypeInfo[_, _] => bati.getComponentInfo
-case pati: PrimitiveArrayTypeInfo[_] => pati.getComponentType
-  }
-
-  override private[flink] def validateInput(): ValidationResult = {
-array.resultType match {
-  case ati: TypeInformation[_] if isArray(ati) => ValidationSuccess
-  case other @ _ => ValidationFailure(s"Array expected but was '$other'.")
-}
-  }
-}
-
-case class Cardinality(container: PlannerExpression) extends PlannerExpression 
{
-
-  override private[flink] def children: Seq[PlannerExpression] = Seq(container)
-
-  override def toString = s"($container).cardinality()"
-
-  override private[flink] def resultType = BasicTypeInfo.INT_TYPE_INFO
-
-  override private[flink] def validateInput(): ValidationResult = {
-container.resultType match {
-  case mti: TypeInformation[_] if isMap(mti) => ValidationSuccess
-  case ati: TypeInformation[_] if isArray(ati) => ValidationSuccess

Review Comment:
   To check my understanding, Multisets were previously and are still work with 
cardinality, right?  (I'm asking since the Multiset type seems to be under 
"maps" here and "collections" elsewhere.  Basically, I'm just making a quick 
sanity check.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

dawidwys commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380400786


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/typeutils/TypeCoercion.scala:
##
@@ -53,85 +53,4 @@ object TypeCoercion {
 
   case _ => false
 }
-
-  /**
-   * All the supported cast types in flink-table.
-   *
-   * Note: No distinction between explicit and implicit conversions Note: This 
is a subset of
-   * SqlTypeAssignmentRule Note: This may lose type during the cast.
-   */
-  def canCast(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (_, VARCHAR | CHAR) => true
-
-  case (VARCHAR | CHAR, _) if isNumeric(to) => true
-  case (VARCHAR | CHAR, BOOLEAN) => true
-  case (VARCHAR | CHAR, DECIMAL) => true
-  case (VARCHAR | CHAR, DATE) => true
-  case (VARCHAR | CHAR, TIME_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITH_LOCAL_TIME_ZONE) => true
-
-  case (BOOLEAN, _) if isNumeric(to) => true
-  case (BOOLEAN, DECIMAL) => true
-  case (_, BOOLEAN) if isNumeric(from) => true
-  case (DECIMAL, BOOLEAN) => true
-
-  case (_, _) if isNumeric(from) && isNumeric(to) => true
-  case (_, DECIMAL) if isNumeric(from) => true
-  case (DECIMAL, _) if isNumeric(to) => true
-  case (DECIMAL, DECIMAL) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (TINYINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (SMALLINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (DOUBLE, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (FLOAT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-
-  case (DATE, TIME_WITHOUT_TIME_ZONE) => false
-  case (TIME_WITHOUT_TIME_ZONE, DATE) => false
-  case (_, _) if isTimePoint(from) && isTimePoint(to) => true
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, TINYINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, SMALLINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, DOUBLE) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, FLOAT) => true
-
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case _ => false
-}
-
-  /** All the supported reinterpret types in flink-table. */
-  def canReinterpret(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case (DATE, BIGINT) => true
-  case (TIME_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTERVAL_YEAR_MONTH, BIGINT) => true
-
-  case _ => false

Review Comment:
   `canCast` method I removed because it was not used
   
   Now, answering you question. The way I understand it. `REINTERPRET_CAST` is 
an internal function, that requires knowledge of the internal data formats. It 
says to change the logical type without touching the underlying data format. It 
can be used if both logical types use the same physical format.
   
   `CAST` can actually modify data. E.g. you can cast an INT to STRING which 
changes the underlying format etc.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-33430) Fix Test Failures

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782164#comment-17782164
 ] 

Matthias Pohl edited comment on FLINK-33430 at 11/2/23 4:33 PM:


Thanks for bringing this up, [~dserfe]. I don't fully understand the intention 
of this Jira issue, though. We're not using 
{{edu.illinois:nondex-maven-plugin:2.1.1:nondex}} as part of our build system. 
Therefore, the test instability your mentioning isn't present. Looking through 
the code, it appears that the production code is also deterministically 
generating the String (I haven't checked the entire code, though, tbh).

Based on what I found about the {{edu.illinois:nondex-maven-plugin}} (from 
[this Github source|https://github.com/TestingResearchIllinois/NonDex]), it 
verifies that the code behaves in the same way among different JVM 
implementations. Is this correct?

If you think that it makes sense to add this plugin to Flink's build pipeline 
we should actually approach this on a more systematic level: 
1. Discussing why this plugin should be added in the first place (e.g. in the 
dev ML). Where are the benefits to the Flink users?
2. If the community agrees we should add the plugin to the build pipeline and 
fix all tests in one go.

WDYT?


was (Author: mapohl):
Thanks for bringing this up, [~dserfe]. I don't fully understand the intention 
of this Jira issue, though. We're not using 
{{edu.illinois:nondex-maven-plugin:2.1.1:nondex}} as part of our build system. 
Therefore, the test instability your mentioning isn't present. Looking through 
the code, it appears that the production code is also deterministically 
generating the String (I haven't checked the entire code, though, tbh).

Based on what I found about the {{edu.illinois:nondex-maven-plugin}} (from 
[this Github source|https://github.com/TestingResearchIllinois/NonDex]), it 
verifies that the code behaves in the same way among different JVM 
implementations. Is this correct?

If you think that it makes sense to add this plugin to Flink's build pipeline 
we should actually approach this on a more systematic level: 
1. Discussing why this plugin should be added in the first place (e.g. in the 
dev ML). Where are the benefits to the Flink users?
2. If the community agrees we should add the plugin to the build pipeline and 
fix all tests in one go.

WDYT?
Your approach of fixing a test that's not failing in our CI right now but just 
running it with a plugin

> Fix Test Failures
> -
>
> Key: FLINK-33430
> URL: https://issues.apache.org/jira/browse/FLINK-33430
> Project: Flink
>  Issue Type: Bug
>Reporter: dserfe
>Priority: Minor
>  Labels: pull-request-available
>
> This issue is to fix test failures caused by flakiness.
> Steps to reproduce test failures:
> 1)Run the following commands
> {code:bash}
> mvn -pl flink-tests edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.test.misc.GenericTypeInfoTest#testSerializerTree
> {code}
>  
> 2) Then we found the following test failures:
> {code:bash}
> [ERROR] GenericTypeInfoTest.testSerializerTree:53 Expected: "GenericTypeInfo 
> (PojoWithCollectionGeneric)\n pojos:java.util.List\n key:int\n 
> sqlDate:java.sql.Date\n bigInt:java.math.BigInteger\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n scalaBigInt:scala.math.BigInt\n bigInteger:java.math.BigInteger\n 
> mixed:java.util.List\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  group:java.lang.String\n date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n" but: was "GenericTypeInfo (PojoWithCollectionGeneric)\n key:int\n 
> bigDecimalKeepItNull:java.math.BigDecimal\n intVal:java.math.BigInteger\n 
> scale:int\n bigInt:java.math.BigInteger\n sqlDate:java.sql.Date\n 
> pojos:java.util.List\n scalaBigInt:scala.math.BigInt\n 
> bigInteger:java.math.BigInteger\n 
> makeMeGeneric:org.apache.flink.test.operators.util.CollectionDataSets$PojoWithDateAndEnum\n
>  date:java.util.Date\n 
> cat:org.apache.flink.test.operators.util.CollectionDataSets$Category (is 
> enum)\n group:java.lang.String\n mixed:java.util.List\n"   {code}
> The root cause of the test failure is due to assuming a consistent order of 
> {{{}serTree{}}}, but the elements of it are actually not consistent, to fix 
> the test failure:
> _change the assertion to compare two lists, which can ignore the orders of 
> elements_



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

jnh5y commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380442730


##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/typeutils/TypeCoercion.scala:
##
@@ -53,85 +53,4 @@ object TypeCoercion {
 
   case _ => false
 }
-
-  /**
-   * All the supported cast types in flink-table.
-   *
-   * Note: No distinction between explicit and implicit conversions Note: This 
is a subset of
-   * SqlTypeAssignmentRule Note: This may lose type during the cast.
-   */
-  def canCast(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (_, VARCHAR | CHAR) => true
-
-  case (VARCHAR | CHAR, _) if isNumeric(to) => true
-  case (VARCHAR | CHAR, BOOLEAN) => true
-  case (VARCHAR | CHAR, DECIMAL) => true
-  case (VARCHAR | CHAR, DATE) => true
-  case (VARCHAR | CHAR, TIME_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (VARCHAR | CHAR, TIMESTAMP_WITH_LOCAL_TIME_ZONE) => true
-
-  case (BOOLEAN, _) if isNumeric(to) => true
-  case (BOOLEAN, DECIMAL) => true
-  case (_, BOOLEAN) if isNumeric(from) => true
-  case (DECIMAL, BOOLEAN) => true
-
-  case (_, _) if isNumeric(from) && isNumeric(to) => true
-  case (_, DECIMAL) if isNumeric(from) => true
-  case (DECIMAL, _) if isNumeric(to) => true
-  case (DECIMAL, DECIMAL) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (TINYINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (SMALLINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (DOUBLE, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (FLOAT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-
-  case (DATE, TIME_WITHOUT_TIME_ZONE) => false
-  case (TIME_WITHOUT_TIME_ZONE, DATE) => false
-  case (_, _) if isTimePoint(from) && isTimePoint(to) => true
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, TINYINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, SMALLINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, DOUBLE) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, FLOAT) => true
-
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case _ => false
-}
-
-  /** All the supported reinterpret types in flink-table. */
-  def canReinterpret(from: LogicalType, to: LogicalType): Boolean =
-(from.getTypeRoot, to.getTypeRoot) match {
-  case (_, _) if from == to => true
-
-  case (DATE, INTEGER) => true
-  case (TIME_WITHOUT_TIME_ZONE, INTEGER) => true
-  case (TIMESTAMP_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTEGER, DATE) => true
-  case (INTEGER, TIME_WITHOUT_TIME_ZONE) => true
-  case (BIGINT, TIMESTAMP_WITHOUT_TIME_ZONE) => true
-  case (INTEGER, INTERVAL_YEAR_MONTH) => true
-  case (BIGINT, INTERVAL_DAY_TIME) => true
-  case (INTERVAL_YEAR_MONTH, INTEGER) => true
-  case (INTERVAL_DAY_TIME, BIGINT) => true
-
-  case (DATE, BIGINT) => true
-  case (TIME_WITHOUT_TIME_ZONE, BIGINT) => true
-  case (INTERVAL_YEAR_MONTH, BIGINT) => true
-
-  case _ => false

Review Comment:
   Ok, ok, I'm still trying to grok a few things here.
   
   `canCast` shows that some of the time types can be cast to TINYINY and 
SMALLINT.  Should `reinterpretCast` be able to do that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33412] Implement type inference for reinterpret_cast function [flink]

[via GitHub]

dawidwys commented on code in PR #23633:
URL: https://github.com/apache/flink/pull/23633#discussion_r1380407814


##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java:
##
@@ -288,6 +288,50 @@ public static boolean supportsExplicitCast(LogicalType 
sourceType, LogicalType t
 return supportsCasting(sourceType, targetType, true);
 }
 
+/**
+ * Returns whether the source type can be reinterpreted as the target type.
+ *
+ * Reinterpret casts correspond to the SQL reinterpret_cast and 
represent the logic behind a
+ * {@code REINTERPRET_CAST(sourceType AS targetType)} operation.
+ */
+public static boolean supportsReinterpretCast(LogicalType sourceType, 
LogicalType targetType) {
+if (sourceType.getTypeRoot() == targetType.getTypeRoot()) {
+return true;
+}
+
+switch (targetType.getTypeRoot()) {
+case INTEGER:
+switch (sourceType.getTypeRoot()) {
+case DATE:
+case TIME_WITHOUT_TIME_ZONE:
+case INTERVAL_YEAR_MONTH:
+return true;
+default:
+return false;
+}
+case BIGINT:
+switch (sourceType.getTypeRoot()) {
+case DATE:
+case TIME_WITHOUT_TIME_ZONE:
+case TIMESTAMP_WITHOUT_TIME_ZONE:
+case INTERVAL_DAY_TIME:
+case INTERVAL_YEAR_MONTH:
+return true;
+default:
+return false;
+}
+case DATE:
+case TIME_WITHOUT_TIME_ZONE:
+case INTERVAL_YEAR_MONTH:
+return sourceType.getTypeRoot() == INTEGER;

Review Comment:
   They are. This branch checks what `INTEGER` can be casted to.
   
   The switch is on the target type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

jnh5y commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380423834


##
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/TypeStrategiesTestBase.java:
##
@@ -154,6 +160,11 @@ public TestSpec expectErrorMessage(String 
expectedErrorMessage) {
 return this;
 }
 
+public TestSpec compareConversionClass() {

Review Comment:
   This appears to be unused.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-18286] Implement type inference for functions on composite types [flink]

[via GitHub]

jnh5y commented on code in PR #23622:
URL: https://github.com/apache/flink/pull/23622#discussion_r1380418763


##
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/ItemAtIndexArgumentTypeStrategyTest.java:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types.inference;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import 
org.apache.flink.table.types.inference.strategies.ItemAtIndexArgumentTypeStrategy;
+import org.apache.flink.table.types.utils.DataTypeFactoryMock;
+
+import java.util.stream.Stream;
+
+/** Tests for {@link ItemAtIndexArgumentTypeStrategy}. */
+class ItemAtIndexArgumentTypeStrategyTest extends InputTypeStrategiesTestBase {
+
+private static final InputTypeStrategy ITEM_AT_INPUT_STRATEGY =
+BuiltInFunctionDefinitions.AT
+.getTypeInference(new DataTypeFactoryMock())
+.getInputTypeStrategy();
+
+@Override
+protected Stream testData() {
+
+return Stream.of(
+TestSpec.forStrategy("Validate integer index for an array", 
ITEM_AT_INPUT_STRATEGY)
+.calledWithArgumentTypes(
+DataTypes.ARRAY(DataTypes.STRING().notNull()),
+DataTypes.SMALLINT().notNull())
+.expectSignature(
+"f([ | ], [ | 
])")
+.expectArgumentTypes(
+DataTypes.ARRAY(DataTypes.STRING().notNull()),
+DataTypes.SMALLINT().notNull()),
+TestSpec.forStrategy(
+"Validate not an integer index for an array",
+ITEM_AT_INPUT_STRATEGY)
+.calledWithArgumentTypes(
+DataTypes.ARRAY(DataTypes.STRING().notNull()), 
DataTypes.STRING())
+.expectErrorMessage(
+"Array can be indexed only using an INTEGER 
NUMERIC type."),
+TestSpec.forStrategy("Validate correct map key", 
ITEM_AT_INPUT_STRATEGY)
+.calledWithArgumentTypes(
+DataTypes.MAP(DataTypes.BIGINT(), 
DataTypes.STRING().notNull()),
+DataTypes.SMALLINT())
+.expectSignature(
+"f([ | ], [ | 
])")
+.expectArgumentTypes(
+DataTypes.MAP(DataTypes.BIGINT(), 
DataTypes.STRING().notNull()),
+DataTypes.BIGINT()),
+TestSpec.forStrategy("Validate incorrect map key", 
ITEM_AT_INPUT_STRATEGY)

Review Comment:
   Is it worth adding a test which shows that implicit casts will work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org