[jira] [Created] (FLINK-33112) Support placement constraint
Junfan Zhang created FLINK-33112: Summary: Support placement constraint Key: FLINK-33112 URL: https://issues.apache.org/jira/browse/FLINK-33112 Project: Flink Issue Type: New Feature Components: Deployment / YARN Reporter: Junfan Zhang Yarn placement constraint is introduced in hadoop3.2.0 , which is useful for specify affinity or anti-affinity or colocation with K8s -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-27550) Remove checking yarn queues before submitting job to Yarn
Junfan Zhang created FLINK-27550: Summary: Remove checking yarn queues before submitting job to Yarn Key: FLINK-27550 URL: https://issues.apache.org/jira/browse/FLINK-27550 Project: Flink Issue Type: Improvement Components: Deployment / YARN Reporter: Junfan Zhang When invoking the method of checkYarnQueues in YarnClusterDescriptor, it will check the specified yarnQueue whether exists in the queues gotten by the YarnClient.QueueInfo. However when using the capacity-scheduler, the yarn queues path should be retrieved by the api of QueueInfo.getQueuePath instead of getQueueName. Due to this, it will always print out the yarn queues log, but it also can be submmitted to Yarn successfully. According to the above description, the process of checking is unnecessary. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-26697) Detailed log with flink app name
Junfan Zhang created FLINK-26697: Summary: Detailed log with flink app name Key: FLINK-26697 URL: https://issues.apache.org/jira/browse/FLINK-26697 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26696) Reuse the method of judgment
Junfan Zhang created FLINK-26696: Summary: Reuse the method of judgment Key: FLINK-26696 URL: https://issues.apache.org/jira/browse/FLINK-26696 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26329) Adjust the order of var initialization in FlinkControllerConfig
Junfan Zhang created FLINK-26329: Summary: Adjust the order of var initialization in FlinkControllerConfig Key: FLINK-26329 URL: https://issues.apache.org/jira/browse/FLINK-26329 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26324) Remove duplicate condition judgment
Junfan Zhang created FLINK-26324: Summary: Remove duplicate condition judgment Key: FLINK-26324 URL: https://issues.apache.org/jira/browse/FLINK-26324 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25536) Minor Fix: Adjust the order of variable declaration and comment in StateAssignmentOperation
Junfan Zhang created FLINK-25536: Summary: Minor Fix: Adjust the order of variable declaration and comment in StateAssignmentOperation Key: FLINK-25536 URL: https://issues.apache.org/jira/browse/FLINK-25536 Project: Flink Issue Type: Improvement Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25495) Client support attach mode when using the deployment of application mode
Junfan Zhang created FLINK-25495: Summary: Client support attach mode when using the deployment of application mode Key: FLINK-25495 URL: https://issues.apache.org/jira/browse/FLINK-25495 Project: Flink Issue Type: Improvement Components: Client / Job Submission Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
Junfan Zhang created FLINK-25410: Summary: Flink CLI should exit when app is accepted with detach mode on Yarn Key: FLINK-25410 URL: https://issues.apache.org/jira/browse/FLINK-25410 Project: Flink Issue Type: Improvement Components: Deployment / YARN Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25398) Show complete stacktrace when requesting thread dump
Junfan Zhang created FLINK-25398: Summary: Show complete stacktrace when requesting thread dump Key: FLINK-25398 URL: https://issues.apache.org/jira/browse/FLINK-25398 Project: Flink Issue Type: Improvement Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25313) Enable flink-web-ui
Junfan Zhang created FLINK-25313: Summary: Enable flink-web-ui Key: FLINK-25313 URL: https://issues.apache.org/jira/browse/FLINK-25313 Project: Flink Issue Type: Improvement Components: Documentation / Training / Exercises Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25268) Support task manager node-label in Yarn deployment
Junfan Zhang created FLINK-25268: Summary: Support task manager node-label in Yarn deployment Key: FLINK-25268 URL: https://issues.apache.org/jira/browse/FLINK-25268 Project: Flink Issue Type: Improvement Components: Deployment / YARN Reporter: Junfan Zhang Now Flink only support application level node label, it's necessary to introduce task manager level node-label on Yarn deployment. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25153) Inappropriate variable naming
Junfan Zhang created FLINK-25153: Summary: Inappropriate variable naming Key: FLINK-25153 URL: https://issues.apache.org/jira/browse/FLINK-25153 Project: Flink Issue Type: Improvement Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24963) Remove the tail separator when outputting yarn queue names
Junfan Zhang created FLINK-24963: Summary: Remove the tail separator when outputting yarn queue names Key: FLINK-24963 URL: https://issues.apache.org/jira/browse/FLINK-24963 Project: Flink Issue Type: Improvement Components: Deployment / YARN Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24555) Incorrectly put empty list to flink configuration
Junfan Zhang created FLINK-24555: Summary: Incorrectly put empty list to flink configuration Key: FLINK-24555 URL: https://issues.apache.org/jira/browse/FLINK-24555 Project: Flink Issue Type: Bug Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23991) Specifying yarn.staging-dir fail when staging scheme is different from default fs scheme
Junfan Zhang created FLINK-23991: Summary: Specifying yarn.staging-dir fail when staging scheme is different from default fs scheme Key: FLINK-23991 URL: https://issues.apache.org/jira/browse/FLINK-23991 Project: Flink Issue Type: Bug Components: Deployment / YARN Affects Versions: 1.13.2 Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23004) Fix misleading log
Junfan Zhang created FLINK-23004: Summary: Fix misleading log Key: FLINK-23004 URL: https://issues.apache.org/jira/browse/FLINK-23004 Project: Flink Issue Type: Improvement Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22919) Remove support for Hadoop1.x in HadoopInputFormatCommonBase.getCredentialsFromUGI
Junfan Zhang created FLINK-22919: Summary: Remove support for Hadoop1.x in HadoopInputFormatCommonBase.getCredentialsFromUGI Key: FLINK-22919 URL: https://issues.apache.org/jira/browse/FLINK-22919 Project: Flink Issue Type: Improvement Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22534) Set delegation token's service name as credential alias
Junfan Zhang created FLINK-22534: Summary: Set delegation token's service name as credential alias Key: FLINK-22534 URL: https://issues.apache.org/jira/browse/FLINK-22534 Project: Flink Issue Type: Improvement Components: Connectors / Hadoop Compatibility Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22485) Support client attach on application mode
Junfan Zhang created FLINK-22485: Summary: Support client attach on application mode Key: FLINK-22485 URL: https://issues.apache.org/jira/browse/FLINK-22485 Project: Flink Issue Type: Improvement Reporter: Junfan Zhang Now, client will not wait until job finish when using application mode. Can we support client attach on application mode? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22329) Missing crendentials in jobconf causes repeated authentication in Hive datasources
Junfan Zhang created FLINK-22329: Summary: Missing crendentials in jobconf causes repeated authentication in Hive datasources Key: FLINK-22329 URL: https://issues.apache.org/jira/browse/FLINK-22329 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22294) Throw IllegalArgumentException when getNumFiles with partitions on different hdfs nameservices
Junfan Zhang created FLINK-22294: Summary: Throw IllegalArgumentException when getNumFiles with partitions on different hdfs nameservices Key: FLINK-22294 URL: https://issues.apache.org/jira/browse/FLINK-22294 Project: Flink Issue Type: Bug Components: Connectors / Hive Affects Versions: 1.12.2 Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22293) Throw IllegalArgumentException when getNumFiles with partitions on different hdfs nameservices
Junfan Zhang created FLINK-22293: Summary: Throw IllegalArgumentException when getNumFiles with partitions on different hdfs nameservices Key: FLINK-22293 URL: https://issues.apache.org/jira/browse/FLINK-22293 Project: Flink Issue Type: Bug Components: Connectors / Hive Affects Versions: 1.12.2 Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21768) Optimize system.exit() logic of CliFrontend
Junfan Zhang created FLINK-21768:
Summary: Optimize system.exit() logic of CliFrontend
Key: FLINK-21768
URL: https://issues.apache.org/jira/browse/FLINK-21768
Project: Flink
Issue Type: Improvement
Components: Command Line Client
Reporter: Junfan Zhang
h2. Why
We encounter a problem when Oozie integerated with Flink Batch Action.
Oozie will use a launcher job to start Flink client used to submit Flink job to
Hadoop Yarn.
And when Flink client finished , Oozie will get its exitCode to determine job
submission status and then do some extra things.
So how Oozie catch {{System.exit()}}? It will implement JDK SecurityManager.
([Oozie related code
link|https://github.com/apache/oozie/blob/f1e01a9e155692aa5632f4573ab1b3ebeab7ef45/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/security/LauncherSecurityManager.java#L24]).
Now when Flink Client finished successfully, it will call
{{System.exit(0)}}([Flink related code
link|https://github.com/apache/flink/blob/195298aea327b3f98d9852121f0f146368696300/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L1133])
method.
And then JVM will use LauncherSecurityManager(Oozie implemented) to handle
{{System.exit(0)}} method and trigger {{LauncherSecurityManager.checkExit()}}
method, and then will throw exception.
Finally Flink Client will catch its {{throwable}} and call
{{System.exit(31)}}([related code
link|https://github.com/apache/flink/blob/195298aea327b3f98d9852121f0f146368696300/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L1139])
method again. It will cause Oozie to misjudge the status of the Fllink job.
Actually it's a corner case. In most scenes, the situation I mentioned will not
happen. But it's still necessary for us to optimize client exit logic.
Besides, i think the problem above may also exist in some other frameworks such
as linkedin/azakaban and apache/airflow, which are using Flink client to submit
batch job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21700) Allow to disable fetching Hadoop delegation token on Yarn
Junfan Zhang created FLINK-21700: Summary: Allow to disable fetching Hadoop delegation token on Yarn Key: FLINK-21700 URL: https://issues.apache.org/jira/browse/FLINK-21700 Project: Flink Issue Type: New Feature Components: Deployment / YARN Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
