[jira] [Commented] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url
[ https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756652#comment-17756652 ] Snoot.io commented on SPARK-44845: -- User 'zekai-li' has created a pull request for this issue: https://github.com/apache/spark/pull/42529 > spark job copies jars repeatedly if fs.defaultFS and application jar are same > url > - > > Key: SPARK-44845 > URL: https://issues.apache.org/jira/browse/SPARK-44845 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 3.4.1 >Reporter: zheju_he >Priority: Major > > In the org.apache.spark.deploy.yarn.Client#compareUri method, > hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different > file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is > that in the last pr, different URIs of user information are also regarded as > different file systems. Uri.getauthority is used to determine the user > information, but authority contains the host so the URI above must be > different from authority. To determine whether the user authentication > information is different, you only need to determine URI.getUserInfo. > > the last pr and issue link: > https://issues.apache.org/jira/browse/SPARK-22587 > https://github.com/apache/spark/pull/19885 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url
[ https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756651#comment-17756651 ] Snoot.io commented on SPARK-44845: -- User 'zekai-li' has created a pull request for this issue: https://github.com/apache/spark/pull/42529 > spark job copies jars repeatedly if fs.defaultFS and application jar are same > url > - > > Key: SPARK-44845 > URL: https://issues.apache.org/jira/browse/SPARK-44845 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 3.4.1 >Reporter: zheju_he >Priority: Major > > In the org.apache.spark.deploy.yarn.Client#compareUri method, > hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different > file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is > that in the last pr, different URIs of user information are also regarded as > different file systems. Uri.getauthority is used to determine the user > information, but authority contains the host so the URI above must be > different from authority. To determine whether the user authentication > information is different, you only need to determine URI.getUserInfo. > > the last pr and issue link: > https://issues.apache.org/jira/browse/SPARK-22587 > https://github.com/apache/spark/pull/19885 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url
[ https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17755404#comment-17755404 ] zheju_he commented on SPARK-44845: -- my pr https://github.com/apache/spark/pull/42529 > spark job copies jars repeatedly if fs.defaultFS and application jar are same > url > - > > Key: SPARK-44845 > URL: https://issues.apache.org/jira/browse/SPARK-44845 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 3.4.1 >Reporter: zheju_he >Priority: Major > > In the org.apache.spark.deploy.yarn.Client#compareUri method, > hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different > file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is > that in the last pr, different URIs of user information are also regarded as > different file systems. Uri.getauthority is used to determine the user > information, but authority contains the host so the URI above must be > different from authority. To determine whether the user authentication > information is different, you only need to determine URI.getUserInfo. > > the last pr and issue link: > https://issues.apache.org/jira/browse/SPARK-22587 > https://github.com/apache/spark/pull/19885 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
