[jira] [Commented] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url
[ https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755404#comment-17755404 ] zheju_he commented on SPARK-44845: -- my pr https://github.com/apache/spark/pull/42529 > spark job copies jars repeatedly if fs.defaultFS and application jar are same > url > - > > Key: SPARK-44845 > URL: https://issues.apache.org/jira/browse/SPARK-44845 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 3.4.1 >Reporter: zheju_he >Priority: Major > > In the org.apache.spark.deploy.yarn.Client#compareUri method, > hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different > file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is > that in the last pr, different URIs of user information are also regarded as > different file systems. Uri.getauthority is used to determine the user > information, but authority contains the host so the URI above must be > different from authority. To determine whether the user authentication > information is different, you only need to determine URI.getUserInfo. > > the last pr and issue link: > https://issues.apache.org/jira/browse/SPARK-22587 > https://github.com/apache/spark/pull/19885 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url
zheju_he created SPARK-44845: Summary: spark job copies jars repeatedly if fs.defaultFS and application jar are same url Key: SPARK-44845 URL: https://issues.apache.org/jira/browse/SPARK-44845 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 3.4.1 Reporter: zheju_he In the org.apache.spark.deploy.yarn.Client#compareUri method, hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is that in the last pr, different URIs of user information are also regarded as different file systems. Uri.getauthority is used to determine the user information, but authority contains the host so the URI above must be different from authority. To determine whether the user authentication information is different, you only need to determine URI.getUserInfo. the last pr and issue link: https://issues.apache.org/jira/browse/SPARK-22587 https://github.com/apache/spark/pull/19885 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42088) Running python3 setup.py sdist on windows reports a permission error
[
https://issues.apache.org/jira/browse/SPARK-42088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677214#comment-17677214
]
zheju_he commented on SPARK-42088:
--
This is my pr address https://github.com/apache/spark/pull/39603
> Running python3 setup.py sdist on windows reports a permission error
>
>
> Key: SPARK-42088
> URL: https://issues.apache.org/jira/browse/SPARK-42088
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 3.4.0
>Reporter: zheju_he
>Priority: Minor
>
> My system version is windows 10, and I can run setup.py with administrator
> permissions, so there will be no error. However, it may be troublesome for us
> to upgrade permissions with Windows Server, so we need to modify the code of
> setup.py to ensure no error. To avoid the hassle of compiling for the user, I
> suggest modifying the following code to enable the out-of-the-box effect
> {code:python}
> def _supports_symlinks():
> """Check if the system supports symlinks (e.g. *nix) or not."""
> return getattr(os, "symlink", None) is not None and
> ctypes.windll.shell32.IsUserAnAdmin() != 0 if sys.platform == "win32" else
> True
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42088) Running python3 setup.py sdist on windows reports a permission error
zheju_he created SPARK-42088:
Summary: Running python3 setup.py sdist on windows reports a
permission error
Key: SPARK-42088
URL: https://issues.apache.org/jira/browse/SPARK-42088
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 3.4.0
Reporter: zheju_he
My system version is windows 10, and I can run setup.py with administrator
permissions, so there will be no error. However, it may be troublesome for us
to upgrade permissions with Windows Server, so we need to modify the code of
setup.py to ensure no error. To avoid the hassle of compiling for the user, I
suggest modifying the following code to enable the out-of-the-box effect
{code:python}
def _supports_symlinks():
"""Check if the system supports symlinks (e.g. *nix) or not."""
return getattr(os, "symlink", None) is not None and
ctypes.windll.shell32.IsUserAnAdmin() != 0 if sys.platform == "win32" else True
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
