[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268892#comment-16268892 ] Jakub Dubovsky commented on SPARK-22585: I have created a PR where we can agree on a code to address this. > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1626#comment-1626 ] Apache Spark commented on SPARK-22585: -- User 'james64' has created a pull request for this issue: https://github.com/apache/spark/pull/19834 > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266941#comment-16266941 ] Sean Owen commented on SPARK-22585: --- URI encoding applies to all of those URI schemes. I don't think they even differ in semantics for the URI path part. I would expect that where users supply URIs, that they are already valid URIs -- properly encoded. Where users supply a local file path, there is no reason to expect that path is URI-encoded. A path is not a URI. Hence if it is made into a URI, it needs to be encoded somehow. > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266885#comment-16266885 ] Jakub Dubovsky commented on SPARK-22585: Additional question. Do you think it is real to make this into 2.2.1 version? > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266859#comment-16266859 ] Jakub Dubovsky commented on SPARK-22585: [~srowen] I'd like to understand more about solution you prefer. So by addJar scaladoc these schemes are supported: null, file, local, hdfs, http, https, ftp. Now semantics of argument to addJar is the same as for URI single arg constructor (with exception of local scheme). This means for example that path is expected to be url encoded. Here as a fix we want to remove requirement for path to be encoded. Do we want to do this for all supported schemes named above? Or only for null, file and local? After we sync in this question I will post a code suggestion how to handle this. Discussion over code will be more fruitful I believe. > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264456#comment-16264456 ] Jakub Dubovsky commented on SPARK-22585: I just tried how URI behaves on some examples and learned that it is doing something else then I think. So yes either encoding a path or using getRawPath are both valid solutions to me. Should I create PR for this or what is next step? > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264392#comment-16264392 ] Sean Owen commented on SPARK-22585: --- Yes, I mean escaping the path only; the host and scheme and so on may use reserved characters for their intended usage. I think that's the context here where it's just paths being turned into local file URIs. A URI is the right representation in this code but needs to represent the right path. I don't see any issue with that, nor decoding. It's not round-tripping for nothing. I don't think the representation depends on usage. The URI's representation simply needs to correctly represent the resource. That's not quite happening here, and it's because special chars aren't escaped in the right places. > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264234#comment-16264234 ] Jakub Dubovsky commented on SPARK-22585: I am not sure about adding encoding step into implementation of addJar method. It's not about encoding whole path as a string since you want to keep some characters literally (':', '/' possibly others). So the code would first need to parse the path to get only path segments and encode those. This most probably leads to using URI again at which point this starts to be circular problem. Moreover I am not sure what is the point of encoding path segments only to ask URI to decode it... I also think that it makes sense to decode the segment only inside of a logic accessing a value of that segment. If I work with url/path as a whole I want to keep it parsable and therefore keep special characters encoded. This is the thinking I would personally use to decide which version (getPath/getRawPath) should be used in particular scenarios across spark code base even though I must admit I have very little insight into these other URI usecases :) > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264129#comment-16264129 ] Sean Owen commented on SPARK-22585: --- Hm, I think the issue is rather than the path needs to be encoded before becoming part of the URI. The URI {{file:/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar}} is not one that names your file; it's {{file:/home/me/.coursier/cache/v1/https/artifactory.com%253A443/path/to.jar}} (escaped %). That may be a workaround here. {{/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar}} is however a valid and correct local path to the file, and that's what the argument is meant to be. So the code should do the encoding. I'm aware that there are a number of places that probably turn paths into URIs, so would be best to try to update all issues of this form. I think it's low-risk as URI encoding won't do anything for most paths, and where it does, it's probably essential. > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263979#comment-16263979 ] Jakub Dubovsky commented on SPARK-22585: Real file path is exactly the same as one I am passing into add jar. It contains "%3A443". > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263512#comment-16263512 ] Sean Owen commented on SPARK-22585: --- Does the real file name contain "%3A443" or ":443"? > Url encoding of jar path expected? > -- > > Key: SPARK-22585 > URL: https://issues.apache.org/jira/browse/SPARK-22585 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Jakub Dubovsky > > I am calling {code}sparkContext.addJar{code} method with path to a local jar > I want to add. Example: > {code}/home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar{code}. > As a result I get an exception saying > {code} > Failed to add > /home/me/.coursier/cache/v1/https/artifactory.com%3A443/path/to.jar to Spark > environment. Stacktrace: > java.io.FileNotFoundException: Jar > /home/me/.coursier/cache/v1/https/artifactory.com:443/path/to.jar not found > {code} > Important part to notice here is that colon character is url encoded in path > I want to use but exception is complaining about path in decoded form. This > is caused by this line of code from implementation ([see > here|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L1833]): > {code} > case null | "file" => addJarFile(new File(uri.getPath)) > {code} > It uses > [getPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getPath()] > method of > [java.net.URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] > which url decodes the path. I believe method > [getRawPath|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getRawPath()] > should be used here which keeps path string in original form. > I tend to see this as a bug since I want to use my dependencies resolved from > artifactory with port directly. Is there some specific reason for this or can > we fix this? > Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org