[jira] [Commented] (SPARK-3769) SparkFiles.get gives me the wrong fully qualified path
[ https://issues.apache.org/jira/browse/SPARK-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157946#comment-14157946 ] Tom Weber commented on SPARK-3769: -- I believe I originally called it on the driver side, but the addfile call makes a local copy, so when you call it there, you get the local copy path which isn't the same path as where it ends up on the remote worker nodes. I'm good with striping the path off and only passing the file name itself to the get call. SparkFiles.get gives me the wrong fully qualified path -- Key: SPARK-3769 URL: https://issues.apache.org/jira/browse/SPARK-3769 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 1.0.2, 1.1.0 Environment: linux host, and linux grid. Reporter: Tom Weber Priority: Minor My spark pgm running on my host, (submitting work to my grid). JavaSparkContext sc =new JavaSparkContext(conf); final String path = args[1]; sc.addFile(path); /* args[1] = /opt/tom/SparkFiles.sas */ The log shows: 14/10/02 16:07:14 INFO Utils: Copying /opt/tom/SparkFiles.sas to /tmp/spark-4c661c3f-cb57-4c9f-a0e9-c2162a89db77/SparkFiles.sas 14/10/02 16:07:15 INFO SparkContext: Added file /opt/tom/SparkFiles.sas at http://10.20.xx.xx:49587/files/SparkFiles.sas with timestamp 1412280434986 those are paths on my host machine. The location that this file gets on grid nodes is: /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/SparkFiles.sas While the call to get the path in my code that runs in my mapPartitions function on the grid nodes is: String pgm = SparkFiles.get(path); And this returns the following string: /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/./opt/tom/SparkFiles.sas So, am I expected to take the qualified path that was given to me and parse it to get only the file name at the end, and then concatenate that to what I get from the SparkFiles.getRootDirectory() call in order to get this to work? Or pass only the parsed file name to the SparkFiles.get method? Seems as though I should be able to pass the same file specification to both sc.addFile() and SparkFiles.get() and get the correct location of the file. Thanks, Tom -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-3769) SparkFiles.get gives me the wrong fully qualified path
Tom Weber created SPARK-3769: Summary: SparkFiles.get gives me the wrong fully qualified path Key: SPARK-3769 URL: https://issues.apache.org/jira/browse/SPARK-3769 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 1.1.0, 1.0.2 Environment: linux host, and linux grid. Reporter: Tom Weber Priority: Minor My spark pgm running on my host, (submitting work to my grid). JavaSparkContext sc =new JavaSparkContext(conf); final String path = args[1]; sc.addFile(path); /* args[1] = /opt/tom/SparkFiles.sas */ The log shows: 14/10/02 16:07:14 INFO Utils: Copying /opt/tom/SparkFiles.sas to /tmp/spark-4c661c3f-cb57-4c9f-a0e9-c2162a89db77/SparkFiles.sas 14/10/02 16:07:15 INFO SparkContext: Added file /opt/tom/SparkFiles.sas at http://10.20.xx.xx:49587/files/SparkFiles.sas with timestamp 1412280434986 those are paths on my host machine. The location that this file gets on grid nodes is: /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/SparkFiles.sas While the call to get the path in my code that runs in my mapPartitions function on the grid nodes is: String pgm = SparkFiles.get(path); And this returns the following string: /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/./opt/tom/SparkFiles.sas So, am I expected to take the qualified path that was given to me and parse it to get only the file name at the end, and then concatenate that to what I get from the SparkFiles.getRootDirectory() call in order to get this to work? Or pass only the parsed file name to the SparkFiles.get method? Seems as though I should be able to pass the same file specification to both sc.addFile() and SparkFiles.get() and get the correct location of the file. Thanks, Tom -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3769) SparkFiles.get gives me the wrong fully qualified path
[ https://issues.apache.org/jira/browse/SPARK-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157263#comment-14157263 ] Tom Weber commented on SPARK-3769: -- Thanks for the quick turnaround! I can see that it wouldn't necessarily make sense that a fully qualified path (relative to the driver programs filesystem) would be what the .get method would take on the worker node systems. But, at the same time, the .get seems like it just takes what you give it and blindly concatenates it to the .getRootDirectory result w/out even validating it or failing if that file doesn't exist. I appreciate the File object methods for pulling the path name apart; I'll use that and that will work just fine. First time playing around with all of this, so sometimes what you expect it to do is just a matter of thinking about it a particular way :) You can close this ticket out, as I'm sure I'll be able to work fine by using the full path on the driver side and only the file name on the worker side. Seems like it might be convenient though if these matched set of routines did this themselves since the driver side needs a qualified path to find the file, and the worker side, by definition, strips that off and only put's the file in the designated work directory (which make sense of course). No big deal though. Thanks again, Tom SparkFiles.get gives me the wrong fully qualified path -- Key: SPARK-3769 URL: https://issues.apache.org/jira/browse/SPARK-3769 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 1.0.2, 1.1.0 Environment: linux host, and linux grid. Reporter: Tom Weber Priority: Minor My spark pgm running on my host, (submitting work to my grid). JavaSparkContext sc =new JavaSparkContext(conf); final String path = args[1]; sc.addFile(path); /* args[1] = /opt/tom/SparkFiles.sas */ The log shows: 14/10/02 16:07:14 INFO Utils: Copying /opt/tom/SparkFiles.sas to /tmp/spark-4c661c3f-cb57-4c9f-a0e9-c2162a89db77/SparkFiles.sas 14/10/02 16:07:15 INFO SparkContext: Added file /opt/tom/SparkFiles.sas at http://10.20.xx.xx:49587/files/SparkFiles.sas with timestamp 1412280434986 those are paths on my host machine. The location that this file gets on grid nodes is: /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/SparkFiles.sas While the call to get the path in my code that runs in my mapPartitions function on the grid nodes is: String pgm = SparkFiles.get(path); And this returns the following string: /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/./opt/tom/SparkFiles.sas So, am I expected to take the qualified path that was given to me and parse it to get only the file name at the end, and then concatenate that to what I get from the SparkFiles.getRootDirectory() call in order to get this to work? Or pass only the parsed file name to the SparkFiles.get method? Seems as though I should be able to pass the same file specification to both sc.addFile() and SparkFiles.get() and get the correct location of the file. Thanks, Tom -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
