[
https://issues.apache.org/jira/browse/MAHOUT-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044335#comment-13044335
]
Hudson commented on MAHOUT-700:
-------------------------------
Integrated in Mahout-Quality #858 (See
[https://builds.apache.org/hudson/job/Mahout-Quality/858/])
> When running 'seqdirectory' on Amazon Elastic MapReduce, FileSystem.get()
> fails due to use of FileSystem.get(conf) instead of FileSystem.get(uri, conf)
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-700
> URL: https://issues.apache.org/jira/browse/MAHOUT-700
> Project: Mahout
> Issue Type: Bug
> Components: Integration
> Environment: Amazon Elastic MapReduce
> Reporter: Dave Lewis
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 0.6
>
> Attachments: MAHOUT-700.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> After submitting a seqdirectory job to EMR:
> ./elastic-mapreduce --enable-debugging -j $JOB_NAME --jar $JAR_URL
> --main-class org.apache.mahout.driver.MahoutDriver --arg seqdirectory --arg
> --input --arg $INPUT_URL --arg --output --arg $OUTPUT_URL --arg --charset
> --arg utf-8
> when $INPUT_URL or $OUTPUT_URL are s3n:// URLs, SequenceFilesFromDirectory
> throws an exception like this:
> Exception in thread "main" java.lang.IllegalArgumentException: This file
> system object (hdfs://ip-10-84-247-151.ec2.internal:9000) does not support
> access to the request path
> 's3n://dall-emr-bucket/output/out-subjot-seqfiles/chunk-0' You possibly
> called FileSystem.get(conf) when you should have called FileSystem.get(uri,
> conf) to obtain a file system supporting your path.
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:351)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:524)
> at
> org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:871)
> at
> org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:859)
> at
> org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:851)
> at org.apache.mahout.text.ChunkedWriter.<init>(ChunkedWriter.java:47)
> at
> org.apache.mahout.text.SequenceFilesFromDirectory.run(SequenceFilesFromDirectory.java:63)
> at
> org.apache.mahout.text.SequenceFilesFromDirectory.run(SequenceFilesFromDirectory.java:106)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.mahout.text.SequenceFilesFromDirectory.main(SequenceFilesFromDirectory.java:81)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> I fixed the problem by changing FileSystem.get(conf) in
> SequenceFilesFromDirectory and ChunkedWriter to FileSystem.get(input.toUri(),
> conf) and FileSystem.get(output.toUri(), conf), respectively. I also had to
> make a couple of changes to SequenceFilesFromDirectoryFilter and
> PrefixAdditionFilter to properly use those FileSystems. I'm building with
> tests now, then I'll add the patch to this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
