Hi Pavan, Thanks for getting back to me. I'm giving this a shot to simplify my example to see if I can reproduce. Any chance you could speak to why this error would keep happening? Exception in thread "main" org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: /compact/tester.orc
The root file path with full permissions is at /compact. I have tried this numerous times with random file names that do not exist and still get the same thing. Is there an explanation for this? On Fri, Apr 16, 2021 at 5:01 PM Pavan Lanka <pla...@apple.com.invalid> wrote: > Hi Ryan, > > In case you have not checked this might be a good starting point for you. > https://orc.apache.org/docs/core-java.html#simple-example < > https://orc.apache.org/docs/core-java.html#simple-example> > > When I follow the code you shared, I don’t quite follow why you are > creating and passing the LocalFileSystem object as you have in your code. > In the link you will see a very simple example on how to instantiate a > writer. The default configuration will give you a LocalFileSystem object > without you having to do any special processing. > > Most likely the error you are seeing is coming from the point that the > configuration object being set is null. > > Try the simple example I am sure that should clear things up for you. > > > On Apr 16, 2021, at 12:32 PM, Ryan Schachte <coderyanschac...@gmail.com> > wrote: > > > > Hi team. Desperate to understand what my issue is here, hoping someone > with > > better knowledge of the ORC writer and how the local hadoop fs is > working. > > > > I'm using the LocalFileSystem for my standalone app (compacts ORC files). > > Logging into the container, I see all my data written to the location I'm > > reading from, but hadoop is failing internally here. > > > > I'm using Owen's code from orc-tools to run schema inference on a given > orc > > file internally in my code and this is what causes this exception. Happy > to > > expand on this were applicable. > > > > Below is my error: > > Exception in thread "main" java.lang.NullPointerException: Cannot invoke > > "org.apache.hadoop.conf.Configuration.getLong(String, long)" because the > > return value of "org.apache.hadoop.fs.FileSystem.getConf()" is null > > at > org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(FileSystem.java:2602) > > at > org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(FileSystem.java:2613) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:989) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:658) > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1729) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:314) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:353) > > at > > > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:416) > > at > > > org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:479) > > at > > > org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:458) > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1164) > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1144) > > at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:115) > > at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:167) > > at org.apache.orc.OrcFile.createWriter(OrcFile.java:1004) > > > > Cheers! > > Ryan > >
