When your code does a listStatus, you can pass a PathFilter object along that can do this filtering for you. See http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.PathFilter) for the API javadocs on that.
On Wed, May 30, 2012 at 7:46 PM, waqas latif <[email protected]> wrote: > I got the problem with I am unable to solve it. I need to apply a filter > for _SUCCESS file while using FileSystem.listStatus method. Can someone > please guide me how to filter _SUCCESS files. Thanks > > On Tue, May 29, 2012 at 1:42 PM, waqas latif <[email protected]> wrote: > >> So my question is that do hadoop 0.20 and 1.0.3 differ in their support of >> writing or reading sequencefiles? same code works fine with hadoop 0.20 but >> problem occurs when run it under hadoop 1.0.3. >> >> >> On Sun, May 27, 2012 at 6:15 PM, waqas latif <[email protected]> wrote: >> >>> But the thing is, it works with hadoop 0.20. even with 100 x100(and even >>> bigger matrices) but when it comes to hadoop 1.0.3 then even there is a >>> problem with 3x3 matrix. >>> >>> >>> On Sun, May 27, 2012 at 12:00 PM, Prashant Kommireddi < >>> [email protected]> wrote: >>> >>>> I have seen this issue with large file writes using SequenceFile writer. >>>> Not found the same issue when testing with writing fairly small files ( < >>>> 1GB). >>>> >>>> On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam >>>> <[email protected]>wrote: >>>> >>>> > Hi, >>>> > If you are using a custom writable object while passing data from the >>>> > mapper to the reducer make sure that the read fields and the write has >>>> the >>>> > same number of variables. It might be possible that you wrote datavtova >>>> > file using custom writable but later modified the custom writable (like >>>> > adding new attribute to the writable) which the old data doesn't have. >>>> > >>>> > It might be a possibility is please check once >>>> > >>>> > On Friday, May 25, 2012, waqas latif wrote: >>>> > >>>> > > Hi Experts, >>>> > > >>>> > > I am fairly new to hadoop MapR and I was trying to run a matrix >>>> > > multiplication example presented by Mr. Norstadt under following link >>>> > > http://www.norstad.org/matrix-multiply/index.html. I can run it >>>> > > successfully with hadoop 0.20.2 but I tried to run it with hadoop >>>> 1.0.3 >>>> > but >>>> > > I am getting following error. Is it the problem with my hadoop >>>> > > configuration or it is compatibility problem in the code which was >>>> > written >>>> > > in hadoop 0.20 by author.Also please guide me that how can I fix this >>>> > error >>>> > > in either case. Here is the error I am getting. >>>> > > >>>> > > in thread "main" java.io.EOFException >>>> > > at java.io.DataInputStream.readFully(DataInputStream.java:180) >>>> > > at java.io.DataInputStream.readFully(DataInputStream.java:152) >>>> > > at >>>> > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508) >>>> > > at >>>> > > >>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486) >>>> > > at >>>> > > >>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475) >>>> > > at >>>> > > >>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470) >>>> > > at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60) >>>> > > at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87) >>>> > > at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112) >>>> > > at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150) >>>> > > at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278) >>>> > > at TestMatrixMultiply.main(TestMatrixMultiply.java:308) >>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> > > at >>>> > > >>>> > > >>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>> > > at >>>> > > >>>> > > >>>> > >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>> > > at java.lang.reflect.Method.invoke(Method.java:597) >>>> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>> > > >>>> > > Thanks in advance >>>> > > >>>> > > Regards, >>>> > > waqas >>>> > > >>>> > >>>> >>> >>> >> -- Harsh J
