[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 1:06 PM:
---
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
long bytesConsumedTotal=0L;
while(bytesConsumedTotal0)
{
//...
bytesConsumedTotal+= bytesConsumed;
}
public class ReversedLinesReader
{
public ReversedLinesReader(InputStream is)
{
//simply start reading from (positioned) is
}
public ReversedLinesReader(File file)
{
//current behaviour seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
public String readLine()
{
//current behaviour
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
long bytesConsumedTotal=0L;
while((bytesConsumed=reader.readLine(str))>0 && bytesConsumedTotal to not depend on
org.apache.hadoop.io.Text
}
public String readLine()
{
//current behaviour
}
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210186#comment-13210186
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:58 PM:
Works like a charm, thanks Georg.
I had a similar requirement. I wanted to "chunk-wise" read a HDFS file
backwards to allow file browsing similar to the Hadoop namenode's web
interface. By clicking a button a user triggers to fetch the previous N lines
starting from a specific offset.
With a few changes to your ReversedLinesFileReader implementation I was able to
implement this functionality. I would suggest to extend your
ReversedLinesFileReader to be able to operate on InputStreams and to return the
number of consumed bytes (i.e. the number of bytes actually read for "line
construction" not the number of buffered bytes). This actually results in a
Reverse org.apache.hadoop.util.LineReader.
was (Author: weihnachtsmann):
Works like a charm, thanks Georg.
I had a similar requirement. I wanted to "chunk-wise" read a HDFS file
backwards to allow file browsing similar to the Hadoop namenode's web
interface. By clicking a button a user triggers to fetch the previous N lines
starting from a specific offset.
With a few changes to your ReversedLinesFileReader implementation I was able to
implement this functionality. I would suggest to extend your
ReversedLinesFileReader to be able to operate on InputStreams and to return the
number of consumed bytes ((i.e. the number of bytes actually read for "line
construction" not the number of buffered bytes). This actually results in a
Reverse org.apache.hadoop.util.LineReader.
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:57 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
long bytesConsumedTotal=0L;
while((bytesConsumed=reader.readLine(str))>0 && bytesConsumedTotal to not depend on
org.apache.hadoop.io.Text
}
public String readLine()
{
//current behaviour
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
public ReversedLinesReader(InputStream is)
{
//simply start reading from (positioned) is
}
public ReversedLinesReader(File file)
{
//current behaviour seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
public String readLine()
{
//current behaviour
}
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:52 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
public ReversedLinesReader(InputStream is)
{
//simply start reading from (positioned) is
}
public ReversedLinesReader(File file)
{
//current behaviour seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
public String readLine()
{
//current behaviour
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//simply start reading from is
}
//current behaviour
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:50 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
//current behaviour
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
//constructor takes (positioned) Input stream
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:51 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//simply start reading from is
}
//current behaviour
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
//current behaviour
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:50 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
//constructor takes (positioned) Input stream
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:46 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:41 PM:
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
//constructor takes (positioned) Input stream
public ReversedLinesFileReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair to not depend on
org.apache.hadoop.io.Text
}
{code}
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (IO-288) Supply a ReversedLinesFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210186#comment-13210186
]
Lars Kolb edited comment on IO-288 at 2/17/12 11:20 AM:
Works like a charm, thanks Georg.
I had a similar requirement. I wanted to "chunk-wise" read a HDFS file
backwards to allow file browsing similar to the Hadoop namenode's web
interface. By clicking a button a user triggers to fetch the previous N lines
starting from a specific offset.
With a few changes to your ReversedLinesFileReader implementation I was able to
implement this functionality. I would suggest to extend your
ReversedLinesFileReader to be able to operate on InputStreams and to return the
number of consumed bytes ((i.e. the number of bytes actually read for "line
construction" not the number of buffered bytes). This actually results in a
Reverse org.apache.hadoop.util.LineReader.
was (Author: weihnachtsmann):
Works like a charm, thanks Georg.
I had a similar requirement. I wanted to "chunk-wise" read a HDFS file
backwards to allow file browsing similar to the Hadoop namenode's web
interface. By clicking a button a user triggers to fetch the previous N lines
starting from a specific offset.
With a few changes to your ReversedLinesFileReader implementation I was able to
implement this functionality. I would suggest to extend your
ReversedLinesFileReader to be able to operate on InputStreams and to return the
number of consumed bytes ((i.e. the number of bytes actually read for "line
construction" not the number of buffered)). This actually results in a Reverse
org.apache.hadoop.util.LineReader.
> Supply a ReversedLinesFileReader
> -
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
