[jira] [Issue Comment Edited] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130101#comment-13130101
]
Georg Henzler edited comment on IO-288 at 10/18/11 10:54 PM:
-
I will provide you with Unit tests if you're interested - I tested the class
with a main class and different input files but it's easy to create Unit tests
from the main class.
Regarding your questions:
- The Tailer class listens to file changes (as the unix tail does) and notifies
a provided Listener passing the added line. The ReverseFileReader starts at the
last line of a file and moves towards the start of the file (ignoring added
lines after it has been instantiated).
- We could subclass FileReader but I'm not sure how to implement e.g.
read(char[] cbuf, int off, int len)... implementing this going backward would
be hard. Mixing going forward and backward is probably not really intuitive. I
would suggest that if we implement FileReader, we throw a
UnsupportedOperationException for most of the Reader inferface's methods.
- I'm not sure of the Filename... is BufferedReverseFileReader a better name to
emphasize on the fact that it's all about the method readLine()? Any other name
suggestions?
was (Author: henzlerg):
I will provide you with Unit tests if you're interested - I tested the
class with a main class and different input files but it's easy to make Unit
tests from the main class.
Regarding your questions:
- The Tailer class listens to file changes (as the unix tail does) and notifies
a provided Listener passing the added line. The ReverseFileReader starts at the
last line of a file and moves towards the start of the file (ignoring added
lines after it has instantiated).
- We could subclass FileReader but I'm not sure how to implement e.g.
read(char[] cbuf, int off, int len)... implementing this going backward would
be hard. Mixing going forward and backward is probably not really intuitive. I
would suggest that if we implement FileReader, we throw a
UnsupportedOperationException for most of the Reader inferface's methods.
- I'm not sure of the Filename... is BufferedReverseFileReader a better name to
emphasize on the fact that it's all about the method readLine()? Any other name
suggestions?
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Attachments: ReverseFileReader.java
>
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130101#comment-13130101
]
Georg Henzler commented on IO-288:
--
I will provide you with Unit tests if you're interested - I tested the class
with a main class and different input files but it's easy to make Unit tests
from the main class.
Regarding your questions:
- The Tailer class listens to file changes (as the unix tail does) and notifies
a provided Listener passing the added line. The ReverseFileReader starts at the
last line of a file and moves towards the start of the file (ignoring added
lines after it has instantiated).
- We could subclass FileReader but I'm not sure how to implement e.g.
read(char[] cbuf, int off, int len)... implementing this going backward would
be hard. Mixing going forward and backward is probably not really intuitive. I
would suggest that if we implement FileReader, we throw a
UnsupportedOperationException for most of the Reader inferface's methods.
- I'm not sure of the Filename... is BufferedReverseFileReader a better name to
emphasize on the fact that it's all about the method readLine()? Any other name
suggestions?
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Attachments: ReverseFileReader.java
>
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Georg Henzler updated IO-288:
-
Attachment: ReverseFileReader.java
Sorry, I didn't notice the radio buttons when I first uploaded...
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Attachments: ReverseFileReader.java
>
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Georg Henzler updated IO-288:
-
Attachment: (was: ReverseFileReader.java)
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130071#comment-13130071
]
Gary D. Gregory commented on IO-288:
Hi Georg,
We cannot consider this for consideration without:
- Grant of license to Apache which is done when an attachment is uploaded.
- Unit tests.
Questions:
- How does this fit in with the Tailer class. Should one reuse the other?
- Should this be a real Reader implementation? Subclassing FileReader?
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Attachments: ReverseFileReader.java
>
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OGNL-29) Replace "getValueBody" and "setValueBody" methods with visitor pattern.
[ https://issues.apache.org/jira/browse/OGNL-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Pitts updated OGNL-29: - Attachment: Get_Set_Value_Body_visitors.patch Attaching patch which will replace getValueBody and setValueBody with visitor pattern. > Replace "getValueBody" and "setValueBody" methods with visitor pattern. > --- > > Key: OGNL-29 > URL: https://issues.apache.org/jira/browse/OGNL-29 > Project: OGNL > Issue Type: Improvement >Reporter: Daniel Pitts > Attachments: Get_Set_Value_Body_visitors.patch > > > I've made yet another patch which cleans up the AST code even further. Please > inspect carefully before committing, as this moves around a lot of the core > functionality. It may also add some overhead to some of the "setValue" > use-cases. I'm not sure if that is acceptable or not, even though the > overhead is likely to be extremely small. > Patch to come... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (OGNL-29) Replace "getValueBody" and "setValueBody" methods with visitor pattern.
Replace "getValueBody" and "setValueBody" methods with visitor pattern. --- Key: OGNL-29 URL: https://issues.apache.org/jira/browse/OGNL-29 Project: OGNL Issue Type: Improvement Reporter: Daniel Pitts Attachments: Get_Set_Value_Body_visitors.patch I've made yet another patch which cleans up the AST code even further. Please inspect carefully before committing, as this moves around a lot of the core functionality. It may also add some overhead to some of the "setValue" use-cases. I'm not sure if that is acceptable or not, even though the overhead is likely to be extremely small. Patch to come... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Georg Henzler updated IO-288:
-
Description:
I needed to analyse a log file today and I was looking for a ReverseFileReader:
A class that behaves exactly like BufferedReader except that it goes from
bottom to top when readLine() is called. I didn't find it in IOUtils and the
internet didn't help a lot either, e.g.
http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
inefficient - the log files I'm analysing are huge and it is not a good idea to
load the whole content in the memory.
So I ended up writing an implementation myself using little memory and the
class RandomAccessFile - see attached file. It's used as follows:
int blockSize = 4096; // only that much memory is needed, no matter how big the
file is
ReverseFileReader reverseFileReader = new ReverseFileReader(myFile, blockSize,
"UTF-8"); // encoding is supported
String line = null;
while((line=reverseFileReader.readLine())!=null) {
... // use the line
if(enoughLinesSeen) {
break;
}
}
reverseFileReader.close();
I believe this could be useful for other people as well!
was:
I needed to analyse a log file today and I was looking for a ReverseFileReader:
A class that behaves exactly like BufferedReader except that it goes from
bottom to top when readLine() is called. I didn't find it in IOUtils and the
internet didn't help a lot either, e.g.
http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
inefficient - the log files I'm analysing is huge and it is not a good idea to
load the whole content in the memory.
So I ended up writing an implementation myself using little memory and the
class RandomAccessFile - see attached file. It's used follows:
int blockSize = 4096; // only that much memory is needed, no matter how big the
file is
ReverseFileReader reverseFileReader = new ReverseFileReader(myFile, blockSize,
"UTF-8");
String line = null;
while((line=reverseFileReader.readLine())!=null) {
... // use the line
if(enoughLinesSeen) {
break;
}
}
reverseFileReader.close();
I believe this could be useful for other people as well!
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Attachments: ReverseFileReader.java
>
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (IO-288) Supply a ReverseFileReader
Supply a ReverseFileReader
--
Key: IO-288
URL: https://issues.apache.org/jira/browse/IO-288
Project: Commons IO
Issue Type: New Feature
Components: Utilities
Reporter: Georg Henzler
Attachments: ReverseFileReader.java
I needed to analyse a log file today and I was looking for a ReverseFileReader:
A class that behaves exactly like BufferedReader except that it goes from
bottom to top when readLine() is called. I didn't find it in IOUtils and the
internet didn't help a lot either, e.g.
http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
inefficient - the log files I'm analysing is huge and it is not a good idea to
load the whole content in the memory.
So I ended up writing an implementation myself using little memory and the
class RandomAccessFile - see attached file. It's used follows:
int blockSize = 4096; // only that much memory is needed, no matter how big the
file is
ReverseFileReader reverseFileReader = new ReverseFileReader(myFile, blockSize,
"UTF-8");
String line = null;
while((line=reverseFileReader.readLine())!=null) {
... // use the line
if(enoughLinesSeen) {
break;
}
}
reverseFileReader.close();
I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (IO-288) Supply a ReverseFileReader
[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Georg Henzler updated IO-288:
-
Attachment: ReverseFileReader.java
> Supply a ReverseFileReader
> --
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
>Reporter: Georg Henzler
> Attachments: ReverseFileReader.java
>
>
> I needed to analyse a log file today and I was looking for a
> ReverseFileReader: A class that behaves exactly like BufferedReader except
> that it goes from bottom to top when readLine() is called. I didn't find it
> in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing is huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReverseFileReader reverseFileReader = new ReverseFileReader(myFile,
> blockSize, "UTF-8");
> String line = null;
> while((line=reverseFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reverseFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-690) Remove methods from "MathUtils"
[ https://issues.apache.org/jira/browse/MATH-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130005#comment-13130005 ] Gilles commented on MATH-690: - Please have a look at revision 1185841. I've implemented "copySign" for "byte" arguments. It can replace the "sign" method; however, the additional functionality comes at the cost of a more complex method that must also check for overflow. Shall I go on and implement "copySign" for the other integral types? Shall I remove "sign"? The "indicator" methods are slightly different: They return -1 or 1 (whereas "sign", and "copySign", can return 0). There is a single use of it in CM (in class "Complex"). Is it safe to remove them, and replace with "copySign"? > Remove methods from "MathUtils" > --- > > Key: MATH-690 > URL: https://issues.apache.org/jira/browse/MATH-690 > Project: Commons Math > Issue Type: Task >Reporter: Gilles >Assignee: Gilles >Priority: Trivial > Labels: api-change > Fix For: 3.0 > > > I propose to remove the following methods from "MathUtils": > public static double sign(final double x) > public static float sign(final float x) > public static double sinh(double x) > public static double cosh(double x) > Also, "sign" and "indicator" functions seem redundant (and the "float" and > "double" versions are not dealing correctly with -0.0). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MATH-692) Cumulative probability and inverse cumulative probability inconsistencies
[
https://issues.apache.org/jira/browse/MATH-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Winter updated MATH-692:
--
Priority: Minor (was: Major)
> Cumulative probability and inverse cumulative probability inconsistencies
> -
>
> Key: MATH-692
> URL: https://issues.apache.org/jira/browse/MATH-692
> Project: Commons Math
> Issue Type: Bug
>Affects Versions: 1.0, 1.1, 1.2, 1.3, 2.0, 2.1, 2.2, 2.2.1, 3.0
>Reporter: Christian Winter
>Priority: Minor
> Fix For: 3.0
>
>
> There are some inconsistencies in the documentation and implementation of
> functions regarding cumulative probabilities and inverse cumulative
> probabilities. More precisely, '<' and '<=' are not used in a consistent way.
> Besides I would move the function inverseCumulativeProbability(double) to the
> interface Distribution. A true inverse of the distribution function does
> neither exist for Distribution nor for ContinuosDistribution. Thus we need to
> define the inverse in terms of quantiles anyway, and this can already be done
> for Distribution.
> On the whole I would declare the (inverse) cumulative probability functions
> in the basic distribution interfaces as follows:
> Distribution:
> - cumulativeProbability(double x): returns P(X <= x)
> - cumulativeProbability(double x0, double x1): returns P(x0 < X <= x1) [see
> also 1)]
> - inverseCumulativeProbability(double p):
> returns the quantile function inf{x in R | P(X<=x) >= p} [see also 2), 3),
> and 4)]
> 1) An aternative definition could be P(x0 <= X <= x1). But this requires to
> put the function probability(double x) or another cumulative probability
> function into the interface Distribution in order be able to calculate P(x0
> <= X <= x1) in AbstractDistribution.
> 2) This definition is stricter than the definition in ContinuousDistribution,
> because the definition there does not specify what to do if there are
> multiple x satisfying P(X<=x) = p.
> 3) A modification could be defined for p=0: Returning sup{x in R | P(X<=x) =
> 0} would yield the infimum of the distribution's support instead of a
> mandatory -infinity.
> 4) This affects issue MATH-540. I'd prefere the definition from above for the
> following reasons:
> - This definition simplifies inverse transform sampling (as mentioned in the
> other issue).
> - It is the standard textbook definition for the quantile function.
> - For integer distributions it has the advantage that the result doesn't
> change when switching to "x in Z", i.e. the result is independent of
> considering the intergers as sole set or as part of the reals.
> ContinuousDistribution:
> nothing to be added regarding (inverse) cumulative probability functions
> IntegerDistribution:
> - cumulativeProbability(int x): returns P(X <= x)
> - cumulativeProbability(int x0, int x1): returns P(x0 < X <= x1) [see also 1)
> above]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MATH-692) Cumulative probability and inverse cumulative probability inconsistencies
Cumulative probability and inverse cumulative probability inconsistencies
-
Key: MATH-692
URL: https://issues.apache.org/jira/browse/MATH-692
Project: Commons Math
Issue Type: Bug
Affects Versions: 2.2, 2.1, 2.0, 1.2, 1.1, 1.0, 1.3, 2.2.1, 3.0
Reporter: Christian Winter
Fix For: 3.0
There are some inconsistencies in the documentation and implementation of
functions regarding cumulative probabilities and inverse cumulative
probabilities. More precisely, '<' and '<=' are not used in a consistent way.
Besides I would move the function inverseCumulativeProbability(double) to the
interface Distribution. A true inverse of the distribution function does
neither exist for Distribution nor for ContinuosDistribution. Thus we need to
define the inverse in terms of quantiles anyway, and this can already be done
for Distribution.
On the whole I would declare the (inverse) cumulative probability functions in
the basic distribution interfaces as follows:
Distribution:
- cumulativeProbability(double x): returns P(X <= x)
- cumulativeProbability(double x0, double x1): returns P(x0 < X <= x1) [see
also 1)]
- inverseCumulativeProbability(double p):
returns the quantile function inf{x in R | P(X<=x) >= p} [see also 2), 3),
and 4)]
1) An aternative definition could be P(x0 <= X <= x1). But this requires to put
the function probability(double x) or another cumulative probability function
into the interface Distribution in order be able to calculate P(x0 <= X <= x1)
in AbstractDistribution.
2) This definition is stricter than the definition in ContinuousDistribution,
because the definition there does not specify what to do if there are multiple
x satisfying P(X<=x) = p.
3) A modification could be defined for p=0: Returning sup{x in R | P(X<=x) = 0}
would yield the infimum of the distribution's support instead of a mandatory
-infinity.
4) This affects issue MATH-540. I'd prefere the definition from above for the
following reasons:
- This definition simplifies inverse transform sampling (as mentioned in the
other issue).
- It is the standard textbook definition for the quantile function.
- For integer distributions it has the advantage that the result doesn't change
when switching to "x in Z", i.e. the result is independent of considering the
intergers as sole set or as part of the reals.
ContinuousDistribution:
nothing to be added regarding (inverse) cumulative probability functions
IntegerDistribution:
- cumulativeProbability(int x): returns P(X <= x)
- cumulativeProbability(int x0, int x1): returns P(x0 < X <= x1) [see also 1)
above]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COMPRESS-36) Add Zip64 Suport
[ https://issues.apache.org/jira/browse/COMPRESS-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129805#comment-13129805 ] Stefan Bodewig commented on COMPRESS-36: As of svn revision 1185722 the zips needed for integration tests are part of a .tar.bz2 archive in svn trunk (which is about two MB of size and expands to more than 100 MB of highly redundant zips). I'll remove the zips from my home dir on people.a.o shortly. > Add Zip64 Suport > > > Key: COMPRESS-36 > URL: https://issues.apache.org/jira/browse/COMPRESS-36 > Project: Commons Compress > Issue Type: New Feature > Components: Archivers >Reporter: Christian Grobmeier >Assignee: Stefan Bodewig > Fix For: 1.3 > > Attachments: 5GB_of_Zeros.zip, 5GB_of_Zeros_7ZIP.zip, > 5GB_of_Zeros_PKZip.zip, 5GB_of_Zeros_WinZip.zip, > 5GB_of_Zeros_WindowsCompressedFolders.zip, 5GB_of_Zeros_jar.zip, > zip64-sample.zip > > > Add Zip64 support. This will make it work to deal with zipfiles > 2 GB. > Planned for compress 1.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SANSELAN-56) proposed enhancement reduces load time for some image files by 40 percent
[
https://issues.apache.org/jira/browse/SANSELAN-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129747#comment-13129747
]
Gary Lucas commented on SANSELAN-56:
Since I am claiming some pretty large performance gains with this change
(and also advocating it as a possible solution for other data formats
such as PNG and JPEG) I thought it appropriate to describe how I got
the numbers I posted.
I test mostly under Windows with some Linux. Since a modern OS is a
noisy test environment, I try to run these tests when there are as
few additional things as possible running on the system.
My test methods always repeat the test procedure in a loop
that performs the same operation multiple times. I disregard
the timing results from the first few iterations of the loop
because (a) the initial operations reflect the timing overhead
for class loaders and the JIT compiler and (b) because
the Windows file cache pretty much guarantees that the
second time a program loads a file takes a lot less time
than the first.
The Sanselan.getBufferedImage(File) method has logic to
determine which of its "image parsers" to use based on the
file extension and other criteria. To eliminate this process
from the time measurements, I always create an instance
of the image parser specifically.
Between iterations, I also explicitly put any objects
created by the test program out of scope and
run the garbage collector. This approach is intended to
reduce the probability of significant garbage collection
running during the main operation that I am trying to time.
I have been testing with larger files (multiple megapixels)
so the resolution and noisiness of the system clock does
not affect the timing.
Here's a snippet of code I use for testing
public void testImageLoadTime(File file)
throws ImageReadException, IOException
{
long nPixel = 0;
long sumTimeMS = 0;
int nTests = 0;
for (int i = 0; i < 5; i++) {
// load the image
// Sanselan has logic to pick the right parser for the
// image format based on the file extension. But, to isolate
// performace costs to specific functionality, we bypass
// all of that and create an instance of a specific parser
HashMap params = new HashMap();
TiffImageParser tiffImageParser = new TiffImageParser();
ByteSourceFile byteSource = new ByteSourceFile(file);
// record the start time
long time0 = System.nanoTime();
BufferedImage bImage =
tiffImageParser.getBufferedImage(byteSource, params);
// record the completion time
long time1 = System.nanoTime();
// compute difference and print elapsed time
// for accumulated statistics, ignore the first two trials
long deltaMS = (time1 - time0) / 100;
if (i > 2) {
nTests++;
sumTimeMS += deltaMS;
}
nPixel = bImage.getWidth() * bImage.getHeight();
System.out.println("time (ms) =" + deltaMS);
// put all relevant objects out-of-scope and
// run garbage collector
bImage = null;
tiffImageParser = null;
byteSource = null;
Runtime.getRuntime().gc();
}
System.out.println("Number of pixels " + nPixel);
System.out.println("Average load time (ms) "
+ ((double) sumTimeMS / nTests));
}
> proposed enhancement reduces load time for some image files by 40 percent
> -
>
> Key: SANSELAN-56
> URL: https://issues.apache.org/jira/browse/SANSELAN-56
> Project: Commons Sanselan
> Issue Type: Improvement
> Environment: Tested in Windows, Linux, MacOS
>Reporter: Gary Lucas
> Labels: api-change
> Attachments: Sanselan-56-SpeedEnhanceTiff.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I have identified an enhancement that reduces the time required to load TIFF
> image by 40 percent. I have tested a modified version of Sanselan under
> Windows, Linux, and MacOS with consistent savings on each platform.
> Additionally, I suspect that this technique may be applicable to other areas
> of the Sanselan code base, including more popular image formats supported by
> Sanselan such as JPEG, PNG, etc.
> I propose to add the relevant code changes to the Sanselan code base. Once
> these modifications are in place, there would be an opportunity for others to
> look at the pro's and cons' of applying the techniques to other data formats.
> The Enhancement
> To load an image from a TIFF file, Sanselan performs extensive data
> processing in ord
[jira] [Commented] (SANDBOX-341) [functor] New components: summarize and aggregate
[ https://issues.apache.org/jira/browse/SANDBOX-341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129700#comment-13129700 ] Liviu Tudor commented on SANDBOX-341: - Hi Simone, I've already replied to you in private, this is just for other maintainers to be aware of this: I'm currently in the process of relocating to Palo Alto, CA (anyone living in the Bay area up for an informal meet at some point? ;) so things a bit hectic but still planning to finish this. ETA is about 2 weeks -- so end of October. Hope this is ok, Liv > [functor] New components: summarize and aggregate > - > > Key: SANDBOX-341 > URL: https://issues.apache.org/jira/browse/SANDBOX-341 > Project: Commons Sandbox > Issue Type: Improvement > Components: Functor > Environment: JDK 1.6.0_25 but should work with any JDK 5+ (possibly > 1.4 though I haven't tested). >Reporter: Liviu Tudor >Priority: Minor > Labels: features > Attachments: commons-functor-aggregate+summarizer.zip, > commons-functor.patch.bz2 > > > This is the next step from https://issues.apache.org/jira/browse/SANDBOX-340 > -- as instructed I'm finally hoping to get the code in the right place and > hopefully this is something that the functor component could do with. > Whereas initially I just started with the summarizer component, I have added > now the second one, the "aggregator" as they are somehow related. If this > code proves to be useful to functor in any way, it would actually be good to > get some feedback on these 2 to see if the class hierarchy can in fact be > changed to share some common functionality as I feel (probably due to the > similar needs that lead to writing/using these components) that somehow they > should share a common base. > In brief, the 2 components: > * aggregator: this just allows for data to be aggregated in a user defined > way (e.g. stored in a list for the purpose of averaging, computing the > arithmetic median etc). The classes provided actually offer the > implementation for storing data in a list and computing the above-mentioned > values or summing up everything. > * timed summarizer: this is another variation of the aggreator, however, it > adds the idea of regular "flushes", so based on a timer it will reset the > value and start summing/aggregating the data again. Rather than using an > aggregator which would store the whole data series (possibly for applying > more complex formulas), this component just computes on the fly on each > request the formula and stores the result of it. (Which does mean things like > computing arithmetic mean, median etc would be difficult to compute without > knowing upfront how many calls will be received -- i.e. how many elements we > will be required to summarize/aggregate.) So the memory footprint of running > this is much smaller -- even though, as I said, it achieves similar results. > I have only provided a summarizer which operates on integers, but obviously > others for float, double etc can be created if we go ahead with this design. > Hopefully the above make sense; this code has resulted from finding myself > writing similar components to these a few times and because it's always been > either one type (e.g. aggregator) or another (summarizer) I haven't given > quite possibly enough thought to the class design to join these 2. Also, > unfortunately, the time I sat down to make these components a bit more > general and submitted issue 340 was nearly 3 months ago so I'm trying to > remember myself all the ideas I had at a time so bear with me please if these > are still a bit fuzzy :) However, if you can make use of these I'm quite > happy to elaborate on areas that are unclear and obviously put some effort > into getting these components to the standards required to put these into a > release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
