Re: [MATH] what a commons-TLP split could look like

[Patrick Meyer]

After reading more about the proposed changes, I am now comfortable with
dividing the library into separate components. However, I have two
concerns. First, a package or class that passes unit tests should not be
abandoned or depricated just because there is no one to support it. It
should only be dropped if it fails a test AND there is no one to support
it. Second, we should be careful about dividing the library into too many
parts. For example, I don't agree with separating array based stats from
matrix or online based stats. That's my two cents. Thanks.
On Jun 20, 2016 5:31 AM, "Eric Barnhill"  wrote:

> Here's a proposed draft for how o.a.c.m might be split into a commons
> component, and more sophisticated spin-out components, around which we
> might develop a new Math TLP or incubator project in which components of
> the project that find backers continue on, and those that do not are frozen
> at their current state for now but not discarded (henceforth I just call
> this TLP).
>
> As a general strategy, I think it works to move the Field and Field Element
> superclasses into the TLP. Methods that use these elements or other
> math-specific data classes (e.g. DerivativeStructure, Neuron,
> AlternativeHypothesis) belong in the TLP. Array-based classes belong in
> commons. I see two exceptions to this. One is Complex objects which seem to
> me to belong in the commons remit. The other is the array of statistics
> objects (like Summary Statistics) as those statistics operations are right
> for commons. I think some refactoring of the stats packages may enable a
> clearer separation of the stats packages into those appropriate for commons
> and those appropriate for a TLP stats package, but I leave that for another
> thread.
>
> Also I call a package "dormant" when to my untrained eye it appears to have
> been underdeveloped and should perhaps be put out of its misery.
>
> Here we go:
> --
> o.a.c.m. Field, FieldElement, RealFieldElement -> TLP
> o.a.c.m. Analysis -> TLP
> o.a.c.m. complex.Complex, ComplexFormat, ComplexUtils -> commons
> o.a.c.m. complex.ComplexField, Quaternion, RootsOfUnity -> TLP
> o.a.c.m. dfp -> dormant
> o.a.c.m. exception -> commons
> o.a.c.m. filter -> dormant
> o.a.c.m. fraction -> all except Big FractionField to commons.
> BigFractionField to TLP
> o.a.c.m. genetics -> TLP
> o.a.c.m. geometry -> TLP
> o.a.c.m. linear -> TLP
> o.a.c.m. ml -> TLP
> o.a.c.m. ode -> TLP
> o.a.c.m. optim and optimization -> combine into one TLP component
> o.a.c.m. random and distribution -> combine into one TLP component
> o.a.c.m. primes -> commons
> o.a.c.m. special -> dormant
> o.a.c.m. stat.frequency -> perhaps belongs with random and distribution?
> o.a.c.m. stat.clustering -> dormant
> o.a.c.m. stat.correlation -> array based methods to commons, Matrix methods
> to TLP
> o.a.c.m. stat.descriptive, inferences, interval, ranking, regression ->
> Matrix methods to TLP. Array methods to commons. Statistics objects should
> be divided between both after a refactoring.
> o.a.c.m.transform -> commons
> o.a.c.m.util -> commons, perhaps refactored so the classes are in more
> informative package names
> --
>
> Eric
>

RE: [Math] Commons Math (r)evolution

[Patrick Meyer]

I think it would be better to spend the time trying to recruit new contributors 
than it would be to alienate existing ones. Also, the effort required to divide 
the library into smaller parts would be better spent creating patches. 

Does anyone have ideas for actively recruiting contributors? Do you know of 
mathematics departments that also teach students Java programming? A recruiting 
campaign with the message like "here's what we do at CM, we'd like your help" 
could attract new contributors. It will take time, but a constructive approach 
like this one will do more to sustain CM.

Thanks,
Patrick

-Original Message-
From: Jörg Schaible [mailto:joerg.schai...@bpm-inspire.com] 
Sent: Friday, June 10, 2016 6:20 AM
To: dev@commons.apache.org
Subject: Re: [Math] Commons Math (r)evolution

Hi Gilles,

Gilles wrote:

> Hi.
> 
> On Fri, 10 Jun 2016 01:48:20 +0200, Jörg Schaible wrote:

[snip]

>> MATH-172 is about an enhancement. Unfortunately no-one can currently 
>> implement it, so we have to wait until someone can or the issue stays 
>> simply unresolved again. You've requested for help and that was the 
>> proper action.
>> However, there has been no problem to release 3.0 in this state, so 
>> why should it be a problem now for 4.0?
> 
> Because the context has changed: in 3.0 there was support, now there 
> isn't.


That does not state the fact, that the code is already released and it does not 
matter at all if users ask questions for release 3.0 or 4.0.


>> MATH-1735 is a bug report for a problem which is currently not
>> reproducible.
>> Again you did the right action, but without more input and without a
>> possibility to reproduce the problem, there's not much we can do.
>> Again, why
>> should this issue prevent a release of 4.0?
> 
> The code in question should not have been in CM. [That was my position
> back
> then (cf. archive).]


Again, you cannot change history, it is already released. And a new release 
of the code does not make the situation worse.

 
> And every bug report for it is a reminder that unmaintainable code
> should
> not be released.


That is your interpretation. For me it is a normal bug report and we can 
eigher solve it on our own or have to wait for a contribution.

 
>>> Moreover what could be true for VFS is not for CM where there are
>>> many,
>>> many different areas that have nothing in common (except perhaps
>>> some
>>> ubiquitous very-low utilities which might be worth their own
>>> component
>>> to serve as a, maybe "internal", dependency).
>>>
>>> Also, compare the source basic statistics (lines of code):
>>>VFS  CM
>>> Java code24215   90834
>>> Unit tests8926   95595
>>>
>>> All in all, CM is more than 5 times larger than VFS (not even
>>> counting
>>> documentation).
>>
>> Any why is size suddenly a problem?
> 
> Not suddenly.
> I've raised this issue for years.  Please look at the archives!


Then: Why is size a problem? It is only a problem if *you* try to support 
all of it at once. Nobody expects that.

[snip]


>> Fine. But talk about artifacts, not components. Apache Commons Math
>> is still
>> *one* component of Apache Commons. It does not matter if you divide
>> the code
>> into different artifacts as long as anything is released together.
> 
> I know.
> 
> What you can't seem to get to is that I consider it a bad idea to
> release
> unsupported code.


You've stated that multiple times now.


> I think that "I do not know (and nobody else does)" is not an
> acceptable
> answer to user requests.


See, this is the difference. To me this *is* acceptible. Especially if users 
have to be kind of experts themselves to use this code.


> If we _know_ that some parts of the code would be unsupported (such
> that it
> would elicit this kind of answer for bug reports), then it's
> _deceiving_ to
> release that code.


A new release does not change the situation at all. With such a definition 
we could move quite some of our components directly to the attic, because 
the original authors are no longer around and we might currently have no 
expert around.


>> Individual release cycles for the different parts can only happen if
>> Math is
>> TLP, but not in Apache Commons. We will certainly not allow and
>> create again
>> any sub-umbrellas (search the Jakarta archives).
> 
> Who talked about sub-umbrellas?  I didn't.


I talk about it, because it is what the result looks to me.

 
> I've explained that
>   1. CM cannot be released as it was before


You've expressed that *you* don't want to release it as it was before (well-
founded however).


>   2. for some parts, the necessary _minimal_ support has "disappeared"


That does not immediately invalidate the existing code. One fact that you 
don't want to see.


>   3. some parts can be turned into independent components (with _full_
>  support)


Have we some kind of agreement here to introduce new Commons components? As 
far as I know, we just did not oppose to br

Re: [math] Name of the new TLP

[Patrick Meyer]

The Apache commons math library already has a reputation and is well kvown.
Any name that does not involve the words Apache and math will require a lot
of rebranding or years of explaining to people that the TLP named X is
really just the library formerly known as commons math. Removing "commons"
from the name is a good way to signal the maturity of the math library
while staying true to its Apache origin. That's why I like Apache math.
On Feb 2, 2016 5:50 PM, "Gilles"  wrote:

> On Tue, 2 Feb 2016 14:17:06 -0800, Gary Gregory wrote:
>
>> On Tue, Feb 2, 2016 at 1:50 PM, Niall Pemberton <
>> niall.pember...@gmail.com>
>> wrote:
>>
>> On Sun, Jan 24, 2016 at 9:08 PM, Phil Steitz 
>>> wrote:
>>>
>>> > We need to agree on a name.  My own preference is for a boring,
>>> > descriptive name, but I am manifestly not a marketing guy, so won't
>>> > be offended if others want to be more creative.
>>> >
>>> > My suggestion is
>>> >
>>> > MathComponents
>>> >
>>> > Hearkens back to HttpComponents, which has worked pretty well.
>>> >
>>>
>>> I've just started learning modern Greek and Math is Μαθηματικά -
>>>  (Mathematika) - so how about "Apache Mathematika"?
>>>
>>>
>> And get a nasty letter from https://www.wolfram.com/mathematica/ perhaps?
>>
>> Gary
>>
>
> Are you implying that they would dare challenge our usage of a name
> that is absolutely not entertaining any confusion?
> As anyone can observe,
> 1. there is a prefix referring to a native American tribe (which thus
>has nothing to do with programming computers), and
> 2. the (common) word referring to "mathematics" is clearly spelled
>differently.
>
> Thus, _nothing_ to worry about.
> 8-}
>
> Gilles
>
>
>>> Niall
>>>
>>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

RE: [math] [POLL] new TLP name

[Patrick Meyer]

Apache Math

-Original Message-
From: Phil Steitz [mailto:phil.ste...@gmail.com] 
Sent: Monday, February 01, 2016 12:07 PM
To: Commons Developers List
Subject: [math] [POLL] new TLP name

Please select your top choice among the following suggested names for the new 
[math]-based TLP.  All are welcome and encouraged to respond.  This POLL will 
be open for 72 hours, at which time two tallies will be presented:  one among 
those who have volunteered for the new PMC and a second among all respondents.  
Hopefully, one name will emerge as consensus winner.  If not, I will kick off 
another
runoff poll among the top choices.   Please respond with your top
choice for the name.

AjaMa
Epsilon
Erdos
Euclid
Euler
Gauss
JAML
Math
MathBlocks
MathComponents (or Math Components)
Mathelactica
MathModules
Megginson
modMath
Nash
Newton
Pythagoras





-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [MATH-1120] Needed opinion about support on variations in percentile calculation

[Patrick Meyer]

There are various ways to compute percentiles and statistical programs use 
different methods by default. I would prefer that we do like R and provide 
multiple options for the type of percentile computation. See 
http://stat.ethz.ch/R-manual/R-patched/library/stats/html/quantile.html 

Patrick

-Original Message-
From: venkatesha murthy [mailto:venkateshamurth...@gmail.com] 
Sent: Wednesday, May 21, 2014 3:18 PM
To: dev@commons.apache.org
Subject: [MATH-1120] Needed opinion about support on variations in percentile 
calculation

Hi All,

The existing Percentile class calculates the percentile based on the quantile 
position of the array fixed as p * (N+1)/100 for a pth Percentile on an Array 
of size N. However if we were to add these numbers in MS Excel to calculate the 
percentile it provides a different result and closely resembeles the formula 
[p*(N-1)/100]+1.

Its imperative at times to match the computations to a standard spreadsheet 
calculations or to a standard tool; which is why i request for allowing the 
quantile position to be customized.
Infact even the kth selection used
can also be refactored as a strategy(than as a private methods) as a further 
step.

So if atleast the Percentile class were to allow the quantile position to be 
customized in the sub classes; then the end user may be helped in providing the 
formula of their choice.

The most minimal change i am proposing here is to just make the quantile 
position setting as a protected method and i have attached a possible patch in 
[MATH-1120] .

Request all to opinionate on this

thanks
venkat


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [VOTE] Release Math 3.3 based on RC3 (second run after mail outage)

[Patrick Meyer]

+1

-Original Message-
From: Luc Maisonobe [mailto:l...@spaceroots.org] 
Sent: Sunday, May 11, 2014 5:56 AM
To: Commons Developers List
Subject: [VOTE] Release Math 3.3 based on RC3 (second run after mail outage)

Hi all,

As I am not sure this message will be recovered after mail outage, I send it 
again, with an update of final date.

I would like to call a vote to release Commons Math 3.3 based on RC3.

Changes since RC2:

 * reverted javadoc fixes for Java 8

Note:

Commons Math 3.3 does compile with Java 8, but creating the site will not work 
due to some incompatibilities:

 * javadoc: known doclint issue
 * findbugs: not ready for java 8


 Math 3.3 RC3 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/math/
(svn revision 5295)

 Maven artifacts are here:


https://repository.apache.org/content/repositories/orgapachecommons-1027/org/apache/commons/commons-math3/3.3/

 Details of changes since 3.2 are in the release notes:
https://dist.apache.org/repos/dist/dev/commons/math/RELEASE-NOTES.txt
http://people.apache.org/builds/commons/math/3.3/RC3/changes-report.html

 The tag is here:
https://svn.apache.org/repos/asf/commons/proper/math/tags/MATH_3_3_RC3
(svn revision 1593137)

 Site:
http://people.apache.org/builds/commons/math/3.3/RC3/
  (note the apidocs for the 3.3 release will be added with the release)

 Clirr Report (compared to 3.2):
http://people.apache.org/builds/commons/math/3.3/RC3/clirr-report.html
   (note that there are 4 false positives where the signature of a method has 
changed such that the parameter type is now the superclass of the previous type)

 RAT Report:
http://people.apache.org/builds/commons/math/3.3/RC3/rat-report.html

 Findbugs Report:
http://people.apache.org/builds/commons/math/3.3/RC3/findbugs.html

  KEYS:
  http://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.

Note that the artifacts have been prepared by Thomas (thanks to him!) and he 
delegated the vote to me. So the signatures correspond to his key and not mine.

This vote will close no sooner that 72 hours from now, that is 
2014-05-14T10:00:00Z (this is UTC time).

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [VOTE] Release Math 3.3 based on RC2

[Patrick Meyer]

+1

-Original Message-
From: Thomas Neidhart [mailto:thomas.neidh...@gmail.com] 
Sent: Thursday, May 1, 2014 12:16 PM
To: dev@commons.apache.org
Subject: [VOTE] Release Math 3.3 based on RC2

Hi all,

I would like to call a vote to release Commons Math 3.3 based on RC2.

Changes since RC1:

 * completed changelog
 * fixed MATH-1110
 * added exclusion filter for findbugs false-positive
 * disabled 2 tests that failed with Java 8
 * javadoc fixes for Java 8 (incomplete, see below)

Note:

Commons Math 3.3 does compile with Java 8, but creating the site will not
work due to some incompatibilities:

 * javadoc: known doclint issue
 * findbugs: not ready for java 8


 Math 3.3 RC2 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/math/
(svn revision 5215)

 Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1026/org
/apache/commons/commons-math3/3.3/

 Details of changes since 3.2 are in the release notes:
https://dist.apache.org/repos/dist/dev/commons/math/RELEASE-NOTES.txt
http://people.apache.org/builds/commons/math/3.3/RC2/changes-report.html

 The tag is here:
https://svn.apache.org/repos/asf/commons/proper/math/tags/MATH_3_3_RC2
(svn revision 1591699)

 Site:
http://people.apache.org/builds/commons/math/3.3/RC2/
  (note the apidocs for the 3.3 release will be added with the release)

 Clirr Report (compared to 3.2):
http://people.apache.org/builds/commons/math/3.3/RC2/clirr-report.html
   (note that there are 4 false positives where the signature of a method
has changed such that the parameter type is now the superclass of the
previous type)

 RAT Report:
http://people.apache.org/builds/commons/math/3.3/RC2/rat-report.html

 Findbugs Report:
http://people.apache.org/builds/commons/math/3.3/RC2/findbugs.html

  KEYS:
  http://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.
This vote will close no sooner that 72 hours from now, i.e. after 1800 GMT
04-May 2014.

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Best regards,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [VOTE] Release Math 3.3 based on RC1

[Patrick Meyer]

Thanks!

-Original Message-
From: Gilles [mailto:gil...@harfang.homelinux.org] 
Sent: Wednesday, April 30, 2014 6:21 PM
To: dev@commons.apache.org
Subject: RE: [VOTE] Release Math 3.3 based on RC1

On Wed, 30 Apr 2014 09:30:37 -0400, Patrick Meyer wrote:
> MATH-1092 appear to be unresolved but a path was submitted. Is it 
> possible to include this patch in 3.3?
>

The patch was included (see the JIRA page), but the issue was not marked as 
resolved.


Regards,
Gilles

P.S. I've just added the missing changelog entry.


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [VOTE] Release Math 3.3 based on RC1

[Patrick Meyer]

MATH-1092 appear to be unresolved but a path was submitted. Is it possible
to include this patch in 3.3?



-Original Message-
From: Thomas Neidhart [mailto:thomas.neidh...@gmail.com] 
Sent: Tuesday, April 29, 2014 4:08 PM
To: dev@commons.apache.org
Subject: [VOTE] Release Math 3.3 based on RC1

Hi all,

I would like to call a vote to release Commons Math 3.3 based on RC1.

 Math 3.3 RC1 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/math/
(svn revision 5199)

 Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1025/org
/apache/commons/commons-math3/3.3/

 Details of changes since 3.2 are in the release notes:
https://dist.apache.org/repos/dist/dev/commons/math/RELEASE-NOTES.txt
http://people.apache.org/builds/commons/math/3.3/RC1/changes-report.html

 The tag is here:
https://svn.apache.org/repos/asf/commons/proper/math/tags/MATH_3_3_RC1
(svn revision 1591059)

 Site:
http://people.apache.org/builds/commons/math/3.3/RC1/
  (note the apidocs for the 3.3 release will be added with the release)

 Clirr Report (compared to 3.2):
http://people.apache.org/builds/commons/math/3.3/RC1/clirr-report.html
   (note that there are 4 false positives where the signature of a method
has changed such that the parameter type is now the superclass of the
previous type)

 RAT Report:
http://people.apache.org/builds/commons/math/3.3/RC1/rat-report.html

 Findbugs Report:
http://people.apache.org/builds/commons/math/3.3/RC1/findbugs.html
  (the listed error is a false positive, the relevant line has not changed
since the last release and the floating point comparison should be correct
in this case).

  KEYS:
  http://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.
This vote will close no sooner that 72 hours from now, i.e. after 1000 GMT
02-May 2014.

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Best regards,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [math] trouble with SingularValueDecomposition

[Patrick Meyer]

Thanks again Ted. 

For what it's worth I have been using the linear algebra features in CM 
recently and they are really nice. Thanks to everyone who contributed to it!

-Original Message-
From: Ted Dunning [mailto:ted.dunn...@gmail.com] 
Sent: Saturday, February 15, 2014 8:02 PM
To: Commons Developers List
Subject: Re: [math] trouble with SingularValueDecomposition

Note that the only reason that the order is unconstrained is because the two 
corresponding singular values are equal.

Strictly speaking, for equal singular values, any unitary transformation of the 
corresponding singular vectors are also valid singular vectors.



On Sat, Feb 15, 2014 at 4:09 AM, Patrick Meyer  wrote:

> Thanks Ted. As I mentioned my knowledge of SVD is limited, and I was 
> not aware that it is OK to have a different order of the first two 
> columns in the results (or the conditions under which the order 
> doesn't matter). I am trying to track down a bug in some code and that’s what 
> led me to the SVD.
> I guess I need to keep looking for the real bug.
>
> For completeness, my results R were the same as you reported. My 
> results from CM are shown below and if you swap the first and second 
> column, the results agree with R.
>
> U:
> 0.9940594018965339  0.06774763124429131  -0.08518312016997649
> 0.10615872136916754  -0.7761401247896214  0.621551704858
> 0.02400481989869077  0.6269104921377042  0.778721390144956
>
> V:
> 0.9963653125425972  0.0  -0.08518312016997495
> 0.0531395658155507  -0.7815621241949481  0.621551704865
> 0.06657590034559915  0.6238274168581248  0.7787213901449556
>
>
>
> -Original Message-
> From: Ted Dunning [mailto:ted.dunn...@gmail.com]
> Sent: Saturday, February 15, 2014 2:17 AM
> To: Commons Developers List
> Subject: Re: [math] trouble with SingularValueDecomposition
>
> For what its worth, I tested the Mahout SVD which shares code lineage 
> with the Commons Math implementation.
>
> The results I got were:
>
>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *sum(abs(m - u * s * v')) = 4.31946146e-16S =1.002319690998
> >  1.0023196909981.
>
> U =0.994059401897 0.067747631244
> > -0.0851831201700.106158721369 -0.776140124790 0.62155170
> >  0.024004819899 0.626910492138 0.778721390145 V =0.996365312543
> > 0. -0.0851831201700.053139565816 -0.781562124195
> > 0.621551700.066575900346 0.623827416858 0.778721390145*
>
>
> Note that the residue of the reconstruction is excellently small.  
> This indicates that the result is correct.
>
>
> If you compare these to the R results,
>
>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *[1] 1.0023196909980066 1.0023196909980066 1.$u
> >   [,1]  [,2]  [,3][1,]
> >  0.067747631244291326 -0.994059401896534967  0.085183120169970525 
> > [2,]
> > -0.776140124789635122 -0.106158721369163295 
> > -0.62155170469113[3,]
> >  0.626910492137687125 -0.024004819898688426 -0.778721390144969994$v
> >  [,1]  [,2]  [,3] [1,]
> >  0.0 -0.996365312542597747  0.085183120169970497[2,]
> > -0.78156212419496163 -0.053139565815546450 -0.62155170469668[3,]
> >  0.62382741685810772 -0.066575900345596822 -0.778721390144969550*
>
>
> These are identical to the previous results except that the first two 
> singular values are equal which means that the order of the 
> corresponding left and right singular vectors are different and there 
> are sign changes in the singular vectors.
>
> My guess is that you will get the same results in Apache Commons Math.
>
>
>
> On Fri, Feb 14, 2014 at 6:07 PM, Patrick Meyer  wrote:
>
> > Hi,
> >
> >
> >
> > I am using the SingularValueDecomposition class with a matrix but it 
> > gives me a different result than R. My knowledge of SVD is limited, 
> > so any advice is welcomed.
> >
> >
> >
> > Here's the method in Java
> >
> >
> >
> > public void svdTest(){
> >
> >
> >
> > double[][] x = {
> >
> > {1.0,  -0.053071807862720116,  0.04236086650321309},
> >
> > {0.05307180786272012,  1.0,  0.0058054424137053435},
> >
> > {-0.04236086650321309,  -0.005805442413705342,  1.0}
> >
> > };
> >
> >
> >
> > RealMatrix X = new Array2DRowRe

RE: [math] trouble with SingularValueDecomposition

[Patrick Meyer]

Thanks Ted. As I mentioned my knowledge of SVD is limited, and I was not aware 
that it is OK to have a different order of the first two columns in the results 
(or the conditions under which the order doesn't matter). I am trying to track 
down a bug in some code and that’s what led me to the SVD. I guess I need to 
keep looking for the real bug.

For completeness, my results R were the same as you reported. My results from 
CM are shown below and if you swap the first and second column, the results 
agree with R.

U:
0.9940594018965339  0.06774763124429131  -0.08518312016997649  
0.10615872136916754  -0.7761401247896214  0.621551704858  
0.02400481989869077  0.6269104921377042  0.778721390144956  

V:
0.9963653125425972  0.0  -0.08518312016997495  
0.0531395658155507  -0.7815621241949481  0.621551704865  
0.06657590034559915  0.6238274168581248  0.7787213901449556



-Original Message-
From: Ted Dunning [mailto:ted.dunn...@gmail.com] 
Sent: Saturday, February 15, 2014 2:17 AM
To: Commons Developers List
Subject: Re: [math] trouble with SingularValueDecomposition

For what its worth, I tested the Mahout SVD which shares code lineage with the 
Commons Math implementation.

The results I got were:


>
>
>
>
>
>
>
>
>
>
>
> *sum(abs(m - u * s * v')) = 4.31946146e-16S =1.002319690998
>  1.0023196909981. 

U =0.994059401897 0.067747631244
> -0.0851831201700.106158721369 -0.776140124790 0.62155170
>  0.024004819899 0.626910492138 0.778721390145 V =0.996365312543
> 0. -0.0851831201700.053139565816 -0.781562124195
> 0.621551700.066575900346 0.623827416858 0.778721390145*


Note that the residue of the reconstruction is excellently small.  This 
indicates that the result is correct.


If you compare these to the R results,


>
>
>
>
>
>
>
>
>
>
> *[1] 1.0023196909980066 1.0023196909980066 1.$u
>   [,1]  [,2]  [,3][1,]
>  0.067747631244291326 -0.994059401896534967  0.085183120169970525 [2,]
> -0.776140124789635122 -0.106158721369163295 -0.62155170469113[3,]
>  0.626910492137687125 -0.024004819898688426 -0.778721390144969994$v
>  [,1]  [,2]  [,3] [1,]
>  0.0 -0.996365312542597747  0.085183120169970497[2,]
> -0.78156212419496163 -0.053139565815546450 -0.62155170469668[3,]
>  0.62382741685810772 -0.066575900345596822 -0.778721390144969550*


These are identical to the previous results except that the first two singular 
values are equal which means that the order of the corresponding left and right 
singular vectors are different and there are sign changes in the singular 
vectors.

My guess is that you will get the same results in Apache Commons Math.



On Fri, Feb 14, 2014 at 6:07 PM, Patrick Meyer  wrote:

> Hi,
>
>
>
> I am using the SingularValueDecomposition class with a matrix but it 
> gives me a different result than R. My knowledge of SVD is limited, so 
> any advice is welcomed.
>
>
>
> Here's the method in Java
>
>
>
> public void svdTest(){
>
>
>
> double[][] x = {
>
> {1.0,  -0.053071807862720116,  0.04236086650321309},
>
> {0.05307180786272012,  1.0,  0.0058054424137053435},
>
> {-0.04236086650321309,  -0.005805442413705342,  1.0}
>
> };
>
>
>
> RealMatrix X = new Array2DRowRealMatrix(x);
>
>
>
> SingularValueDecomposition svd = new 
> SingularValueDecomposition(X);
>
>
>
> RealMatrix U = svd.getU();
>
> for(int i=0;i
> for(int j=0;j
> System.out.print(U.getEntry(i,j) + "  ");
>
> }
>
> System.out.println();
>
> }
>
>
>
> System.out.println();
>
> System.out.println();
>
> RealMatrix V = svd.getV();
>
> for(int i=0;i
> for(int j=0;j
> System.out.print(V.getEntry(i,j) + "  ");
>
> }
>
> System.out.println();
>
> }
>
>
>
>
>
> }
>
>
>
>
>
> And here's the function in R.
>
>
>
> x<-matrix(c(
>
> 1.0,  -0.053071807862720116,  0.04236086650321309,
>
>   0.05307180786272012,  1.0,  0.0058054424137053435,
>
>   -0.04236086650321309,  -0.005805442413705342,  1.0),
>
> nrow=3, byrow=TRUE)
>
> svd(x)
>
>
>
> Does anyone know why I am getting different results for U and V? I am 
> using commons math 3.1.
>
>
>
> Thanks,
>
> Patrick
>
>
>
>
>
>
>
>


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[math] trouble with SingularValueDecomposition

[Patrick Meyer]

Hi,

 

I am using the SingularValueDecomposition class with a matrix but it gives
me a different result than R. My knowledge of SVD is limited, so any advice
is welcomed.

 

Here's the method in Java

 

public void svdTest(){

 

double[][] x = {

{1.0,  -0.053071807862720116,  0.04236086650321309},

{0.05307180786272012,  1.0,  0.0058054424137053435},

{-0.04236086650321309,  -0.005805442413705342,  1.0}

};

 

RealMatrix X = new Array2DRowRealMatrix(x);

 

SingularValueDecomposition svd = new SingularValueDecomposition(X);

 

RealMatrix U = svd.getU();

for(int i=0;i

Re: [Math] Cleaning up the curve fitters

[Patrick Meyer]

My vote was not directed toward the technical merits of the optimization
package. I am not an expert on optimization and I trust you in this regard.
My vote was a comment on the usability of the API. I find it to be
increasing difficult to use.





Maybe I have missed some refactoring, but why all of the depreicated
classes like



On Thu, Jul 18, 2013 at 10:58 AM, Gilles wrote:

> You have done a pretty good job in making the optimization package
>>> non-sensical to user.
>>>
>>
>> +1
>>
>>
> Check your facts ("svn log", JIRA, this ML), please: All modifications
> were done in plain sight, and responded to identified problems which
> I cared to solve.
> I do not deny that it could have raised issues for cases which I did
> not foresee. But all the requests for clearly defined
>   use-cases,
>   benchmarks,
>   actual (i.e. working) code examples,
> have been met with unhelpful "do it yourself" statements.[1]
>
> Whatever we try here, including going somewhat in the direction you
> suggested for singling out algorithms aimed at solving a (non-linear)
> "leastsquares" problem:[2]
>   
> http://markmail.org/message/**nkceta2cskpgbkug
> raises unsubstantiated criticism that never[3] translated into
> _practical_ advice[4] on how to improve the code.
>
>
> Gilles
>
> [1] What _I_ observed and used led me to try out what I think are
> improvements. I you think otherwise, _you_ have to get your hands
> dirty too.
> [2] Except that, at this point, it makes more sense for me, as a user,
> to put it under (model) "fitting" rather than "optim", for the
> reason I gave previously.
> [3] And that means: NEVER.
> [4] That is: an alternative Java code that compiles, runs and produces
> correct answers.
>
>
>
> --**--**-
> To unsubscribe, e-mail: 
> dev-unsubscribe@commons.**apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

RE: [Math] Cleaning up the curve fitters

[Patrick Meyer]

>You have done a pretty good job in making the optimization package
non-sensical to user.

+1



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] Remove "BOBYQAOptimizer"

[Patrick Meyer]

Please do not remove BOBYQAOptimizer from CM. It works quite well for me. I
have a function that takes two parameters. BOBYQAOptimizer finds the right
solution. In 3.0 NonlinearConjugateGradientOptimizer (without the multiple
starts) would never find the right solution. Now that I have switched to
3.1, NonlinearConjugategradientOptimizer (with multiple starts) simply
doesn't work.

I have to express my frustration with all of the changes to the optimization
procedures. The amount of refactoring since 2.2 has been extensive. First,
the refactoring to 3.0 and now to 3.1. It looks like there may be even more
refactoring in 4.0. It is getting very difficult to use the optimization
part of the library because everything is dissected into so many parts.
BOBYQAOptimizer may have been ported from FORTRAN and it may not fit an
object oriented framework, but those older libraries were much easier to
use. 

Some of you may need the flexibility of the flexibility of tailoring every
aspect of the optimization algorithm. Others such as myself just need
something that works as simply as the "optim" function in R. I can do in one
line of R code what takes numerous lines of Java code (that has to be
recoded with every release of CM). It's frustrating.

Patrick



-Original Message-
From: Gilles Sadowski [mailto:gil...@harfang.homelinux.org] 
Sent: Monday, December 31, 2012 8:29 PM
To: dev@commons.apache.org
Subject: [Math] Remove "BOBYQAOptimizer"

Hello.

Cross-posting this message from the the "user" ML:

On Mon, Dec 31, 2012 at 04:23:34PM -0500, enzhong...@pnc.com wrote:
> I use BOBYQAOptimizer to do optimization for Black Scholes formula. 
> maxEval is set to 20, but I still got TooManyEvaluationsException. 
> When it throws exception, I get nothing from the optimizer. 
> CurrentBest is good enough, but it is private and there is no interface to
get it.
> Generally, how do I handle exception with optimizer? I'd like to use 
> currentBest in case of TooManyEvaluationsException.

IMO, it was a bad idea to include "BOBYQAOptimizer" into CM. [For that
matter, any code recognizably ported from other languages (namely FORTRAN
and C) without thorough adapatation to Java has been the source of
problems.]

If nobody is able to work on "BOBYQAOptimizer" so as to make it a
maintainable and documented code, I request that it be removed from the next
major release (4.0).


Regards,
Gilles

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [math] Examples

[Patrick Meyer]

Hi Phil,

I use various parts of Commons Math (CM) for a program I created for
psychometrics called jMetrik, www.ItemAnalysis.com. It does a variety of
statistical procedures from basic descriptive statistics to item response
theory and test equating. jMetrik has a fairly large international user
base. I'm currently working on version 3, which involved a full switch to CM
3.0. Feel free to use the program description on the homepage of the
website. The code is pretty extensive, but I can provide specific examples
of how I use CM. I first found CM several years ago when I was searching for
source code for numerical optimization. I'm happy to help out. Let me know
if you would like more information.

Patrick


-Original Message-
From: Phil Steitz [mailto:phil.ste...@gmail.com] 
Sent: Wednesday, November 28, 2012 8:01 PM
To: Commons Developers List; Commons Users List
Subject: [math] Examples

Apologies for the cross-post.  Please respond to just the dev list.

I am going to do a talk on Commons Math at Apachecon US in Portland next
February.  One of the things I want to do in the talk is to present some
examples of practical use of the library.  I would love to get some examples
from the community that I could use, with acknowledgement, under the ASL.

Ideal examples can be described simply (i.e., not too much arcane
domain-specific theory needed to understand the practical
application) and use [math] APIs simply and directly.  For example, one of
my own applications is a discrete event simulator that uses [math] APIs to
generate event inter-arrival times and to accumulate latency statistics.
Part of the goal of the talk in general and the examples in particular is to
show people how to find the things they need to solve problems using [math],
so some information on how you found your way to the needed goodies (or
helped us improve the library so the goodies you needed were there) will
also be helpful.

Thanks in advance for any contributions.  We can discuss ideas on the dev
list, or you can send me private mail.  I will take code and descriptions in
any form.

Thanks!

Phil

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [math] correlation analysis with NaNs

[Patrick Meyer]

-Original Message-
From: Phil Steitz [mailto:phil.ste...@gmail.com] 
Sent: Monday, November 19, 2012 12:45 PM
To: Commons Developers List
Subject: Re: [math] correlation analysis with NaNs

On 11/19/12 3:31 AM, Gilles Sadowski wrote:
> On Sun, Nov 18, 2012 at 09:27:41PM -0800, Phil Steitz wrote:
>> On 11/18/12 2:01 PM, Thomas Neidhart wrote:
>>> On 11/09/2012 11:14 PM, Phil Steitz wrote:
>>>> On 11/9/12 12:18 AM, Thomas Neidhart wrote:
>>>>> On Thu, Nov 8, 2012 at 7:21 PM, Phil Steitz 
wrote:
>>>>>
>>>>>> On 11/8/12 9:44 AM, Phil Steitz wrote:
>>>>>>> On 11/8/12 8:23 AM, Gilles Sadowski wrote:
>>>>>>>> On Thu, Nov 08, 2012 at 05:00:52PM +0100, Thomas Neidhart wrote:
>>>>>>>>> On 11/08/2012 02:01 PM, Sébastien Brisard wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> 2012/11/8 Gilles Sadowski :
>>>>>>>>>>> On Thu, Nov 08, 2012 at 09:39:00AM +0100, Thomas Neidhart wrote:
>>>>>>>>>>>> Hi Patrick,
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/07/2012 04:37 PM, Patrick Meyer wrote:
>>>>>>>>>>>>> I agree that it would be nice to have a constructor that 
>>>>>>>>>>>>> allows
>>>>>> you to
>>>>>>>>>>>>> specific the ranking algorithm only.
>>>>>>> +1 - patches welcome.
>>>>>>>>>>>>> As far as NaN and the Spearman correlation, maybe we 
>>>>>>>>>>>>> should add a
>>>>>> default
>>>>>>>>>>>>> strategy of NaNStrategy.FAIL so that an exception would 
>>>>>>>>>>>>> occur if
>>>>>> any NaN is
>>>>>>>>>>>>> encountered. R uses this treatment of missing data and 
>>>>>>>>>>>>> forces
>>>>>> users to
>>>>>>>>>>>>> choose how to handle it. If we implemented something like 
>>>>>>>>>>>>> listwise
>>>>>> or
>>>>>>>>>>>>> pairwise deletion it could be used in other classes too. 
>>>>>>>>>>>>> As such,
>>>>>> treatment
>>>>>>>>>>>>> of missing data should be part of a larger discussion and 
>>>>>>>>>>>>> handled
>>>>>> in a more
>>>>>>>>>>>>> comprehensive and systematic way.
>>>>>>> +1 to develop a strategy for representing how to represent and
>>>>>>> handle missing data (see below)
>>>>>>>>>>>> I think this additional option makes sense, but I forward 
>>>>>>>>>>>> this discussion to the dev mailing list where it is better
suited.
>>>>>>>>>>> I'm wary of having CM handle "missing" data.
>>>>>>>>>>> For one thing we'd have to define a "convention" to 
>>>>>>>>>>> represent
>>>>>> missing data.
>>>>>>>>>>> There is no good way to do that in Java. Using NaN for this 
>>>>>>>>>>> purpose
>>>>>> in a
>>>>>>>>>>> low-level library is not a good idea IMHO.
>>>>>>>>>>>
>>>>>>>>>> I agree with Gilles, here. If I remember correctly, R has a 
>>>>>>>>>> special value NA, or something similar, which differs from NaN.
>>>>>>>>>>> Then, any convention might not be suitable for some user 
>>>>>>>>>>> applications, which would lead such an
>>>>>> application's
>>>>>>>>>>> developer to filter the data anyway in order to change his
>>>>>> representation to
>>>>>>>>>>> CM's representation. Rather that calling two redundant 
>>>>>>>>>>> filtering
>>>>>> codes, I'd
>>>>>>>>>>> rather assume that CM gets a clean input on which its 
>>>>>>>>>>> algorithm can
>>>

RE: [Math] MATH-894

[Patrick Meyer]

I misunderstood an earlier email and I though the compute method was public. It 
seems that adding a public compute method may be more complication and effort 
than it is worth. My apologies for the misunderstanding. The compute method 
would only be of value if it can be an easy addition. 

-Original Message-
From: Phil Steitz [mailto:phil.ste...@gmail.com] 
Sent: Thursday, November 15, 2012 1:42 PM
To: Commons Developers List
Subject: Re: [Math] MATH-894

On 11/15/12 10:29 AM, Ted Dunning wrote:
> On Thu, Nov 15, 2012 at 10:04 AM, Phil Steitz  wrote:
>
>> Do you know how to do that with a primitive array?  Can you provide 
>> some sample code?
>>
> You don't.  See my next paragraph.
>
> See the assign method in this class:
>
> https://github.com/apache/mahout/blob/trunk/math/src/main/java/org/apa
> che/mahout/math/VectorView.java

Interesting.  I see no assign method, but I can see what this thing does.  It 
is not clear to me though how this idea could be meaningfully applied to solve 
the problem we have with applying
statistics to an RDA without doing any array copying.   Most likely
I am missing the point.

Phil
>
>
>
>
>
>> Thanks for your help on this.
>>
>> Phil
>>> The Colt/Mahout approach is to define a view object which opaquely 
>>> remembers a reference to the original, an offset and a length.  
>>> Functions and other arguments can be passed to this view object 
>>> which operates on a subset of the original contents by calling the 
>>> function.  Performance is actually quite good.  The JIT seems to 
>>> in-line the view object access to the underlying object and also 
>>> in-lines evaluation of the function so
>> that
>>> the actual code that is executed is pretty much what you would write 
>>> in
>> C,
>>> but you don't have to worry as much since the pattern of access is 
>>> more controlled.
>>>
>>> For completeness, this is essentially what java.nio does with the 
>>> *Buffer classes as well.  You can wrap an array and then you can ask 
>>> for slices
>> out
>>> of that array while retaining the reference semantics.
>>>


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] MATH-894

[Patrick Meyer]

Hi Gilles,

These changes look fine to me and the addition of the compute method is
really nice. Looking more closely at my code, I am using the getElements
method. As long as that remains available, it make sense to deprecate the
getInternalValues method.

My use of ResizeableDoubleArray is related to an earlier discussion -
missing values. My data are stored in a database and it may contain missing
values. I know how many cases are in the database, but I don't know the
amount of missing data. I read the non-missing database values into a
ResizeableDoubleArray, call getElements() and use the nonmissing data array
in my calculations. It may be a bit clunky, but it's one of the ways I
handle missing data without looping over the database twice. I don't have a
solution for comprehensive treatment of missing data yet, but I appreciate
the conversation we are having.

Patrick


-Original Message-
From: Gilles Sadowski [mailto:gil...@harfang.homelinux.org] 
Sent: Thursday, November 15, 2012 6:32 AM
To: dev@commons.apache.org
Subject: Re: [Math] MATH-894

Hi Patrick.

On Mon, Nov 12, 2012 at 02:38:07PM -0500, Patrick Meyer wrote:
> Please keep ResizeableDoubleArray as its own class. I find it very 
> useful for more than descriptive statistics. I do like the idea of 
> adding an apply method to it.

Could you please have a look at the JIRA page:
  https://issues.apache.org/jira/browse/MATH-894
and tell us whether the proposed changes would affect your usage of the
class?


Thanks for the feedback,
Gilles

> [...]

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] MATH-894

[Patrick Meyer]

Please keep ResizeableDoubleArray as its own class. I find it very useful
for more than descriptive statistics. I do like the idea of adding an apply
method to it.

Patrick

-Original Message-
From: Phil Steitz [mailto:phil.ste...@gmail.com] 
Sent: Monday, November 12, 2012 1:23 PM
To: Commons Developers List
Subject: Re: [Math] MATH-894

On 11/12/12 6:05 AM, Gilles Sadowski wrote:
> Hi.
>
> What do you think about deprecating "getInternalValues"?
> Its only use is in "DescriptiveStatistics". I guess that the current 
> usage could save some time (by avoiding the creation of an array and 
> copying the elements), so that acces to the internal representation should
be retained.

That is why it is there.  ResizeableDoubleArray was created to be the
backing store for DescriptiveStatistics.  We don't actually use it anywhere
else, so I am wondering if it might in fact be better to deprecate the
entire class and aim to make it a private inner class of
DescriptiveStatistics.  That way, the broken encapsulation that you point
out will be less of an issue.  The idea behind the class is to support a
"rolling window" of data from a stream that statistics can be applied to.
If we want to keep it separate (or even not, actually), it might be better
for it to expose an apply method that takes a UnivariateStatistic as an
argument and applies the statistic to the currently defined window.  So the
following method from DescriptiveStatistics  public double
apply(UnivariateStatistic stat) {
return stat.evaluate(eDA.getInternalValues(), eDA.start(),
eDA.getNumElements());  } would be implemented in ResizeableDoubleArray
(just removing the eDA. everywhere).  Then in DescriptiveStatistics you
would have

public double apply(UnivariateStatistic stat) {
return eDA.apply(stat);
 }
possibly renaming the version in RDA.  But given all of the fuss over this
class that really is just there to serve DescriptiveStatistics, I think it
may be best to just make it a private inner class of DescriptiveStatistics.

Phil


> If so, I think that the name "getInternalValues" should be changed in 
> order to make it explicit that we are exposing an instance's field 
> (whose modification will be reflected in the object's state). The 
> suffix "...Values" is misleading; I suggest "getInternalArray" since 
> current API (and current usage) anyways forbids that another data
structure be used.
> [The alternative would be to enhance encapsulation by hiding the 
> internal representation altogether, thus removing the methods
"getInternalValues()"
> and "start()".]
>
> I also notice that the "clear()" method reallocates the internal array.
> IMO, it is unnecessarily inefficient. If one wanted to get this 
> behaviour, one could just create a new object. However, when reusing 
> the same object, users could legitimately expect that no allocation 
> occurs and that it is only the _contents_ that is discarded.
>
>
> Regards,
> Gilles
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] MATH-816 (mixture model distribution)

[Patrick Meyer]

Yes, I like the latest changes. It looks cleaner to me.

It seems that the attachments only describe the mixture distribution and do
not provide the EM estimation algorithm. Am I missing something?  Didn't the
original contributor mention the estimation part too? The parts I would like
to add will need the EM part first. 

Thanks!


-Original Message-
From: Gilles Sadowski [mailto:gil...@harfang.homelinux.org] 
Sent: Thursday, October 18, 2012 9:33 AM
To: dev@commons.apache.org
Subject: Re: [Math] MATH-816 (mixture model distribution)

On Thu, Oct 18, 2012 at 08:13:52AM -0400, Patrick Meyer wrote:
> I vote for simplicity. Current practice in the social sciences is to 
> fit multiple models, each with a different number of components, and 
> use fit statistics to choose the best model.

So... Do you vote for the current proposal (as in the latest attachment on
the JIRA page)? [Sorry for being dense. :-)]

[The (simple) "mixture model" code could be in 3.1, due to be out a couple
of weeks _ago_. :-}]

> 
> There are some additional features I would like to see added and I 
> have the code to contribute if it is not currently there. To be 
> consistent with Mplus, we need have the algorithm use multiple random 
> starts and run a few of the best starts to completion. Mplus uses this 
> strategy to effectively overcome local minima.

Proposals welcome; please open a feature request with an outline of the
implementation.


Thanks,
Gilles

> [...]

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] MATH-816 (mixture model distribution) . .

[Patrick Meyer]

I vote for simplicity. Current practice in the social sciences is to fit
multiple models, each with a different number of components, and use fit
statistics to choose the best model. 

There are some additional features I would like to see added and I have the
code to contribute if it is not currently there. To be consistent with
Mplus, we need have the algorithm use multiple random starts and run a few
of the best starts to completion. Mplus uses this strategy to effectively
overcome local minima.


-Original Message-
From: Becksfort, Jared [mailto:jared.becksf...@stjude.org] 
Sent: Wednesday, October 17, 2012 11:37 PM
To: Commons Developers List
Subject: RE: [Math] MATH-816 (mixture model distribution) . .

I see.  I am planning to submit the EM fit for multivariate normal mixture
models in the next couple of weeks (Math-817).  A Gibbs sampling DP fit may
be a bit further out.   I am not opposed to allowing the number of
components to change, but I also like the simplicity of this class.
Whatever you guys decide is probably fine.

Jared

From: Ted Dunning [ted.dunn...@gmail.com]
Sent: Wednesday, October 17, 2012 9:41 PM
To: Commons Developers List
Subject: Re: [Math] MATH-816 (mixture model distribution)
=?utf-8?B?LiAgICAu? ==?utf-8?B?LiAgICAu?=

The issue is that with a fixed number of components, you need to do multiple
runs to find a best fit number of components.  Gibbs sampling against a
Dirichlet process can get you to the same answer in about the same cost as a
single run of EM with a fixed number of models.

On Wed, Oct 17, 2012 at 7:31 PM, Becksfort, Jared <
jared.becksf...@stjude.org> wrote:

> Ted,
>
> I am not sure I understand the problem with the fixed number of 
> components.  My understanding is that CM prefers immutable objects. 
> Adding a component to an object would require reweighting in addition 
> to modifying the component list.  A new mixture model could be 
> instantiated using the getComponents function and then adding or 
> removing more components if necessary.
>
> Jared
> 
> From: Ted Dunning [ted.dunn...@gmail.com]
> Sent: Wednesday, October 17, 2012 5:21 PM
> To: Commons Developers List
> Subject: Re: [Math] MATH-816 (mixture model
> distribution)=?utf-8?B?LiAgICAu?=
>
> Seems fine.
>
> I think that the limitation to a fixed number of mixture components is 
> a bit limiting.  So is the limitation to a uniform set of components.  
> Both limitations can be eased without a huge difficultly.
>
> Avoiding the fixed number of components can be done by using some 
> variant of Dirichlet processes.  Simply picking k_max relatively large 
> and then using an approximate DP over that finite set works well.
>
> That said, mixture models are pretty nice to have.
>
> On Wed, Oct 17, 2012 at 2:13 PM, Gilles Sadowski < 
> gil...@harfang.homelinux.org> wrote:
>
> > Hello.
> >
> > Any objection to commit the code as proposed on the report page?
> >   https://issues.apache.org/jira/browse/MATH-816
> >
> >
> > Regards,
> > Gilles
> >
> > 
> > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> > For additional commands, e-mail: dev-h...@commons.apache.org
> >
> >
>
> Email Disclaimer:  www.stjude.org/emaildisclaimer Consultation 
> Disclaimer:  www.stjude.org/consultationdisclaimer
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] Toward releasing 3.0 ?

[Patrick Meyer]

Looks great, thanks!

-Original Message-
From: Thomas Neidhart [mailto:thomas.neidh...@gmail.com] 
Sent: Wednesday, February 15, 2012 2:28 PM
To: dev@commons.apache.org
Subject: Re: [Math] Toward releasing 3.0 ?

On 02/15/2012 02:41 PM, Patrick Meyer wrote:
> OK, I submited a patch that includes comments and documentation. Let 
> me know if I need to write more, but I think I've covered the 
> functionality of the classes.

Hi Patrick,

thanks for the patch. I have applied it together with additional code
cleanup and javadoc in r1244667.

What I have done:

 - applied your javadoc patch
 - added missing javadoc for variables
 - moved var initialization to constructor
 - changed exceptions to more specific ones
 - simplified getCovarianceMatrix (uses getData() to construct the
   RealMatrix)
 - changed deltaX / deltaY to local variables

Could you please take a look and give feedback about the changes made?

@all: from my point of view this would resolve issue MATH-449 so far as it
is fully documented and has proper unit tests.

Thanks Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] Toward releasing 3.0 ?

[Patrick Meyer]

OK, I submited a patch that includes comments and documentation. Let me know
if I need to write more, but I think I've covered the functionality of the
classes.

Patrick

-Original Message-
From: Gilles Sadowski [mailto:gil...@harfang.homelinux.org] 
Sent: Tuesday, February 14, 2012 7:46 AM
To: dev@commons.apache.org
Subject: Re: [Math] Toward releasing 3.0 ?

On Tue, Feb 14, 2012 at 01:12:50PM +0100, Thomas Neidhart wrote:
> On 02/14/2012 12:50 PM, Patrick Meyer wrote:
> > I can document the StorelessCovariance addition. What is the best 
> > way to add documentation? Do I just add comments to the file and create
a patch?
> 
> Yes this would be fine. I have seen that there is an open issue 
> regarding this contribution, please attach the patch to it (MATH-449).

Yes, of course, that's better than creating a new one as I suggested in my
previous reply!
I've just updated the target version of MATH-449.

Regards,
Gilles

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [Math] Toward releasing 3.0 ?

[Patrick Meyer]

I can document the StorelessCovariance addition. What is the best way to add
documentation? Do I just add comments to the file and create a patch?


-Original Message-
From: Thomas Neidhart [mailto:thomas.neidh...@gmail.com] 
Sent: Monday, February 13, 2012 9:21 AM
To: dev@commons.apache.org
Subject: Re: [Math] Toward releasing 3.0 ?

Hi,

I have seen that there are several classes that are almost undocumented:

 - PivotingQRDecomposition (linear)
 - StorelessCovariance (stat.correlation)
 - StorelessBivariateCovariance (stat.correlation)

both seem to be quite new contributions, is somebody willing to help here?

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] psychometrics

[Patrick Meyer]

Ha ha... I was really hoping someone else would do all the hard work and 
write the MVN distribution function for me.


On 9/3/2011 1:53 PM, Phil Steitz wrote:

On 9/3/11 10:37 AM, Patrick Meyer wrote:

No, I have written very few tests. Until I recently discovered
jUnit, I was using a very crude way of testing my code. I plan to
add more tests over the coming year.

If you are registered with java.net, you can view the source code
at http://java.net/projects/psychometrics. I have requested that
the project be made public but I don't know how long that will take.

There is one dependency in the polychoric correlation package
(com.itemanalysis.polycor) that could be problematic. The maximum
likelihood estimator requires a multivariate normal (or at least
bivariate normal) CDF. I use the one from
http://vadim.kutsyy.com/java/javadoc/com/kutsyy/util/package-summary.html
Are there plans to add a multivaraite normal CDF function to
commons math?

Patches welcome :)

Phil

Patrick



On 9/3/2011 12:36 PM, Greg Sterijevski wrote:

Do you have unit tests written for the different components?

On Sat, Sep 3, 2011 at 10:48 AM, Patrick
Meyer   wrote:


Sounds good. Separating the general purpose stats from the
psychometric
specific stats seems like a natural way to distinguish the two
libraries.
I'll send a link to the source code soon. Hopefully, you will
see the good
ideas in the code.


On 9/3/2011 12:57 AM, Phil Steitz wrote:


On 9/2/11 7:19 AM, Patrick Meyer wrote:


Yes, the math code is separate from the GUI and database for the
most part. I'd be happy to share the code and documentation,
but I
need a few days to add it to a repository and get it online.

There are parts of my library that can be transferred to math
with
very few changes but other parts that need more work to make it
more object oriented in style and less procedural. In any case,
I'll send information about the code once I have it online.


Don't worry too much about making things beautiful.  Bad code +
good
ideas is the combination that works [1].  The thing to think about
is what portions of your code are really general purpose math.
Most
likely kernel regression, polycor, histograms, etc., maybe not so
much test scaling for example.   Luc may have some good advice
having done this already with an space dynamics library (or maybe
Luc is so smart that there was no refactoring necessary :)

What tends to work best is to bring things in incrementally,
talking
about the fit and how to integrate.

Phil

[1] http://s.apache.org/hZ


Patrick

On 9/2/2011 9:26 AM, Gilles Sadowski wrote:


Hello.

   I have been developing an pure Java application that does a
variety

of psychometric methods. I use the commons math library as much
as I
can but I've also had to develop my own library. I'd like to
combine
the two libraries by donating as much of my code to commons
math as
I can. My library includes features for measurement
reliability,
test scaling, test equating, polychoric correlations, histogram
computations, kernel regression, etc. Most of these feature are
specialized to psychometrics, but is there any interest in
adding
them to commons math? Are these methods too specialized for
commons
math?


Can we have a look at the code and documentation?

   If you're interested in seeing the application in action,
you can

download the full program from www.ItemAnalysis.com. You'll
notice
that it also uses an Apache derby database for data management.


Is the math code separate from GUI and data management?


Regards,
Gilles

--**--**
-

To unsubscribe, e-mail:
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org


--**--**

-
To unsubscribe, e-mail:
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org




--**--**-

To unsubscribe, e-mail:
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



--**--**-

To unsubscribe, e-mail:
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org




-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] psychometrics

[Patrick Meyer]

No, I have written very few tests. Until I recently discovered jUnit, I 
was using a very crude way of testing my code. I plan to add more tests 
over the coming year.


If you are registered with java.net, you can view the source code at 
http://java.net/projects/psychometrics. I have requested that the 
project be made public but I don't know how long that will take.


There is one dependency in the polychoric correlation package 
(com.itemanalysis.polycor) that could be problematic. The maximum 
likelihood estimator requires a multivariate normal (or at least 
bivariate normal) CDF. I use the one from 
http://vadim.kutsyy.com/java/javadoc/com/kutsyy/util/package-summary.html Are 
there plans to add a multivaraite normal CDF function to commons math?


Patrick



On 9/3/2011 12:36 PM, Greg Sterijevski wrote:

Do you have unit tests written for the different components?

On Sat, Sep 3, 2011 at 10:48 AM, Patrick Meyer  wrote:


Sounds good. Separating the general purpose stats from the psychometric
specific stats seems like a natural way to distinguish the two libraries.
I'll send a link to the source code soon. Hopefully, you will see the good
ideas in the code.


On 9/3/2011 12:57 AM, Phil Steitz wrote:


On 9/2/11 7:19 AM, Patrick Meyer wrote:


Yes, the math code is separate from the GUI and database for the
most part. I'd be happy to share the code and documentation, but I
need a few days to add it to a repository and get it online.

There are parts of my library that can be transferred to math with
very few changes but other parts that need more work to make it
more object oriented in style and less procedural. In any case,
I'll send information about the code once I have it online.


Don't worry too much about making things beautiful.  Bad code + good
ideas is the combination that works [1].  The thing to think about
is what portions of your code are really general purpose math.  Most
likely kernel regression, polycor, histograms, etc., maybe not so
much test scaling for example.   Luc may have some good advice
having done this already with an space dynamics library (or maybe
Luc is so smart that there was no refactoring necessary :)

What tends to work best is to bring things in incrementally, talking
about the fit and how to integrate.

Phil

[1] http://s.apache.org/hZ


Patrick

On 9/2/2011 9:26 AM, Gilles Sadowski wrote:


Hello.

  I have been developing an pure Java application that does a variety

of psychometric methods. I use the commons math library as much
as I
can but I've also had to develop my own library. I'd like to
combine
the two libraries by donating as much of my code to commons math as
I can. My library includes features for measurement reliability,
test scaling, test equating, polychoric correlations, histogram
computations, kernel regression, etc. Most of these feature are
specialized to psychometrics, but is there any interest in adding
them to commons math? Are these methods too specialized for commons
math?


Can we have a look at the code and documentation?

  If you're interested in seeing the application in action, you can

download the full program from www.ItemAnalysis.com. You'll notice
that it also uses an Apache derby database for data management.


Is the math code separate from GUI and data management?


Regards,
Gilles

--**--**
-

To unsubscribe, e-mail: 
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

  --**--**

-
To unsubscribe, e-mail: 
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org




--**--**-
To unsubscribe, e-mail: 
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



--**--**-
To unsubscribe, e-mail: 
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org




-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] psychometrics

[Patrick Meyer]

Sounds good. Separating the general purpose stats from the psychometric 
specific stats seems like a natural way to distinguish the two 
libraries. I'll send a link to the source code soon. Hopefully, you will 
see the good ideas in the code.


On 9/3/2011 12:57 AM, Phil Steitz wrote:

On 9/2/11 7:19 AM, Patrick Meyer wrote:

Yes, the math code is separate from the GUI and database for the
most part. I'd be happy to share the code and documentation, but I
need a few days to add it to a repository and get it online.

There are parts of my library that can be transferred to math with
very few changes but other parts that need more work to make it
more object oriented in style and less procedural. In any case,
I'll send information about the code once I have it online.

Don't worry too much about making things beautiful.  Bad code + good
ideas is the combination that works [1].  The thing to think about
is what portions of your code are really general purpose math.  Most
likely kernel regression, polycor, histograms, etc., maybe not so
much test scaling for example.   Luc may have some good advice
having done this already with an space dynamics library (or maybe
Luc is so smart that there was no refactoring necessary :)

What tends to work best is to bring things in incrementally, talking
about the fit and how to integrate.

Phil

[1] http://s.apache.org/hZ

Patrick

On 9/2/2011 9:26 AM, Gilles Sadowski wrote:

Hello.


I have been developing an pure Java application that does a variety
of psychometric methods. I use the commons math library as much
as I
can but I've also had to develop my own library. I'd like to
combine
the two libraries by donating as much of my code to commons math as
I can. My library includes features for measurement reliability,
test scaling, test equating, polychoric correlations, histogram
computations, kernel regression, etc. Most of these feature are
specialized to psychometrics, but is there any interest in adding
them to commons math? Are these methods too specialized for commons
math?

Can we have a look at the code and documentation?


If you're interested in seeing the application in action, you can
download the full program from www.ItemAnalysis.com. You'll notice
that it also uses an Apache derby database for data management.

Is the math code separate from GUI and data management?


Regards,
Gilles

-

To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org




-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] psychometrics

[Patrick Meyer]

Yes, the math code is separate from the GUI and database for the most 
part. I'd be happy to share the code and documentation, but I need a few 
days to add it to a repository and get it online.


There are parts of my library that can be transferred to math with very 
few changes but other parts that need more work to make it more object 
oriented in style and less procedural. In any case, I'll send 
information about the code once I have it online.


Patrick

On 9/2/2011 9:26 AM, Gilles Sadowski wrote:

Hello.


I have been developing an pure Java application that does a variety
of psychometric methods. I use the commons math library as much as I
can but I've also had to develop my own library. I'd like to combine
the two libraries by donating as much of my code to commons math as
I can. My library includes features for measurement reliability,
test scaling, test equating, polychoric correlations, histogram
computations, kernel regression, etc. Most of these feature are
specialized to psychometrics, but is there any interest in adding
them to commons math? Are these methods too specialized for commons
math?

Can we have a look at the code and documentation?


If you're interested in seeing the application in action, you can
download the full program from www.ItemAnalysis.com. You'll notice
that it also uses an Apache derby database for data management.

Is the math code separate from GUI and data management?


Regards,
Gilles

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[math] psychometrics

[Patrick Meyer]

Hi All,

I have been developing an pure Java application that does a variety of 
psychometric methods. I use the commons math library as much as I can 
but I've also had to develop my own library. I'd like to combine the two 
libraries by donating as much of my code to commons math as I can. My 
library includes features for measurement reliability, test scaling, 
test equating, polychoric correlations, histogram computations, kernel 
regression, etc. Most of these feature are specialized to psychometrics, 
but is there any interest in adding them to commons math? Are these 
methods too specialized for commons math?


If you're interested in seeing the application in action, you can 
download the full program from www.ItemAnalysis.com. You'll notice that 
it also uses an Apache derby database for data management.


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] StorelessCovariance

[Patrick Meyer]

Thanks for the tip. I was not aware of IntelliJ but I'll certainly look 
into it.


On 8/18/2011 1:27 AM, Ted Dunning wrote:

NetBeans is kind of wasting away for lack of attention.

IntelliJ generally gets these things exactly correct.  SVN, git and mvn are
all mother tongues for it.  There is an excellent community edition and
Apache committers can get a full version.

On Wed, Aug 17, 2011 at 8:09 PM, Patrick Meyer  wrote:


OK, after fidgeting with NetBeans for too long without results, I went with
Phil's suggestion and just used subversion at the command prompt. It was
much easier than trying to configure NetBeans.

I uploaded the patch this evening. It includes two new classes
StorelessCovariance.java and StorelessCovarianceMatrix.**java. I also
included a new test class StorelessCovarianceTest.java.

Interestingly, the one-pass algorithm is slightly less accurate than the
two-pass algorithm. I had to reduce the tolerance from 10E-9 to 10E-7 in
order for tests to pass with the Longley data. The tolerance was not changed
for the Swiss data. It remained 10E-13 for the Swiss data.

The one-pass algorithm is described in a paper by Pebay (2008)
http://prod.sandia.gov/**techlib/access-control.cgi/**2008/086212.pdf<http://prod.sandia.gov/techlib/access-control.cgi/2008/086212.pdf>

Thanks for everyone's help.

Patrick



On 8/17/2011 12:13 PM, Patrick wrote:


I am workin on the StorelessCovariance per jira MATH-449. However, I'm
having a hard time using subversion via Netbeans. I have successfully
created the new classes and a JUnit test. However, when I run the test I get
the following message,

"caused an ERORR: Absent Code attribute in method that is not native or
abstract in class file org/apache/commons/math/**TestUtils"

Any help would be appreciated. I have a lot to contribute to commons math
but my lack of experience with subversion is really hindering me.

Thanks,
Patrick



--**--**-
To unsubscribe, e-mail: 
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



--**--**-
To unsubscribe, e-mail: 
dev-unsubscribe@commons.**apache.org
For additional commands, e-mail: dev-h...@commons.apache.org




-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] StorelessCovariance

[Patrick Meyer]

OK, after fidgeting with NetBeans for too long without results, I went 
with Phil's suggestion and just used subversion at the command prompt. 
It was much easier than trying to configure NetBeans.


I uploaded the patch this evening. It includes two new classes 
StorelessCovariance.java and StorelessCovarianceMatrix.java. I also 
included a new test class StorelessCovarianceTest.java.


Interestingly, the one-pass algorithm is slightly less accurate than the 
two-pass algorithm. I had to reduce the tolerance from 10E-9 to 10E-7 in 
order for tests to pass with the Longley data. The tolerance was not 
changed for the Swiss data. It remained 10E-13 for the Swiss data.


The one-pass algorithm is described in a paper by Pebay (2008) 
http://prod.sandia.gov/techlib/access-control.cgi/2008/086212.pdf


Thanks for everyone's help.

Patrick



On 8/17/2011 12:13 PM, Patrick wrote:
I am workin on the StorelessCovariance per jira MATH-449. However, I'm 
having a hard time using subversion via Netbeans. I have successfully 
created the new classes and a JUnit test. However, when I run the test 
I get the following message,


"caused an ERORR: Absent Code attribute in method that is not native 
or abstract in class file org/apache/commons/math/TestUtils"


Any help would be appreciated. I have a lot to contribute to commons 
math but my lack of experience with subversion is really hindering me.


Thanks,
Patrick



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[math] StorelessCovariance

[Patrick Meyer]

I am workin on the StorelessCovariance per jira MATH-449. However, I'm 
having a hard time using subversion via Netbeans. I have successfully 
created the new classes and a JUnit test. However, when I run the test I 
get the following message,


"caused an ERORR: Absent Code attribute in method that is not native or 
abstract in class file org/apache/commons/math/TestUtils"


Any help would be appreciated. I have a lot to contribute to commons 
math but my lack of experience with subversion is really hindering me.


Thanks,
Patrick

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] Numerical derivatives in Commons Math

[Patrick Meyer]

Thanks for the information Luc. I didn't know those existed. I'm happy 
to keep the discussion at the implementation levels.


On 8/12/2011 6:23 AM, Luc Maisonobe wrote:

Le 12/08/2011 00:30, Patrick Meyer a écrit :

I like the idea of adding this feature. What about an abstract class
that implements DifferentiableMultivariateRealFunction and provides the
method for partialDerivative (). People could then override the
partialDerivative method if they have an analytic derivative.

Here's some code that I'm happy to contribute to Commons-math. It
computes the derivative by the central difference meathod and the
Hessian by finite difference. I can add this to JIRA when it's there.


Hi Patrick,

I think we need to discuss about the API we want and then about the 
implementation. There are already other finite differences 
implementations in some tests for both Apache Commons Math and a 
complete package in Apache Commons Nabla. Your code adds Hessian to 
this which is really a good thing.


Thanks,
Luc



/**
* Numerically compute gradient by the central difference method.
Override this method
* when the analytic gradient is available.
*
*
* @param x
* @return
*/
public double[] derivativeAt(double[] x){
int n = x.length;
double[] grd = new double[n];
double[] u = Arrays.copyOfRange(x, 0, x.length);
double f1 = 0.0;
double f2 = 0.0;
double stepSize = 0.0001;

for(int i=0;i
Le 11/08/2011 23:27, Fran Lattanzio a écrit :

Hello,


Hi Fran,



I have a proposal for a numerical derivatives framework for Commons
Math. I'd like to add the ability to take any UnivariateRealFunction
and produce another function that represents it's derivative for an
arbitrary order. Basically, I'm saying add a factory-like interface
that looks something like the following:

public interface UniverateNumericalDeriver {
public UnivariateRealFunction derive(UnivariateRealFunction f, int
derivOrder);
}


This sound interesting. did you have a look at Commons Nabla
UnivariateDifferentiator interface and its implementations ?

Luc



For an initial implementation of this interface, I propose using
finite differences - either central, forward, or backward. Computing
the finite difference coefficients, for any derivative order and any
error order, is a relatively trivial linear algebra problem. The user
will simply choose an error order and difference type when setting up
an FD univariate deriver - everything else will happen automagically.
You can compute the FD coefficients once the user invokes the function
in the interface above (might be expensive), and determine an
appropriate stencil width when they call evaluate(double) on the
function returned by the aformentioned method - for example, if the
user has asked for the nth derivative, we simply use the nth root of
the machine epsilon/double ulp for the stencil width. It would also be
pretty easy to let the user control this (which might be desirable in
some cases). Wikipedia has decent article on FDs of all flavors:
http://en.wikipedia.org/wiki/Finite_difference

There are, of course, many other univariate numerical derivative
schemes that could be added in the future - using Fourier transforms,
Barak's adaptive degree polynomial method, etc. These could be added
later. We could also add the ability to numerically differentiate at
single point using an arbitrary or user-defined grid (rather than an
automatically generated one, like above). Barak's method and Fornberg
finite difference coefficients could be used in this case:
http://pubs.acs.org/doi/abs/10.1021/ac00113a006
http://amath.colorado.edu/faculty/fornberg/Docs/MathComp_88_FD_formulas.pdf 




It would also make sense to add vectorial and matrix-flavored versions
of interface above. These interfaces would be slightly more complex,
but nothing too crazy. Again, the initial implementation would be
finite differences. This would also be really easy to implement, since
multivariate FD coefficients are nothing more than an outer product of
their univariate cousins. The Wikipedia article also has some good
introductory material on multivariate FDs.

Cheers,
Fran.

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org





-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org





-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] Numerical derivatives in Commons Math

[Patrick Meyer]

I like the idea of adding this feature. What about an abstract class 
that implements DifferentiableMultivariateRealFunction and provides the 
method for partialDerivative (). People could then override the 
partialDerivative method if they have an analytic derivative.


Here's some code that I'm happy to contribute to Commons-math. It 
computes the derivative by the central difference meathod and the 
Hessian by finite difference. I can add this to JIRA when it's there.


/**
 * Numerically compute gradient by the central difference method. 
Override this method

 * when the analytic gradient is available.
 *
 *
 * @param x
 * @return
 */
public double[] derivativeAt(double[] x){
int n = x.length;
double[] grd = new double[n];
double[] u = Arrays.copyOfRange(x, 0, x.length);
double f1 = 0.0;
double f2 = 0.0;
double stepSize = 0.0001;

for(int i=0;istepSize = Math.sqrt(EPSILON)*(Math.abs(x[i])+1.0);//from 
SAS manual on nlp procedure

u[i] = x[i] + stepSize;
f1 = valueAt(u);
u[i] = x[i] - stepSize;
f2 = valueAt(u);
grd[i] = (f1-f2)/(2.0*stepSize);
}
return grd;
}

/**
 * Numerically compute Hessian using a finite difference method. 
Override this

 * method when the analytic Hessian is available.
 *
 * @param x
 * @return
 */
public double[][] hessianAt(double[] x){
int n = x.length;
double[][] hessian = new double[n][n];
double[] gradientAtXpls = null;
double[] gradientAtX = derivativeAt(x);
double xtemp = 0.0;
double stepSize = 0.0001;

for(int j=0;jstepSize = Math.sqrt(EPSILON)*(Math.abs(x[j])+1.0);//from 
SAS manual on nlp procedure

xtemp = x[j];
x[j] = xtemp + stepSize;
double [] x_copy = Arrays.copyOfRange(x, 0, x.length);
gradientAtXpls = derivativeAt(x_copy);
x[j] = xtemp;
for(int i=0;ihessian[i][j] = 
(gradientAtXpls[i]-gradientAtX[i])/stepSize;

}
}
return hessian;
}


On 8/11/2011 5:36 PM, Luc Maisonobe wrote:

Le 11/08/2011 23:27, Fran Lattanzio a écrit :

Hello,


Hi Fran,



I have a proposal for a numerical derivatives framework for Commons
Math. I'd like to add the ability to take any UnivariateRealFunction
and produce another function that represents it's derivative for an
arbitrary order. Basically, I'm saying add a factory-like interface
that looks something like the following:

public interface UniverateNumericalDeriver {
  public UnivariateRealFunction derive(UnivariateRealFunction f, int 
derivOrder);

}


This sound interesting. did you have a look at Commons Nabla 
UnivariateDifferentiator interface and its implementations ?


Luc



For an initial implementation of this interface, I propose using
finite differences - either central, forward, or backward. Computing
the finite difference coefficients, for any derivative order and any
error order, is a relatively trivial linear algebra problem. The user
will simply choose an error order and difference type when setting up
an FD univariate deriver - everything else will happen automagically.
You can compute the FD coefficients once the user invokes the function
in the interface above (might be expensive), and determine an
appropriate stencil width when they call evaluate(double) on the
function returned by the aformentioned method - for example, if the
user has asked for the nth derivative, we simply use the nth root of
the machine epsilon/double ulp for the stencil width. It would also be
pretty easy to let the user control this (which might be desirable in
some cases). Wikipedia has decent article on FDs of all flavors:
http://en.wikipedia.org/wiki/Finite_difference

There are, of course, many other univariate numerical derivative
schemes that could be added in the future - using Fourier transforms,
Barak's adaptive degree polynomial method, etc. These could be added
later. We could also add the ability to numerically differentiate at
single point using an arbitrary or user-defined grid (rather than an
automatically generated one, like above). Barak's method and Fornberg
finite difference coefficients could be used in this case:
http://pubs.acs.org/doi/abs/10.1021/ac00113a006
http://amath.colorado.edu/faculty/fornberg/Docs/MathComp_88_FD_formulas.pdf 



It would also make sense to add vectorial and matrix-flavored versions
of interface above. These interfaces would be slightly more complex,
but nothing too crazy. Again, the initial implementation would be
finite differences. This would also be really easy to implement, since
multivariate FD coefficients are nothing more than an outer product of
their univariate cousins. The Wikipedia article also has some good
introductory material on multivariate FDs.

Cheers,
Fran.

-

Re: [math] adding new classes

[Patrick Meyer]

Thanks. I think my problem may be an IDE issue. In Eclipse, I had no luck
using New > Java Class, but I was able to use New > File. Is that normal?
Sorry for the simple questions, but I really appreciate your help.

On Thu, Dec 9, 2010 at 1:33 PM, Gilles Sadowski <
gil...@harfang.homelinux.org> wrote:

> Hi.
>
> > Excuse my typo, I meant to say if it is possible to add new classes via
> > SUBVERSION, not substance. As I am new to subversion it seems that only
> > patches for existing classes are possible.
>
> Try
>
> $ svn add SomeFile.java
>
>
> Regards,
> Gilles
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [math] adding new classes

[Patrick Meyer]

Excuse my typo, I meant to say if it is possible to add new classes via
SUBVERSION, not substance. As I am new to subversion it seems that only
patches for existing classes are possible.

On Thu, Dec 9, 2010 at 11:45 AM, Patrick Meyer  wrote:

> Is it possible to add new classes via substance patches? I have a storeless
> covariance class and other classes I would like to contribute.
>
> Thanks,
> Patrick
>

[math] adding new classes

[Patrick Meyer]

Is it possible to add new classes via substance patches? I have a storeless
covariance class and other classes I would like to contribute.

Thanks,
Patrick

Re: [math] Frequency getUniqueCount

[Patrick Meyer]

Ted and Sebb,

Thanks for all of the help! I look forward to submitting more contributions.

Thanks,
Patrick




>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [math] Frequency getUniqueCount

[Patrick Meyer]

OK, I generated the patch file using Eclipse and it seems to have the
information you described. I'm not sure if the revision part is correct
though. The latest patch is named MATH-448.patch and it is now attached to
the JIRA issue, https://issues.apache.org/jira/browse/MATH-448. I hope this
latest file is what you need.

Thanks,
Patrick




> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [math] Frequency getUniqueCount

[Patrick Meyer]

I originally used Netbeans to generate the Diff, then copy and pasted it
into Wordpad to save it. I just uploaded a new patch that was created with
the Netbeans "Export Diff Patch" function. Is this new file in the correct
format? (Please excuse the noob - I'll get the process right eventually.)



On Mon, Dec 6, 2010 at 11:55 AM, Ted Dunning  wrote:

> How did you create this diff?
>
> The normal way is to use svn diff in the project root directory.
>
> On Mon, Dec 6, 2010 at 8:29 AM, Patrick Meyer  wrote:
>
> > OK, I attached a diff file to,
> > https://issues.apache.org/jira/browse/MATH-448.
> > However, I am new to subversion and JIRA and I'm not sure I have done
> this
> > correctly. Please let me know if I have done something wrong. I am happy
> to
> > volunteer and contribute to the project. I just want to make sure I am
> > using
> > he right process to submit changes.
> >
> > Thanks,
> > Patrick
> >
> >
> >
> > On Mon, Dec 6, 2010 at 10:46 AM, Ted Dunning 
> > wrote:
> >
> > > If you provide a suggested patch, this will go forward much more
> quickly
> > > than
> > > if you wait for a volunteer to appear out of the mists.
> > >
> > > It is after all a volunteer effort.  Go ahead and volunteer!
> > >
> > > On Mon, Dec 6, 2010 at 6:07 AM, Patrick Meyer 
> > wrote:
> > >
> > > > I have a small feature request for the org.apache.commons.math.stat
> > > > Frequency class. It is often useful to know the unique number of
> > elements
> > > > in
> > > > a frequency table. Would you add this feature? I submitted an issue
> > with
> > > > JIRA with the code needed to make this happen,
> > > > https://issues.apache.org/jira/browse/MATH-448.
> > > >
> > > > Thanks
> > > >
> > >
> >
>

Re: [math] Frequency getUniqueCount

[Patrick Meyer]

OK, I attached a diff file to,https://issues.apache.org/jira/browse/MATH-448.
However, I am new to subversion and JIRA and I'm not sure I have done this
correctly. Please let me know if I have done something wrong. I am happy to
volunteer and contribute to the project. I just want to make sure I am using
he right process to submit changes.

Thanks,
Patrick



On Mon, Dec 6, 2010 at 10:46 AM, Ted Dunning  wrote:

> If you provide a suggested patch, this will go forward much more quickly
> than
> if you wait for a volunteer to appear out of the mists.
>
> It is after all a volunteer effort.  Go ahead and volunteer!
>
> On Mon, Dec 6, 2010 at 6:07 AM, Patrick Meyer  wrote:
>
> > I have a small feature request for the org.apache.commons.math.stat
> > Frequency class. It is often useful to know the unique number of elements
> > in
> > a frequency table. Would you add this feature? I submitted an issue with
> > JIRA with the code needed to make this happen,
> > https://issues.apache.org/jira/browse/MATH-448.
> >
> > Thanks
> >
>

[math] Storeless covariance

[Patrick Meyer]

There are now accurate algorithms for an incremental, storeless covariance
that are extentions of West's algorithm for storeless variance. I submitted
the algorithm and technical paper on its derivation in JIRA,
https://issues.apache.org/jira/browse/MATH-449. It would be nice to have
this added to org.apache.commons.math.covariance.

Thanks,
Patrick

[math] Frequency getUniqueCount

[Patrick Meyer]

I have a small feature request for the org.apache.commons.math.stat
Frequency class. It is often useful to know the unique number of elements in
a frequency table. Would you add this feature? I submitted an issue with
JIRA with the code needed to make this happen,
https://issues.apache.org/jira/browse/MATH-448.

Thanks