Hi.
On Mon, 21 Sep 2015 19:55:15 -0500, Ole Ersoy wrote:
Hola,
On 09/21/2015 04:15 PM, Gilles wrote:
Hi.
On Sun, 20 Sep 2015 15:04:08 -0500, Ole Ersoy wrote:
On 09/20/2015 05:51 AM, Gilles wrote:
On Sun, 20 Sep 2015 01:12:49 -0500, Ole Ersoy wrote:
Wanted to float some ideas for the LeastSquaresOptimizer
(Possibly
General Optimizer) design. For example with the
LevenbergMarquardtOptimizer we would do:
`LevenbergMarquardtOptimizer.optimize(OptimizationContext c);`
Rough optimize() outline:
public static void optimise() {
//perform the optimization
//If successful
c.notify(LevenberMarquardtResultsEnum.SUCCESS, solution);
//If not successful
c.notify(LevenberMarquardtResultsEnum.TOO_SMALL_COST_RELATIVE_TOLERANCE,
diagnostic);
//or
c.notify(LevenberMarquardtResultsEnum.TOO_SMALL_PARAMETERS_RELATIVE_TOLERANCE,
diagnostic)
//etc
}
The diagnostic, when turned on, will contain a trace of the last
N
iterations leading up to the failure. When turned off, the
Diagnostic
instance only contains the parameters used to detect failure. The
diagnostic could be viewed as an indirect way to log optimizer
iterations.
WDYT?
I'm wary of having several different ways to convey information to
the
caller.
It would just be one way.
One way for optimizer, one way for solvers, one way for ...
Yes I see what you mean, but I think on a whole it will be worth it
to add additional sugar code that removes the need for exceptions.
Isn't always possible to wrap exception-generating code so that upper
layers do not see them?
The interface would have to know how to handle them and propagate the
information in some other form (callback).
But the caller may not be the receiver
(It could be). The receiver would be an observer attached to the
OptimizationContext that implements an interface allowing it to
observe
the optimization.
I'm afraid that it will add to the questions of what to put in the
code and how. [We already had sometimes heated discussions just for
the IMHO obvious (e.g. code formatting, documentation,
exception...).]
Hehe. Yes I remember some of these discussions. I wonder how much
time was spent debating the exceptions alone? Surely everyone must
have had this feeling in pit of their stomach that there's got to be
a
better way. On the exception topic, these are some of the issues:
I18N
===================
If you are new to commons math and thinking about designing a commons
math compatible exception you should probably understand the I18N
stuff that's bound to exception (and wonder why it's bound the the
exception). Grab a coffee and spend a few hours, unless you are
obviously fairly new to Java like some ofthe people posting for help.
In this case when the exception occurs, there is going to be a lot of
tutoring going on on the users list.
I already said all I had to say about this; it's in the archive.
Summary: I agree that it shouldn't be here.
Number of Exceptions
===================
Before you do actually design a new exception, you should probably
see if there is an exception that already fits the category of what
you are doing. So you start reading. Exception1...nop
Exception2...nop...Exception3...Exception999..But I think I'm getting
warmer. OK - Did not find it ... but I'm fairly certain that there
is
a elegant place for it somewhere in the exception hierarchy...
On this, I also explained at length my views (assuming that exceptions
are part of the design).
Summary: an exception indicates that something went wrong, and the
caller
should not hope to get anything good out of the call that raised the
exception (i.e. he _must_ craft another call that meets the
requirements
of the code).
Handling of Exceptions
===================
If our app uses several of the commons math classes (That throw
exceptions of the same type), and one of those classes throws an
exception,what is the app supposed to do?
Cf. previous paragraph.
I think most developers would find that question somewhat
challenging. There are numerous strategies. Catch all exceptions
and
log what happened, etc. But what if the requirement is that if an
exception is thrown, the organization that receives it has 0 seconds
to get to the root cause of it and understand the dynamics. Is this
doable? (Yes obviously, but how hard is it...?).
Cf. previous paragraph.
In effect, you describe an upper layer's requirement (handling an
expected
"unexpected(!) failure"). IMHO, it's out CM's realm (CM raises the
exception,
end of story).
It seems that the reporting interfaces could quickly overwhelm
the "actual" code (one type of context per algorithm).
There would one type of Observer interface per algorithm. It would
act on the solution and what are currently exceptions, although
these
would be translated into enums.
Unless I'm mistaken, the most common use-case for codes implemented
in a library such as CM is to provide a correct answer or bail out
in a non-equivocal way.
Most java developers are used to synchronous coding...call the method
get the response...catch the exception if needed. This is changing
with JDK8, and as we evolve and start using lambdas, we become more
accustomed to the functional callback style of programming.
Personally I want to be able to use an API that gives me what I need
when everything works as expected, allows me to resolve unexpected
issues with minimal effort, and is as simple, fluid, and lightweight
as possible.
I've not yet used Java 8; I would have if we were allowed to use it in
CM...
However, I'm not convinced that asynchronicity should be dealt with
at the CM level, beyond making its algorithms multi-thread friendly.
IMO, this is the important change (that can make a big difference,
performance-wise, on machines with multiple cores).
Then developers can use the standard tools in "java.util.concurrent"
to select a runtime policy (single/multi-thread and/or (a)synchronous).
It would make the code more involved to handle a minority of
(undefined) cases. [Actual examples would be welcome in order to
focus the discussion.]
Rough Outline (I've evolved the concept and moved away from the
OptimizationContext in the process of writing):
interface LevenbergMarquardtObserver {
public void hola(Solution s);
public void sugarHoneyIceTea(ResultType rt, Dianostics d);
}
public class LMObserver implements LevenbergMarquardtObserver {
private Application application;
public LMObserver(Application application) {
this.application = application;
}
public void hola(ResultType rt, Solution s) {
application.next(solution);
}
public void sugarHoneyIceTea(ResultType rt, Diagnostic s)
if (rt == ResultType.I_GOT_THIS_ONE) {
//I looked at the commons unit tests for this algorithm
evaluating
//the diagnostics that shows how this failure can occur
//I'm totally fixing this! Steps aside!
}
else if (rt == ResultType.REALLY_COMPLICATED_STUFF)
{
//We need our best engineers...call India.
}
)
public class Application {
//Note nothing is returned.
LevenberMarquardtOptimizer.setOberver(new
LMObserver(this)).setLeastSquaresProblem(new
ClassThatImplementsTheProblem())).start();
public void next(Solution solution) {
//Do cool stuff.
}
}
Or an asynchronous variation:
public class Application {
//This call will not block because async is true
LevenberMarquardtOptimizer.setAsync(true).setOberver(new
LMObserver()).setLeastSquaresProblem(new
ClassThatImplementsTheProblem())).start();
//Do more stuff right away.
public void next(Solution solution) {
//When the thread running the optimization is done, this
method is called back.
//Do whatever comes next
}
}
The above would start the optimization in a separate thread that does
not / SHOULD NOT share data with the main thread.
Cf. previous paragraph: I think that can be done in a layer above CM.
The current reporting is based on exceptions, and assumes that if
no
exception was thrown, then the user's request completed
successfully.
Sure - personally I'd much rather deal with something similar to an
HTTP status code in a callback, than an exception . I think the
code
is cleaner and the calback makes it more elegant to apply an
adaptive
approach to handling the response, like slightly relaxing
constraints,
convergence parameters, etc. Also by getting rid of the
exceptions,
we no longer depend on the I18N layer that they are tied to and now
the messages can be more informative, since they target the root
cause. The observer can also run in the 'main' thread' while the
optimization can run asynchronously. Also WRT JDK9 and modules,
loosing the exceptions would mean one less dependency when the
library
is up into JDK9 modules...which would be more in line with this
philosophy:
https://github.com/substack/browserify-handbook#module-philosophy
I'm not sure I fully understood the philosophy from the text in this
short paragraph.
But I do not agree with the idea that the possibility to quickly
find
some code is more important than standards and best practices.
If you go to npmjs.org and type in Neural Network you will get 56
results all linked to github repositories.
In addition there's meta data indicating number of downloads in the
last day, last month, etc. Try typing in cosine. Odds are you will
find a package that does just want you want and nothing else. This
is
very underwhelming and refreshing in terms of cloning off of github
and getting familar with tests etc. Also eye opening. How many of
us
knew that we could do that much stuff with cosine! :).
I really don't mean to question the quality of any of those
implementations,
but the issue is there: How to choose?
That there are so many of them sort of defeats the purpose of "quickly
find what you need".
It seems (?) that the consequence of this modularity (?) is to
encourage
the creation of many independent/competing/duplicate projects of small
teams (I'd guess, a 1-person-team, in most cases).
I totally agree that in some circumstances, more information on
the
inner working of an algorithm would be quite useful.
... Algorithm iterations become unit testable.
But I don't see the point in devoting resources to reinvent the
wheel:
You mean pimping the wheel? Big pimpin.
I think that logging statements are easy to add, not disruptive at
all,
and come in handy to understand a code's unexpected behaviour.
Assuming that a "logging" feature is useful, it can be added *now*
using
a dependency towards a weight-less (!) framework such as "slf4j".
IMO, it would be a waste of time to implement a new communication
layer
that can do that, and more, if it would be used for logging only in
99%
of the cases.
SLF4J is used by almost every other framework, so why not use it?
Good question: I also asked it quite some time ago.
Didn't get a satisfying answer. Boiled down to "no dependency" policy.
Logging and the diagnostic could be used together. The primary
purpose of the diagnostic though is to collect data that will be
useful in `sugarHoneyIceTea`.
I'm not sure I understand correctly the purpose: if the "Solution" is
found, do you ever need more "context" (i.e. "Result", "Diagnostics")?
If it is only necessary in case of failure, CM's exception can already
carry context information. As I wrote above, such an exception could
be caught by a wrapper (not necessarily part of the CM "core") and
translated into whatever the upper layer expect (e.g. "Diagnostics").
I longed several times for the use of a logging library.
The only show-stopper has been the informal "no-dependency"
policy...
JDK9 Jigsaw should solve dependency hell, so the less coupling
between commons math classes the better.
I wouldn't call "coupling" the dependency towards exception classes:
they are little utilities that can make sense in various parts of
the
library.
If for example the Simplex solver is broken off into it's own module,
then it has to be coupled to the exceptions, unless it is exception
free.
Why is it a problem to be coupled with a few tiny exception classes?
Then if it is really a problem, we can indeed define "local" exceptions
for each package.
[Unless one wants to embark on yet another discussion about
exceptions;
whether there should be one class for each of the "messages" that
exist
in "LocalizedFormats"; whether localization should be done in CM;
etc.]
I think it would be best to just eliminate the exceptions.
I'd think that most users of CM should deem that dangerous.
An exception is relatively difficult to ignore unknowingly (and was
rightfully a better alternative the old "check the return value").
Anyways I'm obviously
interested in playing with this stuff, so when I get something up
into
a repository I'll to do a callback :).
If you are interested in big overhauls, there is one that gathered
relative consensus: rewrite the algorithms in a
"multithread-friendly"
way.
I think that's a tall order that will take us into JDK88 :).
That would be a real pity.
I recall a nit-picking discussion about how to initialize the
"FastMath"
class in order to gain a few _milliseconds_. :-/
But
using callbacks and making potentially long running computations
asynchronous could be a middle ground that would allow simple multi
threaded use without fiddling around under the hood...
Cf. above (this does not need ad-hoc CM code, beyond the relevant
classes
implementing "Runnable" and/or "Callable").
Some ideas were floated (cf. ML archive) but no implementation or
experiment... Perhaps with a well-defined goal such as performance
improvement, your design suggestions will become clearer to more
people.
AFAIK, only the classes in the "o.a.c.m.neuralnet" package are
currently
ready to be used with the "java.util.concurrent" framework.
FWIU Neural Nets are a great fit for concurrency.
Quite true.
But even the optimizers could benefit from just being able to use
more threads: It is often (always?) necessary to evaluate the objective
function "N" times per iteration. So, the computation could be about
min(N, numCores)
times faster.
I think for the
others we will end up having discussions around how users would
control the number of threads, etc. again that makes some of us
nervous.
One additional parameters: numCores.
An asynchronous operation that runs in one separate thread
is easier to reason about.
Sure.
But then we should stop talking about performance on this list. ;-}
If we want to test 10 neural net
configurations, and we have 10 cores, then we can start each by
itself
by doing something like:
Nework.setAsync(true).addNeurons().connectNeurons().addObserver(observer).start().
//Now do 10 more
//If the observer is shared then notifications should be thread safe.
I had a similar argument for not making "FastMath" initialization
faster
(at the cost of a lot of additional code): It was rejected...
Regards,
Gilles
P.S. I think that several issues evoked in this thread could warrant
opening
their own thread, to gather more opinions on actual actions to be
taken.
Cheers,
- Ole
P.S. Dang that was a long email. If I write one more of these, ban
me :)
My fault: I should not keep answering! ;-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
