Re: [math] optimization refactoring
Phil Steitz a écrit : Luc Maisonobe wrote: Hello, Since release of 1.2, several people have asked for advices on using the estimation and optimization packages imported from mantissa. This showed these packages were poorly designed (you can blame me for that). After one of the discussions on this topic, issue MATH-177 (https://issues.apache.org/jira/browse/MATH-177) was set up to ask for a refactoring. I also opened MATH-196 (https://issues.apache.org/jira/browse/MATH-196) for constrained optimization, Gilles contributed Brent minimization (which was appended to MATH-177 and committed recently in the analysis.minimization package), and Benjamin contributed an implementation of Dantzig's simplex method in MATH-246 ((https://issues.apache.org/jira/browse/MATH-246). As we are refactoring many packages for 2.0, I propose to refactor these two packages too. The existing code is : - package o.a.c.math.analysis.minimization: Brent minimization algorithm for univariate scalar valued functions - package o.a.c.math.estimation: Levenberg-Marquardt method for weighted least square minimization of a vector of residuals Gauss-Newton method for weighted least square minimization of a vector of residuals - package o.a.c.math.optimization: Nelder-Mead simplex method for multivariate function minimization without derivatives Virginia Torczon's multidirectional method for multivariate function minimization without derivatives - attached to JIRA issue MATH-246 Dantzig's simplex method for linear systems with linear equality and inequality constraints Some generalization attempts have been made separately in estimation and optimization with a few interfaces (EstimationProblem, Estimator on one side CostFunction and ConvergenceChecker on the other side). These interfaces seems difficult to understand to users. I agree here. They were in fact designed with a specific problem in mind (orbit determination for spacecrafts to be precise) and are not generic enough. Here is what I propose to do : 1) let Brent minimization in the analysis.minimization package where it is now Why not include this in optimization.general below? Gilles proposed to put it in analysis a few weeks ago, on the assumption it is strongly related to roots finding. Of course, it is related to optimization too, so both approaches are logical. In the later case, I would prefer a dedicated package optimization.univariate rather than optimization.general. 2) gather everything else in a package hierarchy: o.a.c.math.optimization with better interfaces (CostFunction may be kept, the other should be replaced) o.a.c.math.optimization.linear for Dantzig's simplex method o.a.c.math.optimization.direct for direct search methods (Nelder-Mead and Torczon) o.a.c.math.optimization.general for everything else 3) some provisions should be made for constrained optimization in the general case, but implementation could probably be delayed until 2.1 (the target revision for MATH-196 is already 2.1) 4) The fancy EstimatedParameter and WeightedMeasurements classes should be dropped (i.e. deprecated) they do not belong to commons-math layer. This is a major change. I think it is better for users to have an easy to understand low level package rather than a big unusable bunch of code. If this option is not chosen, it should be at least possible to use the package at a lower level and provide these classes only as a convenience for those whose needs fit well with this ad-hoc design. +1 5) the possibility to switch a parameter status between should be estimated and should not be touched at all should be removed (I think nobody ever realized it was possible) +0 but I don't think I ever really understood the use cases here. If there is real practical loss of functionality, we should think carefully about this. The original use case was to allow users to try solving a problem first with a set of parameters and later with another set, some parameters being preset to a fixed value. There is a simpler way for users to recover the same functionality without this difficult to understand feature : rebuild the parameters array with a different size in the object representing the problem. This is also safer since once the array is provided to the solver, it is not expect to change size. The current code allow to switch a parameter from bound to free without restriction, even during the run of the solver, which do not expect that. I'm sure this may trigger lots of bugs. 6) the interfaces should define an optimization problem in terms of simple structures (both double[] and RealVector) instead of the dropped parameters and measurements objects. +1 - this is a general topic that applies to other areas of the API as well. I used to think differently, but my own use cases and user comments have convinced me that this approach is both
Re: [math] optimization refactoring
Luc Maisonobe wrote: 1) let Brent minimization in the analysis.minimization package where it is now Why not include this in optimization.general below? Gilles proposed to put it in analysis a few weeks ago, on the assumption it is strongly related to roots finding. Of course, it is related to optimization too, so both approaches are logical. In the later case, I would prefer a dedicated package optimization.univariate rather than optimization.general. I like the idea of optimization.univariate as well since it is more specific - the user knows exactly what they will find in that package. -- View this message in context: http://www.nabble.com/-math--optimization-refactoring-tp22129628p22149219.html Sent from the Commons - Dev mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] optimization refactoring
Hi, The existing code is : - package o.a.c.math.estimation: Levenberg-Marquardt method for weighted least square minimization of a vector of residuals Either one considers the full weighting matrix (including potential correlation between observations) or one does not account for any weight at all. By premultiplying both the function matrix and the observation vector by the square root of the weight matrix, one can forget about it completely in the rest of the computation. So, please, do not restrain weights to a vector. Regards, Dim. Dimitri Pourbaix * Institut d'Astronomie et d'Astrophysique * Don't worry, be happy CP 226, office 2.N4.211, building NO * and CARPE DIEM. Universite Libre de Bruxelles* Boulevard du Triomphe* Tel : +32-2-650.35.71 B-1050 Bruxelles* Fax : +32-2-650.42.26 http://sb9.astro.ulb.ac.be/~pourbaix * mailto:pourb...@astro.ulb.ac.be - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: svn commit: r746804 - /commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
We should note this change of behavior in the changelog. I am OK with
it, but I think we discussed it a while back and decided not to do it
due to concerns that removing the synch could result in hammering the db
engine with concurrent bursts of createConnection requests when load
spikes happen when maxIdle is used. Personally, I don't see a problem
with this, but we should point it out to users. I will also run some
load tests with db engines I have available.
ma...@apache.org wrote:
Author: markt
Date: Sun Feb 22 19:35:59 2009
New Revision: 746804
URL: http://svn.apache.org/viewvc?rev=746804view=rev
Log:
DBCP-212. Remove unnecessary sync. Helps a little with connection churn but the
issue remains.
Modified:
commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
Modified:
commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
URL:
http://svn.apache.org/viewvc/commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java?rev=746804r1=746803r2=746804view=diff
==
---
commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
(original)
+++
commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
Sun Feb 22 19:35:59 2009
@@ -294,7 +294,7 @@
_defaultCatalog = defaultCatalog;
}
-synchronized public Object makeObject() throws Exception {
+public Object makeObject() throws Exception {
Connection conn = _connFactory.createConnection();
if (conn == null) {
throw new IllegalStateException(Connection factory returned null from
createConnection);
-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
Re: svn commit: r746804 - /commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
Phil Steitz wrote: We should note this change of behavior in the changelog. Will do. I'll finish off my work on DBCP-212 first. I don't think there is anything else we can do for DBCP-212 save add some notes to docs. I have patches for this I am about to commit. Mark - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: svn commit: r746804 - /commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java
Mark Thomas wrote: Phil Steitz wrote: We should note this change of behavior in the changelog. Will do. I'll finish off my work on DBCP-212 first. I don't think there is anything else we can do for DBCP-212 save add some notes to docs. I have patches for this I am about to commit. Agreed. Thanks! Phil Mark - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] optimization refactoring
I think that Dmitri overstates his case a bit. This multiplication in observation space works for some algorithms, not for others. Ordinary least squares regression is somewhat of an exception here. Logistic regression is a simple counter-example. It is still useful to have a vector weight and it helps users. It may be useful in some situations to also all a full correlation matrix, but I haven't had a need for that yet. On Sun, Feb 22, 2009 at 11:24 AM, Dimitri Pourbaix pourb...@astro.ulb.ac.be wrote: Either one considers the full weighting matrix (including potential correlation between observations) or one does not account for any weight at all. By premultiplying both the function matrix and the observation vector by the square root of the weight matrix, one can forget about it completely in the rest of the computation. -- Ted Dunning, CTO DeepDyve
Re: svn commit: r746849 - in /commons/proper/dbcp/trunk: src/java/org/apache/commons/dbcp/ src/java/org/apache/commons/jocl/ src/test/ src/test/org/apache/commons/dbcp/ src/test/org/apache/commons/j
On 22/02/2009, pste...@apache.org pste...@apache.org wrote: Author: psteitz Date: Sun Feb 22 23:43:52 2009 New Revision: 746849 URL: http://svn.apache.org/viewvc?rev=746849view=rev Log: Added a connectionInitSqls configuration parameter to BasicDataSource allowing the user to specify a collection of SQL statements to execute one time when a physical database connection is first opened. JIRA: DBCP-175 Thanks to Jiri Melichna and Jerome Lacoste. Modified: commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSourceFactory.java commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java commons/proper/dbcp/trunk/src/java/org/apache/commons/jocl/JOCLContentHandler.java commons/proper/dbcp/trunk/src/test/org/apache/commons/dbcp/TestBasicDataSource.java commons/proper/dbcp/trunk/src/test/org/apache/commons/dbcp/TestBasicDataSourceFactory.java commons/proper/dbcp/trunk/src/test/org/apache/commons/dbcp/TesterStatement.java commons/proper/dbcp/trunk/src/test/org/apache/commons/jocl/TestJOCLContentHandler.java commons/proper/dbcp/trunk/src/test/testpool.jocl commons/proper/dbcp/trunk/xdocs/changes.xml commons/proper/dbcp/trunk/xdocs/configuration.xml Modified: commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java URL: http://svn.apache.org/viewvc/commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java?rev=746849r1=746848r2=746849view=diff == --- commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java (original) +++ commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java Sun Feb 22 23:43:52 2009 @@ -19,6 +19,11 @@ import java.io.PrintWriter; import java.util.Properties; +import java.util.Collection; +import java.util.List; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.Collections; import java.sql.Connection; import java.sql.Driver; import java.sql.DriverManager; @@ -801,6 +806,60 @@ } this.restartNeeded = true; } + +/** + * These SQL statements run once after a Connection is created. + * p + * This property can be used for example to run ALTER SESSION SET + * NLS_SORT=XCYECH in an Oracle Database only once after connection + * creation. + * /p + * + * @since 1.3 + */ +protected List connectionInitSqls; + Why protected and not private? This allows sub-classes to break the thread-safety provided by the synch. get and set methods. - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: svn commit: r746849 - in /commons/proper/dbcp/trunk: src/java/org/apache/commons/dbcp/ src/java/org/apache/commons/jocl/ src/test/ src/test/org/apache/commons/dbcp/ src/test/org/apache/commons/j
sebb wrote: On 22/02/2009, pste...@apache.org pste...@apache.org wrote: Author: psteitz Date: Sun Feb 22 23:43:52 2009 New Revision: 746849 URL: http://svn.apache.org/viewvc?rev=746849view=rev Log: Added a connectionInitSqls configuration parameter to BasicDataSource allowing the user to specify a collection of SQL statements to execute one time when a physical database connection is first opened. JIRA: DBCP-175 Thanks to Jiri Melichna and Jerome Lacoste. Modified: commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSourceFactory.java commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/PoolableConnectionFactory.java commons/proper/dbcp/trunk/src/java/org/apache/commons/jocl/JOCLContentHandler.java commons/proper/dbcp/trunk/src/test/org/apache/commons/dbcp/TestBasicDataSource.java commons/proper/dbcp/trunk/src/test/org/apache/commons/dbcp/TestBasicDataSourceFactory.java commons/proper/dbcp/trunk/src/test/org/apache/commons/dbcp/TesterStatement.java commons/proper/dbcp/trunk/src/test/org/apache/commons/jocl/TestJOCLContentHandler.java commons/proper/dbcp/trunk/src/test/testpool.jocl commons/proper/dbcp/trunk/xdocs/changes.xml commons/proper/dbcp/trunk/xdocs/configuration.xml Modified: commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java URL: http://svn.apache.org/viewvc/commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java?rev=746849r1=746848r2=746849view=diff == --- commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java (original) +++ commons/proper/dbcp/trunk/src/java/org/apache/commons/dbcp/BasicDataSource.java Sun Feb 22 23:43:52 2009 @@ -19,6 +19,11 @@ import java.io.PrintWriter; import java.util.Properties; +import java.util.Collection; +import java.util.List; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.Collections; import java.sql.Connection; import java.sql.Driver; import java.sql.DriverManager; @@ -801,6 +806,60 @@ } this.restartNeeded = true; } + +/** + * These SQL statements run once after a Connection is created. + * p + * This property can be used for example to run ALTER SESSION SET + * NLS_SORT=XCYECH in an Oracle Database only once after connection + * creation. + * /p + * + * @since 1.3 + */ +protected List connectionInitSqls; + Why protected and not private? This allows sub-classes to break the thread-safety provided by the synch. get and set methods. Good point. All of BasicDataSource's properties are set up like this. I was not there at the beginning, but if you look at the earlier revisions, you can see that BasicDataSource was not initially set up to support dynamic reconfiguration so the property getters/setters were not synchronized. Support for dynamic reconfiguration was started in r132132 , but never really completed (the restartNeeded field is never read). It is not clear to me how exactly this would ever work, to be honest. The current setup is a little funny in that some setters have no effect once the pool has been initialized. See comments in the javadoc for e.g. setDriverClassName. None of this really answers your question though. For backward compatability, the existing fields need to remain protected, but I guess we could make this new one private. Phil - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [dbutils] bugfixing/java5 branches ready for merge
Looking at the revisions on the bugfixes branch: 742870 - ?? - Lacking Unit Tests, not liking the catch Exception. RuntimeException throwing needs String arg. Generally not trusting the Java API here to work beautifully and wanting to have covered a bunch of use cases. 743293 - +1 - rename GenericListHandler 743292 - +1 - Minor opt 743278 - +1 - Generics preparation 743269 - +1 - Improved CaseInsensitiveMap 743097 - +1 - Optimization. Small risk of API backwards incompat problems. 742865 - +1 - setNull fix 742704 - +1 - Simple API change 742701 - +1 - Regression bug 742696 - +1 - NPE fix 741987 - +1 - Branching trunk So apart from the one commit, I'm +1 on the changes on the bugfixes branch. Hen On Wed, Feb 11, 2009 at 3:36 AM, Dan Fabulich d...@fabulich.com wrote: I've fixed a bunch of bugs in the bugfixing branch. I think I've fixed all except bug DBUTILS-30/-28 (they're dupes). https://svn.apache.org/repos/asf/commons/sandbox/dbutils/bugfixing http://svn.apache.org/viewvc/commons/sandbox/dbutils/bugfixing/ http://svn.apache.org/viewvc/commons/sandbox/dbutils/bugfixing/?view=log I've also done a bunch of java5-izing (generics/varargs) in the java5 branch, and merged in the bug fixes from the bugfixing branch. https://svn.apache.org/repos/asf/commons/sandbox/dbutils/java5 http://svn.apache.org/viewvc/commons/sandbox/dbutils/java5/ The java5 branch is meant to be exactly like the bugfixing branch, except with generics and varargs. The generics and varargs should be erased at compile time, so the runtime API should be indistinguishable from the runtime API of dbutils-1.2. Review/comments welcome. If I were a commons/dbutils committer right now, I think I'd probably do these things (all of which require committer karma): 1) Merge bugfixing back to trunk 2) Close out all of the bugs I fixed 3) Stage/vote on a dbutils-1.2 release based on bugfixing/trunk 4) Make a dbutils/1.x branch 5) Merge java5 back to trunk 6) Stage/vote on a dbutils-2.0 release based on java5/trunk ... and then start working on crazy API refactorings like my interface bean idea and Liam's RowHolder suggestions, in preparation for a 3.0 release or something. However, I am not, in fact, a commons/dbutils committer, so somewhere in there it might be nice to have a vote about that. ;-) -Dan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
[VFS]HttpConnectionManager
I've been working on getting Commons Configuration to be able to use VFS as its underlying file system and have everything working. However, in the course of this effort I ran into a problem. https://issues.apache.org/jira/browse/VFS-164 states that there were problems under load and so ThreadLocalHttpConnectionManager was written and replaced the use of MultiThreadedHttpConnectionManager. The issue I ran into is that ThreadLocalHttpConnectionManager only allows a single connection open on a thread. This causes some fairly common use cases in Commons Configuration to fail. To get around this I have gone back to MultiThreadedHttpConnectionManager but provided mechanisms to configure maxConnectionsPerHost and maxTotalConnections. Doing this got, along with a few other changes, got everything to work. What I'd like to know is, was there more to VFS-164 than is stated in the issue and is this change sufficient? Or do I need to create yet another HttpConnectionManager? Ralph - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
RE: [VFS]HttpConnectionManager
Hi! -Original Message- From: Ralph Goers [mailto:ralph.go...@dslextreme.com] Sent: Monday, February 23, 2009 7:28 AM What I'd like to know is, was there more to VFS-164 than is stated in the issue and is this change sufficient? Or do I need to create yet another HttpConnectionManager? No, it was just VFS-164 and if you found another solution to that which allows multiple files now it is good! :-) Ciao, Mario - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [VFS]HttpConnectionManager
Thanks Mario. VFS-164 wasn't really clear. Was the problem the limit to 2 connections per host that MultiThreadedHttpConnectionManager has by default? I changed VFS to have it default to 5 and the unit tests pass - including ContentTests.testConcurrentRead() which would hang with only 2 connections. I've added maxConnectionsPerHost and maxTotalConnections to HttpFileSystemConfigBuilder and also allowed them to be specified as system properties. Ralph On Feb 22, 2009, at 11:15 PM, Mario Ivankovits wrote: Hi! -Original Message- From: Ralph Goers [mailto:ralph.go...@dslextreme.com] Sent: Monday, February 23, 2009 7:28 AM What I'd like to know is, was there more to VFS-164 than is stated in the issue and is this change sufficient? Or do I need to create yet another HttpConnectionManager? No, it was just VFS-164 and if you found another solution to that which allows multiple files now it is good! :-) Ciao, Mario - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
