Re: [R] optim seems to be finding a local minimum

[Dimitri Liakhovitski]

One more thing: trying to defend R's honor, I've run optimx instead of
optim (after dividing the IV by its max - same as for optim). I did
not use L-BFGS-B with lower bounds anymore. Instead, I've used
Nelder-Mead (no bounds).
First, it was faster: for a loop across 10 different IVs BFGS took
6.14 sec and Nelder-Mead took just 3.9 sec.
Second, the solution was better - Nelder-Mead fits were ALL better
than L-BFGS-B fits and ALL better than Excel solver's solutions. Of
course, those were small improvements, but still, it's nice!
Dimitri


On Mon, Nov 14, 2011 at 5:26 PM, Dimitri Liakhovitski
 wrote:
> Just to provide some closure:
>
> I ended up dividing the IV by its max so that the input vector (IV) is
> now between zero and one. I still used optim:
> myopt <- optim(fn=myfunc, par=c(1,1), method="L-BFGS-B", lower=c(0,0))
> I was able to get great fit, in 3 cases out of 10 I've beaten Excel
> Solver, but in 7 cases I lost to Excel - but again, by really tiny
> margins (generally less than 1% of Excel's fit value).
>
> Thank you everybody!
> Dimitri
>
> On Fri, Nov 11, 2011 at 10:28 AM, John C Nash  wrote:
>> Some tips:
>>
>> 1) Excel did not, as far as I can determine, find a solution. No point seems 
>> to satisfy
>> the KKT conditions (there is a function kktc in optfntools on R-forge 
>> project optimizer.
>> It is called by optimx).
>>
>> 2) Scaling of the input vector is a good idea given the seeming wide range 
>> of values. That
>> is, assuming this can be done. If the function depends on the relative 
>> values in the input
>> vector rather than magnitude, this may explain the trouble with your 
>> function. That is, if
>> the function depends on the relative change in the input vector and not its 
>> scale, then
>> optimizers will have a lot of trouble if the scale factor for this vector is 
>> implicitly
>> one of the optimization parameters.
>>
>> 3) If you can get the gradient function you will almost certainly be able to 
>> do better,
>> especially in finding whether you have a minimum i.e., null gradient, 
>> positive definite
>> Hessian. When you have gradient function, kktc uses Jacobian(gradient) to 
>> get the Hessian,
>> avoiding one level of digit cancellation.
>>
>> JN
>>
>>
>> On 11/11/2011 10:20 AM, Dimitri Liakhovitski wrote:
>>> Thank you very much to everyone who replied!
>>> As I mentioned - I am not a mathematician, so sorry for stupid
>>> comments/questions.
>>> I intuitively understand what you mean by scaling. While the solution
>>> space for the first parameter (.alpha) is relatively compact (probably
>>> between 0 and 2), the second one (.beta) is "all over the place" -
>>> because it is a function of IV (input vector). And that's, probably,
>>> my main challenge - that I am trying to write a routine for different
>>> possible IVs that I might be facing (they may be in hundreds, in
>>> thousands, in millions). Should I be rescaling the IV somehow (e.g.,
>>> by dividing it by its max) - or should I do something with the
>>> parameter .beta inside my function?
>>>
>>> So far, I've written a loop over many different starting points for
>>> both parameters. Then, I take the betas around the best solution so
>>> far, split it into smaller steps for beta (as starting points) and
>>> optimize again for those starting points. What disappoints me is that
>>> even when I found a decent solution (the minimized value of 336) it
>>> was still worse than the Solver solution!
>>>
>>> And I am trying to prove to everyone here that we should do R, not Excel :-)
>>>
>>> Thanks again for your help, guys!
>>> Dimitri
>>>
>>>
>>> On Fri, Nov 11, 2011 at 9:10 AM, John C Nash  wrote:
 I won't requote all the other msgs, but the latest (and possibly a bit 
 glitchy) version of
 optimx on R-forge

 1) finds that some methods wander into domains where the user function 
 fails try() (new
 optimx runs try() around all function calls). This includes L-BFGS-B

 2) reports that the scaling is such that you really might not expect to 
 get a good solution

 then

 3) Actually gets a better result than the

> xlf<-myfunc(c(0.888452533990788,94812732.0897449))
> xlf
 [1] 334.607
>

 with Kelley's variant of Nelder Mead (from dfoptim package), with

> myoptx
  method                        par       fvalues fns  grs itns conv  KKT1
 4 LBFGSB                     NA, NA 8.988466e+307  NA NULL NULL     NA
 2 Rvmmin           0.1, 200186870.6      25593.83  20    1 NULL    0 FALSE
 3 bobyqa 6.987875e-01, 2.001869e+08      1933.229  44   NA NULL    0 FALSE
 1   nmkb 8.897590e-01, 9.470163e+07      334.1901 204   NA NULL    0 FALSE
   KKT2 xtimes  meths
 4    NA   0.01 LBFGSB
 2 FALSE   0.11 Rvmmin
 3 FALSE   0.24 bobyqa
 1 FALSE   1.08   nmkb

 But do note the terrible scaling. Hardly surprising that this function 
 does not work. I'll
 have to delve deeper to s

Re: [R] optim seems to be finding a local minimum

[Kevin Wright]

Actually, Interval Analysis can be used to find _all_ optima (including the
global optimum) within a starting box.  It's not particularly well-known in
statistical circles.

See this (for example):

http://bib.tiera.ru/ShiZ/math/other/Global%20Optimization%20Using%20Interval%20Analysis%20-%20E.%20Hansen%20(CRC).pdf

Kevin

On Thu, Nov 10, 2011 at 3:20 PM, Rolf Turner  wrote:

> On 11/11/11 08:55, Dimitri Liakhovitski wrote:
>
>> Bert,
>> that's exactly where I started. I found optim in the first paragraph
>> under "General Purpose Continuous Solvers" and used bounded BFGS for a
>> constrained optimization for a situation with more than 1 parameters.
>> Again, not being an engineer / mathematician - would greatly
>> appreciate any pointers.
>>
>
> I'm no expert on numerical optimisation (though I've done quite
> a bit of it in my own ham-fisted way :-) ).  Anyway, it seems to me
> that the only strategy that anyone can use in seeking a global
> optimum when there are multiple local optima is to use a wide
> variety of starting values.  Possibly placed on a (relatively coarse)
> grid.
>
> It can be a tough problem.  Good luck.
>
>cheers,
>
>Rolf Turner
>
>
> __**
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/**listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/**
> posting-guide.html 
> and provide commented, minimal, self-contained, reproducible code.
>



-- 
Kevin Wright

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Dimitri Liakhovitski]

Just to provide some closure:

I ended up dividing the IV by its max so that the input vector (IV) is
now between zero and one. I still used optim:
myopt <- optim(fn=myfunc, par=c(1,1), method="L-BFGS-B", lower=c(0,0))
I was able to get great fit, in 3 cases out of 10 I've beaten Excel
Solver, but in 7 cases I lost to Excel - but again, by really tiny
margins (generally less than 1% of Excel's fit value).

Thank you everybody!
Dimitri

On Fri, Nov 11, 2011 at 10:28 AM, John C Nash  wrote:
> Some tips:
>
> 1) Excel did not, as far as I can determine, find a solution. No point seems 
> to satisfy
> the KKT conditions (there is a function kktc in optfntools on R-forge project 
> optimizer.
> It is called by optimx).
>
> 2) Scaling of the input vector is a good idea given the seeming wide range of 
> values. That
> is, assuming this can be done. If the function depends on the relative values 
> in the input
> vector rather than magnitude, this may explain the trouble with your 
> function. That is, if
> the function depends on the relative change in the input vector and not its 
> scale, then
> optimizers will have a lot of trouble if the scale factor for this vector is 
> implicitly
> one of the optimization parameters.
>
> 3) If you can get the gradient function you will almost certainly be able to 
> do better,
> especially in finding whether you have a minimum i.e., null gradient, 
> positive definite
> Hessian. When you have gradient function, kktc uses Jacobian(gradient) to get 
> the Hessian,
> avoiding one level of digit cancellation.
>
> JN
>
>
> On 11/11/2011 10:20 AM, Dimitri Liakhovitski wrote:
>> Thank you very much to everyone who replied!
>> As I mentioned - I am not a mathematician, so sorry for stupid
>> comments/questions.
>> I intuitively understand what you mean by scaling. While the solution
>> space for the first parameter (.alpha) is relatively compact (probably
>> between 0 and 2), the second one (.beta) is "all over the place" -
>> because it is a function of IV (input vector). And that's, probably,
>> my main challenge - that I am trying to write a routine for different
>> possible IVs that I might be facing (they may be in hundreds, in
>> thousands, in millions). Should I be rescaling the IV somehow (e.g.,
>> by dividing it by its max) - or should I do something with the
>> parameter .beta inside my function?
>>
>> So far, I've written a loop over many different starting points for
>> both parameters. Then, I take the betas around the best solution so
>> far, split it into smaller steps for beta (as starting points) and
>> optimize again for those starting points. What disappoints me is that
>> even when I found a decent solution (the minimized value of 336) it
>> was still worse than the Solver solution!
>>
>> And I am trying to prove to everyone here that we should do R, not Excel :-)
>>
>> Thanks again for your help, guys!
>> Dimitri
>>
>>
>> On Fri, Nov 11, 2011 at 9:10 AM, John C Nash  wrote:
>>> I won't requote all the other msgs, but the latest (and possibly a bit 
>>> glitchy) version of
>>> optimx on R-forge
>>>
>>> 1) finds that some methods wander into domains where the user function 
>>> fails try() (new
>>> optimx runs try() around all function calls). This includes L-BFGS-B
>>>
>>> 2) reports that the scaling is such that you really might not expect to get 
>>> a good solution
>>>
>>> then
>>>
>>> 3) Actually gets a better result than the
>>>
 xlf<-myfunc(c(0.888452533990788,94812732.0897449))
 xlf
>>> [1] 334.607

>>>
>>> with Kelley's variant of Nelder Mead (from dfoptim package), with
>>>
 myoptx
>>>  method                        par       fvalues fns  grs itns conv  KKT1
>>> 4 LBFGSB                     NA, NA 8.988466e+307  NA NULL NULL     NA
>>> 2 Rvmmin           0.1, 200186870.6      25593.83  20    1 NULL    0 FALSE
>>> 3 bobyqa 6.987875e-01, 2.001869e+08      1933.229  44   NA NULL    0 FALSE
>>> 1   nmkb 8.897590e-01, 9.470163e+07      334.1901 204   NA NULL    0 FALSE
>>>   KKT2 xtimes  meths
>>> 4    NA   0.01 LBFGSB
>>> 2 FALSE   0.11 Rvmmin
>>> 3 FALSE   0.24 bobyqa
>>> 1 FALSE   1.08   nmkb
>>>
>>> But do note the terrible scaling. Hardly surprising that this function does 
>>> not work. I'll
>>> have to delve deeper to see what the scaling setup should be because of the 
>>> nature of the
>>> function setup involving some of the data. (optimx includes parscale on all 
>>> methods).
>>>
>>> However, original poster DID include code, so it was easy to do a quick 
>>> check. Good for him.
>>>
>>> JN
>>>
 ## Comparing this solution to Excel Solver solution:
 myfunc(c(0.888452533990788,94812732.0897449))

 -- Dimitri Liakhovitski marketfusionanalytics.com
>>>
>>> __
>>> R-help@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contai

Re: [R] optim seems to be finding a local minimum

[John C Nash]

Some tips:

1) Excel did not, as far as I can determine, find a solution. No point seems to 
satisfy
the KKT conditions (there is a function kktc in optfntools on R-forge project 
optimizer.
It is called by optimx).

2) Scaling of the input vector is a good idea given the seeming wide range of 
values. That
is, assuming this can be done. If the function depends on the relative values 
in the input
vector rather than magnitude, this may explain the trouble with your function. 
That is, if
the function depends on the relative change in the input vector and not its 
scale, then
optimizers will have a lot of trouble if the scale factor for this vector is 
implicitly
one of the optimization parameters.

3) If you can get the gradient function you will almost certainly be able to do 
better,
especially in finding whether you have a minimum i.e., null gradient, positive 
definite
Hessian. When you have gradient function, kktc uses Jacobian(gradient) to get 
the Hessian,
avoiding one level of digit cancellation.

JN


On 11/11/2011 10:20 AM, Dimitri Liakhovitski wrote:
> Thank you very much to everyone who replied!
> As I mentioned - I am not a mathematician, so sorry for stupid
> comments/questions.
> I intuitively understand what you mean by scaling. While the solution
> space for the first parameter (.alpha) is relatively compact (probably
> between 0 and 2), the second one (.beta) is "all over the place" -
> because it is a function of IV (input vector). And that's, probably,
> my main challenge - that I am trying to write a routine for different
> possible IVs that I might be facing (they may be in hundreds, in
> thousands, in millions). Should I be rescaling the IV somehow (e.g.,
> by dividing it by its max) - or should I do something with the
> parameter .beta inside my function?
> 
> So far, I've written a loop over many different starting points for
> both parameters. Then, I take the betas around the best solution so
> far, split it into smaller steps for beta (as starting points) and
> optimize again for those starting points. What disappoints me is that
> even when I found a decent solution (the minimized value of 336) it
> was still worse than the Solver solution!
> 
> And I am trying to prove to everyone here that we should do R, not Excel :-)
> 
> Thanks again for your help, guys!
> Dimitri
> 
> 
> On Fri, Nov 11, 2011 at 9:10 AM, John C Nash  wrote:
>> I won't requote all the other msgs, but the latest (and possibly a bit 
>> glitchy) version of
>> optimx on R-forge
>>
>> 1) finds that some methods wander into domains where the user function fails 
>> try() (new
>> optimx runs try() around all function calls). This includes L-BFGS-B
>>
>> 2) reports that the scaling is such that you really might not expect to get 
>> a good solution
>>
>> then
>>
>> 3) Actually gets a better result than the
>>
>>> xlf<-myfunc(c(0.888452533990788,94812732.0897449))
>>> xlf
>> [1] 334.607
>>>
>>
>> with Kelley's variant of Nelder Mead (from dfoptim package), with
>>
>>> myoptx
>>  methodpar   fvalues fns  grs itns conv  KKT1
>> 4 LBFGSB NA, NA 8.988466e+307  NA NULL NULL NA
>> 2 Rvmmin   0.1, 200186870.6  25593.83  201 NULL0 FALSE
>> 3 bobyqa 6.987875e-01, 2.001869e+08  1933.229  44   NA NULL0 FALSE
>> 1   nmkb 8.897590e-01, 9.470163e+07  334.1901 204   NA NULL0 FALSE
>>   KKT2 xtimes  meths
>> 4NA   0.01 LBFGSB
>> 2 FALSE   0.11 Rvmmin
>> 3 FALSE   0.24 bobyqa
>> 1 FALSE   1.08   nmkb
>>
>> But do note the terrible scaling. Hardly surprising that this function does 
>> not work. I'll
>> have to delve deeper to see what the scaling setup should be because of the 
>> nature of the
>> function setup involving some of the data. (optimx includes parscale on all 
>> methods).
>>
>> However, original poster DID include code, so it was easy to do a quick 
>> check. Good for him.
>>
>> JN
>>
>>> ## Comparing this solution to Excel Solver solution:
>>> myfunc(c(0.888452533990788,94812732.0897449))
>>>
>>> -- Dimitri Liakhovitski marketfusionanalytics.com
>>
>> __
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
> 
> 
>

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Dimitri Liakhovitski]

Thank you very much to everyone who replied!
As I mentioned - I am not a mathematician, so sorry for stupid
comments/questions.
I intuitively understand what you mean by scaling. While the solution
space for the first parameter (.alpha) is relatively compact (probably
between 0 and 2), the second one (.beta) is "all over the place" -
because it is a function of IV (input vector). And that's, probably,
my main challenge - that I am trying to write a routine for different
possible IVs that I might be facing (they may be in hundreds, in
thousands, in millions). Should I be rescaling the IV somehow (e.g.,
by dividing it by its max) - or should I do something with the
parameter .beta inside my function?

So far, I've written a loop over many different starting points for
both parameters. Then, I take the betas around the best solution so
far, split it into smaller steps for beta (as starting points) and
optimize again for those starting points. What disappoints me is that
even when I found a decent solution (the minimized value of 336) it
was still worse than the Solver solution!

And I am trying to prove to everyone here that we should do R, not Excel :-)

Thanks again for your help, guys!
Dimitri


On Fri, Nov 11, 2011 at 9:10 AM, John C Nash  wrote:
> I won't requote all the other msgs, but the latest (and possibly a bit 
> glitchy) version of
> optimx on R-forge
>
> 1) finds that some methods wander into domains where the user function fails 
> try() (new
> optimx runs try() around all function calls). This includes L-BFGS-B
>
> 2) reports that the scaling is such that you really might not expect to get a 
> good solution
>
> then
>
> 3) Actually gets a better result than the
>
>> xlf<-myfunc(c(0.888452533990788,94812732.0897449))
>> xlf
> [1] 334.607
>>
>
> with Kelley's variant of Nelder Mead (from dfoptim package), with
>
>> myoptx
>  method                        par       fvalues fns  grs itns conv  KKT1
> 4 LBFGSB                     NA, NA 8.988466e+307  NA NULL NULL     NA
> 2 Rvmmin           0.1, 200186870.6      25593.83  20    1 NULL    0 FALSE
> 3 bobyqa 6.987875e-01, 2.001869e+08      1933.229  44   NA NULL    0 FALSE
> 1   nmkb 8.897590e-01, 9.470163e+07      334.1901 204   NA NULL    0 FALSE
>   KKT2 xtimes  meths
> 4    NA   0.01 LBFGSB
> 2 FALSE   0.11 Rvmmin
> 3 FALSE   0.24 bobyqa
> 1 FALSE   1.08   nmkb
>
> But do note the terrible scaling. Hardly surprising that this function does 
> not work. I'll
> have to delve deeper to see what the scaling setup should be because of the 
> nature of the
> function setup involving some of the data. (optimx includes parscale on all 
> methods).
>
> However, original poster DID include code, so it was easy to do a quick 
> check. Good for him.
>
> JN
>
>> ## Comparing this solution to Excel Solver solution:
>> myfunc(c(0.888452533990788,94812732.0897449))
>>
>> -- Dimitri Liakhovitski marketfusionanalytics.com
>
> __
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



-- 
Dimitri Liakhovitski
marketfusionanalytics.com

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[John C Nash]

I won't requote all the other msgs, but the latest (and possibly a bit glitchy) 
version of
optimx on R-forge

1) finds that some methods wander into domains where the user function fails 
try() (new
optimx runs try() around all function calls). This includes L-BFGS-B

2) reports that the scaling is such that you really might not expect to get a 
good solution

then

3) Actually gets a better result than the

> xlf<-myfunc(c(0.888452533990788,94812732.0897449))
> xlf
[1] 334.607
>

with Kelley's variant of Nelder Mead (from dfoptim package), with

> myoptx
  methodpar   fvalues fns  grs itns conv  KKT1
4 LBFGSB NA, NA 8.988466e+307  NA NULL NULL NA
2 Rvmmin   0.1, 200186870.6  25593.83  201 NULL0 FALSE
3 bobyqa 6.987875e-01, 2.001869e+08  1933.229  44   NA NULL0 FALSE
1   nmkb 8.897590e-01, 9.470163e+07  334.1901 204   NA NULL0 FALSE
   KKT2 xtimes  meths
4NA   0.01 LBFGSB
2 FALSE   0.11 Rvmmin
3 FALSE   0.24 bobyqa
1 FALSE   1.08   nmkb

But do note the terrible scaling. Hardly surprising that this function does not 
work. I'll
have to delve deeper to see what the scaling setup should be because of the 
nature of the
function setup involving some of the data. (optimx includes parscale on all 
methods).

However, original poster DID include code, so it was easy to do a quick check. 
Good for him.

JN

> ## Comparing this solution to Excel Solver solution:
> myfunc(c(0.888452533990788,94812732.0897449))
> 
> -- Dimitri Liakhovitski marketfusionanalytics.com

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Ben Bolker]

Hans W Borchers  googlemail.com> writes:

> 
> Ben Bolker  gmail.com> writes:
> 
> > 
> >   Simulated annealing and other stochastic global optimization 
> > methods are also possible solutions, although they may or may not
> > work better than the many-starting-points solution -- it depends
> > on the problem, and pretty much everything has to be tuned.  Tabu
> > search  is another possibility,
> > although I don't know much about it ...
> > 
> 
> It is known that the Excel Solver has much improved during recent years.
> Still there are slightly better points such as
> 
> myfunc(c(0.889764228112319, 94701144.5712312))   # 334.18844
> 
> restricting the domain to [0, 1] x [0, 10^9] for an evolutionary approach,
> for instance DEoptim::DEoptim().
> 
> Finding a global optimum in 2 dimensions is not so difficult. Here the scale
> of the second variable could pose a problem as small local minima might be
> overlooked easily.
> 

  Have taken a (quick) second look at the problem, I agree that scaling
and centering are more likely to be useful solutions than stochastic
global optimization stuff.  Even using 'parscale' (see the optim
help page) may clear up the problem.

  Ben Bolker

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Hans W Borchers]

Ben Bolker  gmail.com> writes:

> 
>   Simulated annealing and other stochastic global optimization 
> methods are also possible solutions, although they may or may not
> work better than the many-starting-points solution -- it depends
> on the problem, and pretty much everything has to be tuned.  Tabu
> search  is another possibility,
> although I don't know much about it ...
> 

It is known that the Excel Solver has much improved during recent years.
Still there are slightly better points such as

myfunc(c(0.889764228112319, 94701144.5712312))   # 334.18844

restricting the domain to [0, 1] x [0, 10^9] for an evolutionary approach,
for instance DEoptim::DEoptim().

Finding a global optimum in 2 dimensions is not so difficult. Here the scale
of the second variable could pose a problem as small local minima might be
overlooked easily.

Hans Werner

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Ben Bolker]

Rolf Turner  xtra.co.nz> writes:

> 
> On 11/11/11 08:55, Dimitri Liakhovitski wrote:
> > Bert,
> > that's exactly where I started. I found optim in the first paragraph
> > under "General Purpose Continuous Solvers" and used bounded BFGS for a
> > constrained optimization for a situation with more than 1 parameters.
> > Again, not being an engineer / mathematician - would greatly
> > appreciate any pointers.
> 
> I'm no expert on numerical optimisation (though I've done quite
> a bit of it in my own ham-fisted way  ).  Anyway, it seems to me
> that the only strategy that anyone can use in seeking a global
> optimum when there are multiple local optima is to use a wide
> variety of starting values.  Possibly placed on a (relatively coarse)
> grid.

  Simulated annealing and other stochastic global optimization 
methods are also possible solutions, although they may or may not
work better than the many-starting-points solution -- it depends
on the problem, and pretty much everything has to be tuned.  Tabu
search  is another possibility,
although I don't know much about it ...

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Rolf Turner]

On 11/11/11 08:55, Dimitri Liakhovitski wrote:

Bert,
that's exactly where I started. I found optim in the first paragraph
under "General Purpose Continuous Solvers" and used bounded BFGS for a
constrained optimization for a situation with more than 1 parameters.
Again, not being an engineer / mathematician - would greatly
appreciate any pointers.


I'm no expert on numerical optimisation (though I've done quite
a bit of it in my own ham-fisted way :-) ).  Anyway, it seems to me
that the only strategy that anyone can use in seeking a global
optimum when there are multiple local optima is to use a wide
variety of starting values.  Possibly placed on a (relatively coarse)
grid.

It can be a tough problem.  Good luck.

cheers,

Rolf Turner

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Dimitri Liakhovitski]

Bert,
that's exactly where I started. I found optim in the first paragraph
under "General Purpose Continuous Solvers" and used bounded BFGS for a
constrained optimization for a situation with more than 1 parameters.
Again, not being an engineer / mathematician - would greatly
appreciate any pointers.
Thank you, everyone!
Dimitri

On Thu, Nov 10, 2011 at 2:39 PM, Bert Gunter  wrote:
> Refer to the CRAN "Optimization" task view, please. That is a much more
> appropriate place to begin than posting a query here.
>
> All numerical optimizers only produce local optima.
>
> -- Bert
>
> On Thu, Nov 10, 2011 at 11:24 AM, Dimitri Liakhovitski
>  wrote:
>>
>> Just to add:
>>
>> I also experimented with the starting parameters (par) under optim,
>> especially with the second one. I tried 1, 10, 100, 1000, etc.
>> When I tried 100,000,000 then I got a somewhat better solution (but
>> still not as good as in Excel). However, under message it said:
>>
>> "ERROR: ABNORMAL_TERMINATION_IN_LNSRCH"
>>
>> Dimitri
>>
>> On Thu, Nov 10, 2011 at 1:50 PM, Dimitri Liakhovitski
>>  wrote:
>> > Hello!
>> >
>> > I am trying to create an R optimization routine for a task that's
>> > currently being done using Excel (lots of tables, formulas, and
>> > Solver).
>> > However, otpim seems to be finding a local minimum.
>> > Example data, functions, and comparison with the solution found in
>> > Excel are below.
>> > I am not experienced in optimizations so thanks a lot for your advice!
>> > Dimitri
>> >
>> > ### 2 Inputs:
>> >
>> > IV<-data.frame(IV=c(0,6672895.687,13345791.37,20018687.06,26691582.75,33364478.44,40037374.12,46710269.81,53383165.5,60056061.18,66728956.87,73401852.56,80074748.24,86747643.93,93420539.62,100093435.3,106766331,113439226.7,120112122.4,126785018.1,133457913.7,140130809.4,146803705.1,153476600.8,160149496.5,166822392.2,173495287.9,180168183.5,186841079.2,193513974.9,200186870.6))
>> >
>> > DV<-data.frame(DV=c(0,439.8839775,829.7360945,1176.968757,1487.732038,1767.147276,2019.49499,2248.366401,2456.78592,2647.310413,2822.109854,2983.033036,3131.661246,3269.352233,3397.276321,3516.446162,3627.741311,3731.928591,3829.679009,3921.581866,4008.156537,4089.862363,4167.106955,4240.253215,4309.625263,4375.513474,4438.178766,4497.856259,4554.75841,4609.077705,4660.988983))
>> >
>> > ## Function "transformIV" transforms a data frame column "IV" using
>> > parameters .alpha & .beta:
>> > ## It returns a data frame column IV_transf:
>> > transformIV = function(.alpha,.beta) {
>> >  IV_transf <- as.data.frame(1 - (1/exp((IV/.beta)^.alpha)))
>> >  return(IV_transf)
>> > }
>> >
>> > ### Function "mysum" calculates the sum of absolute residuals after a
>> > regression with a single predictor:
>> > mysum<- function(myIV,myDV){
>> >  regr<-lm(myDV[[1]] ~ 0 + myIV[[1]])
>> >  mysum<-sum(abs(regr$resid))
>> >  return(mysum)
>> > }
>> >
>> > ### Function to be optimized;
>> > ### param is a vector of 2 values (.alpha and .beta)
>> > myfunc <- function(param){
>> >  myalpha<-param[1]
>> >  mybeta<-param[2]
>> >  IVtransf<-transformIV(myalpha, mybeta)
>> >  sumofdevs<-mysum(myIV=IVtransf,myDV=DV)
>> >  return(sumofdevs)
>> > }
>> >
>> > # Optimizing using optim:
>> > myopt <- optim(fn=myfunc, par=c(0.1,max(IV)), method="L-BFGS-B",
>> > lower=0)
>> > (myopt)
>> > myfunc(myopt$par)
>> > ## Comparing this solution to Excel Solver solution:
>> > myfunc(c(0.888452533990788,94812732.0897449))
>> >
>> > --
>> > Dimitri Liakhovitski
>> > marketfusionanalytics.com
>> >
>>
>>
>>
>> --
>> Dimitri Liakhovitski
>> marketfusionanalytics.com
>>
>> __
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>
>
> --
>
> Bert Gunter
> Genentech Nonclinical Biostatistics
>
> Internal Contact Info:
> Phone: 467-7374
> Website:
> http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm
>
>



-- 
Dimitri Liakhovitski
marketfusionanalytics.com

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Bert Gunter]

Refer to the CRAN "Optimization" task view, please. That is a much more
appropriate place to begin than posting a query here.

All numerical optimizers only produce local optima.

-- Bert

On Thu, Nov 10, 2011 at 11:24 AM, Dimitri Liakhovitski <
dimitri.liakhovit...@gmail.com> wrote:

> Just to add:
>
> I also experimented with the starting parameters (par) under optim,
> especially with the second one. I tried 1, 10, 100, 1000, etc.
> When I tried 100,000,000 then I got a somewhat better solution (but
> still not as good as in Excel). However, under message it said:
>
> "ERROR: ABNORMAL_TERMINATION_IN_LNSRCH"
>
> Dimitri
>
> On Thu, Nov 10, 2011 at 1:50 PM, Dimitri Liakhovitski
>  wrote:
> > Hello!
> >
> > I am trying to create an R optimization routine for a task that's
> > currently being done using Excel (lots of tables, formulas, and
> > Solver).
> > However, otpim seems to be finding a local minimum.
> > Example data, functions, and comparison with the solution found in
> > Excel are below.
> > I am not experienced in optimizations so thanks a lot for your advice!
> > Dimitri
> >
> > ### 2 Inputs:
> >
> IV<-data.frame(IV=c(0,6672895.687,13345791.37,20018687.06,26691582.75,33364478.44,40037374.12,46710269.81,53383165.5,
> 60056061.18,66728956.87,73401852.56,80074748.24,86747643.93
> ,93420539.62,100093435.3,106766331,113439226.7,120112122.4,126785018.1,133457913.7,140130809.4,146803705.1,153476600.8,160149496.5,166822392.2,173495287.9,180168183.5,186841079.2,193513974.9,200186870.6))
> > DV<-data.frame(DV=c(0,439.8839775,829.7360945
> ,1176.968757,1487.732038,1767.147276,2019.49499,2248.366401
> ,2456.78592,2647.310413,2822.109854,2983.033036,3131.661246,3269.352233,
> 3397.276321,3516.446162
> ,3627.741311,3731.928591,3829.679009,3921.581866,4008.156537,4089.862363,
> 4167.106955,4240.253215,4309.625263,4375.513474,4438.178766
> ,4497.856259,4554.75841,4609.077705,4660.988983))
> >
> > ## Function "transformIV" transforms a data frame column "IV" using
> > parameters .alpha & .beta:
> > ## It returns a data frame column IV_transf:
> > transformIV = function(.alpha,.beta) {
> >  IV_transf <- as.data.frame(1 - (1/exp((IV/.beta)^.alpha)))
> >  return(IV_transf)
> > }
> >
> > ### Function "mysum" calculates the sum of absolute residuals after a
> > regression with a single predictor:
> > mysum<- function(myIV,myDV){
> >  regr<-lm(myDV[[1]] ~ 0 + myIV[[1]])
> >  mysum<-sum(abs(regr$resid))
> >  return(mysum)
> > }
> >
> > ### Function to be optimized;
> > ### param is a vector of 2 values (.alpha and .beta)
> > myfunc <- function(param){
> >  myalpha<-param[1]
> >  mybeta<-param[2]
> >  IVtransf<-transformIV(myalpha, mybeta)
> >  sumofdevs<-mysum(myIV=IVtransf,myDV=DV)
> >  return(sumofdevs)
> > }
> >
> > # Optimizing using optim:
> > myopt <- optim(fn=myfunc, par=c(0.1,max(IV)), method="L-BFGS-B", lower=0)
> > (myopt)
> > myfunc(myopt$par)
> > ## Comparing this solution to Excel Solver solution:
> > myfunc(c(0.888452533990788,94812732.0897449))
> >
> > --
> > Dimitri Liakhovitski
> > marketfusionanalytics.com
> >
>
>
>
> --
> Dimitri Liakhovitski
> marketfusionanalytics.com
>
> __
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



-- 

Bert Gunter
Genentech Nonclinical Biostatistics

Internal Contact Info:
Phone: 467-7374
Website:
http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] optim seems to be finding a local minimum

[Dimitri Liakhovitski]

Just to add:

I also experimented with the starting parameters (par) under optim,
especially with the second one. I tried 1, 10, 100, 1000, etc.
When I tried 100,000,000 then I got a somewhat better solution (but
still not as good as in Excel). However, under message it said:

"ERROR: ABNORMAL_TERMINATION_IN_LNSRCH"

Dimitri

On Thu, Nov 10, 2011 at 1:50 PM, Dimitri Liakhovitski
 wrote:
> Hello!
>
> I am trying to create an R optimization routine for a task that's
> currently being done using Excel (lots of tables, formulas, and
> Solver).
> However, otpim seems to be finding a local minimum.
> Example data, functions, and comparison with the solution found in
> Excel are below.
> I am not experienced in optimizations so thanks a lot for your advice!
> Dimitri
>
> ### 2 Inputs:
> IV<-data.frame(IV=c(0,6672895.687,13345791.37,20018687.06,26691582.75,33364478.44,40037374.12,46710269.81,53383165.5,60056061.18,66728956.87,73401852.56,80074748.24,86747643.93,93420539.62,100093435.3,106766331,113439226.7,120112122.4,126785018.1,133457913.7,140130809.4,146803705.1,153476600.8,160149496.5,166822392.2,173495287.9,180168183.5,186841079.2,193513974.9,200186870.6))
> DV<-data.frame(DV=c(0,439.8839775,829.7360945,1176.968757,1487.732038,1767.147276,2019.49499,2248.366401,2456.78592,2647.310413,2822.109854,2983.033036,3131.661246,3269.352233,3397.276321,3516.446162,3627.741311,3731.928591,3829.679009,3921.581866,4008.156537,4089.862363,4167.106955,4240.253215,4309.625263,4375.513474,4438.178766,4497.856259,4554.75841,4609.077705,4660.988983))
>
> ## Function "transformIV" transforms a data frame column "IV" using
> parameters .alpha & .beta:
> ## It returns a data frame column IV_transf:
> transformIV = function(.alpha,.beta) {
>  IV_transf <- as.data.frame(1 - (1/exp((IV/.beta)^.alpha)))
>  return(IV_transf)
> }
>
> ### Function "mysum" calculates the sum of absolute residuals after a
> regression with a single predictor:
> mysum<- function(myIV,myDV){
>  regr<-lm(myDV[[1]] ~ 0 + myIV[[1]])
>  mysum<-sum(abs(regr$resid))
>  return(mysum)
> }
>
> ### Function to be optimized;
> ### param is a vector of 2 values (.alpha and .beta)
> myfunc <- function(param){
>  myalpha<-param[1]
>  mybeta<-param[2]
>  IVtransf<-transformIV(myalpha, mybeta)
>  sumofdevs<-mysum(myIV=IVtransf,myDV=DV)
>  return(sumofdevs)
> }
>
> # Optimizing using optim:
> myopt <- optim(fn=myfunc, par=c(0.1,max(IV)), method="L-BFGS-B", lower=0)
> (myopt)
> myfunc(myopt$par)
> ## Comparing this solution to Excel Solver solution:
> myfunc(c(0.888452533990788,94812732.0897449))
>
> --
> Dimitri Liakhovitski
> marketfusionanalytics.com
>



-- 
Dimitri Liakhovitski
marketfusionanalytics.com

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.