Paul, Let's take this offline ... Tell me what you have in mind via email and I'll consider it ...
----- Original Message ----- From: Paul Ho Date: Wednesday, May 7, 2008 12:28 pm Subject: RE: [amibroker] Re: Fitness Criteria that incorporates Walk Forward Result To: amibroker@yahoogroups.com > The purpose of OOS is to make sure there is no over fitting of > data. So all > optimization is done on In sample data. It is always possible to > have more > than 1 set of optimized parameters, because of different > markets, or > different selection of stocks, different parameters being > optimized or even > different fitness criteria. I currently run my optimzation > nearly 3000 > tickers, generating thousands of trades. There needs to be a > systematic way > of choosing the "right" system and I strongly argue that OOS has > a major > role in that. I think it is not invalidating the OOS because, > the amount of > data mining is very small compared to insample. I think by just > examing it > casually, OOS is under utilised, at least in my case. From what > I hear, a > lot of people only optimize on individual or just a few tickers > and the > degree of freedom is comparatively low. What I do here is to run > optimization over the whole ASX exchange, past and present. > Finally, I think you should consider automating the process in > IO and allow > the user a choice. > Cheers > Paul > > > _____ > > From: amibroker@yahoogroups.com > [mailto:[EMAIL PROTECTED] On Behalf > Of Fred Tonetti > Sent: Thursday, 8 May 2008 12:12 AM > To: amibroker@yahoogroups.com > Subject: RE: [amibroker] Re: Fitness Criteria that incorporates > Walk Forward > Result > > > > > > > Paul, > > > > One other word of caution . > > > > If you are using OOS testing to drive the selection process of > parameters to > be used in sample then you run the risk of invalidating the OOS. > > > > I could have automated this process in IO but I didn't for > exactly this > reason. > > > > > _____ > > > From: amibroker@yahoogroups.com > [mailto:[EMAIL PROTECTED] On Behalf > Of Paul Ho > Sent: Wednesday, May 07, 2008 8:37 AM > To: amibroker@yahoogroups.com > Subject: [amibroker] Re: Fitness Criteria that incorporates Walk > ForwardResult > > > > Hi Fred > Yes, I want to use the composite fitness to compare different > systems > and or use it as a feedback in deciding on different parameter > sets > of the same system, This is not too dissimilar to how > sensitivity > analysis is incorporated into the fitness criteria. The only > difference is that sensitivity analysis during optimization, and > walk > forward is done after a new fitness high is found. Instead of > using > the insample fitness as the selection criteria for the best fit > system, the composite is criteria is used to choose among the > various > peak values in one system or in different systems. > > What you said "the capability to automatically reoptimize when > some > condition related to the performance metrics occurs during the > out of > sample period i.e. MDD goes beyond some static threshold or when > it > goes beyond some relationship to the same" is particularly > interesting. Because you are addressing a similar problem but > using a > different method, in your case, you change the time frame and > reoptimize. In my case, I am looking at refining my fitness > criteria > so I might end up in choosing a different optimized parameter > set in > the same time frame. > > Paul. > > --- In [EMAIL PROTECTED] > ps.com, Fred > Tonetti wrote: > > > > Paul, > > > > > > > > I understand what you are saying but I'm not sure what you do > with > the > > combined fitness when you get it . Do you use it to compare > different > > systems to each other ? > > > > > > > > Personally from the perspective of multiple automated WF's I > am more > > interested in . When to reoptimize . > > > > > > > > IO already has the capability to reoptimize based on: > > > > > > > > - Some static amount of time occurring during the OOS i.e. > > > > > > > > //IO: WFAuto: Rolling: 2: Weeks > > > > > > > > - or in some undefined amount of time based on some number of > long/short > > entries/exits etc i.e. > > > > > > > > //IO: WFAuto: Rolling: 2: LongEntrys > > > > > > > > What I've been playing with recently is something a little > different that is > > also based on a variable amount of time in the OOS i.e. the > capability to > > automatically reoptimize when some condition related to the > performance > > metrics occurs during the out of sample period i.e. MDD goes > beyond > some > > static threshold or when it goes beyond some relationship to > the > same or > > different performance metrics of in sample. > > > > > > > > For example . > > > > > > > > Assume the In Sample Performance Metrics are prefaced by IS > and Out > of > > Sample Performance Metrics are prefaced by OS then one should > be > able to > > write ( in terms of IO Directives ) > > > > //IO: WFAuto: Rolling: Condition: OSMDD > 10 or OSMDD > 0.75 * ISMDD > > > > > > > > In reality I suspect this is what most people actually do i.e. > find > some > > yardstick(s) that tell them their system is broken or about to > be > broken and > > then reoptimize at that time. > > > > > > > > > > > > _____ > > > > From: [EMAIL PROTECTED] > ps.com[mailto:[EMAIL PROTECTED] > ps.com] > On Behalf > > Of Paul Ho > > Sent: Tuesday, May 06, 2008 10:41 AM > > To: [EMAIL PROTECTED] ps.com > > Subject: [amibroker] Fitness Criteria that incorporates Walk > Forward Result > > > > > > > > Howard calls it the objective function. Fred calls it Fitness. > What > I > > meant by Fitness Criteria is a mathematical function on which > fitness > > or goodness of the system is judged, and is used as an > objective > > criteria to compare different systems, as a score in > optimization. > > > > My currrent question is - So why not incorporate the fitness > in > walk > > forward analysis into our fitness criteria? What I am talking > about > > is to formalise the visual inspection process. I am not > proposing > to > > use out of sample data for optimization purposes. Rather the > > parameter set that has been previously optimized is forward > tested > > and a fitness is obtained and incorporated into the original > criteria > > to form a composite fitness. > > > > For example. My current composite fitness is the geometric > average > of > > In sample fitness and Out of sample fitness divided by the > standard > > deviation (?) of In sample and out of sample fitness. > > > > Are there anybody doing something is this area? What are your > > thoughts? > > > > If you are wondering why not use visual inspection. My plan is > to > use > > the computer to do most of the work and thats why I need a > fitness > > criteria. > > > > Cheers > > Paul. > > > > > > > > > >
