Hi Mike, Thanks for your reply. I understand what you say, but personnaly I would appreciate to see other results than best results, because it makes it possible to adjust a particular system. As an example, let's say that my 20,50 cross was first and now is second, hence the walk-forward chooses something else for the next OOS. But maybe over the long run - that is, many years - 20,50 would be in the top 5 and I only have to adjust to what changed in the market (e.g. volatility, market going down, etc.) to make it better.
You see, in this particular case, I don't have to reject the 20,50 crossover as the walk-forward would propose, but only to make my 20,50 better because in the long run it is more stable (I would see this if I could see the 5 best instead of only the best one). Maybe I am not making sense, but knowing that 13,31 is better than 20,50 which is now better than 14,18 is not helping me a lot; I'd prefer to see which of those systems performed well in what kind of market and then adjust the parameters to the market. Do you understand what I mean? Thanks! Louis 2008/4/18, Mike <[EMAIL PROTECTED]>: > > Louis, > > I don't think that you are understanding how walk forward works. You > always want the *best* values for use in the next OOS. What you have > to understand is that after the last IS period, you will have a new > set of parameter values that you start trading *live* tomorrow. > > You don't want 2nd best. You don't care if last month you were > trading 20/50. What you care about is that starting tomorrow you want > to trade 10/25 (for example). 20/50 is *no longer best*, so why do > you care if it's 2nd, 3rd or 100th best? It's no longer meaningful > going forward. This is why it is critical that you use an objective > function that represents your definition of "best". > > As for your comments regarding random tests. It sounds like you are > confusing two issues. Monte Carlo is not Walk Forward. > > If you want to apply Monte Carlo when first developing your strategy, > that's fine. Monte Carlo is used to see how much your performance > differs when adding noise to the *same set of data* as the original > optimization. Was it pure chance that your strategy perfomed well > over that data, or did your strategy still do well even when noise > was added to the data (i.e. strategy recognized the signal and was > able to ignore the noise)? > > Walk forward is used to validate that your strategy will adapt to the > market and continue to give positive returns on data that it has > *never seen before*. > > If you want to add sensitivity analysis to your walk forward process > such that you end up with parameter values that are surrounded by > other well performing values (as opposed to values isolated at the > top of a peak surrounded by poor performing values), then you can use > the walk forward capabilities of IO (See Intelligent Optimizer in > Files section of this group). > > http://www.amibroker.org/userkb/?s=intelligent+optimizer > > Mike > > --- In amibroker@yahoogroups.com <amibroker%40yahoogroups.com>, "Louis > Préfontaine" > <[EMAIL PROTECTED]> wrote: > > > > Hi Mike (and everyone), > > > > The problem I have with actual walk-forward is this: each > parameters are > > tested only once. As an example, if I choose a Cross ma (c,20), ma > (c,50); > > and optimize the two variables, each time it will only be tested > once. But > > there could be many results with the same variables; I would prefer > to be > > able to do a random test of the same variables let's say 1000 times > to get > > an average of the results. But with walk-forward that would take > WEEKS to > > get it done! > > > > The other problem is this one. Let's say that an IS and OOS > results show > > that 20 and 50 (in the previous example) are the two best results > (based on > > only one test, as I talked previously, but still...). Then, the > next IS and > > OOS shows something different (let's say 10 and 25). But maybe 20 > and 50 is > > still good, been the second best of the list. But the problem is > that I > > only see the ultimate best results in the walk-forward! I'd like > to see, > > let's say the 5 best results so if a parameter is in the 5 best > every time I > > have something stable even if it's not the best parameter at the > particular > > IS-OOS time! > > > > Sorry again if I gace someone a headache here. I wish I could > explain > > better what I mean, but maybe you understand my problem. > > > > Thanks, > > > > Louis > > > > 2008/4/18, Mike <[EMAIL PROTECTED]>: > > > > > > Louis, > > > > > > That's the whole point of walk forward analysis! The goal is to > enter > > > each new OOS period with parameter values *most relevant* to what > the > > > market is about to offer. When parameter values change, it is not > a > > > sign of failure. It is a sign of adaptation to changing market > > > conditions. > > > > > > If your parameter values stay the same, that suggests that the > market > > > is continuing as it had been. If your parameter values change, > that > > > suggests that the market has changed and that your strategy has > > > *adapted* to the change. > > > > > > The frequency at which your parameter values change will largely > be > > > dependant upon the length of your IS period. If you have a 10 > year IS > > > period and a 2 week OOS period, it will be a loooooooong time > before > > > your parameter values change from one OOS to the next OOS. > > > > > > That is why Howard advised you in your earlier thread to > experiment > > > with different IS period lengths to determine what the ideal IS > > > period length is for your strategy. > > > > > > If you are unable to find a suitable IS:OOS period lengths > > > combination, that suggests that your strategy is poor and needs > to be > > > reworked or abandoned. > > > > > > Mike > > > > > > --- In amibroker@yahoogroups.com <amibroker%40yahoogroups.com><amibroker% > 40yahoogroups.com>, "Louis > > > Préfontaine" > > > <rockprog80@> wrote: > > > > > > > > Hi Thomas, > > > > > > > > I understand what you mean and I agree with you on what you say. > > > My concern > > > > is more about what to do when each IS and the following OOS uses > > > some > > > > parameters and then each new IS-OOS uses another parameter. > > > Moreover, all > > > > those parameters are (as far as I know) the best-of-many and > they > > > are not > > > > resampled, so I see a potential problem and I have trouble > > > identifying where > > > > my system fails because if I use a walk-forward of 2 weeks each > 2 > > > weeks has > > > > new parameters... > > > > > > > > Do you understand what I mean? Sorry if I am confusing... > English > > > is not > > > > my first language. > > > > > > > > Louis > > > > > > > > 2008/4/18, Thomas Ludwig <Thomas.Ludwig@>: > > > > > > > > > > > > > > The problem with that is the following: let's say my signal > is > > > a MA > > > > > > crossover, and I optimized each MA. I apply a walk-forward > of 3 > > > > > > months, and each time the MA Crossover is different. So, in > > > the end, > > > > > > if the OOS is worse than IS, I don't know much more because > > > each time > > > > > > the walk-forward was acting on different MA Crossover. > > > > > > > > > > > > > > > I disagree. Look at it this way: You perform an optimization > ovet > > > your > > > > > IS period, and then you usually select the best parameter > > > combination > > > > > based on the metric you chose (like K-ratio). The basic > > > *assumtion* is > > > > > that this "best" parameter combination gives you a trading > system > > > that > > > > > can generalize - i.e., it will be able to be profitable even > if > > > market > > > > > conditions change somewhat. But you can't be sure before you > > > apply this > > > > > optimized system to NEW data different from the IS data - > that's > > > the > > > > > OOS period. That's exactly what walk-forward is doing. It just > > > adds one > > > > > more assumption: That you normally wouldn't stick to the same > > > > > parameters over several years. That's why it walks forward > though > > > the > > > > > time comparing the optimized result of the respective IS > period > > > with > > > > > the NEW data of the subsequent OOS period. Now, if you compare > > > your IS > > > > > results with the OOS results and you find the latter ones > > > considerably > > > > > worse that shows that your system is not robust. Think about > the > > > 3D > > > > > optimization graph available in Amibroker: Such a poor system > > > would > > > > > probably produce a 3D surface plot with big changes (spikes, > > > drops) > > > > > instead of stable plateaus - i.e. it is *very* dependent on > the > > > > > parameters you chose. A small change in the parameters for the > > > tested > > > > > period would lead to poor results as the system has > > > only "learned" the > > > > > market structure in that period and will fail if that changes > over > > > > > time. > > > > > > > > > > > > > > > > Do you > > > > > > understand what I mean? And because the walk-forward only > > > shows the > > > > > > BEST result of ONE optimized portfolio result (RAR, RRR, > CAR, > > > etc.) I > > > > > > just can't make it better because I can't see what is result > > > number > > > > > > 2, or 3. In that way, optimization seems superior, but > maybe I > > > am > > > > > > not using walk-forward correctly. That is my main concern! > > > > > > > > > > > > Louis > > > > > > > > > > > > 2008/4/17, Louis Préfontaine <rockprog80@>: > > > > > > > Hi Thomas, > > > > > > > > > > > > > > Do you use Walk-forward as a random optimizer like Monte > Carlo > > > > > > > Simulation or do you use it as a parameter optimization > > > (let's say > > > > > > > you want to know the best numbers for a MA crossover). Or > > > maybe > > > > > > > both? > > > > > > > > > > > > > > I ask this because my feeling is that if I use it only as > a > > > > > > > parameter optimization, each parameter would be tested > only > > > one > > > > > > > time in each IS or OOS, hence this could not be > > > significative. I > > > > > > > tried to add a random simulation 5 times to get the best > out > > > of 5 > > > > > > > results, but I was wondering if this was correct or > simply a > > > waste > > > > > > > of time. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Louis > > > > > > > > > > > > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@>: > > > > > > > > > Thank you very much Thomas. So in fact the walk- > forward > > > > > > > > > measures the data-mining bias in some way? I will read > > > what > > > > > > > > > you say I should read, and I will look at chapter 20 > in > > > > > > > > > Howard's book... > > > > > > > > > > > > > > > > Yes, it's explained there in detail. It's great that > > > Amibroker > > > > > > > > now automates this process (that wasn't the case when > > > Howard's > > > > > > > > book was published). > > > > > > > > > > > > > > > > > But still, so far I get the impression that if I > > > backtested > > > > > > > > > let's say cross (ma,ma...) for 2000 to 2008 and I take > > > this > > > > > > > > > best result and it is 100% CAR, then eben if OOS is > 50% > > > CAR I > > > > > > > > > guess there can still be place for data-ming bias (or > > > > > > > > > curve-fitting) because the optimization was done with > the > > > best > > > > > > > > > result. > > > > > > > > > > > > > > > > If your IS period is 2000-2008 I'm afraid that there is > no > > > time > > > > > > > > left for the OOS period ;-) Both periods must not > overlap! > > > In > > > > > > > > the book (and you can configure Amibroker accordingly) > the > > > IS > > > > > > > > period is chosen to be 2 years and OOS is one year. If > you > > > start > > > > > > > > the process in 2000, the first IS period would be 2000- > 2001 > > > and > > > > > > > > the first OOS period would be 2002. The second IS > period is > > > > > > > > 2001-2002 and OOS is 2003 - and so forth. In my > > > understanding > > > > > > > > this approach simulates the fact that you normally don't > > > trade a > > > > > > > > sytem with the same parameters unchanged for many years > > > (unless > > > > > > > > it's a really long-term system). Rather, you would re- > > > optimize > > > > > > > > the system every 1-2 years to adjust it to changing > market > > > > > > > > conditions. That's what the walk-forward test is > actually > > > doing. > > > > > > > > Every new optimization is compared with the results of > an > > > OOS > > > > > > > > period. If the OOS results are considerably worse than > the > > > IS > > > > > > > > results this is a strong hint that the sytem will not > work > > > in > > > > > > > > real trading. > > > > > > > > > > > > > > > > > Anyway, if I understand correctly what you say, the > OOS > > > will > > > > > > > > > ALWAYS be less than IS because the IS is optimized - > that > > > is, > > > > > > > > > it will take the best-of-100 (or 200, etc.) result, > and > > > compare > > > > > > > > > it with a more random result that would occure in real > > > life. > > > > > > > > > Am I wrong? > > > > > > > > > > > > > > > > The OOS results are not necessarily worse - but most > often > > > they > > > > > > > > are. I've analyzed a couple of systems (which I don't > > > trade) that > > > > > > > > had beautiful IS equity curves - they all failed the > walk- > > > forward > > > > > > > > test spectacularly. So it seems they were all overfitted > > > and were > > > > > > > > not able to generalize. > > > > > > > > > > > > > > > > Greetings, > > > > > > > > > > > > > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Louis > > > > > > > > > > > > > > > > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@>: > > > > > > > > > > Louis, > > > > > > > > > > > > > > > > > > > > in the IS period your system is optimized, then the > best > > > > > > > > > > values from the optimization are used to perform a > test > > > over > > > > > > > > > > the IS and OOS periods. > > > > > > > > > > > > > > > > > > > > If the OOS results are worse than the IS results, > this > > > means > > > > > > > > > > that the system doesn't generalize well enough. BTW: > > > This > > > > > > > > > > topic is very well explained in chapter 20 of > Howard's > > > book. > > > > > > > > > > I also suggest to look at > > > > > > > > > > > > > > > > > > > > http://www.amibroker.com/kb/2008/02/12/getting- > started- > > > with-a > > > > > > > > > > > > > > >utomat ic-walk-forward-optimization/. > > > > > > > > > > > > > > > > > > > > > > > > > I must say, that walk forward testing has completely > > > changed > > > > > > > > > > my way of thinking. It's much easier to see now if a > > > trading > > > > > > > > > > system is worth a second look. > > > > > > > > > > > > > > > > > > > > Greetings, > > > > > > > > > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I've been experimenting with walking-forward, and > I > > > have > > > > > > > > > > > some questions regarding how it works. > > > > > > > > > > > > > > > > > > > > > > I ran a complete random optimization or > buying/selling > > > > > > > > > > > using the variables I set (a MCS in fact), and > > > > > > > > > > > systematically OOS results were worst than IS. I > > > don't > > > > > > > > > > > understand how it works, because whatever if the > > > sampling > > > > > > > > > > > is IS or OOS it is always the same variables that > are > > > in > > > > > > > > > > > place. > > > > > > > > > > > > > > > > > > > > > > Anyone could explain how this work? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Louis > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > > > > > > > > > > > Please note that this group is for discussion > between > > > users > > > > > > > > > > only. > > > > > > > > > > > > > > > > > > > > To get support from AmiBroker please send an e-mail > > > directly > > > > > > > > > > to SUPPORT {at} amibroker.com > > > > > > > > > > > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always > > > check > > > > > > > > > > DEVLOG: http://www.amibroker.com/devlog/ > > > > > > > > > > > > > > > > > > > > For other support material please check also: > > > > > > > > > > http://www.amibroker.com/support.html > > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > > > > > > > Please note that this group is for discussion between > users > > > only. > > > > > > > > > > > > > > > > To get support from AmiBroker please send an e-mail > > > directly to > > > > > > > > SUPPORT {at} amibroker.com > > > > > > > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always > check > > > DEVLOG: > > > > > > > > http://www.amibroker.com/devlog/ > > > > > > > > > > > > > > > > For other support material please check also: > > > > > > > > http://www.amibroker.com/support.html > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > ------------------------------------ > > > > > > > > > > Please note that this group is for discussion between users > only. > > > > > > > > > > To get support from AmiBroker please send an e-mail directly > to > > > > > SUPPORT {at} amibroker.com > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check > DEVLOG: > > > > > http://www.amibroker.com/devlog/ > > > > > > > > > > For other support material please check also: > > > > > http://www.amibroker.com/support.html > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
