I am wondering if anyone here has already created a CBT procedure that
solves this problem. And if not, wondering if anyone actually would
volunteer to create one. It's a bit beyond my scope at the present time.
When one does an In-Sample / Out-Of-Sample walk forward test in AB's AA,
the AA engine will use the "optimum" variables from the IS period to use
on the OOS period. But more often than not, these values are not really
stable.
In a two-variable 3D optimization graph, for example, the 'best' values
are usually on some sharp peak somewhere. If you use these values next
month (i.e. for the OOS period), they can easily fall of that peak and
result in a big loss. One would have been better off choosing values
that were on a more stable plateau, even if they represented a
significantly lower CAR/MDD or Net Profit or whatever value you are
optimizing against.
19th Century Light-Table Method
What I have currently found useful is to optimize my system over six
individual months. I then flatten the 3D optimization graphs to 2D for
each month, and print out the six charts on thin sheets of paper. I
stack the papers one on top of the other and place them on a makeshift
light table. This essentially gives me superimposed Venn diagrams and it
is easy to circle the best common area for the six months. I then choose
the center of that common area, and get my X and Y coordinates which
corresponds to values of Profit and Stop (e.g. 30/100). When I test
these values on the OOS 7th month, I always end up with a significant
profit. I've done a lot of laborious runs like this, and it always
works.
Now of course my mechanical method of printing out graphs and placing
them over a light is very primitive (perhaps shamefully so). In order to
truly validate this, I essentially need to program this method so I can
use it for computerized statistical validation over my historical data.
I essentially want to be able to automatically:
1. Optimize my system on six individual months (not a single
six-month period)
2. Find the the stable plateaus for Profit/Loss for each month, that
correspond to a minimum acceptable optimization level
3. Compare the six stable plateaus to find the "Common Plateau"
4. Find the center of the six-month Common Plateau, to obtain best
Profit/Loss values
5. Walk-Forward to the OOS month (i.e. the 7th month) and backtest
with the Profit/Loss values found in step 4
6. Repeat steps 1-5 for as many steps in my Walk-Forward analysis.
Of course the above is easier said than done. Or is it? Maybe there is
some simple way to fandangle this, that I have not thought of.
Any suggestions on how to approach this? Or better yet, anyone feeling
up to the challenge of actually slapping it together as a CBT/Jscript
piece of coding brilliance?
In its simplest form, it really only need to do steps 1-4 above,perhaps
callable by a jscript. It can then easily be used with AB's WF
capability.
