Hi Mark, >1. Survivorship bias; >A lot of dips just dip down to 0, especially around 2001-2003.
This is indeed a problem which I couldn't come up with a solution. Though I don't hold for a long period of time, it's possible some declined sharply and I wouldn't know about it. >2. Position sizing pitfalls This is a difficult issue. I set it to 0% (no limit) deliberately because if I set to some limit, once my equity becomes so big it wasn't hurting my overall CAR despite making bad trades toward the end of backtested period. (Imagine you constantly lose 100% of 0.001% of your equity...you still end up with 99% of your equity despite making series of terrible trades.) >3. Test period consideration This one is also hard to assess. I used to backtest against 1996-2006 data, that's 10 years. The problem was...NASDAQ experienced historical growth between 1996-2000, so it wouldn't take a genius to design a system that works during that span. I have a feeling we are better off backtesting against a down market to truely assess your system though. Lots of work to do on my part to come up with a system that keeps on working, but thank you for your advice. intermilan04 --- In amibroker@yahoogroups.com, "Mark H" <[EMAIL PROTECTED]> wrote: > > intermilan04: > > Here are some possibilities: > > 1. Survivorship bias; > A lot of dips just dip down to 0, especially around 2001-2003. > > 2. Position sizing pitfalls: > If you position size is big compared to the daily volume, you will not be able to get the open price in real trading as in historical testing since you moving in/out of the market would affect the price itself. Limit to 0.5% of EMA(V,50). Don't use percentage of bar volume since you can not follow that in real trading. Also use volume and price filter to avoid illiquid stocks. > > 3. Test period consideration: > If the test period is too long, the capital would become too big to be realistic. Try to test year by year, this will give you a feel of CAR, though you may not get the real MDD if you do so. > > 4. Over-optimization: > Make sure the system is based on sound market observation and real possibility. Limit number of variables to below 5. Make the portfolio size sufficiently big. > > Hope this helps a bit, > > - Mark H. > > > ----- Original Message ----- > From: intermilan04 > To: amibroker@yahoogroups.com > Sent: Wednesday, August 30, 2006 4:27 PM > Subject: [amibroker] Re: Backtest vs Forwardtest > > > Hi Duke, > > I have minimum price and volume to avoid penny stocks and > nobody-is-caring stocks...that's about it. > > It could be that the way I designed my system might be picking > small-cap stocks over large ones more often, though. It is a long > system which tries to pick up in a dip. > > intermilan04 > > --- In amibroker@yahoogroups.com, "Lists" <Lists@> wrote: > > > > I will second Dingo's comments about sample size. However, the other > factors to consider (aong many) are what I call sample "segments." For > example if you have a size bias built into your screening process > there will be times when your system will produce above/below average > returns relative to longer historical averages. An example of that is > a system who results have been focused in the small cap space. We have > had 6+ straight years of small cap out-performance - a period that > coincides with your backtest period ( the last period was 1976-1982.) > You have now perhaps a transitory period where both volatility and > rotation is affecting your current results. > > > > Just something to ponder.... > > > > Regards, > > > > Duke Jones, CMT > > > > > > > -------Original Message------- > > > From: dingo <dingo@> > > > Subject: RE: [amibroker] Re: Backtest vs Forwardtest > > > Sent: 30 Aug '06 19:37 > > > > > > In most circles 5 years is not enough. Plus profits are not all > you should > > > be looking at imho. How about drawdowns? I get nausea just > typing that > > > word. > > > > > > If you have access to past emails on this list then you might > want to search > > > for "robust". There have been several nice exchanges on this > term - one > > > that your have just tripped over. > > > > > > d > > > > > > > -----Original Message----- > > > > From: amibroker@yahoogroups.com > > > > [mailto:[EMAIL PROTECTED] On Behalf Of intermilan04 > > > > Sent: Wednesday, August 30, 2006 2:37 PM > > > > To: amibroker@yahoogroups.com > > > > Subject: [amibroker] Re: Backtest vs Forwardtest > > > > > > > > Hi dingo, > > > > > > > > Thank you for your prompt reply. > > > > > > > > 75% is really good, if I can snatch it. Often times I see > > > > that number when I backtest, but it drops once I start > > > > following my system... > > > > > > > > The past results are indeed too good to be true but my system > > > > is not looking at future quotes. I guess it could be the > > > > curve-fitting problem though, if 5 years of data isn't enough. > > > > > > > > I had tried with longer data range but what happened was this > > > > system racked up 1900% and 3700% in 1999-2000 and 2000-2001, > > > > which skewed my average CAR... > > > > > > > > Are there ways to avoid the curve-fitting issue other than > > > > backtesting against longer data range? > > > > > > > > Regards, > > > > > > > > intermilan04 > > > > > > > > --- In amibroker@yahoogroups.com, "dingo" <dingo@> wrote: > > > > > > > > > > And 75% isn't good enough for you? > > > > > > > > > > And don't you think the past results just might be too good > > > > to be true? > > > > > > > > > > And you checked your formula to see if it is looking into > > > > the future? > > > > > > > > > > And if it isn't then it just could be so optimized that it > > > > is "curve > > > > > fitting". > > > > > > > > > > And without your code its hard to pinpoint what's going on.... > > > > > > > > > > d > > > > > > > > > > > -----Original Message----- > > > > > > From: amibroker@yahoogroups.com > > > > > > [mailto:[EMAIL PROTECTED] On Behalf Of intermilan04 > > > > > > Sent: Wednesday, August 30, 2006 2:11 PM > > > > > > To: amibroker@yahoogroups.com > > > > > > Subject: [amibroker] Backtest vs Forwardtest > > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'm having a puzzling situation where my backtest results are > > > > > > fantastic yet my forwardtest result is nowhere near it. > > > > > > > > > > > > My system is optimized between 2001/1/1 and 2006/1/1. > > > > > > Results YTD is "forwardtest" since it is beyond the scope of > > > > > > optimized data range. > > > > > > > > > > > > Here are some numbers of backtests: > > > > > > Year-by-year-results (CAR) > > > > > > 2001/1/1-2002/1/1: 393.70% > > > > > > 2002/1/1-2003/1/1: 232.64% > > > > > > 2003/1/1-2004/1/1: 721.79% > > > > > > 2004/1/1-2005/1/1: 400.82% > > > > > > 2005/1/1-2006/1/1: 490.72% > > > > > > > > > > > > and at last--forwardtest > > > > > > 2006/1/1-2006/8/29: 74.64% > > > > > > > > > > > > I am at a loss to explain this. It's very sad that I > > > > work hard to > > > > > > come up with a system that has worked, only to see it not > working > > > > > > nearly as good as it should be. > > > > > > > > > > > > Any analysis/suggestions to fix the problem above is greatly > > > > > > appreciated. > > > > > > > > > > > > Sincerely, > > > > > > > > > > > > intermilan04 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please note that this group is for discussion between users > only. > > > > > > > > > > > > To get support from AmiBroker please send an e-mail directly to > > > > > > SUPPORT {at} amibroker.com > > > > > > > > > > > > For other support material please check also: > > > > > > http://www.amibroker.com/support.html > > > > > > > > > > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > No virus found in this incoming message. > > > > > > Checked by AVG Free Edition. > > > > > > Version: 7.1.405 / Virus Database: 268.11.7/433 - Release > > > > > > Date: 8/30/2006 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please note that this group is for discussion between users only. > > > > > > > > To get support from AmiBroker please send an e-mail directly > > > > to SUPPORT {at} amibroker.com > > > > > > > > For other support material please check also: > > > > http://www.amibroker.com/support.html > > > > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > No virus found in this incoming message. > > > > Checked by AVG Free Edition. > > > > Version: 7.1.405 / Virus Database: 268.11.7/433 - Release > > > > Date: 8/30/2006 > > > > > > > > > > > > > > > > > > > > Please note that this group is for discussion between users only. > > > > > > To get support from AmiBroker please send an e-mail directly to > > > SUPPORT {at} amibroker.com > > > > > > For other support material please check also: > > > http://www.amibroker.com/support.html > > > > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please note that this group is for discussion between users only. To get support from AmiBroker please send an e-mail directly to SUPPORT {at} amibroker.com For other support material please check also: http://www.amibroker.com/support.html Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/amibroker/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
