Yes Mich, I noticed that as well, In addition, Currently, memcof seems to calculate the rmserror as sum(data^2) - sum(1 - reflection Coeff^2). Is this valid? if not what do you use to calculate it recursively. Cheers Paul.
_____ From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of Tom Tom Sent: Thursday, 16 November 2006 7:56 AM To: [email protected] Subject: Re: [amibroker] Re: Polynomial Trendlines Hi ! Thanks Paul ! It is around the same for MEM yes. I find a way to compute it during the recursive process (as you tell it). I have made comparaison between MEM in Numerical Recipes and formula i make from original mathematical recursive formula from Burg. In NR, they make the recurrent loop to compute the Num and Den (use to calculate the coefficient of reflexion k), loop from 1 to M-i (M is number of quotes data, i is incrementing from 1 to ORDER_AR). So for high order AR, most recent data are not taken in consideration !? Same for updating the forward and backward error from the lattice filter, they just considere from 1 to M-i. Original burg formula goes loop from i to M-1, so last data are always here even for high order. -> memcof on Numerical Recipes doesn't respect original algorithm. I don't know why they do this on NR mem algo !? i don't find any source stating than taking [1:M-i] (memcof NR) is better than [i:M-1] (original burg). Mich. ----- Original Message ----- From: Paul Ho To: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com Sent: Wednesday, November 15, 2006 10:05 AM Subject: RE: [amibroker] Re: Polynomial Trendlines Problem is more (for us and for general AR modeling) to find the right order. Order to low = bad resolution and dominants frequency estimated are not good, to high order = to much frequency and some of them don't exist in fact and are only created by overfit the model. So you need to know wich AR order to use for correct dermining of the number of sinusoidale componant on the signal and have a good resolution. I will compute on the AFL some automatic AR order selection from classic algorithm for AR : FPE, MDL, AIC, CAT, and the best alternative for shot data order estimation seems : CIC (Combined Information Criterion). Newest method are CIC, Modified FPE and Modified MDL (I wil not compute the last two ones because i did'nt find easy algorithm to implement them). I presume FPE = Final Prediction Error. I have had a algorithm from Anthony Warren from TASC in 83'. He proposed minimising FPE through by minimising the rmserror = sum(e(t)^2); FPE = (N+M)/(N-M) * rmserror. In the calculation of AR coeff, Coefficients or order 1.....M is also obtained as part of the recursive process. Also there are few extra calculations to obtain FPE and select the optimium order from 1....M with minimum FPE. I havent done it in AFL for MEM, but have done so for polynomial curve fitting. the following code illustrate my algorithm, though I believe in MEM it is more compact if(auto) { Ym = Nz(PolyFit(PF_Y, PF_BegBar, PF_EndBar, 1, PF_ExtraB, PF_ExtraF)); PF_Y2 = IIf(BI < PF_BegBar - PF_ExtraB - PF_ExtraF, 0, IIf(BI > PF_EndBar, 0, PF_Y)); PF_N = PF_BegBar - PF_EndBar + 1; mfpe = (PF_N + 1)/(PF_N - 1) * LastValue(Cum((PF_Y2 - Ym)^2)); AddColumn(mfpe, "fpe - 1", 1.3); PF_order = 1; for(m = 2; m <=12;m++) { Ym = Nz(PolyFit(PF_Y, PF_BegBar, PF_EndBar, m, PF_ExtraB, PF_ExtraF)); fpe = (PF_N + m )/(PF_N - m) * LastValue(Cum((PF_Y2 - Ym)^2)); AddColumn(fpe, "fpe"+m, 1.3); if(fpe < mfpe) { mfpe = mfpe; PF_order = m; } } } Yn = PolyFit(PF_Y, PF_BegBar, PF_EndBar, PF_Order, PF_ExtraB, PF_ExtraF); cheers Paul. From: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com [mailto:[EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com] On Behalf Of Tom Tom Sent: Wednesday, 15 November 2006 3:57 AM To: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com Subject: Re: [amibroker] Re: Polynomial Trendlines Hello Paul ---- i.e : Long post, but i think interresting for those who are trying to extract sinus out of noise (Non parametric method like FFT (periodogram) or DSP (correlogram), and Parametric Method like Linear Prediction like AR, MA, ARMA, etc..., TrigFit). Scientific littertature seems agree that Parametric Method offer better resolution than classic direct non parametric FFT/DSP, so why we continue to use FFT directly applied on the data... --- Paul, Biased/Not biased is just only for Yuke Walker (you can call it too Durbin or autoccorelation) AR method. It is not for Burg (you can call it MEM or Maximum Likelihood), nor covariance and modified covariance. More information about those : http://www.mathwork <http://www.mathworks.com/access/helpdesk/help/toolbox/dspblks/index.html?/a ccess/helpdesk/help/toolbox/dspblks/yulewalkerarestimator.html> s.com/access/helpdesk/help/toolbox/dspblks/index.html?/access/helpdesk/help/ toolbox/dspblks/yulewalkerarestimator.html For stability of the modeled filter you have to choose : Biased one. But is is biased. If you choose not biased tou got the right freqency estimation and so poorer resolotion. You need to correct the pole position to place it inside the unit circle if you choose not biased (code is in NR to replace pole on the unit circle, just after Burg/MEM (function memcof) ). For Burg not need to use memcof because filter produced is already stable. I just finished developping too my own Burg (MEM) AR Model, but not from NR source but from another one where it gives mathematic algorithm for burg. I choose this one because it provides a hamming window on the reflexion coefficient of the lattice filter (resolution is improved and variance is less, from the theoric paper ; ) ). So there is no copyright (take code from scratch from mathematic formula) and it is 100% AFL. I will post it as soon as i corrected some problem like barcount héhé I will have to see how it compare to traditionnal burg, and burg wompared to yuke-walker (from what i have read burg as a lot better spectral resolution than YW for short data history). I test the traditional burg and it is very efficient to find sinus inside some noise... i was very impressive. A lonely 50 bar sinus is detected with only 20 bar data, Order 5 AR Burg ! Even 3 sinus added are near perfect matching fit (and to find frequency so) in some noise ! I test with theoric generated AR 2 model, and it find back the 2 right coef, with less than 0.1% error without noise (with noise another story héhé) : ) So formula AFL formula i will post seems ok. Problem is more (for us and for general AR modeling) to find the right order. Order to low = bad resolution and dominants frequency estimated are not good, to high order = to much frequency and some of them don't exist in fact and are only created by overfit the model. So you need to know wich AR order to use for correct dermining of the number of sinusoidale componant on the signal and have a good resolution. I will compute on the AFL some automatic AR order selection from classic algorithm for AR : FPE, MDL, AIC, CAT, and the best alternative for shot data order estimation seems : CIC (Combined Information Criterion). Newest method are CIC, Modified FPE and Modified MDL (I wil not compute the last two ones because i did'nt find easy algorithm to implement them). ARMA, like you said is another story, and is uselless in fact because we are looking to extract only sinus (or peridic behaviour). MA is use for better colored noise spectral representation. For ARMA only Yule Walker modified (on what i was working last week, and seems not to produce filter with stability 100%) is easy to implement. Between AR Burg (the best solution for AR who produce 100% stable filter) and ARMA YW Modified, it seems AR Burg has a lot better resolution for our purpose (multiple sinus in noise). So I won't bother with ARMA for now : ) For mean substract i find two technics in letterature : - substract classic mean on the data horizon wich are used to be modeled and after add it back - same but not with classic mean. we can use an autocorrelation-weighted mean. Problem for this method is we need to know "a priori" the noise variance on the data... and it is not he case for use. For detrending like you mention it we can : - substract linear fit (least square, or poly order1, same) - substract order n (n>1) polyfit - make differenciation one by one on the sample (so the mean is 0, and the linear trend become a constant... two in one héhé) I will just finish by putting that AR method and TrigFit method (proposed by Fred) are two different way to achieve same thing: find multiple sinus in noise. Fred, recursively check correlation by sinus, substarct them to the signal each time a new one is find until the signal is estimated to be only noise left in it (criterion is used here to know when to stop). AR find fitting model by fitting AR model, so by fitting pole (1 pole = 1 sinus), creterion is maximum likelihood. Maybe they can be use together for better spectral resolution in noise, or maybe they are the same... arh. stop ... i go eating, my stomach call me ! Sorry for long post, but there are not a lot of post on spectral analsys (or cycle finding same thing) in the forum, and now than FFT is available, maybe we can go to the next step : Parametric Spectral Estimation, and compare them. Cheers, Mich. ----- Original Message ----- From: Paul Ho To: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com Sent: Tuesday, November 14, 2006 8:31 AM Subject: RE: [amibroker] Re: Polynomial Trendlines Mich I wasn't aware of the AFL that you posted in library, so after reading your reply and your code, I have a fair idea. I have been working on both the MEM method by Burg as well as Yule-Walker equation using least square - Gaussian Elimination. I have a number of alorgithms for either, including one of my own, The results appear reasonable in re-generation of the series (rmserror is small enough) but the prediction is not good, neither is the spectrum. So my problem right now is not coding but verification and validation. There remains a lot of work to verify that the implementation is bug free, and then validate that the approach is the correct one. for example - how does deducting the series from its mean affect its prediction. or what would happen if sampling is only taking place every 2nd or 3rd bar. I am developing using C++ pluggins so in a lot of cases I can just write a wrapper around C codes posted in NR or other sources and the debugging environment is one I'm comfortable with. I see that in another post you're asking for codes of those NR functions - perhaps doing it in C will solve that problem. I'm staying away from ARMA for the moment, just AR itself is challenging enough. However, there might be different detrending techniques that I would like to consider, including using a 1st order polynomial ax + b, and perhaps the techniques you have shown Could you explain a bit about the difference between biasd and the non-biased estimators. /Paul. From: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com [mailto:[EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com] On Behalf Of Tom Tom Sent: Monday, 13 November 2006 7:16 AM To: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com Subject: Re: [amibroker] Re: Polynomial Trendlines Hello, Paul, by this link http://web.gfi. <http://web.gfi.uib.no/~ngbnk/kurs/notes/node122.html> uib.no/~ngbnk/kurs/notes/node122.html i was poitning to the fact than : 1- From Signal Y(t), we compute Autocorrelation Corr(t). 2- From the FFT from the Autocorr you got the DSP. (1) is included in the AR cose i posted (2 possibility: biased and not biased). (2) is included in the beta version of AB So for now we can have DSP (= averaging FFT, considering it infinite with finite component). But i was pointing to the fact that, we can skip (2) and maximise spectral representation by make some "estimation on the signal" or "use MEM". So MEM (Maximum Entropy Method) leads mathematicaly to AR Spectral Method. It is exactly same thing. AR Spectral Method is a parametric spectral estimation method. You compute an AR Model in time dimension, and after dirrectly from the coefficients and from the variance of the innovation (error), you got the DSP (Density Spectral Power), wich is same than Correlograme or Average Periodogramme. The fact is there is not need of FFT or DFT : ))) and the noise is filtered... and this extract pretty well exctract resonancy from the spectrum. AR (Auto Regressive) is good for signal presenting some main resonant frequency. On the oser side, you have MA (Moving Average), wich is good for inverse (big lack of some frequency). A long AR model can model a MA model. ARMA (Auto Regressive Moving Average) model (combine both aspect and can make great spectral estimation of more large variety of signal... ARMA = AR + MA)... I am trying woking on ARMA, but very difficult to assume stable/robust/convergent filter... need to compute some criterions and test pole/zero to assume stability... to much work... I think i will just use AR. With some pre-processing on the data (denoising with classic MA used for trading) and choose a big number of coefficient for AR Model, we converge to ARMA : ))) ARIMA is arma, but with trend/some seasonality consideration. Can be computer by adding linear trend or polyfit. ARIMA = preprocessing (classic MA, detrending) + AR long. Should do the trick maybe ; ) ... to test. SARIMA, ARMAX, ARX, ARCH, GARCH, are all some different version of AR modeling i thing (i don't know very well ARH, GARCH, seems good to estimate volatility for trading options). I just finish this technic post by a link : http://irevues. <http://irevues.inist.fr/bitstream/2042/3241/1/004.PDF+TEXTE.pdf> inist.fr/bitstream/2042/3241/1/004.PDF+TEXTE.pdf it's in french, sorry... but just look at page 40. It show difference with AR Spectral Parmaetric Method (i.e: MEM) and classic FFT for finding sinus in noised data. Pretty good isn't it ? Finished, hope i didn't bother you too much... this was just to give some idea to work on for futur indicator based on cycle determination and which go further than classic FFT wich make zero assumption about signal spectrum pattern (AR make assumption that resonant frequency are present in signal). If I manage to make good sinus extraction from AR Spectrum Estimation, I will post some code. But stock quotes data spectrum seems more to refer to AR model than MA i thing, so this presumption on the data can bring us more clean cycle determination. Maybe we need in Amibroker in some near future special plot window for frequency domain in real time... who knows : ) Cheers, Mich. ----- Original Message ----- From: Paul Ho To: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com Sent: Sunday, November 12, 2006 6:15 AM Subject: RE: [amibroker] Re: Polynomial Trendlines Mich, So maybe someone can help me : is it possible to compute a DSP from AR coefficient based on the formula for AR i post (there is code to compute biased or not biased autocorrelation function inside) Do someone try ? What link are you referring to, http://web.gfi. <http://web.gfi.uib.no/~ngbnk/kurs/notes/node122.html,> uib.no/~ngbnk/kurs/notes/node122.html, there is no code with this link. Anyway, the power spectrum function is given in a number of references including the one that i posted. I am still currently working on mine but is not complete. Cheers Paul. From: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com [mailto:[EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com] On Behalf Of tomy_frenchy Sent: Sunday, 12 November 2006 12:14 PM To: [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com Subject: [amibroker] Re: Polynomial Trendlines MESA, MEM seems to use the DSP (Density Spectrum Power) of the autocorrelation function. Seem it is corresponding in some sort to AR modeling. http://web.gfi. <http://web.gfi.uib.no/~ngbnk/kurs/notes/node122.html> uib.no/~ngbnk/kurs/notes/node122.html It is mentionned to the link you gave too : http://www.library. <http://www.library.cornell.edu/nr/bookcpdf/c13-7.pdf> cornell.edu/nr/bookcpdf/c13-7.pdf AR = MEM ... So maybe someone can help me : is it possible to compute a DSP from AR coefficient based on the formula for AR i post (there is code to compute biased or not biased autocorrelation function inside). Do someone try ? Does it exhibit more clearly cycles ? Seems there is some interresting link between AR and MEM to be exploited maybe ? Thanks, Mich. __________________________________________________________ Windows Live Messenger sur i-mode : dialoguez avec vos amis depuis votre mobile comme sur PC ! http://mobile. <http://mobile.live.fr/messenger/bouygues/> live.fr/messenger/bouygues/ __________________________________________________________ Ten : profite de ton Messenger en illimité sur ton mobile ! http://mobile. <http://mobile.live.fr/messenger/ten/> live.fr/messenger/ten/ __________________________________________________________ Retrouvez tout en un clin d'oeil avec Live Search ! http://www.live. <http://www.live.com/?mkt=fr-fr> com/?mkt=fr-fr
