Tomasz Dollars and cents aside, i would like to understand the criteria that is used in the data that would be purged. Here is the case that i am trying to understand.
My AB has # o fbars 50000 in base interval 1 minute The data feed comign in is a tick level data feed. Questions are 1. Does AB store it as a Tick data internally. The reason is that if i want to change to see tick data later on, could i see it for the timeframe without it going to the plugin to get the tick data 2. AB tells me 50k bars = 104 days with 1562 kb per symbol. So does it use FIFO to purge the bar that are >104 days. This means that i can backup my database files every 104 days to get some older data if i needed. Seede --- In amibroker@yahoogroups.com, "Tomasz Janeczko" <[EMAIL PROTECTED]> wrote: > > That's special price for AmiBroker users :-) > > Best regards, > Tomasz Janeczko > amibroker.com > ----- Original Message ----- > From: Don Lindberg > To: amibroker@yahoogroups.com > Sent: Thursday, September 13, 2007 6:37 AM > Subject: RE: [AmiBroker] Re: Is 500,000 the maximum number of bars allowable in a database? > > > Tomasz, > > Your to cheap. Programming consultants are in the $80 - $100 hr range! > > > > Donald F Lindberg > > > -------------------------------------------------------------------- ---------- > > From: amibroker@yahoogroups.com [mailto:[EMAIL PROTECTED] On Behalf Of eric tao > Sent: Wednesday, September 12, 2007 8:55 PM > To: amibroker@yahoogroups.com > Subject: [amibroker] Re: Is 500,000 the maximum number of bars allowable in a database? > > > > $60/hours, Good to know :) > > --- In amibroker@yahoogroups.com, "Tomasz Janeczko" <groups@> wrote: > > > > > This isn't even a major change. It just removes the direct > > > relationship of per symbol data on disk to the amount of symbol data > > > Amibroker processes. > > > > I will tell you something: you have NO idea, absolutely NO IDEA, if > it is major change or not. > > As far as AmiBroker internals are considered your guesses are as > accurate as forecasting the value of SP500 in December 2017. > > > > The one you think isn't major is in fact big (requires in fact LOTS > of changes) and no-one except 0.5% of AB user base would benefit > > from it. > > > > Are you willing to pay for custom development? $60/hour is the rate. > > It will take about 10 days of development and testing. Are you > willing to pay $4800 to develop it for you? > > If so, no problem, I will do that for $4800. > > > > Best regards, > > Tomasz Janeczko > > amibroker.com > > ----- Original Message ----- > > From: "scourt2000" <stevehite@> > > To: <amibroker@yahoogroups.com> > > Sent: Wednesday, September 12, 2007 4:43 PM > > Subject: [amibroker] Re: Is 500,000 the maximum number of bars > allowable in a database? > > > > > > > > > > Tomasz, > > > > > > Just because I store large amounts of data in one file (aka, a > > > symbol), it doesn't mean that Amibroker has to process all of that > > > data. Take the example of sqlite, the free database stores all of its > > > information in one file but users don't process all of that data just > > > because it's in one physical file on a disk. The stock data in > > > Amibroker is a simple database. > > > > > > Please address this point: > > > > > > 1. Amibroker should never delete old stock/futures data just because > > > new data is coming in. All that is needed is for Amibroker to limit > > > the amount of data it is willing to pull in historically from some > > > specified start point in the past. Whether that is a certain number > > > of bars or a specific date range, that's fine. Like I alluded to, if > > > I have 5 years worth of something like ES tick data as a continuous > > > contract in one file and I want to backtest (or view) a date range in > > > that data, then I should have some way in Amibroker of specifying a > > > start date and end date for the data segment to be pulled in for > > > processing. > > > > > > Here's what Amibroker currently 'says': "I'm going to use all of the > > > data you have for a symbol. If you have too much, then I am going to > > > blow out the PC's memory. Game over." > > > > > > Here's what Amibroker should do: If there's too much data for a > > > symbol, let the user specify a date range for testing/viewing or so > > > many bars leading up to the present bar for testing/viewing. The > > > actual physical size of that data on disk remains intact, just not all > > > available for processing due to memory limitations. > > > > > > This isn't even a major change. It just removes the direct > > > relationship of per symbol data on disk to the amount of symbol data > > > Amibroker processes. > > > > > > And like I said previously, there is a workaround for this. I have to > > > chop up my data into separate, artificial symbol names. Where does > > > this put the burden of the workload? On me, of course. What is > > > software supposed to do? It's supposed to take the burden of the > > > workload off of the user as much as possible. > > > > > > Steve > > > > > > > > > --- In amibroker@yahoogroups.com, "Tomasz Janeczko" <groups@> wrote: > > >> > > >> Hello, > > >> > > >> Why don't you move to Metastock that has 65000 bar limit ? > > >> > > >> The truth is that AmiBroker is the most capable in this area because > > >> it stores way more data than other softwares. Also, by changing the > > > registry > > >> limit you can set it to any value you need. The limit is not the > > > software but the hardware > > >> > > >> It is funny when people make statements like yours without taking > > > calculator and > > >> actually checking how much memory you need to store that. > > >> > > >> 20 million bars - OK, but did you actually count how much memory you > > > need for that? > > >> Each quote is 64 bytes, now 20 million of bars is 1.2GB of memory > > > per symbol. > > >> > > >> You may say "but you don't need to load it at once". But > > > unfortunatelly you do need that. > > >> How you can backtest without loading the data ? > > >> And each AFL array in such case would need 4 * 20 million = 80 MB of > > > RAM. > > >> Usually user formulas use hundreds of such arrays. > > >> > > >> Wise old words are: "You can't eat your cake and have it too". > > >> > > >> If you want more data stored - the price for that is increased > > > memory requirements > > >> and more CPU power needed to process it. > > >> > > >> Now you can say: you can process one bar at a time - yes you can - > > > but you need > > >> to store intermediate results anyway for next iteration and that > > > means that memory requirements > > >> would not be even one byte less than they are. Plus changing > > > execution model to 1-bar at a time will > > >> mean (number-of-bars)-times slow down. > > >> > > >> So you would end up with system that is slow, slow and even more > slow. > > >> > > >> Best regards, > > >> Tomasz Janeczko > > >> amibroker.com > > >> ----- Original Message ----- > > >> From: "scourt2000" <stevehite@> > > >> To: <amibroker@yahoogroups.com> > > >> Sent: Wednesday, September 12, 2007 2:00 PM > > >> Subject: [amibroker] Re: Is 500,000 the maximum number of bars > > > allowable in a database? > > >> > > >> > > >> > > > >> > And this is my biggest pet peeve about Amibroker: > > >> > > > >> > If I have the disk space available, Amibroker should never, > ever, EVER > > >> > just silently blow away my historical data because of some > arbitrary > > >> > system limit like 500,000 bars or 1,000,000. If I want 20 million > > >> > bars historical and I have the disk space to store it, then > fine. If > > >> > Amibroker needs far less to function well, then that's fine too. > > >> > > > >> > The solution is NOT to delete old data. The solution is to > have a way > > >> > for Amibroker to limit the number of past bars (for calculation > > >> > purposes) or even have me set a beginning and end date in an > > >> > historical range WITHOUT destroying that historical data (i.e., the > > >> > present day should not have to be the last bar that I view or > use for > > >> > backtesting). > > >> > > > >> > Currently, the only solution is to set some registry value to > go over > > >> > 500,000, but that in itself is not solution. Because of > Amibroker's > > >> > implementation, you'll get your memory stressed out big time and, > > >> > again, if your new limit gets reached, Amibroker will remove your > > >> > oldest data to make room for new data. > > >> > > > >> > No one ever said that you had to look at stock/futures data > from the > > >> > complete past to today. If I had 20 years of one symbol's > data and > > >> > that data was far too much to make available at one time because of > > >> > memory problems, then it should be segmentable without me having to > > >> > split of the data in artificial symbols. > > >> > > > >> > Of course there's a workaround (there usually is). You have to > split > > >> > your data up (e.g., tick data) into segments with different > artificial > > >> > symbol names (e.g., I use eSignal continuous futures contracts). > > >> > > > >> > > > >> > --- In amibroker@yahoogroups.com, Graham <kavemanperth@> wrote: > > >> >> > > >> >> I believe the database limit is set to 500,000 but can be > changed in > > >> >> the windows registry. TJ has mentioned this in a previous post > here, > > >> >> somewhere sometime. > > >> >> > > >> >> > > >> >> -- > > >> >> Cheers > > >> >> Graham Kav > > >> >> AFL Writing Service > > >> >> http://www.aflwriting.com > > >> >> > > >> >> > > >> >> On 12/09/2007, Ara Kaloustian <ara1@> wrote: > > >> >> > > > >> >> > > > >> >> > You can have as much as you want, but performance degrades. > > >> >> > > > >> >> > > >> > > > >> > > > >> > > > >> > > > >> > Please note that this group is for discussion between users only. > > >> > > > >> > To get support from AmiBroker please send an e-mail directly to > > >> > SUPPORT {at} amibroker.com > > >> > > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > >> > http://www.amibroker.com/devlog/ > > >> > > > >> > For other support material please check also: > > >> > http://www.amibroker.com/support.html > > >> > > > >> > Yahoo! Groups Links > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > > > > > Please note that this group is for discussion between users only. > > > > > > To get support from AmiBroker please send an e-mail directly to > > > SUPPORT {at} amibroker.com > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > http://www.amibroker.com/devlog/ > > > > > > For other support material please check also: > > > http://www.amibroker.com/support.html > > > > > > Yahoo! Groups Links > > > > > > > > > > > > > > > > > >
