Hello Tomasz, Unbelievable!!!! What took me hours (and even days), your method has reduced to minutes!!!! (For anyone who is reading this, in order to concatenate many files into one, I suggest using the dos Copy command, with * as wild card; also use the Dos copy command to sort that huge file - it does it in seconds.) Thank you again - another happy customer!
--- In [email protected], "Tomasz Janeczko" <[EMAIL PROTECTED]> wrote: > > Hello, > > Your response confirms my guesses: you have separate files per each day > which is the "worst case". > > To speed import up you should join all your files into SINGLE file > and then sort by ticker. If it ticker is first column of your data file > - it would be just alphabetic sorting. > > The other way to speed it up (but less) is to go to Tools- >Preferences > "Data" and make sure that in memory cache is greater than > maximum number of symbols you import. > > Otherwise it needs to swap to the disk all symbols for all days you are importing. > > Best regards, > Tomasz Janeczko > amibroker.com > ----- Original Message ----- > From: "jacklweinberg" <[EMAIL PROTECTED]> > To: <[email protected]> > Sent: Tuesday, May 15, 2007 5:26 PM > Subject: [amibroker] Re: Massive data loading? > > > > Tomasz, > > I think that when "Import Files" is sued, it sorts my selection (I > > select an entire years worth of data files at once). My file names > > are like this: "nasdaq_20010201.txt" (that is with yyyymmdd in the > > name). Is AmiBroker sorting those file names? or does it import in > > the order that was selected? > > > > --- In [email protected], "Tomasz Janeczko" <groups@> > > wrote: > >> > >> Hello, > >> > >> 5 years of data on my end imports in *minutes* not hours. > >> > >> a) you have small size of "in-memory" cache so it requires a lot > > of disk mangling > >> b) the data you import are not sorted or sorted improperly (in > > reverse order) and AB needs to re-sort. > >> > >> For fast import It is advised to use data that is sorted in OLDEST > > QUOTES FIRST - NEWEST LAST > >> and to have one file per symbol. > >> > >> The worst possible case (it's probably yours) is having sorted in > > opposite direction > >> (newest first) and different symbol in every line. > >> > >> Best regards, > >> Tomasz Janeczko > >> amibroker.com > >> ----- Original Message ----- > >> From: "jacklweinberg" <jacklweinberg@> > >> To: <[email protected]> > >> Sent: Tuesday, May 15, 2007 2:59 PM > >> Subject: [amibroker] Re: Massive data loading? > >> > >> > >> > Tomasz, > >> > Perhaps my original message wasn't clear. > >> > I have the data in txt files, and am using AmiBroker "Import > > Files" > >> > to load them into the database. That is the process that is > > taking > >> > over 32 hours! > >> > > >> > --- In [email protected], "Tomasz Janeczko" <groups@> > >> > wrote: > >> >> > >> >> Hello, > >> >> > >> >> It is MUCH quicker if you change the settings. > >> >> Default settings are fail-safe. You can however > >> >> change "number of simultaneous downloads" to 5 or more. > >> >> BY setting it say to 10 you can get 10x speed up, > >> >> however Yahoo may block you after some excessive downloading > >> >> and you will need to revert to defaults and wait a couple of > >> > minutes > >> >> for Yahoo to unblock you. > >> >> > >> >> Best regards, > >> >> Tomasz Janeczko > >> >> amibroker.com > >> >> ----- Original Message ----- > >> >> From: "jacklweinberg" <jacklweinberg@> > >> >> To: <[email protected]> > >> >> Sent: Monday, May 14, 2007 7:33 PM > >> >> Subject: [amibroker] Massive data loading? > >> >> > >> >> > >> >> > Does anyone have experience at loading up an entirely new > >> > database > >> >> > with AmiBroker? > >> >> > For example, loading AB with all of Nasdaq with 5 years of > > data? > >> >> > I used the file import command, and selected a series of .txt > >> > files, > >> >> > and it takes around 32 hours of elapsed time per year!!! (On > > a > >> > fast > >> >> > machine, no less!) > >> >> > Does anyone know of a quicker method? > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Please note that this group is for discussion between users > > only. > >> >> > > >> >> > To get support from AmiBroker please send an e-mail directly > > to > >> >> > SUPPORT {at} amibroker.com > >> >> > > >> >> > For NEW RELEASE ANNOUNCEMENTS and other news always check > > DEVLOG: > >> >> > http://www.amibroker.com/devlog/ > >> >> > > >> >> > For other support material please check also: > >> >> > http://www.amibroker.com/support.html > >> >> > > >> >> > Yahoo! Groups Links > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > >> > > >> > > >> > > >> > > >> > Please note that this group is for discussion between users only. > >> > > >> > To get support from AmiBroker please send an e-mail directly to > >> > SUPPORT {at} amibroker.com > >> > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > >> > http://www.amibroker.com/devlog/ > >> > > >> > For other support material please check also: > >> > http://www.amibroker.com/support.html > >> > > >> > Yahoo! Groups Links > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > > > Please note that this group is for discussion between users only. > > > > To get support from AmiBroker please send an e-mail directly to > > SUPPORT {at} amibroker.com > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > http://www.amibroker.com/devlog/ > > > > For other support material please check also: > > http://www.amibroker.com/support.html > > > > Yahoo! Groups Links > > > > > > > > > > >
