Hi.
As much as I understand your desire to hand the project over, I wouldn't
necessarily do that [:D]
My python scripts are not meant to replace the US Stocks.zip
distributions. In particular they will not be zip archives of an
AmiBroker database folder.
Instead they let someone initialize their own new AmiBroker database.
And as explained earlier they will only be useful to people comfortable
with using the Command Prompt.
Since there doesn't seem to be a burning interest in this topic, I'm
taking my time with releasing these scripts. Here's the outstanding
issues:
* I can't seem to get the Industry categories to be correctly sorted
within their Sectors. See my thread How to add Industries in correct
order to new database via COM
<http://finance.groups.yahoo.com/group/amibroker/message/140695> for
details,
* Would it be nice to create some sort of AmiBroker database
Comparer? (See my thread Database data quality
<http://finance.groups.yahoo.com/group/amibroker/message/140694> ). If
the Yahoo stock list doesn't change very much then maybe it's not all
that important how fast it runs (or how easy it is to run it).
* Yahoo's quotes.csv api to get ticker Exchanges seems to have fixed
itself. Earlier this week it was erroneously assigning all NASDAQ traded
stocks to what the api calls nasdaqnm (actually the NASDAQ Global Select
Market).
Using the quotes.csv api (and grabbing Exchange information for 200
stocks at a time), it only takes about 5 minutes to get a list of Yahoo
stocks that is 99% accurate. This might be good enough for most people,
in which case the scripts are essentially finished (except unfortunately
for a readme.txt on how to run them).
If, on the other hand, you really want to see only stocks that Yahoo
will actually give you quotes on and what Yahoo really thinks their
exchange is then you will have to download each stock's quotes page (a
lengthy process).
To speed that up the I'm currently investigating:
1. Using Yahoo YQL <http://developer.yahoo.com/yql/> and/or Yahoo
Pipes <http://pipes.yahoo.com/pipes/> to see if it's possible to get
Company Name, Exchange, Sector, Industry, # employees, and Historical
Price Start & End dates with a single request.
2. Doing the entire stock list generation within a Google App Engine
<http://code.google.com/appengine/> application. The tricky part is an
App Engine app has to generate its response in under 30 seconds but
maybe some caching would get around that. At a minimum I think I should
be able to get all the stock info mentioned above using just a single
App Engine request.
3. Switching the validation part of my script to use multiple threads
(less cool than the above options, slightly complicated, but could
definitely be done).
--- In [email protected] <mailto:[email protected]> ,
"jrswindle2001" <jr...@...> wrote:
>
> Hey, this is great!
>
> Since you got it up and running I'm going to quit trying to maintain
my code. Too many breaks in the code due to Yahoo changing their html
every few months. And you are right, you have to double check a lot of
entries as they have a lot of incorrect data, which is why I did opt to
grab the company profile page. But when I cross checked it with other
open sources (like Google finance), came up with still yet other errors.
>
> Anyway, good luck and keep up the good work. Glad to see someone else
picked up the task!
>
> Thanks!
> Jim
