Hi Kurasake,
When you say you don't "see" the spikes in QT, do you mean that a chart in QT
doesn't show them, or that when you open a data window (again to do so click on
the chart of the symbol and press <CTRL> D ) and scroll to the time that AB
says there is a spike there is no corresponding spike in the QT data?
I ask because I often do not "see" a spike on a QT chart, however the data when
examined in a data window has the spike. This may be a function of the spike
filter setting in QT, however I've never asked Jerry Medved about it. The bad
data must be removed from QT before editing the AB database or it will be
reimported to AB on the next exploration. Once the data in AB is older than
the coverage in QT (in my case, 20 days), the AB data can be edited safely as
there is no chance for reimporting the bad data via an exploration.
I'm genuinely curious about what you are "seeing" as I have always found the
bad spikes in QT when I look at the data.
Another possible solution would be to delete the data in QT for a particular
symbol (or that period which is affected) and then backfill the symbol, however
as I collect data at an interval of less than a minute and the backfill is
limited to a minute period, this is not an optimal solution for my situation.
Hope this helps with your mystery.
Peace and Justice --- Patrick
----- Original Message -----
From: kurasake
To: amibroker@yahoogroups.com
Sent: Friday, April 30, 2010 11:23 AM
Subject: [amibroker] Re: Easy data refresh?
Hi Patrick.
Thanks for taking the time to write so much.
I've cross checked a few symbols in AB vs QT and I'm definitely only seeing
the spikes in AB. For example for WDC I'll see the spike in the daily chart in
AB, I'll switch to the 1-minute chart in AB and see the spike(s), note the
date/time and look at the symbol in QT and scan through the entire range of
date/time in QT for WDC and everything is normal. Another way of looking at it
for me is that while the bad data spikes don't happen all the time or on every
symbol in AB, I've never seen bad data spikes in QT (minute view, historical or
raw data window) so I'm fairly certain the issue has to be in in AB or
something related only to AB.
I would also tend to believe that if the bad data were consistent across QT
and AB that it would most likely be a problem in the IQFeed data stream or the
IQ agent in which case I would think it would be affecting a large number of
their customers.
Curiously, no matter what symbol, the spikes always seem to pop up 3-5 days
prior and after I perform an explore in AA. So I could have looked at WDC
earlier, and everything could've been fine. Then I'll run an Explore and then
look at WDC again and the spikes would appear. I also don't think I've ever
seen the spikes occur for the current day or for any day >5 days ago which is
strange (though I'm not 100% positive about this part). This further increases
my suspicion that it is something in AB.
The spikes also only seem to appear (or at least I notice it more) when I've
already looked at a large number of symbols, so I'm thinking it might be an
issue with the DB/memory cache but it's still hard for me to rationalize
how/why it happens. I've tried playing around with the database and memory
cache settings before but I guess I'll try other combinations.
As for the large number of symbols, I wish I could settle on 200 or so but my
"system" (for lack of a better word) relies on a combination of technicals and
candles and I don't always get the right signals triggered if the number of
symbols I'm scanning across is too small. I don't always have that many
symbols though and there are days when I don't need to perform an Explore and
the spikes aren't so numerous as to be overwhelming (though some days are worse
then others).
I'll figure this out eventually.
Cheers!
--- In amibroker@yahoogroups.com, "NW Trader" <pk47har...@...> wrote:
>
> Hi Kurasake,
>
> I know exactly what you are experiencing. However, the bad data is in QT,
but then once you have imported it to AB, it will persist. It will most
frequently occur when you do a backfill of some or all stocks in QT, but
depending on your data source and the spike settings you have defined in QT, it
can happen anytime -- especially on volatile opening, closing, or FOMC days.
And I see spikes virtually every day on symbols like SPY just after the close.
Be aware that the spikes do not always appear on a QT chart, especially the
instances of a 0 value for a close on an intraday bar (like a one minute bar).
But they are there and you can find them by opening a QT data window (<CTRL>D)
and scrolling to the time of the bad data.
>
> Here is my solution. First I've created an exploration which I run (daily
setting in AA) on all symbols, at least once a day. The output of the
exploration is the current Close, the HHV of high for the past 21 days, the LLV
of low for the past 21 days, and a 5 period ATR divided by the close. Why 21?
Well I have a version of QT which allows 20 days of historic data before it
starts to drop off, so this will cover all QT data available on a given day.
Why the ATR/C ? This is a quick volatility tool which can help spot when a
stock may have a split, an outlier data point, or some other radical move.
>
> So you run the scan and identify the stocks that have data that needs
correcting. Now open the Quote Editor in AB and sort the columns (e.g., if you
have a LLV(L,21) of 0, sort the lows and find the bars where 0 exists. You
could delete or edit those bars, and exit the Quote Editor, however this won't
fix things so long as the bad data is in QT. So once you've identified the
time of the bad data in AB, you have to find and delete the bad data in QT.
This can be accomplished either with the QT data window (go to the bad data and
delete), or zoom the QT chart in on the time period with the bad quote (even
tho it may not be visible on a QT chart) and press <SHIFT> <CTRL> L to remove a
low point (may have to do this more than once if multiple minutes are bad) or
<SHIFT> <CTRL> H to remove a high quote.
>
> Splits can also be adjusted in QT, as well as in AB. See the help files for
each program, tho if you fix data in QT it should be ok in AB. Again, always
fix the QT data first, then AB.
>
> Finally a gratuitous observation & suggestion. IMHO, trying to track 1500
stocks intra day is neither worth it (given all the data maintenance you will
be doing) nor going to leave you much time to trade. Consider narrowing the
stocks you are following intraday to those you realistically would trade based
on price, volume, and your other trading metrics. I now track approximately
200 symbols intraday, importing them to AB, but I trade only about - 2 to 3
dozen regularly (based on my Price/Volume and other metrics). There are an
additional 4 dozen or so that are possible trade candidates that are not
regularly traded but may move into the regular camp as others drop out. The
remaining symbols are indexes, sector ETFs, and longer term holds and swings
that I want to generally keep an eye on. For me this works well as I can
thoroughly know the stocks I trade regularly; I study their intraday support
and resistance levels, daily patterns, know when they report earnings or have
other news releases, and such. I didn't always have 3 dozen traders -- years
ago I tried to follow hundreds, but realized that the concept of span of
control was causing me to miss many trades and make some bad decisions on
trades I took. So I focused on about a dozen stocks and over the years I've
expanded the number of my traders based on experience, knowledge of those
stocks, and developing a trading rhythm for how and when I trade.
>
> Of course if you want to track 1500 stocks for end of day study purposes,
consider either a separate data base set to daily data, or a different data
provider.
>
> This works for me, but your mileage may vary.
>
> Peace and Justice --- Patrick
> ----- Original Message -----
> From: kurasake
> To: amibroker@yahoogroups.com
> Sent: Thursday, April 29, 2010 12:19 PM
> Subject: [amibroker] Easy data refresh?
>
>
>
> I've been having a problem with bad data appearing in AB (abnormally high
volume or wrong OHLC) in AB when I perform an "Explore" from the "Automatic
Analysis" on a large number of symbols (>1500) when I have "wait for backfill"
checked even for something as simple as filter=C>O.
>
> Fortunately the bad data points are very easy to identify and seem to
occur on the same days for the affected symbols and since I haven't been able
to resolve it, I've sort of learned to live with it (Note, I don't get any bad
data in QuoteTracker running on the same machine using the same RT data feed so
I'm pretty sure it's in AB).
>
> Anyway, when I see the bad data, I right click on the "Plug-in Status"
area of the status bar and do a "Force Backfill" and all the data is correctly
filled in. My question is, is there an easier way, either from the menu (so I
can setup a macro) or better yet, via some code/formula that will
> 1. Keep all the symbols but delete all the EOD and RT data
> 2. Automatically "Force Backfill" from my RT feed?
>
> If someone has an idea on what's causing the problem in the first place,
that would be helpful too but I've posted this regarding this problem before
and haven't seen a solution so I'm guessing I'm a unique case.
>
> I've also tried flushing the DB cache and playing around with different
database parameters, in-memory cache size, setting up a new database etc.
>
> Thanks.
>
> Roy
>
> System:
> Windows XP Pro/Sp3
> Intel Core i7, 4GB Ram, many gigs of HD free
> AB v5.3 (tried uninstall/reinstalling)
> RT Data from IQFeed using the latest client
>
>
>
>
> ------------------------------------
>
> **** IMPORTANT PLEASE READ ****
> This group is for the discussion between users only.
> This is *NOT* technical support channel.
>
> TO GET TECHNICAL SUPPORT send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> http://www.amibroker.com/feedback/
> (submissions sent via other channels won't be considered)
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> Yahoo! Groups Links
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
