Hi Kurasake,
Still following. Lunch break well over, but things are so messed up
today that it's hard to trade. I spent most of the NY lunch hour
trying to clean up a few key charts.
I still maintain that the issue is not with AB, but with the data
source and/or data reporter. Sidney's comment will help you see the
"bad" data in QT. In eSig the data is also visible on a T&S chart,
but the outlier data is flagged and is excluded from their charts.
Today there's a lot of "bad" data. I say "bad" because in many
instances there is actually a trade well away from the market that
is reported by dark pools and which is way out of time sequence.
That's the explanation I've gotten both from tech support persons at
both AMTD and at eSignal. Today it's driving me up a wall as many of
the commodity issues are extremely jittery -- most are on the NYSE,
but some are also on the NASDAQ. Silver, coal and steel sector stocks
like SLW, PCX, VALE, FCX, ANR, MTL, X and AKS are really making me
nuts as I trade them often. And ETFs like SPY, SLV, FAZ and FXE are
also badly tweaked. Lot of bad spikes came in around 10:26 and others
have been occurring ever since, especially right around 12:00 to
12:15. This is not caused by AB and I am running no explorations or
scans in AB, just the live feed via plug in. The data supplied to AB
is bad and (for me) the only cure will be to wait until well after the
close, replace the AB database with a backup (I always maintain a
current clean one) and redownload today's intraday data. A pain.
Peace and Justice --- Patrick
----- Original Message -----
*From:* kurasake <mailto:kuras...@yahoo.com>
*To:* amibroker@yahoogroups.com <mailto:amibroker@yahoogroups.com>
*Sent:* Wednesday, May 05, 2010 7:48 AM
*Subject:* [amibroker] Re: Easy data refresh?
Hi Patrick,
In case you're still following this thread I just wanted to let
you know that I'm still having problems, despite the rebuild.
And the problem seems to come up regardless of whether or not I
have QT running so I think the problem (at least in my case) is
isolated to AB and/or IQFeed. As far as seeing if it happens if
QT is the only thing running, it's not going to be realistic for
me to confirm if the problem happens in QT if AB isn't running
since I never get a visual cue in QT that the data is corrupt,
which would mean I'd have to constantly monitor the QT data window
to numerically check for spikes. And since it doesn't happen
frequently enough or in every symbol, it becomes impractical.
Anyway, no need to reply. Just wanted to give you an update. If
I can't narrow it down with tweaks in the memory/db cache then
I'll consider opening a trouble ticket with AB.
Thanks again.
--- In amibroker@yahoogroups.com
<mailto:amibroker@yahoogroups.com>, "kurasake" <kuras...@...> wrote:
>
>
> Hi Patrick
>
> Thanks for taking the time to reply.
>
> I took the "option of last resort" and nuked my computer and
reinstalled everything from scratch over the weekend. It was
painful but hopefully (fingers crossed) worth it. I haven't
re-imported all my symbols/charts and stuff etc yet but so far
running an explore on >1000 seems to be working so I guess it was
"something" with my computer in general... weird.
>
> To answer your question though, by "see" in QT, I meant
visually. I didn't actually look at the data in the data window
(I assumed the data in the chart in QT = the data in the data
window, though I suppose that may not necessarily be the case).
>
> If this problem happens again, I suppose another option I could
try (and probably should've tried) is to run only AB and see if
the spikes come up when QT isn't running and vice-versa. Or to
run another program which can tap into the IQfeed and see if there
are any abnormalities in the data in that app as well.
>
> Thanks again for your feedback.
>
> Cheers!
>
>
>
> --- In amibroker@yahoogroups.com
<mailto:amibroker@yahoogroups.com>, "NW Trader" <pk47hargus@> wrote:
> >
> > Hi Kurasake,
> >
> > When you say you don't "see" the spikes in QT, do you mean
that a chart in QT doesn't show them, or that when you open a data
window (again to do so click on the chart of the symbol and press
<CTRL> D ) and scroll to the time that AB says there is a spike
there is no corresponding spike in the QT data?
> >
> > I ask because I often do not "see" a spike on a QT chart,
however the data when examined in a data window has the spike.
This may be a function of the spike filter setting in QT, however
I've never asked Jerry Medved about it. The bad data must be
removed from QT before editing the AB database or it will be
reimported to AB on the next exploration. Once the data in AB is
older than the coverage in QT (in my case, 20 days), the AB data
can be edited safely as there is no chance for reimporting the bad
data via an exploration.
> >
> > I'm genuinely curious about what you are "seeing" as I have
always found the bad spikes in QT when I look at the data.
> >
> > Another possible solution would be to delete the data in QT
for a particular symbol (or that period which is affected) and
then backfill the symbol, however as I collect data at an interval
of less than a minute and the backfill is limited to a minute
period, this is not an optimal solution for my situation.
> >
> > Hope this helps with your mystery.
> >
> > Peace and Justice --- Patrick
> > ----- Original Message -----
> > From: kurasake
> > To: amibroker@yahoogroups.com
<mailto:amibroker@yahoogroups.com>
> > Sent: Friday, April 30, 2010 11:23 AM
> > Subject: [amibroker] Re: Easy data refresh?
> >
> >
> > Hi Patrick.
> >
> > Thanks for taking the time to write so much.
> >
> > I've cross checked a few symbols in AB vs QT and I'm
definitely only seeing the spikes in AB. For example for WDC I'll
see the spike in the daily chart in AB, I'll switch to the
1-minute chart in AB and see the spike(s), note the date/time and
look at the symbol in QT and scan through the entire range of
date/time in QT for WDC and everything is normal. Another way of
looking at it for me is that while the bad data spikes don't
happen all the time or on every symbol in AB, I've never seen bad
data spikes in QT (minute view, historical or raw data window) so
I'm fairly certain the issue has to be in in AB or something
related only to AB.
> >
> > I would also tend to believe that if the bad data were
consistent across QT and AB that it would most likely be a problem
in the IQFeed data stream or the IQ agent in which case I would
think it would be affecting a large number of their customers.
> >
> > Curiously, no matter what symbol, the spikes always seem to
pop up 3-5 days prior and after I perform an explore in AA. So I
could have looked at WDC earlier, and everything could've been
fine. Then I'll run an Explore and then look at WDC again and the
spikes would appear. I also don't think I've ever seen the spikes
occur for the current day or for any day >5 days ago which is
strange (though I'm not 100% positive about this part). This
further increases my suspicion that it is something in AB.
> >
> > The spikes also only seem to appear (or at least I notice it
more) when I've already looked at a large number of symbols, so
I'm thinking it might be an issue with the DB/memory cache but
it's still hard for me to rationalize how/why it happens. I've
tried playing around with the database and memory cache settings
before but I guess I'll try other combinations.
> >
> > As for the large number of symbols, I wish I could settle on
200 or so but my "system" (for lack of a better word) relies on a
combination of technicals and candles and I don't always get the
right signals triggered if the number of symbols I'm scanning
across is too small. I don't always have that many symbols though
and there are days when I don't need to perform an Explore and the
spikes aren't so numerous as to be overwhelming (though some days
are worse then others).
> >
> > I'll figure this out eventually.
> >
> > Cheers!
> >
> >
> > --- In amibroker@yahoogroups.com
<mailto:amibroker@yahoogroups.com>, "NW Trader" <pk47hargus@> wrote:
> > >
> > > Hi Kurasake,
> > >
> > > I know exactly what you are experiencing. However, the bad
data is in QT, but then once you have imported it to AB, it will
persist. It will most frequently occur when you do a backfill of
some or all stocks in QT, but depending on your data source and
the spike settings you have defined in QT, it can happen anytime
-- especially on volatile opening, closing, or FOMC days. And I
see spikes virtually every day on symbols like SPY just after the
close. Be aware that the spikes do not always appear on a QT
chart, especially the instances of a 0 value for a close on an
intraday bar (like a one minute bar). But they are there and you
can find them by opening a QT data window (<CTRL>D) and scrolling
to the time of the bad data.
> > >
> > > Here is my solution. First I've created an exploration
which I run (daily setting in AA) on all symbols, at least once a
day. The output of the exploration is the current Close, the HHV
of high for the past 21 days, the LLV of low for the past 21 days,
and a 5 period ATR divided by the close. Why 21? Well I have a
version of QT which allows 20 days of historic data before it
starts to drop off, so this will cover all QT data available on a
given day. Why the ATR/C ? This is a quick volatility tool which
can help spot when a stock may have a split, an outlier data
point, or some other radical move.
> > >
> > > So you run the scan and identify the stocks that have data
that needs correcting. Now open the Quote Editor in AB and sort
the columns (e.g., if you have a LLV(L,21) of 0, sort the lows and
find the bars where 0 exists. You could delete or edit those
bars, and exit the Quote Editor, however this won't fix things so
long as the bad data is in QT. So once you've identified the time
of the bad data in AB, you have to find and delete the bad data in
QT. This can be accomplished either with the QT data window (go
to the bad data and delete), or zoom the QT chart in on the time
period with the bad quote (even tho it may not be visible on a QT
chart) and press <SHIFT> <CTRL> L to remove a low point (may have
to do this more than once if multiple minutes are bad) or <SHIFT>
<CTRL> H to remove a high quote.
> > >
> > > Splits can also be adjusted in QT, as well as in AB. See the
help files for each program, tho if you fix data in QT it should
be ok in AB. Again, always fix the QT data first, then AB.
> > >
> > > Finally a gratuitous observation & suggestion. IMHO, trying
to track 1500 stocks intra day is neither worth it (given all the
data maintenance you will be doing) nor going to leave you much
time to trade. Consider narrowing the stocks you are following
intraday to those you realistically would trade based on price,
volume, and your other trading metrics. I now track approximately
200 symbols intraday, importing them to AB, but I trade only about
- 2 to 3 dozen regularly (based on my Price/Volume and other
metrics). There are an additional 4 dozen or so that are possible
trade candidates that are not regularly traded but may move into
the regular camp as others drop out. The remaining symbols are
indexes, sector ETFs, and longer term holds and swings that I want
to generally keep an eye on. For me this works well as I can
thoroughly know the stocks I trade regularly; I study their
intraday support and resistance levels, daily patterns, know when
they report earnings or have other news releases, and such. I
didn't always have 3 dozen traders -- years ago I tried to follow
hundreds, but realized that the concept of span of control was
causing me to miss many trades and make some bad decisions on
trades I took. So I focused on about a dozen stocks and over the
years I've expanded the number of my traders based on experience,
knowledge of those stocks, and developing a trading rhythm for how
and when I trade.
> > >
> > > Of course if you want to track 1500 stocks for end of day
study purposes, consider either a separate data base set to daily
data, or a different data provider.
> > >
> > > This works for me, but your mileage may vary.
> > >
> > > Peace and Justice --- Patrick
> > > ----- Original Message -----
> > > From: kurasake
> > > To: amibroker@yahoogroups.com
<mailto:amibroker@yahoogroups.com>
> > > Sent: Thursday, April 29, 2010 12:19 PM
> > > Subject: [amibroker] Easy data refresh?
> > >
> > >
> > >
> > > I've been having a problem with bad data appearing in AB
(abnormally high volume or wrong OHLC) in AB when I perform an
"Explore" from the "Automatic Analysis" on a large number of
symbols (>1500) when I have "wait for backfill" checked even for
something as simple as filter=C>O.
> > >
> > > Fortunately the bad data points are very easy to identify
and seem to occur on the same days for the affected symbols and
since I haven't been able to resolve it, I've sort of learned to
live with it (Note, I don't get any bad data in QuoteTracker
running on the same machine using the same RT data feed so I'm
pretty sure it's in AB).
> > >
> > > Anyway, when I see the bad data, I right click on the
"Plug-in Status" area of the status bar and do a "Force Backfill"
and all the data is correctly filled in. My question is, is there
an easier way, either from the menu (so I can setup a macro) or
better yet, via some code/formula that will
> > > 1. Keep all the symbols but delete all the EOD and RT data
> > > 2. Automatically "Force Backfill" from my RT feed?
> > >
> > > If someone has an idea on what's causing the problem in
the first place, that would be helpful too but I've posted this
regarding this problem before and haven't seen a solution so I'm
guessing I'm a unique case.
> > >
> > > I've also tried flushing the DB cache and playing around
with different database parameters, in-memory cache size, setting
up a new database etc.
> > >
> > > Thanks.
> > >
> > > Roy
> > >
> > > System:
> > > Windows XP Pro/Sp3
> > > Intel Core i7, 4GB Ram, many gigs of HD free
> > > AB v5.3 (tried uninstall/reinstalling)
> > > RT Data from IQFeed using the latest client
> > >
> > >
> > >
> > >
> > > ------------------------------------
> > >
> > > **** IMPORTANT PLEASE READ ****
> > > This group is for the discussion between users only.
> > > This is *NOT* technical support channel.
> > >
> > > TO GET TECHNICAL SUPPORT send an e-mail directly to
> > > SUPPORT {at} amibroker.com
> > >
> > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> > > http://www.amibroker.com/feedback/
> > > (submissions sent via other channels won't be considered)
> > >
> > > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > > http://www.amibroker.com/devlog/
> > >
> > > Yahoo! Groups Links
> > >
> >
> >
> >
> >
> > ------------------------------------
> >
> > **** IMPORTANT PLEASE READ ****
> > This group is for the discussion between users only.
> > This is *NOT* technical support channel.
> >
> > TO GET TECHNICAL SUPPORT send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> > http://www.amibroker.com/feedback/
> > (submissions sent via other channels won't be considered)
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > Yahoo! Groups Links
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
amibroker-fullfeatu...@yahoogroups.com
<mailto:amibroker-fullfeatu...@yahoogroups.com>
