Re: Revisiting problems with price updates
On Montag, 28. Februar 2022 17:04:10 CET Jack via KMyMoney-devel wrote: > On 2/28/22 01:46, Thomas Baumgart via KMyMoney-devel wrote: > > On Mittwoch, 23. Februar 2022 00:15:26 CET Jack via KMyMoney-devel wrote: > >> As this is getting rather long winded, I'm going to trim down to mainly > >> stuff I'm directly replying to. > >> > >> On 2022.02.22 02:34, Thomas Baumgart via KMyMoney-devel wrote: > >>> On Dienstag, 22. Februar 2022 01:47:21 CET Jack via KMyMoney-devel > >>> wrote: > >> [snip] > As one example, I have an equity E416 for symbol CARR (Carrier > Global Corp.) > If I look in the Price Editor for CARR, I see two entries - one is > from KMyMOney Currency and the other from Yahoo Finance. It is not > clear why there are two en tries for the same ID and Name. > >>> See below. What are the dates of these entries? > >> 1/6/21 Currency and 2/4/22 Yahoo Finance. Probable the most recent > >> price of each of the two pricepairs. > > I added code to remove those 'invalid' price pairs. Please have a look at > > the > > console next time you save the file. It will not save those entries anymore > > and afterwards they should be gone. If they re-appear, we need to figure out > > why and what causes it. Again, having a look at the console for those > > debug messages about price pairs is important. > I confirm this works as you describe. The problem with console messages > is that unless you are explicitly looking for them, things like this do > not stand out much from the usual, info messages. Another future thought > - to distinguish info from warning from error in console output, > although I'm not sure there is any good, reliable way. Neiter do I :( I know, but for now it is just a debug thingy we need to look at. -- Regards Thomas Baumgart https://www.signal.org/ Signal, the better WhatsApp - Life would be so much easier if we could just look at the source code. -- Dave Olson - signature.asc Description: This is a digitally signed message part.
Re: Revisiting problems with price updates
On 2/28/22 01:46, Thomas Baumgart via KMyMoney-devel wrote: On Mittwoch, 23. Februar 2022 00:15:26 CET Jack via KMyMoney-devel wrote: As this is getting rather long winded, I'm going to trim down to mainly stuff I'm directly replying to. On 2022.02.22 02:34, Thomas Baumgart via KMyMoney-devel wrote: On Dienstag, 22. Februar 2022 01:47:21 CET Jack via KMyMoney-devel wrote: [snip] As one example, I have an equity E416 for symbol CARR (Carrier Global Corp.) If I look in the Price Editor for CARR, I see two entries - one is from KMyMOney Currency and the other from Yahoo Finance. It is not clear why there are two en tries for the same ID and Name. See below. What are the dates of these entries? 1/6/21 Currency and 2/4/22 Yahoo Finance. Probable the most recent price of each of the two pricepairs. I added code to remove those 'invalid' price pairs. Please have a look at the console next time you save the file. It will not save those entries anymore and afterwards they should be gone. If they re-appear, we need to figure out why and what causes it. Again, having a look at the console for those debug messages about price pairs is important. I confirm this works as you describe. The problem with console messages is that unless you are explicitly looking for them, things like this do not stand out much from the usual, info messages. Another future thought - to distinguish info from warning from error in console output, although I'm not sure there is any good, reliable way. I did not try to follow the details you described (maybe at a later time when we have more evidence when all this happens). [...]
Re: Revisiting problems with price updates
On Mittwoch, 23. Februar 2022 00:15:26 CET Jack via KMyMoney-devel wrote: > As this is getting rather long winded, I'm going to trim down to mainly > stuff I'm directly replying to. > > On 2022.02.22 02:34, Thomas Baumgart via KMyMoney-devel wrote: > > On Dienstag, 22. Februar 2022 01:47:21 CET Jack via KMyMoney-devel > > wrote: > [snip] > >> As one example, I have an equity E416 for symbol CARR (Carrier > >> Global Corp.) > >> If I look in the Price Editor for CARR, I see two entries - one is > >> from KMyMOney Currency and the other from Yahoo Finance. It is not > >> clear why there are two en tries for the same ID and Name. > > > > See below. What are the dates of these entries? > > 1/6/21 Currency and 2/4/22 Yahoo Finance. Probable the most recent > price of each of the two pricepairs. I added code to remove those 'invalid' price pairs. Please have a look at the console next time you save the file. It will not save those entries anymore and afterwards they should be gone. If they re-appear, we need to figure out why and what causes it. Again, having a look at the console for those debug messages about price pairs is important. I did not try to follow the details you described (maybe at a later time when we have more evidence when all this happens). [...] -- Regards Thomas Baumgart https://www.signal.org/ Signal, the better WhatsApp - Laziness led to the invention of the most useful tools. - signature.asc Description: This is a digitally signed message part.
Re: Revisiting problems with price updates
As this is getting rather long winded, I'm going to trim down to mainly stuff I'm directly replying to. On 2022.02.22 02:34, Thomas Baumgart via KMyMoney-devel wrote: On Dienstag, 22. Februar 2022 01:47:21 CET Jack via KMyMoney-devel wrote: [snip] As one example, I have an equity E416 for symbol CARR (Carrier Global Corp.) If I look in the Price Editor for CARR, I see two entries - one is from KMyMOney Currency and the other from Yahoo Finance. It is not clear why there are two en tries for the same ID and Name. See below. What are the dates of these entries? 1/6/21 Currency and 2/4/22 Yahoo Finance. Probable the most recent price of each of the two pricepairs. If I show all stored prices, the sources include Transaction, User, Yahoo Finance, Finance::Quote yahoo_json, Statement Importer, and KMyMoney Currency. All make sense to me except for the last. In addition, if I look in my kmy file, I have five entries for including one User and four KMyMoney Currency. Those are the ones causing the trouble (I think). Agreed, but then why would there be KMyMoney Currency prices in the other batch? See below for info on those price entries. The User one seems to be entered by the user (either directly as price or by a transaction). Do you see a relationship between this price entry and a transaction? The ones showing KMyMoney Currency are a cause of this if they have a later date. Can you confirm? Here is a review of all the prices. My first shares of this stock were obtained as a dividend from a different stock which I manually entered as "Add shares" on 4/7/20. The User entry on that date was my entry of the price indicated for the date in my broker statement. That is NOT the User entry mentioned above. The F::Q entries from 5/5 to 10/30/20 were normal online price updates. There were sales in three accounts on 9/21/20, thus the transaction price, but I don't know why I would have manually entered the transactions instead of using ofx import. I cannot think of why I would have manually entered a price on that date, but that is the User entry mentioned above. KMyMoney Currency (KMMC) prices on 10/1 and 10/30/20 also had F::Q prices on the same dates. KMM Currency price 12/1/20 had a Yahoo Finance price the same date. These are the three KMMC prices mentioned above. KMMC price 1/6/21 had Statement Importer price the same date, and was for sales in four accounts. This and all further KMMC prices are in the "right" direction pricepair. KMMC prices from 2/7/21 through 1/4/22 were probably normal online price updates. Yahoo Finance update 2/4/22 was my most recent online price update. I'm pretty sure I set Yahoo Finance set as the price source far longer than a month ago. I do have a bunch of backups, but I'd rather not go hunting through them manually. If there is specific info to look for, I can put together a shell script or maybe a Perl script using the XML parser. Then, in the Price Editor, the Stock name for CARR is "Carrier Global Corp,Carrier Global Corp,Carrier Global Corp,Carrier Global Corp,Carrier Global Corp". How did the name get repeated five times? Good question. I don't have an answer (yet). Leaving so we don't forget, although it's only a cosmetic issue (I hope.) Then, if I look in the Price Updater, I see an entry for ID: CARR with Internal ID: E000416 and Source: Yahoo Finance. However, if I scroll down, I see ID: USD > E000416 with internal ID: "USE E000416" a price of "CARR0.0218 and Source KMyMOney Currency. The internal ID should probably be "USD E000416" and is a typo on your end. Sorry, that was a typo, the InternalID is USD E000614 (not USE). The price of "CARR0.0218" is certainly weird. Also note, there is no "CARR0" string anywhere in my data file, according to grep. There are four other similar entries, where the price is the stock symbol followed by "0." and a four digit number which doesn't match anything I can think of. Regarding the names Yahoo Finance and KMyMoney Currency I can point to https://invent.kde.org/office/kmymoney/-/commit/8559f9ec which we changed when Yahoo was not available a few years ago. Still, I don't understand why the currency exchange rate source is used for an investment price update. That's what we call the $64,000 question. (There was long ago a TV quiz show in the US with that name, and that was the grand prize if you answered all their questions.) I originally used Yaoo, and when it started failing, inconsistently changed between F::Q yahoo_json, Stooq, and a small use of Financial Times. Searching my kmy file for E000416 shows only the Security definition, the five Accounts holding that stock, and the two PRICEPAIR entries already mentioned. In particular, I don't see what generates that USD > E000416 entry in the price updater. My currency list includes only USD and EUR. > > Any though
Re: Revisiting problems with price updates
On Dienstag, 22. Februar 2022 01:47:21 CET Jack via KMyMoney-devel wrote:
> The latest is https://bugs.kde.org/show_bug.cgi?id=450426. I think
> this is just another example of an issue that has been mentioned before
> on the mailing list, forum, and bug reports. The common complaint is
> that the price updater fails - but if you look carefully, it seems to
> be trying to retrieve a pair that doesn't make sense - it is reversed
> from what you would expect - the value of a currency in terms of a
> stock, instead of the price of the stock in the currency.
Indeed, that does not make sense and it is what is causing the failure during
online download. As of now, I have no idea how these entries get created.
In terms of the reported "I was able to delete them but once I ran
the update they appeared again" I think the downloader checks which
combinations are on file and creates them as it goes. I have to investigate
this.
> When i started looking into my own kmy file, I've found things I can't
> explain, that I think are related. Hopefully something here will
> trigger some thought and give a hint where we might look for the
> underlying problem..
>
> As one example, I have an equity E416 for symbol CARR (Carrier
> Global Corp.)
> If I look in the Price Editor for CARR, I see two entries - one is from
> KMyMOney Currency and the other from Yahoo Finance. It is not clear
> why there are two en tries for the same ID and Name.
See below. What are the dates of these entries?
> If I show all
> stored prices, the sources include Transaction, User, Yahoo Finance,
> Finance::Quote yahoo_json, Statement Importer, and KMyMoney Currency.
> All make sense to me except for the last. In addition, if I look in my
> kmy file, I have five entries for
> including one User and four KMyMoney Currency.
Those are the ones causing the trouble (I think). The User one seems to
be entered by the user (either directly as price or by a transaction).
Do you see a relationship between this price entry and a transaction? The
ones showing KMyMoney Currency are a cause of this if they have a later
date. Can you confirm?
> from="E000416" to="USD"> includes all the rest. I had almost expected
> all the KMyMoney Currency entries to be in one group. Can anyone
> explain why a stock price would ever get looked up through KMyMoney
> Currency? If I sort those price entries by date, all the KMyMoney
> Currency entries are in the latter half of the list. In a few cases
> where there is also another entry on the same date, they differ by a
> few cents - in only one case are they identical.
>
> Then, in the Price Editor, the Stock name for CARR is "Carrier Global
> Corp,Carrier Global Corp,Carrier Global Corp,Carrier Global
> Corp,Carrier Global Corp". How did the name get repeated five times?
Good question. I don't have an answer (yet).
> It only appears once in the Investment Views (once under Securities,
> and once under Equities tab for each account which held that stock (all
> not closed except one.) If I scroll through the complete Price Editor,
> there are several other entries with the Stock Name repeated multiple
> times, but I see no pattern.
>
> Then, if I look in the Price Updater, I see an entry for ID: CARR with
> Internal ID: E000416 and Source: Yahoo Finance. However, if I scroll
> down, I see ID: USD > E000416 with internal ID: "USE E000416" a price
> of "CARR0.0218 and Source KMyMOney Currency.
The internal ID should probably be "USD E000416" and is a typo on your end.
The price of "CARR0.0218" is certainly weird.
Regarding the names Yahoo Finance and KMyMoney Currency I can point to
https://invent.kde.org/office/kmymoney/-/commit/8559f9ec which we changed
when Yahoo was not available a few years ago. Still, I don't understand
why the currency exchange rate source is used for an investment price
update.
> Searching my kmy file for E000416 shows only the Security definition,
> the five Accounts holding that stock, and the two PRICEPAIR entries
> already mentioned. In particular, I don't see what generates that
> USD > E000416 entry in the price updater. My currency list includes
> only USD and EUR.
>
> Any thoughts or suggestions for further troubleshooting?
I scanned the term "KMyMoney Currency" and only found it in a few spots:
in WebPriceQuote::launchCSV it is only used as reference:
// for historical exchange rates we switch to Stooq
if (sourcename == QLatin1String("KMyMoney Currency"))
sourcename = QLatin1String("Stooq Currency");
and for the generation of the source information to select from:
result["KMyMoney Currency"] = WebPriceQuoteSource("KMyMoney Currency",
"https://fx-rate.net/%1/%2";,
QString(),
"https://fx-rate.net/([^/]+/[^/]+)",
the third location is the one introduced by above mentioned
Revisiting problems with price updates
The latest is https://bugs.kde.org/show_bug.cgi?id=450426. I think this is just another example of an issue that has been mentioned before on the mailing list, forum, and bug reports. The common complaint is that the price updater fails - but if you look carefully, it seems to be trying to retrieve a pair that doesn't make sense - it is reversed from what you would expect - the value of a currency in terms of a stock, instead of the price of the stock in the currency. When i started looking into my own kmy file, I've found things I can't explain, that I think are related. Hopefully something here will trigger some thought and give a hint where we might look for the underlying problem.. As one example, I have an equity E416 for symbol CARR (Carrier Global Corp.) If I look in the Price Editor for CARR, I see two entries - one is from KMyMOney Currency and the other from Yahoo Finance. It is not clear why there are two en tries for the same ID and Name. If I show all stored prices, the sources include Transaction, User, Yahoo Finance, Finance::Quote yahoo_json, Statement Importer, and KMyMoney Currency. All make sense to me except for the last. In addition, if I look in my kmy file, I have five entries for including one User and four KMyMoney Currency. from="E000416" to="USD"> includes all the rest. I had almost expected all the KMyMoney Currency entries to be in one group. Can anyone explain why a stock price would ever get looked up through KMyMoney Currency? If I sort those price entries by date, all the KMyMoney Currency entries are in the latter half of the list. In a few cases where there is also another entry on the same date, they differ by a few cents - in only one case are they identical. Then, in the Price Editor, the Stock name for CARR is "Carrier Global Corp,Carrier Global Corp,Carrier Global Corp,Carrier Global Corp,Carrier Global Corp". How did the name get repeated five times? It only appears once in the Investment Views (once under Securities, and once under Equities tab for each account which held that stock (all not closed except one.) If I scroll through the complete Price Editor, there are several other entries with the Stock Name repeated multiple times, but I see no pattern. Then, if I look in the Price Updater, I see an entry for ID: CARR with Internal ID: E000416 and Source: Yahoo Finance. However, if I scroll down, I see ID: USD > E000416 with internal ID: "USE E000416" a price of "CARR0.0218 and Source KMyMOney Currency. Searching my kmy file for E000416 shows only the Security definition, the five Accounts holding that stock, and the two PRICEPAIR entries already mentioned. In particular, I don't see what generates that USD > E000416 entry in the price updater. My currency list includes only USD and EUR. Any thoughts or suggestions for further troubleshooting? Jack
