William, I have experienced the importer trying to match data out of the range of dates in the current import. It only occurred from memory when I first changed over to version 3.0. The matcher appeared to have lost all memory of what accounts to assign in the changeover form 2.6. However I found after importing 1-2 months data it was functioning normally again. I have been using the OFX importer for 3-4 years with OFX without any significant problems.
Your point about large data files sounds valid. I havent looked at the code for the match picker so I don't know how it works or whether it works on the historical data to extract the information it needs to make a choice of an accounts to assign or data to match. As it is a Bayesian mechanism at some point it has to examine the existing data and construct some sort of probability table, so my guess would be that this could be a step which is taking so long. Being able to set a preference for a date range or period to use in constructing the initial probability tables is probably a good idea if this is the case. My experience on the changeover from 2.6 to 3.0 when it appeared to have lost any memory of previous import assignments indicated that the importer was constructing those tables from the data it imports and not from the historical data, but I could be wrong. I would expect it to be using a Kalman filtering approach on the input data but can't be sure until I get a good look at the code. It did attempt to match transactions that were otherwise similar to transactions in the previous month or two initially. I only have data going back~8 years and have been retired for a large percentage of that so my files aren't huge so I may not be hitting your problem if it is the case that it does look further back. I think the decision about whether to import a small number of transactions by hand is really one for the user and not the importer to make. I would import small batches, maybe 20-30 to test the importer function and ensure it was working as expected before attempting to import 10k. On Wed, 2018-08-29 at 22:00 +0100, Wm via gnucash-devel wrote: > On 25/08/2018 07:22, David Cousens wrote: > > i thank David for his posting which i have read, I don't address all > he said > > > Keep trying. Tthe brain dead importer does get less brain dead with > > repeated > > use. > > i'm not sure it does get better as implemented because 2 of the bits > of > brain dead-ity are > > 1. the universe against which the importer is comparing imported tx > is > going to be growing so as a strategy it is doO0MED to sluggishness > and > eventually not being used unless there is some limit to the universe > (week / month / quarter / year / decade) > > 2. unless there is something better users are going to try and use > it > and become more frustrated and stop using it. > > ==== > > fairly easy to think about ways of fixing 1. like "do you want the > importer to really, really, really compare the imported tx against > your > stuff from the 1980's ? y/N" at the moment this is defaulting to Y > without asking and I don't think that makes sense. > > I mean, think of inflation? Why would one of anything in 2018 be > sensibly matched against the same thing 30 years ago? > > There isn't even the opportunity to time limit the universe and some > folk have stuff going back much longer than me and have many more tx > than me. > > fixing 2. just involves some thought about the user, almost no > programming. Redundant questions for the user would be, "you are > importing 3 tx, you have 10K tx in your file, this could take > fucking > hours, do you want to continue or just type them in by hand? if you > want my advice by hand is quicker" > > See? the importer has no idea of scale, 3 tx incoming ? I'll do it > by > hand. > > > > > > _______________________________________________ > gnucash-devel mailing list > gnucash-devel@gnucash.org > https://lists.gnucash.org/mailman/listinfo/gnucash-devel _______________________________________________ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel