Re: [GNC] Fwd: [GNC-dev] Fwd: Failed - import of QIF files from Quicken
You're right that scheme code logging is useful. The scheme way is to use pk, such as (pk "qif acct map is" qif-acct-map) But often you'll find that gnc objects are opaque; you can use (gnc:pk "qif acct map is" qif-acct-map) instead. You can combine multiple objects with pk or gnc:pk (gnc:pk "in qif-import:qif-to-gnc. File list is " qif-files-list " acct map is " qif-acct-map) etc. Good luck! On Wed, 19 Feb 2020, 8:36 am James Peterson, wrote: > Well, with no hint of what the problem is, even smaller > test sets won't necessarily find anything, so it could be > a lot of work for no benefit. For example, I could export > each account separately (although there are about 80 of them > so that's a lot of work), and each account could load > okay separately, because maybe the problem is an interaction > between two different accounts. So then I have to do 80*79 > possible pairs of accounts, and so on. > > My poking around in the > source code to try to find the problem suggests that the > problem is in the Scheme part that seems to be working with > all the accounts at the same time: > > ;; Build a local account tree to hold converted transactions. > > ;; Sort the account list on the depth of the account path. > > ;; Make all the accounts. > > ;; Run through the markable transactions marking any > ;; duplicates. marked transactions/splits won't get imported. > >;; Convert into a GnuCash transaction. >;; rebalance and commit everything > > So this looks like multiple runs thru the data, and even printing > the line number won't narrow it down -- you have to know what > step of the process we are doing, and where we are in the data > at that point. > > As things stand, we don't even know what the immediate problem > is -- a divide by zero? Looking for an account that should exist, > but doesn't?, an account type that it doesn't know about? > > The code is not written to allow it to even explain what it is > doing. If we are going to use this code base, someone who actually > understands, and is comfortable with the code will need to add > a "verbose input processing" level or something that will create > a log file explaining what it is going to try to do, then do it, > then explain what it got and how it interprets that. I've done > that with code before, but only C code, where, if need be, there is > the equivalent of a printf to a log file between every major step, > or each iteration of a loop, explaining what it thinks it is doing > and with what input and with what result. I just don't know how to > add that to scheme. > > jim > > > On Tue, 2020-02-18 at 16:53 -0600, Tommy Trussell wrote: > > On Tue, Feb 18, 2020 at 4:07 PM crazylyle wrote: > > > So you are suggesting reducing the size of our QIF files to a small > > > debuggable size. > > > > > > My QIF file is 653,808 lines long. About 2^20. So just using a binary > > > search would > > > take at least 20 trials to find the first line that it fails on. Not > > > exactly something that > > > seems practical. And that's just for my input file. > > > > > > And of course, just feeding a partial file might produce a failure, > but a > > > failure of a > > > different kind, so this won't necessarily get us anywhere. > > > > What bisections or limited imports have you tried? > > > > If you haven't already, try going into Quicken and exporting one day, one > > month, one year, one account... a smaller test batch and see if you get > the > > same error. > > > > I know years ago (maybe 2012 or earlier) when I switched from Quicken to > > GnuCash I found that Quicken created many odd little problems that were > just > > not worth the effort to clean up. SO I exported all my data (going back > to > > about 1989) into one QIF file that I never bothered to import, but the > data > > is still there if I need it. As you know, QIF is a text file, so you can > > search for transactions with a text editor. > > > > Then I went back and exported just the previous one or two years from > > Quicken (so for you it might be 2019 and 2020-to-date) and imported only > > that data into GnuCash. In order to make the import work its best, I > used a > > multi-step process, where I created the most-used accounts first, did a > few > > test imports, fixed a few small things in Quicken, then did my import. I > > cleaned up the few remaining import problems in GnuCash and haven't > looked > > back. > > > > UNFORTUNATELY for you, most of us who make the conversion never go back > and > > try the latest (there's no need) so the developers depend upon folks who > > encounter problems to fix them as they arise. > > > > > > > Please remember to CC this list on all your replies. > > > You can do this by using Reply-To-List or Reply-All. > > ___ > gnucash-user mailing list > gnucash-user@gnucash.org > To update your subscription preferences or to unsubscribe: >
Re: [GNC] Fwd: [GNC-dev] Fwd: Failed - import of QIF files from Quicken
Well, with no hint of what the problem is, even smaller test sets won't necessarily find anything, so it could be a lot of work for no benefit. For example, I could export each account separately (although there are about 80 of them so that's a lot of work), and each account could load okay separately, because maybe the problem is an interaction between two different accounts. So then I have to do 80*79 possible pairs of accounts, and so on. My poking around in the source code to try to find the problem suggests that the problem is in the Scheme part that seems to be working with all the accounts at the same time: ;; Build a local account tree to hold converted transactions. ;; Sort the account list on the depth of the account path. ;; Make all the accounts. ;; Run through the markable transactions marking any ;; duplicates. marked transactions/splits won't get imported. ;; Convert into a GnuCash transaction. ;; rebalance and commit everything So this looks like multiple runs thru the data, and even printing the line number won't narrow it down -- you have to know what step of the process we are doing, and where we are in the data at that point. As things stand, we don't even know what the immediate problem is -- a divide by zero? Looking for an account that should exist, but doesn't?, an account type that it doesn't know about? The code is not written to allow it to even explain what it is doing. If we are going to use this code base, someone who actually understands, and is comfortable with the code will need to add a "verbose input processing" level or something that will create a log file explaining what it is going to try to do, then do it, then explain what it got and how it interprets that. I've done that with code before, but only C code, where, if need be, there is the equivalent of a printf to a log file between every major step, or each iteration of a loop, explaining what it thinks it is doing and with what input and with what result. I just don't know how to add that to scheme. jim On Tue, 2020-02-18 at 16:53 -0600, Tommy Trussell wrote: > On Tue, Feb 18, 2020 at 4:07 PM crazylyle wrote: > > So you are suggesting reducing the size of our QIF files to a small > > debuggable size. > > > > My QIF file is 653,808 lines long. About 2^20. So just using a binary > > search would > > take at least 20 trials to find the first line that it fails on. Not > > exactly something that > > seems practical. And that's just for my input file. > > > > And of course, just feeding a partial file might produce a failure, but a > > failure of a > > different kind, so this won't necessarily get us anywhere. > > What bisections or limited imports have you tried? > > If you haven't already, try going into Quicken and exporting one day, one > month, one year, one account... a smaller test batch and see if you get the > same error. > > I know years ago (maybe 2012 or earlier) when I switched from Quicken to > GnuCash I found that Quicken created many odd little problems that were just > not worth the effort to clean up. SO I exported all my data (going back to > about 1989) into one QIF file that I never bothered to import, but the data > is still there if I need it. As you know, QIF is a text file, so you can > search for transactions with a text editor. > > Then I went back and exported just the previous one or two years from > Quicken (so for you it might be 2019 and 2020-to-date) and imported only > that data into GnuCash. In order to make the import work its best, I used a > multi-step process, where I created the most-used accounts first, did a few > test imports, fixed a few small things in Quicken, then did my import. I > cleaned up the few remaining import problems in GnuCash and haven't looked > back. > > UNFORTUNATELY for you, most of us who make the conversion never go back and > try the latest (there's no need) so the developers depend upon folks who > encounter problems to fix them as they arise. > > > > Please remember to CC this list on all your replies. > > You can do this by using Reply-To-List or Reply-All. ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Fwd: [GNC-dev] Fwd: Failed - import of QIF files from Quicken
On Tue, Feb 18, 2020 at 4:07 PM crazylyle wrote: > So you are suggesting reducing the size of our QIF files to a small > debuggable size. > > My QIF file is 653,808 lines long. About 2^20. So just using a binary > search would > take at least 20 trials to find the first line that it fails on. Not > exactly something that > seems practical. And that's just for my input file. > > And of course, just feeding a partial file might produce a failure, but a > failure of a > different kind, so this won't necessarily get us anywhere. > What bisections or limited imports have you tried? If you haven't already, try going into Quicken and exporting one day, one month, one year, one account... a smaller test batch and see if you get the same error. I know years ago (maybe 2012 or earlier) when I switched from Quicken to GnuCash I found that Quicken created many odd little problems that were just not worth the effort to clean up. SO I exported all my data (going back to about 1989) into one QIF file that I never bothered to import, but the data is still there if I need it. As you know, QIF is a text file, so you can search for transactions with a text editor. Then I went back and exported just the previous one or two years from Quicken (so for you it might be 2019 and 2020-to-date) and imported only that data into GnuCash. In order to make the import work its best, I used a multi-step process, where I created the most-used accounts first, did a few test imports, fixed a few small things in Quicken, then did my import. I cleaned up the few remaining import problems in GnuCash and haven't looked back. UNFORTUNATELY for you, most of us who make the conversion never go back and try the latest (there's no need) so the developers depend upon folks who encounter problems to fix them as they arise. > Please remember to CC this list on all your replies. > You can do this by using Reply-To-List or Reply-All. > ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Fwd: [GNC-dev] Fwd: Failed - import of QIF files from Quicken
So you are suggesting reducing the size of our QIF files to a small debuggable size. My QIF file is 653,808 lines long. About 2^20. So just using a binary search would take at least 20 trials to find the first line that it fails on. Not exactly something that seems practical. And that's just for my input file. And of course, just feeding a partial file might produce a failure, but a failure of a different kind, so this won't necessarily get us anywhere. -- Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
[GNC] Fwd: [GNC-dev] Fwd: Failed - import of QIF files from Quicken
Forward from devel -- Forwarded message - From: Geert Janssens Date: Tue, 18 Feb 2020, 3:48 pm Subject: Re: [GNC-dev] Fwd: [GNC] Failed - import of QIF files from Quicken To: Cc: Christopher Lam Hi Tom, We don't need your complete financial history, just the bit that causes the import to fail. Or even 'a' bit that fails. So if you can create a small Qif file that illustrates the failure that would help already. Perhaps you're able to generate that by a process of elimination ? For example you could start by exporting each category in Quicken to a separate qif file and then test which one fail(s) to import. For each one, that fails, redo the export, this time exporting half of the transactions to one file and the other half to another. Continue to reduce the number of exported transactions until you're either down to only one failing transaction or there are no more errors. In case of one failing transaction, you can then tweak the qif file to anonymize it (and retest if the anonymized file continues to error out on import) and share it with us. Regards, Geert > -- Forwarded message - > From: Tom Hatzigeorgiou > Date: Tue, 18 Feb 2020, 1:20 pm > Subject: Re: [GNC] Failed - import of QIF files from Quicken > To: Christopher Lam > > > I understand that you want to recreate the problem so that you can fix it > but this is my whole financial history and I wouldn't know where to start > to anonymize it > so I wouldn't be comfortable giving it away > > You should know that I am a programmer so I think I could follow > instructions and could debug the importer for you to find out where it > breaks at least with my data > I would be wiling to do that because I like GNUCASH, I hate what Quicken is > doing to long time users (like me), and I would keep control of my data. > > > > -- > *From:* Christopher Lam > *Sent:* Monday, February 17, 2020 5:55 PM > *To:* Tom Hatzigeorgiou > *Cc:* gnucash-user@gnucash.org > *Subject:* Re: [GNC] Failed - import of QIF files from Quicken > > Hello > If you'll be kind to offer the QIF file in bugzilla or privately -- feel > free to amend it manually first to anonymize details, it will be useful to > debug. > C > > > On Mon, 17 Feb 2020 at 15:09, Tom Hatzigeorgiou wrote: > > I have seen few emails on this subject. > > Please see if the below statements would help you. > > > I've been having a problem with importing my data from Quicken and into > GNUCASH for Windows using a QIF file created by Quicken. I’ve over 30 years > of data with thousands of transactions spanning multiple accounts and > categories. > > I have found that there is an issue with the importer starting with version > 3.2 (I think). > > As mentioned by others, the importer would fail without giving an error. > > I have solved the issue by installing GNUCASH 3.1 importing my data and > then installing the latest version (currently 3.8). Of course this assumes > that you will be able to find the 3.1 install file. I hope you can find it > in one of the archives. > > I don’t see any problem with my data after I open the file (in GNU 3.8) > created by the GNUCASH 3.1. > > I was wondering if the developers could replace the importer (of the latest > version) with the version that existed in the 3.1 version (until the import > issue is fixed). I believe it would solved a lot of problems for a lot of > people. > > I have another issue getting security prices and bank transaction. I have > read a lot of solutions but none has help me. When I find a simple solution > to this problem I will post here. > > > ___ > gnucash-user mailing list > gnucash-user@gnucash.org > To update your subscription preferences or to unsubscribe: > https://lists.gnucash.org/mailman/listinfo/gnucash-user > If you are using Nabble or Gmane, please see > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > - > Please remember to CC this list on all your replies. > You can do this by using Reply-To-List or Reply-All. > ___ > gnucash-devel mailing list > gnucash-de...@gnucash.org > https://lists.gnucash.org/mailman/listinfo/gnucash-devel ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
