Re: [GNC] Why do Imported Transactions NEED to be Matched?
Good information and very readable code on Bayesian matching - https://wiki.gnucash.org/wiki/Bayes ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
Hi, flywire writes: > Agree that import works in small chunks (as posted previously). Does that > response cover any way of getting GUIDs on the data? The GUIDs are 100% internal to GnuCash and unique per user per object (account, transaction, etc). There is no way to add GUIDs to imports, and even if you did it would never make sense because there is no way your bank or any other entity would know what GUID to use in your personal gnucash database. Just a case in point, even if you and I have the same account name, e.g. Expenses:Groceries, the GUID in my book and the GUID in your book would be different. Having said that, there *ARE* ways to map from textual account name to GUID, and the CSV and QIF importers already do that to some degree. > Please remember to CC this list on all your replies. > You can do this by using Reply-To-List or Reply-All. -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
A quick look through a copy of the data file suggests that it is somehow incorporated in the data for the Bayesian matching for an account but I can't be sure without looking through the code. I use Bayesian matching rather than the older import name mapping which can be used optionally. It is always possible to edit the datafile directly but that of course is fraught with danger. If you do, work on a copy of your data file only. I do this a fair bit while working on the documentation, creating a simple account structure in a new file, importing a couple of typical examples and then looking at how it is stored. Keeping the account structure simple makes finding things a lot easier. A lot of the data is stored in the XML data file as key-value pairs in a structure with tags around it. I haven't explored the import mapping procedure at all yet in any detail so I have no real idea of how it works. It was on the todo list to take a closer look at how it functions. The following is an extract from one of my import test files which I think indicates how the mapping is stored within the structure for each account: Acme Corp c39d8f38121b4ea0b6dcfbc65039ef77 STOCK ASX ACM 1 import-map csv-account-map * Assets:Investments:Brokerage Account:Stock:Acme Corp c39d8f38121b4ea0b6dcfbc65039ef77* f43d06a3681a42beaac15bfda775df7f I think the bolded bit is where the string in the transfer field is mapped onto the guid for an account. You would need to create a similar structure for each account which would be used as a transfer account, Note this is for CSV importing only, QIF more than likely has its own import map structure which i haven't explored at all. I don't normally import QIF data as a matter of course but have had my bank generate a couple I use for tesing the importer while documenting it. I honestly think it will be far simpler and less likely for you to end up with a corrupted data file by just mapping the files the first time they come up in the import matcher which will create these data structures automatically. I am deliberately discouraging you. I would never attempt this on my main accounting data files only on test files. David - David Cousens -- Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
Unfortunately knowing the GUID's won't help as the importer currently doesn't import them and use them directly only via that mapping process. - David Cousens -- Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
On 4/26/20 6:18 PM, flywire wrote: >> Do you know other (including closed source) apps don’t use something > similar? > > I've used many other financial apps that just import Transfer Account > directly in as listed in the csv. > >> Why would a user normally need to know this implementation detail? > Because it's counter-intuitive to load a Transfer Account field that is > exactly the same as the CoA in the GnuCash file and then have to match the > Transfer Account. But that is the sweet idea -- use a short mnemonic and have it mapped once. I do this to encode a bunch of transactions for a non-profit I manage. Keeps from having to type in the entire account path. > Agree that import works in small chunks (as posted previously). Does that > response cover any way of getting GUIDs on the data? > ___ > gnucash-user mailing list > gnucash-user@gnucash.org > To update your subscription preferences or to unsubscribe: > https://lists.gnucash.org/mailman/listinfo/gnucash-user > If you are using Nabble or Gmane, please see > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > - > Please remember to CC this list on all your replies. > You can do this by using Reply-To-List or Reply-All. -- Stephen M Butler, PMP, PSM stephen.m.butle...@gmail.com kg...@arrl.net 253-350-0166 --- GnuPG Fingerprint: 8A25 9726 D439 758D D846 E5D4 282A 5477 0385 81D8 ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
> Do you know other (including closed source) apps don’t use something similar? I've used many other financial apps that just import Transfer Account directly in as listed in the csv. > Why would a user normally need to know this implementation detail? Because it's counter-intuitive to load a Transfer Account field that is exactly the same as the CoA in the GnuCash file and then have to match the Transfer Account. Agree that import works in small chunks (as posted previously). Does that response cover any way of getting GUIDs on the data? ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
> On Apr 26, 2020 w18d117, at 7:28 PM, flywire wrote: > > @Tom Seesink - Will you log that as an enhancement bug at > https://bugs.gnucash.org/enter_bug.cgi?product=GnuCash (Import - other)? > Please link back here. > > @David Cousens, that excellent explanation about how the Import Matcher > works helps answer my question. The documentation needs to explain > the match is GUIDs rather than Transfer Accounts like other > financial software. Do you know other (including closed source) apps don’t use something similar? Why would a user normally need to know this implementation detail? If the import is done in small chunks, as advised (rather than one giant import with thousands of transactions) then the method behind the scenes isn’t an issue. Of course, GnuCash is open source, so anyone can go see for themselves that GUIDs are used without that fact being put in the docs. > > An enhancement bug fix will take too long. Is there any way of getting > GUIDs on the data? Perhaps reformatting it to QIF or OFX? Ultimately it is > stored in a table somewhere. Is the code the only place to find the details > on the import process? There are many threads on this list discussing the process. Also, bugs which were addressed by the changes will have some discussion in them. Until the docs are updated, this would be the only source of info other than the code itself. Regards, Adrien ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
@Tom Seesink - Will you log that as an enhancement bug at https://bugs.gnucash.org/enter_bug.cgi?product=GnuCash (Import - other)? Please link back here. @David Cousens, that excellent explanation about how the Import Matcher works helps answer my question. The documentation needs to explain the match is GUIDs rather than Transfer Accounts like other financial software. An enhancement bug fix will take too long. Is there any way of getting GUIDs on the data? Perhaps reformatting it to QIF or OFX? Ultimately it is stored in a table somewhere. Is the code the only place to find the details on the import process? ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
Tom Please Reply to the gnucash-user@gnucash.org (Reply to List or Reply to All) mailing list rather than individual contributors. The issue will get a wider response and the replies are then useful to all of the list participants. I am not a major contributor to the development but I think you will find the main reason for having to associate the external file names with the internal file names is that GnuCash does not use the account name strings internally for referring to accounts in the code but uses GUIDs as a searches for the GUID is likely more rapid than a search for a string. Therefore when the importer hits an account name you have entered in the transfer account column, and there is not a mapping of that string to the GUID, the software currently does not know where to direct it. A string search through the whole account structure which has to match a series of sub-strings is likely to be fairly expensive in computer time which is one possible reason the manual mapping was implemented. It should be just as efficient to map the Quicken categories directly onto the GnuCash account structure. In principle it is obviously possible to have GnuCash search through its list of account data structures for a match to the entered string and construct that mapping, but that code doesn't exist at the moment. The best way to raise this is using bugs.gnucash.org and to file a bug as a feature request against the program. This will at least get it on the developer group's radar. After that it is a matter of priorities and someone with the requisite skill sets and knowledge of the code picking it up and whether it can be implemented without too much of a computing time cost within the current architecture of GnuCash. The core of GnuCash is quite old (circa 1998) and I think was originally written in a LISP variant (Scheme) but it is very reliable. There are projects underway to convert the Scheme to C, C++, which fit in around the fixing of bugs in the existing code, but this has to take place without impact on the core functionality so progress is not necessarily all that rapid. Unless someone in the development team sees a way to implement this fairly quickly and easily it may take some considerable time to be up and running. Both the development and documentation teams are all volunteers and the resources to implement things we would like to see are not always available. I wouldn't be holding my breath. It is more productive to come to terms with the limitations in GnuCash and seek workarounds. The CSV importer was recently completely rewritten and a rewrite of the documentation is underway. The Help manual component in Ch 6 is largely complete. The Guide component which will address importing strategies and importing multiline multisplit data is still being written. Is your data CSV or QIF? In a given data file, that mapping process should be a one off, but like many things with the importer, AFAIK the map is probably only written to the data file when you successfully complete the import and if you don't do that or you start a new data file, it will have no mapping information stored, which is likely why you end up back at square one. Are you working with the full data file or have you pulled out a typical subset, e.g. one that has a transaction to each of the accounts/categories in the heirarchy, to get the importer setup and the map file constructed? If you do this first, you should then be able to import larger sections of the files with no problem. When I transferred from MYOB around 12 years ago I brought it in initally a month at a time and then a year at a time once I had the process sorted. If you go back through the gnucash-user archives you will find many references to initially splitting up large imports into small maneagable chunks initially and then only increasing the size of the chunks when you have the process running adequately. You will also find many discussions around importing Quicken data and optimal strategies. I also worked backwards importing the last year's data first, the next to last year's data, resetting the opening balances to the accounts as I entered each batch of prior data. I still had my old version of MYOB accessible then so I was able to generate the balance sheets at the start of each imported chunk which helped a lot. David On Sun, 2020-04-26 at 20:16 +1000, t...@sgml.net.au wrote: > David, > (This is not about matching during import.) > Over the last six+ weeks I have been importing my Quicken ledger and for > various reasons I had to do it over and over > again. > Because I purchased Quicken 2007 (with no export facility) I needed to find a > way to get all transactions and all > accounts used for the transactions. > So I achieved the export of Quicken accounts and Quicken categories and > created an accounts list for GC. Then I > exported all transactions from Quicken (and I had to split it into two >
Re: [GNC] Why do Imported Transactions NEED to be Matched?
The import was successful but I imagine the experience is like going through a binder and reaper blindfolded in the dark. The issues weren't all GnuCash. I reduced the form size, imported 50 (?) transactions (into a sqlite file) and then the rest. The account matching still occurred :(( but the transaction matching was all green :)) I'm still curious about why the account matching occurred so I'll test that with another dataset. My recollection was no feedback with windows popping up the "Application.exe has stopped working” error. Despite the process, the csv import process in other software is usually next to instant rather than tens of minutes, and there is almost always feedback. There was a lot of disk thrashing after the transaction matching. At the end of the process the message was file [name] successfully imported without any reporting of the number of transactions or errors, an anticlimax after that experience. Fairly sure no help buttons either. This is not really a new user experience as I've tried this on a couple of occasions previously and given up because I wasn't comfortable with the support system. I went to other software (but I'm back for the sql access). ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
Mainly because your particular use case is not the only one that the importing process has been designed to meet and the import process has to survive at least some common problems in the implementation by various sources of files and a considerable variety in the amount of information supplied by different sources of files. AFAIK you can't skip the initial step of assigning account names in the transfer field to accounts in the GnuCash account heirarchy which is a one off mapping step which occurs the first time an account name appears in a transfer account field. Once that mapping is created it is applied to all future import with that name in the transfer field. There is little value in assigning full account paths here as they still have to be mapped once (although there is no problem in doing so). If all your data contains a transfer account then you can effectively skip the matching process. If all the backgrounds to all the rows are green and have the A column checked, just hit OK. A possible new feature here would be an option which could be set in the importer user preferences to automatically trigger the OK if the above criteria were met on all rows. That doesn't exist at the moment. If any have a gold background a transfer account has not been assigned (could be a typo in the field for example) but this should not happen if you are specifying transfer accounts as the account mapping process should be initiated and you will assign an account. If either the U+R or R ( recently changed to U+C and C) checkboxes are checked and the backgound is green then an imported transaction has been matched against an existing transaction. The matching takes place within a date range window and some tolerances on the amount. If you are confident that these have not been imported previously just change the checkbox to A. A red background to a row indicates that the transaction did not meet the criteria for automatic assignment and that manual intervention is required to complete import of that row. David - David Cousens -- Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Why do Imported Transactions NEED to be Matched?
I have imported thousands of entries in CSV format--years of transactions--and found it helpful to split the original CSV into multiple parts (in my case, by year), saving the gnucash file after each import. For the very first import of a new bank account, I use a CSV with only three months of data, enough to teach gnucash the pattern of regular transactions without needing too many manual account assignments. Once that is imported the program has a better start at correctly assigning transactions. -- Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
