Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs
Christian, After you create an enhancement ticket, presumably generalized for all applications of the undocumented feature upgrade details for typical cases, please report the bug number here so others that do not get all new bug reports automatically can find it more easily. David Carlson On Sun, Nov 10, 2019 at 4:29 PM Christian Gruber wrote: > > Am 10.11.19 um 22:39 schrieb John Ralls: > > > >> On Nov 10, 2019, at 12:07 PM, Christian Gruber < > christian.gru...@posteo.de> wrote: > >> > >> > >> Am 08.11.19 um 23:08 schrieb John Ralls: > On Nov 8, 2019, at 1:58 PM, Christian Gruber < > christian.gru...@posteo.de> wrote: > > Am 08.11.19 um 04:39 schrieb John Ralls: > > Christian, > > > > It's not that it's not prepared for Bayesian matching, it's that > older versions of GnuCash stored the Bayesian match tokens hierarchically. > Aaron Laws (lmat) changed it to a flatter structure with somewhat better > memory locality for faster access. imap_convert_bayes_to_flat should run > once to convert the data and set the feature, after which > check_import_map_data will see the flag and return. A file created with 3.x > and Baysian maps would already have the feature set. > > > > > > With that background, to your questions: > > > > Why does it take so long? Because it traverses the entire tree of > accounts, every time. The test book has 1127 accounts. Add to that that > there are some things inside the loop that shouldn't be and that > convert_imap_account_bayes_to_flat doesn't use some obvious short circuits > and you get taking a long time. > > > > Why does it run twice? Because there aren't any accounts with > import-map-bayes slots, so it does no conversions so it doesn't set the > feature. > Why isn't the feature set in any case after conversion is done, > whether there was any slot to convert or not? > > I would expect that, if the hierarchical structure should not be used > anymore. I wouldn't only set the feature, if there has been any error > during conversion. But it's not an error, if > convert_imap_account_bayes_to_flat() returns false. > >>> The feature isn't set because that would prevent using the file with > an older version of GnuCash for no good reason: There aren't any > import-map-bayes tags in the new format so there's nothing for the older > version to mess up. > >>> > >>> Regards, > >>> John Ralls > >>> > >> Sounds meaningful, but it's not obvious. The conversion process is not > much transparent to the user. The user is not informed about conversion and > additionally the conditions, when the GnuCash file is converted and when > not, are not quite intuitive, I guess. And the second question for me is, > if this is really a relevant use case to keep the GnuCash file unconverted > if possible, when it is used with a newer GnuCash version. Maybe it's > relevant, if several people, who use different GnuCash versions work on the > same GnuCash file. But what happens, when one with a newer GnuCash version > uses the bayesian import matcher for the first time, which silently > converts the GnuCash file? Will it still be usable with an older GnuCash > version? > >> > >> But actually this is another discussion, which is not important for the > bug ticket. What's more important, I created a new fresh GnuCash file with > the SKR04 accounts template using my GnuCash version 3.7 and if I checked > this correctly, even with this GnuCash version the feature > GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you > confirm this? If yes, then the bug report is relevant not only for > migration from older to current GnuCash versions but even for current > GnuCash versions in general. > > No, it's the same discussion. If you don't use the feature then the flag > for it won't be set. That's how we designed the feature mechanism. The use > case is straightforward: Some user has two computers, a laptop running > Windows and a desktop running Ubuntu 16.04. She updates the Windows machine > with each release but doesn't know how to build GnuCash on her Ubuntu > machine so leaves it at whatever they shipped on 16.04; 2.6.12 IIRC. As > long as she doesn't use any of the features supported only in 3.x she can > continue to use both systems to keep her books and regardless of which > machine she uses to create a book. > > > > The "silently" part could be better. It would indeed be a nice > enhancement to pop a dialog box when GnuCash is about to set a feature flag > explaining that it's use will preclude using the file on a GnuCash version > older than X and giving the user a chance to bail out. It would be nicer > still to persist that decision so that it need ask only once, and if the > user assents it could set the feature flag even if the feature isn't used, > as is the case here. Of course there would then be the need to reset > negative responses so that when our hypothetical user upgrades her desktop > to Ubuntu 20.04
Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs
Am 10.11.19 um 22:39 schrieb John Ralls: On Nov 10, 2019, at 12:07 PM, Christian Gruber wrote: Am 08.11.19 um 23:08 schrieb John Ralls: On Nov 8, 2019, at 1:58 PM, Christian Gruber wrote: Am 08.11.19 um 04:39 schrieb John Ralls: Christian, It's not that it's not prepared for Bayesian matching, it's that older versions of GnuCash stored the Bayesian match tokens hierarchically. Aaron Laws (lmat) changed it to a flatter structure with somewhat better memory locality for faster access. imap_convert_bayes_to_flat should run once to convert the data and set the feature, after which check_import_map_data will see the flag and return. A file created with 3.x and Baysian maps would already have the feature set. With that background, to your questions: Why does it take so long? Because it traverses the entire tree of accounts, every time. The test book has 1127 accounts. Add to that that there are some things inside the loop that shouldn't be and that convert_imap_account_bayes_to_flat doesn't use some obvious short circuits and you get taking a long time. Why does it run twice? Because there aren't any accounts with import-map-bayes slots, so it does no conversions so it doesn't set the feature. Why isn't the feature set in any case after conversion is done, whether there was any slot to convert or not? I would expect that, if the hierarchical structure should not be used anymore. I wouldn't only set the feature, if there has been any error during conversion. But it's not an error, if convert_imap_account_bayes_to_flat() returns false. The feature isn't set because that would prevent using the file with an older version of GnuCash for no good reason: There aren't any import-map-bayes tags in the new format so there's nothing for the older version to mess up. Regards, John Ralls Sounds meaningful, but it's not obvious. The conversion process is not much transparent to the user. The user is not informed about conversion and additionally the conditions, when the GnuCash file is converted and when not, are not quite intuitive, I guess. And the second question for me is, if this is really a relevant use case to keep the GnuCash file unconverted if possible, when it is used with a newer GnuCash version. Maybe it's relevant, if several people, who use different GnuCash versions work on the same GnuCash file. But what happens, when one with a newer GnuCash version uses the bayesian import matcher for the first time, which silently converts the GnuCash file? Will it still be usable with an older GnuCash version? But actually this is another discussion, which is not important for the bug ticket. What's more important, I created a new fresh GnuCash file with the SKR04 accounts template using my GnuCash version 3.7 and if I checked this correctly, even with this GnuCash version the feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you confirm this? If yes, then the bug report is relevant not only for migration from older to current GnuCash versions but even for current GnuCash versions in general. No, it's the same discussion. If you don't use the feature then the flag for it won't be set. That's how we designed the feature mechanism. The use case is straightforward: Some user has two computers, a laptop running Windows and a desktop running Ubuntu 16.04. She updates the Windows machine with each release but doesn't know how to build GnuCash on her Ubuntu machine so leaves it at whatever they shipped on 16.04; 2.6.12 IIRC. As long as she doesn't use any of the features supported only in 3.x she can continue to use both systems to keep her books and regardless of which machine she uses to create a book. The "silently" part could be better. It would indeed be a nice enhancement to pop a dialog box when GnuCash is about to set a feature flag explaining that it's use will preclude using the file on a GnuCash version older than X and giving the user a chance to bail out. It would be nicer still to persist that decision so that it need ask only once, and if the user assents it could set the feature flag even if the feature isn't used, as is the case here. Of course there would then be the need to reset negative responses so that when our hypothetical user upgrades her desktop to Ubuntu 20.04 next spring she can turn on all of those 3.x only features she'd previously declined. It occurs to me that another improvement to the existing flat Bayesian conversion would be to check the state of the GNC_PREF_USE_BAYES and skip the conversion if it's not true. This is already implemented. Flag GNC_PREF_USE_BAYES is checked in functions matchmap_find_destination() and matchmap_store_destination() and if not set, than functions gnc_account_imap_find_account_bayes() and gnc_account_imap_add_account_bayes() aren't invoked and no conversion will be done. Regards, John Ralls Thanks for the revealing explanation of the feature mechanism. In
Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs
> On Nov 10, 2019, at 12:07 PM, Christian Gruber > wrote: > > > Am 08.11.19 um 23:08 schrieb John Ralls: >> >>> On Nov 8, 2019, at 1:58 PM, Christian Gruber >>> wrote: >>> >>> Am 08.11.19 um 04:39 schrieb John Ralls: Christian, It's not that it's not prepared for Bayesian matching, it's that older versions of GnuCash stored the Bayesian match tokens hierarchically. Aaron Laws (lmat) changed it to a flatter structure with somewhat better memory locality for faster access. imap_convert_bayes_to_flat should run once to convert the data and set the feature, after which check_import_map_data will see the flag and return. A file created with 3.x and Baysian maps would already have the feature set. With that background, to your questions: Why does it take so long? Because it traverses the entire tree of accounts, every time. The test book has 1127 accounts. Add to that that there are some things inside the loop that shouldn't be and that convert_imap_account_bayes_to_flat doesn't use some obvious short circuits and you get taking a long time. Why does it run twice? Because there aren't any accounts with import-map-bayes slots, so it does no conversions so it doesn't set the feature. >>> Why isn't the feature set in any case after conversion is done, whether >>> there was any slot to convert or not? >>> >>> I would expect that, if the hierarchical structure should not be used >>> anymore. I wouldn't only set the feature, if there has been any error >>> during conversion. But it's not an error, if >>> convert_imap_account_bayes_to_flat() returns false. >> The feature isn't set because that would prevent using the file with an >> older version of GnuCash for no good reason: There aren't any >> import-map-bayes tags in the new format so there's nothing for the older >> version to mess up. >> >> Regards, >> John Ralls >> > Sounds meaningful, but it's not obvious. The conversion process is not much > transparent to the user. The user is not informed about conversion and > additionally the conditions, when the GnuCash file is converted and when not, > are not quite intuitive, I guess. And the second question for me is, if this > is really a relevant use case to keep the GnuCash file unconverted if > possible, when it is used with a newer GnuCash version. Maybe it's relevant, > if several people, who use different GnuCash versions work on the same > GnuCash file. But what happens, when one with a newer GnuCash version uses > the bayesian import matcher for the first time, which silently converts the > GnuCash file? Will it still be usable with an older GnuCash version? > > But actually this is another discussion, which is not important for the bug > ticket. What's more important, I created a new fresh GnuCash file with the > SKR04 accounts template using my GnuCash version 3.7 and if I checked this > correctly, even with this GnuCash version the feature > GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you > confirm this? If yes, then the bug report is relevant not only for migration > from older to current GnuCash versions but even for current GnuCash versions > in general. No, it's the same discussion. If you don't use the feature then the flag for it won't be set. That's how we designed the feature mechanism. The use case is straightforward: Some user has two computers, a laptop running Windows and a desktop running Ubuntu 16.04. She updates the Windows machine with each release but doesn't know how to build GnuCash on her Ubuntu machine so leaves it at whatever they shipped on 16.04; 2.6.12 IIRC. As long as she doesn't use any of the features supported only in 3.x she can continue to use both systems to keep her books and regardless of which machine she uses to create a book. The "silently" part could be better. It would indeed be a nice enhancement to pop a dialog box when GnuCash is about to set a feature flag explaining that it's use will preclude using the file on a GnuCash version older than X and giving the user a chance to bail out. It would be nicer still to persist that decision so that it need ask only once, and if the user assents it could set the feature flag even if the feature isn't used, as is the case here. Of course there would then be the need to reset negative responses so that when our hypothetical user upgrades her desktop to Ubuntu 20.04 next spring she can turn on all of those 3.x only features she'd previously declined. It occurs to me that another improvement to the existing flat Bayesian conversion would be to check the state of the GNC_PREF_USE_BAYES and skip the conversion if it's not true. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs
Am 08.11.19 um 23:08 schrieb John Ralls: On Nov 8, 2019, at 1:58 PM, Christian Gruber wrote: Am 08.11.19 um 04:39 schrieb John Ralls: Christian, It's not that it's not prepared for Bayesian matching, it's that older versions of GnuCash stored the Bayesian match tokens hierarchically. Aaron Laws (lmat) changed it to a flatter structure with somewhat better memory locality for faster access. imap_convert_bayes_to_flat should run once to convert the data and set the feature, after which check_import_map_data will see the flag and return. A file created with 3.x and Baysian maps would already have the feature set. With that background, to your questions: Why does it take so long? Because it traverses the entire tree of accounts, every time. The test book has 1127 accounts. Add to that that there are some things inside the loop that shouldn't be and that convert_imap_account_bayes_to_flat doesn't use some obvious short circuits and you get taking a long time. Why does it run twice? Because there aren't any accounts with import-map-bayes slots, so it does no conversions so it doesn't set the feature. Why isn't the feature set in any case after conversion is done, whether there was any slot to convert or not? I would expect that, if the hierarchical structure should not be used anymore. I wouldn't only set the feature, if there has been any error during conversion. But it's not an error, if convert_imap_account_bayes_to_flat() returns false. The feature isn't set because that would prevent using the file with an older version of GnuCash for no good reason: There aren't any import-map-bayes tags in the new format so there's nothing for the older version to mess up. Regards, John Ralls Sounds meaningful, but it's not obvious. The conversion process is not much transparent to the user. The user is not informed about conversion and additionally the conditions, when the GnuCash file is converted and when not, are not quite intuitive, I guess. And the second question for me is, if this is really a relevant use case to keep the GnuCash file unconverted if possible, when it is used with a newer GnuCash version. Maybe it's relevant, if several people, who use different GnuCash versions work on the same GnuCash file. But what happens, when one with a newer GnuCash version uses the bayesian import matcher for the first time, which silently converts the GnuCash file? Will it still be usable with an older GnuCash version? But actually this is another discussion, which is not important for the bug ticket. What's more important, I created a new fresh GnuCash file with the SKR04 accounts template using my GnuCash version 3.7 and if I checked this correctly, even with this GnuCash version the feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you confirm this? If yes, then the bug report is relevant not only for migration from older to current GnuCash versions but even for current GnuCash versions in general. Regards, Christian ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Translators: Re: To colon or not to colon
Hi Bob, Thanks for working on this. I would propose to do the string changes in maint. That's what is currently being referred to for translation work, so getting string changes in there will allow tranlsators to start working on them already now. The other gui redesign work (converting from boxes to grids, alignment changes,...) I'd keep for master. That would make for one big consistency change. Regards, Geert Op maandag 4 november 2019 18:45:04 CET schreef Robert Fewell: > Frank, > So it looks like get rid of trailing colon/space and right align text for > normal dialogues. > > For RTL, GTK already swaps the label and control along with the alignment > which I have observed on the 'View Invoice page'. > > The preference dialogue may need to be treated differently, not sure yet > and there is also no indents for RTL, will need to be looked at separately > along with some other RTL stuff I have been thinking about. > > For removing the trailing colon, 35 out of 66 glade files and maybe some > source files would need to be changed which is probably too many changes > for maint in one hit all be it in a few commits. > Then on top of that, changing from GTK boxes to grids and the alignment > changes, I am just not sure what to do on maint and what to do on master? > > Regards, > Bob > > On Mon, 4 Nov 2019 at 16:05, Frank H. Ellenberger < > > frank.h.ellenber...@gmail.com> wrote: > > Hi Bob, > > > > Am 04.11.19 um 12:01 schrieb Robert Fewell: > > > So, the only other question as part of this is whether the labels should > > > > be > > > > > left aligned or right and again there is a mix of the two in various > > > dialogs. > > > The page you linked to says they should be right aligned so there is the > > > minimum space between the label and control but I quite like it being > > > > left. > > > > > As part of the changes to use grids, I will probably need to set this so > > > > I > > > > > just want it to be consistent and only change it once. > > > > > > Any thoughts, left or right aligned ? > > > > The benefit of the right alignment: It is easily comprehensible which > > label belongs to which control element. This would be harder, if you > > have one long and many short labels and align them left. > > > > Their exception is the existence of options, which depend on a > > previous option, i.e. in Edit->Preferences>Scheduled Transactions > > "Notify before transactions are created" is only vailable, if > > "Auto-create new transactions" is enabled. I am wondering, if there is > > a smarter solution. > > > > Another question: How will the elements behave for RTL writing? > > > > > Regards, > > > Bob > > > > > >> https://developer.gnome.org/hig/stable/visual-layout.html.en > > > > Regards > > Frank > > ___ > gnucash-devel mailing list > gnucash-devel@gnucash.org > https://lists.gnucash.org/mailman/listinfo/gnucash-devel ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
