[GNC-dev] New bayesian account matching algorithm

[Christian Gruber]

Hi devs,


I'd like to propose a new algorithm for matching imported transactions 
to accounts. I've been dealing with the bayesian account matcher for 
several months now during my spare free time. Starting point of my 
efforts was the observation, that sometimes matching fails, where I 
wouldn't expect that. For instance some recurrent transactions, which 
could be matched successfully for a long time, suddenly started to fail 
matching. Additionally several users in the german mailing list reported 
similar problems and asked for help on how to "improve" the matching 
results. The mostly recommended workaround was to "clean up" the import 
mapping table using the corresponding import map editor.



Therefore I started trying to understand the current implementation. See 
my mails from that thread:



   * 
http://gnucash.1415818.n4.nabble.com/GNC-dev-Understanding-the-bayesian-import-matching-algorithm-td4719812.html



I could fix some bugs already. See this issue on Bugzilla:


   * https://bugs.gnucash.org/show_bug.cgi?id=797587


Another issue is still open:


   * https://bugs.gnucash.org/show_bug.cgi?id=797744


Unfortunatelly I was not able to fully understand the implementation, 
see my mails from july in the mail thread mentioned above. Moreover it 
seemed, that nobody else could explain the current implementation anymore.



Therefore I stopped trying to further understand the algorithm and 
prepared a new and improved approach. I did a little research and tried 
different approaches. The final approach, I want to propose now, is a 
probalistic (bayesian) approach as well and should solve the following 
two problems of the current algorithm:



   * it should be possible to understand and explain the matching results

   * the account matching probabilities should be "real" probabilities 
and not only result to 1 or 0 (see my last mail from october in the mail 
thread mentioned above)



This new algorithm is even simpler, but still as effective as the 
current algorithm. And the current algorithm can be easily replaced 
using the existing import mapping table (token frequency table). It is 
provided as PR #839 on Github:



   * https://github.com/Gnucash/gnucash/pull/839


I tested the algorithm with the transactions from my personal account. I 
wrote a simple test application, which simulates the import of all my 
transactions one by one in chronological order. For each transaction 
this application compares the matched account calculated by the 
algorithm with the account I manually assigned the transaction to. I 
used this application to compare the new proposed algorithm with the 
current algorithm.



At first all transactions, which did match correctly before still match 
correctly using the new algorithm. Additionally several transactions, 
which did not match correcly before, match correctly now. The ratio of 
correct matches could be improved from approx. 60 % to 70 %.



But the most important improvement is, that false matching results can 
be explained now.



And this is a first draft of the new approach only. There is still space 
for further improvement. The algorithm has a new feature. You can select 
or exclude individual tokens from being considered for calculation of 
account matching probability. I already proposed this idea on Bugzilla



   * https://bugs.gnucash.org/show_bug.cgi?id=797779


From my point of view this is the key for further improvement. The 
current as well as the new proposed algorithm can solve all the unique 
cases, i.e. transactions with have at least one token, which is unique 
to exactly one account (see also my last mail from october in the mail 
thread mentioned above). The more complicated cases are transactions, 
which can not be matched uniquely to one account, since similar 
transactions have been assigned to different accounts in the past by the 
user.



I played around with these transactions and could already achieve 
promising results by reducing the 'token_account_probability_threshold' 
used for automatic token selection. But this raises new problems, since 
there are some tokens, which can distract the matching result from the 
correct account. Furthermore it wasn't that easy anymore to find a 
meaningful threshold to detect "new" transactions, which haven't occured 
before and shouldn't be tried to match.



Therefore I decided for a very conservative threshold in the first draft 
and left this open for future improvements. I prefered a robust solution 
at the moment.



Ok, now you can have a look and try out the new approach on your own. 
And I'm curious about your feedback.



Christian



___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Understanding the bayesian import matching algorithm

[Christian Gruber]

Hi Derek,

I further investigated the implemented algorithm during the last months 
using the transactions from my personal account. I wrote a simple 
application which simulates the import of all my transactions one by one 
in chronological order. For each transaction this application compares 
the matched account calculated by the implemented bayes algorithm with 
the account I manually assigned the transaction to. In case of a 
mismatch the application provides several debug information like the 
calculated probabilities for each account and the frequencies of each 
token for instance to see what's going wrong.


What I found out was very surprising to me. Here are my results:

* the calculated probabilities for each account P(A_k | T_1, ..., T_N) are
  always whether exactly 1 or almost 0, i.e. the matched account is actually a
  binary decision
* sometimes there is more than one account with probability 1 and the finally
  selected matching account (function highest_probability()) can be the
  correct one or not, it's random
* the condition for a probability of 1 is, that there exists a token, which is
  unique for a single account, i.e. all transactions containing this token have
  always been assigned to the same account
* if there are several tokens, which are unique for different accounts, all
  these accounts have probability 1
* if there is no token, which is unique to one account, the probability is
  almost 0 for all accounts

That means, as soon as there is no unique token, which can be used for 
classification, the implemented algorithm fails. Moreover if the 
relevant tokens are not unique, but there are unique tokens, which are 
actually irrelevant, these irrelevant tokens are used for 
classification. This was the most common observed case for wrong account 
selection, since the irrelevant tokens where mostly not assigned to the 
correct account. In many cases these irrelevant tokens had frequencies 
of 1, i.e. there was only one transaction before with randomly the same 
token.


In addition this would mean, that the bayes classifier is actually 
needless, since the same classification results can be achieved with a 
much simpler rule-based decision algorithm and without any probalistic 
approach.


These observations encourage my suspicion, that something is wrong with 
the implemented algorithm.


Christian


Am 06.07.20 um 01:05 schrieb Christian Gruber:


Am 02.07.20 um 21:28 schrieb Derek Atkins:

Hi,

On Thu, July 2, 2020 3:10 pm, Christian Gruber wrote:

Hi,

while further studying the bayesian import matching algorithm I'm 
now at

the point, where I wanted to understand, how the bayes formula is
applied to the problem of matching transactions to accounts using
tokens. But I need further information, since it doesn't come clear to
me what is really calculated there.

The implementation can be found in the following functions in 
Account.cpp:


   * get_first_pass_probabilities()
   * build_probabilities()
   * highest_probability()

Actually, the latter could be omitted as it only selects the account
with the highest matching probability.

Studying the code and the rare comments on the implementation it seems
to be a variant of the naive bayes classifier
<https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Probabilistic_model> 


with the tokens used as (independent) "features" and the accounts used
as "classes". But comparing this algorithm to the code leaves several
questions open.

Does anybody know a more precise algorithm description, on which the
implementation in GnuCash is based on?

I'm not sure how detailed you need right now; I helped with some of the
initial implementations but I'm sure it's all been rewritten by now.  
The


No, I don't think so. The basic algorithm seems to be unchanged, since 
it was implemented the first time by you with commit b2ccbf6 in 2003. 
Yes, the code was moved between files and files have been renamed. 
Also the code was restructured in commit b3667c76f "Implement flat 
bayes kvp" and there were lots of smaller code changes over the time. 
But the algorithm still seems to be the same.


idea is that the description/memo strings are tokenized and used as 
inputs

into the probabilities that the transaction would go into the target
account.  If you have a high-enough probability it will auto-select that
account for that transaction.

When you assign an account (during import), it adds those tokens to the
account's list of tokens for future guessing.

The main principle is clear to me.


Did you have a specific question about the process?  For the complete
algorithm you can look at the code.  It's really not all that 
complicated

(or at least it wasn't when first implemented).


Ok, I try to be more precise.

I. The implemented algorithm I read from the code is the following:

    product_k
P(A_k | T_1, ..., T_N) = --
 produc

Re: [GNC-dev] Understanding the bayesian import matching algorithm

[Christian Gruber]

Am 02.07.20 um 21:28 schrieb Derek Atkins:

Hi,

On Thu, July 2, 2020 3:10 pm, Christian Gruber wrote:

Hi,

while further studying the bayesian import matching algorithm I'm now at
the point, where I wanted to understand, how the bayes formula is
applied to the problem of matching transactions to accounts using
tokens. But I need further information, since it doesn't come clear to
me what is really calculated there.

The implementation can be found in the following functions in Account.cpp:

   * get_first_pass_probabilities()
   * build_probabilities()
   * highest_probability()

Actually, the latter could be omitted as it only selects the account
with the highest matching probability.

Studying the code and the rare comments on the implementation it seems
to be a variant of the naive bayes classifier
<https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Probabilistic_model>
with the tokens used as (independent) "features" and the accounts used
as "classes". But comparing this algorithm to the code leaves several
questions open.

Does anybody know a more precise algorithm description, on which the
implementation in GnuCash is based on?

I'm not sure how detailed you need right now; I helped with some of the
initial implementations but I'm sure it's all been rewritten by now.  The


No, I don't think so. The basic algorithm seems to be unchanged, since 
it was implemented the first time by you with commit b2ccbf6 in 2003. 
Yes, the code was moved between files and files have been renamed. Also 
the code was restructured in commit b3667c76f "Implement flat bayes kvp" 
and there were lots of smaller code changes over the time. But the 
algorithm still seems to be the same.



idea is that the description/memo strings are tokenized and used as inputs
into the probabilities that the transaction would go into the target
account.  If you have a high-enough probability it will auto-select that
account for that transaction.

When you assign an account (during import), it adds those tokens to the
account's list of tokens for future guessing.

The main principle is clear to me.


Did you have a specific question about the process?  For the complete
algorithm you can look at the code.  It's really not all that complicated
(or at least it wasn't when first implemented).


Ok, I try to be more precise.

I. The implemented algorithm I read from the code is the following:

product_k
P(A_k | T_1, ..., T_N) = --
 product_k + product_diff_k
with

product_k  =  P(A_k | T_1)  * ... *  P(A_k | T_N)
product_diff_k = [1 - P(A_k | T_1)] * ... * [1 - P(A_k | T_N)]

and with

P(A_k | T_n)   ... probability of account A_k under the
   condition of token T_n
P(A_k | T_1, ..., T_N) ... probability of account A_k under the
   condition of a list of tokens T_1, ..., T_N
N  ... number of tokens


II. By applying the formulas of the Bayes classifier (see e.g. Wikipedia) I get 
the following:

 product_k
P(A_k | T_1, ..., T_N) = ---
 product_1 + ... + product_K
with

product_k = P(T_1 | A_k) * ... * P(T_N | A_k)

and with

P(T_n | A_k) ... probability of token T_n under the condition
 of account A_k
K... number of accounts
N... number of tokens

assuming equal total probability P(A_k) for all accounts.


III. Simplifying these formulas by only distinguishing between account A_k and 
all other accounts leads to the following formulas:

product_k
P(A_k | T_1, ..., T_N) = -
 product_k + product_not_k
with

product_k = P(T_1 |  A_k) * ... * P(T_N |  A_k)
product_not_k = P(T_1 | ¬A_k) * ... * P(T_N | ¬A_k)

and with

P(T_n |  A_k) ... probability of token T_n under the condition
  of account A_k
P(T_n | ¬A_k) ... probability of Token T_n under the condition
  of any other account than A_k
K ... number of accounts
N ... number of tokens

assuming equal total probability for P(A_k) and P(¬A_k).


As you can see neither the formulas in derivation II nor the formulas in 
derivation III fit to formulas in derivation I.

One obvious difference is, that the implemented code uses probabilities P(A_k | 
T_n), whereas the formulas derived from bayes classification use P(T_n | A_k).




Regards,
Christian

-derek

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] Understanding the bayesian import matching algorithm

[Christian Gruber]

Hi,

while further studying the bayesian import matching algorithm I'm now at 
the point, where I wanted to understand, how the bayes formula is 
applied to the problem of matching transactions to accounts using 
tokens. But I need further information, since it doesn't come clear to 
me what is really calculated there.


The implementation can be found in the following functions in Account.cpp:

 * get_first_pass_probabilities()
 * build_probabilities()
 * highest_probability()

Actually, the latter could be omitted as it only selects the account 
with the highest matching probability.


Studying the code and the rare comments on the implementation it seems 
to be a variant of the naive bayes classifier 
 
with the tokens used as (independent) "features" and the accounts used 
as "classes". But comparing this algorithm to the code leaves several 
questions open.


Does anybody know a more precise algorithm description, on which the 
implementation in GnuCash is based on?


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Is the import match map still required?

[Christian Gruber]

I created two enhancement issues on Bugzilla regarding this topic:

 * https://bugs.gnucash.org/show_bug.cgi?id=797778
 * https://bugs.gnucash.org/show_bug.cgi?id=797779


Am 30.05.20 um 14:37 schrieb Christian Gruber:

David,

thanks for your detailed explanations. Implementing a procedure, which 
could be run as needed and which updates the frequency table according 
to the current transactions for an account, seems to be a meaningful 
first step. This could be used to measure performance next. Then it 
could be decided, if this procedure can also run on the fly.


I also thought more about the user's part of interaction with the 
frequency table. The current situation seems to be a bit like 
"hacking" the frequency table to achieve better matching results. You 
can remove some entries, which seem to be wrong or seem to corrupt the 
matching results or whatever. If the user would not change the 
frequency table directly, but could instead set some personal 
preferences on how the data is used, this would solve the problem, 
that these preferences are not influenced by running the procedure 
updating the frequency table. And by regular updates of the frequency 
table, wrong or outdated entries are removed reliably and the data is 
up-to-date. The user could for example exclude some tokens from the 
bayesian algorithm, which are not relevant for him.


Christian


Am 25.05.20 um 01:13 schrieb David Cousens:

Christian,

I haven't experimented to know whether constructing the frequency 
table on
the fly creates a performance bottleneck or not but am guessing the 
original

developer thought it might. It would require a detailed look at the code
involved but my suspicion would be that the performance penalty is 
likely to

be significant.

My comment about bloat is that at present data is only maintained for
accounts you specifically import data into and if that data is 
stored. If it
isn't then bloat doesn't apply obviously. Any sort of generalized 
procedure
could allow selection of accounts for which Bayesian matching is 
required,
i.e. those for which importing is used to input data. My initial 
thought was
that you would run it for all accounts but it is really only 
necessary for
the specific subset of accounts into which you import data. It would 
then

require the ability to run the procedure on an account if it occurred in
import data but didn't have existing account matching data. If it is 
on the
fly then no problem it can run whenever a new account being imported 
into
appears in the imported data. The most common use case is probably 
importing
data to one specific account but GnuCash is also able to specify the 
account
being imported into in the import data itself.  I haven't looked at 
how the

frequency table is currently stored in memory but I am guessing it is
constructed in memory when the data file is read in.

The up-to-date aspect is one advantage and if the current procedure  is
changed to improve performance then that is not hampered by the 
presence of
historical data which would be updated automatically when the 
procedure is

run. If the table is stored as it is at present and a procedure was
available to trawl the current transactions for an account then it 
can be
kept up to date by running that procedure periodically. the use of 
data from
manually entered transactions would then be incorporated whether on 
the fly

or just run as required.

Having a standalone procedure to trawl an existing file to update the 
stored
data for an account  would allow exploration of whether this is 
likely to be
a significant performance hit if it was run on the fly so that could 
perhaps
be a first step.  The core part of the code to store the data has to 
exist
in the matcher code already and it will be a case of wrapping this in 
a loop

through the transactions existing in an account and setting up the gui
interface to select accounts to run on.

The problem with pruning the data is that GnuCash has no way of knowing
apriori which tokens are most relevant. I would think that date 
information

is not really relevant and amount/value information does little in most
cases to identify a transfer account.

The main difficulty I have  with transfer account assignment is that 
some

regular transactions use a unique code in the description each time they
occur with no separate unique identifier of the transaction source. 
My wife

and I both have separte gym membership subscriptions and the transaction
descriptions neither identify the gym or for which of us  the 
transaction
applies. Options are to persuade the source to include specific data 
or only
use a single account to record both but I like to track both our 
individual

and joint expenses

Some regular transactions also get matched to previous payments in the
transaction matching within the date range window where the amounts and
descriptions are usually identical. The current 42 day window 
captures both
fortnightly and month

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 01.06.20 um 22:47 schrieb John Ralls:



On Jun 1, 2020, at 1:25 PM, Christian Gruber  wrote:


Am 31.05.20 um 02:47 schrieb John Ralls:

On May 30, 2020, at 4:45 AM, Christian Gruber  
wrote:


Am 29.05.20 um 01:49 schrieb Frank H. Ellenberger:

Christian,

did you clean ~/.cache/guile ? That is the place, where the precompiled
.go bytecode from the .scm files is stored.

Thanks for this advise, I haven't tried this yet. Unfortunately it didn't help.

Frank

Am 28.05.20 um 17:42 schrieb Christian Gruber:

Am 28.05.20 um 12:24 schrieb Frank H. Ellenberger:

Hi,

Am 28.05.20 um 08:13 schrieb Christian Gruber:

I have to search for left *.go files, right? There are a lot in
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete
directory without hesitation?

The easier way is to uninstall guile-modules-2_2 to get rid of them. ;-)
They are part of that package.

Ok, I checked that with inverse file search ("apt-file search"). On my
Ubuntu system these files belong to package guile-2.2-libs. And the
*.scm files in /usr/share/guile/2.2 belong to that package as well. But
after uninstalling guile-2.2-libs CMake complains, when building GnuCash:

CMake Error at CMakeLists.txt:311 (message):
   Neither guile 2.2 nor guile 2.0 were found GnuCash can't run without
one of
   them.  Ensure that one is installed and can be found with pkg-config.

Therefore I installed guile-2.2-libs again. And the build error is the
same again as at the beginning:

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)

Is this maybe a problem of a missing CMake dependency?

Another idea, I updated my Ubuntu system from 19.10 Eoan Ermine to 20.04
Focal Fossa a few days ago. Could this be a reason?

Can anybody please try to reproduce the build process on Linux from the current 
master branch in a completely clean build directory and with a clean guile 
cache (~/.cache/guile)?

I did the following:

cd ~/gnucash/build
rm rf *
cmake -DCMAKE_BUILD_TYPE=Debug ~/gnucash/src
make all

Can anybody tell me more precisely, what this error message actually means? Does "no code for 
module (gnucash core-utils)" mean, that something is not built yet? If yes, what is not built? 
Target "gnc-core-utils" is already built.

Christian,

Sorry, I gave you bad advice Wednesday and that damaged your guile installation.

"no code for module (gnucash core-utils)" means that guile wants to load 
/lib/guile/2.2/site-ccache/gnucash/core-utils.go

Is this directory system dependent? On my Ubuntu the *.go files are in 
/lib/x86_64-linux-gnu/guile/2.2/site-ccache.

Well, it's more distro-dependent, with the Debian-based distros using the 
`uname` subdirectory for application-specific libraries. Some others use lib 
for 32-bit and lib64 for 64-bit libraries.


and it's either not there, is empty, or can't find lib/libgnc-core-utils.so, 
the library on which it depends...

That was finally the missing hint. This file was missing.

And the reason for that really seems to be a missing CMake dependency. After building target 
"scm-core-utils", the file /x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/core-utils.go 
was built. And now I could go on with "make all" successfully until the end.

Very good. I've just pushed that change to master.


*unless* it finds an old core-utils.go that wants to link to an installed 
libgnc-core-utils.so or if it's from maint libgncmod-core-utils.go. It searches 
system directories (i.e. /usr/lib*) first which is why you can't develop 
gnucash on a Linux system with an installed gnucash.

I've just finished clean builds of master on Ubuntu 18.04 and freshly updated 
Arch Linux and Debian Unstable, the last of which I'd think would be pretty 
close to your Ubuntu 20.04. Note that we run continuous integration tests after 
every push, you can see the results at 
https://travis-ci.org/github/Gnucash/gnucash/builds/. Those run on dockers with 
Ubuntu 18.04 and Arch Linux that are spun up and configured from scratch for 
each test run.

I know and that puzzles me. Why do the CI builds run successfully, if this is 
really a missing CMake dependency? Maybe the problem doesn't occur, when doing 
parallel builds on several cores, since there are other targets, which depend 
on scm-core-utils. And if these targets are built in parallel and finish before 
target scm-engine-2 is built, the missing dependency doesn't lead to a build 
error.

Absent dependencies the build order of targets is somewhat random. It was 
working for you until you upgraded your OS and got different versions of cmake 
and make (have you tried ninja? It's about 4x faster than make) and the build 
order changed exposing the missing dependency.
I have tried ninja, but I can see no benefit for building GnuCash. The 
build time of "ninja -j8 all" is almost the same as "make -j8 all" from 
a clean build directory (2m28s vs. 2m29s). I 

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 31.05.20 um 02:47 schrieb John Ralls:



On May 30, 2020, at 4:45 AM, Christian Gruber  
wrote:


Am 29.05.20 um 01:49 schrieb Frank H. Ellenberger:

Christian,

did you clean ~/.cache/guile ? That is the place, where the precompiled
.go bytecode from the .scm files is stored.

Thanks for this advise, I haven't tried this yet. Unfortunately it didn't help.

Frank

Am 28.05.20 um 17:42 schrieb Christian Gruber:

Am 28.05.20 um 12:24 schrieb Frank H. Ellenberger:

Hi,

Am 28.05.20 um 08:13 schrieb Christian Gruber:

I have to search for left *.go files, right? There are a lot in
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete
directory without hesitation?

The easier way is to uninstall guile-modules-2_2 to get rid of them. ;-)
They are part of that package.

Ok, I checked that with inverse file search ("apt-file search"). On my
Ubuntu system these files belong to package guile-2.2-libs. And the
*.scm files in /usr/share/guile/2.2 belong to that package as well. But
after uninstalling guile-2.2-libs CMake complains, when building GnuCash:

CMake Error at CMakeLists.txt:311 (message):
   Neither guile 2.2 nor guile 2.0 were found GnuCash can't run without
one of
   them.  Ensure that one is installed and can be found with pkg-config.

Therefore I installed guile-2.2-libs again. And the build error is the
same again as at the beginning:

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)

Is this maybe a problem of a missing CMake dependency?

Another idea, I updated my Ubuntu system from 19.10 Eoan Ermine to 20.04
Focal Fossa a few days ago. Could this be a reason?

Can anybody please try to reproduce the build process on Linux from the current 
master branch in a completely clean build directory and with a clean guile 
cache (~/.cache/guile)?

I did the following:

cd ~/gnucash/build
rm rf *
cmake -DCMAKE_BUILD_TYPE=Debug ~/gnucash/src
make all

Can anybody tell me more precisely, what this error message actually means? Does "no code for 
module (gnucash core-utils)" mean, that something is not built yet? If yes, what is not built? 
Target "gnc-core-utils" is already built.

Christian,

Sorry, I gave you bad advice Wednesday and that damaged your guile installation.

"no code for module (gnucash core-utils)" means that guile wants to load 
/lib/guile/2.2/site-ccache/gnucash/core-utils.go
Is this directory system dependent? On my Ubuntu the *.go files are in 
/lib/x86_64-linux-gnu/guile/2.2/site-ccache.

and it's either not there, is empty, or can't find lib/libgnc-core-utils.so, 
the library on which it depends...


That was finally the missing hint. This file was missing.

And the reason for that really seems to be a missing CMake dependency. 
After building target "scm-core-utils", the file 
/x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/core-utils.go was built. 
And now I could go on with "make all" successfully until the end.



*unless* it finds an old core-utils.go that wants to link to an installed 
libgnc-core-utils.so or if it's from maint libgncmod-core-utils.go. It searches 
system directories (i.e. /usr/lib*) first which is why you can't develop 
gnucash on a Linux system with an installed gnucash.

I've just finished clean builds of master on Ubuntu 18.04 and freshly updated 
Arch Linux and Debian Unstable, the last of which I'd think would be pretty 
close to your Ubuntu 20.04. Note that we run continuous integration tests after 
every push, you can see the results at 
https://travis-ci.org/github/Gnucash/gnucash/builds/. Those run on dockers with 
Ubuntu 18.04 and Arch Linux that are spun up and configured from scratch for 
each test run.


I know and that puzzles me. Why do the CI builds run successfully, if 
this is really a missing CMake dependency? Maybe the problem doesn't 
occur, when doing parallel builds on several cores, since there are 
other targets, which depend on scm-core-utils. And if these targets are 
built in parallel and finish before target scm-engine-2 is built, the 
missing dependency doesn't lead to a build error.




Regards,
John Ralls


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Is the import match map still required?

[Christian Gruber]

David,

thanks for your detailed explanations. Implementing a procedure, which 
could be run as needed and which updates the frequency table according 
to the current transactions for an account, seems to be a meaningful 
first step. This could be used to measure performance next. Then it 
could be decided, if this procedure can also run on the fly.


I also thought more about the user's part of interaction with the 
frequency table. The current situation seems to be a bit like "hacking" 
the frequency table to achieve better matching results. You can remove 
some entries, which seem to be wrong or seem to corrupt the matching 
results or whatever. If the user would not change the frequency table 
directly, but could instead set some personal preferences on how the 
data is used, this would solve the problem, that these preferences are 
not influenced by running the procedure updating the frequency table. 
And by regular updates of the frequency table, wrong or outdated entries 
are removed reliably and the data is up-to-date. The user could for 
example exclude some tokens from the bayesian algorithm, which are not 
relevant for him.


Christian


Am 25.05.20 um 01:13 schrieb David Cousens:

Christian,

I haven't experimented to know whether constructing the frequency table on
the fly creates a performance bottleneck or not but am guessing the original
developer thought it might. It would require a detailed look at the code
involved but my suspicion would be that the performance penalty is likely to
be significant.

My comment about bloat is that at present data is only maintained for
accounts you specifically import data into and if that data is stored. If it
isn't then bloat doesn't apply obviously. Any sort of generalized procedure
could allow selection of accounts for which Bayesian matching is required,
i.e. those for which importing is used to input data. My initial thought was
that you would run it for all accounts but it is really only necessary for
the specific subset of accounts into which you import data. It would then
require the ability to run the procedure on an account if it occurred in
import data but didn't have existing account matching data. If it is on the
fly then no problem it can run whenever a new account being imported into
appears in the imported data. The most common use case is probably importing
data to one specific account but GnuCash is also able to specify the account
being imported into in the import data itself.  I haven't looked at how the
frequency table is currently stored in memory but I am guessing it is
constructed in memory when the data file is read in.

The up-to-date aspect is one advantage and if the current procedure  is
changed to improve performance then that is not hampered by the presence of
historical data which would be updated automatically when the procedure is
run. If the table is stored as it is at present and a procedure was
available to trawl the current transactions for an account then it can be
kept up to date by running that procedure periodically. the use of data from
manually entered transactions would then be incorporated whether on the fly
or just run as required.

Having a standalone procedure to trawl an existing file to update the stored
data for an account  would allow exploration of whether this is likely to be
a significant performance hit if it was run on the fly so that could perhaps
be a first step.  The core part of the code to store the data has to exist
in the matcher code already and it will be a case of wrapping this in a loop
through the transactions existing in an account and setting up the gui
interface to select accounts to run on.

The problem with pruning the data is that GnuCash has no way of knowing
apriori which tokens are most relevant. I would think that date information
is not really relevant and amount/value information does little in most
cases to identify a transfer account.

The main difficulty I have  with transfer account assignment is that some
regular transactions use a unique code in the description each time they
occur with no separate unique identifier of the transaction source. My wife
and I both have separte gym membership subscriptions and the transaction
descriptions neither identify the gym or for which of us  the transaction
applies. Options are to persuade the source to include specific data or only
use a single account to record both but I like to track both our individual
and joint expenses

Some regular transactions also get matched to previous payments in the
transaction matching within the date range window where the amounts and
descriptions are usually identical. The current 42 day window captures both
fortnightly and monthly regular income transactions for example.  This only
affects a few transactions each month and I don't have huge numbers of
transactions to process now that I have retired but that may not be the case
for other users. Maybe making the date range window adjustable rather 

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 27.05.20 um 02:49 schrieb David Cousens:

Christian

 From memory I think that error occurs when you have a problem with the cmake
relative addressing from the build directory to the gnucash source directory
although that should be the same no matter whether you have the master or
maint checked out with git.
Thanks for the tip, David. But the build error really occurs only when 
building from master branch, not from maint branch.


David Cousens



-
David Cousens
--
Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-Dev-f1435356.html
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 29.05.20 um 23:03 schrieb Geert Janssens:


Op donderdag 28 mei 2020 08:13:14 CEST schreef Christian Gruber:

> Am 28.05.20 um 06:31 schrieb John Ralls:

> >> On May 27, 2020, at 2:15 PM, Christian Gruber

> >>  wrote:>>

> >> Am 27.05.20 um 22:16 schrieb John Ralls:

> >>>> On May 27, 2020, at 1:09 PM, Christian Gruber

> >>>>  wrote:>>>>

> >>>> Am 27.05.20 um 00:28 schrieb John Ralls:

> >>>>>> On May 26, 2020, at 1:34 PM, Christian Gruber

> >>>>>>  wrote:

> >>>>>>

> >>>>>> Hi,

> >>>>>>

> >>>>>> I tried to build GnuCash from master (currently fde6be6e0) 
for the


> >>>>>> first time. Before I always built from maint.

> >>>>>>

> >>>>>> I get a strange build error:

> >>>>>>

> >>>>>> Scanning dependencies of target scm-engine-2

> >>>>>> [ 30%] Generating

> >>>>>> 
../../lib/x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/utilities.g


> >>>>>> o Backtrace:

> >>>>>>

> >>>>>> In scripts/compile.scm:

> >>>>>> 264:26 19 (_ _)

> >>>>>>

> >>>>>> In system/base/target.scm:

> >>>>>> 57:6 18 (with-target _ _)

> >>>>>>

> >>>>>> In system/base/compile.scm:

> >>>>>> 152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _

> >>>>>> …)

> >>>>>>

> >>>>>> 43:4 16 (call-once _)

> >>>>>>

> >>>>>> In ice-9/boot-9.scm:

> >>>>>> 841:4 15 (with-throw-handler _ _ _)

> >>>>>>

> >>>>>> In system/base/compile.scm:

> >>>>>> 59:11 14 (_)

> >>>>>>

> >>>>>> 155:11 13 (_ #)

> >>>>>> 235:18 12 (read-and-compile # #:from _ #

> >>>>>> …)

> >>>>>> 183:32 11 (compile-fold (#)

> >>>>>> …)

> >>>>>>

> >>>>>> In ice-9/boot-9.scm:

> >>>>>> 2312:4 10 (save-module-excursion # >>>>>> lang…>)

> >>>>>>

> >>>>>> In language/scheme/compile-tree-il.scm:

> >>>>>> 31:15 9 (_)

> >>>>>>

> >>>>>> In ice-9/psyntax.scm:

> >>>>>> 1262:36 8 (expand-top-sequence ((use-modules (gnucash #))) _ _ #f

> >>>>>> …)

> >>>>>> 1209:24 7 (parse _ (("placeholder" placeholder)) ((top) #(# # …))

> >>>>>> …)

> >>>>>>

> >>>>>> 285:10 6 (parse _ (("placeholder" placeholder)) (()) _ c (# #)

> >>>>>> #)

> >>>>>>

> >>>>>> In ice-9/boot-9.scm:

> >>>>>> 3377:20 5 (process-use-modules _)

> >>>>>>

> >>>>>> 222:17 4 (map1 (((gnucash core-utils

> >>>>>>

> >>>>>> 3378:31 3 (_ ((gnucash core-utils)))

> >>>>>>

> >>>>>> 2803:6 2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _

> >>>>>> …)

> >>>>>>

> >>>>>> In unknown file:

> >>>>>> 1 (scm-error misc-error #f "~A ~S" ("no code for modu…"

> >>>>>> …) …)

> >>>>>>

> >>>>>> In ice-9/boot-9.scm:

> >>>>>> 752:25 0 (dispatch-exception _ _ _)

> >>>>>>

> >>>>>> ice-9/boot-9.scm:752:25: In procedure dispatch-exception:

> >>>>>> no code for module (gnucash core-utils)

> >>>>>>

> >>>>>>

> >>>>>>

> >>>>>> I get the same build error even when using earlier commits from

> >>>>>> master, tag 3.902 for instance.

> >>>>>>

> >>>>>> Do I do something wrong?

> >>>>>

> >>>>> Christian,

> >>>>>

> >>>>> Did you remember to uninstall GnuCash and build in a 
completely clean


> >>>>> (as in rm -rf *) build directory?>>>>

> >>>> Yes, I do remember.

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 29.05.20 um 01:49 schrieb Frank H. Ellenberger:

Christian,

did you clean ~/.cache/guile ? That is the place, where the precompiled
.go bytecode from the .scm files is stored.
Thanks for this advise, I haven't tried this yet. Unfortunately it 
didn't help.


Frank

Am 28.05.20 um 17:42 schrieb Christian Gruber:

Am 28.05.20 um 12:24 schrieb Frank H. Ellenberger:

Hi,

Am 28.05.20 um 08:13 schrieb Christian Gruber:

I have to search for left *.go files, right? There are a lot in
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete
directory without hesitation?

The easier way is to uninstall guile-modules-2_2 to get rid of them. ;-)
They are part of that package.

Ok, I checked that with inverse file search ("apt-file search"). On my
Ubuntu system these files belong to package guile-2.2-libs. And the
*.scm files in /usr/share/guile/2.2 belong to that package as well. But
after uninstalling guile-2.2-libs CMake complains, when building GnuCash:

CMake Error at CMakeLists.txt:311 (message):
   Neither guile 2.2 nor guile 2.0 were found GnuCash can't run without
one of
   them.  Ensure that one is installed and can be found with pkg-config.

Therefore I installed guile-2.2-libs again. And the build error is the
same again as at the beginning:

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)

Is this maybe a problem of a missing CMake dependency?

Another idea, I updated my Ubuntu system from 19.10 Eoan Ermine to 20.04
Focal Fossa a few days ago. Could this be a reason?


Can anybody please try to reproduce the build process on Linux from the 
current master branch in a completely clean build directory and with a 
clean guile cache (~/.cache/guile)?


I did the following:

cd ~/gnucash/build
rm rf *
cmake -DCMAKE_BUILD_TYPE=Debug ~/gnucash/src
make all

Can anybody tell me more precisely, what this error message actually 
means? Does "no code for module (gnucash core-utils)" mean, that 
something is not built yet? If yes, what is not built? Target 
"gnc-core-utils" is already built.


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 28.05.20 um 12:24 schrieb Frank H. Ellenberger:

Hi,

Am 28.05.20 um 08:13 schrieb Christian Gruber:

I have to search for left *.go files, right? There are a lot in
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete
directory without hesitation?

The easier way is to uninstall guile-modules-2_2 to get rid of them. ;-)
They are part of that package.


Ok, I checked that with inverse file search ("apt-file search"). On my 
Ubuntu system these files belong to package guile-2.2-libs. And the 
*.scm files in /usr/share/guile/2.2 belong to that package as well. But 
after uninstalling guile-2.2-libs CMake complains, when building GnuCash:


CMake Error at CMakeLists.txt:311 (message):
  Neither guile 2.2 nor guile 2.0 were found GnuCash can't run without 
one of

  them.  Ensure that one is installed and can be found with pkg-config.

Therefore I installed guile-2.2-libs again. And the build error is the 
same again as at the beginning:


ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)

Is this maybe a problem of a missing CMake dependency?

Another idea, I updated my Ubuntu system from 19.10 Eoan Ermine to 20.04 
Focal Fossa a few days ago. Could this be a reason?



___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 28.05.20 um 06:31 schrieb John Ralls:



On May 27, 2020, at 2:15 PM, Christian Gruber  
wrote:


Am 27.05.20 um 22:16 schrieb John Ralls:

On May 27, 2020, at 1:09 PM, Christian Gruber  
wrote:


Am 27.05.20 um 00:28 schrieb John Ralls:

On May 26, 2020, at 1:34 PM, Christian Gruber  
wrote:

Hi,

I tried to build GnuCash from master (currently fde6be6e0) for the first time. 
Before I always built from maint.

I get a strange build error:

Scanning dependencies of target scm-engine-2
[ 30%] Generating 
../../lib/x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/utilities.go
Backtrace:
In scripts/compile.scm:
264:26 19 (_ _)
In system/base/target.scm:
  57:6 18 (with-target _ _)
In system/base/compile.scm:
 152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _ …)
  43:4 16 (call-once _)
In ice-9/boot-9.scm:
 841:4 15 (with-throw-handler _ _ _)
In system/base/compile.scm:
 59:11 14 (_)
155:11 13 (_ #)
235:18 12 (read-and-compile # #:from _ # …)
183:32 11 (compile-fold (#) …)
In ice-9/boot-9.scm:
2312:4 10 (save-module-excursion #)
In language/scheme/compile-tree-il.scm:
 31:15  9 (_)
In ice-9/psyntax.scm:
   1262:36  8 (expand-top-sequence ((use-modules (gnucash #))) _ _ #f …)
   1209:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
285:10  6 (parse _ (("placeholder" placeholder)) (()) _ c (# #) #)
In ice-9/boot-9.scm:
   3377:20  5 (process-use-modules _)
222:17  4 (map1 (((gnucash core-utils
   3378:31  3 (_ ((gnucash core-utils)))
2803:6  2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ …)
In unknown file:
1 (scm-error misc-error #f "~A ~S" ("no code for modu…" …) …)
In ice-9/boot-9.scm:
752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)



I get the same build error even when using earlier commits from master, tag 
3.902 for instance.

Do I do something wrong?

Christian,

Did you remember to uninstall GnuCash and build in a completely clean (as in rm 
-rf *) build directory?

Yes, I do remember. And I already did uninstall GnuCash and built in a 
completely clean build directory. Nevertheless I still get this build error. 
Maybe the uninstall process was not complete and there are some files left.

I have to search for left *.go files, right? There are a lot in 
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete directory 
without hesitation?

Yes, but if they're in /usr that should mean that you have at least the 
remnants of a package manager installation. You should probably check that the 
package manager thinks that you've uninstalled gnucash.

I checked, that the package manager does not expect an existing GnuCash 
installation and removed the complete directory 
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. But it seems, that this was too 
much. Now I see a lot of the following warnings for different *.scm files:

;;; WARNING: compilation of /usr/share/guile/2.2/language/bytecode/spec.scm 
failed:
;;; no such language bytecode

Another directory to nuke.

I should delete *.scm files?


Regards,
John Ralls


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 27.05.20 um 22:16 schrieb John Ralls:



On May 27, 2020, at 1:09 PM, Christian Gruber  
wrote:


Am 27.05.20 um 00:28 schrieb John Ralls:

On May 26, 2020, at 1:34 PM, Christian Gruber  
wrote:

Hi,

I tried to build GnuCash from master (currently fde6be6e0) for the first time. 
Before I always built from maint.

I get a strange build error:

Scanning dependencies of target scm-engine-2
[ 30%] Generating 
../../lib/x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/utilities.go
Backtrace:
In scripts/compile.scm:
264:26 19 (_ _)
In system/base/target.scm:
  57:6 18 (with-target _ _)
In system/base/compile.scm:
 152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _ …)
  43:4 16 (call-once _)
In ice-9/boot-9.scm:
 841:4 15 (with-throw-handler _ _ _)
In system/base/compile.scm:
 59:11 14 (_)
155:11 13 (_ #)
235:18 12 (read-and-compile # #:from _ # …)
183:32 11 (compile-fold (#) …)
In ice-9/boot-9.scm:
2312:4 10 (save-module-excursion #)
In language/scheme/compile-tree-il.scm:
 31:15  9 (_)
In ice-9/psyntax.scm:
   1262:36  8 (expand-top-sequence ((use-modules (gnucash #))) _ _ #f …)
   1209:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
285:10  6 (parse _ (("placeholder" placeholder)) (()) _ c (# #) #)
In ice-9/boot-9.scm:
   3377:20  5 (process-use-modules _)
222:17  4 (map1 (((gnucash core-utils
   3378:31  3 (_ ((gnucash core-utils)))
2803:6  2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ …)
In unknown file:
1 (scm-error misc-error #f "~A ~S" ("no code for modu…" …) …)
In ice-9/boot-9.scm:
752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)



I get the same build error even when using earlier commits from master, tag 
3.902 for instance.

Do I do something wrong?

Christian,

Did you remember to uninstall GnuCash and build in a completely clean (as in rm 
-rf *) build directory?

Yes, I do remember. And I already did uninstall GnuCash and built in a 
completely clean build directory. Nevertheless I still get this build error. 
Maybe the uninstall process was not complete and there are some files left.

I have to search for left *.go files, right? There are a lot in 
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete directory 
without hesitation?

Yes, but if they're in /usr that should mean that you have at least the 
remnants of a package manager installation. You should probably check that the 
package manager thinks that you've uninstalled gnucash.


I checked, that the package manager does not expect an existing GnuCash 
installation and removed the complete directory 
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. But it seems, that this was 
too much. Now I see a lot of the following warnings for different *.scm 
files:


;;; WARNING: compilation of 
/usr/share/guile/2.2/language/bytecode/spec.scm failed:

;;; no such language bytecode



Regards,
John Ralls


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Am 27.05.20 um 00:28 schrieb John Ralls:



On May 26, 2020, at 1:34 PM, Christian Gruber  
wrote:

Hi,

I tried to build GnuCash from master (currently fde6be6e0) for the first time. 
Before I always built from maint.

I get a strange build error:

Scanning dependencies of target scm-engine-2
[ 30%] Generating 
../../lib/x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/utilities.go
Backtrace:
In scripts/compile.scm:
264:26 19 (_ _)
In system/base/target.scm:
  57:6 18 (with-target _ _)
In system/base/compile.scm:
 152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _ …)
  43:4 16 (call-once _)
In ice-9/boot-9.scm:
 841:4 15 (with-throw-handler _ _ _)
In system/base/compile.scm:
 59:11 14 (_)
155:11 13 (_ #)
235:18 12 (read-and-compile # #:from _ # …)
183:32 11 (compile-fold (#) …)
In ice-9/boot-9.scm:
2312:4 10 (save-module-excursion #)
In language/scheme/compile-tree-il.scm:
 31:15  9 (_)
In ice-9/psyntax.scm:
   1262:36  8 (expand-top-sequence ((use-modules (gnucash #))) _ _ #f …)
   1209:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
285:10  6 (parse _ (("placeholder" placeholder)) (()) _ c (# #) #)
In ice-9/boot-9.scm:
   3377:20  5 (process-use-modules _)
222:17  4 (map1 (((gnucash core-utils
   3378:31  3 (_ ((gnucash core-utils)))
2803:6  2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ …)
In unknown file:
1 (scm-error misc-error #f "~A ~S" ("no code for modu…" …) …)
In ice-9/boot-9.scm:
752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)



I get the same build error even when using earlier commits from master, tag 
3.902 for instance.

Do I do something wrong?

Christian,

Did you remember to uninstall GnuCash and build in a completely clean (as in rm 
-rf *) build directory?


Yes, I do remember. And I already did uninstall GnuCash and built in a 
completely clean build directory. Nevertheless I still get this build 
error. Maybe the uninstall process was not complete and there are some 
files left.


I have to search for left *.go files, right? There are a lot in 
/usr/lib/x86_64-linux-gnu/guile/2.2/ccache. Can I remove the complete 
directory without hesitation?




Regards,
John Ralls


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] I need help building GnuCash on master

[Christian Gruber]

Hi,

I tried to build GnuCash from master (currently fde6be6e0) for the first time. 
Before I always built from maint.

I get a strange build error:

Scanning dependencies of target scm-engine-2
[ 30%] Generating 
../../lib/x86_64-linux-gnu/guile/2.2/site-ccache/gnucash/utilities.go
Backtrace:
In scripts/compile.scm:
   264:26 19 (_ _)
In system/base/target.scm:
 57:6 18 (with-target _ _)
In system/base/compile.scm:
    152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _ …)
 43:4 16 (call-once _)
In ice-9/boot-9.scm:
    841:4 15 (with-throw-handler _ _ _)
In system/base/compile.scm:
    59:11 14 (_)
   155:11 13 (_ #)
   235:18 12 (read-and-compile # #:from _ # …)
   183:32 11 (compile-fold (#) …)
In ice-9/boot-9.scm:
   2312:4 10 (save-module-excursion #)
In language/scheme/compile-tree-il.scm:
    31:15  9 (_)
In ice-9/psyntax.scm:
  1262:36  8 (expand-top-sequence ((use-modules (gnucash #))) _ _ #f …)
  1209:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
   285:10  6 (parse _ (("placeholder" placeholder)) (()) _ c (# #) #)
In ice-9/boot-9.scm:
  3377:20  5 (process-use-modules _)
   222:17  4 (map1 (((gnucash core-utils
  3378:31  3 (_ ((gnucash core-utils)))
   2803:6  2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ …)
In unknown file:
   1 (scm-error misc-error #f "~A ~S" ("no code for modu…" …) …)
In ice-9/boot-9.scm:
   752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash core-utils)



I get the same build error even when using earlier commits from master, tag 
3.902 for instance.

Do I do something wrong?

Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Is the import match map still required?

[Christian Gruber]

Am 24.05.20 um 01:52 schrieb David Cousens:

Christian,

I guess it depends on whether there is a performance advantage in using the
previously stored data for the transfer account associations over
constructing the frequency table on the fly. The search for matching
transactions only takes place within a narrow time window around the date of
import, so it is unlikely to canvas enough transactions to be able to
construct a valid frequency table from tokenized data within that window.
The stored frequency table would generally contain data from a much wider
range of transactions and would take much longer to construct on the fly
each time it was needed.
I'm only thinking about account matching (bayesian matching), not 
transaction matching. For this of course it would be necessary to work 
with all historical data, not only with a few transactions within a 
narrow time window. Can you tell, if it would be a considerable 
performance load to construct the frequency table on the fly from all 
historical transactions related to a transfer account?

I have also pondered whether it could be usefully augmented by using data
from transactions entered manually which have not been imported for the file
associations.  Could be of value where you have a good set of historical
records but it would only need a one off run through the existing
transactions to gather the data. Unless you confined it to running on a
specific set of accounts to which you import data it might cause bloat of
the data file with unnecessary and unused information.


A possible advantage of constructing the frequency table on the fly 
could be, that it is always up-to-date. If the user sets the "wrong" 
other account during import for instance and corrects this after the 
import, the import match map still contains the wrong matching 
information at the moment and will also not be corrected after the import.


Also manually entered transactions would be considered, right.

A one-off manual run through all transactions to update the import match 
map could be a good alternative to constructing it on the fly. Sounds good.


Why do you think, a run through all transactions "might cause a bloat of 
the data file"? The current import match map also contains all, maybe 
unused or unnecessary data from all matched accounts. I still assume in 
this case, that the import match map is related to one transfer account 
only, which already limits the set of accounts from which the import 
match map is constructed.



I have examined the stored data in my  data file with the import map editor
and found that there was a lot of data stored which contributes little to
the matching for the transfer account ( dates, connectors (a, and, the
etc.), transaction amounts ?) which often have a fairly uniform frequency
for all accounts which were used as transfer accounts. After a bit of
pruning of the stored data my matching reliability seemed to improve a bit.
Ok, I see. If the import match map has to be pruned to get reliable 
results from the bayesian matching algorithm, a frequency table, which 
is constructed on the fly or is rebuilt on a one-off run, is a big 
disadvantage. If it is constructed on the fly, nothing can be pruned. 
And if it is rebuilt, all pruned data will back after the run.

I don't know at the moment if the tokens stored for transfer account
matching are a subset of the tokens used for transaction matching (haven't
checked) but restricting the set of tokens used may possibly improve
performance and reduce the amount of data stored if all tokens associated
with a transaction are currently being stored in the frequency table which
is what I suspect from examining my import map data.
Yes, this is the current situation, every token is stored. Do you have 
suggestions, how tokens could be automatically pruned in a meaningful way?


David Cousens



-
David Cousens
--
Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-Dev-f1435356.html
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] Is the import match map still required?

[Christian Gruber]

Hi devs,

Meanwhile, I'm studying the bayesian import matching algorithm quite 
intensively. There is one question, I was often asking myself and which 
I want to ask you now. The same question was also asked in the german 
mailing list a few days ago.


Is it really necessary to work with a separate "import match map" 
instead of the imported transaction data directly? The import match map 
contains matching information from earlier imports, but do the imported 
transactions not contain the same information? I mean instead of 
querying token occurrences in this import match map for instance, 
couldn't the bayesian algorithm get the same information by querying the 
already imported transactions? Is this a performance consideration or 
are there further reasons?


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Dependency graphs

[Christian Gruber]

Hi Geert,

I tried to generate a dependency graph now as well using CMake. And I 
played around a little bit with configuration using a 
CMakeGraphVizOptions.cmake 
 file.


I have two suggestions:

 * set(GRAPHVIZ_EXTERNAL_LIBS OFF)
 * set(GRAPHVIZ_IGNORE_TARGETS "test-*")

Furthermore it seems useful to set graph attribute rankdir to "LR" when 
invoking dot (-Grankdir=LR).


I attached a dependency graph using these settings built from current 
maint branch.


I was wondering why there are no scheme targets included. But then I saw 
that CMake is currently not able to include custom targets into the 
dependency graphs and scheme targets are custom targets.


Regards,
Christian


Am 08.12.19 um 14:55 schrieb Geert Janssens:

While discussing cmake target dependencies on irc with Chris I started
wondering how hard it would be to visualize our current dependency tree.

Turns out with cmake it's pretty straightforward to export the dependency info
in a format that's understood by the graphviz tool.

So I have created two graphs: one for current maint and one for current
master.

My plan is to put them in the wiki for reference, together with the commands
used to create them. Currently the wiki won't allow uploads of svg files, so
that bit has to wait some more.

Meanwhile I'll post them here for those interested.

Regards,

Geert

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

Am 11.11.19 um 11:22 schrieb Geert Janssens:

Op zondag 10 november 2019 23:28:06 CET schreef Christian Gruber:

Am 10.11.19 um 22:39 schrieb John Ralls:

On Nov 10, 2019, at 12:07 PM, Christian Gruber
 wrote:>>
Am 08.11.19 um 23:08 schrieb John Ralls:

On Nov 8, 2019, at 1:58 PM, Christian Gruber
 wrote:>>>>
Am 08.11.19 um 04:39 schrieb John Ralls:

Christian,

It's not that it's not prepared for Bayesian matching, it's that older
versions of GnuCash stored the Bayesian match tokens hierarchically.
Aaron Laws (lmat) changed it to a flatter structure with somewhat
better memory locality for faster access. imap_convert_bayes_to_flat
should run once to convert the data and set the feature, after which
check_import_map_data will see the flag and return. A file created
with 3.x and Baysian maps would already have the feature set.


With that background, to your questions:

Why does it take so long? Because it traverses the entire tree of
accounts, every time. The test book has 1127 accounts. Add to that
that there are some things inside the loop that shouldn't be and that
convert_imap_account_bayes_to_flat doesn't use some obvious short
circuits and you get taking a long time.

Why does it run twice? Because there aren't any accounts with
import-map-bayes slots, so it does no conversions so it doesn't set
the feature.>>>>

Why isn't the feature set in any case after conversion is done, whether
there was any slot to convert or not?

I would expect that, if the hierarchical structure should not be used
anymore. I wouldn't only set the feature, if there has been any error
during conversion. But it's not an error, if
convert_imap_account_bayes_to_flat() returns false.>>>

The feature isn't set because that would prevent using the file with an
older version of GnuCash for no good reason: There aren't any
import-map-bayes tags in the new format so there's nothing for the
older version to mess up.

Regards,
John Ralls

Sounds meaningful, but it's not obvious. The conversion process is not
much transparent to the user. The user is not informed about conversion
and additionally the conditions, when the GnuCash file is converted and
when not, are not quite intuitive, I guess. And the second question for
me is, if this is really a relevant use case to keep the GnuCash file
unconverted if possible, when it is used with a newer GnuCash version.
Maybe it's relevant, if several people, who use different GnuCash
versions work on the same GnuCash file. But what happens, when one with
a newer GnuCash version uses the bayesian import matcher for the first
time, which silently converts the GnuCash file? Will it still be usable
with an older GnuCash version?

But actually this is another discussion, which is not important for the
bug ticket. What's more important, I created a new fresh GnuCash file
with the SKR04 accounts template using my GnuCash version 3.7 and if I
checked this correctly, even with this GnuCash version the feature
GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you
confirm this? If yes, then the bug report is relevant not only for
migration from older to current GnuCash versions but even for current
GnuCash versions in general.>

No, it's the same discussion. If you don't use the feature then the flag
for it won't be set. That's how we designed the feature mechanism. The
use case is straightforward: Some user has two computers, a laptop
running Windows and a desktop running Ubuntu 16.04. She updates the
Windows machine with each release but doesn't know how to build GnuCash
on her Ubuntu machine so leaves it at whatever they shipped on 16.04;
2.6.12 IIRC. As long as she doesn't use any of the features supported
only in 3.x she can continue to use both systems to keep her books and
regardless of which machine she uses to create a book.

The "silently" part could be better. It would indeed be a nice enhancement
to pop a dialog box when GnuCash is about to set a feature flag
explaining that it's use will preclude using the file on a GnuCash
version older than X and giving the user a chance to bail out. It would
be nicer still to persist that decision so that it need ask only once,
and if the user assents it could set the feature flag even if the feature
isn't used, as is the case here. Of course there would then be the need
to reset negative responses so that when our hypothetical user upgrades
her desktop to Ubuntu 20.04 next spring she can turn on all of those 3.x
only features she'd previously declined.

It occurs to me that another improvement to the existing flat Bayesian
conversion would be to check the state of the GNC_PREF_USE_BAYES and skip
the conversion if it's not true.

This is already implemented. Flag GNC_PREF_USE_BAYES is checked in
functions matchmap_find_destination() and matchmap_store_destination()
and if not set, than functions gnc_account_imap_find_account_bayes() and
gnc_account_imap_add_account_bayes() aren't inv

Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

Am 10.11.19 um 23:28 schrieb Christian Gruber:


Am 10.11.19 um 22:39 schrieb John Ralls:


On Nov 10, 2019, at 12:07 PM, Christian Gruber 
 wrote:



Am 08.11.19 um 23:08 schrieb John Ralls:
On Nov 8, 2019, at 1:58 PM, Christian Gruber 
 wrote:


Am 08.11.19 um 04:39 schrieb John Ralls:

Christian,

It's not that it's not prepared for Bayesian matching, it's that 
older versions of GnuCash stored the Bayesian match tokens 
hierarchically. Aaron Laws (lmat) changed it to a flatter 
structure with somewhat better memory locality for faster access. 
imap_convert_bayes_to_flat should run once to convert the data 
and set the feature, after which check_import_map_data will see 
the flag and return. A file created with 3.x and Baysian maps 
would already have the feature set.



With that background, to your questions:

Why does it take so long? Because it traverses the entire tree of 
accounts, every time. The test book has 1127 accounts. Add to 
that that there are some things inside the loop that shouldn't be 
and that convert_imap_account_bayes_to_flat doesn't use some 
obvious short circuits and you get taking a long time.


Why does it run twice? Because there aren't any accounts with 
import-map-bayes slots, so it does no conversions so it doesn't 
set the feature.
Why isn't the feature set in any case after conversion is done, 
whether there was any slot to convert or not?


I would expect that, if the hierarchical structure should not be 
used anymore. I wouldn't only set the feature, if there has been 
any error during conversion. But it's not an error, if 
convert_imap_account_bayes_to_flat() returns false.
The feature isn't set because that would prevent using the file 
with an older version of GnuCash for no good reason: There aren't 
any import-map-bayes tags in the new format so there's nothing for 
the older version to mess up.


Regards,
John Ralls

Sounds meaningful, but it's not obvious. The conversion process is 
not much transparent to the user. The user is not informed about 
conversion and additionally the conditions, when the GnuCash file is 
converted and when not, are not quite intuitive, I guess. And the 
second question for me is, if this is really a relevant use case to 
keep the GnuCash file unconverted if possible, when it is used with 
a newer GnuCash version. Maybe it's relevant, if several people, who 
use different GnuCash versions work on the same GnuCash file. But 
what happens, when one with a newer GnuCash version uses the 
bayesian import matcher for the first time, which silently converts 
the GnuCash file? Will it still be usable with an older GnuCash 
version?


But actually this is another discussion, which is not important for 
the bug ticket. What's more important, I created a new fresh GnuCash 
file with the SKR04 accounts template using my GnuCash version 3.7 
and if I checked this correctly, even with this GnuCash version the 
feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash 
file. Can you confirm this? If yes, then the bug report is relevant 
not only for migration from older to current GnuCash versions but 
even for current GnuCash versions in general.
No, it's the same discussion. If you don't use the feature then the 
flag for it won't be set. That's how we designed the feature 
mechanism. The use case is straightforward: Some user has two 
computers, a laptop running Windows and a desktop running Ubuntu 
16.04. She updates the Windows machine with each release but doesn't 
know how to build GnuCash on her Ubuntu machine so leaves it at 
whatever they shipped on 16.04; 2.6.12 IIRC. As long as she doesn't 
use any of the features supported only in 3.x she can continue to use 
both systems to keep her books and regardless of which machine she 
uses to create a book.


The "silently" part could be better. It would indeed be a nice 
enhancement to pop a dialog box when GnuCash is about to set a 
feature flag explaining that it's use will preclude using the file on 
a GnuCash version older than X and giving the user a chance to bail 
out. It would be nicer still to persist that decision so that it need 
ask only once, and if the user assents it could set the feature flag 
even if the feature isn't used, as is the case here. Of course there 
would then be the need to reset negative responses so that when our 
hypothetical user upgrades her desktop to Ubuntu 20.04 next spring 
she can turn on all of those 3.x only features she'd previously 
declined.


It occurs to me that another improvement to the existing flat 
Bayesian conversion would be to check the state of the 
GNC_PREF_USE_BAYES and skip the conversion if it's not true.
This is already implemented. Flag GNC_PREF_USE_BAYES is checked in 
functions matchmap_find_destination() and matchmap_store_destination() 
and if not set, than functions gnc_account_imap_find_account_bayes() 
and gnc_account_imap_add_account_bayes() aren't invoked and no 
conversion will be done.


Regards,

Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

Am 10.11.19 um 22:39 schrieb John Ralls:



On Nov 10, 2019, at 12:07 PM, Christian Gruber  
wrote:


Am 08.11.19 um 23:08 schrieb John Ralls:

On Nov 8, 2019, at 1:58 PM, Christian Gruber  wrote:

Am 08.11.19 um 04:39 schrieb John Ralls:

Christian,

It's not that it's not prepared for Bayesian matching, it's that older versions 
of GnuCash stored the Bayesian match tokens hierarchically. Aaron Laws (lmat) 
changed it to a flatter structure with somewhat better memory locality for 
faster access. imap_convert_bayes_to_flat should run once to convert the data 
and set the feature, after which check_import_map_data will see the flag and 
return. A file created with 3.x and Baysian maps would already have the feature 
set.


With that background, to your questions:

Why does it take so long? Because it traverses the entire tree of accounts, 
every time. The test book has 1127 accounts. Add to that that there are some 
things inside the loop that shouldn't be and that 
convert_imap_account_bayes_to_flat doesn't use some obvious short circuits and 
you get taking a long time.

Why does it run twice? Because there aren't any accounts with import-map-bayes 
slots, so it does no conversions so it doesn't set the feature.

Why isn't the feature set in any case after conversion is done, whether there 
was any slot to convert or not?

I would expect that, if the hierarchical structure should not be used anymore. 
I wouldn't only set the feature, if there has been any error during conversion. 
But it's not an error, if convert_imap_account_bayes_to_flat() returns false.

The feature isn't set because that would prevent using the file with an older 
version of GnuCash for no good reason: There aren't any import-map-bayes tags 
in the new format so there's nothing for the older version to mess up.

Regards,
John Ralls


Sounds meaningful, but it's not obvious. The conversion process is not much 
transparent to the user. The user is not informed about conversion and 
additionally the conditions, when the GnuCash file is converted and when not, 
are not quite intuitive, I guess. And the second question for me is, if this is 
really a relevant use case to keep the GnuCash file unconverted if possible, 
when it is used with a newer GnuCash version. Maybe it's relevant, if several 
people, who use different GnuCash versions work on the same GnuCash file. But 
what happens, when one with a newer GnuCash version uses the bayesian import 
matcher for the first time, which silently converts the GnuCash file? Will it 
still be usable with an older GnuCash version?

But actually this is another discussion, which is not important for the bug 
ticket. What's more important, I created a new fresh GnuCash file with the 
SKR04 accounts template using my GnuCash version 3.7 and if I checked this 
correctly, even with this GnuCash version the feature 
GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you confirm 
this? If yes, then the bug report is relevant not only for migration from older 
to current GnuCash versions but even for current GnuCash versions in general.

No, it's the same discussion. If you don't use the feature then the flag for it 
won't be set. That's how we designed the feature mechanism. The use case is 
straightforward: Some user has two computers, a laptop running Windows and a 
desktop running Ubuntu 16.04. She updates the Windows machine with each release 
but doesn't know how to build GnuCash on her Ubuntu machine so leaves it at 
whatever they shipped on 16.04; 2.6.12 IIRC. As long as she doesn't use any of 
the features supported only in 3.x she can continue to use both systems to keep 
her books and regardless of which machine she uses to create a book.

The "silently" part could be better. It would indeed be a nice enhancement to 
pop a dialog box when GnuCash is about to set a feature flag explaining that it's use 
will preclude using the file on a GnuCash version older than X and giving the user a 
chance to bail out. It would be nicer still to persist that decision so that it need ask 
only once, and if the user assents it could set the feature flag even if the feature 
isn't used, as is the case here. Of course there would then be the need to reset negative 
responses so that when our hypothetical user upgrades her desktop to Ubuntu 20.04 next 
spring she can turn on all of those 3.x only features she'd previously declined.

It occurs to me that another improvement to the existing flat Bayesian 
conversion would be to check the state of the GNC_PREF_USE_BAYES and skip the 
conversion if it's not true.
This is already implemented. Flag GNC_PREF_USE_BAYES is checked in 
functions matchmap_find_destination() and matchmap_store_destination() 
and if not set, than functions gnc_account_imap_find_account_bayes() and 
gnc_account_imap_add_account_bayes() aren't invoked and no conversion 
will be done.


Regards,
John Ralls

Thanks for the revealing explanation of the feature

Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

Am 08.11.19 um 23:08 schrieb John Ralls:



On Nov 8, 2019, at 1:58 PM, Christian Gruber  wrote:

Am 08.11.19 um 04:39 schrieb John Ralls:

Christian,

It's not that it's not prepared for Bayesian matching, it's that older versions 
of GnuCash stored the Bayesian match tokens hierarchically. Aaron Laws (lmat) 
changed it to a flatter structure with somewhat better memory locality for 
faster access. imap_convert_bayes_to_flat should run once to convert the data 
and set the feature, after which check_import_map_data will see the flag and 
return. A file created with 3.x and Baysian maps would already have the feature 
set.


With that background, to your questions:

Why does it take so long? Because it traverses the entire tree of accounts, 
every time. The test book has 1127 accounts. Add to that that there are some 
things inside the loop that shouldn't be and that 
convert_imap_account_bayes_to_flat doesn't use some obvious short circuits and 
you get taking a long time.

Why does it run twice? Because there aren't any accounts with import-map-bayes 
slots, so it does no conversions so it doesn't set the feature.

Why isn't the feature set in any case after conversion is done, whether there 
was any slot to convert or not?

I would expect that, if the hierarchical structure should not be used anymore. 
I wouldn't only set the feature, if there has been any error during conversion. 
But it's not an error, if convert_imap_account_bayes_to_flat() returns false.

The feature isn't set because that would prevent using the file with an older 
version of GnuCash for no good reason: There aren't any import-map-bayes tags 
in the new format so there's nothing for the older version to mess up.

Regards,
John Ralls

Sounds meaningful, but it's not obvious. The conversion process is not 
much transparent to the user. The user is not informed about conversion 
and additionally the conditions, when the GnuCash file is converted and 
when not, are not quite intuitive, I guess. And the second question for 
me is, if this is really a relevant use case to keep the GnuCash file 
unconverted if possible, when it is used with a newer GnuCash version. 
Maybe it's relevant, if several people, who use different GnuCash 
versions work on the same GnuCash file. But what happens, when one with 
a newer GnuCash version uses the bayesian import matcher for the first 
time, which silently converts the GnuCash file? Will it still be usable 
with an older GnuCash version?


But actually this is another discussion, which is not important for the 
bug ticket. What's more important, I created a new fresh GnuCash file 
with the SKR04 accounts template using my GnuCash version 3.7 and if I 
checked this correctly, even with this GnuCash version the feature 
GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the GnuCash file. Can you 
confirm this? If yes, then the bug report is relevant not only for 
migration from older to current GnuCash versions but even for current 
GnuCash versions in general.


Regards,
Christian


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

Am 08.11.19 um 04:39 schrieb John Ralls:

Christian,

It's not that it's not prepared for Bayesian matching, it's that older versions 
of GnuCash stored the Bayesian match tokens hierarchically. Aaron Laws (lmat) 
changed it to a flatter structure with somewhat better memory locality for 
faster access. imap_convert_bayes_to_flat should run once to convert the data 
and set the feature, after which check_import_map_data will see the flag and 
return. A file created with 3.x and Baysian maps would already have the feature 
set.


With that background, to your questions:

Why does it take so long? Because it traverses the entire tree of accounts, 
every time. The test book has 1127 accounts. Add to that that there are some 
things inside the loop that shouldn't be and that 
convert_imap_account_bayes_to_flat doesn't use some obvious short circuits and 
you get taking a long time.

Why does it run twice? Because there aren't any accounts with import-map-bayes 
slots, so it does no conversions so it doesn't set the feature.


Why isn't the feature set in any case after conversion is done, whether 
there was any slot to convert or not?


I would expect that, if the hierarchical structure should not be used 
anymore. I wouldn't only set the feature, if there has been any error 
during conversion. But it's not an error, if 
convert_imap_account_bayes_to_flat() returns false.




Why is the feature not set after the import? It should be if it's actually 
setting any matches. That's done at the end of change_imap_entry. If you're 
quitting the matcher without associating the transactions to accounts it won't 
call change_imap_entry and set the feature.

Regards,
John Ralls



On Nov 7, 2019, at 2:41 PM, Christian Gruber  wrote:

Can anybody provide help?

The last change on the relevant functions was done in commit fbf4843f31 by 
"lmat" in Dec 2017 between GnuCash versions 2.6 and 2.7. And the commit message 
seems to fit.

Christian

Am 04.11.19 um 20:28 schrieb Christian Gruber:

I have some questions related to Bug 797463 
<https://bugs.gnucash.org/show_bug.cgi?id=797463>, which I have analyzed.

The author wrote, that "Gnu Cash hangs", when importing (only two) transactions 
into a Gnu Cash file with standard accounts list SKR04.

My analysis showed, that actually Gnu Cash does not hang, but needs really long 
time for import (several minutes). The problem is, that the author has bayesian 
matching enabled in his preferences, but the Gnu Cash file is not prepared for 
bayesian matching (feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the Gnu 
Cash file). Moreover the Gnu Cash file contains a lot of accounts (approx. 
1000).

Most time is spent in function check_import_map_data(), which is called from 
gnc_account_imap_find_account_bayes() (Account.cpp). In this function 
imap_convert_bayes_to_flat() is called, which AFAICS prepares all accounts for 
bayesian matching.

I'm not familiar with that conversion step and therefore have several questions:

Why does the conversion need so much CPU time?

If the conversion needs so much CPU time, why is it done only temporarily? The 
conversion is done for each of the two transactions again.

Why is the conversion even not persistent after the import is done? The feature 
GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the Gnu Cash file, even not after 
the import is finished.

Regards,
Christian


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

Can anybody provide help?

The last change on the relevant functions was done in commit fbf4843f31 
by "lmat" in Dec 2017 between GnuCash versions 2.6 and 2.7. And the 
commit message seems to fit.


Christian

Am 04.11.19 um 20:28 schrieb Christian Gruber:
I have some questions related to Bug 797463 
<https://bugs.gnucash.org/show_bug.cgi?id=797463>, which I have analyzed.


The author wrote, that "Gnu Cash hangs", when importing (only two) 
transactions into a Gnu Cash file with standard accounts list SKR04.


My analysis showed, that actually Gnu Cash does not hang, but needs 
really long time for import (several minutes). The problem is, that 
the author has bayesian matching enabled in his preferences, but the 
Gnu Cash file is not prepared for bayesian matching (feature 
GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the Gnu Cash file). 
Moreover the Gnu Cash file contains a lot of accounts (approx. 1000).


Most time is spent in function check_import_map_data(), which is 
called from gnc_account_imap_find_account_bayes() (Account.cpp). In 
this function imap_convert_bayes_to_flat() is called, which AFAICS 
prepares all accounts for bayesian matching.


I'm not familiar with that conversion step and therefore have several 
questions:


Why does the conversion need so much CPU time?

If the conversion needs so much CPU time, why is it done only 
temporarily? The conversion is done for each of the two transactions 
again.


Why is the conversion even not persistent after the import is done? 
The feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the Gnu Cash 
file, even not after the import is finished.


Regards,
Christian


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] What about outdated open bugs in Gnucash Bugzilla?

[Christian Gruber]

Ok, I'll start doing that on all bugs older than 10 years. If there is 
no reaction after at least one month, I'll close them as RESOLVED.


By the way, I found some bugs, which were already commented this way by 
"Wm", but without setting status to NEEDINFO. And he/she didn't close 
the bugs. I'll close these as well, if there is no reaction after at 
least one month. See https://bugs.gnucash.org/show_bug.cgi?id=459278 for 
instance.


Christian

Am 28.10.19 um 04:03 schrieb John Ralls:

Colin,

That's a worthwhile idea that could be easily applied to all bugs over n years 
old. One could start with n around 10, perhaps meaning everything against 
versions before 2.4.0.  Care to spend some quality time in Bugzilla?

Regards,
John Ralls


On Oct 27, 2019, at 3:14 PM, Colin Law  wrote:

Possibly an alternative, used in Ubuntu for example, is when a version
goes out of support that any bugs against that version have a comment
added saying the version is out of support, saying that if the bug is
relevant to a later version then please to post a comment, and marking
the bug as needing info, so it will expire after a time if there is no
input,

Colin

On Sun, 27 Oct 2019 at 21:46, David Cousens  wrote:

John,

Is there perhaps a need to place maintenance limits on GnuCash release
versions,  i.e. at a specified time after release they become unsupported,
as is bugs and all. This is more than likely what actually happens in
practice given limited skilled developer time. As bug reports are often tied
to a specific version, bugs could then be removed from bugzilla when the
version they apply to becomes unsupported.  Most bugs are not show stoppers
and those that are or are particularly inconvenient are usually fixed fairly
quickly after release.

Enhancement requests could possibly have a longer lifetime but perhaps there
is a need for expiry on those as well. E.g. if they didn't make it into the
next major release and you really need that feature you raise the issue
again (or work on it yourself).

David Cousens



-
David Cousens
--
Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-Dev-f1435356.html
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] Bug 797463 - CSV Import of transactions into a new file hangs

[Christian Gruber]

I have some questions related to Bug 797463 
, which I have analyzed.


The author wrote, that "Gnu Cash hangs", when importing (only two) 
transactions into a Gnu Cash file with standard accounts list SKR04.


My analysis showed, that actually Gnu Cash does not hang, but needs 
really long time for import (several minutes). The problem is, that the 
author has bayesian matching enabled in his preferences, but the Gnu 
Cash file is not prepared for bayesian matching (feature 
GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the Gnu Cash file). 
Moreover the Gnu Cash file contains a lot of accounts (approx. 1000).


Most time is spent in function check_import_map_data(), which is called 
from gnc_account_imap_find_account_bayes() (Account.cpp). In this 
function imap_convert_bayes_to_flat() is called, which AFAICS prepares 
all accounts for bayesian matching.


I'm not familiar with that conversion step and therefore have several 
questions:


Why does the conversion need so much CPU time?

If the conversion needs so much CPU time, why is it done only 
temporarily? The conversion is done for each of the two transactions again.


Why is the conversion even not persistent after the import is done? The 
feature GNC_FEATURE_GUID_FLAT_BAYESIAN is not set in the Gnu Cash file, 
even not after the import is finished.


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] What about outdated open bugs in Gnucash Bugzilla?

[Christian Gruber]

Am 27.10.19 um 03:40 schrieb John Ralls:



On Oct 26, 2019, at 2:34 PM, Christian Gruber  
wrote:

Hi, I'm currently looking through the (quite long) buglist on Gnucash Bugzilla 
<https://bugs.gnucash.org/> to see, where I can provide help. Unfortunatelly 
I'm a little bit frustrated, because of many entries, which are still open (STATUS != 
RESOLVED), but haven't changed for years.

Which of them are still relevant? When is a bug outdated? Are some of them 
maybe already resolved, but haven't been closed? Who actually closes a bug, the 
author or the Gnucash maintainers? Are there any plans to close outdated bugs?

I had a look on Bugzilla Administration 
<https://wiki.gnucash.org/wiki/Bugzilla_Administration> page, but this didn't 
answer my questions.


Christian,

The overall problem is too many bugs, not enough developer time to go through 
them, especially old ones. Yes, it's entirely possible that many of them have 
been subsequently fixed by someone working on something else and that someone 
didn't think to go looking for related bugs. Unfortunately it's equally likely 
that some of those old bugs are too hard to fix or even to find the cause, or 
that no developer ever even got interested in looking into them.

Which ones are still relevant is hard to determine: GnuCash is complex and there have 
been lots of bugs over the years that were revealed because of a corner case in someone's 
accounts, so just because a developer or tester can't reproduce a bug doesn't mean that 
it's invalid. I suppose one could declare a bug obsolete if the module that would have 
caused it is easily identifiable and that code has been substantially rewritten since, 
but determining that isn't necessarily easy. The only policy we have is that if a bug is 
marked "NEEDSINFO" and none has been provided after 3 Months it can be resolved 
as INCOMPLETE, usually with a note telling the reporter to reopen it if they ever get 
around to caring about it again.

It's nice but rare when the reporter closes a bug, so it usually falls to a 
developer.

Note that of the 1364 open GnuCash bugs, 489 are enhancement requests.

Non-enhancement bugs 2001-01-01 to 2010-12-31:  155
  2011-01-01 to 2015-12-31:  296
  2016-01-01 to 2016-12-31:   48
  2017-01-01 to 2017-12-31:   77
  2018-01-01 to 2018-12-31:  117
  2019-01-01 to Now: 210

As an illustration of how old bugs can still be relevant, 
https://bugs.gnucash.org/show_bug.cgi?id=88517 is the oldest open bug (from 
July 2002). It's about copying and pasting transactions, was last commented on 
in 2017, and I think
is still valid.

Regards,
John Ralls

Ok, I suspected something like that. I'll do my best to provide 
meaningful help.

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] What about outdated open bugs in Gnucash Bugzilla?

[Christian Gruber]

Hi, I'm currently looking through the (quite long) buglist on Gnucash 
Bugzilla  to see, where I can provide help. 
Unfortunatelly I'm a little bit frustrated, because of many entries, 
which are still open (STATUS != RESOLVED), but haven't changed for years.


Which of them are still relevant? When is a bug outdated? Are some of 
them maybe already resolved, but haven't been closed? Who actually 
closes a bug, the author or the Gnucash maintainers? Are there any plans 
to close outdated bugs?


I had a look on Bugzilla Administration 
 page, but this 
didn't answer my questions.


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Build error with scm-engine on current master

[Christian Gruber]

Oh I missed that. I actually fetched current state without rebasing.

Now everything is fine.

Regards,
Christian

Am 13.09.19 um 21:38 schrieb Geert Janssens:

John pushed a fix for your make check error earlier today.

As for the load path, I'm not sure. I know we have had issues with this in the
past. Perhaps those are solved by now.

Regards,

Geert

Op vrijdag 13 september 2019 21:27:41 CEST schreef Christian Gruber:

I tried again.

"make all" works fine now, but "make check" still generates errors. But this
is probably due to some still missing dependencies, as I already mentioned
in the previous thread "Need help - several tests fail". When I invoke
"make check" in a fresh build dir directly after CMake, then the following
error is generated when compiling gnucash/report/html-chart.scm:

Backtrace:
In scripts/compile.scm:
 259:26 19 (_ _)
In system/base/target.scm:
   57:6 18 (with-target _ _)
In system/base/compile.scm:
  152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _ …)
   43:4 16 (call-once _)
In ice-9/boot-9.scm:
  841:4 15 (with-throw-handler _ _ _)
In system/base/compile.scm:
  59:11 14 (_)
 155:11 13 (_ #)
 235:18 12 (read-and-compile # #:from _ # …)
 183:32 11 (compile-fold (#) …)
In ice-9/boot-9.scm:
 2312:4 10 (save-module-excursion #)
In language/scheme/compile-tree-il.scm:
  31:15  9 (_)
In ice-9/psyntax.scm:
1235:36  8 (expand-top-sequence ((use-modules (gnucash json #))) _ …)
1182:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
 285:10  6 (parse _ (("placeholder" placeholder)) (()) _ c (# #) #)
In ice-9/boot-9.scm:
3377:20  5 (process-use-modules _)
 222:17  4 (map1 (((gnucash json builder
3378:31  3 (_ ((gnucash json builder)))
 2803:6  2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ …)
In unknown file:
 1 (scm-error misc-error #f "~A ~S" ("no code for modu…" …) …)
In ice-9/boot-9.scm:
 752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash json builder)



Instead if I invoke "make all" first after CMake and then "make "check",
everything builds correctly.

The target with missing dependencies seems to be scm-report-2.



Finally I tried to clarify the question, if and how an existing GnuCash
installation can interfere with the build process, just out of interest.
In PR #574 <https://github.com/Gnucash/gnucash/pull/574> you wrote:

the linker picks up a 3.x version of gnucash installed system-wide

And John Ralls wrote:

The problem is that Guile loads C libraries with dlopen and that looks on
the system path [...] before it looks in the build directory

But as far as I can see when guile is invoked LD_LIBRARY_PATH is always set
to "${LIBDIR_BUILD}:${LIBDIR_BUILD}/gnucash", i.e. that these directories
are searched first before any system paths (see
http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html).

Did I miss something?

Christian

Am 12.09.19 um 10:45 schrieb Geert Janssens:

Hi Christian,

With the last commit on master it works here now.

Can you retry ?

Geert

Op donderdag 12 september 2019 09:47:43 CEST schreef Geert Janssens:

Hi Christian,

Thanks to your detailed description I can now reproduce the problem when
building master with "make" rather than "ninja".

Turns out there's a hidden circular dependency between engine.scm and
engine- utilities.scm via the call to
(gnc:module-begin-syntax (gnc:module-load "gnucash/engine" 0))

I have a fix ready for this particular case, but there are more of these
so
I'm still working through the others.

Regards,

Geert

Op donderdag 12 september 2019 00:46:13 CEST schreef Christian Gruber:

This thread is related to the build issue I encountered when finally
building PR #574 <https://github.com/Gnucash/gnucash/pull/574> locally.

Invoking "make all" on current master lead to the following build error
when compiling libgnucash/engine/engine-utilities.scm:

Backtrace:1 (primitive-load-path
"gnucash/engine/gnc-numeric") In ice-9/boot-9.scm:752:25  0
(dispatch-exception _ _ _) ice-9/boot-9.scm:752:25: In procedure
dispatch-exception: In procedure primitive-load-path: Unable to find
file "gnucash/engine/gnc-numeric" in load path

File libgnucash/engine/engine-utilities.scm is built with target
scm-engine-1, therefore the same build error can be reproduced by
invoking "make scm-engine-1" or "make scm-engine" as well.

According to PR #574 <https://github.com/Gnucash/gnucash/pull/574>  I
can preclude, that this is due to reusing a maint build directory, since
I built in a clean and fresh build dir.

Next I tried to build again after moving directories
/usr/lib/x86_64-linux-gnu/gnucash/gnucash a

Re: [GNC-dev] Build error with scm-engine on current master

[Christian Gruber]

I tried again.

"make all" works fine now, but "make check" still generates errors. But this is probably due to 
some still missing dependencies, as I already mentioned in the previous thread "Need help - several tests 
fail". When I invoke "make check" in a fresh build dir directly after CMake, then the following error is 
generated when compiling gnucash/report/html-chart.scm:

Backtrace:
In scripts/compile.scm:
   259:26 19 (_ _)
In system/base/target.scm:
 57:6 18 (with-target _ _)
In system/base/compile.scm:
    152:6 17 (compile-file _ #:output-file _ #:from _ #:to _ #:env _ …)
 43:4 16 (call-once _)
In ice-9/boot-9.scm:
    841:4 15 (with-throw-handler _ _ _)
In system/base/compile.scm:
    59:11 14 (_)
   155:11 13 (_ #)
   235:18 12 (read-and-compile # #:from _ # …)
   183:32 11 (compile-fold (#) …)
In ice-9/boot-9.scm:
   2312:4 10 (save-module-excursion #)
In language/scheme/compile-tree-il.scm:
    31:15  9 (_)
In ice-9/psyntax.scm:
  1235:36  8 (expand-top-sequence ((use-modules (gnucash json #))) _ …)
  1182:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # …)) …)
   285:10  6 (parse _ (("placeholder" placeholder)) (()) _ c (# #) #)
In ice-9/boot-9.scm:
  3377:20  5 (process-use-modules _)
   222:17  4 (map1 (((gnucash json builder
  3378:31  3 (_ ((gnucash json builder)))
   2803:6  2 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ …)
In unknown file:
   1 (scm-error misc-error #f "~A ~S" ("no code for modu…" …) …)
In ice-9/boot-9.scm:
   752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
no code for module (gnucash json builder)



Instead if I invoke "make all" first after CMake and then "make "check", 
everything builds correctly.

The target with missing dependencies seems to be scm-report-2.



Finally I tried to clarify the question, if and how an existing GnuCash 
installation can interfere with the build process, just out of interest.

In PR #574 <https://github.com/Gnucash/gnucash/pull/574> you wrote:


the linker picks up a 3.x version of gnucash installed system-wide


And John Ralls wrote:


The problem is that Guile loads C libraries with dlopen and that looks on the 
system path [...] before it looks in the build directory


But as far as I can see when guile is invoked LD_LIBRARY_PATH is always set to 
"${LIBDIR_BUILD}:${LIBDIR_BUILD}/gnucash", i.e. that these directories are 
searched first before any system paths (see 
http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html).

Did I miss something?

Christian



Am 12.09.19 um 10:45 schrieb Geert Janssens:

Hi Christian,

With the last commit on master it works here now.

Can you retry ?

Geert

Op donderdag 12 september 2019 09:47:43 CEST schreef Geert Janssens:

Hi Christian,

Thanks to your detailed description I can now reproduce the problem when
building master with "make" rather than "ninja".

Turns out there's a hidden circular dependency between engine.scm and
engine- utilities.scm via the call to
(gnc:module-begin-syntax (gnc:module-load "gnucash/engine" 0))

I have a fix ready for this particular case, but there are more of these so
I'm still working through the others.

Regards,

Geert

Op donderdag 12 september 2019 00:46:13 CEST schreef Christian Gruber:

This thread is related to the build issue I encountered when finally
building PR #574 <https://github.com/Gnucash/gnucash/pull/574> locally.

Invoking "make all" on current master lead to the following build error
when compiling libgnucash/engine/engine-utilities.scm:

Backtrace:1 (primitive-load-path
"gnucash/engine/gnc-numeric") In ice-9/boot-9.scm:752:25  0
(dispatch-exception _ _ _) ice-9/boot-9.scm:752:25: In procedure
dispatch-exception: In procedure primitive-load-path: Unable to find
file "gnucash/engine/gnc-numeric" in load path

File libgnucash/engine/engine-utilities.scm is built with target
scm-engine-1, therefore the same build error can be reproduced by
invoking "make scm-engine-1" or "make scm-engine" as well.

According to PR #574 <https://github.com/Gnucash/gnucash/pull/574>  I
can preclude, that this is due to reusing a maint build directory, since
I built in a clean and fresh build dir.

Next I tried to build again after moving directories
/usr/lib/x86_64-linux-gnu/gnucash/gnucash and files
/usr/lib/x86_64-linux-gnu/gnucash/libgnc*.so.

After that I uninstalled my GnuCash 3.5, which was installed before from
distros package manager.

Finally I searched my whole system for gnucash directories and removed
them in /usr/local/... where I previously installed GnuCash built from
source repo.

Everytime I built in a clean and fresh build dir, but the build error
was always the same.

Finally I further investigated sources and found 

[GNC-dev] Build error with scm-engine on current master

[Christian Gruber]

This thread is related to the build issue I encountered when finally 
building PR #574  locally.


Invoking "make all" on current master lead to the following build error 
when compiling libgnucash/engine/engine-utilities.scm:


Backtrace:    1 (primitive-load-path 
"gnucash/engine/gnc-numeric") In ice-9/boot-9.scm:    752:25  0 
(dispatch-exception _ _ _) ice-9/boot-9.scm:752:25: In procedure 
dispatch-exception: In procedure primitive-load-path: Unable to find 
file "gnucash/engine/gnc-numeric" in load path


File libgnucash/engine/engine-utilities.scm is built with target 
scm-engine-1, therefore the same build error can be reproduced by 
invoking "make scm-engine-1" or "make scm-engine" as well.


According to PR #574   I 
can preclude, that this is due to reusing a maint build directory, since 
I built in a clean and fresh build dir.


Next I tried to build again after moving directories 
/usr/lib/x86_64-linux-gnu/gnucash/gnucash and files 
/usr/lib/x86_64-linux-gnu/gnucash/libgnc*.so.


After that I uninstalled my GnuCash 3.5, which was installed before from 
distros package manager.


Finally I searched my whole system for gnucash directories and removed 
them in /usr/local/... where I previously installed GnuCash built from 
source repo.


Everytime I built in a clean and fresh build dir, but the build error 
was always the same.


Finally I further investigated sources and found a maybe important 
change in file libgnucash/engine/engine.scm made in commit 53f59f7. 
Therefore I tried two builds at the end, one with commit 53f59f7 and the 
second with commit cb50f7a, which is one commit before 53f59f7. The 
result was, that commit 53f59f7 leads to the mentioned build error, but 
commit cb50f7a could successfully be built.


Any ideas?

Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Need help - several tests fail

[Christian Gruber]

Am 22.08.19 um 09:21 schrieb Geert Janssens:

Op donderdag 22 augustus 2019 05:37:12 CEST schreef John Ralls:

On Aug 21, 2019, at 3:28 PM, Christian Gruber 
wrote:

Ok, that was a useful hint. Clearing the guile cache helped. I'm one step
further. I have no idea, what led to this situation. I couldn't reproduce
this again.

Next problem is that still several tests fail with the following output:

ERROR: no code for module (gnucash engine test srfi64-extras)

Which means that they're not finding
libgnucash/engine/test/srfi64-extras.scm or perhaps srfi64-extras.go Can
you figure out why? It might help to run make with VERBOSE=1 so that the
whole command is printed for each step or to switch to ninja which dumps
any failed commands without being asked and is also about 10x faster than
make.

Regards,
John Ralls

It can also be an as of yet undiscovered build dependency issue (there still
are a few on maint).

You didn't post any error output (or full build log) so it's hard to tell.
In the build logs do you find a message logging the build of engine/test/
srfi64-extras.go before you get your errors ?

Geert


I fixed the problem. It is a missing build dependency. I did only call 
"make check" directly after generating a fresh build dir with CMake. So 
not every dependency of the tests has been built yet, scm-srfi64-extras 
is one of them. After calling "make all" and then "make check" again and 
after installing missing french locale, all tests run successfully now. 
Maybe there are even more missing dependencies, which are all built with 
"make all".


Is this a known issue, that the test applications are missing some 
required dependencies in the CMakeLists.txt? I'll have a look, what is 
actually missing and try to fix it.


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Need help - several tests fail

[Christian Gruber]

Am 19.08.19 um 00:54 schrieb John Ralls:



On Aug 18, 2019, at 1:20 PM, Christian Gruber  
wrote:

Hi,

Since I'm new to GnuCash development, I still need some help getting everything 
running. Several tests currently fail on my platform:

   1 - test-exp-parser (Failed)
   2 - test-link-module-app-utils (Failed)
   4 - test-scm-query-string (Failed)
   8 - test-date-utilities (Failed)
   9 - test-options (Failed)
  10 - test-app-utils (Failed)
  15 - test-load-example-account (Failed)
  36 - test-qof (Child aborted)
  59 - test-gnc-numeric (Failed)
  64 - test-scm-query (Failed)
  65 - test-business-core (Failed)
  81 - test-libgnucash-scm-utilities (Failed)
  83 - test-link-module-gnome-utils (Failed)
  85 - test-import-parse (Failed)
  93 - test-qif-imp (Failed)
  94 - test-qif-parse (Failed)
  95 - test-qif-merge-groups (Failed)
  99 - test-invoice (Failed)
 100 - test-owner-report (Failed)
 102 - test-link-module-report-gnome (Failed)
 104 - test-link-module-report-system (Failed)
 106 - test-test-extras (Failed)
 107 - test-commodity-utils (Failed)
 108 - test-report-utilities (Failed)
 109 - test-html-utilities-srfi64 (Failed)
 110 - test-html-fonts (Failed)
 111 - test-report-html (Failed)
 112 - test-report-system (Failed)
 113 - test-standard-category-report (Failed)
 114 - test-standard-net-linechart (Failed)
 115 - test-standard-net-barchart (Failed)
 116 - test-cashflow-barchart (Failed)
 117 - test-charts (Failed)
 118 - test-transaction (Failed)
 119 - test-balsheet-pnl (Failed)
 120 - test-income-gst (Failed)
 121 - test-budget (Failed)
 122 - test-register (Failed)
 123 - test-average-balance (Failed)
 124 - test-stress-options (Failed)
 125 - test-cash-flow (Failed)

At many tests I saw the following or similar output:

1: ;;; WARNING: loading compiled file 
/home/christian/dev/build/gnucash/lib/gnucash/scm/ccache/2.0/fin.go failed:
1: ;;; ERROR: In procedure make_objcode_from_file: bad header on object file: 
"\x7fELF\x02\x01\x01�\x00\x00\x00\x00\x00\x00\x00\x00"
[...]
1: ERROR: In procedure primitive-load-path: Unable to find file "fin" in load 
path

There is always a warning, that some *.go files could not be loaded. Next there is an 
error about "bad header on object file". And at the end there is another error, 
that the file, which could not be loaded before, could not be found in the the load path 
or that there is no code for this module.

Any ideas?

My system is an Ubuntu 19.04.

That's a symptom of stale guile caches. If "make clean && make && make check" 
doesn't resolve the problem go hunting for .go files outside your build dir. ~/.caches/guile is a common 
hiding place. If you have an installed GnuCash make sure that Guile can't find its .go files.

Regards,
John Ralls

Ok, that was a useful hint. Clearing the guile cache helped. I'm one 
step further. I have no idea, what led to this situation. I couldn't 
reproduce this again.


Next problem is that still several tests fail with the following output:

ERROR: no code for module (gnucash engine test srfi64-extras)

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] Need help - several tests fail

[Christian Gruber]

Hi,

Since I'm new to GnuCash development, I still need some help getting 
everything running. Several tests currently fail on my platform:


      1 - test-exp-parser (Failed)
      2 - test-link-module-app-utils (Failed)
      4 - test-scm-query-string (Failed)
      8 - test-date-utilities (Failed)
      9 - test-options (Failed)
     10 - test-app-utils (Failed)
     15 - test-load-example-account (Failed)
     36 - test-qof (Child aborted)
     59 - test-gnc-numeric (Failed)
     64 - test-scm-query (Failed)
     65 - test-business-core (Failed)
     81 - test-libgnucash-scm-utilities (Failed)
     83 - test-link-module-gnome-utils (Failed)
     85 - test-import-parse (Failed)
     93 - test-qif-imp (Failed)
     94 - test-qif-parse (Failed)
     95 - test-qif-merge-groups (Failed)
     99 - test-invoice (Failed)
    100 - test-owner-report (Failed)
    102 - test-link-module-report-gnome (Failed)
    104 - test-link-module-report-system (Failed)
    106 - test-test-extras (Failed)
    107 - test-commodity-utils (Failed)
    108 - test-report-utilities (Failed)
    109 - test-html-utilities-srfi64 (Failed)
    110 - test-html-fonts (Failed)
    111 - test-report-html (Failed)
    112 - test-report-system (Failed)
    113 - test-standard-category-report (Failed)
    114 - test-standard-net-linechart (Failed)
    115 - test-standard-net-barchart (Failed)
    116 - test-cashflow-barchart (Failed)
    117 - test-charts (Failed)
    118 - test-transaction (Failed)
    119 - test-balsheet-pnl (Failed)
    120 - test-income-gst (Failed)
    121 - test-budget (Failed)
    122 - test-register (Failed)
    123 - test-average-balance (Failed)
    124 - test-stress-options (Failed)
    125 - test-cash-flow (Failed)

At many tests I saw the following or similar output:

1: ;;; WARNING: loading compiled file 
/home/christian/dev/build/gnucash/lib/gnucash/scm/ccache/2.0/fin.go failed:
1: ;;; ERROR: In procedure make_objcode_from_file: bad header on object 
file: "\x7fELF\x02\x01\x01�\x00\x00\x00\x00\x00\x00\x00\x00"

[...]
1: ERROR: In procedure primitive-load-path: Unable to find file "fin" in 
load path


There is always a warning, that some *.go files could not be loaded. 
Next there is an error about "bad header on object file". And at the end 
there is another error, that the file, which could not be loaded before, 
could not be found in the the load path or that there is no code for 
this module.


Any ideas?

My system is an Ubuntu 19.04.

Regards,
Christian


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Rework GoogleTest integration

[Christian Gruber]

Am 14.08.19 um 05:09 schrieb John Ralls:



On Aug 13, 2019, at 2:54 PM, Christian Gruber  
wrote:

Am 13.08.19 um 02:45 schrieb John Ralls:

On Aug 12, 2019, at 3:12 PM, Christian Gruber  
wrote:

Following my previous thread "[GNC-dev] Contribute to GnuCash development" I 
opened a new topic thread about reworking GoogleTest integration.


At first some investigation results on bug 797344 
<https://bugs.gnucash.org/show_bug.cgi?id=797344>.

In function gnc_gtest_configure() in file common/cmake_modules/GncAddTest.cmake 
the two CMake variables GTEST_INCLUDE_DIR and GMOCK_INCLUDE_DIR are set as 
follows:

find_path(GTEST_INCLUDE_DIR gtest/gtest.h HINTS ${GTEST_ROOT}/include 
${GMOCK_ROOT}/gtest/include /usr/include)
find_path(GMOCK_INCLUDE_DIR gmock/gmock.h HINTS ${GMOCK_ROOT}/include 
/usr/include)

This means, as long as GTEST_ROOT and GMOCK_ROOT are defined and refer to a 
valid GoogleTest repository, header files gtest.h and gmock.h are found there 
and the two variables GTEST_INCLUDE_DIR and GMOCK_INCLUDE_DIR will refer to the 
include directories within this GoogleTest repository.

In contrast to that the four CMake variables GTEST_MAIN_LIB, GTEST_SHARED_LIB, 
GMOCK_MAIN_LIB and GMOCK_SHARED_LIB are set in the same function 
gnc_gtest_configure() as follows:

find_library(GTEST_MAIN_LIB gtest_main)
find_library(GTEST_SHARED_LIB gtest)
find_library(GMOCK_MAIN_LIB gmock_main)
find_library(GMOCK_SHARED_LIB gmock)

This means, libraries are always searched in the default CMake search paths. 
Therefore any preinstalled GoogleTest libraries will be found this way.

This explains the behaviour described in my bug report.


Now let's come to my ideas regarding rework of GoogleTest integration. After 
further studying CMake files I have two general questions at first.

1. Why are library targets "gtest" and "gmock" defined in GnuCash build system 
instead of importing them from GoogleTest build results?

In file common/test-core/CMakeLists.txt these two targets are defined via 
add_library(). The same is done within GoogleTest build system in files 
googletest/CMakeLists.txt and googlemock/CMakeLists.txt. So part of the CMake 
build system from GoogleTest repository is copied into GnuCash build system.

My idea is to build and install GoogleTest completely independent from GnuCash 
using its own build system and then importing targets via find_package() into 
GnuCash build system. GoogleTest build system provides CMake configuration 
files after installation ready for importing targets into other projects.

CMake function find_package() is able to find package GTest via CMake configuration 
files, i.e. prebuilt from source code repository, as well as preinstalled libraries 
from distro using internal CMake module FindGTest 
<https://cmake.org/cmake/help/v3.5/module/FindGTest.html>. Beginning from CMake 
version 3.5 module FindGTest provides imported targets GTest::GTest and GTest::Main.

This would avoid mixing of libraries and header files from different locations.

Additionally the user doesn't have to define extra environment variables 
GTEST_ROOT and GMOCK_ROOT anymore. Instead the user can configure CMake search 
path using CMake variable CMAKE_PREFIX_PATH, if desired.

And the user can choose, which GoogleTest installation should be used by 
influencing the CMake search path. So the requirement, that one can decide 
whether to use preinstalled GoogleTest libraries from distro or GoogleTest 
installation prebuilt from sources would be fulfilled.


2. Why is library target "gtest" not used directly instead of variables 
GTEST_LIB, GTEST_INCLUDE_DIR?

Several test applications are defined via function gnc_add_test() by passing a 
list of sources, include directories and libraries to that function. This 
function gnc_add_test() is defined in file 
common/cmake_modules/GncAddTest.cmake and passes the list of libraries 
(TEST_LIBS_VAR_NAME) after resolving the variable names to CMake function 
target_link_libraries().

My idea is to add library target "gtest" directly to that list of libraries 
instead of variable GTEST_LIB, which is possible since CMake function 
target_link_libraries() can handle libraries as well as CMake targets. The advantage is, 
that you don't have to handle include directories separately on your own. When passing 
CMake targets to function target_link_libraries() CMake does all the stuff for you. 
Include directories defined for that target are added automatically, i.e. 
target_include_directories() doesn't have to be invoked to add GTEST_INCLUDE_DIR.

A further advantage is, that all test application targets depend on library target "gtest". And if library 
target "gtest" is not an imported target, it would be rebuilt automatically if necessary, when building test 
applications. Currently after any change to GoogleTest sources, e.g. checking out another version, you have to rebuilt 
Googl

Re: [GNC-dev] Contribute to GnuCash development

[Christian Gruber]

Am 12.08.19 um 00:48 schrieb John Ralls:



On Aug 11, 2019, at 3:31 PM, Christian Gruber  
wrote:

Hi,

I'm new on this mailing list. I'm using GnuCash for several months now and 
think, I can also spend some time for GnuCash bugfixing and maybe also 
development later on.

When I started to build GnuCash from source code the first time, I already 
detected two bugs, which I reported on Bugzilla:

* https://bugs.gnucash.org/show_bug.cgi?id=797342
* https://bugs.gnucash.org/show_bug.cgi?id=797344

Maybe I can fix the second bug at first. However I had a deeper look into the 
GnuCash CMake files regarding integration of GoogleTest into the GnuCash build 
system and it looked a little bit strange to me. From my CMake experience I 
would choose a different way. And I would say, that not only this bug should be 
fixed, but the whole GoogleTest integration should be updated.

If you like, we can discuss about that. I have several years of CMake 
experience.

Sure. The only constraint is that in spite of the Googletest recommendation to 
build directly from source for each application some of the Linux distro 
packagers insist that the packaged shared library from their distro must be 
used.

Regards,
John Ralls


Ok, I'll open a new topic thread.
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

[GNC-dev] Contribute to GnuCash development

[Christian Gruber]

Hi,

I'm new on this mailing list. I'm using GnuCash for several months now 
and think, I can also spend some time for GnuCash bugfixing and maybe 
also development later on.


When I started to build GnuCash from source code the first time, I 
already detected two bugs, which I reported on Bugzilla:


* https://bugs.gnucash.org/show_bug.cgi?id=797342
* https://bugs.gnucash.org/show_bug.cgi?id=797344

Maybe I can fix the second bug at first. However I had a deeper look 
into the GnuCash CMake files regarding integration of GoogleTest into 
the GnuCash build system and it looked a little bit strange to me. From 
my CMake experience I would choose a different way. And I would say, 
that not only this bug should be fixed, but the whole GoogleTest 
integration should be updated.


If you like, we can discuss about that. I have several years of CMake 
experience.


Regards,
Christian

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel