Re: (AUDIT?) Re: r14892 - gnucash/trunk - Add a new QOF_TYPE_NUMSTRING to add numeric sorts. (#150799).
Quoting Chris Shoemaker [EMAIL PROTECTED]: What about...? 3) Leave sorting out of the type-system altogether and sort only in the view. Save the sort preference in GConf. (BTW, that's how it works in all the GtkTreeModel/View pairs, including on the register-rewrite branch.) That doesn't work with SQL backends when you want to return a subset of the responses. For example, if I wanted to do something like: select * from Split where ... order by ... limit 10; In this case, the order by clause is important in the underlying query.If you don't get the 'order by' correct then you'll get the wrong objects returned, which probably isn't what you want. Either that or you need to full working copy of all your data in core, which can get very expensive in large data sets. -chris -derek -- Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory Member, MIT Student Information Processing Board (SIPB) URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH [EMAIL PROTECTED]PGP key available ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Reporting Errors
== 12 of 12 tests failed Please report to gnucash-devel@gnucash.org == make[7]: *** [check-TESTS] Error 1 make[7]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module/test' make[6]: *** [check-am] Error 2 make[6]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module/test' make[5]: *** [check-recursive] Error 1 make[5]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module/test' make[4]: *** [check-recursive] Error 1 make[4]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module' make[3]: *** [check] Error 2 make[3]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module' make[2]: *** [check-recursive] Error 1 make[2]: Leaving directory `/usr/local/gnucash-2.0.1/src' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/usr/local/gnucash-2.0.1' make: *** [check] Error 2 ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: (AUDIT?) Re: r14892 - gnucash/trunk - Add a new QOF_TYPE_NUMSTRING to add numeric sorts. (#150799).
Quoting Christian Stimming [EMAIL PROTECTED]: Okay. After your additional explanation I understand that either solution required a large invasive change. In that case we should probably just stick to the existing API and use your implementation Well, I think we've already decided to fork the QOF API, so API Stability isn't all that important. Personally, I'm beginning to think that option #2 is better than what I actually implemented.. 1) Add a new type, which keeps all the class definitions the same as they are (except changing TRANS_NUM from QOF_TYPE_STRING to QOF_TYPE_NUMSTRING). which is just what you did in r14892, right? The other solution sounds even more error-prone in terms of the number of places that need changes. Yes. 2) Add a new (optional) QofParam member of type QofSortFunc, modify each and every place where we define QofParam (which is every QofClass definition everywhere in GnuCash), and change the code to prefer a QofParam QofSortFunc over the default type QofSortFunc. I think both approaches are invasive. The former is more QOF API friendly, the latter is probably better in that you can let each parameter define its own sort function. I guess if we had some more object-oriented syntax avaible, it would have made some sense to define the TYPE_NUMSTRING as a kind of TYPE_STRING, i.e. define NUMSTRING as a subtype (inherited class) of STRING. But this is C and we have to stick with what we can write here. So eventually I think your implementation is just fine and I would agree to a back-port, too. *nods* But unfortunately we don't really have a class hierarchy. :( Christian -derek -- Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory Member, MIT Student Information Processing Board (SIPB) URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH [EMAIL PROTECTED]PGP key available ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting Errors
Hi, It would help if you actually reported the errors! It would also help if you said something like, oh, what version of gnucash and what OS/Distro you're using.. -derek Quoting Mark Wade [EMAIL PROTECTED]: == 12 of 12 tests failed Please report to gnucash-devel@gnucash.org == make[7]: *** [check-TESTS] Error 1 make[7]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module/test' make[6]: *** [check-am] Error 2 make[6]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module/test' make[5]: *** [check-recursive] Error 1 make[5]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module/test' make[4]: *** [check-recursive] Error 1 make[4]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module' make[3]: *** [check] Error 2 make[3]: Leaving directory `/usr/local/gnucash-2.0.1/src/gnc-module' make[2]: *** [check-recursive] Error 1 make[2]: Leaving directory `/usr/local/gnucash-2.0.1/src' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/usr/local/gnucash-2.0.1' make: *** [check] Error 2 ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel -- Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory Member, MIT Student Information Processing Board (SIPB) URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH [EMAIL PROTECTED]PGP key available ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: (AUDIT?) Re: r14892 - gnucash/trunk - Add a new QOF_TYPE_NUMSTRING to add numeric sorts. (#150799).
On Tue, Sep 26, 2006 at 09:19:12AM -0400, Derek Atkins wrote: Quoting Chris Shoemaker [EMAIL PROTECTED]: What about...? 3) Leave sorting out of the type-system altogether and sort only in the view. Save the sort preference in GConf. (BTW, that's how it works in all the GtkTreeModel/View pairs, including on the register-rewrite branch.) That doesn't work with SQL backends when you want to return a subset of the responses. Playing the devil's advocate... A) We don't really have a working SQL backend. B) and no one is really working on one. But ignoring that for now... For example, if I wanted to do something like: select * from Split where ... order by ... limit 10; In this case, the order by clause is important in the underlying query.If you don't get the 'order by' correct then you'll get the wrong objects returned, which probably isn't what you want. Well, you get the right objects, just in the wrong order. If the user changes the sort from ascending to descending, do you want to requery the backend with different SQL? Of course not. You just reorder all the objects you already have. This is true for any sorting operation. Either that or you need to full working copy of all your data in core, which can get very expensive in large data sets. By core do you mean L1 data cache or just RAM? Either way, I'm _very_ skeptical of design decisions made with this motivation. Assuming you mean RAM, I would assert that the number of users who: a) would consider using GnuCash and b) have a financial dataset whose in memory (not on disk) representation is even 1/10 of the amount of RAM that came in the machine they want to use for GnuCash is actually zero. Yes, I understand that QOF was designed to handle NASA's multi petabyte image databases. I just think it's unnecessarily burdonsome to perpetuate that design requirement in GnuCash's QOF when it doesn't benefit any GnuCash users. I think it's _especially_ beneficial to drop the our database might be bigger than RAM ideas as we consider options for extending/rewriting QOF in the future. BTW, I don't object to this current changeset, or even backporting it. This is just the way QOF is today. I'm only concerned that we re-evaluate that design decision going forward. Just my $(small number)/$(xaccCommodityGetFraction(comm)) $(gnc_get_default_currency()). -chris ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: (AUDIT?) Re: r14892 - gnucash/trunk - Add a new QOF_TYPE_NUMSTRING to add numeric sorts. (#150799).
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Iff this change is okay for trunk, then IMHO it can very well be
back-ported completely. However, are you sure the problem is solved in
the optimum way by introducing a completely new QOF type? IMHO changes
like these
Index: gnucash/trunk/lib/libqof/qof/qofquery.c
===
--- gnucash/trunk/lib/libqof/qof/qofquery.c (revision 13747)
+++ gnucash/trunk/lib/libqof/qof/qofquery.c (revision 14892)
@@ -1694,5 +1694,6 @@
return;
}
- if (!safe_strcmp (pd-type_name, QOF_TYPE_STRING))
+ if (!safe_strcmp (pd-type_name, QOF_TYPE_STRING) ||
+ !safe_strcmp (pd-type_name, QOF_TYPE_NUMSTRING))
{
query_string_t pdata = (query_string_t) pd;
really suck. I readily admit I don't understand the QOF type system
anyway, but from a naive understanding of the term type I would expect
the sorting order *not* to be part of the definition of a type. And I
would additionally predict that the next time someone needs to write
strcmp(type_name, QOF_TYPE_STRING) somewhere, he/she will almost surely
forget to additionally check for QOF_TYPE_NUMSTRING. Wasn't there any
way to only change the sorting semantics as opposed to introducing a
new type?
Christian
Derek Atkins schrieb:
I don't know if this changeset should get back-ported or not.
What do you all think?
Derek Atkins [EMAIL PROTECTED] writes:
Author: warlord
Date: 2006-09-25 20:36:30 -0400 (Mon, 25 Sep 2006)
New Revision: 14892
Trac: http://svn.gnucash.org/trac/changeset/14892
Modified:
gnucash/trunk/
[snip]
Log:
Add a new QOF_TYPE_NUMSTRING to add numeric sorts. (#150799).
This new type is like QOF_TYPE_STRING but it sorts numerically
(first) and then sorts alphanumerically (by the tail of the
number). Added the QOF Type, the gnome-search support, and
modified TRANS_NUM to use the new type.
-derek
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBRRjcAGXAi+BfhivFAQLKZgP/b6mfqWlOoE3cri4U1jPnkwgFXvdZzFLl
fMk4qU/NjbkD5fZeKnQ/U6ol93fQ/Orb/zd3KxvJ8z+eUR3+LjJAP/MhYAYfkri+
qkyDx5k79uerDkJozyllg7QlqmaTIMGHLqGPPgLn9DqVX4VDQSZ3Zibdq4czklqX
UvTbklM4Mco=
=JOd5
-END PGP SIGNATURE-
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: (AUDIT?) Re: r14892 - gnucash/trunk - Add a new QOF_TYPE_NUMSTRING to add numeric sorts. (#150799).
On Tue, Sep 26, 2006 at 11:26:04AM -0400, Derek Atkins wrote: Quoting Chris Shoemaker [EMAIL PROTECTED]: That doesn't work with SQL backends when you want to return a subset of the responses. Playing the devil's advocate... A) We don't really have a working SQL backend. B) and no one is really working on one. But ignoring that for now... I concede A, but B is certainly in my mind.. If I can gain up the energy I certainly plan to work on it, but probably not in the timeframe for 2.2. For example, if I wanted to do something like: select * from Split where ... order by ... limit 10; In this case, the order by clause is important in the underlying query.If you don't get the 'order by' correct then you'll get the wrong objects returned, which probably isn't what you want. Well, you get the right objects, just in the wrong order. If the user changes the sort from ascending to descending, do you want to requery the backend with different SQL? Of course not. You just reorder all the objects you already have. This is true for any sorting operation. Not really. Assume you have 100 objects in the database, but you want to see the most recent 10 objects. If you only ask SQL for 10 objects, then the 10 objects it returns may not be the 10 objects you want to display unless the 'sort' matches. For example, if the sort is backwards, you might want to see objects 1-10 but it gives you 91-100. Or even worse, if you're sorting on the wrong thing it might give you some random set of the items between 1 and 100. Oh, I missed that limit 10 part. This is really conflating filtering with sorting. Does _GnuCash_ really have a use for filter N? _Even_ if we want to support remote datasets larger than RAM, you already have filtering by where. So, you're describing a case when you don't even want to return full query results! I just don't see this being even remotely possible for personal and small-business accounting software. Now, one approach to work around this is to assume you have regular checkpointing in the database (e.g. when you close the books) and then you always pull in all objects since the last checkpoint. Then you don't have to worry about it, except in the cases where you want to go back in time and see things that happened in the closed-out periods.. Then you just need to pull in periods atomically -- i.e. you always grab a full period of data from the database. Either that or you need to full working copy of all your data in core, which can get very expensive in large data sets. By core do you mean L1 data cache or just RAM? Either way, I'm _very_ skeptical of design decisions made with this motivation. Assuming you mean RAM, I would assert that the number of users who: I'm not thinking about it in terms of CPU cache usage. I'm thinking about it in terms of what's stored in QOF, and what QOF has to do in order to give you results. a) would consider using GnuCash and b) have a financial dataset whose in memory (not on disk) representation is even 1/10 of the amount of RAM that came in the machine they want to use for GnuCash is actually zero. I dunno. Go grab Don Paolo's data set.. 1000 accounts. 100,000 transaction. Well, I figure the on-disk representation is probably 2-4 times larger than the in memory size (totally a guess). So I wouldn't worry unless his datafiles are .5GB. Then tell me that it's okay to have it all in QOF's RAM cache.. I would say it's okay to have it all in RAM, and I don't think it needs any special cache at all. Now imagine going out to 20 years of data, hundreds of thousands of transactions... 10 years, 20 years, 100 years... Datasets grow linearly. RAM doesn't. To find the cross-over point when personal and small-business accounting data approached sizes larger than average RAM, I think we'd have to go back to the 1980s. Wouldn't you rather have a smaller working set? I know I would. From a user's POV, smaller memory requirements traded for increased latency isn't a clear win. From a developer's POV, having uniform access to the whole dataset is a clear benefit. Yes, I understand that QOF was designed to handle NASA's multi petabyte image databases. I just think it's unnecessarily burdonsome to perpetuate that design requirement in GnuCash's QOF when it doesn't benefit any GnuCash users. I wasn't really thinking in those terms... But I do think that requiring QOF to operate on 20 years of data for every operation is sub-optimal. I don't really think of it as QOF operating. I think of it as GnuCash operating. And I think GnuCash should have immediate access to all of the data in a book, even if that's 20 years. Now, book closing is a nice feature, too I think it's _especially_ beneficial to drop the our database might be bigger than RAM ideas as we consider options for extending/rewriting QOF in the future. I disagree... but perhaps
Re: is it possible to refine an existing QofQuery multiple times?
Derek,
by the way, this enhancement request to the QOF API is still valid. I'd love
if a subquery on a qof query were implemented, because then I'd subsequently
be able to speed up the generic importer (hopefully) quite a lot.
Is it possible for you sometime in the future to implement what you've
outlined? Thanks a lot in advance.
Christian
Am Sonntag, 6. August 2006 16:20 schrieb Derek Atkins:
No, there's no way to take the results of a query and run a subquery on it.
HOWEVER, I think adding that functionality might be pretty easy. We could
add something like:
GList* qof_query_run_subquery(QofQuery* subq, QofQuery* q);
Note that running qof_query_run() on the same query multiple times will
re-iterate over the full book, but you can use qof_query_last_run()
to just return the last set.
I think adding this new API would be relatively simple.
What you've provided below will still iterate multiple times. Using
this new API you'd do something like:
...
[ setup query ]
...
(void)qof_query_run(stored_query);
...
loop {
...
[ setup subquery ]
results = qof_query_run_subquery(subq, stored_q);
...
}
-derek
Quoting Christian Stimming [EMAIL PROTECTED]:
Just a question about the qofquery.h interface: Is it possible to refine
a query multiple times?
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: is it possible to refine an existing QofQuery multiple times?
Sure. Once my current tests are finished I'll work on it. It should
be relatively straightforward.
-derek
Quoting Christian Stimming [EMAIL PROTECTED]:
Derek,
by the way, this enhancement request to the QOF API is still valid.
I'd love
if a subquery on a qof query were implemented, because then I'd subsequently
be able to speed up the generic importer (hopefully) quite a lot.
Is it possible for you sometime in the future to implement what you've
outlined? Thanks a lot in advance.
Christian
Am Sonntag, 6. August 2006 16:20 schrieb Derek Atkins:
No, there's no way to take the results of a query and run a subquery on it.
HOWEVER, I think adding that functionality might be pretty easy. We could
add something like:
GList* qof_query_run_subquery(QofQuery* subq, QofQuery* q);
Note that running qof_query_run() on the same query multiple times will
re-iterate over the full book, but you can use qof_query_last_run()
to just return the last set.
I think adding this new API would be relatively simple.
What you've provided below will still iterate multiple times. Using
this new API you'd do something like:
...
[ setup query ]
...
(void)qof_query_run(stored_query);
...
loop {
...
[ setup subquery ]
results = qof_query_run_subquery(subq, stored_q);
...
}
-derek
Quoting Christian Stimming [EMAIL PROTECTED]:
Just a question about the qofquery.h interface: Is it possible to refine
a query multiple times?
--
Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
Member, MIT Student Information Processing Board (SIPB)
URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH
[EMAIL PROTECTED]PGP key available
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Alternate patch to handle the Number Column sort
Here's an attempt to implement choice #2. As you can see
from this patch, it's much simpler. With approval I can
back out r14892 and apply this patch instead. I've tested
that this patch works, and it does (successfully). It turns
out that I do NOT need to edit every QofParam to get it to
work right! :)
Let me know what you think. Do you prefer this patch or the
one I committed earlier?
-derek
=== ChangeLog
==
--- ChangeLog (revision 13354)
+++ ChangeLog (local)
@@ -1,3 +1,20 @@
+2006-09-26 Derek Atkins [EMAIL PROTECTED]
+
+ Add the ability to override the default sort (#150799).
+ Override the Transaction Number to sort numerically.
+ * lib/libqof/qof/qofclass.h:
+ * lib/libqof/qof/qofquery.c:
+ * lib/libqof/qof/qofquerycore-p.h:
+ * lib/libqof/qof/qofquerycore.[ch]:
+ - Publish the QofCompareFunc prototype.
+ - Add a new QofParam param_compfcn parameter
+ - Change QofQuery to use the param_compfcn over the
+ default type compare function.
+ - create (and publish) a qof API to compare strings
+ as numbers: qof_string_number_compare_func()
+ * src/engine/Transaction.c:
+ Assign TRANS_NUM to use qof_string_numer_compare_func()
+
2006-09-23 Derek Atkins [EMAIL PROTECTED]
* src/ledger-core/split-register*.c:
=== lib/libqof/qof/qofclass.h
==
--- lib/libqof/qof/qofclass.h (revision 13354)
+++ lib/libqof/qof/qofclass.h (local)
@@ -151,6 +151,14 @@
*/
typedef void (*QofSetterFunc) (gpointer, gpointer);
+/* A callback for how to compare two (same-type) objects based on a
+ * common getter (parameter member), using the provided comparison
+ * options (which are the type-specific options).
+ */
+typedef gint (*QofCompareFunc) (gpointer a, gpointer b,
+gint compare_options,
+QofParam *getter);
+
/** This structure is for each queriable parameter in an object
*
* -- param_name is the name of the parameter.
@@ -174,6 +182,7 @@
QofTypeparam_type;
QofAccessFunc param_getfcn;
QofSetterFunc param_setfcn;
+ QofCompareFunc param_compfcn;
gpointer param_userdata;
};
=== lib/libqof/qof/qofquery.c
==
--- lib/libqof/qof/qofquery.c (revision 13354)
+++ lib/libqof/qof/qofquery.c (local)
@@ -473,13 +473,17 @@
*/
if (sort-param_fcns)
{
-sort-comp_fcn = qof_query_core_get_compare (resObj-param_type);
+/* First, check if this parameter has a sort function override.
+ * if not then check if there's a global compare function for the type
+ */
+if (resObj-param_compfcn)
+ sort-comp_fcn = resObj-param_compfcn;
+else
+ sort-comp_fcn = qof_query_core_get_compare (resObj-param_type);
-/* Hrm, perhaps this is an object compare, not a core compare? */
+/* Next, perhaps this is an object compare, not a core type compare? */
if (sort-comp_fcn == NULL)
-{
sort-obj_cmp = qof_class_get_default_sort (resObj-param_type);
-}
}
else if (!safe_strcmp (sort-param_list-data, QUERY_DEFAULT_SORT))
{
=== lib/libqof/qof/qofquerycore-p.h
==
--- lib/libqof/qof/qofquerycore-p.h (revision 13354)
+++ lib/libqof/qof/qofquerycore-p.h (local)
@@ -45,14 +45,6 @@
QofParam *getter,
QofQueryPredData *pdata);
-/* A callback for how to compare two (same-type) objects based on a
- * common getter (parameter member), using the provided comparison
- * options (which are the type-specific options).
- */
-typedef gint (*QofCompareFunc) (gpointer a, gpointer b,
- gint compare_options,
- QofParam *getter);
-
/* Lookup functions */
QofQueryPredicateFunc qof_query_core_get_predicate (gchar const *type);
QofCompareFunc qof_query_core_get_compare (gchar const *type);
=== lib/libqof/qof/qofquerycore.c
==
--- lib/libqof/qof/qofquerycore.c (revision 13354)
+++ lib/libqof/qof/qofquerycore.c (local)
@@ -24,6 +24,7 @@
#include config.h
#include glib.h
+#include stdlib.h
#include qof.h
#include qofquerycore-p.h
@@ -182,6 +183,36 @@
return safe_strcmp (s1, s2);
}
+int
+qof_string_number_compare_func (gpointer a, gpointer b, gint options,
+QofParam *getter)
+{
+ const char *s1, *s2;
+ char *sr1, *sr2;
+ long i1, i2;
+ g_return_val_if_fail (a b getter getter-param_getfcn, COMPARE_ERROR);
+
+ s1 = ((query_string_getter)getter-param_getfcn) (a, getter);
+ s2 = ((query_string_getter)getter-param_getfcn) (b, getter);
+
+ // Deal with NULL strings
+ if (s1 == s2) return 0;
+ if (!s1 s2) return -1;
+ if (s1 !s2) return 1;
+
+ // Convert to integers and test
+ i1 = strtol(s1, sr1, 0);
+ i2 =
Re: Alternate patch to handle the Number Column sort
On Tue, Sep 26, 2006 at 07:20:11PM -0400, Derek Atkins wrote: Here's an attempt to implement choice #2. As you can see from this patch, it's much simpler. With approval I can back out r14892 and apply this patch instead. I've tested that this patch works, and it does (successfully). It turns out that I do NOT need to edit every QofParam to get it to work right! :) Let me know what you think. Do you prefer this patch or the one I committed earlier? I think this one is much better. -chris ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
