Re: [GNC-dev] Import roadmap

[john]

> On Feb 5, 2024, at 03:40, Kevin Buckley via gnucash-devel 
>  wrote:
> 
> On Monday, February 5th, 2024 at 12:36, john  wrote:
>> 
>> Those XML changes sound like the wrong direction.
>> XML code needs to stay in libgnucash/backend, not get mixed in with 
>> engine/QOF,
>> especially since the long-range plan is to turn XML into a backup format
>> with SQL being the primary storage and data access mechanism.
> 
> As someone who's always liked the ability to view/edit the XML (OK, make 
> that any plain text format) might I ask what is meant by "backup format",
> within the context of that plan?
> 
> I'm struggling to think of what use a "backup" file, XML or otherwise,
> that one wouldn't be able to "restore" from, as in with File->Open,
> would be.
> 
> Either an XML file encapsulates/serialises the GNC Book, as it was when 
> it was backed up, or it doesn't, in which case how would such a file be 
> used to do a "restore from backup"?

It would function as a portable alternative to the various SQL implementations' 
backup mechanisms. While in theory a SQL DUMP query should be readable by any 
SQL implementation the reality is that subtle differences in datatypes between 
implementations mean that's not quite true. 

You'd restore with File>Open.

Regards,
John Ralls


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Import roadmap

[Brian Rater]

It needed more thought and experiment, but isolating data structures to the
individual classes would still be the best way to go if they can be
provided with a simple interface to read and write basic data types to/from
whatever our backend structure(s) would be while hiding those details.
Whether or not a simple interface could be created to allow classes to
read/write to a backend with a hidden structure was what I still needed to
look into,  but ran out of time.  There are undoubtedly going to be many
design questions like this moving forward.


On Sun, Feb 4, 2024, 11:37 PM john  wrote:

> Those XML changes sound like the wrong direction. XML code needs to stay
> in libgnucash/backend, not get mixed in with engine/QOF, especially since
> the long-range plan is to turn XML into a backup format with SQL being the
> primary storage and data access mechanism.
>
> I understand the reluctance to cause a regression, but no matter how you
> go about it you're going to be writing new code. Unlike a C -> C++
> conversion you can't do a function at a time with Scheme -> C++, nor can
> you use the same tests for both the old and new code. If you wire into the
> existing C import code you'll at least avoid creating bugs in those parts
> of the process.
>
> Regards,
> John Ralls
>
>
>
>
> On Feb 4, 2024, at 16:50, Brian Rater  wrote:
>
> Great.  Thank you for the insights.  Before I got overwhelmed with work
> and other commitments last year, I was experimenting with a set of base
> classes that would replace the lower level qof infrastructure.  As part of
> that, I was also looking at an alternative xml concept where xml provided
> some fundamental utilities, but each class such as book and transaction
> would use those to read and write themselves.  This keeps the data
> structures contained in the class itself rather than xml w needing to know
> about engine class data structures.  That effort did not progress far
> enough to bring up for discussion here or do a development branch pr,
> however.
>
> I’m comfortable with redesigning and rearchitecting, but I am planning to
> go cautiously with import/QIF as it sounds like a lot people use that and
> the format is not well defined.  I don’t want to introduce new bugs.
>
> On Sun, Feb 4, 2024 at 6:50 PM John Ralls  wrote:
>
>>
>>
>> > On Feb 4, 2024, at 12:11 PM, Brian Rater  wrote:
>> >
>> > I've been looking at the import code and documentation and I'm wondering
>> > what the overall plan and path is.  It looks like there is a "generic"
>> > implementation in the top level directory which has utilities that are
>> > being converted to C++.  I can see the importers making some calls into
>> > this.  What is the plan and where are we at?
>> >
>> > In particular, what needs to happen with QIF, which is still written in
>> > guile?
>> >
>> > I'm asking because I'm looking for a task to work on.  My particular
>> > passion is writing C++ classes to replace legacy code along with adding
>> > tests and documentation.  I've been reading over the source code and
>> > following the development list for several months, but I've had limited
>> > bandwidth until recently to make contributions.
>> >
>> > I've been looking at converting Import/QIF to C++ because it is written
>> in
>> > guile (higher priority than converting the C code in my opinion), is
>> fairly
>> > self contained, and the only alternative guile conversion code is
>> reports,
>> > which is way too large in scope for where I am.  Converting QIF has
>> > drawbacks such as needing extensive testing, and having limited existing
>> > tests for regression testing, however.
>> >
>> > My current thought is to directly translate the existing algorithms and
>> not
>> > use the generic import to minimize introducing bugs, but others have
>> > probably put more thought into how to handle this.
>> >
>> > So I'm looking for some direction either on what the end goal is and
>> how to
>> > achieve it, or direction to leave QIF alone for now and work on some
>> other
>> > area of the codebase (suggestions welcome).
>> >
>>
>> Hi Brian,
>>
>> Welcome to GnuCash.
>>
>> Rewriting the QIF importer to C++ is a reasonable task if that's what
>> motivates you. It's not on the roadmap in more than the most generic "get
>> rid of Guile" sense, but it does seem like a good project to take on. It's
>> as good a legacy conversion project as any at this point. The rest are kind
>> of blocked by the XML->in-memory SQL DB so that we can lose QOFQuery and
>> stop creating huge linked lists of every object in the book. That's
>> design-and-implement rather than rewrite an existing design so maybe not
>> something you'd be passionate about.
>>
>> IMO it would be better to rewrite the QIF parser and transaction creator
>> into C++ and then wire it in to the "generic" infrastructure so that it
>> presents the same user experience as other imports.
>> I'm not sure how well that fits with your passion either, but I'm also
>> not convinced that it 

Re: [GNC-dev] Import roadmap

[Kevin Buckley via gnucash-devel]

On Monday, February 5th, 2024 at 12:36, john  wrote:
> 
> Those XML changes sound like the wrong direction.
> XML code needs to stay in libgnucash/backend, not get mixed in with 
> engine/QOF,
> especially since the long-range plan is to turn XML into a backup format
> with SQL being the primary storage and data access mechanism.

As someone who's always liked the ability to view/edit the XML (OK, make 
that any plain text format) might I ask what is meant by "backup format",
within the context of that plan?

I'm struggling to think of what use a "backup" file, XML or otherwise,
that one wouldn't be able to "restore" from, as in with File->Open,
would be.

Either an XML file encapsulates/serialises the GNC Book, as it was when 
it was backed up, or it doesn't, in which case how would such a file be 
used to do a "restore from backup"?


___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Import roadmap

[john]

Those XML changes sound like the wrong direction. XML code needs to stay in 
libgnucash/backend, not get mixed in with engine/QOF, especially since the 
long-range plan is to turn XML into a backup format with SQL being the primary 
storage and data access mechanism.

I understand the reluctance to cause a regression, but no matter how you go 
about it you're going to be writing new code. Unlike a C -> C++ conversion you 
can't do a function at a time with Scheme -> C++, nor can you use the same 
tests for both the old and new code. If you wire into the existing C import 
code you'll at least avoid creating bugs in those parts of the process.

Regards,
John Ralls




> On Feb 4, 2024, at 16:50, Brian Rater  wrote:
> 
> Great.  Thank you for the insights.  Before I got overwhelmed with work and 
> other commitments last year, I was experimenting with a set of base classes 
> that would replace the lower level qof infrastructure.  As part of that, I 
> was also looking at an alternative xml concept where xml provided some 
> fundamental utilities, but each class such as book and transaction would use 
> those to read and write themselves.  This keeps the data structures contained 
> in the class itself rather than xml w needing to know about engine class data 
> structures.  That effort did not progress far enough to bring up for 
> discussion here or do a development branch pr, however.
> 
> I’m comfortable with redesigning and rearchitecting, but I am planning to go 
> cautiously with import/QIF as it sounds like a lot people use that and the 
> format is not well defined.  I don’t want to introduce new bugs.
> 
> On Sun, Feb 4, 2024 at 6:50 PM John Ralls  > wrote:
>> 
>> 
>> > On Feb 4, 2024, at 12:11 PM, Brian Rater > > > wrote:
>> > 
>> > I've been looking at the import code and documentation and I'm wondering
>> > what the overall plan and path is.  It looks like there is a "generic"
>> > implementation in the top level directory which has utilities that are
>> > being converted to C++.  I can see the importers making some calls into
>> > this.  What is the plan and where are we at?
>> > 
>> > In particular, what needs to happen with QIF, which is still written in
>> > guile?
>> > 
>> > I'm asking because I'm looking for a task to work on.  My particular
>> > passion is writing C++ classes to replace legacy code along with adding
>> > tests and documentation.  I've been reading over the source code and
>> > following the development list for several months, but I've had limited
>> > bandwidth until recently to make contributions.
>> > 
>> > I've been looking at converting Import/QIF to C++ because it is written in
>> > guile (higher priority than converting the C code in my opinion), is fairly
>> > self contained, and the only alternative guile conversion code is reports,
>> > which is way too large in scope for where I am.  Converting QIF has
>> > drawbacks such as needing extensive testing, and having limited existing
>> > tests for regression testing, however.
>> > 
>> > My current thought is to directly translate the existing algorithms and not
>> > use the generic import to minimize introducing bugs, but others have
>> > probably put more thought into how to handle this.
>> > 
>> > So I'm looking for some direction either on what the end goal is and how to
>> > achieve it, or direction to leave QIF alone for now and work on some other
>> > area of the codebase (suggestions welcome).
>> > 
>> 
>> Hi Brian,
>> 
>> Welcome to GnuCash.
>> 
>> Rewriting the QIF importer to C++ is a reasonable task if that's what 
>> motivates you. It's not on the roadmap in more than the most generic "get 
>> rid of Guile" sense, but it does seem like a good project to take on. It's 
>> as good a legacy conversion project as any at this point. The rest are kind 
>> of blocked by the XML->in-memory SQL DB so that we can lose QOFQuery and 
>> stop creating huge linked lists of every object in the book. That's 
>> design-and-implement rather than rewrite an existing design so maybe not 
>> something you'd be passionate about.
>> 
>> IMO it would be better to rewrite the QIF parser and transaction creator 
>> into C++ and then wire it in to the "generic" infrastructure so that it 
>> presents the same user experience as other imports. 
>> I'm not sure how well that fits with your passion either, but I'm also not 
>> convinced that it makes sense to reimplement functional algorithms as OO 
>> classes. I reimplemented the Scheme report options system into C++ a couple 
>> of years ago and it turned out to be a largely start-from-scratch redesign.
>> 
>> Regards,
>> John Ralls
>> 

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Import roadmap

[Brian Rater]

Great.  Thank you for the insights.  Before I got overwhelmed with work and
other commitments last year, I was experimenting with a set of base classes
that would replace the lower level qof infrastructure.  As part of that, I
was also looking at an alternative xml concept where xml provided some
fundamental utilities, but each class such as book and transaction would
use those to read and write themselves.  This keeps the data structures
contained in the class itself rather than xml needing to know about engine
class data structures.  That effort did not progress far enough to bring up
for discussion here or do a development branch pr, however.

I’m comfortable with redesigning and rearchitecting, but I am planning to
go cautiously with import/QIF as it sounds like a lot people use that and
the format is not well defined.  I don’t want to introduce new bugs.

On Sun, Feb 4, 2024 at 6:50 PM John Ralls  wrote:

>
>
> > On Feb 4, 2024, at 12:11 PM, Brian Rater  wrote:
> >
> > I've been looking at the import code and documentation and I'm wondering
> > what the overall plan and path is.  It looks like there is a "generic"
> > implementation in the top level directory which has utilities that are
> > being converted to C++.  I can see the importers making some calls into
> > this.  What is the plan and where are we at?
> >
> > In particular, what needs to happen with QIF, which is still written in
> > guile?
> >
> > I'm asking because I'm looking for a task to work on.  My particular
> > passion is writing C++ classes to replace legacy code along with adding
> > tests and documentation.  I've been reading over the source code and
> > following the development list for several months, but I've had limited
> > bandwidth until recently to make contributions.
> >
> > I've been looking at converting Import/QIF to C++ because it is written
> in
> > guile (higher priority than converting the C code in my opinion), is
> fairly
> > self contained, and the only alternative guile conversion code is
> reports,
> > which is way too large in scope for where I am.  Converting QIF has
> > drawbacks such as needing extensive testing, and having limited existing
> > tests for regression testing, however.
> >
> > My current thought is to directly translate the existing algorithms and
> not
> > use the generic import to minimize introducing bugs, but others have
> > probably put more thought into how to handle this.
> >
> > So I'm looking for some direction either on what the end goal is and how
> to
> > achieve it, or direction to leave QIF alone for now and work on some
> other
> > area of the codebase (suggestions welcome).
> >
>
> Hi Brian,
>
> Welcome to GnuCash.
>
> Rewriting the QIF importer to C++ is a reasonable task if that's what
> motivates you. It's not on the roadmap in more than the most generic "get
> rid of Guile" sense, but it does seem like a good project to take on. It's
> as good a legacy conversion project as any at this point. The rest are kind
> of blocked by the XML->in-memory SQL DB so that we can lose QOFQuery and
> stop creating huge linked lists of every object in the book. That's
> design-and-implement rather than rewrite an existing design so maybe not
> something you'd be passionate about.
>
> IMO it would be better to rewrite the QIF parser and transaction creator
> into C++ and then wire it in to the "generic" infrastructure so that it
> presents the same user experience as other imports.
> I'm not sure how well that fits with your passion either, but I'm also not
> convinced that it makes sense to reimplement functional algorithms as OO
> classes. I reimplemented the Scheme report options system into C++ a couple
> of years ago and it turned out to be a largely start-from-scratch redesign.
>
> Regards,
> John Ralls
>
>
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel

Re: [GNC-dev] Import roadmap

[John Ralls]

> On Feb 4, 2024, at 12:11 PM, Brian Rater  wrote:
> 
> I've been looking at the import code and documentation and I'm wondering
> what the overall plan and path is.  It looks like there is a "generic"
> implementation in the top level directory which has utilities that are
> being converted to C++.  I can see the importers making some calls into
> this.  What is the plan and where are we at?
> 
> In particular, what needs to happen with QIF, which is still written in
> guile?
> 
> I'm asking because I'm looking for a task to work on.  My particular
> passion is writing C++ classes to replace legacy code along with adding
> tests and documentation.  I've been reading over the source code and
> following the development list for several months, but I've had limited
> bandwidth until recently to make contributions.
> 
> I've been looking at converting Import/QIF to C++ because it is written in
> guile (higher priority than converting the C code in my opinion), is fairly
> self contained, and the only alternative guile conversion code is reports,
> which is way too large in scope for where I am.  Converting QIF has
> drawbacks such as needing extensive testing, and having limited existing
> tests for regression testing, however.
> 
> My current thought is to directly translate the existing algorithms and not
> use the generic import to minimize introducing bugs, but others have
> probably put more thought into how to handle this.
> 
> So I'm looking for some direction either on what the end goal is and how to
> achieve it, or direction to leave QIF alone for now and work on some other
> area of the codebase (suggestions welcome).
> 

Hi Brian,

Welcome to GnuCash.

Rewriting the QIF importer to C++ is a reasonable task if that's what motivates 
you. It's not on the roadmap in more than the most generic "get rid of Guile" 
sense, but it does seem like a good project to take on. It's as good a legacy 
conversion project as any at this point. The rest are kind of blocked by the 
XML->in-memory SQL DB so that we can lose QOFQuery and stop creating huge 
linked lists of every object in the book. That's design-and-implement rather 
than rewrite an existing design so maybe not something you'd be passionate 
about.

IMO it would be better to rewrite the QIF parser and transaction creator into 
C++ and then wire it in to the "generic" infrastructure so that it presents the 
same user experience as other imports. 
I'm not sure how well that fits with your passion either, but I'm also not 
convinced that it makes sense to reimplement functional algorithms as OO 
classes. I reimplemented the Scheme report options system into C++ a couple of 
years ago and it turned out to be a largely start-from-scratch redesign.

Regards,
John Ralls

___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel