Re: Report wrapper to call python scripts?
Hello, Having read this thread I wondered if there could be some sort of interface to export data from gnucash so that any kind of external report generator could work on it. Exporting the selected data to i.e. an xml file. The external program would then handle the data it's way and could be distributed separately. To ease such process something like a plugin-system for gnucash could be introduced. Having the option to insert a button to start this program or preferences dialog in the gui. regards, Christoph Holtermann Am 24.12.2014 um 12:25 schrieb Geert Janssens: On Wednesday 24 December 2014 12:22:46 Dmitry Pavlov wrote: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)). The report system is indeed in dire need of some attention. The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Absolutely. That's why a couple of years back one developer started to rewrite the reports based on eguile which is a templating system written in guile. The developer ran out of time so only a few reports have been written based on eguile: there is a balance sheet and a tax invoice report. And I have just recently added a payment receipt report based on eguile as well. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. Yes. That's one reason it hasn't happened yet. Other reasons are that the active developers are mostly focused on the core rewrite in c++ making the core a moving target. That's probably not the best moment to rewrite a report system. The current report system - while very outdated - is rather stable and well understood. That makes it easier to keep working on top of a heavily changing core. Yet another reason why I personally have been holding off of rewriting the report system (other than insufficient time) is that the developers intend to move towards a more sql driven data model. If that succeeds the report system could perhaps be replaced with one of the many existing sql based reporting systems. So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). That could be an interesting approach. However it's important to keep in mind that gnucash supports multiple platforms: linux, Windows and OS X. If you want to write reports intended to be included with gnucash and in a language other than guile that should be a language that is supported on all three platforms. You mention python. While there are versions of python for all of them, there are no python bindings for gnucash on Windows and OS X. There have been several attempts to get it to work on Windows, but no success just yet. The same goes for OS X. There is no reliable way yet to install working python bindings on all supported versions of OS X. This is mostly due python's own incompatibilities between python versions. Next if at some point the python bindings are working on all platforms, there is the distribution issue: if we start to depend on python for some features, we would need to ship python with our Windows installer. That would mean a considerable increase in size. That is only justified if the functionality would be greatly enhanced by this. So as things *currently* stand python is IMO not an option (yet) to replace the reporting system. Sorry about that. The only languages we have natively available on all supported platforms are c, c++, guile and javascript (via webkit). For guile there are bindings to many of the c libraries. For javascript there aren't (yet?). In it's settings we can point to specific script and all guile invocation would just 1. include execution of that script with passing parameters from options If I understand you correctly you want to separate the options from the report generating code ? So your wrapper script would be responsible for displaying the options to the user and the actual report script only gets the values passed in ? The way I understand the code that will already be a big work because the options for each report are also defined in the same report scheme file. And the whole options handling code is 90% scheme code. The options themselves live in the guile context, not in C. Displaying the options dialog is about 70% guile code which expects the options to live in guile. (I happened to look at that code flow yesterday,
Re: Report wrapper to call python scripts?
On Jan 3, 2015, at 8:35 AM, Christoph Holtermann c.holterm...@gmx.de wrote: Am 03.01.2015 um 16:37 schrieb John Ralls: On Jan 3, 2015, at 2:44 AM, Christoph Holtermann c.holterm...@gmx.de wrote: Hello, Having read this thread I wondered if there could be some sort of interface to export data from gnucash so that any kind of external report generator could work on it. Exporting the selected data to i.e. an xml file. The external program would then handle the data it's way and could be distributed separately. To ease such process something like a plugin-system for gnucash could be introduced. Having the option to insert a button to start this program or preferences dialog in the gui. The simplest export is to use the SQL backend. There are a variety of report generators available for SQL databases. Since the next release of GnuCash will be SQL-centric (replacing QOF for queries and providing record-level locking for simultaneous multi-user access), I think this is the way to go. I don't really understand this yet. Is the idea to give other applications access to the main SQL database itself ? There's no good way to stop them. ;-) It's not why we're doing it, but it's a useful side effect. Several users are already using the SQL database for custom reports, and Wm has already posted a couple of how-tos to the user list with the promise of more to follow. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Jan 3, 2015, at 2:44 AM, Christoph Holtermann c.holterm...@gmx.de wrote: Hello, Having read this thread I wondered if there could be some sort of interface to export data from gnucash so that any kind of external report generator could work on it. Exporting the selected data to i.e. an xml file. The external program would then handle the data it's way and could be distributed separately. To ease such process something like a plugin-system for gnucash could be introduced. Having the option to insert a button to start this program or preferences dialog in the gui. The simplest export is to use the SQL backend. There are a variety of report generators available for SQL databases. Since the next release of GnuCash will be SQL-centric (replacing QOF for queries and providing record-level locking for simultaneous multi-user access), I think this is the way to go. GnuCash already has a plugin system, though it's a bit cumbersome. You can find some examples in src/experimental. Regards, John RAlls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
Am 03.01.2015 um 16:37 schrieb John Ralls: On Jan 3, 2015, at 2:44 AM, Christoph Holtermann c.holterm...@gmx.de wrote: Hello, Having read this thread I wondered if there could be some sort of interface to export data from gnucash so that any kind of external report generator could work on it. Exporting the selected data to i.e. an xml file. The external program would then handle the data it's way and could be distributed separately. To ease such process something like a plugin-system for gnucash could be introduced. Having the option to insert a button to start this program or preferences dialog in the gui. The simplest export is to use the SQL backend. There are a variety of report generators available for SQL databases. Since the next release of GnuCash will be SQL-centric (replacing QOF for queries and providing record-level locking for simultaneous multi-user access), I think this is the way to go. I don't really understand this yet. Is the idea to give other applications access to the main SQL database itself ? GnuCash already has a plugin system, though it's a bit cumbersome. You can find some examples in src/experimental. nice. I'll have a look at it. Regards, John RAlls regards, Christroph Holterman -- --- Nachricht gesendet von C. Holtermann --- - - - Verschlüsselte Nachrichten können über - - den öffentlichen Schlüssel auf folgendem - - Keyserver an mich gesendet werden: - http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x4DD9CF0482B0620B ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Jan 1, 2015, at 2:52 PM, Sébastien de Menten sdemen...@gmail.com wrote: Hello John, I went to explore a bit this path to see its feasibility. I came with a concrete example of the idea with this single exe (https://github.com/sdementen/piecash/blob/master/piecash_interpreter/piecash_interpreter.exe?raw=true) generated by pyinstaller that takes as argument a python file (and any extra argument) and run it through python 2.7. In this example exe, the piecash module is included so that the piecash_ledger script (https://github.com/sdementen/piecash/blob/master/piecash_interpreter/piecash_interpreter.exe?raw=true) can be run with: piecash_interpreter.exe piecash_ledger.py mybook.gnucash Could this be also an option for the official python bindings to ease their installation on windows (and maybe OS X but I have not OS X machine...) ? And as possible interface with gnucash and the official python bindings (or any other bindgins/python executable like piecash ;-) ) sebastien On Sun, Dec 28, 2014 at 3:59 PM, John Ralls jra...@ceridwen.us wrote: On Dec 27, 2014, at 11:54 PM, Sébastien de Menten sdemen...@gmail.com wrote: Just a thought regarding the need for a python distribution for the python binding on Windows/OS X, would it be an option to build a single executable with the gnucash bindings (see http://www.orbitals.com/programs/pyexe.html or http://www.decalage.info/en/python/py2exe) ? This would give a complete control on the required python version/package distribution. Those solutions are for distributing single applications written in Python. They wouldn't do any good for python bindings, where the user supplies code. For that we'd have to bundle the entire Python distribution. Because of the constraints of linking to a particular libpython on OSX -- the interpreter and bindings must link to the same libpython, and different versions of OSX provide different versions of python, were in the same boat there. We'd need to distribute a complete Python installation in the GnuCash bundle, and generally users would have to use the python interpreter we would ship. And if the user is more knowledgeable re python, it could go with its own distribution (+ other relevant comment in this thread) That would require somehow coercing the packages shipped with GnuCash to link the library that the interpreter is using. That's not something the typical Python programmer thinks much about. No. The python bindings are part of GnuCash, and GnuCash isn’t a python program, so neither pyexe nor py2exe will work. They work with piecash because it’s pure python. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
Hello John, I went to explore a bit this path to see its feasibility. I came with a concrete example of the idea with this single exe ( https://github.com/sdementen/piecash/blob/master/piecash_interpreter/piecash_interpreter.exe?raw=true) generated by pyinstaller that takes as argument a python file (and any extra argument) and run it through python 2.7. In this example exe, the piecash module is included so that the piecash_ledger script ( https://github.com/sdementen/piecash/blob/master/piecash_interpreter/piecash_interpreter.exe?raw=true) can be run with: piecash_interpreter.exe piecash_ledger.py mybook.gnucash Could this be also an option for the official python bindings to ease their installation on windows (and maybe OS X but I have not OS X machine...) ? And as possible interface with gnucash and the official python bindings (or any other bindgins/python executable like piecash ;-) ) sebastien On Sun, Dec 28, 2014 at 3:59 PM, John Ralls jra...@ceridwen.us wrote: On Dec 27, 2014, at 11:54 PM, Sébastien de Menten sdemen...@gmail.com wrote: Just a thought regarding the need for a python distribution for the python binding on Windows/OS X, would it be an option to build a single executable with the gnucash bindings (see http://www.orbitals.com/programs/pyexe.html or http://www.decalage.info/en/python/py2exe) ? This would give a complete control on the required python version/package distribution. Those solutions are for distributing single applications written in Python. They wouldn't do any good for python bindings, where the user supplies code. For that we'd have to bundle the entire Python distribution. Because of the constraints of linking to a particular libpython on OSX -- the interpreter and bindings must link to the same libpython, and different versions of OSX provide different versions of python, were in the same boat there. We'd need to distribute a complete Python installation in the GnuCash bundle, and generally users would have to use the python interpreter we would ship. And if the user is more knowledgeable re python, it could go with its own distribution (+ other relevant comment in this thread) That would require somehow coercing the packages shipped with GnuCash to link the library that the interpreter is using. That's not something the typical Python programmer thinks much about. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Dec 27, 2014, at 11:54 PM, Sébastien de Menten sdemen...@gmail.com wrote: Just a thought regarding the need for a python distribution for the python binding on Windows/OS X, would it be an option to build a single executable with the gnucash bindings (see http://www.orbitals.com/programs/pyexe.html or http://www.decalage.info/en/python/py2exe) ? This would give a complete control on the required python version/package distribution. Those solutions are for distributing single applications written in Python. They wouldn't do any good for python bindings, where the user supplies code. For that we'd have to bundle the entire Python distribution. Because of the constraints of linking to a particular libpython on OSX -- the interpreter and bindings must link to the same libpython, and different versions of OSX provide different versions of python, were in the same boat there. We'd need to distribute a complete Python installation in the GnuCash bundle, and generally users would have to use the python interpreter we would ship. And if the user is more knowledgeable re python, it could go with its own distribution (+ other relevant comment in this thread) That would require somehow coercing the packages shipped with GnuCash to link the library that the interpreter is using. That's not something the typical Python programmer thinks much about. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
Just a thought regarding the need for a python distribution for the python binding on Windows/OS X, would it be an option to build a single executable with the gnucash bindings (see http://www.orbitals.com/programs/pyexe.html or http://www.decalage.info/en/python/py2exe) ? This would give a complete control on the required python version/package distribution. And if the user is more knowledgeable re python, it could go with its own distribution (+ other relevant comment in this thread) ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
Hello, using python and jinjaII templating as I tried for invoices could be an interesting thing (just for linux). For reports SciPy would also be tempting. regards, Christoph Holtermann Am 24.12.2014 um 12:49 schrieb Dmitry Pavlov: Geert, thank you for a very detailed comments. I did't know that situation with python bindings is so complicated on Windows and OSX, and that can be a real show stopper. For the details of implementations my idea was slightly different: options can be preserved for guile, as options definitions is really the simplest part of any current report. The python part was planned to only include report generation and guille part should just pass options to that script, maybe as command line options or file with that parameters (exact way is implemtation details, really). While I was working on budget report enhancement I've looked at eguile reports: 1. they are still using scheme to implement report logic, and changes can be painfull :) 2. template part is full of small scriptlets (scheme code insertions) and at least it's worth to make some taglib (sorry for that java world term) so that common formatting and type conversion failures can be hidden for template authors by simple formatting constructs. It seems that until python problems can't be solved such report implementation can only be a local linux version feature, not included in distro. Anyway if some linux user/developer want to participate, I'm ready :). As for more global change, personally me not sure to find enough time and scheme knowledge to rewrite many reports at least to eguile, but if I'll manage to find time, I'll let you know. Thank you all for comments! 2014-12-24 14:25 GMT+03:00 Geert Janssens geert.gnuc...@kobaltwit.be: On Wednesday 24 December 2014 12:22:46 Dmitry Pavlov wrote: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)). The report system is indeed in dire need of some attention. The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Absolutely. That's why a couple of years back one developer started to rewrite the reports based on eguile which is a templating system written in guile. The developer ran out of time so only a few reports have been written based on eguile: there is a balance sheet and a tax invoice report. And I have just recently added a payment receipt report based on eguile as well. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. Yes. That's one reason it hasn't happened yet. Other reasons are that the active developers are mostly focused on the core rewrite in c++ making the core a moving target. That's probably not the best moment to rewrite a report system. The current report system - while very outdated - is rather stable and well understood. That makes it easier to keep working on top of a heavily changing core. Yet another reason why I personally have been holding off of rewriting the report system (other than insufficient time) is that the developers intend to move towards a more sql driven data model. If that succeeds the report system could perhaps be replaced with one of the many existing sql based reporting systems. So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). That could be an interesting approach. However it's important to keep in mind that gnucash supports multiple platforms: linux, Windows and OS X. If you want to write reports intended to be included with gnucash and in a language other than guile that should be a language that is supported on all three platforms. You mention python. While there are versions of python for all of them, there are no python bindings for gnucash on Windows and OS X. There have been several attempts to get it to work on Windows, but no success just yet. The same goes for OS X. There is no reliable way yet to install working python bindings on all supported versions of OS X. This is mostly due python's own incompatibilities between python versions. Next if at some point the python bindings are working on all platforms, there is the distribution issue: if we start to depend on python for some features, we would need to ship python with our Windows installer. That would mean a considerable increase in size. That is only justified if the functionality would be greatly enhanced by this. So as
Re: Report wrapper to call python scripts?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, Am 24.12.2014 um 10:20 schrieb Marcus Wellnitz: Hello Dimity, we discussed that at the german mailinglist more than once ;-) I have made solution wirth python-bindings and latex for my invoices and the output is much better. https://github.com/mwellnitz/gnucash-latex/ https://github.com/mwellnitz/gnucash-latex/wiki Actually I have a problem with the python-bindings (Error) but that's maybe only at my system. Texts are in german, sorry :-( so maybe It's not so helpful for you? Also Chistoph made his solution: https://github.com/c-holtermann/gnucash/tree/python-kvp this is temporary. KVP-access via Python is not intended by the core developers. I just wanted to access company data this way. I'm trying to get access by creating a company object in https://github.com/c-holtermann/gnucash/tree/Company Still needs a lot of work. latex invoices using jinja2 templating system have been accepted for master. It's gncinvoice_jinja.py which works fine for me. And I hope to find some time to join Marcus and my approaches. Marcus - why don't you create a fork of gnucash ? I think that way it would be easier to compare our versions of files like latex_invoices.py. Or is it maybe possible in git to have a fork of just one directory ? Maybe the example-scripts directory ? Or is it even possible to just link in one file ? As time is limited it's impossible for us to improve or even merge our solutions but if the gnucash core-team will support any new solution we will participate in that project. I wish you a happy christmas Marcus regards and merry christmas Christoph Am 24.12.2014 um 10:22 schrieb Dmitry Pavlov: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)) . The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). In it's settings we can point to specific script and all guile invocation would just 1. include execution of that script with passing parameters from options 2. grab output that is supposed to be report content (html for now) and include that as it's own result In that case we can have one more language to implement reports, because scheme is not so popular now, and many people find it not so easy to use, especially when we are talking about reporting :) I'm not sure that I can implement all that stuff myself, but if someone find that idea good enough I'll be glad to discuss that and collaborate to implement that wrapper script. - -- - --- Nachricht gesendet von C. Holtermann --- - - - - - Verschlüsselte Nachrichten können über - - - den öffentlichen Schlüssel auf folgendem - - - Keyserver an mich gesendet werden: - http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x4DD9CF0482B0620B -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlScQmsACgkQTdnPBIKwYgv8RACgkZIInY0oxRlzIxlmWDQg9gwE G4QAn3dY6MNDFNmPtjAdUXjTtlJQuSA1 =HhGN -END PGP SIGNATURE- ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello Dimity, we discussed that at the german mailinglist more than once ;-) I have made solution wirth python-bindings and latex for my invoices and the output is much better. https://github.com/mwellnitz/gnucash-latex/ https://github.com/mwellnitz/gnucash-latex/wiki Actually I have a problem with the python-bindings (Error) but that's maybe only at my system. Texts are in german, sorry :-( so maybe It's not so helpful for you? Also Chistoph made his solution: https://github.com/c-holtermann/gnucash/tree/python-kvp As time is limited it's impossible for us to improve or even merge our solutions but if the gnucash core-team will support any new solution we will participate in that project. I wish you a happy christmas Marcus Am 24.12.2014 um 10:22 schrieb Dmitry Pavlov: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)) . The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). In it's settings we can point to specific script and all guile invocation would just 1. include execution of that script with passing parameters from options 2. grab output that is supposed to be report content (html for now) and include that as it's own result In that case we can have one more language to implement reports, because scheme is not so popular now, and many people find it not so easy to use, especially when we are talking about reporting :) I'm not sure that I can implement all that stuff myself, but if someone find that idea good enough I'll be glad to discuss that and collaborate to implement that wrapper script. - -- Marcus Wellnitz GPG Fingerprint: 07D4 C8D7 F5FB CC00 EDF1 0FCB DEED 2417 9BF6 4022 -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJUmoV6AAoJEN7tJBeb9kAi0ioP/ApHGPQ9qdM98mR610UBuTmc AomzNgXDwVOYHsmIz7GuOFm/Xn9zLKOOQmZchoE5U8/Zt0NkROwVPcWlxdW6PUG4 f9th10UJfcVW/IMsCk2MtL8kZSSNKDZFWPFYeokyvodZDhmOeuMIr3wr0H1z2bhz cXUqAcMi4sARgo3wPeG+nb4RR4K03fUPkwUe6yu5pbBFXKfRLp19VLkAXuYsgr2I eTLLZswBRwK9P3kmtUXE6YF0qaP+g2FjYfKtvc6skwfhX+S1cmbnI9HJ9zLVYSs0 vIPTHDCG2vusNiMpJO5Il1fFYYCSC33IaEQ3n2bnBV+9yCBC4xKzGmXUSBsMfWCt pvP+RYVvsE0S/f0Ob5nqyq70fWr+qUfS43ugSH01zppkpHZMWBH6B2CRRnaPYKgd z/qJaQtJzQBMJneFEWLMCJTWqLWU3SKDRplmmHdiGOUZluEUKY1bzs9x3BGOSmEB o6LwF57Y1R3zgO0u+1obfRVzWaAo2YN4g7Lgml0OL8XT91IgInD0UsRPKWObW1yH DW/AawZ+Cja7geiryGewJaGw8aaMB4VID+bzr3uJa4YHD/FR7NasS8WMvevtlPZn vq/nL5Ns7dVhXBTk5AOdCUR6W1UFF9LzD2ZaCR8GSS7+wKUe5UjSNDEZkGEXDSva TWUXhJZZC+yr3zjjiuxh =CSPU -END PGP SIGNATURE- ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Wednesday 24 December 2014 12:22:46 Dmitry Pavlov wrote: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)). The report system is indeed in dire need of some attention. The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Absolutely. That's why a couple of years back one developer started to rewrite the reports based on eguile which is a templating system written in guile. The developer ran out of time so only a few reports have been written based on eguile: there is a balance sheet and a tax invoice report. And I have just recently added a payment receipt report based on eguile as well. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. Yes. That's one reason it hasn't happened yet. Other reasons are that the active developers are mostly focused on the core rewrite in c++ making the core a moving target. That's probably not the best moment to rewrite a report system. The current report system - while very outdated - is rather stable and well understood. That makes it easier to keep working on top of a heavily changing core. Yet another reason why I personally have been holding off of rewriting the report system (other than insufficient time) is that the developers intend to move towards a more sql driven data model. If that succeeds the report system could perhaps be replaced with one of the many existing sql based reporting systems. So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). That could be an interesting approach. However it's important to keep in mind that gnucash supports multiple platforms: linux, Windows and OS X. If you want to write reports intended to be included with gnucash and in a language other than guile that should be a language that is supported on all three platforms. You mention python. While there are versions of python for all of them, there are no python bindings for gnucash on Windows and OS X. There have been several attempts to get it to work on Windows, but no success just yet. The same goes for OS X. There is no reliable way yet to install working python bindings on all supported versions of OS X. This is mostly due python's own incompatibilities between python versions. Next if at some point the python bindings are working on all platforms, there is the distribution issue: if we start to depend on python for some features, we would need to ship python with our Windows installer. That would mean a considerable increase in size. That is only justified if the functionality would be greatly enhanced by this. So as things *currently* stand python is IMO not an option (yet) to replace the reporting system. Sorry about that. The only languages we have natively available on all supported platforms are c, c++, guile and javascript (via webkit). For guile there are bindings to many of the c libraries. For javascript there aren't (yet?). In it's settings we can point to specific script and all guile invocation would just 1. include execution of that script with passing parameters from options If I understand you correctly you want to separate the options from the report generating code ? So your wrapper script would be responsible for displaying the options to the user and the actual report script only gets the values passed in ? The way I understand the code that will already be a big work because the options for each report are also defined in the same report scheme file. And the whole options handling code is 90% scheme code. The options themselves live in the guile context, not in C. Displaying the options dialog is about 70% guile code which expects the options to live in guile. (I happened to look at that code flow yesterday, that's why I know). So making this part language agnostic means rewriting the options code. Which happens to be something I *am* considering if I find the time :) 2. grab output that is supposed to be report content (html for now) and include that as it's own result If I understand things correctly this second part is what is done now already. The guile reports generate an html output file that is then parsed by the integrated webkit engine. In that case we can have one more language to implement reports, because scheme is not so popular now, and many people find it not so easy to use, especially when we are talking
Re: Report wrapper to call python scripts?
Can you send link to your git repo with that changes? 24 дек. 2014 г. 18:39 пользователь David Osguthorpe david.osgutho...@gmail.com написал: On Wed, Dec 24, 2014 at 12:25:23PM +0100, Geert Janssens wrote: If I understand you correctly you want to separate the options from the report generating code ? So your wrapper script would be responsible for displaying the options to the user and the actual report script only gets the values passed in ? The way I understand the code that will already be a big work because the options for each report are also defined in the same report scheme file. And the whole options handling code is 90% scheme code. The options themselves live in the guile context, not in C. Displaying the options dialog is about 70% guile code which expects the options to live in guile. (I happened to look at that code flow yesterday, that's why I know). I have been following the various conversations about python - something I am more than interested in. (I did try to allow python calls from guile using a guile module I found by googling - and returning data to guile using another such module - but had issues with random crashes so gave that up - the modules were also not being maintained). However I then found the python module and used that as the basis so that I now have the ability to write reports in python The only change I made to gnucash was in the python module init script to call another python script stored in a users .gnucash directory under a python directory - all python and compiled python modules are stored there (ie no need for distribution in gnucash). There are 5 compiled modules currently based on pre-gobject introspection codegen (as used for eg pygtk), pygkeyfile (as apparently there are no distributed GKeyFile python bindings), python bindings generated by codegen for GncPluginPage (for subclassing in python to create report pages), GncHtml and GncHtmlWebkit - for final report output. there is also a swighelper python module for moving objects between python ctypes and swig instances (python ctypes is also used to access some functionality/data). These are setup for compiling in a gnucash source distribution - currently in a plugin directory under the python module. The rest is all python - and yes this does mean I re-implemented options in python - although in a JIT way so options I have not needed are not implemented. GnuCash objects are accessed using the distributed python binding classes. This needs pygtk installed in order to insert into GnuCashs Gtk based GUI - but the python code is able to install additional menus for the python reports. (By the way note that GnuCash is multithreaded because Gtk is multi-threaded - in the initial implementation attempts I had issues with python GIL crashes but these dissappeared when I started using proper pygobject based implementations of GnuCashs GTypes). So far I have the basic hello world report in python (with all options implemented that it uses), cash_flow and a multi-commodity price_scatter (which is what started all this - around gnucash 1.8 I had a multi-commodity price scatter plot in scheme (the distributed version only did 1 commodity) which failed (as I know now) when the plotting switched to Javascript based graphs - the python reports are able to use the Javascript plotting features. The HTML is generated using the python ElementTree library - all scheme ignored here although there is a partial implementation of stylesheets - the access to GncHtmlWebkit is used for the final display of the HTML string. I have also ignored the KVP code for options - note that options in the python reports do not apply to the scheme reports - there is no cross-over. This is all done under OSX (Snow Leopard) using MacPorts as the primary installer of GnuCash, python 2.7, pygtk etc. - although I have had a small look at working under Linux - my laptop dual boots OSX/Linux. The code is in a git repository which can be available if there is interest - although it contains a lot of history as tried differing approaches. David ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Wed, Dec 24, 2014 at 08:07:45PM +0400, Dmitry Pavlov wrote: Can you send link to your git repo with that changes? Supposedly this is it https://github.com/davidjo/gnucash_python.git David ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Dec 24, 2014, at 7:38 AM, David Osguthorpe david.osgutho...@gmail.com wrote: (By the way note that GnuCash is multithreaded because Gtk is multi-threaded - in the initial implementation attempts I had issues with python GIL crashes but these dissappeared when I started using proper pygobject based implementations of GnuCashs GTypes). No, Gtk supports use in a multi-threaded environment, but is not itself multithreaded. In particular, Gtk requires that all calls to Gtk functions occur in a single thread, and on OSX (and perhaps Windows) that thread must be the main thread, the one in which the g_main_loop is executing. Worker threads should use idle events to coordinate with the Gtk thread. All of the main parts of GnuCash also operate in that one thread. WebKit, however, is multi-threaded. As long as you just hand it HTML and Javascript to render or execute it takes care of everything itself including posting events back to Gtk via the GtkWidget you give it to draw on. That will likely change in the course of the C++11 rewrite, though probably not this development cycle: We first need to have code that's threadable. The current code is too laden with static variables for that to be possible. I'm frankly amazed that GnuCash's malformed GObjects work with PyGObject considering that they generally ignore GObject's memory management and are directly allocated and freed. Regardless, the C++ rewrite will break all of that because we're removing all GLib dependencies from everything except the GUI. OTOH, since we'll be using C++11 std::shared_ptr and std::unique_ptr for memory management the resulting Python bindings should in the end be a lot more stable than what we have now, although the SWIG folks still have some work to do to fully assimilate C++11 [1]. Another potential problem for your addon: WebKit keeps getting more bloated and harder to build for Windows and OSX; in both cases we're using rather old versions (1.8.3 and 1.6.2 respectively) because of the difficulty of building newer versions on those platforms. We're looking for a lighter-weight replacement. If you're just making HTML strings and handing them to WebKit, no problem, but if you're interacting directly with the WebKitGtk API you'll have to rewrite that as well. Regards, John Ralls [1] http://www.swig.org/Doc3.0/CPlusPlus11.html ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Dec 24, 2014, at 12:22 AM, Dmitry Pavlov zeldi...@gmail.com wrote: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)) . The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. Not only no offense, no kidding! So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). In it's settings we can point to specific script and all guile invocation would just 1. include execution of that script with passing parameters from options 2. grab output that is supposed to be report content (html for now) and include that as it's own result Isn't our report system already too complicated? Geert has already mentioned that we're going to re-engineer GnuCash to use a SQL database internally; this is our plan for implementing concurrent multi-user access. That alone will vastly simplify the reporting architecture. It will also mean that SQLite3 rather than XML will be the preferred file format for most users and MySql or PostgresQL for those who need multi-user capabilities or who like running a DB Server. That in turn moves the preferred mechanism for custom reporting entirely out of GnuCash: Simply fire up your SQL report generator of choice, write your query, and run the report. You can actually do that now, as long as you don't trip over one of the corners where GnuCash writes to KVP without doing an edit-commit on the object owning the KVP. I can't say that it's the recommended approach for all custom reporting because of those corners, but if you can satisfy yourself that the SQL backend works correctly for your usage it's certainly easier and more future-resistant that writing a custom report in Scheme. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
Personally, i'm still on file storage so based on that maybe it's a good time now to look at sql one and try migrate to it and gain advantage of sql. 24 дек. 2014 г. 19:54 пользователь John Ralls jra...@ceridwen.us написал: On Dec 24, 2014, at 12:22 AM, Dmitry Pavlov zeldi...@gmail.com wrote: A brief search through list did not bring any results, so I decided to start a new one. First of all: no offence but most gnucash reports are poorly implemented. It's not because they useless or looks not pretty (most of them are useful and good, calm down :)) . The reason is that a model (i.e. data of the report) is completely messed up with the view (html tags) in report generation code + html creation tag by tag is really outdated now, there are more proper tools like templates for that. Of course it's a really huge work to rewrite that completely in more model-view style or rewrite that in different language. Not only no offense, no kidding! So I have idea: Gnucash already have an infrustructure of invoking scheme reports, saving settings, etc. What about implementing some wrapper report that can just invoke some script (for example that use python bindings). In it's settings we can point to specific script and all guile invocation would just 1. include execution of that script with passing parameters from options 2. grab output that is supposed to be report content (html for now) and include that as it's own result Isn't our report system already too complicated? Geert has already mentioned that we're going to re-engineer GnuCash to use a SQL database internally; this is our plan for implementing concurrent multi-user access. That alone will vastly simplify the reporting architecture. It will also mean that SQLite3 rather than XML will be the preferred file format for most users and MySql or PostgresQL for those who need multi-user capabilities or who like running a DB Server. That in turn moves the preferred mechanism for custom reporting entirely out of GnuCash: Simply fire up your SQL report generator of choice, write your query, and run the report. You can actually do that now, as long as you don't trip over one of the corners where GnuCash writes to KVP without doing an edit-commit on the object owning the KVP. I can't say that it's the recommended approach for all custom reporting because of those corners, but if you can satisfy yourself that the SQL backend works correctly for your usage it's certainly easier and more future-resistant that writing a custom report in Scheme. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Report wrapper to call python scripts?
On Wed, Dec 24, 2014 at 08:38:06AM -0800, John Ralls wrote: On Dec 24, 2014, at 7:38 AM, David Osguthorpe david.osgutho...@gmail.com wrote: (By the way note that GnuCash is multithreaded because Gtk is multi-threaded - in the initial implementation attempts I had issues with python GIL crashes but these dissappeared when I started using proper pygobject based implementations of GnuCashs GTypes). No, Gtk supports use in a multi-threaded environment, but is not itself multithreaded. In particular, Gtk requires that all calls to Gtk functions occur in a single thread, and on OSX (and perhaps Windows) that thread must be the main thread, the one in which the g_main_loop is executing. Worker threads should use idle events to coordinate with the Gtk thread. well according to gdb on OSX there are multiple threads running (at least 3) - and something was clobbering the Python GIL - the stack traces all pointed back to un-coordinated writes to the python GIL - and the crashes were all associated with menu clicks All of the main parts of GnuCash also operate in that one thread. WebKit, however, is multi-threaded. As long as you just hand it HTML and Javascript to render or execute it takes care of everything itself including posting events back to Gtk via the GtkWidget you give it to draw on. That will likely change in the course of the C++11 rewrite, though probably not this development cycle: We first need to have code that's threadable. The current code is too laden with static variables for that to be possible. I'm frankly amazed that GnuCash's malformed GObjects work with PyGObject considering that they generally ignore GObject's memory management and are directly allocated and freed. Regardless, the C++ rewrite will break all of that because we're removing all GLib dependencies from everything except the GUI. OTOH, since we'll be using C++11 std::shared_ptr and std::unique_ptr for memory management the resulting Python bindings should in the end be a lot more stable than what we have now, although the SWIG folks still have some work to do to fully assimilate C++11 [1]. I know but I wanted something that worked now with a stable gnucash - and most of the python wont change - only 3 modules (plus whatever you replace GKeyFile with) - which its not clear will change as they are directly associated with the GUI - GncPluginPage, GncHtml, GncHtmlWebKit - all other objects are handled using the existing python bindings Another potential problem for your addon: WebKit keeps getting more bloated and harder to build for Windows and OSX; in both cases we're using rather old versions (1.8.3 and 1.6.2 respectively) because of the difficulty of building newer versions on those platforms. We're looking for a lighter-weight replacement. If you're just making HTML strings and handing them to WebKit, no problem, but if you're interacting directly with the WebKitGtk API you'll have to rewrite that as well. I am just handing the HTML string to webkit - using the show_data function - this seemed to be the only pre-scheme place to use to inject HTML for display (and is presumably needed to use the Javascript plotting functions) - if you think there is a better way to inject the HTML please let me know David ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
