Re: [Qgis-developer] Creating large datasets with PyQGIS
Did you have a look at spatialite? It is a portable database format like dbf, has good support by qgis, supports indexing... Best Matthias On 03/24/2015 07:49 PM, Pedro Camargo wrote: > Hi, >I am currently working with some large geographic files (~130Gb > divided in thousands of files) and I need to output the result of the > analysis in some sort of data format, which cannot be a relational > database (because of the people that will be using this processing > framework in the future). > >Right now I am reading and writing to text files, which is > EXTREMELY slow. DBF is not really a good option because its many > legacy issues, so I was thinking about hdf5 or something of that sort. > > Is there anything in QGIS that would me allow to work with that? > >I am on a Windows environment, so using my pytables installation is > a pain in the neck. Any suggestions? > >BTW any advancements on how to install Python packages to QGIS own > python installation? > > Thanks, > > Pedro > > PS - Sorry for using windows ;-) > > > ___ > Qgis-developer mailing list > Qgis-developer@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/qgis-developer signature.asc Description: OpenPGP digital signature ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Creating large datasets with PyQGIS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Pedro! I think I cannot help you with large data.. (unwritten law of IT that it is pushed to its limits everytime). But regarding a python modules probably take a look here: http://quantumofgis.blogspot.co.at/2014/11/qgis-standalone-and-python-mo dules.html regards Werner On 03/24/2015 07:49 PM, Pedro Camargo wrote: > Hi, I am currently working with some large geographic files > (~130Gb divided in thousands of files) and I need to output the > result of the analysis in some sort of data format, which cannot be > a relational database (because of the people that will be using > this processing framework in the future). > > Right now I am reading and writing to text files, which is > EXTREMELY slow. DBF is not really a good option because its many > legacy issues, so I was thinking about hdf5 or something of that > sort. > > Is there anything in QGIS that would me allow to work with that? > > I am on a Windows environment, so using my pytables installation is > a pain in the neck. Any suggestions? > > BTW any advancements on how to install Python packages to QGIS own > python installation? > > Thanks, > > Pedro > > PS - Sorry for using windows ;-) > > > ___ Qgis-developer > mailing list Qgis-developer@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/qgis-developer > -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) iEYEARECAAYFAlURtYsACgkQDAH1YiCxBgkwgACcDOZJzerVW53ihx9EBQxf7qzg cagAnjlMl+BZGqvdIyTQgTnqn28if64g =l81R -END PGP SIGNATURE- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] Creating large datasets with PyQGIS
Hi, I am currently working with some large geographic files (~130Gb divided in thousands of files) and I need to output the result of the analysis in some sort of data format, which cannot be a relational database (because of the people that will be using this processing framework in the future). Right now I am reading and writing to text files, which is EXTREMELY slow. DBF is not really a good option because its many legacy issues, so I was thinking about hdf5 or something of that sort. Is there anything in QGIS that would me allow to work with that? I am on a Windows environment, so using my pytables installation is a pain in the neck. Any suggestions? BTW any advancements on how to install Python packages to QGIS own python installation? Thanks, Pedro PS - Sorry for using windows ;-) ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
On Tue, Mar 24, 2015 at 1:30 PM, Pietro wrote: > Hi Vaclav, > > On Tue, Mar 24, 2015 at 3:54 PM, Vaclav Petras > wrote: > > On Tue, Mar 24, 2015 at 2:00 AM, Pietro wrote: > >> together with Francesco Geri, last week we started to write a first > >> draft that generate the GUI for the GRASS module dynamically (based on > >> the xml generated with: --interface-description), so far it is based > >> on grass7 but should work also on grass6 (not tested, should be enough > >> to backport the pygrass.modules.Module class). > > > > Hi Pietro and Francesco, > > > > this sounds good. Are you speaking about QGIS GUI (in Python) generated > from > > --interface-description? Do you think it would be possible to write it > in a > > way which would be GUI framework independent so that it is usable in both > > GRASS and QGIS? The GUIs for modules in GRASS are quite good now but the > > code should be rewritten and this would be a great way how to get this > new > > implementation. > > yes, actually what I wrote it is completely independent from QGIS, it > is compatible with pyqt/pyside and with python2/python3, > It is just a proof of concept. Now Francesco it is working to better > integrate this draft with QGIS stuff. > > That's good to hear. > The idea was heavily based on formlayout code: > > https://code.google.com/p/formlayout/ > > Basically each parameter is a GUI field that it is add to the main > window, therefore based on the parameter type the field could be: > > if "raster"/"vector" => a combo box with the rasters/vectors available > if "integer" => a spinbox > if "string"/"double" => a text field > if "file" => a file dialog > > based on the guisection parameter all the options are organized in tabs. > > Both QGIS and GRASS are adding different extra features like setting region according to a selected raster or dependent fields for vector GRASS layers and attributes. That's why it gets complicated at the end. And of course then there is the extra information which is not stored in XML such as "GRASS standard options". Anyway, skipping qgm is quite important feature for QGIS since it is necessary for GRASS addons as you say. > The code is under 500 lines of code. > > I guess should be possible to follow the same approach using the wxpython. > I think that with a bit more effort should be possible to make it > general enough to make it independent from the GUI framework (Qt/Wx) > that we want to use. > > I have not link to share at this stage because we are still working on it. > But I guess we can share the code as soon as we have something that it > is able to cover the basic needs. > > I'm looking forward to that. > > Pietro > ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
On Tue, Mar 24, 2015 at 7:00 AM, Pietro wrote: > Hi Radim, > > together with Francesco Geri, last week we started to write a first > draft that generate the GUI for the GRASS module dynamically (based on > the xml generated with: --interface-description), so far it is based > on grass7 but should work also on grass6 (not tested, should be enough > to backport the pygrass.modules.Module class). The GUI for GRASS module in the plugin is also generated based on --interface-description, qgm files are only used to define a subset of module's options or to override option attributes (like label) if necessary. So it is already implemented, but always combined with qgm, skipping qgm, it maybe used to generate generic UI for any module. Using qgm gives more power, for example table field combo options based on current layer selected as input, setting input type from input layer, selecting feature categories in map canvas and so on. You may be interested in https://github.com/qgis/QGIS/blob/master/src/plugins/grass/qgsgrassmodule.cpp Do you see a possibility how we could join the effort? Radim > Best regards > > Pietro > > On Mon, Mar 23, 2015 at 7:56 PM, Radim Blazek wrote: >> Hi all, >> >> I have finally launched the crowdfunding campaign to support the GRASS >> plugin upgrade. Briefly, it covers upgrade to GRASS 7, browser >> integration, drag-and-drop import and new vector editing. All the >> details are available here: >> >> http://www.gissula.eu/qgis-grass-plugin-crowdfunding/ >> >> Please propagate this info to all relevant channels, national mailing lists >> etc. >> >> Radim >> ___ >> grass-dev mailing list >> grass-...@lists.osgeo.org >> http://lists.osgeo.org/mailman/listinfo/grass-dev ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
On Tue, Mar 24, 2015 at 2:00 AM, Pietro wrote: > Hi Radim, > > together with Francesco Geri, last week we started to write a first > draft that generate the GUI for the GRASS module dynamically (based on > the xml generated with: --interface-description), so far it is based > on grass7 but should work also on grass6 (not tested, should be enough > to backport the pygrass.modules.Module class). > > Hi Pietro and Francesco, this sounds good. Are you speaking about QGIS GUI (in Python) generated from --interface-description? Do you think it would be possible to write it in a way which would be GUI framework independent so that it is usable in both GRASS and QGIS? The GUIs for modules in GRASS are quite good now but the code should be rewritten and this would be a great way how to get this new implementation. Vaclav > Best regards > > Pietro > > On Mon, Mar 23, 2015 at 7:56 PM, Radim Blazek > wrote: > > Hi all, > > > > I have finally launched the crowdfunding campaign to support the GRASS > > plugin upgrade. Briefly, it covers upgrade to GRASS 7, browser > > integration, drag-and-drop import and new vector editing. All the > > details are available here: > > > > http://www.gissula.eu/qgis-grass-plugin-crowdfunding/ > > > > Please propagate this info to all relevant channels, national mailing > lists etc. > > > > Radim > > ___ > > grass-dev mailing list > > grass-...@lists.osgeo.org > > http://lists.osgeo.org/mailman/listinfo/grass-dev > ___ > grass-dev mailing list > grass-...@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/grass-dev > ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
On Tue, Mar 24, 2015 at 5:56 AM, Radim Blazek wrote: > >> Are there functions in time series implementation which need to be > >> called directly from the plugin or everything may be done just calling > >> t.rast.* modules? > > > > Most of the temporal functionality is available through the temporal > > modules. However some important algorithms (temporal re-sampling) are > > available only in the Python framework. This is needed for time series > > animation creation. Using the framework directly will speed things up, > > because the module calls, the parsing and interpretation of the module > > outputs can be avoided. > > If it should be used for dynamic animation in QGIS canvas you could > consider the possibility to subclass raster renderer in Python and > insert it into raster layer pipe from Python plugin. > Speaking about animations, some things from GRASS GIS GUI could be perhaps used directly in the same was as Tcl/Tk NVIZ is used in processing for GRASS 6. Animation tool is one of them. This would be great since we would get al least some functionality/code sharing between GRASS and QGIS GUIs which is otherwise not possible due to Python/wxPython and C++/Qt (and would be only possible if both things would be at least in the same language). This is of course not fulfilling the requirement to be general, i.e. work with other data providers in QGIS, but surely some things just have to be like that if they are using GRASS-specific formats (temporal data) or algorithms (e.g. algorithms to work with temporal data, their topology, ...). Vaclav ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] GRASS rasters in Processing?
Hi all, I noticed that GRASS rasters are not usable directly in Processing: is there a technical reason for this, or they are just filtered out from the list of available layers? On the other hand, vectors are available for analysis, but QGIS gets stuck at 100% CPU forever for a command (Split lines with lines, and an error[0] for gdalogr:gridinvdist. GDAL execution console output Unable to open input datasource "/home/paolo/Documents/Didattica/Corsi_Formazione_Faunalia/QGIS/QGIS_data/grass/Toscana_corso/PERMANENT/dtm_points/1_point". To me this seems both a bug (do not show layers for which an analysis cannot be run) and a feature request (support GRASS raster and vectors as Processing input). Am I wrong? If confirmed, I'm going to open the relevant tickets. All the best. -- Paolo Cavallini - www.faunalia.eu QGIS & PostGIS courses: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
Hi Soeren On Tue, Mar 24, 2015 at 10:04 AM, Sören Gebbert wrote: > 2015-03-24 9:40 GMT+01:00 Radim Blazek : >> I completely agree that Python would be better, the advantages of >> Python are obvious and that would be definitely my choice if I had to >> start from scratch. Un(fortunately) the GRASS plugin + qgsgrassgislib >> have already 22500 lines of C++ so porting to Python is not an option >> and mixing C++ in single plugin either (as far to my knowledge). > > Indeed, porting the C++ code to Python is a large effort. However, > maybe you can define a stretch-goal in the crowd funding campaign? If > this goal is met, then you have enough funds to port the C++ code to > Python and you can add more features? I don't have any serious estimation how much porting from C++ to Python costs, but new line of code costs 10-50euro (according to quick internet search). To be really very modest, say that porting would cost 2 euro per line, i.e. 22500*2 = 45000 euro for somethings which brings no new features to users. That is not something I would ever propose. > I think that using C++ and Python in a Plugin shouldn't be a big > problem in my humble opinion. The main issue would be that the C++ > code of the data provide will be part of QGIS and the Python code that > makes use of the GRASS data provider will be a separate GRASS Python > QGIS plugin. The plugin and the provider are sharing some C++ code (qgsgrass and qgsgrasslib). To port the plugin to Python you also have to write and maintain Python bindings for that shared classes which is just extra work. > Maybe this approach will allow to implement several > independent Python plugins that make use of the GRASS data provider to > implement specific algorithms? That should not depend on the GRASS plugin in C++. If you write Python bindings for the provider, you can use it (non standard) in your Python plugin. I believe however that plugins implementing algorithms should be preferably provider independent. >> Are there functions in time series implementation which need to be >> called directly from the plugin or everything may be done just calling >> t.rast.* modules? > > Most of the temporal functionality is available through the temporal > modules. However some important algorithms (temporal re-sampling) are > available only in the Python framework. This is needed for time series > animation creation. Using the framework directly will speed things up, > because the module calls, the parsing and interpretation of the module > outputs can be avoided. If it should be used for dynamic animation in QGIS canvas you could consider the possibility to subclass raster renderer in Python and insert it into raster layer pipe from Python plugin. >>> Btw: Otto Dassau and i mentioned your crowd funding idea at the >>> FOSSGIS in Germany two weeks ago. It is on Youtube[2] but only in >>> German. Thank you a lot! Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
Hi Radim, 2015-03-24 9:40 GMT+01:00 Radim Blazek : > Hi Soeren, > thanks for your reaction. I remember we already discussed the > possibility to move to Python. > > On Mon, Mar 23, 2015 at 9:19 PM, Sören Gebbert > wrote: >> Hi Radim, >> this is a beautiful idea and i hope you will get plenty of funds. >> >> I have some questions regarding the implementation, since this is not >> mentioned in the project description: >> >> Do you plan to implement the plugin in C++ only, or will you try to >> combine C++ (data provider) and Python (all the rest)? The reason i am >> asking is, that using Python for the user interaction, module calling, >> vector editing and mapset/location handling would allow us GRASS >> developer to provide possible improvements and bugfixes for the plugin >> more easily. For example, the time series handling [1] in GRASS GIS is >> mainly implemented in Python and provides a Python API that could be >> used in the QGIS GRASS Plugin to implement time series analysis >> support. >> Using the QGIS Python plugin approach will reduce the need for >> compilation, which allows much faster development of modifications and >> bugfix testing. > > > I completely agree that Python would be better, the advantages of > Python are obvious and that would be definitely my choice if I had to > start from scratch. Un(fortunately) the GRASS plugin + qgsgrassgislib > have already 22500 lines of C++ so porting to Python is not an option > and mixing C++ in single plugin either (as far to my knowledge). Indeed, porting the C++ code to Python is a large effort. However, maybe you can define a stretch-goal in the crowd funding campaign? If this goal is met, then you have enough funds to port the C++ code to Python and you can add more features? I think that using C++ and Python in a Plugin shouldn't be a big problem in my humble opinion. The main issue would be that the C++ code of the data provide will be part of QGIS and the Python code that makes use of the GRASS data provider will be a separate GRASS Python QGIS plugin. Maybe this approach will allow to implement several independent Python plugins that make use of the GRASS data provider to implement specific algorithms? > Are there functions in time series implementation which need to be > called directly from the plugin or everything may be done just calling > t.rast.* modules? Most of the temporal functionality is available through the temporal modules. However some important algorithms (temporal re-sampling) are available only in the Python framework. This is needed for time series animation creation. Using the framework directly will speed things up, because the module calls, the parsing and interpretation of the module outputs can be avoided. Many thanks for the QGIS GRASS Plugin Best regards Soeren > Radim > >> The data provider and vector editing helper classes must be of course >> implemented in C++ and should stay in the QGIS source tree. >> >> Best regards >> Soeren >> >> [1] http://ifgi.uni-muenster.de/~epebe_01/tgrass.pdf >> >> Btw: Otto Dassau and i mentioned your crowd funding idea at the >> FOSSGIS in Germany two weeks ago. It is on Youtube[2] but only in >> German. >> >> [2] https://www.youtube.com/watch?v=rxmPbh2igmM&t=1407 >> >> 2015-03-23 19:56 GMT+01:00 Radim Blazek : >>> Hi all, >>> >>> I have finally launched the crowdfunding campaign to support the GRASS >>> plugin upgrade. Briefly, it covers upgrade to GRASS 7, browser >>> integration, drag-and-drop import and new vector editing. All the >>> details are available here: >>> >>> http://www.gissula.eu/qgis-grass-plugin-crowdfunding/ >>> >>> Please propagate this info to all relevant channels, national mailing lists >>> etc. >>> >>> Radim >>> ___ >>> grass-dev mailing list >>> grass-...@lists.osgeo.org >>> http://lists.osgeo.org/mailman/listinfo/grass-dev ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [GRASS-dev] QGIS GRASS Plugin Upgrade Crowdfunding
Hi Soeren, thanks for your reaction. I remember we already discussed the possibility to move to Python. On Mon, Mar 23, 2015 at 9:19 PM, Sören Gebbert wrote: > Hi Radim, > this is a beautiful idea and i hope you will get plenty of funds. > > I have some questions regarding the implementation, since this is not > mentioned in the project description: > > Do you plan to implement the plugin in C++ only, or will you try to > combine C++ (data provider) and Python (all the rest)? The reason i am > asking is, that using Python for the user interaction, module calling, > vector editing and mapset/location handling would allow us GRASS > developer to provide possible improvements and bugfixes for the plugin > more easily. For example, the time series handling [1] in GRASS GIS is > mainly implemented in Python and provides a Python API that could be > used in the QGIS GRASS Plugin to implement time series analysis > support. > Using the QGIS Python plugin approach will reduce the need for > compilation, which allows much faster development of modifications and > bugfix testing. I completely agree that Python would be better, the advantages of Python are obvious and that would be definitely my choice if I had to start from scratch. Un(fortunately) the GRASS plugin + qgsgrassgislib have already 22500 lines of C++ so porting to Python is not an option and mixing C++ in single plugin either (as far to my knowledge). Are there functions in time series implementation which need to be called directly from the plugin or everything may be done just calling t.rast.* modules? Radim > The data provider and vector editing helper classes must be of course > implemented in C++ and should stay in the QGIS source tree. > > Best regards > Soeren > > [1] http://ifgi.uni-muenster.de/~epebe_01/tgrass.pdf > > Btw: Otto Dassau and i mentioned your crowd funding idea at the > FOSSGIS in Germany two weeks ago. It is on Youtube[2] but only in > German. > > [2] https://www.youtube.com/watch?v=rxmPbh2igmM&t=1407 > > 2015-03-23 19:56 GMT+01:00 Radim Blazek : >> Hi all, >> >> I have finally launched the crowdfunding campaign to support the GRASS >> plugin upgrade. Briefly, it covers upgrade to GRASS 7, browser >> integration, drag-and-drop import and new vector editing. All the >> details are available here: >> >> http://www.gissula.eu/qgis-grass-plugin-crowdfunding/ >> >> Please propagate this info to all relevant channels, national mailing lists >> etc. >> >> Radim >> ___ >> grass-dev mailing list >> grass-...@lists.osgeo.org >> http://lists.osgeo.org/mailman/listinfo/grass-dev ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] Sponsorship page and logos
Hi all. I'm glad announcing several new sponsors, and many donors: http://www.qgis.org/en/site/about/sponsorship.html These are vital for the progress of the project: please keep on donating, and spread out the work, suggesting others to do the same. All the best, and thanks to all! -- Paolo Cavallini - www.faunalia.eu QGIS & PostGIS courses: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
