Re: [Qgis-developer] Delimited text issues and ideas
Hi Alexander Quite agree that it is still not user friendly. I have a patch which allows multiselect in the dialog. It is a little unclean in that it may ask for the coordinate system for each file (depending on current settings), but that would require changes outside of the provider I think. I've not pushed this to master - I think it should wait till after 2.0 now as it will add a few more translations. That said, it would be nice to have it consistent with the AddVectorLayer dialog. It is available at.. https://github.com/ccrook/Quantum-GIS/tree/csv_multi_select Cheers Chris From: Alexander Bruy [alexander.b...@gmail.com] Sent: 27 May 2013 04:22 To: Chris Crook Cc: qgis-developer@lists.osgeo.org Subject: Re: [Qgis-developer] Delimited text issues and ideas Hi Chris, 2013/5/26 Chris Crook : > Good thought. If you are loading several files at the moment QGIS should > remember the settings between files, so you only have to select them and add > them, not redo the settings. But it is still a lot of clicks for each file. Right, now settings are saved between files, but users still need to select file one by one. Not userfriendly IMO. > 1) Once a list of files are selected from browse buttin, how to display the > selected file names in the text box. Would this become a drop down list? Or > would we use a list of file names with a separator (and if so, what > separator)? Should the text box allow wild card file names? There must be a > precedent for this somewhere, just can't think of one straight away. Why not show full file names with separator like semicolon? If I'm not wrong, same approach used in GDALTools plugin when multiple files selected. > 2) Handling of layer names for each file. Could just be the file name (which > is the default layer name anyway). Could have some way of entering the > filename into a string (eg ). Whatever is used it should be > consistent with the rest of QGIS (expression builder? not sure there is > anywhere variables are used in this way), and obvious for the user... For me using filename as layer name is fine -- Alexander Bruy This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or i...@linz.govt.nz) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Delimited text issues and ideas
Hi Chris, 2013/5/26 Chris Crook : > Good thought. If you are loading several files at the moment QGIS should > remember the settings between files, so you only have to select them and add > them, not redo the settings. But it is still a lot of clicks for each file. Right, now settings are saved between files, but users still need to select file one by one. Not userfriendly IMO. > 1) Once a list of files are selected from browse buttin, how to display the > selected file names in the text box. Would this become a drop down list? Or > would we use a list of file names with a separator (and if so, what > separator)? Should the text box allow wild card file names? There must be a > precedent for this somewhere, just can't think of one straight away. Why not show full file names with separator like semicolon? If I'm not wrong, same approach used in GDALTools plugin when multiple files selected. > 2) Handling of layer names for each file. Could just be the file name (which > is the default layer name anyway). Could have some way of entering the > filename into a string (eg ). Whatever is used it should be > consistent with the rest of QGIS (expression builder? not sure there is > anywhere variables are used in this way), and obvious for the user... For me using filename as layer name is fine -- Alexander Bruy ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Delimited text issues and ideas
Hi Alexander Good thought. If you are loading several files at the moment QGIS should remember the settings between files, so you only have to select them and add them, not redo the settings. But it is still a lot of clicks for each file. It would be fairly easy to add multiple files. I guess you'd just use the first selected file in the preview and then take those settings for all the files. Only a couple of messy user interface things .. 1) Once a list of files are selected from browse buttin, how to display the selected file names in the text box. Would this become a drop down list? Or would we use a list of file names with a separator (and if so, what separator)? Should the text box allow wild card file names? There must be a precedent for this somewhere, just can't think of one straight away. 2) Handling of layer names for each file. Could just be the file name (which is the default layer name anyway). Could have some way of entering the filename into a string (eg ). Whatever is used it should be consistent with the rest of QGIS (expression builder? not sure there is anywhere variables are used in this way), and obvious for the user... Any thoughts on this? Cheers Chris ___ _ From: Alexander Bruy [alexander.b...@gmail.com] Sent: 23 May 2013 18:35 To: Tim Sutton Cc: qgis-developer@lists.osgeo.org; HAUBOURG; Chris Crook Subject: Re: [Qgis-developer] Delimited text issues and ideas Hi, also another useful option will be ability lo load multiple CSV files at once using same settings. Now, if I want load many CSV file I should open them one by one, set settings and then load. I know, this can be implemented as script but not all users like scripting. -- Alexander Bruy This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or i...@linz.govt.nz) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Delimited text issues and ideas
I just added a mockup based on nathan's work, and actual appearance of Qgis Browser. Forgive me for the poor image work, it's a draft to help understanding what's in my mind sith "unified data access" see https://dl.dropboxusercontent.com/u/72368800/unified_data_access_improved.png -- View this message in context: http://osgeo-org.1560.x6.nabble.com/Delimited-text-issues-and-ideas-tp5055027p5055218.html Sent from the Quantum GIS - Developer mailing list archive at Nabble.com. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Delimited text issues and ideas
Hi Chris, thanks for enlarging the discussion here, that's always better. I really like the idea to interpret CSVT, but users will need a gui to create it. That remains a good patch for 2.0 if possible. We could also add cvst / vrt writing in a standalone function (vector menu) .. not urgent to me, these are workaround of a good import assistant. We kept on digging all import problems, and have no simple solution. MM QGIS gives some functions, but needs to write at least two intermediate shapes (01 for import + XY, 2 to convert text fields to real when needed) . Additionnal problem, almost no common tool is exporting csv correctly according to csv specs, we then have to build good import tools, dealing with all existing variations of txt and csv files (including BOM problems).. Excel does good when dealing with decimal delimiters, but skips text delimiters, so GDAL will interpret all numeric-like identifier as identifier. My "09" is read as 9 when imported.. No possibility for me to reinvent french administrative codes. Libre Office is hard to use when dealing with commas/point as text delimiters What I have in mind to let my user work: *1- Short term proposal for actual users :* QGIS 1.8: Me (funding or coding) : developp a quick plugin to add in vector menu: Create point / geometry from text into a memory layer. I will advice users to import with a clean xls 2003 format when dealing with such data sources. GDAL works Ok (better than csv, with no csvt needed) CSV is OK with your import tool, but will put numeric fields in string. Use of MM qgis or add field + update in shp is mandatory *2 -QGIS 2.0 :* well, feature freeze is her, so having the create point plugin compatible with new API. 3- QGIS 2.1 : fund (me certainly, if a consensus is reached) what have been previously discussed here [0] with Nathan mockups of unified data access dialog, and migrate your importer transparently there. 1: Unify data access : previsualization of data / character enconding choice / text delimiter / decimal delimiter / data type chooser assistant / advanced options to write csvt and/or vrt for power users 2: port to C++ create point or geometry plugin 3: add basic ETL functions to Core : rename, reorder, change type, change names, save through OGR or mem Layer [0] http://osgeo-org.1560.x6.nabble.com/Clarifying-access-to-non-geographical-datas-td4986567.html#a4987225 Thoughts? -- View this message in context: http://osgeo-org.1560.x6.nabble.com/Delimited-text-issues-and-ideas-tp5055027p5055180.html Sent from the Quantum GIS - Developer mailing list archive at Nabble.com. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Delimited text issues and ideas
Hi, also another useful option will be ability lo load multiple CSV files at once using same settings. Now, if I want load many CSV file I should open them one by one, set settings and then load. I know, this can be implemented as script but not all users like scripting. -- Alexander Bruy ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Delimited text issues and ideas
Hi On Wed, May 22, 2013 at 10:48 PM, Chris Crook wrote: > Hi Régis > > Interesting thoughts. I've renamed this thread and copied it in to > qgis-developer, as I think it is worth getting broader input on this. Hope > you don't mind my putting your email there > > If I've understood the main points of your suggestion are: > > 1) The delimited text provider and the GUI should be able to read a VRT/CSVT > file if one is present to determine field types etc > > 2) The delimited text provider GUI should be able to save settings to a > VRT/CSVT > > 3) The user should be able to explicitly set data types for each column > > This is inline with some of my thinking in this. I was planning to add some > way of saving settings as a "delimited text file type", that could then be > selected when adding a new text layer. At the moment (ie in master, not 1.8) > the plugin remembers settings based on file extension, but that doesn't > provide enough granularity for me. This would be analogous to saving styles > in QGIS. I hadn't decided where to store the settings yet (ie whether in a > file, or in the QGIS settings, or... ) > > Like you, I am also not very happy with the way that the provider determines > field types by scanning the file when it loads it. So if you put different > data in the file, and reload it into QGIS, then the data types may change. > > I hadn't thought about using VRT or CSVT files and I really like the idea. > > As a really simple first step, which would not be a major refactoring, the > provider could check for a CSVT file when it loads as CSV and use it to > determine field types - this would not require any UI or API changes at all. > That on its own could be very useful. The only difficulty I can foresee with > this is how to manage files with names other than ".csv". Should it just > look for a file matching the name of the input file with a "t" at the end? > Or should it only use this options for files that are named ".csv". Or > should it look for ".csvt" whatever the name of the file. Other than that > this really isn't much work, and I'd be keen to implement it. I guess the > simplest approach would be the first options, looking for a file named the > same as the data file but with a "t". If it exists and can be interpreted, > then it will be used to define field types. This would make it really easy > to manage when creating data, and would be compatible with GDAL/OGR. > > The VRT file is much more work as you suggest. The mapping between VRT and > the CSV options is not complete. So the options around delimiters, skipped > lines, regular expressions, and so on are not available for the VRT file. > Conversely several of the VRT options don't apply within QGIS. So there is > quite a lot of work in specifying how this should work. Also it would entail > a major reworking in terms of how the file is opened (ie you would select the > VRT file in the GUI, which would then have to identify the CSV file, and also > handle politely VRT files which did not define CSV files, but which defined > other data source types. This seems a lot of work and would end up > re-engineering a lot of what is already in OGR. So (even as I'm writing > this) I'm becoming less clear that this is a good approach. > > Returning to the CSVT idea - once the provider can use the CSVT then there > remains the question of what GUI/API changes should be made to support it. > Two thoughts come to mind immediately. > > One is, should the CSVT idea be extended to support the other metadata > information required in setting up the file, such as the delimiter, etc. The > OGR specification for CSVT just defines the field types in the first line. I > don't know if it would ignore subsequent lines, in which case additional > metadata could go there and still be compatible with OGR usage. Or should > another metadata file (eg .metadata, .qgs, .dlt, ...) be used to hold all the > information specifying how the file should be used (sounds really messy). > But it would be really nice to be able to just select a file and have all > these options automatically populated if the metadata file existed. > > The other thought is around your suggestion of writing the CSVT/metadata > file. The main extra work involved in this is the user interface for > defining field types. > > The dialog box is already quite busy, but I guess a simple approach would be > just to add a row to the preview box under the column headings with a field > type selector for each column, and values > "Auto,Text,Integer,Real,Date,Time,DateTime", or something like that. The > field types could then be passed through to the provider in the datasource > URI (ie with a parameter such as "fieldtypes=text,text,integer,..."). This > also doesn't sound like too much work. > > Once this is done it would be simple to add a "save settings to metadata > file" type button to the GUI, wh
