My first suggestion would be for, after importing from csv, save it to geopackages for the rest of the operations. Then, in the end export back to the necessary file format. I think working directly on text files is not optimal.
Best regards, Alexandre Neto A ter, 9/10/2018, 20:02, Rebecca Bartlett <rebeccabartl...@cunet.carleton.ca> escreveu: > Hello, > > > > A couple of colleagues and I have been wrestling with a similar issue to > this and I’ve been hunting around to see if it’s documented, and happily > saw this email string. I’ve viewed the bug reports (both Kelley’s and the > related #19771) and while they capture most of what we’ve been running > into, it’s not an exact match. Here’s our situation and any input would be > greatly appreciated – including viable workarounds. > > > > 1. Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File > (5721 features; 9 fields; no geometry) > > 2. Using Field Calculator via the attribute table to add a Decimal > field (length: 10; precision: 2 – the precision is important which is why > the virtual field workaround without opening the attribute table hasn’t > worked well with its precision of 3, plus the precision isn’t retained upon > table export although we may be missing an export option somewhere) and > input values from an already-existing text field > > a. Copying text field to numeric field for a join > > 3. Field calculation takes less than a second > > 4. Saving the table doesn’t work. I left it running for 90 minutes > the first time and it has crashed QGIS on colleagues’ computers multiple > times > > a. Notable that it also doesn’t work on 2.18.24 on Windows machines > but it’s fine on 2.18 with Macs (ran the process in a class of 20 after we > had them use 2.18, and only the Windows users had issues) > > b. Still takes a few moments to save on Mac 2.18 – but it works! > > > > We’ve tried several workarounds and the best one (a.k.a. the one that > works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via > the Source Fields tab in Properties, but it still takes >6 minutes to save > which seems quite long for 5700 features. > > > > I’m assuming this is the same issue but if there are any reliable > workarounds we’re all ears! > > > > Sincerely, > > Rebecca > > > > Rebecca Bartlett > > GIS and Digital Resources Librarian > > Research Support Services > > Carleton University Library > > 613-520-2600 x4084 > > rebecca.bartl...@carleton.ca > > > ------------------------------ > > Okay, I've created an issue for this situation. Hopefully I've done it > > satisfactorily. Thanks again! > > > > On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net > <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote: > > > > >* Hi,* > > > > > >* Glad to hear that it helped to run the operations on a closed table.* > > >* Nevertheless, as Luigi said, it would make sense to submit an issue report* > > >* about this slowness.* > > > > > >* The other thing one has to keep on mind:* > > > > > >* The SQLite API is kind of limited. Only until recently, if you wanted to* > > >* rename a field, one had to copy the whole table with the renamed field* > > >* changed in the new table - then the old table had to be removed (all in > >the* > > >* background), because there was API to rename columns in SQLite. I think* > > >* with the very recent version, which is not rolled out into most OS, this* > > >* has changed. You'd also need a very recent GDAL/OGR.* > > >* Greetings,* > > >* Andreas* > > > > > >* Am 03.10.2018 um 21:20 schrieb Kelley Jabr:* > > > > > >* Luigi, I'm pretty sure you just solved my problem. I tried a few* > > >* operations with the table closed and it was like night and day. That 400k* > > >* field calculator finished in mere moments! Thank you!!* > > > > > >* On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:* > > > > > >>* can you test doing operations with field calculator without having the* > > >>* attribute table opened? just a workaround could be to use a virtual layer* > > >>* then save the entire layer later.* > > >> > > >>* Luigi Pirelli* > > >> > > >> > > >>* > >>*************************************************************************************************** > > >>* * LinkedIn: https://www.linkedin.com/in/luigipirelli > >><https://www.linkedin.com/in/luigipirelli>* > > >>* * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli > >><http://gis.stackexchange.com/users/19667/luigi-pirelli>* > > >>* * GitHub: https://github.com/luipir <https://github.com/luipir>* > > >>* * Mastering QGIS 2nd Edition:* > > >>* ** > > >>* > >>https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition > >> > >><https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition>* > > >>* * Hire me: http://goo.gl/BYRQKg <http://goo.gl/BYRQKg>* > > >> > > >>* > >>*************************************************************************************************** > > >> > > >> > > >>* On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com > >><https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:* > > >> > > >>>* Hello,* > > >>> > > >>>* I'm working with a geopackage on ~530k features, and just about* > > >>>* everything I try to do in the table takes way longer than expected.* > > >>> > > >>>* - Adding a field took 5 minutes (table now has 22 total fields, so* > > >>>* not massive)* > > >>>* - Field Calculator on ~32k selected records took 8 minutes* > > >>>* (literally all I was putting in the cells was: N)* > > >>>* - Field calculator on ~480k records took so many hours that I had to* > > >>>* leave it running over the weekend and when I came back, the Field* > > >>>* Calculator had "closed unexpectedly"* > > >>> > > >>>* (I'm running on a linux machine with an Intel Core i7 and 32GB of* > > >>>* memory.)* > > >>> > > >>>* Has anyone else experience similar issues? Any advice or recommendations* > > >>>* for how to improve performance here?* > > >>> > > >>>* Thank you,* > > >>> > > >>>* --* > > >>> > > >>>* Kelley Jabr* > > >>> > > >>>* Analyst, Geographic Data Operations* > > >>> > > >>>* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >>> > > >>>* _______________________________________________* > > >>>* Qgis-user mailing list* > > >>>* Qgis-user at lists.osgeo.org > >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >>>* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >>>* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >> > > >> > > > > > >* --* > > > > > >* Kelley Jabr* > > > > > >* Analyst, Geographic Data Operations* > > > > > >* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > > > > >* Indeed - We help people get jobs.* > > > > > >* Indeed.com <http://www.indeed.com <http://www.indeed.com/>>* > > > > > > > > > > > >* _______________________________________________* > > >* Qgis-user mailing listQgis-user at lists.osgeo.org > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > > > > > > > >* _______________________________________________* > > >* Qgis-user mailing list* > > >* Qgis-user at lists.osgeo.org > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* > > > > > > > > -- > > > > Kelley Jabr > > > > Analyst, Geographic Data Operations > > > > kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user> > > > > Indeed - We help people get jobs. > > > > Indeed.com <http://www.indeed.com> > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html> > > > > > _______________________________________________ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user -- Alexandre Neto --------------------- @AlexNetoGeo http://sigsemgrilhetas.wordpress.com http://gisunchained.wordpress.com
_______________________________________________ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
