Hello, I've found the refactor fields process to be a reliable and fast backup option (plus being a process it now works in the background on a separate thread and shows a progress meter so you've a good idea if it's actually working!) if you're happy to have entirely new fields added.
I've had similar issues recently, tried to add a virtual field onto a geopackage layer of about ~5000 features consisting of a attribute(get_feature()) and ended up freezing up with no sign of progress after 15min. Refactor field did 6 new fields at once in about 20 seconds! James On Tue, 9 Oct 2018 at 22:32, Alexandre Neto <senhor.n...@gmail.com> wrote: > My first suggestion would be for, after importing from csv, save it to > geopackages for the rest of the operations. Then, in the end export back to > the necessary file format. I think working directly on text files is not > optimal. > > Best regards, > > Alexandre Neto > > > A ter, 9/10/2018, 20:02, Rebecca Bartlett < > rebeccabartl...@cunet.carleton.ca> escreveu: > >> Hello, >> >> >> >> A couple of colleagues and I have been wrestling with a similar issue to >> this and I’ve been hunting around to see if it’s documented, and happily >> saw this email string. I’ve viewed the bug reports (both Kelley’s and the >> related #19771) and while they capture most of what we’ve been running >> into, it’s not an exact match. Here’s our situation and any input would be >> greatly appreciated – including viable workarounds. >> >> >> >> 1. Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File >> (5721 features; 9 fields; no geometry) >> >> 2. Using Field Calculator via the attribute table to add a Decimal >> field (length: 10; precision: 2 – the precision is important which is why >> the virtual field workaround without opening the attribute table hasn’t >> worked well with its precision of 3, plus the precision isn’t retained upon >> table export although we may be missing an export option somewhere) and >> input values from an already-existing text field >> >> a. Copying text field to numeric field for a join >> >> 3. Field calculation takes less than a second >> >> 4. Saving the table doesn’t work. I left it running for 90 minutes >> the first time and it has crashed QGIS on colleagues’ computers multiple >> times >> >> a. Notable that it also doesn’t work on 2.18.24 on Windows >> machines but it’s fine on 2.18 with Macs (ran the process in a class of 20 >> after we had them use 2.18, and only the Windows users had issues) >> >> b. Still takes a few moments to save on Mac 2.18 – but it works! >> >> >> >> We’ve tried several workarounds and the best one (a.k.a. the one that >> works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via >> the Source Fields tab in Properties, but it still takes >6 minutes to save >> which seems quite long for 5700 features. >> >> >> >> I’m assuming this is the same issue but if there are any reliable >> workarounds we’re all ears! >> >> >> >> Sincerely, >> >> Rebecca >> >> >> >> Rebecca Bartlett >> >> GIS and Digital Resources Librarian >> >> Research Support Services >> >> Carleton University Library >> >> 613-520-2600 x4084 >> >> rebecca.bartl...@carleton.ca >> >> >> ------------------------------ >> >> Okay, I've created an issue for this situation. Hopefully I've done it >> >> satisfactorily. Thanks again! >> >> >> >> On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net >> <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote: >> >> >> >> >* Hi,* >> >> > >> >> >* Glad to hear that it helped to run the operations on a closed table.* >> >> >* Nevertheless, as Luigi said, it would make sense to submit an issue >> >report* >> >> >* about this slowness.* >> >> > >> >> >* The other thing one has to keep on mind:* >> >> > >> >> >* The SQLite API is kind of limited. Only until recently, if you wanted to* >> >> >* rename a field, one had to copy the whole table with the renamed field* >> >> >* changed in the new table - then the old table had to be removed (all in >> >the* >> >> >* background), because there was API to rename columns in SQLite. I think* >> >> >* with the very recent version, which is not rolled out into most OS, this* >> >> >* has changed. You'd also need a very recent GDAL/OGR.* >> >> >* Greetings,* >> >> >* Andreas* >> >> > >> >> >* Am 03.10.2018 um 21:20 schrieb Kelley Jabr:* >> >> > >> >> >* Luigi, I'm pretty sure you just solved my problem. I tried a few* >> >> >* operations with the table closed and it was like night and day. That 400k* >> >> >* field calculator finished in mere moments! Thank you!!* >> >> > >> >> >* On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:* >> >> > >> >> >>* can you test doing operations with field calculator without having the* >> >> >>* attribute table opened? just a workaround could be to use a virtual >> >>layer* >> >> >>* then save the entire layer later.* >> >> >> >> >> >>* Luigi Pirelli* >> >> >> >> >> >> >> >> >>* >> >>*************************************************************************************************** >> >> >>* * LinkedIn: https://www.linkedin.com/in/luigipirelli >> >><https://www.linkedin.com/in/luigipirelli>* >> >> >>* * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli >> >><http://gis.stackexchange.com/users/19667/luigi-pirelli>* >> >> >>* * GitHub: https://github.com/luipir <https://github.com/luipir>* >> >> >>* * Mastering QGIS 2nd Edition:* >> >> >>* ** >> >> >>* >> >>https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition >> >> >> >><https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition>* >> >> >>* * Hire me: http://goo.gl/BYRQKg <http://goo.gl/BYRQKg>* >> >> >> >> >> >>* >> >>*************************************************************************************************** >> >> >> >> >> >> >> >> >>* On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com >> >><https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:* >> >> >> >> >> >>>* Hello,* >> >> >>> >> >> >>>* I'm working with a geopackage on ~530k features, and just about* >> >> >>>* everything I try to do in the table takes way longer than expected.* >> >> >>> >> >> >>>* - Adding a field took 5 minutes (table now has 22 total fields, so* >> >> >>>* not massive)* >> >> >>>* - Field Calculator on ~32k selected records took 8 minutes* >> >> >>>* (literally all I was putting in the cells was: N)* >> >> >>>* - Field calculator on ~480k records took so many hours that I had to* >> >> >>>* leave it running over the weekend and when I came back, the Field* >> >> >>>* Calculator had "closed unexpectedly"* >> >> >>> >> >> >>>* (I'm running on a linux machine with an Intel Core i7 and 32GB of* >> >> >>>* memory.)* >> >> >>> >> >> >>>* Has anyone else experience similar issues? Any advice or >> >>>recommendations* >> >> >>>* for how to improve performance here?* >> >> >>> >> >> >>>* Thank you,* >> >> >>> >> >> >>>* --* >> >> >>> >> >> >>>* Kelley Jabr* >> >> >>> >> >> >>>* Analyst, Geographic Data Operations* >> >> >>> >> >> >>>* kjabr at indeed.com >> >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >>> >> >> >>>* _______________________________________________* >> >> >>>* Qgis-user mailing list* >> >> >>>* Qgis-user at lists.osgeo.org >> >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >>>* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user >> >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >>>* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user >> >>><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >> >> >> >> >> >> > >> >> >* --* >> >> > >> >> >* Kelley Jabr* >> >> > >> >> >* Analyst, Geographic Data Operations* >> >> > >> >> >* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> > >> >> >* Indeed - We help people get jobs.* >> >> > >> >> >* Indeed.com <http://www.indeed.com <http://www.indeed.com/>>* >> >> > >> >> > >> >> > >> >> >* _______________________________________________* >> >> >* Qgis-user mailing listQgis-user at lists.osgeo.org >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> > >> >> > >> >> >* _______________________________________________* >> >> >* Qgis-user mailing list* >> >> >* Qgis-user at lists.osgeo.org >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user >> ><https://lists.osgeo.org/mailman/listinfo/qgis-user>* >> >> >> >> >> >> >> >> -- >> >> >> >> Kelley Jabr >> >> >> >> Analyst, Geographic Data Operations >> >> >> >> kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user> >> >> >> >> Indeed - We help people get jobs. >> >> >> >> Indeed.com <http://www.indeed.com> >> >> -------------- next part -------------- >> >> An HTML attachment was scrubbed... >> >> URL: >> <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html> >> >> >> >> >> _______________________________________________ >> Qgis-user mailing list >> Qgis-user@lists.osgeo.org >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > > -- > Alexandre Neto > --------------------- > @AlexNetoGeo > http://sigsemgrilhetas.wordpress.com > http://gisunchained.wordpress.com > _______________________________________________ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user