Re: [Qgis-user] clip geojason QGIS
> Convert it to a geopackage first -- geojson is a HORRIBLE format for large datasets like this, as it's completely unoptimised and has no spatial index Just want to add that shapefile itself has size limitations and is not a good format for these types of large datasets (or anything really), so I definitely recommended using gpkg. On Thu, 22 Oct 2020 at 03:52, Nicolas Cadieux wrote: > Hi, > > If the clipping operations takes too much time, you could try a spatial > query. Once you have a field with the states for each building, then you > can split the file with that field. This may, or may not be faster. Maybe > also be a problem with building sitting on the border. > > Nicolas Cadieux > > > Le 21 oct. 2020 à 20:28, Nyall Dawson a écrit : > > > > On Thu, 22 Oct 2020 at 10:23, Nyall Dawson > wrote: > >> > >>> On Thu, 22 Oct 2020 at 10:15, wrote: > >>> > >>> Nyall, > >>> > >>> Thanks for your quick reply. Would I load it first and then save the > layer as a geopackage to do that? Or do I have to do something more > elaborate? > >> > >> I'd do it from the QGIS browser panel -- right click on the geojson > >> file and save as. > > > > Just to clarify -- that's the "export" option, not "save as". > > > > Nyall > > > > That'll save some of the initial time required to > >> load this into QGIS. > >> > >> (alternatively the fastest way to convert would be to use ogr2ogr > >> directly from the command line, if you're comfortable with that!) > >> > >> Nyall > >> > >>> > >>> Cheers > >>> > >>> Dr Iain Stuart > >>> JCIS Consultants > >>> P.O. Box 2397 > >>> Burwood North > >>> NSW, 2134 > >>> > >>> (02) 9701 0191 > >>> (0413) 380116 (m) > >>> > >>> -Original Message- > >>> From: Nyall Dawson > >>> Sent: Thursday, 22 October 2020 10:41 AM > >>> To: i...@jcis.net.au > >>> Cc: qgis-user > >>> Subject: Re: [Qgis-user] clip geojason QGIS > >>> > >>> On Thu, 22 Oct 2020 at 09:33, wrote: > >>>> > >>>> It seems that Microsoft (of all people) is creating this database of > building shapes for the known universe. They recently completed Australia > and made it available. It is in a zipped geojason file – a format that I > have no idea about. > >>>> > >>>> > >>>> > >>>> I have loaded it to QGIS and it is huge. It took over 30min to load > and it takes about 10min to redraw every time and I am not running it on a > Pentium 4 with 8bits of ram either. > >>> > >>> Convert it to a geopackage first -- geojson is a HORRIBLE format for > large datasets like this, as it's completely unoptimised and has no spatial > index. Effectively on every redraw QGIS is having to loop through the > entire 6gb geojson file every time! ouch > >>> > >>> Nyall > >>> > >>> > >>> > >>>> > >>>> > >>>> > >>>> I would like to clip the data to the states of Australia which I have > a vector files or maybe regions in order to get the dataset smaller so it > can be more usable. > >>>> > >>>> > >>>> > >>>> Can you clip geojason files ? or would I have to covert them into > something like shapefiles? Any ideas on a workflow to do this? > >>>> > >>>> > >>>> > >>>> BTW I am running an I7-8700 with 16mb RAM and a NVIDIA Quadro P2000. > >>>> > >>>> > >>>> > >>>> Cheers > >>>> > >>>> > >>>> > >>>> Dr Iain Stuart > >>>> > >>>> JCIS Consultants > >>>> > >>>> P.O. Box 2397 > >>>> > >>>> Burwood North > >>>> > >>>> NSW, 2134 > >>>> > >>>> > >>>> > >>>> (02) 9701 0191 > >>>> (0413) 380116 (m) > >>>> > >>>> > >>>> > >>>> ___ > >>>> Qgis-user mailing list > >>>> Qgis-user@lists.osgeo.org > >>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > >>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > >>> > >>> > > ___ > > Qgis-user mailing list > > Qgis-user@lists.osgeo.org > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] clip geojason QGIS
Hi, If the clipping operations takes too much time, you could try a spatial query. Once you have a field with the states for each building, then you can split the file with that field. This may, or may not be faster. Maybe also be a problem with building sitting on the border. Nicolas Cadieux > Le 21 oct. 2020 à 20:28, Nyall Dawson a écrit : > > On Thu, 22 Oct 2020 at 10:23, Nyall Dawson wrote: >> >>> On Thu, 22 Oct 2020 at 10:15, wrote: >>> >>> Nyall, >>> >>> Thanks for your quick reply. Would I load it first and then save the layer >>> as a geopackage to do that? Or do I have to do something more elaborate? >> >> I'd do it from the QGIS browser panel -- right click on the geojson >> file and save as. > > Just to clarify -- that's the "export" option, not "save as". > > Nyall > > That'll save some of the initial time required to >> load this into QGIS. >> >> (alternatively the fastest way to convert would be to use ogr2ogr >> directly from the command line, if you're comfortable with that!) >> >> Nyall >> >>> >>> Cheers >>> >>> Dr Iain Stuart >>> JCIS Consultants >>> P.O. Box 2397 >>> Burwood North >>> NSW, 2134 >>> >>> (02) 9701 0191 >>> (0413) 380116 (m) >>> >>> -Original Message- >>> From: Nyall Dawson >>> Sent: Thursday, 22 October 2020 10:41 AM >>> To: i...@jcis.net.au >>> Cc: qgis-user >>> Subject: Re: [Qgis-user] clip geojason QGIS >>> >>> On Thu, 22 Oct 2020 at 09:33, wrote: >>>> >>>> It seems that Microsoft (of all people) is creating this database of >>>> building shapes for the known universe. They recently completed Australia >>>> and made it available. It is in a zipped geojason file – a format that I >>>> have no idea about. >>>> >>>> >>>> >>>> I have loaded it to QGIS and it is huge. It took over 30min to load and it >>>> takes about 10min to redraw every time and I am not running it on a >>>> Pentium 4 with 8bits of ram either. >>> >>> Convert it to a geopackage first -- geojson is a HORRIBLE format for large >>> datasets like this, as it's completely unoptimised and has no spatial >>> index. Effectively on every redraw QGIS is having to loop through the >>> entire 6gb geojson file every time! ouch >>> >>> Nyall >>> >>> >>> >>>> >>>> >>>> >>>> I would like to clip the data to the states of Australia which I have a >>>> vector files or maybe regions in order to get the dataset smaller so it >>>> can be more usable. >>>> >>>> >>>> >>>> Can you clip geojason files ? or would I have to covert them into >>>> something like shapefiles? Any ideas on a workflow to do this? >>>> >>>> >>>> >>>> BTW I am running an I7-8700 with 16mb RAM and a NVIDIA Quadro P2000. >>>> >>>> >>>> >>>> Cheers >>>> >>>> >>>> >>>> Dr Iain Stuart >>>> >>>> JCIS Consultants >>>> >>>> P.O. Box 2397 >>>> >>>> Burwood North >>>> >>>> NSW, 2134 >>>> >>>> >>>> >>>> (02) 9701 0191 >>>> (0413) 380116 (m) >>>> >>>> >>>> >>>> ___ >>>> Qgis-user mailing list >>>> Qgis-user@lists.osgeo.org >>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user >>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user >>> >>> > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] clip geojason QGIS
On Thu, 22 Oct 2020 at 10:23, Nyall Dawson wrote: > > On Thu, 22 Oct 2020 at 10:15, wrote: > > > > Nyall, > > > > Thanks for your quick reply. Would I load it first and then save the layer > > as a geopackage to do that? Or do I have to do something more elaborate? > > I'd do it from the QGIS browser panel -- right click on the geojson > file and save as. Just to clarify -- that's the "export" option, not "save as". Nyall That'll save some of the initial time required to > load this into QGIS. > > (alternatively the fastest way to convert would be to use ogr2ogr > directly from the command line, if you're comfortable with that!) > > Nyall > > > > > Cheers > > > > Dr Iain Stuart > > JCIS Consultants > > P.O. Box 2397 > > Burwood North > > NSW, 2134 > > > > (02) 9701 0191 > > (0413) 380116 (m) > > > > -----Original Message- > > From: Nyall Dawson > > Sent: Thursday, 22 October 2020 10:41 AM > > To: i...@jcis.net.au > > Cc: qgis-user > > Subject: Re: [Qgis-user] clip geojason QGIS > > > > On Thu, 22 Oct 2020 at 09:33, wrote: > > > > > > It seems that Microsoft (of all people) is creating this database of > > > building shapes for the known universe. They recently completed Australia > > > and made it available. It is in a zipped geojason file – a format that I > > > have no idea about. > > > > > > > > > > > > I have loaded it to QGIS and it is huge. It took over 30min to load and > > > it takes about 10min to redraw every time and I am not running it on a > > > Pentium 4 with 8bits of ram either. > > > > Convert it to a geopackage first -- geojson is a HORRIBLE format for large > > datasets like this, as it's completely unoptimised and has no spatial > > index. Effectively on every redraw QGIS is having to loop through the > > entire 6gb geojson file every time! ouch > > > > Nyall > > > > > > > > > > > > > > > > > > I would like to clip the data to the states of Australia which I have a > > > vector files or maybe regions in order to get the dataset smaller so it > > > can be more usable. > > > > > > > > > > > > Can you clip geojason files ? or would I have to covert them into > > > something like shapefiles? Any ideas on a workflow to do this? > > > > > > > > > > > > BTW I am running an I7-8700 with 16mb RAM and a NVIDIA Quadro P2000. > > > > > > > > > > > > Cheers > > > > > > > > > > > > Dr Iain Stuart > > > > > > JCIS Consultants > > > > > > P.O. Box 2397 > > > > > > Burwood North > > > > > > NSW, 2134 > > > > > > > > > > > > (02) 9701 0191 > > > (0413) 380116 (m) > > > > > > > > > > > > ___ > > > Qgis-user mailing list > > > Qgis-user@lists.osgeo.org > > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > > > > ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] clip geojason QGIS
On Thu, 22 Oct 2020 at 10:15, wrote: > > Nyall, > > Thanks for your quick reply. Would I load it first and then save the layer as > a geopackage to do that? Or do I have to do something more elaborate? I'd do it from the QGIS browser panel -- right click on the geojson file and save as. That'll save some of the initial time required to load this into QGIS. (alternatively the fastest way to convert would be to use ogr2ogr directly from the command line, if you're comfortable with that!) Nyall > > Cheers > > Dr Iain Stuart > JCIS Consultants > P.O. Box 2397 > Burwood North > NSW, 2134 > > (02) 9701 0191 > (0413) 380116 (m) > > -Original Message- > From: Nyall Dawson > Sent: Thursday, 22 October 2020 10:41 AM > To: i...@jcis.net.au > Cc: qgis-user > Subject: Re: [Qgis-user] clip geojason QGIS > > On Thu, 22 Oct 2020 at 09:33, wrote: > > > > It seems that Microsoft (of all people) is creating this database of > > building shapes for the known universe. They recently completed Australia > > and made it available. It is in a zipped geojason file – a format that I > > have no idea about. > > > > > > > > I have loaded it to QGIS and it is huge. It took over 30min to load and it > > takes about 10min to redraw every time and I am not running it on a Pentium > > 4 with 8bits of ram either. > > Convert it to a geopackage first -- geojson is a HORRIBLE format for large > datasets like this, as it's completely unoptimised and has no spatial index. > Effectively on every redraw QGIS is having to loop through the entire 6gb > geojson file every time! ouch > > Nyall > > > > > > > > > > > I would like to clip the data to the states of Australia which I have a > > vector files or maybe regions in order to get the dataset smaller so it can > > be more usable. > > > > > > > > Can you clip geojason files ? or would I have to covert them into something > > like shapefiles? Any ideas on a workflow to do this? > > > > > > > > BTW I am running an I7-8700 with 16mb RAM and a NVIDIA Quadro P2000. > > > > > > > > Cheers > > > > > > > > Dr Iain Stuart > > > > JCIS Consultants > > > > P.O. Box 2397 > > > > Burwood North > > > > NSW, 2134 > > > > > > > > (02) 9701 0191 > > (0413) 380116 (m) > > > > > > > > ___ > > Qgis-user mailing list > > Qgis-user@lists.osgeo.org > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > > ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] clip geojason QGIS
On Thu, 22 Oct 2020 at 09:33, wrote: > > It seems that Microsoft (of all people) is creating this database of building > shapes for the known universe. They recently completed Australia and made it > available. It is in a zipped geojason file – a format that I have no idea > about. > > > > I have loaded it to QGIS and it is huge. It took over 30min to load and it > takes about 10min to redraw every time and I am not running it on a Pentium 4 > with 8bits of ram either. Convert it to a geopackage first -- geojson is a HORRIBLE format for large datasets like this, as it's completely unoptimised and has no spatial index. Effectively on every redraw QGIS is having to loop through the entire 6gb geojson file every time! ouch Nyall > > > > I would like to clip the data to the states of Australia which I have a > vector files or maybe regions in order to get the dataset smaller so it can > be more usable. > > > > Can you clip geojason files ? or would I have to covert them into something > like shapefiles? Any ideas on a workflow to do this? > > > > BTW I am running an I7-8700 with 16mb RAM and a NVIDIA Quadro P2000. > > > > Cheers > > > > Dr Iain Stuart > > JCIS Consultants > > P.O. Box 2397 > > Burwood North > > NSW, 2134 > > > > (02) 9701 0191 > (0413) 380116 (m) > > > > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
