[R-sig-Geo] Plotting USA map with each state's shape or size reflective of a metric

[Vinh Nguyen]

Hi everyone,

Does anyone know if this is possible with R? An example can be found here:
http://projects.fivethirtyeight.com/2016-election-forecast/ (a few plots
down).

Appreciate any references you point me to. Thanks!

Vinh

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] best practice for reading large shapefiles?

[Vinh Nguyen]

On Tue, Apr 26, 2016 at 1:12 PM, Roger Bivand <roger.biv...@nhh.no> wrote:
> On Tue, 26 Apr 2016, Vinh Nguyen wrote:
>
>> Would loading the shapefile into postgresql first and then use readOGR
>> to read from postgres be a recommended approach?  That is, would the
>> bottleneck still occur?  Thank you.
>
>
> Most likely, as both use the respective OGR drivers. With data this size,
> you'll need a competent platform (probably Linux, say 128GB RAM) as
> everything is in memory. I find it hard to grasp what the point of doing
> this might be - visualization won't work as none of the considerable detail
> certainly in these files will be visible. Can you put the lot into an SQLite
> file and access the attributes as SQL queries? I don't see the analysis or
> statistics here.
>

- I can't tell from your response whether you are recommending PostGIS
is a recommended approach or not.  Could you clarify?

- I am working on a Windows server with 64gb ram, so not too weak,
especially for some files that are a few gb in size.  Again, not sure
if the job just halted or it's still running, but just rather slow.
I've killed it for now as the memory usage still has not grown after a
few hours.

- Yes, the shapes are quite granular and many in quantity.  The use
case was not to visualize them all at once.  Wanted a master file so
that when I get a data set of interest, I could intersect the two and
then subset the areas of interest (eg, within a state or county).
Then visualize/analyze from there.  The master shapefile was meant to
make it easy (reading in one file) as opposed to deciding which
shapefile to read in depending on the project.

- I just looked back at the 30 PLSS zip files, and they provide shapes
for 3 levels of granularity.  I went with the smallest.  I just
realized that the mid-size one would be sufficient for now, which
results in dbf=138mb and shp=501mb.  Attempting to read this in now (~
30 minutes), which I assume will read in fine after some time.  Will
respond to this thread if this is not the case.

Thanks for responding Roger.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] best practice for reading large shapefiles?

[Vinh Nguyen]

Would loading the shapefile into postgresql first and then use readOGR
to read from postgres be a recommended approach?  That is, would the
bottleneck still occur?  Thank you.

-- Vinh


On Tue, Apr 26, 2016 at 11:18 AM, Vinh Nguyen <vinhdi...@gmail.com> wrote:
> Hi,
>
> I have a very large shapefile that I would like to read into R
> (dbf=5.6gb and shp=2.3gb).
>
> For reference, I downloaded the 30 shapefiles of the [Public Land
> Survey System](http://www.geocommunicator.gov/GeoComm/lsis_home/home/)
> and combined them into a single national file via gdal (ogr2ogr) as
> described 
> [here](http://www.northrivergeographic.com/ogr2ogr-merge-shapefiles);
> I originally attempted to combine the files in R as described
> [here](https://stat.ethz.ch/pipermail/r-sig-geo/2011-May/011814.html),
> but ran out of memory about 80% in, but luckily discovered ogr2ogr.
>
> I'm reading in the combined file in R via readOGR, and it's been over
> an hour and R appears to hang.  When I check the task manager, the R
> session currently consumes <10% CPU and 245MB.  Not sure if any
> productive activity is going on, so I'm just waiting it out.
> [This](http://r-sig-geo.2731867.n2.nabble.com/Long-time-to-load-shapefiles-td7584869.html)
> thread describes that readOGR can be slow for large shapefiles, and
> suggested that the SpatialDataFrame be saved in an R format.  My
> problem is getting the entire shapefile read in the first place before
> I could save it as an R object.
>
> Does anyone have any suggestions for reading this large shapefile into
> R?  Thank you for your help.
>
> -- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[R-sig-Geo] best practice for reading large shapefiles?

[Vinh Nguyen]

Hi,

I have a very large shapefile that I would like to read into R
(dbf=5.6gb and shp=2.3gb).

For reference, I downloaded the 30 shapefiles of the [Public Land
Survey System](http://www.geocommunicator.gov/GeoComm/lsis_home/home/)
and combined them into a single national file via gdal (ogr2ogr) as
described [here](http://www.northrivergeographic.com/ogr2ogr-merge-shapefiles);
I originally attempted to combine the files in R as described
[here](https://stat.ethz.ch/pipermail/r-sig-geo/2011-May/011814.html),
but ran out of memory about 80% in, but luckily discovered ogr2ogr.

I'm reading in the combined file in R via readOGR, and it's been over
an hour and R appears to hang.  When I check the task manager, the R
session currently consumes <10% CPU and 245MB.  Not sure if any
productive activity is going on, so I'm just waiting it out.
[This](http://r-sig-geo.2731867.n2.nabble.com/Long-time-to-load-shapefiles-td7584869.html)
thread describes that readOGR can be slow for large shapefiles, and
suggested that the SpatialDataFrame be saved in an R format.  My
problem is getting the entire shapefile read in the first place before
I could save it as an R object.

Does anyone have any suggestions for reading this large shapefile into
R?  Thank you for your help.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

On Apr 7, 2016 12:58 AM, "Roger Bivand" <roger.biv...@nhh.no> wrote:
>
> On Wed, 6 Apr 2016, Vinh Nguyen wrote:
>
>> On Wed, Apr 6, 2016 at 10:42 AM, Edzer Pebesma
>> <edzer.pebe...@uni-muenster.de> wrote:
>>>
>>> When I replace the .prj file of an arbitrary shapefile with these
>>> contents, it gets read well. Have you tried with current R (3.2.4) and
>>> rgdal (1.1.7)?
>>
>>
>>
>> Thanks Edzer, upgrading rgdal to 1.1.8 works.  Consider this issue close.
>
>
> For completeness, which OS was this, and how was GDAL installed - do you
recall what GDAL version was reported when rgdal loaded?
>
> If Windows CRAN binary, it is possible that the underlying GDAL version
of the spatial reference code and EPSG databases changed - there were no
relevant changes in rgdal itself, I think.
>

Windows. I don't have the log with gdal version. Your assessment is
probably correct. Thanks.

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

On Wed, Apr 6, 2016 at 10:42 AM, Edzer Pebesma
 wrote:
> When I replace the .prj file of an arbitrary shapefile with these
> contents, it gets read well. Have you tried with current R (3.2.4) and
> rgdal (1.1.7)?


Thanks Edzer, upgrading rgdal to 1.1.8 works.  Consider this issue close.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

On Apr 6, 2016 10:27 AM, "Edzer Pebesma" 
wrote:
>
> What are the contents of the 2013Spring_Flood_Drawn.prj file?
>

Here's the content of the prj file:
PROJCS["WGS_1984_Web_Mercator_Auxiliary_Sphere",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Mercator_Auxiliary_Sphere"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],PARAMETER["Standard_Parallel_1",0.0],PARAMETER["Auxiliary_Sphere_Type",0.0],UNIT["Meter",1.0]]

Thanks so much for your help.

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

On Wed, Apr 6, 2016 at 8:58 AM, Denis mutiibwa  wrote:
> readOGR(dir.path, "argv2")
> it seems "argv2" has to be the shapefile name not a variable name


My 'floodshapedir' and 'shapebase' are both variables with path stored
in.  Here's the function call with the strings explicitly passed.  I
don't think it's an issue with these paths.

> dFloodRisk <- readOGR('FloodShapes', '2013Spring_Flood_Drawn', verbose=TRUE)
OGR data source with driver: ESRI Shapefile
Source: "FloodShapes", layer: "2013Spring_Flood_Drawn"
with 8 features
It has 2 fields
> proj4string(dFloodRisk)
[1] NA

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

On Wed, Apr 6, 2016 at 8:41 AM, Denis mutiibwa  wrote:
> Where is "shapebase" coming from?
> If those are the files in the directory, your scripts should as below:
> dir.path <- directory path
> dFloodRisk <- readOGR(dir.path, "2014Spring_Flood_Drawn")
> proj4string(dFloodRisk)

I'm reading the files in exactly as you are suggesting (floodshapedir
and shapebase are variables) as seen in the output below:

> dFloodRisk <- readOGR(floodshapedir, shapebase)
OGR data source with driver: ESRI Shapefile
Source: "FloodShapes", layer: "2013Spring_Flood_Drawn"

Just not sure why the prj file is not being read in.  Thanks.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

On Wed, Apr 6, 2016 at 8:24 AM, Denis mutiibwa  wrote:
> Hi Vinh,
> proj4string function is giving you NA because "shapebase" shapefile doesn't
> have an auxiliary file "shapebase.prj"
>
> Denis Mutiibwa,


Thanks for your response Denis.  The following files are all in the directory:
2014Spring_Flood_Drawn.cpg  2014Spring_Flood_Drawn.sbx
2014Spring_Flood_Drawn.dbf  2014Spring_Flood_Drawn.shp
2014Spring_Flood_Drawn.prj  2014Spring_Flood_Drawn.shp.xml
2014Spring_Flood_Drawn.sbn  2014Spring_Flood_Drawn.shx

So, the prj file is there...or am I misunderstanding?  Thanks.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[R-sig-Geo] can't read in shapefile

[Vinh Nguyen]

Hi,

I have two shapefiles, both generated from ArcGIS using similar
methods.  I was able to read in one file using readOGR but not the
other.  Any thoughts on how best to figure out why I can't read in the
file?  Here's the error:

> dFloodRisk <- readOGR(floodshapedir, shapefilebase, verbose=TRUE)
Error in ogrInfo(dsn = dsn, layer = layer, encoding = encoding,
use_iconv = use_iconv,  :
  Cannot open layer
> traceback()
5: .Call("ogrInfo", as.character(dsn), as.character(layer), PACKAGE = "rgdal")
4: ogrInfo(dsn = dsn, layer = layer, encoding = encoding, use_iconv = use_iconv,
   swapAxisOrder = swapAxisOrder, require_geomType = require_geomType)
3: withCallingHandlers(expr, message = function(c)
invokeRestart("muffleMessage"))
2: suppressMessages(ogr_info <- ogrInfo(dsn = dsn, layer = layer,
   encoding = encoding, use_iconv = use_iconv, swapAxisOrder =
swapAxisOrder,
   require_geomType = require_geomType))
1: readOGR(floodshapedir, shapefilebase,
   verbose = TRUE)

Thanks for your help.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[R-sig-Geo] readOGR not reading in prj file?

[Vinh Nguyen]

Hi,

I'm currently using rgdal v 1.1-1 on R 3.2.2.  It appears my
projection file is not being read in by readOGR:

> dFloodRisk <- readOGR(floodshapedir, shapebase)
OGR data source with driver: ESRI Shapefile
Source: "FloodShapes", layer: "2013Spring_Flood_Drawn"
with 8 features
It has 2 fields
> proj4string(dFloodRisk)
[1] NA

I have to manually pass in the CRS:
> dFloodRisk <- readOGR(floodshapedir, paste0(curYear, 'Spring_Flood_Drawn'), 
> p4s='+init=EPSG:3857')
OGR data source with driver: ESRI Shapefile
Source: "FloodShapes", layer: "2013Spring_Flood_Drawn"
with 8 features
It has 2 fields
> proj4string(dFloodRisk)
[1] "+init=EPSG:3857 +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0
+lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs"

Any thoughts on why readOGR doesn't automatically read in the prj
file?  Here's the content of the prj file:
PROJCS["WGS_1984_Web_Mercator_Auxiliary_Sphere",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Mercator_Auxiliary_Sphere"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],PARAMETER["Standard_Parallel_1",0.0],PARAMETER["Auxiliary_Sphere_Type",0.0],UNIT["Meter",1.0]]

Thanks so much for your help.

-- Vinh

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo