Re: [R-sig-Geo] geodata package: CMIP6 climate model data downloading error

[Bede-Fazekas Ákos]

Dear Manuel,

There are some broken links in the WorldClim website. If you find one, 
you can write to i...@worldclim.org. Although I have negative 
experiences, you should try this way.

Anyway, the global, non-tiled raster can be downloaded from this link:
https://geodata.ucdavis.edu/cmip6/30s/ACCESS-CM2/ssp126/wc2.1_30s_bioc_ACCESS-CM2_ssp126_2021-2040.tif

Have a nice week,
Ákos

_
Ákos Bede-Fazekas
Centre for Ecological Research, Hungary

2023.01.24. 23:29 keltezéssel, Manuel Spínola írta:

Dear list members,

I am trying to download CMIP6 climate model data using the R package
geodata but I got this error message:

bio <- cmip6_tile(lon = -84, lat = 10,"ACCESS-CM2", "126", "2021-2040",
var="bioc", path = "climate_ckange")
trying URL '
https://geodata.ucdavis.edu/cmip6/tiles/ACCESS-CM2/ssp126/wc2.1_30s_bioc_ACCESS-CM2_ssp126_2021-2040_tile-28.tif
'
Error in utils::download.file(url = url, destfile = filename, quiet =
quiet,  :
   cannot open URL '
https://geodata.ucdavis.edu/cmip6/tiles/ACCESS-CM2/ssp126/wc2.1_30s_bioc_ACCESS-CM2_ssp126_2021-2040_tile-28.tif
'
download failed

I cannot even download the data using the worldclim website.




___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[R-sig-Geo] geodata package: CMIP6 climate model data downloading error

[Manuel Spínola]

Dear list members,

I am trying to download CMIP6 climate model data using the R package
geodata but I got this error message:

bio <- cmip6_tile(lon = -84, lat = 10,"ACCESS-CM2", "126", "2021-2040",
var="bioc", path = "climate_ckange")
trying URL '
https://geodata.ucdavis.edu/cmip6/tiles/ACCESS-CM2/ssp126/wc2.1_30s_bioc_ACCESS-CM2_ssp126_2021-2040_tile-28.tif
'
Error in utils::download.file(url = url, destfile = filename, quiet =
quiet,  :
  cannot open URL '
https://geodata.ucdavis.edu/cmip6/tiles/ACCESS-CM2/ssp126/wc2.1_30s_bioc_ACCESS-CM2_ssp126_2021-2040_tile-28.tif
'
download failed

I cannot even download the data using the worldclim website.


-- 
*Manuel Spínola, Ph.D.*
Instituto Internacional en Conservación y Manejo de Vida Silvestre
Universidad Nacional
Apartado 1350-3000
Heredia
COSTA RICA
mspin...@una.cr 
mspinol...@gmail.com
Teléfono: (506) 8706 - 4662
Institutional website: ICOMVIS

Blog sobre Ciencia de Datos: https://mspinola-ciencia-de-datos.netlify.app

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] Extract CRU data

[Grzegorz Sapijaszko]

On Tue, 2023-01-24 at 12:13 +0100, Miluji Sb wrote:
> Greetings everyone,
> 
> I have a question on extracting country-level data from CRU (
> https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.06/crucy.2205251923.v4.06/countries/tmp/
> ).


Something like:

To get all links/filenames in one table:

a <-
rvest::read_html("https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.06/crucy.2205251923.v4.06/countries/tmp/
") 

tbl <- a |>
  rvest::html_table() |>
  as.data.frame()

tbl <- tbl[-c(1,2),]

To download them all to specific directory

my_download_function <- function(myurl ="", output_dir = "data") {
  if(!dir.exists({{output_dir}})) {dir.create({{output_dir}})}
  .destfile = paste0({{output_dir}}, "/", {{myurl}})
  .myurl <-
paste0("https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.06/crucy.2205251923.v4.06/countries/tmp/
", {{myurl}})
  download.file(url = .myurl, destfile = .destfile, method = "wget",
extra = "-c --progress=bar:force")  
  NULL
}

invisible(lapply(seq(nrow(tbl)), function(i)
my_download_function(tbl[i,1], "data")))

Now, having it locally you can read them one by one with read.csv,
like:

f <- list.files(path = "data", pattern = "crucy*", full.names = TRUE)
read.csv(f[i], skip = 3, header = TRUE)

It doesn't make sense without adding additional information about
country/territotry, but at least you have starting point.

Regards,
Grzegorz

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] strange issue of memory converting a raster to a data frame

[Sebastiano Trevisani]

Dear Vlad and Loic,


Some more info on the issue; the error ("Error: cannot allocate vector of
size 1.7 Gb") shows a random behaviour, sometimes things work (e.g., after
a reboot) and sometimes not.

 Anyway this is a reproducible example:

theTest=rast(nrows=5000, ncols=5000, nlyrs=18, xmin=0, ymin=0,
vals=.333)
mem_info(theTest)
theDataFrame=as.data.frame(theTest)



Sebastiano




On Mon, Jan 23, 2023 at 7:18 PM Sebastiano Trevisani 
wrote:

> Thank you Vlad!
> This is a good solution, I keep it for future.
>
> My issue is that I would like to understand why things don't work in the
> best performing PC.
>
> I'll do more experiments, to isolate the issue.
>
> Sebastiano
>
>
>
> Sebastiano
>
>
>
> Il lun 23 gen 2023, 18:48 Vlad Amihaesei  ha
> scritto:
>
>> You can try converting each layer to data frame instead of all the layers
>> at once time. You can do this using a loop.
>>
>> r <- terra::rast(file.nc)
>>
>> df<- NULL
>>
>> for( i in 1:nlyr(r)){
>> r1 <-  r[[i]]
>> r1.df <- as.data.frame(r1,xy =T)
>> df <- rbind(df, r1.df)
>> }
>>
>> Best  regards
>>
>> Vlad Alexandru AMIHAESEI
>>
>>
>> On Mon, Jan 23, 2023 at 10:33 AM Sebastiano Trevisani 
>> wrote:
>>
>>> Hi  Loïc,
>>>
>>> thank you for your suggestion! this was one of the possibilities I
>>> checked.
>>> However, I tried with terra.options() to change the memfrac parameter
>>> from
>>> 0.6 to 0.1, so as to force the writing on disc, but things didn't work.
>>> It seems like it is working as 32 bit system, but the R the installation
>>> seems right to 64. Probably I'm missing something very obvious.
>>> I will try to build a reproducible example without the need to upload all
>>> the file.
>>>
>>> Thanks again,
>>>
>>> Sebastiano
>>>
>>> *  Sebastiano Trevisani, Ph.D.*
>>> *Associate Professor*
>>> *Applied and Environmental Geology*
>>>
>>>
>>> *  https://orcid.org/-0001-8436-7798
>>> *
>>> *IUAV University of Venice: www.iuav.it *
>>>
>>> *Address: Dorsoduro 2206,  Venice 30123, Italy Tel:+39. 041. 257
>>> 1299Mail:strevis...@iuav.it  *
>>> *"Le opinioni espresse sono riferibili esclusivamente all'autore e non *
>>> * riflettono in alcun modo una posizione ufficiale dello IUAV "*
>>> *"The views expressed are purely those of the writer and may not in
>>> any circumstances be regarded as stating an official position of the
>>> IUAV."*
>>>
>>>
>>> On Mon, Jan 23, 2023 at 5:36 PM DUTRIEUX Loic <
>>> loic.dutri...@ec.europa.eu>
>>> wrote:
>>>
>>> > Hi Sebastiano,
>>> >
>>> > Try comparing terra::terraOptions() and terra::mem_info(r) on both
>>> > systems. Just guessing there but it could be that terra is trying to do
>>> > everything in memory and failing on your 16GB system, while taking a
>>> memory
>>> > friendly route on your 8GB system.
>>> >
>>> > Cheers,
>>> > Loïc
>>> > 
>>> > From: R-sig-Geo  on behalf of
>>> Sebastiano
>>> > Trevisani 
>>> > Sent: 23 January 2023 12:30:14
>>> > To: r-sig-geo
>>> > Subject: [R-sig-Geo] strange issue of memory converting a raster to a
>>> data
>>> > frame
>>> >
>>> > Dear list members,
>>> >
>>> > In converting a raster as the one below to a data frame, I have an
>>> issue of
>>> > memory allocation only when I use a specific pc, and in particular the
>>> one
>>> > with more memory. To be more precise (using same version of R 4.2.2 and
>>> > Terra terra 1.6.47, by means of as.data.frame() function):
>>> >
>>> > with a I5 processor with 8Gb of memory, windows 10 it works
>>> >
>>> > with a I7 processor with 16gb of memory, windows 11 it doesn't
>>> >
>>> > I'm not able to figure out a possible explanation!
>>> > Any suggestions on what to check?
>>> >
>>> >
>>> > SpatRaster
>>> > dimensions  : 5000, 5000, 18  (nrow, ncol, nlyr)
>>> > resolution  : 30, 30  (x, y)
>>> > extent  : -8526.878, 141473.1, 4110588, 4260588  (xmin, xmax, ymin,
>>> > ymax)
>>> > coord. ref. : WGS 84 / UTM zone 45N (EPSG:32645)
>>> > source  : multyMadRes2c.tif
>>> >
>>> >
>>> >
>>> > Sebastiano
>>> >
>>> > *  Sebastiano Trevisani, Ph.D.*
>>> > *Associate Professor*
>>> > *Applied and Environmental Geology*
>>> >
>>> >
>>> > *
>>> >
>>> https://urldefense.com/v3/__https://orcid.org/-0001-8436-7798__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9YxpzJmj$
>>> > <
>>> >
>>> https://urldefense.com/v3/__https://orcid.org/-0001-8436-7798__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9YxpzJmj$
>>> > >*
>>> > *IUAV University of Venice:
>>> >
>>> https://urldefense.com/v3/__http://www.iuav.it__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9ag_1m0F$
>>> > <
>>> >
>>> https://urldefense.com/v3/__http://www.iuav.it/__;!!DOxrgLBm!H86VAGwO3fNqAu4LS2hmv13aheZ9hGMiMmD7ZTN2sVxV-iuyV04Q_dV_lK5KNQauD3mFclNlC8a2LSlyZehq9cV-rY9s$
>>> > >*
>>> >
>

Re: [R-sig-Geo] Extract CRU data

[Barry Rowlingson]

Are you asking if there's a way to automate the download of a list of links
from that page? You could write an R script to get the HTML, then find all
the HTML  tags, and then get the URLs in the link addresses, and there's
packages for doing this kind of web scraping.

But for this kind of thing it might be easier to use a web browser add-on -
I have "Down Them All" set up on Firefox, and with a click or two I can get
a list of all the link URLs and hit a button that downloads everything to a
single folder. Once done, I can use standard R functions to list all the
downloaded files and read them. Took about 20 seconds to do for this page,
and now I have a folder of 292 .tmp.per files.

Barry


On Tue, Jan 24, 2023 at 11:13 AM Miluji Sb  wrote:

> Greetings everyone,
>
> I have a question on extracting country-level data from CRU (
>
> https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.06/crucy.2205251923.v4.06/countries/tmp/
> ).
> The data for each variable are available for individual countries and I am
> struggling to download all of them. Can I extract all the files in R then
> merge? Thanks so much.
>
> Best,
>
> Milu
>
> [[alternative HTML version deleted]]
>
> ___
> R-sig-Geo mailing list
> R-sig-Geo@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

[R-sig-Geo] Extract CRU data

[Miluji Sb]

Greetings everyone,

I have a question on extracting country-level data from CRU (
https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.06/crucy.2205251923.v4.06/countries/tmp/).
The data for each variable are available for individual countries and I am
struggling to download all of them. Can I extract all the files in R then
merge? Thanks so much.

Best,

Milu

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo