[R] web scraping image
I'm working on a script that downloads data from the USGS NWIS server. dataRetrieval makes it easy to quickly get the data in a neat tabular format, but I was also interested in getting the tabular text files - also fairly easy for me using download.file. However, I'm not skilled enough to work out how to download the nice graphic files that can be produced dynamically from the USGS NWIS server (for example: http://nwis.waterdata.usgs.gov/nwis/peak?site_no=12144500&agency_cd=USGS&format=img) My question is how do I get the image from this web page and save it to a local directory? scrapeR returns the information from the page and I suspect this is a possible solution path, but I don't know what the next step is. My code provided below works from a list I've created of USGS flow gauging stations. Curtis ## Code to process USGS daily flow data for high and low flow analysis ## Need to start with list of gauge ids to process ## Can't figure out how to automate download of images require(dataRetrieval) require(data.table) require(scrapeR) df <- read.csv("usgs_stations.csv", header=TRUE) lstas <-length(df$siteno) #length of locator list print(paste('Processsing...',df$name[1],' ',df$siteno[1], sep = "")) datall <- readNWISpeak(df$siteno[1]) for (a in 2:lstas) { # Print station being processed print(paste('Processsing...',df$name[a],' ',df$siteno[a], sep = "")) dat<- readNWISpeak(df$siteno[a]) datall <- rbind(datall,dat) } write.csv(datall, file = "usgs_peaks.csv") # Retrieve ascii text files and graphics for (a in 1:lstas) { print(paste('Processsing...',df$name[1],' ',df$siteno[1], sep = "")) graphic.url <- paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',df$siteno[a],'&agency_cd=USGS&format=img', sep = "") peakfq.url <- paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',df$siteno[a],'&agency_cd=USGS&format=hn2', sep = "") tab.url <- paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',df$siteno[a],'&agency_cd=USGS&format=rdb', sep = "") graphic.fn <- paste('graphic_',df$siteno[a],'.gif', sep = "") peakfq.fn <- paste('peakfq_',df$siteno[a],'.txt', sep = "") tab.fn <- paste('tab_',df$siteno[a],'.txt', sep = "") download.file(graphic.url,graphic.fn,mode='wb') # This apparently doesn't work - file is empty download.file(peakfq.url,peakfq.fn) download.file(tab.url,tab.fn) } # scrapeR pageSource<-scrape(url="http://nwis.waterdata.usgs.gov/nwis/peak?site_no=12144500&agency_cd=USGS&format=img",headers=TRUE, parse=FALSE) page<-scrape(object="pageSource") __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] web scraping image
Hi Chris, I don't have the packages you are using, but tracing this indicates that the page source contains the relative path of the graphic, in this case: /nwisweb/data/img/USGS.12144500.19581112.20140309..0.peak.pres.gif and you already have the server URL: nwis.waterdata.usgs.gov getting the path out of the page source isn't difficult, just split the text at double quotes and get the token following "img src=". If I understand the arguments of "download.file" correctly, the path is the graphic.fn argument and the server URL is the graphic.url argument. I would paste them together and display the result to make sure that it matches the image you want. When I did this, the correct image appeared in my browser. I'm using Google Chrome, so I don't have to prepend the http:// Jim On Fri, Jun 5, 2015 at 2:31 AM, Curtis DeGasperi wrote: > I'm working on a script that downloads data from the USGS NWIS server. > dataRetrieval makes it easy to quickly get the data in a neat tabular > format, but I was also interested in getting the tabular text files - > also fairly easy for me using download.file. > > However, I'm not skilled enough to work out how to download the nice > graphic files that can be produced dynamically from the USGS NWIS > server (for example: > http://nwis.waterdata.usgs.gov/nwis/peak?site_no=12144500&agency_cd=USGS&format=img) > > My question is how do I get the image from this web page and save it > to a local directory? scrapeR returns the information from the page > and I suspect this is a possible solution path, but I don't know what > the next step is. > > My code provided below works from a list I've created of USGS flow > gauging stations. > > Curtis > > ## Code to process USGS daily flow data for high and low flow analysis > ## Need to start with list of gauge ids to process > ## Can't figure out how to automate download of images > > require(dataRetrieval) > require(data.table) > require(scrapeR) > > df <- read.csv("usgs_stations.csv", header=TRUE) > > lstas <-length(df$siteno) #length of locator list > > print(paste('Processsing...',df$name[1],' ',df$siteno[1], sep = "")) > > datall <- readNWISpeak(df$siteno[1]) > > for (a in 2:lstas) { > # Print station being processed > print(paste('Processsing...',df$name[a],' ',df$siteno[a], sep = "")) > > dat<- readNWISpeak(df$siteno[a]) > > datall <- rbind(datall,dat) > > } > > write.csv(datall, file = "usgs_peaks.csv") > > # Retrieve ascii text files and graphics > > for (a in 1:lstas) { > > print(paste('Processsing...',df$name[1],' ',df$siteno[1], sep = "")) > > graphic.url <- > paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',df$siteno[a],'&agency_cd=USGS&format=img', > sep = "") > peakfq.url <- > paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',df$siteno[a],'&agency_cd=USGS&format=hn2', > sep = "") > tab.url <- > paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',df$siteno[a],'&agency_cd=USGS&format=rdb', > sep = "") > > graphic.fn <- paste('graphic_',df$siteno[a],'.gif', sep = "") > peakfq.fn <- paste('peakfq_',df$siteno[a],'.txt', sep = "") > tab.fn <- paste('tab_',df$siteno[a],'.txt', sep = "") > > download.file(graphic.url,graphic.fn,mode='wb') # This apparently > doesn't work - file is empty > download.file(peakfq.url,peakfq.fn) > download.file(tab.url,tab.fn) > } > > # scrapeR > pageSource<-scrape(url="http://nwis.waterdata.usgs.gov/nwis/peak?site_no=12144500&agency_cd=USGS&format=img",headers=TRUE, > parse=FALSE) > page<-scrape(object="pageSource") > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] web scraping image
Thanks to Jim's prompting, I think I came up with a fairly painless way to parse the HTML without having to write any parsing code myself using the function getHTMLExternalFiles in the XML package. A working version of the code follows: ## Code to process USGS peak flow data require(dataRetrieval) require(XML) ## Need to start with list of gauge ids to process siteno <- c('12142000','12134500','12149000') lstas <-length(siteno) #length of locator list print(paste('Processsing...',siteno[1],' ',siteno[1], sep = "")) datall <- readNWISpeak(siteno[1]) for (a in 2:lstas) { # Print station being processed print(paste('Processsing...',siteno[a], sep = "")) dat<- readNWISpeak(siteno[a]) datall <- rbind(datall,dat) } write.csv(datall, file = "usgs_peaks.csv") # Retrieve ascii text files and graphics for (a in 1:lstas) { print(paste('Processsing...',siteno[a], sep = "")) graphic.url <- paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',siteno[a],'&agency_cd=USGS&format=img', sep = "") usgs.img <- getHTMLExternalFiles(graphic.url) graphic.img <- paste('http://nwis.waterdata.usgs.gov',usgs.img, sep = "") peakfq.url <- paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',siteno[a],'&agency_cd=USGS&format=hn2', sep = "") tab.url <- paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',siteno[a],'&agency_cd=USGS&format=rdb', sep = "") graphic.fn <- paste('graphic_',siteno[a],'.gif', sep = "") peakfq.fn <- paste('peakfq_',siteno[a],'.txt', sep = "") tab.fn <- paste('tab_',siteno[a],'.txt', sep = "") download.file(graphic.img,graphic.fn,mode='wb') download.file(peakfq.url,peakfq.fn) download.file(tab.url,tab.fn) } > -- > > Message: 34 > Date: Fri, 5 Jun 2015 08:59:04 +1000 > From: Jim Lemon > To: Curtis DeGasperi > Cc: r-help mailing list > Subject: Re: [R] web scraping image > Message-ID: > < ca+8x3fv0ajw+e22jayv1gfm6jr_tazua5fwgd3t_mfgfqy2...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Hi Chris, > I don't have the packages you are using, but tracing this indicates > that the page source contains the relative path of the graphic, in > this case: > > /nwisweb/data/img/USGS.12144500.19581112.20140309..0.peak.pres.gif > > and you already have the server URL: > > nwis.waterdata.usgs.gov > > getting the path out of the page source isn't difficult, just split > the text at double quotes and get the token following "img src=". If I > understand the arguments of "download.file" correctly, the path is the > graphic.fn argument and the server URL is the graphic.url argument. I > would paste them together and display the result to make sure that it > matches the image you want. When I did this, the correct image > appeared in my browser. I'm using Google Chrome, so I don't have to > prepend the http:// > > Jim > > On Fri, Jun 5, 2015 at 2:31 AM, Curtis DeGasperi > wrote: >> I'm working on a script that downloads data from the USGS NWIS server. >> dataRetrieval makes it easy to quickly get the data in a neat tabular >> format, but I was also interested in getting the tabular text files - >> also fairly easy for me using download.file. >> >> However, I'm not skilled enough to work out how to download the nice >> graphic files that can be produced dynamically from the USGS NWIS >> server (for example: >> http://nwis.waterdata.usgs.gov/nwis/peak?site_no=12144500&agency_cd=USGS&format=img ) >> >> My question is how do I get the image from this web page and save it >> to a local directory? scrapeR returns the information from the page >> and I suspect this is a possible solution path, but I don't know what >> the next step is. >> >> My code provided below works from a list I've created of USGS flow >> gauging stations. >> >> Curtis >> >> ## Code to process USGS daily flow data for high and low flow analysis >> ## Need to start with list of gauge ids to process >> ## Can't figure out how to automate download of images >> >> require(dataRetrieval) >> require(data.table) >> require(scrapeR) >> >> df <- read.csv("usgs_stations.csv", header=TRUE) >> >> lstas <-length(df$siteno) #length of locator list >> >> print(paste('P
Re: [R] web scraping image
You can also do it with rvest & httr (but that does involve some "parsing"): library(httr) library(rvest) url <- "http://nwis.waterdata.usgs.gov/nwis/peak?site_no=12144500&agency_cd=USGS&format=img"; html(url) %>% html_nodes("img") %>% html_attr("src") %>% paste0("http://nwis.waterdata.usgs.gov";, .) %>% GET(write_disk("12144500.gif")) -> status Very readable and can be made programmatic pretty easily, too. Plus: avoids direct use of the XML library. Future versions will no doubt swap xml2 for XML as well. -Bob On Mon, Jun 8, 2015 at 2:09 PM, Curtis DeGasperi wrote: > Thanks to Jim's prompting, I think I came up with a fairly painless way to > parse the HTML without having to write any parsing code myself using the > function getHTMLExternalFiles in the XML package. A working version of the > code follows: > > ## Code to process USGS peak flow data > > require(dataRetrieval) > require(XML) > > ## Need to start with list of gauge ids to process > > siteno <- c('12142000','12134500','12149000') > > lstas <-length(siteno) #length of locator list > > print(paste('Processsing...',siteno[1],' ',siteno[1], sep = "")) > > datall <- readNWISpeak(siteno[1]) > > for (a in 2:lstas) { > # Print station being processed > print(paste('Processsing...',siteno[a], sep = "")) > > dat<- readNWISpeak(siteno[a]) > > datall <- rbind(datall,dat) > > } > > write.csv(datall, file = "usgs_peaks.csv") > > # Retrieve ascii text files and graphics > for (a in 1:lstas) { > > print(paste('Processsing...',siteno[a], sep = "")) > > graphic.url <- > paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',siteno[a],'&agency_cd=USGS&format=img', > sep = "") > usgs.img <- getHTMLExternalFiles(graphic.url) > graphic.img <- paste('http://nwis.waterdata.usgs.gov',usgs.img, sep = "") > > peakfq.url <- > paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',siteno[a],'&agency_cd=USGS&format=hn2', > sep = "") > tab.url <- > paste('http://nwis.waterdata.usgs.gov/nwis/peak?site_no=',siteno[a],'&agency_cd=USGS&format=rdb', > sep = "") > > graphic.fn <- paste('graphic_',siteno[a],'.gif', sep = "") > peakfq.fn <- paste('peakfq_',siteno[a],'.txt', sep = "") > tab.fn <- paste('tab_',siteno[a],'.txt', sep = "") > download.file(graphic.img,graphic.fn,mode='wb') > download.file(peakfq.url,peakfq.fn) > download.file(tab.url,tab.fn) > } > >> -- >> >> Message: 34 >> Date: Fri, 5 Jun 2015 08:59:04 +1000 >> From: Jim Lemon >> To: Curtis DeGasperi >> Cc: r-help mailing list >> Subject: Re: [R] web scraping image >> Message-ID: >> < > ca+8x3fv0ajw+e22jayv1gfm6jr_tazua5fwgd3t_mfgfqy2...@mail.gmail.com> >> Content-Type: text/plain; charset=UTF-8 >> >> Hi Chris, >> I don't have the packages you are using, but tracing this indicates >> that the page source contains the relative path of the graphic, in >> this case: >> >> /nwisweb/data/img/USGS.12144500.19581112.20140309..0.peak.pres.gif >> >> and you already have the server URL: >> >> nwis.waterdata.usgs.gov >> >> getting the path out of the page source isn't difficult, just split >> the text at double quotes and get the token following "img src=". If I >> understand the arguments of "download.file" correctly, the path is the >> graphic.fn argument and the server URL is the graphic.url argument. I >> would paste them together and display the result to make sure that it >> matches the image you want. When I did this, the correct image >> appeared in my browser. I'm using Google Chrome, so I don't have to >> prepend the http:// >> >> Jim >> >> On Fri, Jun 5, 2015 at 2:31 AM, Curtis DeGasperi >> wrote: >>> I'm working on a script that downloads data from the USGS NWIS server. >>> dataRetrieval makes it easy to quickly get the data in a neat tabular >>> format, but I was also interested in getting the tabular text files - >>> also fairly easy for me using download.file. >>> >>> However, I'm not skilled enough to work out how to download the nice >>> graph