[R] Listing the contents of an FTP directory via R?
R-helpers: I'd like to be able to store all the file information from an ftp site (e.g. file and foldernames) through an R command. Any ideas how to do this? Here's an example site to use: ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005 --j -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Listing the contents of an FTP directory via R?
A couple of ways. using Rcurl you can use the curlOption of dirlistonly. otherwise you can read the page and parse. I've got some code around here to do that. Steve On Mon, Apr 9, 2012 at 11:27 AM, Jonathan Greenberg j...@illinois.eduwrote: R-helpers: I'd like to be able to store all the file information from an ftp site (e.g. file and foldernames) through an R command. Any ideas how to do this? Here's an example site to use: ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005 --j -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Listing the contents of an FTP directory via R?
Steven: Thanks -- I seem to be running into the problem with the link I sent along: getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,dirlistonly = TRUE) Error in function (type, msg, asError = TRUE) : RETR response: 550 I'm wondering if it might be a passive ftp issue, but: getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,ftp.use.epsv=TRUE, dirlistonly = TRUE) Error in function (type, msg, asError = TRUE) : FTP response reading failed Does not seem to work... Thoughts? --j On Mon, Apr 9, 2012 at 1:32 PM, steven mosher mosherste...@gmail.com wrote: A couple of ways. using Rcurl you can use the curlOption of dirlistonly. otherwise you can read the page and parse. I've got some code around here to do that. Steve On Mon, Apr 9, 2012 at 11:27 AM, Jonathan Greenberg j...@illinois.edu wrote: R-helpers: I'd like to be able to store all the file information from an ftp site (e.g. file and foldernames) through an R command. Any ideas how to do this? Here's an example site to use: ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005 --j -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Listing the contents of an FTP directory via R?
Ya I hit the same error with my code that reads directories. I'll try some other stuff . I think I hit this error before I with usgs. On Mon, Apr 9, 2012 at 11:40 AM, Jonathan Greenberg j...@illinois.eduwrote: Steven: Thanks -- I seem to be running into the problem with the link I sent along: getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,dirlistonly = TRUE) Error in function (type, msg, asError = TRUE) : RETR response: 550 I'm wondering if it might be a passive ftp issue, but: getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,ftp.use.epsv=TRUE, dirlistonly = TRUE) Error in function (type, msg, asError = TRUE) : FTP response reading failed Does not seem to work... Thoughts? --j On Mon, Apr 9, 2012 at 1:32 PM, steven mosher mosherste...@gmail.com wrote: A couple of ways. using Rcurl you can use the curlOption of dirlistonly. otherwise you can read the page and parse. I've got some code around here to do that. Steve On Mon, Apr 9, 2012 at 11:27 AM, Jonathan Greenberg j...@illinois.edu wrote: R-helpers: I'd like to be able to store all the file information from an ftp site (e.g. file and foldernames) through an R command. Any ideas how to do this? Here's an example site to use: ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005 --j -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Listing the contents of an FTP directory via R?
Its kinda gross, but you can just download.file() and then parse the result !DOCTYPE HTML PUBLIC -//IETF//DTD HTML//EN HTML HEAD TITLEFTP directory /MOTA/MCD15A3.005 at e4ftl01.cr.usgs.gov/TITLE /HEAD BODY H2 ID=WinINetFtpDirectoryFTP directory /MOTA/MCD15A3.005 at e4ftl01.cr.usgs.gov/H2 HR H4PRE ## ## *** WARNING TO USERS OF THIS SYSTEM *** THIS COMPUTER SYSTEM, INCLUDING ALL RELATED EQUIPMENT, NETWORKS, AND NETWORK DEVICES (INCLUDING INTERNET ACCESS), IS PROVIDED BY THE DEPARTMENT OF THE INTERIOR (DOI) IN ACCORDANCE WITH THE AGENCY POLICY FOR OFFICIAL USE AND LIMITED PERSONAL USE. ALL AGENCY COMPUTER SYSTEMS MAY BE MONITORED FOR ALL LAWFUL PURPOSES, INCLUDING BUT NOT LIMITED TO, ENSURING THAT USE IS AUTHORIZED, FOR MANAGEMENT OF THE SYSTEM, TO FACILITATE PROTECTION AGAINST UNAUTHORIZED ACCESS, AND TO VERIFY SECURITY PROCEDURES, SURVIVABILITY AND OPERATIONAL SECURITY. ANY INFORMATION ON THIS COMPUTER SYSTEM MAY BE EXAMINED, RECORDED, COPIED AND USED FOR AUTHORIZED PURPOSES AT ANY TIME. ALL INFORMATION, INCLUDING PERSONAL INFORMATION, PLACED OR SENT OVER THIS SYSTEM MAY BE MONITORED, AND USERS OF THIS SYSTEM ARE REMINDED THAT SUCH MONITORING DOES OCCUR. THEREFORE, THERE SHOULD BE NO EXPECTATION OF PRIVACY WITH RESPECT TO USE OF THIS SYSTEM. BY LOGGING INTO THIS AGENCY COMPUTER SYSTEM, YOU ACKNOWLEDGE AND CONSENT TO THE MONITORING OF THIS SYSTEM. EVIDENCE OF YOUR USE, AUTHORIZED OR UNAUTHORIZED, COLLECTED DURING MONITORING MAY BE USED FOR CIVIL, CRIMINAL, ADMINISTRATIVE, OR OTHER ADVERSE ACTION. UNAUTHORIZED OR ILLEGAL USE MAY SUBJECT YOU TO PROSECUTION. ## ## /PRE/H4 HR A HREF=..Up to higher level directory/ABRPRE 10/18/2011 05:23PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.04/B2002.07.04/B/A 10/18/2011 05:08PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.08/B2002.07.08/B/A 10/18/2011 05:05PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.12/B2002.07.12/B/A 10/18/2011 06:00PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.16/B2002.07.16/B/A 10/18/2011 06:00PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.20/B2002.07.20/B/A 10/18/2011 06:00PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.24/B2002.07.24/B/A 10/18/2011 06:01PM Directory A HREF=/MOTA/MCD15A3.005/2002.07.28/B2002.07.28/B/A 10/18/2011 06:01PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.05/B2002.08.05/B/A 10/18/2011 06:01PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.09/B2002.08.09/B/A 10/18/2011 05:55PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.13/B2002.08.13/B/A 10/18/2011 05:54PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.17/B2002.08.17/B/A 10/18/2011 05:37PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.21/B2002.08.21/B/A 10/18/2011 04:26PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.25/B2002.08.25/B/A 10/18/2011 05:19PM Directory A HREF=/MOTA/MCD15A3.005/2002.08.29/B2002.08.29/B/A 10/18/2011 05:19PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.02/B2002.09.02/B/A 10/18/2011 05:14PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.06/B2002.09.06/B/A 10/18/2011 03:39PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.10/B2002.09.10/B/A 10/18/2011 05:23PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.14/B2002.09.14/B/A 10/18/2011 05:25PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.18/B2002.09.18/B/A 10/18/2011 05:25PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.22/B2002.09.22/B/A 10/18/2011 05:21PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.26/B2002.09.26/B/A 10/18/2011 05:59PM Directory A HREF=/MOTA/MCD15A3.005/2002.09.30/B2002.09.30/B/A 10/18/2011 06:00PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.04/B2002.10.04/B/A 10/18/2011 04:37PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.08/B2002.10.08/B/A 10/18/2011 04:03PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.12/B2002.10.12/B/A 10/18/2011 06:00PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.16/B2002.10.16/B/A 10/18/2011 05:36PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.20/B2002.10.20/B/A 10/18/2011 03:54PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.24/B2002.10.24/B/A 10/18/2011 03:54PM Directory A HREF=/MOTA/MCD15A3.005/2002.10.28/B2002.10.28/B/A 10/18/2011 03:50PM Directory A HREF=/MOTA/MCD15A3.005/2002.11.01/B2002.11.01/B/A 10/18/2011 01:25PM Directory A HREF=/MOTA/MCD15A3.005/2002.11.05/B2002.11.05/B/A 10/18/2011 01:27PM Directory A HREF=/MOTA/MCD15A3.005/2002.11.09/B2002.11.09/B/A 10/18/2011 01:29PM Directory A HREF=/MOTA/MCD15A3.005/2002.11.13/B2002.11.13/B/A 10/18/2011 01:31PM Directory A
Re: [R] Listing the contents of an FTP directory via R?
It seems to be choking on NLST: require(RCurl) getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,ftp.use.epsv=TRUE, dirlistonly = TRUE) ... 230 Guest login ok, access restrictions apply. PWD 257 / is current directory. * Entry path is '/' CWD MOTA 250 CWD command successful. CWD MCD15A3.005 250 CWD command successful. EPSV * Connect data stream passively 500 'EPSV': command not understood. * disabling EPSV usage PASV 227 Entering Passive Mode (152,61,133,5,177,17) * Trying 152.61.133.5... * connected * Connecting to 152.61.133.5 (152.61.133.5) port 45329 TYPE A 200 Type set to A. NLST 550 No files found. * RETR response: 550 * Remembering we are in dir MOTA/MCD15A3.005/ * Connection #0 to host e4ftl01.cr.usgs.gov left intact Error in function (type, msg, asError = TRUE) : RETR response: 550 Thanks for looking into this! --j On Mon, Apr 9, 2012 at 1:45 PM, steven mosher mosherste...@gmail.com wrote: Ya I hit the same error with my code that reads directories. I'll try some other stuff . I think I hit this error before I with usgs. On Mon, Apr 9, 2012 at 11:40 AM, Jonathan Greenberg j...@illinois.edu wrote: Steven: Thanks -- I seem to be running into the problem with the link I sent along: getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,dirlistonly = TRUE) Error in function (type, msg, asError = TRUE) : RETR response: 550 I'm wondering if it might be a passive ftp issue, but: getURL(ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005/,verbose=TRUE,ftp.use.epsv=TRUE, dirlistonly = TRUE) Error in function (type, msg, asError = TRUE) : FTP response reading failed Does not seem to work... Thoughts? --j On Mon, Apr 9, 2012 at 1:32 PM, steven mosher mosherste...@gmail.com wrote: A couple of ways. using Rcurl you can use the curlOption of dirlistonly. otherwise you can read the page and parse. I've got some code around here to do that. Steve On Mon, Apr 9, 2012 at 11:27 AM, Jonathan Greenberg j...@illinois.edu wrote: R-helpers: I'd like to be able to store all the file information from an ftp site (e.g. file and foldernames) through an R command. Any ideas how to do this? Here's an example site to use: ftp://e4ftl01.cr.usgs.gov/MOTA/MCD15A3.005 --j -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html -- Jonathan A. Greenberg, PhD Assistant Professor Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 415-763-5476 AIM: jgrn307, MSN: jgrn...@hotmail.com, Gchat: jgrn307, Skype: jgrn3007 http://www.geog.illinois.edu/people/JonathanGreenberg.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
