Re: [R] Closing FTP sessions with RCurl
Fantastic! That did the trick. I still suspect that there may be a command
within RCurl that accomplishes the same thing, but in any case I’m very
grateful to have a solution that works.
Best,
John
From: William Dunlap <wdun...@tibco.com>
Date: Wednesday, June 15, 2016 at 12:57 PM
To: J Payne <jcpa...@uw.edu>
Cc: Tom Wright <t...@maladmin.com>, R help list <r-help@r-project.org>
Subject: Re: [R] Closing FTP sessions with RCurl
>>rm(curl) # release the curl! (does this end the session???)
Try adding a call to gc() immediately after this removal. That will force an
immediate run of any finalizer associated with the object just removed.
With the call to gc(), the garbage collector will be called some time in the
future and the finalizers will be run then.
(I don't know if curl has a finalizer that closes the session.)
Bill Dunlap
TIBCO Software
wdunlap tibco.com
On Wed, Jun 15, 2016 at 12:34 PM, J Payne <jcpa...@uw.edu> wrote:
Thanks Tom! I tried that and it didn’t work, but perhaps there are other
problems. The system administrator wrote “I think closing sessions immediately
upon completion of the transfer,
or continuing to use the same session for additional requests would both have a
similar and positive effect.” However, I’ve now tried your solution to close
the session, and also tried using one curl handle throughout, but the
throttling persists.
John
On 6/14/16, 1:00 PM, "Tom Wright" <t...@maladmin.com> wrote:
>No expert here, and this isn't tested. It seems you can set the
>forbid.reuse option which will cause curl to shutdown the connection
>after transfer is complete.
>
>if(url.exists("http://www.omegahat.net/RCurl/index.html;)) {
> curl <- getCurlHandle()
> curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
> getURL("http://www.omegahat.net/RCurl/index.html;, curl = curl)
>}
>
>On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpa...@uw.edu> wrote:
>> Does anyone know how to close an FTP session with RCurl? I am trying to
>> automate the process of downloading snow data from a government website, and
>> their server is throttling the connection after a few files are downloaded.
>> I contacted their system administrator, who wrote: “My suspicion at this
>> point is that the getURL commands are opened and perform the function asked,
>> then linger in wait for 15 minutes until or ftp server closes the idle
>> sessions. Is there a way to tell R to close the sessions?”
>>
>>
>>
>> I’ve perused the RCurl manual but I don’t see a way to close sessions. I
>> tried copying the following example from the RCurl manual, but it didn’t
>> solve the problem. I’m a novice at this and I don’t understand the
>> relationship between handles and sessions, so I am probably missing
>> something.
>>
>>
>>
>> #EXAMPLE from getCurl(), p. 39
>>
>> if(url.exists("http://www.omegahat.net/RCurl/index.html;)) {
>>
>>curl = getCurlHandle()
>>
>>getURL("http://www.omegahat.net/RCurl/index.html;, curl = curl)
>>
>>#getCurlInfo(curl) # I skipped this step
>>
>>rm(curl) # release the curl! (does this end the session???)
>>
>> }
>>
>>
>>
>> Thanks!
>>
>>
>>
>> John
>>
>>
>> [[alternative HTML version deleted]]
>>
>> __
>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Closing FTP sessions with RCurl
Thanks Tom! I tried that and it didn’t work, but perhaps there are other
problems. The system administrator wrote “I think closing sessions immediately
upon completion of the transfer,
or continuing to use the same session for additional requests would both have a
similar and positive effect.” However, I’ve now tried your solution to close
the session, and also tried using one curl handle throughout, but the
throttling persists.
John
On 6/14/16, 1:00 PM, "Tom Wright" <t...@maladmin.com> wrote:
>No expert here, and this isn't tested. It seems you can set the
>forbid.reuse option which will cause curl to shutdown the connection
>after transfer is complete.
>
>if(url.exists("http://www.omegahat.net/RCurl/index.html;)) {
> curl <- getCurlHandle()
> curlSetOpt(.opts=list(forbid.reuse=1),curl=curl)
> getURL("http://www.omegahat.net/RCurl/index.html;, curl = curl)
>}
>
>On Tue, Jun 14, 2016 at 2:41 PM, J Payne <jcpa...@uw.edu> wrote:
>> Does anyone know how to close an FTP session with RCurl? I am trying to
>> automate the process of downloading snow data from a government website, and
>> their server is throttling the connection after a few files are downloaded.
>> I contacted their system administrator, who wrote: “My suspicion at this
>> point is that the getURL commands are opened and perform the function asked,
>> then linger in wait for 15 minutes until or ftp server closes the idle
>> sessions. Is there a way to tell R to close the sessions?”
>>
>>
>>
>> I’ve perused the RCurl manual but I don’t see a way to close sessions. I
>> tried copying the following example from the RCurl manual, but it didn’t
>> solve the problem. I’m a novice at this and I don’t understand the
>> relationship between handles and sessions, so I am probably missing
>> something.
>>
>>
>>
>> #EXAMPLE from getCurl(), p. 39
>>
>> if(url.exists("http://www.omegahat.net/RCurl/index.html;)) {
>>
>>curl = getCurlHandle()
>>
>>getURL("http://www.omegahat.net/RCurl/index.html;, curl = curl)
>>
>>#getCurlInfo(curl) # I skipped this step
>>
>>rm(curl) # release the curl! (does this end the session???)
>>
>> }
>>
>>
>>
>> Thanks!
>>
>>
>>
>> John
>>
>>
>> [[alternative HTML version deleted]]
>>
>> __
>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Closing FTP sessions with RCurl
Does anyone know how to close an FTP session with RCurl? I am trying to
automate the process of downloading snow data from a government website, and
their server is throttling the connection after a few files are downloaded. I
contacted their system administrator, who wrote: “My suspicion at this point is
that the getURL commands are opened and perform the function asked, then linger
in wait for 15 minutes until or ftp server closes the idle sessions. Is there a
way to tell R to close the sessions?”
I’ve perused the RCurl manual but I don’t see a way to close sessions. I tried
copying the following example from the RCurl manual, but it didn’t solve the
problem. I’m a novice at this and I don’t understand the relationship between
handles and sessions, so I am probably missing something.
#EXAMPLE from getCurl(), p. 39
if(url.exists("http://www.omegahat.net/RCurl/index.html;)) {
curl = getCurlHandle()
getURL("http://www.omegahat.net/RCurl/index.html;, curl = curl)
#getCurlInfo(curl) # I skipped this step
rm(curl) # release the curl! (does this end the session???)
}
Thanks!
John
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] system() command not working
I forgot to add that setting the environment variable in .Renviron in my home directory *also* works. I have updated my question on Stackoverflow to include these answers. John On 6/5/16, 1:17 PM, "J Payne" <jcpa...@uw.edu> wrote: >Thanks Berend, that’s super useful. In summary, here is what I found: > >The problem (if I understand correctly) is that R was not passing environment >variables to OSX El Capitan successfully. In any case: > >These do not work: >1. Setting the environment variable in my .bash_profile (for example >“MTR_DATA_DIR="/Applications/MRT/data"); or >2. Adding the same environment variable to .Rprofile in my home directory. > >These do work (hooray!): >3. Changing the environment variable using >Sys.setenv(MRT_DATA_DIR="/Applications/MRT/data") inside R at the R command >line; or >4. Typing “MRT_DATA_DIR="/Applications/MRT/data" open -a Rstudio” in the >Terminal. This latter method is an effective workaround and wonderful to have >in my bag of tricks, but is slightly clumsier since I have to remember to open >RStudio this way each time. > >I’m very grateful to you and Roy for your help. > >John > >On 6/5/16, 3:28 AM, "Berend Hasselman" <b...@xs4all.nl> wrote: > >> >>> On 4 Jun 2016, at 22:12, Roy Mendelssohn - NOAA Federal >>> <roy.mendelss...@noaa.gov> wrote: >>> >>> Hi John: >>> >>> When El Capitan first came out there was a discussion in the R-SIg-Mac >>> list about environmental variables not being passed down to applications >>> (not just R abut in general). I believe a work around was suggested, but I >>> would search the archives for that. >>> >>> So what is happening, when you run from the command line, the variables >>> MRT_DATA_DIR and MRTDATADIR which are defined somewhere in your environment >>> are found in the terminal, but when the same command is run from the >>> application they are not being found. I would search the R-SIg-Mac archive >>> or post to that list, because i can’t remember what the work around was for >>> it. >>> >> >>I can't find the discussion on R-SIG-Mac list. But you can try this: >> >>MRT_DATA_DIR= open -a Rstudio >> >>or >> >>MRT_DATA_DIR= open -a R >> >>Try it and see what happens. >>It may even be possible to put something in .Rprofile setting your >>environment variables. >> >>Berend Hasselman >> >>> HTH, >>> >>> -Roy >>> >>>> On Jun 4, 2016, at 11:59 AM, J Payne <jcpa...@uw.edu> wrote: >>>> >>>> I’ve posted this question on StackExchange at >>>> http://stackoverflow.com/questions/37604466/r-system-not-working-with-modis-reprojection-tool, >>>> but haven’t received any replies. I’m hoping that someone who >>>> understands the operation of the R system() command can help. >>>> >>>> >>>> >>>> I have a command that works when typed into the Terminal on a Mac (OSX El >>>> Cap), but exactly the same command fails when called from R using >>>> `system()`. >>>> >>>> >>>> >>>> I am trying to batch-process MODIS satellite files using a small program >>>> called the MODIS Reprojection Tool >>>> (https://lpdaac.usgs.gov/tools/modis_reprojection_tool). My software is >>>> all up to date. >>>> >>>> >>>> >>>> This is a simple example in which I mosaic two files. The names of the >>>> two files are in a text input file called `input.list`. The command just >>>> tells the `mrtmosaic` routine where to find the input list and where to >>>> put the output. >>>> >>>> >>>> >>>> This command works correctly in the Terminal: >>>> >>>> >>>> >>>>/Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i >>>> ~/temp/input.list -o ~/temp/output.hdf >>>> >>>> >>>> >>>> However, if I put exactly the same string into a variable and run it from >>>> R (using RStudio), it fails: >>>> >>>> >>>> >>>>comstring<-"/Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i >>>> ~/temp/input.list -o ~/temp/output.hdf" >>>> >>>>system(comstring) >>>> >>>> >>
Re: [R] system() command not working
Thanks Berend, that’s super useful. In summary, here is what I found: The problem (if I understand correctly) is that R was not passing environment variables to OSX El Capitan successfully. In any case: These do not work: 1. Setting the environment variable in my .bash_profile (for example “MTR_DATA_DIR="/Applications/MRT/data"); or 2. Adding the same environment variable to .Rprofile in my home directory. These do work (hooray!): 3. Changing the environment variable using Sys.setenv(MRT_DATA_DIR="/Applications/MRT/data") inside R at the R command line; or 4. Typing “MRT_DATA_DIR="/Applications/MRT/data" open -a Rstudio” in the Terminal. This latter method is an effective workaround and wonderful to have in my bag of tricks, but is slightly clumsier since I have to remember to open RStudio this way each time. I’m very grateful to you and Roy for your help. John On 6/5/16, 3:28 AM, "Berend Hasselman" <b...@xs4all.nl> wrote: > >> On 4 Jun 2016, at 22:12, Roy Mendelssohn - NOAA Federal >> <roy.mendelss...@noaa.gov> wrote: >> >> Hi John: >> >> When El Capitan first came out there was a discussion in the R-SIg-Mac >> list about environmental variables not being passed down to applications >> (not just R abut in general). I believe a work around was suggested, but I >> would search the archives for that. >> >> So what is happening, when you run from the command line, the variables >> MRT_DATA_DIR and MRTDATADIR which are defined somewhere in your environment >> are found in the terminal, but when the same command is run from the >> application they are not being found. I would search the R-SIg-Mac archive >> or post to that list, because i can’t remember what the work around was for >> it. >> > >I can't find the discussion on R-SIG-Mac list. But you can try this: > >MRT_DATA_DIR= open -a Rstudio > >or > >MRT_DATA_DIR= open -a R > >Try it and see what happens. >It may even be possible to put something in .Rprofile setting your >environment variables. > >Berend Hasselman > >> HTH, >> >> -Roy >> >>> On Jun 4, 2016, at 11:59 AM, J Payne <jcpa...@uw.edu> wrote: >>> >>> I’ve posted this question on StackExchange at >>> http://stackoverflow.com/questions/37604466/r-system-not-working-with-modis-reprojection-tool, >>> but haven’t received any replies. I’m hoping that someone who understands >>> the operation of the R system() command can help. >>> >>> >>> >>> I have a command that works when typed into the Terminal on a Mac (OSX El >>> Cap), but exactly the same command fails when called from R using >>> `system()`. >>> >>> >>> >>> I am trying to batch-process MODIS satellite files using a small program >>> called the MODIS Reprojection Tool >>> (https://lpdaac.usgs.gov/tools/modis_reprojection_tool). My software is >>> all up to date. >>> >>> >>> >>> This is a simple example in which I mosaic two files. The names of the two >>> files are in a text input file called `input.list`. The command just tells >>> the `mrtmosaic` routine where to find the input list and where to put the >>> output. >>> >>> >>> >>> This command works correctly in the Terminal: >>> >>> >>> >>>/Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i ~/temp/input.list >>> -o ~/temp/output.hdf >>> >>> >>> >>> However, if I put exactly the same string into a variable and run it from R >>> (using RStudio), it fails: >>> >>> >>> >>>comstring<-"/Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i >>> ~/temp/input.list -o ~/temp/output.hdf" >>> >>>system(comstring) >>> >>> >>> >>>> Warning: gctp_call : Environmental Variable Not Found: >>> >>>MRT_DATA_DIR nor MRTDATADIR not defined >>> >>>Error: GetInputGeoCornerMosaic : General Processing Error converting >>> lat/long coordinates to input projection coordinates. >>> >>>Fatal Error, Terminating... >>> >>> >>> >>> The strange thing is that the system knows what the environment variables >>> are. In the terminal, the command >>> >>> `echo $MRT_DATA_DIR` >>> >>> shows the correct directory: /Applications/Modis_Reprojection_Tool/
Re: [R] system() command not working
Thanks very much Roy! I will post to R-Sig-Mac. I browsed their archives but didn’t see anything about the issue – I might have missed something, though. John On 6/4/16, 1:12 PM, "Roy Mendelssohn - NOAA Federal" <roy.mendelss...@noaa.gov> wrote: >Hi John: > >When El Capitan first came out there was a discussion in the R-SIg-Mac list >about environmental variables not being passed down to applications (not just >R abut in general). I believe a work around was suggested, but I would search >the archives for that. > >So what is happening, when you run from the command line, the variables >MRT_DATA_DIR and MRTDATADIR which are defined somewhere in your environment >are found in the terminal, but when the same command is run from the >application they are not being found. I would search the R-SIg-Mac archive or >post to that list, because i can’t remember what the work around was for it. > >HTH, > >-Roy > >> On Jun 4, 2016, at 11:59 AM, J Payne <jcpa...@uw.edu> wrote: >> >> I’ve posted this question on StackExchange at >> http://stackoverflow.com/questions/37604466/r-system-not-working-with-modis-reprojection-tool, >> but haven’t received any replies. I’m hoping that someone who understands >> the operation of the R system() command can help. >> >> >> >> I have a command that works when typed into the Terminal on a Mac (OSX El >> Cap), but exactly the same command fails when called from R using >> `system()`. >> >> >> >> I am trying to batch-process MODIS satellite files using a small program >> called the MODIS Reprojection Tool >> (https://lpdaac.usgs.gov/tools/modis_reprojection_tool). My software is all >> up to date. >> >> >> >> This is a simple example in which I mosaic two files. The names of the two >> files are in a text input file called `input.list`. The command just tells >> the `mrtmosaic` routine where to find the input list and where to put the >> output. >> >> >> >> This command works correctly in the Terminal: >> >> >> >> /Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i ~/temp/input.list >> -o ~/temp/output.hdf >> >> >> >> However, if I put exactly the same string into a variable and run it from R >> (using RStudio), it fails: >> >> >> >> comstring<-"/Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i >> ~/temp/input.list -o ~/temp/output.hdf" >> >> system(comstring) >> >> >> >>> Warning: gctp_call : Environmental Variable Not Found: >> >> MRT_DATA_DIR nor MRTDATADIR not defined >> >> Error: GetInputGeoCornerMosaic : General Processing Error converting >> lat/long coordinates to input projection coordinates. >> >> Fatal Error, Terminating... >> >> >> >> The strange thing is that the system knows what the environment variables >> are. In the terminal, the command >> >> `echo $MRT_DATA_DIR` >> >> shows the correct directory: /Applications/Modis_Reprojection_Tool/data >> >> >> >> I don't see why it would have trouble finding the variables from an `R >> system()` call when it has no trouble in the Terminal. I'm very stumped! >> >> >> >> >> >> >> >> >> [[alternative HTML version deleted]] >> >> __ >> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. > >** >"The contents of this message do not reflect any position of the U.S. >Government or NOAA." >** >Roy Mendelssohn >Supervisory Operations Research Analyst >NOAA/NMFS >Environmental Research Division >Southwest Fisheries Science Center >***Note new address and phone*** >110 Shaffer Road >Santa Cruz, CA 95060 >Phone: (831)-420-3666 >Fax: (831) 420-3980 >e-mail: roy.mendelss...@noaa.gov www: http://www.pfeg.noaa.gov/ > >"Old age and treachery will overcome youth and skill." >"From those who have been given much, much will be expected" >"the arc of the moral universe is long, but it bends toward justice" -MLK Jr. > __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] system() command not working
I’ve posted this question on StackExchange at http://stackoverflow.com/questions/37604466/r-system-not-working-with-modis-reprojection-tool, but haven’t received any replies. I’m hoping that someone who understands the operation of the R system() command can help. I have a command that works when typed into the Terminal on a Mac (OSX El Cap), but exactly the same command fails when called from R using `system()`. I am trying to batch-process MODIS satellite files using a small program called the MODIS Reprojection Tool (https://lpdaac.usgs.gov/tools/modis_reprojection_tool). My software is all up to date. This is a simple example in which I mosaic two files. The names of the two files are in a text input file called `input.list`. The command just tells the `mrtmosaic` routine where to find the input list and where to put the output. This command works correctly in the Terminal: /Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i ~/temp/input.list -o ~/temp/output.hdf However, if I put exactly the same string into a variable and run it from R (using RStudio), it fails: comstring<-"/Applications/Modis_Reprojection_Tool/bin/mrtmosaic -i ~/temp/input.list -o ~/temp/output.hdf" system(comstring) >Warning: gctp_call : Environmental Variable Not Found: MRT_DATA_DIR nor MRTDATADIR not defined Error: GetInputGeoCornerMosaic : General Processing Error converting lat/long coordinates to input projection coordinates. Fatal Error, Terminating... The strange thing is that the system knows what the environment variables are. In the terminal, the command `echo $MRT_DATA_DIR` shows the correct directory: /Applications/Modis_Reprojection_Tool/data I don't see why it would have trouble finding the variables from an `R system()` call when it has no trouble in the Terminal. I'm very stumped! [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
