Re: [R] RCurl and cookies in POST requests
Hi Duncan. Duncan Temple Lang (Sunday 21 November 2010, 16:23): > [...] > There is a new version of the RCurl package on the Omegahat repository > and that handles this case. > [...] I just downloaded the new RCurl version (1.5-0.1, for Windows) from: http://www.stats.ox.ac.uk/pub/RWin/bin/windows/contrib/2.12/RCurl_1.5-0.1.zip and works perfectly. Thanks for your work! chr -- SDF Public Access UNIX System - http://sdf.lonestar.org __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] RCurl and cookies in POST requests
Hi Christian There is a new version of the RCurl package on the Omegahat repository and that handles this case. The issue was running the finalizer to garbage collect the curl handle, and it was correctly not being released as the dynCurlReader() update function was precious and had a reference to the curl handle. The new package has finer grained control in curlSetOpt() to control whether functions are made 'precious' or not and we can use this when we know we will leave the curl handle in a correct and consistent state even when the values of the function options are to be garbage collected before the curl handle. D. On 11/15/10 7:06 AM, Christian M. wrote: > Hello Duncan. > > Thanks for having a look at this. As soon as I get home I'll try > your suggestion. > > BTW, the link to the omega-help mailing list seems to be broken: > http://www.omegahat.org/mailman/listinfo/ > > Thank you. > > chr > > > Duncan Temple Lang (Monday 15 November 2010, 01:02): >> Hi Christian >> >> Thanks for finding this. The problem seems to be that the finalizer >> on the curl handle seems to disappear and so is not being called >> when the handle is garbage collected. So there is a bug somewhere >> and I'll try to hunt it down quickly. >> >> In the meantime, you can achieve the same effect by calling the >> C routine curl_easy_cleanup. You can't do this directly with a >> .Call() or .C() as there is no explicit interface in the RCurl >> package to this routine. However, you can use the Rffi package >> (on the omegahat repository) >> >> library(Rffi) >> cif = CIF(voidType, list(pointerType)) >> callCIF(cif, "curl_easy_cleanup", c...@ref) >> >> I'll keep looking for why the finalizer is getting discarded. >> >> Thanks again, >> >> D. >> >> On 11/14/10 6:30 AM, Christian M. wrote: >>> Hello. >>> >>> I know that it's usually possible to write cookies to a cookie >>> file by removing the curl handle and doing a gc() call. I can do >>> this with getURL(), but I just can't obtain the same results with >>> postForm(). >>> >>> If I use: >>> >>> curlHandle <- getCurlHandle(cookiefile=FILE, cookiejar=FILE) >>> >>> and then do: >>> >>> getURL(http://example.com/script.cgi, curl=curlHandle) >>> rm(curlHandle) >>> gc() >>> >>> it's OK, the cookie is there. But, if I do (same handle; the >>> parameter is a dummy): >>> >>> postForm(site, .params=list(par="cookie"), curl=curlHandle, >>> style="POST") >>> rm(curlHandle) >>> gc() >>> >>> no cookie is written. >>> >>> Probably I'm doing something wrong, but don't know what. >>> >>> Is it possible to store cookies read from the output of a >>> postForm() call? How? >>> >>> Thanks. >>> >>> Christian >>> >>> PS.: I'm attaching a script that can be sourced (and its .txt >>> version). It contains an example. The expected result is a file >>> (cookies.txt) with two cookies. The script currently uses >>> getURL() and two cookies are stored. If postForm() is used >>> (currently commented), only 1 cookie is written. >>> > __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] RCurl and cookies in POST requests
Hello Duncan. Thanks for having a look at this. As soon as I get home I'll try your suggestion. BTW, the link to the omega-help mailing list seems to be broken: http://www.omegahat.org/mailman/listinfo/ Thank you. chr Duncan Temple Lang (Monday 15 November 2010, 01:02): > Hi Christian > > Thanks for finding this. The problem seems to be that the finalizer > on the curl handle seems to disappear and so is not being called > when the handle is garbage collected. So there is a bug somewhere > and I'll try to hunt it down quickly. > > In the meantime, you can achieve the same effect by calling the > C routine curl_easy_cleanup. You can't do this directly with a > .Call() or .C() as there is no explicit interface in the RCurl > package to this routine. However, you can use the Rffi package > (on the omegahat repository) > > library(Rffi) > cif = CIF(voidType, list(pointerType)) > callCIF(cif, "curl_easy_cleanup", c...@ref) > > I'll keep looking for why the finalizer is getting discarded. > > Thanks again, > > D. > > On 11/14/10 6:30 AM, Christian M. wrote: > > Hello. > > > > I know that it's usually possible to write cookies to a cookie > > file by removing the curl handle and doing a gc() call. I can do > > this with getURL(), but I just can't obtain the same results with > > postForm(). > > > > If I use: > > > > curlHandle <- getCurlHandle(cookiefile=FILE, cookiejar=FILE) > > > > and then do: > > > > getURL(http://example.com/script.cgi, curl=curlHandle) > > rm(curlHandle) > > gc() > > > > it's OK, the cookie is there. But, if I do (same handle; the > > parameter is a dummy): > > > > postForm(site, .params=list(par="cookie"), curl=curlHandle, > > style="POST") > > rm(curlHandle) > > gc() > > > > no cookie is written. > > > > Probably I'm doing something wrong, but don't know what. > > > > Is it possible to store cookies read from the output of a > > postForm() call? How? > > > > Thanks. > > > > Christian > > > > PS.: I'm attaching a script that can be sourced (and its .txt > > version). It contains an example. The expected result is a file > > (cookies.txt) with two cookies. The script currently uses > > getURL() and two cookies are stored. If postForm() is used > > (currently commented), only 1 cookie is written. > > -- SDF Public Access UNIX System - http://sdf.lonestar.org __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] RCurl and cookies in POST requests
Hi Christian Thanks for finding this. The problem seems to be that the finalizer on the curl handle seems to disappear and so is not being called when the handle is garbage collected. So there is a bug somewhere and I'll try to hunt it down quickly. In the meantime, you can achieve the same effect by calling the C routine curl_easy_cleanup. You can't do this directly with a .Call() or .C() as there is no explicit interface in the RCurl package to this routine. However, you can use the Rffi package (on the omegahat repository) library(Rffi) cif = CIF(voidType, list(pointerType)) callCIF(cif, "curl_easy_cleanup", c...@ref) I'll keep looking for why the finalizer is getting discarded. Thanks again, D. On 11/14/10 6:30 AM, Christian M. wrote: > Hello. > > I know that it's usually possible to write cookies to a cookie > file by removing the curl handle and doing a gc() call. I can do > this with getURL(), but I just can't obtain the same results with > postForm(). > > If I use: > > curlHandle <- getCurlHandle(cookiefile=FILE, cookiejar=FILE) > > and then do: > > getURL(http://example.com/script.cgi, curl=curlHandle) > rm(curlHandle) > gc() > > it's OK, the cookie is there. But, if I do (same handle; the > parameter is a dummy): > > postForm(site, .params=list(par="cookie"), curl=curlHandle, > style="POST") > rm(curlHandle) > gc() > > no cookie is written. > > Probably I'm doing something wrong, but don't know what. > > Is it possible to store cookies read from the output of a > postForm() call? How? > > Thanks. > > Christian > > PS.: I'm attaching a script that can be sourced (and its .txt > version). It contains an example. The expected result is a file > (cookies.txt) with two cookies. The script currently uses > getURL() and two cookies are stored. If postForm() is used > (currently commented), only 1 cookie is written. > > > > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] RCurl and cookies in POST requests
Hello.
I know that it's usually possible to write cookies to a cookie
file by removing the curl handle and doing a gc() call. I can do
this with getURL(), but I just can't obtain the same results with
postForm().
If I use:
curlHandle <- getCurlHandle(cookiefile=FILE, cookiejar=FILE)
and then do:
getURL(http://example.com/script.cgi, curl=curlHandle)
rm(curlHandle)
gc()
it's OK, the cookie is there. But, if I do (same handle; the
parameter is a dummy):
postForm(site, .params=list(par="cookie"), curl=curlHandle,
style="POST")
rm(curlHandle)
gc()
no cookie is written.
Probably I'm doing something wrong, but don't know what.
Is it possible to store cookies read from the output of a
postForm() call? How?
Thanks.
Christian
PS.: I'm attaching a script that can be sourced (and its .txt
version). It contains an example. The expected result is a file
(cookies.txt) with two cookies. The script currently uses
getURL() and two cookies are stored. If postForm() is used
(currently commented), only 1 cookie is written.
--
SDF Public Access UNIX System - http://sdf.lonestar.org
# The script will call no_cookie() if no cookies file exists.
# It will then read the cookie XXX (its value is "cookie1").
# When XXX has been read, it will be written to "c_file" with
# "rm(curlHandle) ; gc()". Finally the script will test whether
# cookie ZZZ (its value is "cookie2") exists. If it doesn't,
# then the script will retrive the same URL as before, which
# will reply with ZZZ if XXX was sent.
#
# If XXX and ZZZ are in "cookies.txt", this should be removed
# to test again (otherwise the script will find both cookies
# and won't do anything).
library(RCurl)
site <- "http://chr.tx0.org/arch/ml/r/cookie-20101114.cgi";
c_file <- "cookies.txt"
no_cookie <- function() {
curlHandle <- getCurlHandle(cookiefile=c_file, cookiejar=c_file)
getURL(site, curl=curlHandle)
rm(curlHandle)
gc()
}
if ( file.exists(c_file) == FALSE ) {
file.create(c_file)
no_cookie()
}
cookie_1 <- sub(".*(XXX)\t", "", grep("XXX",readLines(c_file),value=T))
if ( length(grep("ZZZ",readLines(c_file))) == 0 ) {
curlHandle <- getCurlHandle(cookiefile=c_file, cookiejar=c_file)
# Either getURL OR postForm:
aaa <- getURL(site, curl=curlHandle)
#aaa <- postForm(site, .params=list(par=cookie_1), curl=curlHandle,
style="POST")
# Debug POST:
#d = debugGatherer()
#postForm(site, .params=list(par=cookie_1), .opts=list(httpheader = c(
# header = "foo: bar"), debugfunction = d$update, verbose=T),
# curl=curlHandle, style="POST")
#write.table(d$value()[["headerIn"]], file="debug_in.txt")
#write.table(d$value()[["headerOut"]], file="debug_out.txt")
rm(curlHandle)
gc()
}
# The script will call no_cookie() if no cookies file exists.
# It will then read the cookie XXX (its value is "cookie1").
# When XXX has been read, it will be written to "c_file" with
# "rm(curlHandle) ; gc()". Finally the script will test whether
# cookie ZZZ (its value is "cookie2") exists. If it doesn't,
# then the script will retrive the same URL as before, which
# will reply with ZZZ if XXX was sent.
#
# If XXX and ZZZ are in "cookies.txt", this should be removed
# to test again (otherwise the script will find both cookies
# and won't do anything).
library(RCurl)
site <- "http://chr.tx0.org/arch/ml/r/cookie-20101114.cgi";
c_file <- "cookies.txt"
no_cookie <- function() {
curlHandle <- getCurlHandle(cookiefile=c_file, cookiejar=c_file)
getURL(site, curl=curlHandle)
rm(curlHandle)
gc()
}
if ( file.exists(c_file) == FALSE ) {
file.create(c_file)
no_cookie()
}
cookie_1 <- sub(".*(XXX)\t", "", grep("XXX",readLines(c_file),value=T))
if ( length(grep("ZZZ",readLines(c_file))) == 0 ) {
curlHandle <- getCurlHandle(cookiefile=c_file, cookiejar=c_file)
# Either getURL OR postForm:
aaa <- getURL(site, curl=curlHandle)
#aaa <- postForm(site, .params=list(par=cookie_1), curl=curlHandle,
style="POST")
# Debug POST:
#d = debugGatherer()
#postForm(site, .params=list(par=cookie_1), .opts=list(httpheader = c(
# header = "foo: bar"), debugfunction = d$update, verbose=T),
# curl=curlHandle, style="POST")
#write.table(d$value()[["headerIn"]], file="debug_in.txt")
#write.table(d$value()[["headerOut"]], file="debug_out.txt")
rm(curlHandle)
gc()
}
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
