[R] postForm() in RCurl and library RHTMLForms
Hi RUsers,
Suppose I want to see the data on the website
url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
for the index "S&P CNX NIFTY" for
dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
then read the html table from the page using readHTMLtable()
I am using this code
webpage <- postForm(url,.params=list(
"FromDate"="01-11-2010",
"ToDate"="02-11-2010",
"IndexType"="S&P CNX NIFTY",
"Indicesdata"="Get Details"),
.opts=list(useragent = getOption("HTTPUserAgent")))
But it doesn't give me desired result
Also I was trying to use the function getHTMLFormDescription from the
package RHTMLForms but there we can't use the argument
.opts=list(useragent = getOption("HTTPUserAgent")) which is needed for this
particular website
Thanks and Regard
-
TO GET MORE DETAILS CLICK HERE
--
View this message in context:
http://r.789695.n4.nabble.com/postForm-in-RCurl-and-library-RHTMLForms-tp4651644.html
Sent from the R help mailing list archive at Nabble.com.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] postForm() in RCurl and library RHTMLForms
Hi RUsers,
Suppose I want to see the data on the website
url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
for the index "S&P CNX NIFTY" for
dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
then read the html table from the page using readHTMLtable()
I am using this code
webpage <- postForm(url,.params=list(
"FromDate"="01-11-2010",
"ToDate"="02-11-2010",
"IndexType"="S&P CNX NIFTY",
"Indicesdata"="Get Details"),
.opts=list(useragent = getOption("HTTPUserAgent")))
But it doesn't give me desired result
Also I was trying to use the function getHTMLFormDescription from the
package RHTMLForms but there we can't use the argument
.opts=list(useragent = getOption("HTTPUserAgent")) which is needed for this
particular website
-
TO GET MORE DETAILS CLICK HERE
--
View this message in context:
http://r.789695.n4.nabble.com/postForm-in-RCurl-and-library-RHTMLForms-tp4651655.html
Sent from the R help mailing list archive at Nabble.com.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] postForm() in RCurl and library RHTMLForms
Why I am getting this error? Error in getHTMLFormDescription(docNifty)[[1]] : subscript out of bounds -- View this message in context: http://r.789695.n4.nabble.com/postForm-in-RCurl-and-library-RHTMLForms-tp3026742p4650636.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] postForm() in RCurl and library RHTMLForms
Hi R Users,
I want to get the data from the url given from 10/09/2012 to 15/10/2012.
I don't know how to pass the parameters .
...
library(RHTMLForms)
>
> ff = getHTMLFormDescription("
http://www.bseindia.com/markets/equity/EQReports/slbshortcell.aspx?expandable=3¶m=0")
> ff
$aspnetForm
HTML Form:
http://www.bseindia.com/markets/equity/EQReports/slbshortcell.aspx?expandable=3¶m=0
ctl00$ContentPlaceHolder1$GetQuote1$txtscrip_code: [ Scrip code/Scrip Name
]
ctl00$ContentPlaceHolder1$txtDate: [ ]
ctl00$ContentPlaceHolder1$txtTodate: [ ]
myDestination: #,
http://www.bseindia.com/static/markets/mutualfunds/BSEStarMF.aspx?expandable=1,
http://www.bsebti.com/, http://www.bsesme.com/, http://www.cdslindia.com/,
http://www.bseindia.com/iccl/about_iccl.aspx, http://www.mkttech.in/,
http://www.bseindia.com/about/members_portal.asp, Group Websites, BSE STAR
MF, BSE Institute Ltd, BSE SME Platform, CDSL, ICCL, Marketplace
Technologies, Members Portal
myDestination1: #, http://hindi.bseindia.com/, http://marathi.bseindia.com/,
http://gujarati.bseindia.com/, Select Language, Hindi, Marathi, Gujarati
txtscrip: [ Scrip Name / Code ]
.
g = createFunction(ff[[1]])
here after how to call the function like this and getting the table out of
it.
xx = g("10/09/2012 ", "15/10/2012")
expecting your reply.
thanks
veepsirtt
On Tue, Sep 25, 2012 at 12:10 PM, wrote:
> Hi Duncan
>
> How to pass the parameters to this web form?.
> year="2012" ,month="August"
> I am not getting the table values.
>
> thanking you
> veepsirtt
> options(RCurlOptions = list(useragent = "R"))
> library(RCurl)
> url <- "http://www.bseindia.com/histdata/categorywise_turnover.asp";
> wp = getURLContent(url)
>
> library(RHTMLForms)
> library(XML)
> doc = htmlParse(wp, asText = TRUE)
> form = getHTMLFormDescription(doc)[[1]]
> fun = createFunction(form)
> o = fun(mmm = "9", yyy = "2012",url="
> http://www.bseindia.com/histdata/categorywise_turnover.asp";)
>
> table = readHTMLTable(htmlParse(o, asText = TRUE),
> header = TRUE,
> stringsAsFactors = FALSE)
> table
>
>
>
>
> On 11/4/10 2:39 AM, sayan dasgupta wrote:
> > Hi RUsers,
> >
> > Suppose I want to see the data on the website
> > url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
> >
> > for the index "S&P CNX NIFTY" for
> > dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
> >
> > then read the html table from the page using readHTMLtable()
> >
> > I am using this code
> > webpage <- postForm(url,.params=list(
> >"FromDate"="01-11-2010",
> >"ToDate"="02-11-2010",
> >"IndexType"="S&P CNX NIFTY",
> >"Indicesdata"="Get Details"),
> > .opts=list(useragent = getOption("HTTPUserAgent")))
> >
> > But it doesn't give me desired result
>
> You need to be more specific about how it fails to give the desired result.
>
> You are in fact posting to the wrong URL. The form is submitted to a
> different
> URL -
> http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp
>
>
>
> >
> > Also I was trying to use the function getHTMLFormDescription from the
> > package RHTMLForms but there we can't use the argument
> > .opts=list(useragent = getOption("HTTPUserAgent")) which is needed for
> > this
> > particular website
>
> That's not the case. The function RHTMLForms will generate for you does
> support
> the .opts parameter.
>
> What you want is something along the lines:
>
>
> # Set default options for RCurl
> # requests
> options(RCurlOptions = list(useragent = "R"))
> library(RCurl)
>
> # Read the HTML page since we cannot use htmlParse() directly
> # as it does not specify the user agent or an
> # Accept:*.*
>
> url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
> wp = getURLContent(url)
>
> # Now that we have the page, parse it and use the RHTMLForms
> # package to create an R function that will act as an interface
> # to the form.
> library(RHTMLForms)
> library(XML)
> doc = htmlParse(wp, asText = TRUE)
> # need to set the URL for this document since we read it from
> # text, rather than from the URL directly
>
> docName(doc) = url
>
> # Create the form description and generate the R
> # function "call" the
>
> form = getHTMLFormDescription(doc)[[1]]
> fun = createFunction(form)
>
>
> # now we can invoke the form from R. We only need 2
> # inputs - FromDate and ToDate
>
> o = fun(FromDate = "01-11-2010", ToDate = "04-11-2010")
>
> # Hav
Re: [R] postForm() in RCurl and library RHTMLForms
On 11/4/10 11:31 PM, sayan dasgupta wrote:
> Thanks a lot thats exactly what I was looking for
>
> Just a quick question I agree the form gets submitted to the URL
> "http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp";
>
> and I am filling up the form in the page
> "http://www.nseindia.com/content/indices/ind_histvalues.htm";
>
> How do I submit the arguments like FromDate, ToDate, Symbol using postForm()
> and submit the query to get the similar table.
>
Well that is what the function that RHTMLForms creates does.
So you can look at that code and see that it calls formQuery()
which ends in a call to postForm(). You could use
debug(postForm)
and examine the arguments to it.
postForm("...jsp", FromDate = "10-"
The answer is
o =
postForm("http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp";,
FromDate = "01-11-2010", ToDate = "04-11-2010",
IndexType = "S&P CNX NIFTY", check = "new",
style = "POST" )
>
>
>
>
>
>
> On Fri, Nov 5, 2010 at 6:43 AM, Duncan Temple Lang
> wrote:
>
>>
>>
>> On 11/4/10 2:39 AM, sayan dasgupta wrote:
>>> Hi RUsers,
>>>
>>> Suppose I want to see the data on the website
>>> url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
>>>
>>> for the index "S&P CNX NIFTY" for
>>> dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
>>>
>>> then read the html table from the page using readHTMLtable()
>>>
>>> I am using this code
>>> webpage <- postForm(url,.params=list(
>>>"FromDate"="01-11-2010",
>>>"ToDate"="02-11-2010",
>>>"IndexType"="S&P CNX NIFTY",
>>>"Indicesdata"="Get Details"),
>>> .opts=list(useragent = getOption("HTTPUserAgent")))
>>>
>>> But it doesn't give me desired result
>>
>> You need to be more specific about how it fails to give the desired result.
>>
>> You are in fact posting to the wrong URL. The form is submitted to a
>> different
>> URL -
>> http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp
>>
>>
>>
>>>
>>> Also I was trying to use the function getHTMLFormDescription from the
>>> package RHTMLForms but there we can't use the argument
>>> .opts=list(useragent = getOption("HTTPUserAgent")) which is needed for
>> this
>>> particular website
>>
>> That's not the case. The function RHTMLForms will generate for you does
>> support
>> the .opts parameter.
>>
>> What you want is something along the lines:
>>
>>
>> # Set default options for RCurl
>> # requests
>> options(RCurlOptions = list(useragent = "R"))
>> library(RCurl)
>>
>> # Read the HTML page since we cannot use htmlParse() directly
>> # as it does not specify the user agent or an
>> # Accept:*.*
>>
>> url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
>> wp = getURLContent(url)
>>
>> # Now that we have the page, parse it and use the RHTMLForms
>> # package to create an R function that will act as an interface
>> # to the form.
>> library(RHTMLForms)
>> library(XML)
>> doc = htmlParse(wp, asText = TRUE)
>> # need to set the URL for this document since we read it from
>> # text, rather than from the URL directly
>>
>> docName(doc) = url
>>
>> # Create the form description and generate the R
>> # function "call" the
>>
>> form = getHTMLFormDescription(doc)[[1]]
>> fun = createFunction(form)
>>
>>
>> # now we can invoke the form from R. We only need 2
>> # inputs - FromDate and ToDate
>>
>> o = fun(FromDate = "01-11-2010", ToDate = "04-11-2010")
>>
>> # Having looked at the tables, I think we want the the 3rd
>> # one.
>> table = readHTMLTable(htmlParse(o, asText = TRUE),
>>which = 3,
>>header = TRUE,
>>stringsAsFactors = FALSE)
>> table
>>
>>
>>
>>
>> Yes it is marginally involved. But that is because we cannot simply read
>> the HTML document directly from htmlParse() because the lack of Accept(&
>> useragent)
>> HTTP header.
>>
>>>
>>>
>>> Thanks and Regards
>>> Sayan Dasgupta
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> __
>>> R-help@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>
>> __
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guid
Re: [R] postForm() in RCurl and library RHTMLForms
Thanks a lot thats exactly what I was looking for
Just a quick question I agree the form gets submitted to the URL
"http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp";
and I am filling up the form in the page
"http://www.nseindia.com/content/indices/ind_histvalues.htm";
How do I submit the arguments like FromDate, ToDate, Symbol using postForm()
and submit the query to get the similar table.
On Fri, Nov 5, 2010 at 6:43 AM, Duncan Temple Lang
wrote:
>
>
> On 11/4/10 2:39 AM, sayan dasgupta wrote:
> > Hi RUsers,
> >
> > Suppose I want to see the data on the website
> > url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
> >
> > for the index "S&P CNX NIFTY" for
> > dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
> >
> > then read the html table from the page using readHTMLtable()
> >
> > I am using this code
> > webpage <- postForm(url,.params=list(
> >"FromDate"="01-11-2010",
> >"ToDate"="02-11-2010",
> >"IndexType"="S&P CNX NIFTY",
> >"Indicesdata"="Get Details"),
> > .opts=list(useragent = getOption("HTTPUserAgent")))
> >
> > But it doesn't give me desired result
>
> You need to be more specific about how it fails to give the desired result.
>
> You are in fact posting to the wrong URL. The form is submitted to a
> different
> URL -
> http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp
>
>
>
> >
> > Also I was trying to use the function getHTMLFormDescription from the
> > package RHTMLForms but there we can't use the argument
> > .opts=list(useragent = getOption("HTTPUserAgent")) which is needed for
> this
> > particular website
>
> That's not the case. The function RHTMLForms will generate for you does
> support
> the .opts parameter.
>
> What you want is something along the lines:
>
>
> # Set default options for RCurl
> # requests
> options(RCurlOptions = list(useragent = "R"))
> library(RCurl)
>
> # Read the HTML page since we cannot use htmlParse() directly
> # as it does not specify the user agent or an
> # Accept:*.*
>
> url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
> wp = getURLContent(url)
>
> # Now that we have the page, parse it and use the RHTMLForms
> # package to create an R function that will act as an interface
> # to the form.
> library(RHTMLForms)
> library(XML)
> doc = htmlParse(wp, asText = TRUE)
> # need to set the URL for this document since we read it from
> # text, rather than from the URL directly
>
> docName(doc) = url
>
> # Create the form description and generate the R
> # function "call" the
>
> form = getHTMLFormDescription(doc)[[1]]
> fun = createFunction(form)
>
>
> # now we can invoke the form from R. We only need 2
> # inputs - FromDate and ToDate
>
> o = fun(FromDate = "01-11-2010", ToDate = "04-11-2010")
>
> # Having looked at the tables, I think we want the the 3rd
> # one.
> table = readHTMLTable(htmlParse(o, asText = TRUE),
>which = 3,
>header = TRUE,
>stringsAsFactors = FALSE)
> table
>
>
>
>
> Yes it is marginally involved. But that is because we cannot simply read
> the HTML document directly from htmlParse() because the lack of Accept(&
> useragent)
> HTTP header.
>
> >
> >
> > Thanks and Regards
> > Sayan Dasgupta
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
> __
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] postForm() in RCurl and library RHTMLForms
On 11/4/10 2:39 AM, sayan dasgupta wrote:
> Hi RUsers,
>
> Suppose I want to see the data on the website
> url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
>
> for the index "S&P CNX NIFTY" for
> dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
>
> then read the html table from the page using readHTMLtable()
>
> I am using this code
> webpage <- postForm(url,.params=list(
>"FromDate"="01-11-2010",
>"ToDate"="02-11-2010",
>"IndexType"="S&P CNX NIFTY",
>"Indicesdata"="Get Details"),
> .opts=list(useragent = getOption("HTTPUserAgent")))
>
> But it doesn't give me desired result
You need to be more specific about how it fails to give the desired result.
You are in fact posting to the wrong URL. The form is submitted to a different
URL - http://www.nseindia.com/marketinfo/indices/histdata/historicalindices.jsp
>
> Also I was trying to use the function getHTMLFormDescription from the
> package RHTMLForms but there we can't use the argument
> .opts=list(useragent = getOption("HTTPUserAgent")) which is needed for this
> particular website
That's not the case. The function RHTMLForms will generate for you does support
the .opts parameter.
What you want is something along the lines:
# Set default options for RCurl
# requests
options(RCurlOptions = list(useragent = "R"))
library(RCurl)
# Read the HTML page since we cannot use htmlParse() directly
# as it does not specify the user agent or an
# Accept:*.*
url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
wp = getURLContent(url)
# Now that we have the page, parse it and use the RHTMLForms
# package to create an R function that will act as an interface
# to the form.
library(RHTMLForms)
library(XML)
doc = htmlParse(wp, asText = TRUE)
# need to set the URL for this document since we read it from
# text, rather than from the URL directly
docName(doc) = url
# Create the form description and generate the R
# function "call" the
form = getHTMLFormDescription(doc)[[1]]
fun = createFunction(form)
# now we can invoke the form from R. We only need 2
# inputs - FromDate and ToDate
o = fun(FromDate = "01-11-2010", ToDate = "04-11-2010")
# Having looked at the tables, I think we want the the 3rd
# one.
table = readHTMLTable(htmlParse(o, asText = TRUE),
which = 3,
header = TRUE,
stringsAsFactors = FALSE)
table
Yes it is marginally involved. But that is because we cannot simply read
the HTML document directly from htmlParse() because the lack of Accept(&
useragent)
HTTP header.
>
>
> Thanks and Regards
> Sayan Dasgupta
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] postForm() in RCurl and library RHTMLForms
I dont have the implementation in the way you want it
. Sorry
but
someone here will definitely know
The group showed me to do it this way though
.
library(zoo)
library("RCurl")
sNiftyURL =
"http://nseindia.com/content/indices/histdata/S&P%20CNX%20NIFTY01-01-2000-02
-11-2010.csv"
Nifty_Dat = getURLContent(sNiftyURL, verbose = TRUE, useragent =
getOption("HTTPUserAgent"))
tblNifty <- read.csv(textConnection(Nifty_Dat))
tblNifty <- subset(tblNifty,select=c(Date,Close))
tblNifty$Date <- as.Date(tblNifty$Date, format ="%d-%b-%Y")
tblNifty <-read.zoo((tblNifty))
closeAllConnections()
HTH.
S
From: sayan dasgupta [mailto:kitt...@gmail.com]
Sent: 04 November 2010 15:09
To: r-help@r-project.org
Cc: dun...@wald.ucdavis.edu; santosh.srini...@gmail.com
Subject: postForm() in RCurl and library RHTMLForms
Hi RUsers,
Suppose I want to see the data on the website
url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
for the index "S&P CNX NIFTY" for
dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
then read the html table from the page using readHTMLtable()
I am using this code
webpage <- postForm(url,.params=list(
"FromDate"="01-11-2010",
"ToDate"="02-11-2010",
"IndexType"="S&P CNX NIFTY",
"Indicesdata"="Get Details"),
.opts=list(useragent = getOption("HTTPUserAgent")))
But it doesn't give me desired result
Also I was trying to use the function getHTMLFormDescription from the
package RHTMLForms but there we can't use the argument
.opts=list(useragent = getOption("HTTPUserAgent")) which is needed for this
particular website
Thanks and Regards
Sayan Dasgupta
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] postForm() in RCurl and library RHTMLForms
Hi RUsers,
Suppose I want to see the data on the website
url <- "http://www.nseindia.com/content/indices/ind_histvalues.htm";
for the index "S&P CNX NIFTY" for
dates "FromDate"="01-11-2010","ToDate"="02-11-2010"
then read the html table from the page using readHTMLtable()
I am using this code
webpage <- postForm(url,.params=list(
"FromDate"="01-11-2010",
"ToDate"="02-11-2010",
"IndexType"="S&P CNX NIFTY",
"Indicesdata"="Get Details"),
.opts=list(useragent = getOption("HTTPUserAgent")))
But it doesn't give me desired result
Also I was trying to use the function getHTMLFormDescription from the
package RHTMLForms but there we can't use the argument
.opts=list(useragent = getOption("HTTPUserAgent")) which is needed for this
particular website
Thanks and Regards
Sayan Dasgupta
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
