Re: [Bug-wget] Fwd: Regarding wget to download webpage

2012-01-30 Thread Bhargavi N
Hello Henrik,

   Thanks Thanks Thanks a LOT!!

Thanks Entire wget team for helping me so generously. Even in your busy
schedules.

Issue is now resolved.

I used Live HTTP headers add on as suggested by you and it really helped me.

Initially i passed all the headers as displayed in the capture window,
with "--header" option of wget.

Later i understood that session details have to be captured.

I used "--keep-session-cookies" option of wget, since sessions details have
to be saved.

I saved the session details in cookies.txt, which i used later to load
ADHOC query page and run the query.

Below are the commands i used to get the query results.

wget -O logon.html --keep-session-cookies --save-cookies cookies.txt
 --post-data 'loginID=mylogon&password=mypasswd'
https://www.collabnet.timeinc.net

wget -O query_res.html --load-cookies cookies.txt "
https://www.collabnet.timeinc.net/servlets/AdHocQuery?query=select+count%28*%29+from+HELM_USER&Button=Run+query
"


Special thanks to you and whole team for sparing your time to help me out
with the issue.

Please let me know if i can be part of your team or i can be of any help to
you.

HAVE NICE DAY !!

With wishes,
Bhargavi

On Tue, Jan 24, 2012 at 3:20 PM, Henrik Holst
wrote:

> I looked at your web site and it does not perform standard http
> authentication so --username and --password cannot be used to logon to
> that page.
>
> You have to supply the username and password using --post-data there
> aswell. If you had followed my advice to use the Live Headers extension
> with Firefox you would have seen exactly what to do, please use that
> tool aswell as learn some http basics and you will soon learn how to
> perform what you want. Because since we do not have access to that site
> of yours (no username or password) we as a community will have quite
> some hard time to tell you exactly how to proceed since we cannot test
> things at our end.
>
> Anyways, as I wrote I tested to perform a logon attempt with the site
> and with Live HTTP Headers extension active I could see that the
> authentication should be performed like this:
>
> wget --post-data "detour=https%3A%2F%2Fwww.collabnet.timeinc.net%
> 2F&loginID=username&password=password&Login=Login" --save-cookies
> cookies.txt "https://www.collabnet.timeinc.net/servlets/TLogin";
>
> Replace the "username" and "password" in the post data with your account
> details.
>
> However since I have no account on that site of yours, I do not know if
> this really works and whether the detour=xxx thing is really needed and
> whether you have to also add a Referer: header or not to the request.
>
> So if I where you:
> 1. Install Live HTTPS Headers extension in Firefox
> 2. Open it's capture window
> 3. Perform a logon and a sql query
> 4. Study the results in the capture window and try to mimic what you see
> with wget.
>
> /HH
>
>
> tis 2012-01-24 klockan 10:11 +0530 skrev Bhargavi N:
> > Hello Henrik,
> >
> >
> >I am saving the log on details in the cookies same way as you have
> > mentioned.
> >
> >
> > But cookie file looks empty..
> >
> >
> > Next i loaded the cookie.txt with the
> > URL: https://www.collabnet.timeinc.net/servlets/AdHocQuery
> >
> >
> > I saved the results page and then opened in firefox, i get the same
> > logon page to ollabnet instead of results page.
> >
> >
> > Steps to run the query:
> >
> >
> > 1) I need to logon to collabnet website:
> > https://www.collabnet.timeinc.net
> >
> >
> > 2) In the collabnet website, we have an option to run the query:
> >
> >
> > https://www.collabnet.timeinc.net/servlets/AdHocQuery
> >
> >
> > I need to go to this page and then need to provide SQL query in the
> > query text area.
> >
> >
> > Then need to submit the button "Run Query", which will submit the
> > query.
> >
> >
> > 3) Next need to download the results page to my local directoy on
> > LINUX box.
> >
> >
> > Please help me regarding this.
> >
> >
> > Thanks !
> >
> >
> > Regards,
> > Bhargavi
> >
> >
> > On Mon, Jan 23, 2012 at 12:43 PM, Henrik Holst
> >  wrote:
> > Ok,
> >
> >   most probably the first site (where you logon) returns a
> > cookie which you must present to the other site (where you
> > perform the sql-query). So
> >
> > wget --username xx --password yy "https://collabnet website"
> > -O /dev/null --save-cookies cookies.txt
> >
> > That will logon and save the resulting cookie in the
> > cookies.txt file. Next you have to send this cookie and your
> > query to the other site:
> >
> > whet --load-cookies cookies.txt "https://the query site"
> > --post-data "the query" -O result
> >
> > And the result should be in the result file.
> >
> > Now as Angel Gonzales wrote you probably have to send the
> > query using the form like --post-data "query=Select%20*%20From
> > %20table" where the sql query in question has to be
> > url-encoded (something that wget ca

Re: [Bug-wget] Fwd: Regarding wget to download webpage

2012-01-24 Thread Henrik Holst
I looked at your web site and it does not perform standard http
authentication so --username and --password cannot be used to logon to
that page.

You have to supply the username and password using --post-data there
aswell. If you had followed my advice to use the Live Headers extension
with Firefox you would have seen exactly what to do, please use that
tool aswell as learn some http basics and you will soon learn how to
perform what you want. Because since we do not have access to that site
of yours (no username or password) we as a community will have quite
some hard time to tell you exactly how to proceed since we cannot test
things at our end.

Anyways, as I wrote I tested to perform a logon attempt with the site
and with Live HTTP Headers extension active I could see that the
authentication should be performed like this:

wget --post-data "detour=https%3A%2F%2Fwww.collabnet.timeinc.net%
2F&loginID=username&password=password&Login=Login" --save-cookies
cookies.txt "https://www.collabnet.timeinc.net/servlets/TLogin";

Replace the "username" and "password" in the post data with your account
details.

However since I have no account on that site of yours, I do not know if
this really works and whether the detour=xxx thing is really needed and
whether you have to also add a Referer: header or not to the request.

So if I where you:
1. Install Live HTTPS Headers extension in Firefox
2. Open it's capture window
3. Perform a logon and a sql query
4. Study the results in the capture window and try to mimic what you see
with wget.

/HH


tis 2012-01-24 klockan 10:11 +0530 skrev Bhargavi N:
> Hello Henrik,
> 
> 
>I am saving the log on details in the cookies same way as you have
> mentioned.
> 
> 
> But cookie file looks empty..
> 
> 
> Next i loaded the cookie.txt with the
> URL: https://www.collabnet.timeinc.net/servlets/AdHocQuery
> 
> 
> I saved the results page and then opened in firefox, i get the same
> logon page to ollabnet instead of results page.
> 
> 
> Steps to run the query: 
> 
> 
> 1) I need to logon to collabnet website:
> https://www.collabnet.timeinc.net
> 
> 
> 2) In the collabnet website, we have an option to run the query:
> 
> 
> https://www.collabnet.timeinc.net/servlets/AdHocQuery
> 
> 
> I need to go to this page and then need to provide SQL query in the
> query text area.
> 
> 
> Then need to submit the button "Run Query", which will submit the
> query.
> 
> 
> 3) Next need to download the results page to my local directoy on
> LINUX box.
> 
> 
> Please help me regarding this.
> 
> 
> Thanks !
> 
> 
> Regards,
> Bhargavi
> 
> 
> On Mon, Jan 23, 2012 at 12:43 PM, Henrik Holst
>  wrote:
> Ok,
> 
>   most probably the first site (where you logon) returns a
> cookie which you must present to the other site (where you
> perform the sql-query). So
> 
> wget --username xx --password yy "https://collabnet website"
> -O /dev/null --save-cookies cookies.txt
> 
> That will logon and save the resulting cookie in the
> cookies.txt file. Next you have to send this cookie and your
> query to the other site:
> 
> whet --load-cookies cookies.txt "https://the query site"
> --post-data "the query" -O result
> 
> And the result should be in the result file.
> 
> Now as Angel Gonzales wrote you probably have to send the
> query using the form like --post-data "query=Select%20*%20From
> %20table" where the sql query in question has to be
> url-encoded (something that wget cannot do for you) which is
> mostly replave all spaces with %20 and also all occurences of
> +, &, " and ' with their respective %hex code.
> 
> A easy way to see exactly which formfield to use in the
> post-data is to install the Live Headers extension to Firefox
> and open it's capture window and peform the query on the site
> using Firefox, you'll then get to see the exact query, what
> the GET string is, what the post-data is etc. It's a very good
> start to go from.
> 
> 
> /HH
> 
> 2012/1/20 Bhargavi N 
> I meant i am unable to logon with to the website.
> 
> 
> Sorry, to confuse you with lot of questions.
> 
> 
> I have to logon to collabnet website first, providing
> username and passwd.
> 
> 
> Next i want to go to servlet page where i can provide
> SQL query in the textarea of the form.
> 
> 
> Then i need to submit the form and download the
> results page to my local directory in linux box.
> 
> 
> I will be running wget command on linux box.

Re: [Bug-wget] Fwd: Regarding wget to download webpage

2012-01-20 Thread Henrik Holst
Well, it normally should work with this:
  wget "https://collabnet.net/servlets/adhocquery"; --user="username"
--password="password" --post-data "select * from emp" -O output_filename

But then according to the man page, wget only supports post-data this is in
urlencoded format like "key=value&key2=value2" so wget might not support
sql queries at all. Perhaps cURL works better for your use case.

/HH

2012/1/20 Bhargavi N 

> Including the team to this mail chain..
>
> -- Forwarded message --
> From: Bhargavi N 
> Date: Fri, Jan 20, 2012 at 4:30 PM
> Subject: Re: [Bug-wget] Regarding wget to download webpage
> To: Dayanand Kumbhar , tle...@exelana.com,
> keis...@gmail.com
>
>
>
> Hello Everyone,
>
>  I am extremely thankful to all of you for the help regarding wget. But
> still i am unable to get the right page downloaded.
>
> Requirement:
>
>  I need to run adhoc sql query on remote web page. ie; collabnet. The
> webpage has text area where i can enter sql query and then click submit.
>
> Once i submit sql runs and results are displayed in the page.
>
> This REMOTE page is call to servlet ie;
> https://collabnet.net/servlets/adhocquery for example.
>
> I want to pass SQL query as form data through POST/GET method and get the
> sql results page downloaded to my local directory on LINUX box. I am
> running wget on LINUX host on commandline / shell script.
>
> I will invoke wget with the URL https://collabnet.net/servlets/adhocqueryand
> formdata ie; SQL as --post-data "select * from emp". Finally i want to
> download the sql query results page to my local directory.
>
> I tried all the options that you suggested me, but still nable to get the
> right page. Every time it is displaying me the index page for collabnet,
> which i do not need. I need to run the servelet page with the formdata
> passed to it.
>
> Then i want to get the results page downloaded to my local directory on my
> linux host.
>
> Please help me regarding this.
>
> Again thanking all of you for your reply and helping me out.
>
> Thanks !
>
> Regards,
> Bhargavi
>
> On Wed, Jan 18, 2012 at 12:40 PM, Dayanand Kumbhar  >wrote:
>
> > Hi
> >
> > i am assuming you have to post the query string
> > You can use this
> >
> > wget -qO- http://example.com/page.php --post-data "SQL QUERY" | cat
> >
> >
> > On Wed, Jan 18, 2012 at 12:24 PM, Bhargavi N  wrote:
> >
> >> Hello Team,
> >>
> >>   My name is Bhargavi and i am working on UNIX for over 7years.
> >>
> >> I work on project, that needs to download results from remote web page.
> >> This web page has adhoc sql querying option. I need to run this sql
> query
> >> on webpage and get the results downloaded using wget.
> >>
> >> How can i do that ?
> >>
> >> Please suggest me way to download the results page using wget.
> >>
> >> Thanks !
> >>
> >> Warm Regards,
> >> Bhargavi
> >>
> >
> >
> >
> > --
> > Dayanand Kumbhar
> >
>


[Bug-wget] Fwd: Regarding wget to download webpage

2012-01-20 Thread Bhargavi N
Including the team to this mail chain..

-- Forwarded message --
From: Bhargavi N 
Date: Fri, Jan 20, 2012 at 4:30 PM
Subject: Re: [Bug-wget] Regarding wget to download webpage
To: Dayanand Kumbhar , tle...@exelana.com,
keis...@gmail.com



Hello Everyone,

  I am extremely thankful to all of you for the help regarding wget. But
still i am unable to get the right page downloaded.

Requirement:

 I need to run adhoc sql query on remote web page. ie; collabnet. The
webpage has text area where i can enter sql query and then click submit.

Once i submit sql runs and results are displayed in the page.

This REMOTE page is call to servlet ie;
https://collabnet.net/servlets/adhocquery for example.

I want to pass SQL query as form data through POST/GET method and get the
sql results page downloaded to my local directory on LINUX box. I am
running wget on LINUX host on commandline / shell script.

I will invoke wget with the URL https://collabnet.net/servlets/adhocquery and
formdata ie; SQL as --post-data "select * from emp". Finally i want to
download the sql query results page to my local directory.

I tried all the options that you suggested me, but still nable to get the
right page. Every time it is displaying me the index page for collabnet,
which i do not need. I need to run the servelet page with the formdata
passed to it.

Then i want to get the results page downloaded to my local directory on my
linux host.

Please help me regarding this.

Again thanking all of you for your reply and helping me out.

Thanks !

Regards,
Bhargavi

On Wed, Jan 18, 2012 at 12:40 PM, Dayanand Kumbhar wrote:

> Hi
>
> i am assuming you have to post the query string
> You can use this
>
> wget -qO- http://example.com/page.php --post-data "SQL QUERY" | cat
>
>
> On Wed, Jan 18, 2012 at 12:24 PM, Bhargavi N  wrote:
>
>> Hello Team,
>>
>>   My name is Bhargavi and i am working on UNIX for over 7years.
>>
>> I work on project, that needs to download results from remote web page.
>> This web page has adhoc sql querying option. I need to run this sql query
>> on webpage and get the results downloaded using wget.
>>
>> How can i do that ?
>>
>> Please suggest me way to download the results page using wget.
>>
>> Thanks !
>>
>> Warm Regards,
>> Bhargavi
>>
>
>
>
> --
> Dayanand Kumbhar
>