Re: [Bug-wget] Fwd: Regarding wget to download webpage
Hello Henrik, Thanks Thanks Thanks a LOT!! Thanks Entire wget team for helping me so generously. Even in your busy schedules. Issue is now resolved. I used Live HTTP headers add on as suggested by you and it really helped me. Initially i passed all the headers as displayed in the capture window, with "--header" option of wget. Later i understood that session details have to be captured. I used "--keep-session-cookies" option of wget, since sessions details have to be saved. I saved the session details in cookies.txt, which i used later to load ADHOC query page and run the query. Below are the commands i used to get the query results. wget -O logon.html --keep-session-cookies --save-cookies cookies.txt --post-data 'loginID=mylogon&password=mypasswd' https://www.collabnet.timeinc.net wget -O query_res.html --load-cookies cookies.txt " https://www.collabnet.timeinc.net/servlets/AdHocQuery?query=select+count%28*%29+from+HELM_USER&Button=Run+query " Special thanks to you and whole team for sparing your time to help me out with the issue. Please let me know if i can be part of your team or i can be of any help to you. HAVE NICE DAY !! With wishes, Bhargavi On Tue, Jan 24, 2012 at 3:20 PM, Henrik Holst wrote: > I looked at your web site and it does not perform standard http > authentication so --username and --password cannot be used to logon to > that page. > > You have to supply the username and password using --post-data there > aswell. If you had followed my advice to use the Live Headers extension > with Firefox you would have seen exactly what to do, please use that > tool aswell as learn some http basics and you will soon learn how to > perform what you want. Because since we do not have access to that site > of yours (no username or password) we as a community will have quite > some hard time to tell you exactly how to proceed since we cannot test > things at our end. > > Anyways, as I wrote I tested to perform a logon attempt with the site > and with Live HTTP Headers extension active I could see that the > authentication should be performed like this: > > wget --post-data "detour=https%3A%2F%2Fwww.collabnet.timeinc.net% > 2F&loginID=username&password=password&Login=Login" --save-cookies > cookies.txt "https://www.collabnet.timeinc.net/servlets/TLogin"; > > Replace the "username" and "password" in the post data with your account > details. > > However since I have no account on that site of yours, I do not know if > this really works and whether the detour=xxx thing is really needed and > whether you have to also add a Referer: header or not to the request. > > So if I where you: > 1. Install Live HTTPS Headers extension in Firefox > 2. Open it's capture window > 3. Perform a logon and a sql query > 4. Study the results in the capture window and try to mimic what you see > with wget. > > /HH > > > tis 2012-01-24 klockan 10:11 +0530 skrev Bhargavi N: > > Hello Henrik, > > > > > >I am saving the log on details in the cookies same way as you have > > mentioned. > > > > > > But cookie file looks empty.. > > > > > > Next i loaded the cookie.txt with the > > URL: https://www.collabnet.timeinc.net/servlets/AdHocQuery > > > > > > I saved the results page and then opened in firefox, i get the same > > logon page to ollabnet instead of results page. > > > > > > Steps to run the query: > > > > > > 1) I need to logon to collabnet website: > > https://www.collabnet.timeinc.net > > > > > > 2) In the collabnet website, we have an option to run the query: > > > > > > https://www.collabnet.timeinc.net/servlets/AdHocQuery > > > > > > I need to go to this page and then need to provide SQL query in the > > query text area. > > > > > > Then need to submit the button "Run Query", which will submit the > > query. > > > > > > 3) Next need to download the results page to my local directoy on > > LINUX box. > > > > > > Please help me regarding this. > > > > > > Thanks ! > > > > > > Regards, > > Bhargavi > > > > > > On Mon, Jan 23, 2012 at 12:43 PM, Henrik Holst > > wrote: > > Ok, > > > > most probably the first site (where you logon) returns a > > cookie which you must present to the other site (where you > > perform the sql-query). So > > > > wget --username xx --password yy "https://collabnet website" > > -O /dev/null --save-cookies cookies.txt > > > > That will logon and save the resulting cookie in the > > cookies.txt file. Next you have to send this cookie and your > > query to the other site: > > > > whet --load-cookies cookies.txt "https://the query site" > > --post-data "the query" -O result > > > > And the result should be in the result file. > > > > Now as Angel Gonzales wrote you probably have to send the > > query using the form like --post-data "query=Select%20*%20From > > %20table" where the sql query in question has to be > > url-encoded (something that wget ca
Re: [Bug-wget] Fwd: Regarding wget to download webpage
I looked at your web site and it does not perform standard http authentication so --username and --password cannot be used to logon to that page. You have to supply the username and password using --post-data there aswell. If you had followed my advice to use the Live Headers extension with Firefox you would have seen exactly what to do, please use that tool aswell as learn some http basics and you will soon learn how to perform what you want. Because since we do not have access to that site of yours (no username or password) we as a community will have quite some hard time to tell you exactly how to proceed since we cannot test things at our end. Anyways, as I wrote I tested to perform a logon attempt with the site and with Live HTTP Headers extension active I could see that the authentication should be performed like this: wget --post-data "detour=https%3A%2F%2Fwww.collabnet.timeinc.net% 2F&loginID=username&password=password&Login=Login" --save-cookies cookies.txt "https://www.collabnet.timeinc.net/servlets/TLogin"; Replace the "username" and "password" in the post data with your account details. However since I have no account on that site of yours, I do not know if this really works and whether the detour=xxx thing is really needed and whether you have to also add a Referer: header or not to the request. So if I where you: 1. Install Live HTTPS Headers extension in Firefox 2. Open it's capture window 3. Perform a logon and a sql query 4. Study the results in the capture window and try to mimic what you see with wget. /HH tis 2012-01-24 klockan 10:11 +0530 skrev Bhargavi N: > Hello Henrik, > > >I am saving the log on details in the cookies same way as you have > mentioned. > > > But cookie file looks empty.. > > > Next i loaded the cookie.txt with the > URL: https://www.collabnet.timeinc.net/servlets/AdHocQuery > > > I saved the results page and then opened in firefox, i get the same > logon page to ollabnet instead of results page. > > > Steps to run the query: > > > 1) I need to logon to collabnet website: > https://www.collabnet.timeinc.net > > > 2) In the collabnet website, we have an option to run the query: > > > https://www.collabnet.timeinc.net/servlets/AdHocQuery > > > I need to go to this page and then need to provide SQL query in the > query text area. > > > Then need to submit the button "Run Query", which will submit the > query. > > > 3) Next need to download the results page to my local directoy on > LINUX box. > > > Please help me regarding this. > > > Thanks ! > > > Regards, > Bhargavi > > > On Mon, Jan 23, 2012 at 12:43 PM, Henrik Holst > wrote: > Ok, > > most probably the first site (where you logon) returns a > cookie which you must present to the other site (where you > perform the sql-query). So > > wget --username xx --password yy "https://collabnet website" > -O /dev/null --save-cookies cookies.txt > > That will logon and save the resulting cookie in the > cookies.txt file. Next you have to send this cookie and your > query to the other site: > > whet --load-cookies cookies.txt "https://the query site" > --post-data "the query" -O result > > And the result should be in the result file. > > Now as Angel Gonzales wrote you probably have to send the > query using the form like --post-data "query=Select%20*%20From > %20table" where the sql query in question has to be > url-encoded (something that wget cannot do for you) which is > mostly replave all spaces with %20 and also all occurences of > +, &, " and ' with their respective %hex code. > > A easy way to see exactly which formfield to use in the > post-data is to install the Live Headers extension to Firefox > and open it's capture window and peform the query on the site > using Firefox, you'll then get to see the exact query, what > the GET string is, what the post-data is etc. It's a very good > start to go from. > > > /HH > > 2012/1/20 Bhargavi N > I meant i am unable to logon with to the website. > > > Sorry, to confuse you with lot of questions. > > > I have to logon to collabnet website first, providing > username and passwd. > > > Next i want to go to servlet page where i can provide > SQL query in the textarea of the form. > > > Then i need to submit the form and download the > results page to my local directory in linux box. > > > I will be running wget command on linux box.
Re: [Bug-wget] Fwd: Regarding wget to download webpage
Well, it normally should work with this: wget "https://collabnet.net/servlets/adhocquery"; --user="username" --password="password" --post-data "select * from emp" -O output_filename But then according to the man page, wget only supports post-data this is in urlencoded format like "key=value&key2=value2" so wget might not support sql queries at all. Perhaps cURL works better for your use case. /HH 2012/1/20 Bhargavi N > Including the team to this mail chain.. > > -- Forwarded message -- > From: Bhargavi N > Date: Fri, Jan 20, 2012 at 4:30 PM > Subject: Re: [Bug-wget] Regarding wget to download webpage > To: Dayanand Kumbhar , tle...@exelana.com, > keis...@gmail.com > > > > Hello Everyone, > > I am extremely thankful to all of you for the help regarding wget. But > still i am unable to get the right page downloaded. > > Requirement: > > I need to run adhoc sql query on remote web page. ie; collabnet. The > webpage has text area where i can enter sql query and then click submit. > > Once i submit sql runs and results are displayed in the page. > > This REMOTE page is call to servlet ie; > https://collabnet.net/servlets/adhocquery for example. > > I want to pass SQL query as form data through POST/GET method and get the > sql results page downloaded to my local directory on LINUX box. I am > running wget on LINUX host on commandline / shell script. > > I will invoke wget with the URL https://collabnet.net/servlets/adhocqueryand > formdata ie; SQL as --post-data "select * from emp". Finally i want to > download the sql query results page to my local directory. > > I tried all the options that you suggested me, but still nable to get the > right page. Every time it is displaying me the index page for collabnet, > which i do not need. I need to run the servelet page with the formdata > passed to it. > > Then i want to get the results page downloaded to my local directory on my > linux host. > > Please help me regarding this. > > Again thanking all of you for your reply and helping me out. > > Thanks ! > > Regards, > Bhargavi > > On Wed, Jan 18, 2012 at 12:40 PM, Dayanand Kumbhar >wrote: > > > Hi > > > > i am assuming you have to post the query string > > You can use this > > > > wget -qO- http://example.com/page.php --post-data "SQL QUERY" | cat > > > > > > On Wed, Jan 18, 2012 at 12:24 PM, Bhargavi N wrote: > > > >> Hello Team, > >> > >> My name is Bhargavi and i am working on UNIX for over 7years. > >> > >> I work on project, that needs to download results from remote web page. > >> This web page has adhoc sql querying option. I need to run this sql > query > >> on webpage and get the results downloaded using wget. > >> > >> How can i do that ? > >> > >> Please suggest me way to download the results page using wget. > >> > >> Thanks ! > >> > >> Warm Regards, > >> Bhargavi > >> > > > > > > > > -- > > Dayanand Kumbhar > > >
[Bug-wget] Fwd: Regarding wget to download webpage
Including the team to this mail chain.. -- Forwarded message -- From: Bhargavi N Date: Fri, Jan 20, 2012 at 4:30 PM Subject: Re: [Bug-wget] Regarding wget to download webpage To: Dayanand Kumbhar , tle...@exelana.com, keis...@gmail.com Hello Everyone, I am extremely thankful to all of you for the help regarding wget. But still i am unable to get the right page downloaded. Requirement: I need to run adhoc sql query on remote web page. ie; collabnet. The webpage has text area where i can enter sql query and then click submit. Once i submit sql runs and results are displayed in the page. This REMOTE page is call to servlet ie; https://collabnet.net/servlets/adhocquery for example. I want to pass SQL query as form data through POST/GET method and get the sql results page downloaded to my local directory on LINUX box. I am running wget on LINUX host on commandline / shell script. I will invoke wget with the URL https://collabnet.net/servlets/adhocquery and formdata ie; SQL as --post-data "select * from emp". Finally i want to download the sql query results page to my local directory. I tried all the options that you suggested me, but still nable to get the right page. Every time it is displaying me the index page for collabnet, which i do not need. I need to run the servelet page with the formdata passed to it. Then i want to get the results page downloaded to my local directory on my linux host. Please help me regarding this. Again thanking all of you for your reply and helping me out. Thanks ! Regards, Bhargavi On Wed, Jan 18, 2012 at 12:40 PM, Dayanand Kumbhar wrote: > Hi > > i am assuming you have to post the query string > You can use this > > wget -qO- http://example.com/page.php --post-data "SQL QUERY" | cat > > > On Wed, Jan 18, 2012 at 12:24 PM, Bhargavi N wrote: > >> Hello Team, >> >> My name is Bhargavi and i am working on UNIX for over 7years. >> >> I work on project, that needs to download results from remote web page. >> This web page has adhoc sql querying option. I need to run this sql query >> on webpage and get the results downloaded using wget. >> >> How can i do that ? >> >> Please suggest me way to download the results page using wget. >> >> Thanks ! >> >> Warm Regards, >> Bhargavi >> > > > > -- > Dayanand Kumbhar >