Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, Jun 09, 2016 at 10:14:53AM +, Bob wrote:

[...]

> Hello Tomas & Jeremy,
> 
> Finally I got the combination. The redirected links along with
> username/password is not providing any cookie jar. So I went back to
> the main page i.e. form page. I collected the cookie there, then
> fire the curl with post data.
> 
> curl -c mycookie -s "http:///login.jsp"
> 
> curl -X POST -c mycookie \
> -d username= \
> -d password=  \
>  http:///login
> 
> thanks to all of you for your kind support

Ah. A pre-emptive session cookie. Turtles all the way down :-)

regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldZQvEACgkQBcgs9XrR2kZmUwCff1O/XAKtmMNMcPluldYsq0Zs
c+YAn1ZQGkQ1S09kWio9xy6jJWVVl1uI
=yP+h
-END PGP SIGNATURE-

Re: curl and form submission

[Bob]

On Thursday 09 June 2016 08:10 AM, to...@tuxteam.de wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, Jun 09, 2016 at 07:49:28AM +, Bob wrote:


On Thursday 09 June 2016 07:13 AM, to...@tuxteam.de wrote:

[...]


Thanks for your explanation. I did "curl -c" but no luck as it
doesn't have any cookie.

curl -c mycookie " 
http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
"

# ls mycookie
ls: cannot access 'mycookie': No such file or directory

whereas I can see during curl operation

[]








[/...]

The form leads explicitly to a POST request. This is one where
the data is sent in the HTTP body (and not as query parameters
in the URL, as your curl command above is trying).

Some web frameworks accept (equivalently) both variants, some
not.

Try sending as a POST request:

   curl -X POST -c mycookie \
-d username= -d password= \
http:///login

(replacing, of course , ,  by more interesting
values. I've broken up the long line with backslashes, don't type
in those).

For more on the -d option and its cousins, look them up in the
curl man page under --data, --data-ascii, --data-binary and
friends.

Is anything in the cookiejar yet?


Issue is, this direct link  (which is a redirect link from actual
form page) doesn't create any cookie jar


curl -c mycookie " 
http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
"

though same is running well on browser and logging in immediately.

--trace-ascii investigation shows as below


: GET /login1.html?a=%3F%2B%2F%40=%3F%2B%2F%4
0040: 0 HTTP/1.1
004c: Host: 
005b: User-Agent: curl/7.47.0
0074: Accept: */*
...
..
0300: /head>..http://1.1.1.1/login;>.<

Ouch.

This is, as far as I can see, a "javascript redirect". It reflects on
the sad state of our so-called "industry".

As far as I can see, the browser "bounces" the request to
http://1.1.1.1/login with the same fields. So try to direct your
curl request at this (new?) URI. You don't see the browser doing
that (so it seems there's one more back-and-forth)

Of course, the browser is sending more/different stuff in the header,
like the referer, some string for the user agent, etc. The server
might be acting on some of that information.

That's why I suggested spying on what the browser does and trying
to work from there. It's easier to cut off superfluous stuff from
a "known good" case than to try adding stuff blindly.

Get acquainted either with Wireshark (only if your connection isn't
HTTPS! [1]) or with Firebug. It's both free software.


[1] It's possible to spy on an HTTPS connection with Wireshark. But
for that, you'd have to learn how to get your browser to cough
up the session key and to insert that in Wireshark. That raises
the (already pretty high) bar quite a bit.



Hello Tomas & Jeremy,

Finally I got the combination. The redirected links along with 
username/password is not providing any cookie jar. So I went back to the 
main page i.e. form page. I collected the cookie there, then fire the 
curl with post data.


curl -c mycookie -s "http:///login.jsp"

curl -X POST -c mycookie \
-d username= \
-d password=  \
 http:///login

thanks to all of you for your kind support

Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, Jun 09, 2016 at 07:49:28AM +, Bob wrote:
> 
> 
> On Thursday 09 June 2016 07:13 AM, to...@tuxteam.de wrote:
> >[...]
> >
> >>Thanks for your explanation. I did "curl -c" but no luck as it
> >>doesn't have any cookie.
> >>
> >>curl -c mycookie " 
> >>http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
> >>"
> >>
> >># ls mycookie
> >>ls: cannot access 'mycookie': No such file or directory
> >>
> >>whereas I can see during curl operation
> >>
> >>[]
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>[/...]
> >The form leads explicitly to a POST request. This is one where
> >the data is sent in the HTTP body (and not as query parameters
> >in the URL, as your curl command above is trying).
> >
> >Some web frameworks accept (equivalently) both variants, some
> >not.
> >
> >Try sending as a POST request:
> >
> >   curl -X POST -c mycookie \
> >-d username= -d password= \
> >http:///login
> >
> >(replacing, of course , ,  by more interesting
> >values. I've broken up the long line with backslashes, don't type
> >in those).
> >
> >For more on the -d option and its cousins, look them up in the
> >curl man page under --data, --data-ascii, --data-binary and
> >friends.
> >
> >Is anything in the cookiejar yet?
> >
> Issue is, this direct link  (which is a redirect link from actual
> form page) doesn't create any cookie jar
> 
> 
> curl -c mycookie " 
> http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
> "
> 
> though same is running well on browser and logging in immediately.
> 
> --trace-ascii investigation shows as below
> 
> 
> : GET /login1.html?a=%3F%2B%2F%40=%3F%2B%2F%4
> 0040: 0 HTTP/1.1
> 004c: Host: 
> 005b: User-Agent: curl/7.47.0
> 0074: Accept: */*
> ...
> ..
> 0300: /head>.. 0340:  action="http://1.1.1.1/login;>.<

Ouch.

This is, as far as I can see, a "javascript redirect". It reflects on
the sad state of our so-called "industry".

As far as I can see, the browser "bounces" the request to
http://1.1.1.1/login with the same fields. So try to direct your
curl request at this (new?) URI. You don't see the browser doing
that (so it seems there's one more back-and-forth)

Of course, the browser is sending more/different stuff in the header,
like the referer, some string for the user agent, etc. The server
might be acting on some of that information.

That's why I suggested spying on what the browser does and trying
to work from there. It's easier to cut off superfluous stuff from
a "known good" case than to try adding stuff blindly.

Get acquainted either with Wireshark (only if your connection isn't
HTTPS! [1]) or with Firebug. It's both free software.


[1] It's possible to spy on an HTTPS connection with Wireshark. But
   for that, you'd have to learn how to get your browser to cough
   up the session key and to insert that in Wireshark. That raises
   the (already pretty high) bar quite a bit.

regards
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldZJGoACgkQBcgs9XrR2kZTwACfd28LnYp7qyqAJonUntc+8P/M
XaYAnRrsPSSqV9vJgHjbMf2wqtKr0zxN
=ckK0
-END PGP SIGNATURE-

Re: curl and form submission

[Bob]

On Thursday 09 June 2016 07:13 AM, to...@tuxteam.de wrote:

[...]


Thanks for your explanation. I did "curl -c" but no luck as it
doesn't have any cookie.

curl -c mycookie " 
http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
"

# ls mycookie
ls: cannot access 'mycookie': No such file or directory

whereas I can see during curl operation

[]








[/...]

The form leads explicitly to a POST request. This is one where
the data is sent in the HTTP body (and not as query parameters
in the URL, as your curl command above is trying).

Some web frameworks accept (equivalently) both variants, some
not.

Try sending as a POST request:

   curl -X POST -c mycookie \
-d username= -d password= \
http:///login

(replacing, of course , ,  by more interesting
values. I've broken up the long line with backslashes, don't type
in those).

For more on the -d option and its cousins, look them up in the
curl man page under --data, --data-ascii, --data-binary and
friends.

Is anything in the cookiejar yet?

Issue is, this direct link  (which is a redirect link from actual form 
page) doesn't create any cookie jar



curl -c mycookie " 
http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40 
"


though same is running well on browser and logging in immediately.

--trace-ascii investigation shows as below


: GET /login1.html?a=%3F%2B%2F%40=%3F%2B%2F%4
0040: 0 HTTP/1.1
004c: Host: 
005b: User-Agent: curl/7.47.0
0074: Accept: */*
...
..
0300: /head>..http://1.1.1.1/login;>.<

Don't know why cookie is not coming with this link.

Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, Jun 09, 2016 at 05:34:25AM +, Bob wrote:
> 
> On Wednesday 08 June 2016 05:01 PM, Jeremy Nicoll wrote:
> >On Wed, 8 Jun 2016, at 17:16, Bob wrote:

[...]

> Thanks for your explanation. I did "curl -c" but no luck as it
> doesn't have any cookie.
> 
> curl -c mycookie " 
> http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
> "
> 
> # ls mycookie
> ls: cannot access 'mycookie': No such file or directory
> 
> whereas I can see during curl operation
> 
> []
> 
> 
> 
> 
> 
> 
> 
> 
> [/...]

The form leads explicitly to a POST request. This is one where
the data is sent in the HTTP body (and not as query parameters
in the URL, as your curl command above is trying).

Some web frameworks accept (equivalently) both variants, some
not.

Try sending as a POST request:

  curl -X POST -c mycookie \
   -d username= -d password= \
   http:///login

(replacing, of course , ,  by more interesting
values. I've broken up the long line with backslashes, don't type
in those).

For more on the -d option and its cousins, look them up in the
curl man page under --data, --data-ascii, --data-binary and
friends.

Is anything in the cookiejar yet?

regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldZFx8ACgkQBcgs9XrR2kZuTQCfY/gdtNzYZwKBw8P2BBqLRqaG
K4IAn35AMRryXkB+h/tNU+QLxFsjIN9j
=8a03
-END PGP SIGNATURE-

Re: curl and form submission

[Bob]

On Wednesday 08 June 2016 05:01 PM, Jeremy Nicoll wrote:

On Wed, 8 Jun 2016, at 17:16, Bob wrote:


Well Tomas, I like to share with you that your guess is again correct.
This link quickly redirects to http:// (seems the session checked
here) and then to home page. I'm tying to figure out the cookie part now.

It's important that at the start of this process you have an empty
cookie file.  Then
you give its name to curl on every call; curl will store any cookies
that the site sends
to you in that file (you shouldn't have to pay any attention to what
they actually
are).  In every subsequent curl command, you again specify the same file
and the
cookies whose values are stored in it are sent back to the server by
curl.  Each time
the server might change cookies but the whole thing should just work.
At the very
end of the process, throw the cookie file away.

Also, in the curl command you can specify names of files into which curl
will place
the HTML of the page you just fetched, and you can sepcify a second file
to have the
headers (which accompany every page sent to a browser, but users don't
ever see
them) stored.  It's good to do this because those headers will include
any server
error codes - useful for diagnosing problems in the whole process - and
they also
contain eg redirection instructions, if the server wants you to make
another request
for a different page.  if you don't have the headers file for your
script to read and check
it can be really difficult to make this sort of thing work!


Hello Jeremy,

Thanks for your explanation. I did "curl -c" but no luck as it doesn't 
have any cookie.


curl -c mycookie " 
http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40 
"


# ls mycookie
ls: cannot access 'mycookie': No such file or directory

whereas I can see during curl operation

[]








[/...]

Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Wed, Jun 08, 2016 at 01:06:32PM -0700, A wrote:
> I'm no expert on wget, but have you tried RTFM:
> 
> ‘--user=user’
> ‘--password=password’
> 
>Specify the username user and password password for both FTP and
>HTTP file retrieval. These parameters can be overridden using the
>‘--ftp-user’ and ‘--ftp-password’ options for FTP connections and
>the ‘--http-user’ and ‘--http-password’ options for HTTP connections.
> 
> 
> And just for relative completeness:
> 
> ‘--ask-password’
> 
>Prompt for a password for each connection established. Cannot be
>specified when ‘--password’ is being used, because they are mutually
>exclusive.

Unfortunately, this is for "standard" http authentification. I
believe that the server is doing its own thing here with auth, as
is, (unfortunately) custom.

It might be worth a try, though: it'd save one http turnaround.

Regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldYgCAACgkQBcgs9XrR2kbxDQCePw5lcKpTN1XpgyDVmY7gMPVv
7VwAn0CxKDv+j7Bw7nDvAgXAnVpy3BLj
=vWQ3
-END PGP SIGNATURE-

Re: curl and form submission

[A]

I'm no expert on wget, but have you tried RTFM:

‘--user=user’
‘--password=password’

   Specify the username user and password password for both FTP and
   HTTP file retrieval. These parameters can be overridden using the
   ‘--ftp-user’ and ‘--ftp-password’ options for FTP connections and
   the ‘--http-user’ and ‘--http-password’ options for HTTP connections.


And just for relative completeness:

‘--ask-password’

   Prompt for a password for each connection established. Cannot be
   specified when ‘--password’ is being used, because they are mutually
   exclusive.


- A


On 06/07/2016 05:23 PM, Oliver Briscbois wrote:

On 2016-06-07, Bob  wrote:

Hello list,

I'm trying to automate my internet login which is based on a web form.

Good question. I hope someone follows up with an answer. I'll be
following this thread. I've been trying the same thing with wget without
success

Oliver.

Re: curl and form submission

[Jeremy Nicoll]

On Wed, 8 Jun 2016, at 17:16, Bob wrote:

> Well Tomas, I like to share with you that your guess is again correct.
> This link quickly redirects to http:// (seems the session checked 
> here) and then to home page. I'm tying to figure out the cookie part now.

It's important that at the start of this process you have an empty
cookie file.  Then 
you give its name to curl on every call; curl will store any cookies
that the site sends
to you in that file (you shouldn't have to pay any attention to what
they actually 
are).  In every subsequent curl command, you again specify the same file
and the 
cookies whose values are stored in it are sent back to the server by
curl.  Each time
the server might change cookies but the whole thing should just work. 
At the very 
end of the process, throw the cookie file away.

Also, in the curl command you can specify names of files into which curl
will place
the HTML of the page you just fetched, and you can sepcify a second file
to have the
headers (which accompany every page sent to a browser, but users don't
ever see 
them) stored.  It's good to do this because those headers will include
any server
error codes - useful for diagnosing problems in the whole process - and
they also 
contain eg redirection instructions, if the server wants you to make
another request
for a different page.  if you don't have the headers file for your
script to read and check
it can be really difficult to make this sort of thing work!

-- 
Jeremy Nicoll - my opinions are my own.

Re: curl and form submission

[Bob]

On Wednesday 08 June 2016 01:28 PM, to...@tuxteam.de wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Wed, Jun 08, 2016 at 01:02:54PM +, Bob wrote:

[...]


Hello Tomas,

Thanks for your explanation. At my end a funny thing happens which
now questions the whole web based authentication of this provider. I
have discovered a simple link (which happens during a quick
redirection after pressing login button)

  http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40

I logout to kill my session and then put the same on browser, I
logged in straight  :-)

Again, your browser is pulling your leg. What you are doing with this
command line is equivalent to sending your filled in login form (you
are sending what the browser would do via "URL query parameters".

The next step is the server sending back a fresh session ID in a cookie
(and possibly a redirect to the "real" page). From then on you're in
business.


How can I send that link through command line then ?

Simple: just "curl" it:

curl 
'http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40'

but as I said, this'll just give you the first step in a multi-step
process. You'd have possibly to follow (at least) one redirect,
offering the cookie you just harvested. That's where curl's options
- -c (set cookie jar) and -L (follow redirect) come in.

Try to find a way to watch what your browser's doing. It's much
more instructive than reading my incomplete guesses :-)



Well Tomas, I like to share with you that your guess is again correct.
This link quickly redirects to http:// (seems the session checked 
here) and then to home page. I'm tying to figure out the cookie part now.

Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Wed, Jun 08, 2016 at 01:02:54PM +, Bob wrote:

[...]

> Hello Tomas,
> 
> Thanks for your explanation. At my end a funny thing happens which
> now questions the whole web based authentication of this provider. I
> have discovered a simple link (which happens during a quick
> redirection after pressing login button)
> 
>  http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40
> 
> I logout to kill my session and then put the same on browser, I
> logged in straight  :-)

Again, your browser is pulling your leg. What you are doing with this
command line is equivalent to sending your filled in login form (you
are sending what the browser would do via "URL query parameters".

The next step is the server sending back a fresh session ID in a cookie
(and possibly a redirect to the "real" page). From then on you're in
business.

> How can I send that link through command line then ?

Simple: just "curl" it:

   curl 
'http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40'

but as I said, this'll just give you the first step in a multi-step
process. You'd have possibly to follow (at least) one redirect,
offering the cookie you just harvested. That's where curl's options
- -c (set cookie jar) and -L (follow redirect) come in.

Try to find a way to watch what your browser's doing. It's much
more instructive than reading my incomplete guesses :-)

regards
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldYHW8ACgkQBcgs9XrR2kZ9qACfQ+n5GTVFsO22i060kFqu3OJR
RWoAmwcgPTy21GVVCjaZ6+UuDVfzCgT2
=qaKj
-END PGP SIGNATURE-

Re: curl and form submission

[Bob]

On Wednesday 08 June 2016 10:20 AM, to...@tuxteam.de wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

[please keep replies to list, so that others can chime in]

On Wed, Jun 08, 2016 at 10:03:38AM +, Bob wrote:

On Wednesday 08 June 2016 07:11 AM, to...@tuxteam.de wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, Jun 07, 2016 at 03:33:40PM +, Bob wrote:

Hello list,

I'm trying to automate my internet login which is based on a web
form [...]
Your clue will be helpful.

Most probably yes: those "modern" login forms elicit a response with
some session key which the browser uses from then on to "prove" to
the server that authentication happened.

You have to simulate that.

[...]


You are correct Tomas,

When I call the curl and investigate very first few lines I get the session

~
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Set-Cookie: JSESSIONID=A22B8981871A8A4779F5DEB8ACC5D1B8; Path=/loginarea
< Content-Type: text/html
< Transfer-Encoding: chunked

So, I wonder If I can make my previous command [curl -sd
"Username===Login"
http:///Login.jsp
] work with correct session. How can I collect the session then ? I
like to share, if any other tool can help me to autologin with the
web based form, I am open with that too.

First you have to realize that the whole process consists of several
back-and-forths (your web browser hides this fact from you). The
client contacts the server, the server sees "no session" and redirects
to the login page, this comes back with a session ID, then the
browser goes to do useful work, providing the just-received cookie.

Then, there's the option -c to curl (cf. curl's excellent, albeit a
bit terse documentation), in which you provide a file name where
curl is supposed to manage cookies for a request series.

So the rough schema will be (at least):

   curl -c  login-page   # this stores above cookie
   curl -c  some-useful-page # this uses the cookie

perhaps more back-and-forths.

Go install firebug (if you're using firefox as your browser) and have
a look what your browser is doing: this will give you an idea of what
it'd take to teach curl.

Alternatively, and if the connection is not encrypted, you can spy on
it with wireshark.

In any case you'll need a bit of patience to be able to control all
this. Just don't hesitate to come back!

HTH
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldX8XYACgkQBcgs9XrR2kY+sgCfdQwmRSlyTt7zTwkYXqgaLN1/
5fAAniEEP+DfkNeKDaj4dBSN7MFr6Vf6
=qh5B
-END PGP SIGNATURE-



Hello Tomas,

Thanks for your explanation. At my end a funny thing happens which now 
questions the whole web based authentication of this provider. I have 
discovered a simple link (which happens during a quick redirection after 
pressing login button)


 http:///login1.html?a=%3F%2B%2F%40=%3F%2B%2F%40

I logout to kill my session and then put the same on browser, I logged 
in straight  :-)


How can I send that link through command line then ?

regards,
Bob

Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

[please keep replies to list, so that others can chime in]

On Wed, Jun 08, 2016 at 10:03:38AM +, Bob wrote:
> 
> On Wednesday 08 June 2016 07:11 AM, to...@tuxteam.de wrote:
> >-BEGIN PGP SIGNED MESSAGE-
> >Hash: SHA1
> >
> >On Tue, Jun 07, 2016 at 03:33:40PM +, Bob wrote:
> >>Hello list,
> >>
> >>I'm trying to automate my internet login which is based on a web
> >>form [...]

> >>Your clue will be helpful.
> >Most probably yes: those "modern" login forms elicit a response with
> >some session key which the browser uses from then on to "prove" to
> >the server that authentication happened.
> >
> >You have to simulate that.

[...]

> You are correct Tomas,
> 
> When I call the curl and investigate very first few lines I get the session
> 
> ~
> < HTTP/1.1 200 OK
> < Server: Apache-Coyote/1.1
> < Set-Cookie: JSESSIONID=A22B8981871A8A4779F5DEB8ACC5D1B8; Path=/loginarea
> < Content-Type: text/html
> < Transfer-Encoding: chunked
> 
> So, I wonder If I can make my previous command [curl -sd
> "Username===Login"
> http:///Login.jsp
> ] work with correct session. How can I collect the session then ? I
> like to share, if any other tool can help me to autologin with the
> web based form, I am open with that too.

First you have to realize that the whole process consists of several
back-and-forths (your web browser hides this fact from you). The
client contacts the server, the server sees "no session" and redirects
to the login page, this comes back with a session ID, then the
browser goes to do useful work, providing the just-received cookie.

Then, there's the option -c to curl (cf. curl's excellent, albeit a
bit terse documentation), in which you provide a file name where
curl is supposed to manage cookies for a request series.

So the rough schema will be (at least):

  curl -c  login-page   # this stores above cookie
  curl -c  some-useful-page # this uses the cookie

perhaps more back-and-forths.

Go install firebug (if you're using firefox as your browser) and have
a look what your browser is doing: this will give you an idea of what
it'd take to teach curl.

Alternatively, and if the connection is not encrypted, you can spy on
it with wireshark.

In any case you'll need a bit of patience to be able to control all
this. Just don't hesitate to come back!

HTH
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldX8XYACgkQBcgs9XrR2kY+sgCfdQwmRSlyTt7zTwkYXqgaLN1/
5fAAniEEP+DfkNeKDaj4dBSN7MFr6Vf6
=qh5B
-END PGP SIGNATURE-

Re: curl and form submission

[Jude DaShiell]

For the wget users, you may want to try using the wput package which is 
intended to fill web forms or was when it was written.  I can't speak 
for how current wput is in terms of web technology you talk to but that 
package may be a little more effective than wget.


On Wed, 8 Jun 2016, Oliver Briscbois wrote:


Date: Tue, 7 Jun 2016 20:23:27
From: Oliver Briscbois <l...@zapto.roth.ca>
To: debian-user@lists.debian.org
Subject: Re: curl and form submission
Resent-Date: Wed,  8 Jun 2016 00:24:48 + (UTC)
Resent-From: debian-user@lists.debian.org

On 2016-06-07, Bob <bobnli...@gmail.com> wrote:

Hello list,

I'm trying to automate my internet login which is based on a web form.


Good question. I hope someone follows up with an answer. I'll be
following this thread. I've been trying the same thing with wget without
success

Oliver.




--

Re: curl and form submission

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, Jun 07, 2016 at 03:33:40PM +, Bob wrote:
> Hello list,
> 
> I'm trying to automate my internet login which is based on a web
> form. I have already checked few tutorial/posts on form submission
> by curl. I have tried various combinations too but nothing worked in
> my case.
> 
> When I inspect the form I get following for username/password text
> box and submit button

[...]

> which is not working unfortunately. Have I missed anything ?
> 
> Your clue will be helpful.

Most probably yes: those "modern" login forms elicit a response with
some session key which the browser uses from then on to "prove" to
the server that authentication happened.

You have to simulate that.

Most of the time, this session key is hidden in a cookie. You can
let curl manage the cookies for you with option -c: it stores the
set cookies in the file you give it and uses it in subsequent requests
appropriately.

Sometimes servers fall back to a session ID embedded in the URL
when the client refuses to use cookies (the ending .jsp suggests
you are talking to a java server pages thingmajig: those do this
kind of fallback -- sometimes). There, you would have to follow
redirects judiciously and possibly enrich your URL with whatever
session ID is there (perhaps embedded in the path, perhaps in
the url params after the ?).

Why those &$%#*$? web designers gave up on "traditional"
authentication is something I've kept asking myself the last
15 years. Some things are "turtles all the way down", the
Intertubes is just a disgusting tangle of turtles.

But there you go.

regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldXxRUACgkQBcgs9XrR2kbZ4ACeKjvXD6ORJKl2Lz0fyaBvs0B5
ClUAnjwWlI38ws4JEkRTcccsCqdOtmdg
=74TJ
-END PGP SIGNATURE-

Re: curl and form submission

[Oliver Briscbois]

On 2016-06-07, Bob  wrote:
> Hello list,
>
> I'm trying to automate my internet login which is based on a web form.

Good question. I hope someone follows up with an answer. I'll be
following this thread. I've been trying the same thing with wget without
success

Oliver.

Re: curl and form submission

[Jeremy Nicoll]

On Tue, 7 Jun 2016, at 16:33, Bob wrote:
> Hello list,
> 
> I'm trying to automate my internet login which is based on a web form. I 
> have already checked few tutorial/posts on form submission by curl. I 
> have tried various combinations too but nothing worked in my case.
> 
> When I inspect the form I get following for username/password text box 
> and submit button
> 
> 
> 
> 
> 
> 


Don't you have to find out what validateForm() would do if the values
provided are 
ok, and make curl do the next step?  That may also depend on whether
there's
code to run for form submit, even if validation has been ok.

-- 
Jeremy Nicoll - my opinions are my own.

curl and form submission

[Bob]

Hello list,

I'm trying to automate my internet login which is based on a web form. I 
have already checked few tutorial/posts on form submission by curl. I 
have tried various combinations too but nothing worked in my case.


When I inspect the form I get following for username/password text box 
and submit button









```

when I check by curl I get below




# curl /Login.jsp | grep Username


  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   Spent Left  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- 
--:--:-- 0function newCookie(Username,value,days) {

   document.cookie = Username+"="+value+expires+"; path=/"; }
function readCookie(Username) {
   var nameSG = Username + "=";
function eraseCookie(Username) {
  newCookie(Username,"",1); }
newCookie('theName', document.login.Username.value); // add a 
new cookie as shown at left for every

   document.login.Username.value = '';   // add a line for every field
document.login.Username.value = readCookie("theName");
document.login.Username.value = trim(document.login.Username.value);
if(document.login.Username.value.length==0){
 document.login.Username.focus();
var uname = "?"+document.login.Username.value+"+/@";
  valign="middle" nowrap="nowrap">style="color:#FF">Username
  name="Username" size="20" >





# curl /Login.jsp | grep Pass


  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   Spent Left  Speed
  0 00 00 0  0  0 --:--:--  0:00:03 
--:--:-- 0newCookie('theEmail', 
document.login.Password.value);   // field you wish to have the script 
remember

   document.login.Password.value = ''; }
document.login.Password.value = readCookie("theEmail");
if(document.login.Password.value.length==0){
 alert("Password is required.");
  document.login.Password.focus();
var pwd = "?"+document.login.Password.value+"+/@";
document.login.LoginPassword.value=encodeURIComponent(pwd);
valign="middle" nowrap="nowrap">style="color:#FF">Password
  name="Password" size="20" >
100 120040 120040 0   3405  0 --:--:--  0:00:03 
--:--:--  3405
valign="middle">


^


Hence I construct a curl command like

curl -sd "Username===Login" 
http:///Login.jsp


which is not working unfortunately. Have I missed anything ?

Your clue will be helpful.


Regards,

Bob