Dear Scott,
thank you. The new improvements to the test scripts were committed to
rivet. I plan to propose a new rivet release soon
-- Massimo
On 30/04/23 13:08, Scott Pitcher wrote:
I've finished adding a raw_post test to the test suite. Patch is attached.
There is a small change to tests/apachetest/apachetest.tcl. I got it to
read IncludeOptional config files and to respect the ServerRoot, which
helped load config.
Also changed tests/rivet.test to get it to look in the environment for
TestList and the 3 testgroup? variables, so I could override them more
easily.
I added the new test on the end of post.test:
::tcltest::test postvariables-6.1 {::rivet::raw_post} {
set payload "\x00\x00\x00\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39"
set page [::http::geturl "${urlbase}post.rvt" \
-query $payload \
-type application/octet-stream]
regexp -line {^\[::rivet::raw_post\] =
00000030313233343536373839$} [ ::http::data $page ] match
set match
} {[::rivet::raw_post] = 00000030313233343536373839}
And post.rvt has a condition check to handle that and return the
raw_post result. The leading zeros in the test data will cause the test
to fail without the bug fix (will return an empty string).
Test passed fine here.
The post.rvt and post.test changes are the important ones and you can
discard the other changes to apachetest.tcl and rivet.test. I had to add
those to get the tests to run.
Kind regards,
Scott
On 27/4/23 09:53, Scott Pitcher wrote:
Thank you. That's good news. I haven't finished with the unit test
yet. Will come back to it this weekend.
Kind regards,
Scott
On 27/4/23 09:01, Massimo MANGHI wrote:
Your patch fixing the issue with the ::rivet::raw_post command was
commited in the master branch of mod_rivet.
-- Massimo
------------------------------------------------------------------------
*Da:* Scott Pitcher
*Inviato:* Giovedì, 20 Aprile, 2023 14:59
*A:* Massimo MANGHI; [email protected]
*Oggetto:* Re: R: R: Retrieving the body portion of a POST request
No problem at all. I'm working my way through it anyway.
I found the authz_core_module module error is caused by the
IncludeOptional directives in my /etc/apache2/apache2.conf, which are
missed by the apachetest::getallincludes proc. It regexp's against
Include but not IncludeOptional. I'll rework it to accept both.
Kind regards,
Scott
On 20/4/23 19:16, Massimo MANGHI wrote:
Hi Scott
sorry for reacting to your message so late. If you test rivet
against your Debian installed apache webserver you have to tweak
your configuration in order to have all the required modules loaded.
If you don't want to change it you have to build your own apache
webserver and have it installed in a custom directory with all the
configuration (make install does it for you when you build the
apache HTTP web server)
runtests.tcl run all the tests. If you need to work on your own
tests then you may disable temporarily the others by putting in the
TestList variable in tests/rivet.test only what you're actually
working on
-- Massimo
------------------------------------------------------------------------
*Da:* Scott Pitcher
*Inviato:* Giovedì, 20 Aprile, 2023 05:42
*A:* Massimo MANGHI; [email protected]
<mailto:[email protected]>
*Oggetto:* Re: R: Retrieving the body portion of a POST request
I've managed to get beyond this problem by upgrading Apache from
2.4.25 to 2.4.46 via Debian stretch-backports-sloppy.
Now I have an error -
checking /etc/apache2/apache2.conf
including files from ports.conf
check conf line for authz_core_module module...No LoadModule line
for authz_core_module!
while executing
"error "No LoadModule line for $mod\!""
(procedure "getloadmodules" line 12)
invoked from within ...
scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
But I'll work on this when I have some time free later this week.
Kind regards,
Scott
On 17/4/23 10:57, Scott Pitcher wrote:
I've added a test to tests/post.test and .../post.rvt. I'm having a
little trouble invoking the test scripts. I get an error in
gethttpdconf:
scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
./runtests.tcl /usr/sbin/apache2
runtests.tcl is running with auto_path: ./apachetest
/usr/share/tcltk/tcl8.6 /usr/share/tcltk /usr/lib
/usr/local/lib/tcltk /usr/local/share/tcltk
/usr/lib/tcltk/x86_64-linux-gnu /usr/lib/tcltk /usr/lib/tcltk/t
cl8.6
n arguments: 0
httpd_bin: /usr/sbin/apache2
httpd_args:
httpd_version: 2.4.25
mpm: prefork, bridge: default
Tests will be run against apache 2.4.25 version with the prefork
module and the default bridge
DEBUG apachetest::gethttpdconf: options= binname=/usr/sbin/apache2
bin=file3
can't read "filename": no such variable
while executing
"file exists $filename"
(procedure "gethttpdconf" line 8)
invoked from within
"gethttpdconf $binname"
(procedure "determinemodules" line 4)
invoked from within
"determinemodules $binname"
(procedure "apachetest::makeconf" line 16)
invoked from within
"apachetest::makeconf server.conf $bridge {
LoadModule rivet_module [file join $CWD .. src/.libs
mod_rivet[info sharedlibextension]]
# User an..."
(file "./runtests.tcl" line 124)
scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
This seems to be because invoking apache2 with -V option fails due
to missing environment configuration:
scotty@server1:~/src/rivet/tcl-rivet/tests$ /usr/sbin/apache2 -V
[Mon Apr 17 10:50:31.830756 2023] [core:warn] [pid 997] AH00111:
Config variable ${APACHE_RUN_DIR} is not defined
apache2: Syntax error on line 80 of /etc/apache2/apache2.conf:
DefaultRuntimeDir must be a valid directory, absolute or relative
to ServerRoot
scotty@server1:~/src/rivet/tcl-rivet/tests$
Looking at /etc/init.d/apache2 there is a little work to do with
running apache and setting up environment.
Perhaps it's better to run apache with "...-V -f server.conf"?
Your thoughts?
Kind regards,
Scott
On 14/4/23 18:05, Massimo MANGHI wrote:
Hi Scott
I get the point, if you don't have variable section in the POST
request you can't access the data except if you call raw_post.
Thank you for the patch, I will integrate it into Rivet. I guess
this patch is worth a bugfix release of Rivet.
Since you're working on this type of POST requests do you have at
hand some test cases we may add to our test suite? Thank you
-- Massimo
------------------------------------------------------------------------
*Da:* Scott Pitcher
*Inviato:* Martedì, 11 Aprile, 2023 12:06
*A:* Massimo MANGHI; [email protected]
<mailto:[email protected]>
*Oggetto:* Re: Retrieving the body portion of a POST request
Hi Massimo,
I tested "::rivet::var_post all" by inserting it into my client
session code:
# DEBUG: What does var_post return? if {[info exists
env(REQUEST_METHOD)] && [string match "POST"
$env(REQUEST_METHOD)]} { ::rivet::apache_log_error debug
"clientSession: POST, ::rivet::var_post=[::rivet::var_post all]"
}
... but when a Fossil POST request arrives, this function returns
a blank string. Probably because the POST request uses no
variables. It's main purpose appears to be to pass Fossil the 100
or so bytes in the body of the POST request.
When I ask Fossil to do a http clone and observe it with Wireshark
at the server I see the following:
Frame 261: 164 bytes on wire (1312 bits), 164 bytes captured
(1312 bits) on interface 0
Ethernet II, Src: d8:5e:d3:0e:20:aa (d8:5e:d3:0e:20:aa), Dst:
AsrockIn_e6:6b:35 (bc:5f:f4:e6:6b:35)
Internet Protocol Version 4, Src: 192.168.1.12, Dst: 192.168.1.1
Transmission Control Protocol, Src Port: 33820, Dst Port: 80,
Seq: 180, Ack: 1, Len: 98
[2 Reassembled TCP Segments (277 bytes): #259(179), #261(98)]
[Frame: 259, payload: 0-178 (179 bytes)]
[Frame: 261, payload: 179-276 (98 bytes)]
[Segment count: 2]
[Reassembled TCP length: 277]
[Reassembled TCP Data:
504f5354202f666f73732f67697432666f7373696c2e6367...]
Hypertext Transfer Protocol
POST /foss/git2fossil.cgi HTTP/1.0\r\n
Host:
[[[http://www.server2.svpts\r\n]http://www.server2.svpts\r\n][http://www.server2.svpts\r\n]http://www.server2.svpts\r\n]www.server2.svpts\r\n
User-Agent: Fossil/2.20 (2022-11-16 18:46:32 [210e89a059])\r\n
Content-Type: application/x-fossil\r\n
Content-Length: 98\r\n
\r\n
[Full request URI:
http://www.server2.svpts/foss/git2fossil.cgi
<http://www.server2.svpts/foss/git2fossil.cgi>]
[HTTP request 1/1]
[Response in frame: 273]
File Data: 98 bytes
Media Type
Media type: application/x-fossil (98 bytes)
0000 00 00 00 61 78 da 05 c1 41 0a 80 20 10 05 d0 bd ...ax...
A.. ....
0010 a7 18 68 1d cc fc 19 35 d7 69 f7 88 90 08 ca a2 ..h....5
.i......
0020 a2 f3 f7 de 75 cf eb 31 d3 b2 6f b5 bd fd 57 ef ....u..1
..o...W.
0030 67 3b 1b 01 cc 4c 60 40 44 02 c9 60 41 e1 96 fd g;...L`@
D..`A...
0040 6c 95 94 c4 75 c4 d1 4f 01 69 9c 3c 52 56 68 41 l...u..O
.i.<RVhA
0050 31 d3 1c 63 32 b3 11 52 82 64 5f 72 74 3f 31 1c 1..c2..R
.d_rt?1.
0060 18 6d .m
The 98 bytes in the body of the request after the last "\r\n" is
the part I'm trying to get hold of in my website code. With Rivet
patched (attached to this email) the ::rivet::raw_post returns the
98 bytes as a Tcl byte array, and I'm able to write that directly
through the pipe to Fossil's, and I had the first successful
"fossil clone http..." via Rivet and the website last night. I was
very happy!
For reference I've also attached my execcgi.tcl module. Much of it
is notes and documentation I've gleaned and used as a guide. I've
tested it with a few CGI programs not just Fossil. It's really a
work in progress but also seems to doing the job now.
Thanks for your help :)
Kind regards,
Scott
On 11/4/23 02:31, Massimo Manghi wrote:
Hi Scott
Thank you for the extensive explanation of your problem. I'm
certainly interested in reviewing your patch
Why using the usual command ::rivet::var_post (see
https://tcl.apache.org/rivet/manual3.2/var.html
<https://tcl.apache.org/rivet/manual3.2/var.html>) to retrieve
the posted data didn't work for you?
-- Massimo
On 09/04/23 01:55, Scott Pitcher wrote:
Hi,
I've been writing a system for our website using Rivet as the
foundation. I'd written a proc for running CGI programs from the
client request, or, passing the request over. This works well
but I'd recently started to incorporate some fossil archives in
the website. The fossil CGI method worked except when I tried to
clone the repo from a shell. Fossil would report an error code
and clone would fail.
I examined the website logs and it Fossil makes a POST request
to the CGI program. With wireshark I could see around 100 or so
binary bytes in the body of the POST request, below the headers:
0000 50 4f 53 54 20 2f 66 6f 73 73 2f 75 6e 69 6d 61 POST
/foss/unima 0010 6b 65 73 63 72 69 70 74 73 2e 63 67 69 2f 69
6e kescripts.cgi/in 0020 64 65 78 20 48 54 54 50 2f 31 2e
30 0d 0a 48 6f dex HTTP/1.0..Ho 0030 73 74 3a 20 77 77 77
2e 73 65 72 76 65 72 32 2e st: www.server2
<http://www.server2>. 0040 73 76 70 74 73 0d 0a 55 73 65 72
2d 41 67 65 6e svpts..User-Agen 0050 74 3a 20 46 6f 73 73
69 6c 2f 32 2e 32 30 20 28 t: Fossil/2.20 ( 0060 32 30 32
32 2d 31 31 2d 31 36 20 31 38 3a 34 36 2022-11-16 18:46
0070 3a 33 32 20 5b 32 31 30 65 38 39 61 30 35 39 5d :32
[210e89a059] 0080 29 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70
65 3a )..Content-Type: 0090 20 61 70 70 6c 69 63 61 74 69 6f
6e 2f 78 2d 66 application/x-f 00a0 6f 73 73 69 6c 0d 0a
43 6f 6e 74 65 6e 74 2d 4c ossil..Content-L 00b0 65 6e 67
74 68 3a 20 31 30 30 0d 0a 0d 0a 00 00 ength: 100......
00c0 00 61 78 da 05 c1 41 0a 80 20 10 00 c0 bb af 58
.axÚ.ÁA.. ..À»¯X 00d0 e8 1c ec 6e ba e9 d1 4c ff 21 21 11 94
86 45 ef è.ìnºéÑLÿ!!...Eï 00e0 6f e6 ee 79 bf 32 6c e7 51
ea 3b 7e a5 3f 47 ab oæîy¿2lçQê;~¥?G« 00f0 c0 8c 88 c0 c8
4c 44 02 64 b5 4c ac b6 b3 d5 02 À..ÀÈLD.dµL¬¶³Õ. 0100 13 90
1a 20 a1 5d 30 08 e9 e0 8c 8e 5e 4b 8c 22 ... ¡]0.éà..^K."
0110 c6 27 f4 f3 6c 53 10 71 d6 38 b7 92 fa 01 36 4f
Æ'ôólS.qÖ8·.ú.6O 0120 18 9a
I was able to retrieve the headers using ::rivet::apache_table
array_get headers_in. I wasn't able to retrieve the body of the
POST request. I tried ::rivet::raw_post but it returns an empty
string each time.
I checked out the Rivet source and delved into src/mod_rivet_ng
and the Rivet_RawPost() function, and traced down to
apache_request.h and ApacheRequest_get_raw_post().
The problem is that Rivet_RawPost() uses Tcl_NewStringObj() to
pass the body back to Tcl, and given that the first 3 bytes are
NUL, we end with an empty string. I changed it to
Tcl_NewByteArrayObj() and added a length parameter which was
passed by address all the way down to ApacheRequest_get_raw_post():
data = TclWeb_GetRawPost(private->req, &length); if
(!data) { data = ""; } retval =
Tcl_NewByteArrayObj((const unsigned char *)data, length);
Tcl_SetObjResult(interp, retval);
... and added a raw_length field to the ApacheRequest structure:
const char* temp_dir; char* raw_post; /* Raw post data. */ int
raw_length; request_rec* r; int nargs; } ApacheRequest;
The ApacheRequest_parse_urlencoded() function will fill this
field in just after filling in the raw_post pointer, and
util_read needed a length pointer parameter as well to get that
working:
int ApacheRequest_parse_urlencoded(ApacheRequest *req) { .....
if ((rc = util_read(req, &data, &length)) != OK) { return rc; }
if (data) { req->raw_post = (char*) data; /* Give people a way
of getting at the raw data. */ req->raw_length = length;
split_to_parms(req, data); } } return OK; }
I went to the effort because it's a simple change and I think it
completes raw_post , if that function can return the complete body.
That's just my opinion. It's important to me because I have a
website structure that can handle content, both HTTP and HTTPS
at the same time, as well as user base with login and
differentiation between local and remote access. It's been
developed so we can include private content for our business
only and host files for customers both public (e.g.
documentation) and private (firmware and software downloads).
I really wanted to get fossil working under it as it streamlines
that part of the development for us. I was going to give up and
host the fossil archives separately but I persisted and came up
with this. I still haven't had time to pass the POST request
into fossil but I'll do so when I have time this weekend, but I
do expect it to work, one way or another.
Anyway. That's my situation and if it's at all any use to
anyone, I'd be more than happy to give you a patch. It's a
really a minor set of changes but they give raw_post a bit more
utility.
Further: I'd really like Rivet to have a kind of "run_cgi"
method, but I've coded one in tcl at the moment and that's good
for testing. At some stage I'll write a Rivet command for doing
the same which might make environment setup and processing of
the CGI output, headers etc, a bit more efficient.
Kind regards, Scott Pitcher
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
<mailto:[email protected]>
For additional commands, e-mail: [email protected]
<mailto:[email protected]>
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water
...|
+--------------------------------------------------------------------------------+
| Scott [email protected]
<mailto:[email protected]> |
| +61-3-9008-5695 (B) +61-479-042-834
(M) |
| SVP Technical Services ABN 79 220 496
603 |
|PGP:https://gnupg.org/ <https://gnupg.org/>
|
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
<https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any
copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/
<https://gnupg.org/>.
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water
...|
+--------------------------------------------------------------------------------+
| Scott [email protected]
<mailto:[email protected]> |
| +61-3-9008-5695 (B) +61-479-042-834
(M) |
| SVP Technical Services ABN 79 220 496
603 |
|PGP:https://gnupg.org/ <https://gnupg.org/>
|
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
<https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any
copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/
<https://gnupg.org/>.
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water
...|
+--------------------------------------------------------------------------------+
| Scott [email protected]
<mailto:[email protected]> |
| +61-3-9008-5695 (B) +61-479-042-834
(M) |
| SVP Technical Services ABN 79 220 496
603 |
|PGP:https://gnupg.org/ <https://gnupg.org/>
|
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
<https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any
copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/
<https://gnupg.org/>.
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water
...|
+--------------------------------------------------------------------------------+
| Scott [email protected]
<mailto:[email protected]> |
| +61-3-9008-5695 (B) +61-479-042-834
(M) |
| SVP Technical Services ABN 79 220 496
603 |
|PGP:https://gnupg.org/ <https://gnupg.org/>
|
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
<https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any
copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/
<https://gnupg.org/>.
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water
...|
+--------------------------------------------------------------------------------+
| Scott [email protected] |
| +61-3-9008-5695 (B) +61-479-042-834
(M) |
| SVP Technical Services ABN 79 220 496
603 |
|PGP:https://gnupg.org/
|
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
|
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any
copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/.
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water
...|
+--------------------------------------------------------------------------------+
| Scott [email protected] |
| +61-3-9008-5695 (B) +61-479-042-834
(M) |
| SVP Technical Services ABN 79 220 496
603 |
|PGP:https://gnupg.org/
|
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
|
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any
copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
