Re: [squid-users] External C program
On Thu, Apr 30, 2009 at 4:40 AM, Amos Jeffries squ...@treenet.co.nz wrote:
Julien Philibin wrote:
On Wed, Apr 29, 2009 at 11:15 PM, Amos Jeffries squ...@treenet.co.nz
wrote:
Very interesting Bharath !!!
Yes thank you. You have identified the issue and we can now tell Julien
exactly what he has to do.
What would be your advice to get my program working ?!
Use fgets(). The scan() family apparently do not handle EOF in the way
needed.
Thus to work your code must be:
char line[8196];
char ip[45];
char url[8196];
ip[0] = '\0';
url[0] = '\0';
while( fgets(line, 8196, stdin) != NULL ) {
snscanf(sbuf, 8196, %s %s ip, url);
// happy joy
}
Amos
Hey that's smart! :)
I'm going to go for that and if things go wrong, I'll let you know ...
It is slightly wrong. The sbuf there should be 'line'.
I hope your compiler catches that also.
Yep I found it out :)
And please do use snscanf instead of scanf. It will save you from many
security and segfault bugs over your coding time.
You are talking about snscanf, but nor man snscanf nor google are
showing me revelant stuff about this function ... Am I missing
something ? I am using sscanf instead, for now ...
Thank you everyone!
btw: Amos, any idea why I get a randomly 127.0.0.1 instead of my real
Ip in the logs ?
As someone said earlier 127.0.0.1 is one of the IPs assigned to your
machine. It is a special IPv4 address assigned as localhost. Every machine
with networking has that same IP for private non-Internet traffic use.
Most machines will have two of these; 127.0.0.1 for IPv4 and ::1 for IPv6.
They are identical in use and purpose for their own IP protocols.
Why you get it randomly I don't know. I expect it to show up consistently
for requests the OS identifies as local-machine only. And never for requests
the OS thinks are remote global.
If your testing uses localhost:3128 as the proxy it will connect to
127.0.0.1 privately. If it uses the public IP or name resolving to the
public IP it will use a global public connections.
I am using the public IP address to access my proxy. I'll try to
remove the 127.0.0.1 from the hosts file and see how things go on ...
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
Current Beta Squid 3.1.0.7
Thank you everyone again for your time, and sorry for the delay in
getting back to you, I've been doing some researches and making my
hands dirty with my external helpers all week long!
Julien
Re: [squid-users] External C program
On Wed, Apr 29, 2009 at 1:22 AM, Amos Jeffries squ...@treenet.co.nz wrote:
Julien Philibin wrote:
Hi John,
thanks for your reply.
I'll give a shot with your skeleton and see how things are going on ...
On Tue, Apr 28, 2009 at 1:59 AM, John Doe jd...@yahoo.com wrote:
From: Julien Philibin jul...@philibin.fr
Hi, I've been trying to find a typical external ACL C program skeleton
for a while, but I wasn't able to find anything very interesting ...
What I would like to do, is to read to different strings and process
them in order to allow/disallow access to a website.
The thing is, after a while I get two processes that use around 10 Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and so
on, everytime I restart squid ...
Personaly, I use fgets/fflush and I did not see any problem (memory leak,
etc) so far...
Something like:
#define INPUTSIZE 4096
FYI: I've just had to start bumping my own custom helpers to using 8196 or
more for their buffers. Current Squid allow up to 8196 for URL length and
many more for possible headers length so watch that on inputs.
char input[INPUTSIZE];
while (fgets(input, sizeof(input), stdin)) {
if ((cp=strchr(input, '\n')) == NULL) {
fprintf(stderr, filter: input too big: %s\n, input);
} else {
*cp = '\0';
}
...
fflush(stderr);
fflush(stdout);
}
Do you use any malloc or functions that malloc... and that would need a
free?
Yes I do, but I also free them (the memory usage doesn't change). I
also made a mistake, it is not 10Mb but 1 ...
THe only weird thing is that after a restart (of squid), it looks like
squid doesn't have any control anymore on the externals programs and
they (both of external programs) start to use a lot of CPU...
Maybe it has something to do with stdin that was not flushed correctly
and creates an infinite loop or something ...
Probably. Squid simply closes its connection to the pipes and abandons the
old helper. Leaving the pipe close with a '\0' I believe.
From the docs of scanf() I don't get a clear idea of the return value when
empty string is received (is it 1/0/EOF?).
I'll try to figure it out as soon as my helper is working properly :-)
Also scanf() you were using earlier has no concept of length and opens the
possibility of buffer over-runs.
Prefer fgets or snscanf() as input methods.
Hi guys, so, I've been trying to implement the source code you gave to
me. I am running into an issue.
my first string is supposed to be a source (lenght = 16)
and the second one the URl of the website that the user is trying to access.
When I use the fgets method: fgets(source, sizeof(source), stdin) it
doesn't work. if the Ip address is less than 15, the program simply
takes the beginning of the destination URL and everything goes wrong
So I was wondering what would you guys use ?
sscanf(stdin, %s, s);
or
scanf(%s, source); //as I was doing before, and double check the
buffer's size
or
Something else?
I have to admit, all this is confusing me a little bit :-)
There must be an easy/secure way to catch two strings from stdin ...
Thanks for your time guys.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
Current Beta Squid 3.1.0.7
Julien
Re: [squid-users] External C program
And also, when I take a look at the source, I don't understand why
sometimes I have 127.0.0.1 instead of my real IP showing up ... ?!
Any clue ? I wasn't able to find anything about that on internet ...
Thanks everybody
On Wed, Apr 29, 2009 at 9:02 PM, Julien Philibin jul...@philibin.fr wrote:
On Wed, Apr 29, 2009 at 1:22 AM, Amos Jeffries squ...@treenet.co.nz wrote:
Julien Philibin wrote:
Hi John,
thanks for your reply.
I'll give a shot with your skeleton and see how things are going on ...
On Tue, Apr 28, 2009 at 1:59 AM, John Doe jd...@yahoo.com wrote:
From: Julien Philibin jul...@philibin.fr
Hi, I've been trying to find a typical external ACL C program skeleton
for a while, but I wasn't able to find anything very interesting ...
What I would like to do, is to read to different strings and process
them in order to allow/disallow access to a website.
The thing is, after a while I get two processes that use around 10 Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and so
on, everytime I restart squid ...
Personaly, I use fgets/fflush and I did not see any problem (memory leak,
etc) so far...
Something like:
#define INPUTSIZE 4096
FYI: I've just had to start bumping my own custom helpers to using 8196 or
more for their buffers. Current Squid allow up to 8196 for URL length and
many more for possible headers length so watch that on inputs.
char input[INPUTSIZE];
while (fgets(input, sizeof(input), stdin)) {
if ((cp=strchr(input, '\n')) == NULL) {
fprintf(stderr, filter: input too big: %s\n, input);
} else {
*cp = '\0';
}
...
fflush(stderr);
fflush(stdout);
}
Do you use any malloc or functions that malloc... and that would need a
free?
Yes I do, but I also free them (the memory usage doesn't change). I
also made a mistake, it is not 10Mb but 1 ...
THe only weird thing is that after a restart (of squid), it looks like
squid doesn't have any control anymore on the externals programs and
they (both of external programs) start to use a lot of CPU...
Maybe it has something to do with stdin that was not flushed correctly
and creates an infinite loop or something ...
Probably. Squid simply closes its connection to the pipes and abandons the
old helper. Leaving the pipe close with a '\0' I believe.
From the docs of scanf() I don't get a clear idea of the return value when
empty string is received (is it 1/0/EOF?).
I'll try to figure it out as soon as my helper is working properly :-)
Also scanf() you were using earlier has no concept of length and opens the
possibility of buffer over-runs.
Prefer fgets or snscanf() as input methods.
Hi guys, so, I've been trying to implement the source code you gave to
me. I am running into an issue.
my first string is supposed to be a source (lenght = 16)
and the second one the URl of the website that the user is trying to access.
When I use the fgets method: fgets(source, sizeof(source), stdin) it
doesn't work. if the Ip address is less than 15, the program simply
takes the beginning of the destination URL and everything goes wrong
So I was wondering what would you guys use ?
sscanf(stdin, %s, s);
or
scanf(%s, source); //as I was doing before, and double check the
buffer's size
or
Something else?
I have to admit, all this is confusing me a little bit :-)
There must be an easy/secure way to catch two strings from stdin ...
Thanks for your time guys.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
Current Beta Squid 3.1.0.7
Julien
Re: [squid-users] External C program
Very interesting Bharath !!!
What would be your advice to get my program working ?!
Thanks
On Wed, Apr 29, 2009 at 9:19 PM, Bharath Raghavendran
rbharat...@gmail.com wrote:
Hi,
I was playing around with scanf just now. scanf seems to be able to
input 2 strings :
scanf(%s %s, str1, str2);
Moreover, I also noticed that while( scanf( ..blah.. ) ) doesn't
work as intended. i.e., its unable to detect an EOF. Hence, when squid
quits/restarts and it sends an EOF to the program, the program does
not quit. This could be the cause for getting multiple children when
squid restarts.
-Bharath
2009/4/30 Julien Philibin jul...@philibin.fr:
On Wed, Apr 29, 2009 at 1:22 AM, Amos Jeffries squ...@treenet.co.nz wrote:
Julien Philibin wrote:
Hi John,
thanks for your reply.
I'll give a shot with your skeleton and see how things are going on ...
On Tue, Apr 28, 2009 at 1:59 AM, John Doe jd...@yahoo.com wrote:
From: Julien Philibin jul...@philibin.fr
Hi, I've been trying to find a typical external ACL C program skeleton
for a while, but I wasn't able to find anything very interesting ...
What I would like to do, is to read to different strings and process
them in order to allow/disallow access to a website.
The thing is, after a while I get two processes that use around 10 Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and so
on, everytime I restart squid ...
Personaly, I use fgets/fflush and I did not see any problem (memory leak,
etc) so far...
Something like:
#define INPUTSIZE 4096
FYI: I've just had to start bumping my own custom helpers to using 8196 or
more for their buffers. Current Squid allow up to 8196 for URL length and
many more for possible headers length so watch that on inputs.
char input[INPUTSIZE];
while (fgets(input, sizeof(input), stdin)) {
if ((cp=strchr(input, '\n')) == NULL) {
fprintf(stderr, filter: input too big: %s\n, input);
} else {
*cp = '\0';
}
...
fflush(stderr);
fflush(stdout);
}
Do you use any malloc or functions that malloc... and that would need a
free?
Yes I do, but I also free them (the memory usage doesn't change). I
also made a mistake, it is not 10Mb but 1 ...
THe only weird thing is that after a restart (of squid), it looks like
squid doesn't have any control anymore on the externals programs and
they (both of external programs) start to use a lot of CPU...
Maybe it has something to do with stdin that was not flushed correctly
and creates an infinite loop or something ...
Probably. Squid simply closes its connection to the pipes and abandons the
old helper. Leaving the pipe close with a '\0' I believe.
From the docs of scanf() I don't get a clear idea of the return value when
empty string is received (is it 1/0/EOF?).
I'll try to figure it out as soon as my helper is working properly :-)
Also scanf() you were using earlier has no concept of length and opens the
possibility of buffer over-runs.
Prefer fgets or snscanf() as input methods.
Hi guys, so, I've been trying to implement the source code you gave to
me. I am running into an issue.
my first string is supposed to be a source (lenght = 16)
and the second one the URl of the website that the user is trying to access.
When I use the fgets method: fgets(source, sizeof(source), stdin) it
doesn't work. if the Ip address is less than 15, the program simply
takes the beginning of the destination URL and everything goes wrong
So I was wondering what would you guys use ?
sscanf(stdin, %s, s);
or
scanf(%s, source); //as I was doing before, and double check the
buffer's size
or
Something else?
I have to admit, all this is confusing me a little bit :-)
There must be an easy/secure way to catch two strings from stdin ...
Thanks for your time guys.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
Current Beta Squid 3.1.0.7
Julien
Re: [squid-users] External C program
On Wed, Apr 29, 2009 at 9:30 PM, Bharath Raghavendran
rbharat...@gmail.com wrote:
2009/4/30 Julien Philibin jul...@philibin.fr:
Very interesting Bharath !!!
What would be your advice to get my program working ?!
Thanks
Unfortunately, I have no advice. I have made external ACLs using C++
and know how to handle this using the cin stream. With my knowledge
of C, the only thing I can think of is using file handling for stdin.
I am sure file handling should have ways to detect EOF (I need to
search net if you need more details :P )
Could you please show me your skeleton of your C++ Program ? I
developed mine in C, because I use to develop a lot in C many years
ago. But I don't mind developing it in C++ as it is not a very
complicated program (Basically just using the SQL Lib) ...
127.0.0.1 or localhost is a connection to loopback interface which
is used to make connections to your own computer. If you have entered
the proxy address (assuming you are using squid as a proxy server) as
any of these two,
Yes I am using it as proxy server. And it is directly set up in my
browser's config.
your computer connects to squid using loopback
interface and hence your ip is shown up as 127.0.0.1.
You can avoid this by putting your actual IP in the proxy address.
What do you mean ? I checked in my documentation books and I wasn't
able to find anything about the proxy's address.
And the thing is, it doesn't show up every time with 127.0.0.1, it is
(apparently) shown as 127.0.0.1 randomly
-Bharath
2009/4/30 Julien Philibin jul...@philibin.fr:
And also, when I take a look at the source, I don't understand why
sometimes I have 127.0.0.1 instead of my real IP showing up ... ?!
Any clue ? I wasn't able to find anything about that on internet ...
Thanks everybody
On Wed, Apr 29, 2009 at 9:02 PM, Julien Philibin jul...@philibin.fr wrote:
On Wed, Apr 29, 2009 at 1:22 AM, Amos Jeffries squ...@treenet.co.nz wrote:
Julien Philibin wrote:
Hi John,
thanks for your reply.
I'll give a shot with your skeleton and see how things are going on ...
On Tue, Apr 28, 2009 at 1:59 AM, John Doe jd...@yahoo.com wrote:
From: Julien Philibin jul...@philibin.fr
Hi, I've been trying to find a typical external ACL C program skeleton
for a while, but I wasn't able to find anything very interesting ...
What I would like to do, is to read to different strings and process
them in order to allow/disallow access to a website.
The thing is, after a while I get two processes that use around 10 Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and so
on, everytime I restart squid ...
Personaly, I use fgets/fflush and I did not see any problem (memory leak,
etc) so far...
Something like:
#define INPUTSIZE 4096
FYI: I've just had to start bumping my own custom helpers to using 8196 or
more for their buffers. Current Squid allow up to 8196 for URL length and
many more for possible headers length so watch that on inputs.
char input[INPUTSIZE];
while (fgets(input, sizeof(input), stdin)) {
if ((cp=strchr(input, '\n')) == NULL) {
fprintf(stderr, filter: input too big: %s\n, input);
} else {
*cp = '\0';
}
...
fflush(stderr);
fflush(stdout);
}
Do you use any malloc or functions that malloc... and that would need a
free?
Yes I do, but I also free them (the memory usage doesn't change). I
also made a mistake, it is not 10Mb but 1 ...
THe only weird thing is that after a restart (of squid), it looks like
squid doesn't have any control anymore on the externals programs and
they (both of external programs) start to use a lot of CPU...
Maybe it has something to do with stdin that was not flushed correctly
and creates an infinite loop or something ...
Probably. Squid simply closes its connection to the pipes and abandons the
old helper. Leaving the pipe close with a '\0' I believe.
From the docs of scanf() I don't get a clear idea of the return value when
empty string is received (is it 1/0/EOF?).
I'll try to figure it out as soon as my helper is working properly :-)
Also scanf() you were using earlier has no concept of length and opens the
possibility of buffer over-runs.
Prefer fgets or snscanf() as input methods.
Hi guys, so, I've been trying to implement the source code you gave to
me. I am running into an issue.
my first string is supposed to be a source (lenght = 16)
and the second one the URl of the website that the user is trying to access.
When I use the fgets method: fgets(source, sizeof(source), stdin) it
doesn't work. if the Ip address is less than 15, the program simply
takes the beginning of the destination URL and everything goes wrong
So I was wondering what would you guys use ?
sscanf(stdin, %s, s);
or
scanf(%s, source); //as I was doing before, and double check the
buffer's size
or
Something else?
I have to admit, all this is confusing me a little
Re: [squid-users] External C program
On Wed, Apr 29, 2009 at 11:15 PM, Amos Jeffries squ...@treenet.co.nz wrote:
Very interesting Bharath !!!
Yes thank you. You have identified the issue and we can now tell Julien
exactly what he has to do.
What would be your advice to get my program working ?!
Use fgets(). The scan() family apparently do not handle EOF in the way
needed.
Thus to work your code must be:
char line[8196];
char ip[45];
char url[8196];
ip[0] = '\0';
url[0] = '\0';
while( fgets(line, 8196, stdin) != NULL ) {
snscanf(sbuf, 8196, %s %s ip, url);
// happy joy
}
Amos
Hey that's smart! :)
I'm going to go for that and if things go wrong, I'll let you know ...
Thank you everyone!
btw: Amos, any idea why I get a randomly 127.0.0.1 instead of my real
Ip in the logs ?
Thanks
On Wed, Apr 29, 2009 at 9:19 PM, Bharath Raghavendran
rbharat...@gmail.com wrote:
Hi,
I was playing around with scanf just now. scanf seems to be able to
input 2 strings :
scanf(%s %s, str1, str2);
Moreover, I also noticed that while( scanf( ..blah.. ) ) doesn't
work as intended. i.e., its unable to detect an EOF. Hence, when squid
quits/restarts and it sends an EOF to the program, the program does
not quit. This could be the cause for getting multiple children when
squid restarts.
-Bharath
2009/4/30 Julien Philibin jul...@philibin.fr:
On Wed, Apr 29, 2009 at 1:22 AM, Amos Jeffries squ...@treenet.co.nz
wrote:
Julien Philibin wrote:
Hi John,
thanks for your reply.
I'll give a shot with your skeleton and see how things are going on
...
On Tue, Apr 28, 2009 at 1:59 AM, John Doe jd...@yahoo.com wrote:
From: Julien Philibin jul...@philibin.fr
Hi, I've been trying to find a typical external ACL C program
skeleton
for a while, but I wasn't able to find anything very interesting
...
What I would like to do, is to read to different strings and
process
them in order to allow/disallow access to a website.
The thing is, after a while I get two processes that use around 10
Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and
so
on, everytime I restart squid ...
Personaly, I use fgets/fflush and I did not see any problem (memory
leak,
etc) so far...
Something like:
#define INPUTSIZE 4096
FYI: I've just had to start bumping my own custom helpers to using
8196 or
more for their buffers. Current Squid allow up to 8196 for URL length
and
many more for possible headers length so watch that on inputs.
char input[INPUTSIZE];
while (fgets(input, sizeof(input), stdin)) {
if ((cp=strchr(input, '\n')) == NULL) {
fprintf(stderr, filter: input too big: %s\n, input);
} else {
*cp = '\0';
}
...
fflush(stderr);
fflush(stdout);
}
Do you use any malloc or functions that malloc... and that would
need a
free?
Yes I do, but I also free them (the memory usage doesn't change). I
also made a mistake, it is not 10Mb but 1 ...
THe only weird thing is that after a restart (of squid), it looks
like
squid doesn't have any control anymore on the externals programs and
they (both of external programs) start to use a lot of CPU...
Maybe it has something to do with stdin that was not flushed
correctly
and creates an infinite loop or something ...
Probably. Squid simply closes its connection to the pipes and abandons
the
old helper. Leaving the pipe close with a '\0' I believe.
From the docs of scanf() I don't get a clear idea of the return value
when
empty string is received (is it 1/0/EOF?).
I'll try to figure it out as soon as my helper is working properly :-)
Also scanf() you were using earlier has no concept of length and opens
the
possibility of buffer over-runs.
Prefer fgets or snscanf() as input methods.
Hi guys, so, I've been trying to implement the source code you gave to
me. I am running into an issue.
my first string is supposed to be a source (lenght = 16)
and the second one the URl of the website that the user is trying to
access.
When I use the fgets method: fgets(source, sizeof(source), stdin) it
doesn't work. if the Ip address is less than 15, the program simply
takes the beginning of the destination URL and everything goes wrong
So I was wondering what would you guys use ?
sscanf(stdin, %s, s);
or
scanf(%s, source); //as I was doing before, and double check the
buffer's size
or
Something else?
I have to admit, all this is confusing me a little bit :-)
There must be an easy/secure way to catch two strings from stdin ...
Thanks for your time guys.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
Current Beta Squid 3.1.0.7
Julien
Re: [squid-users] External C program
Hi John,
thanks for your reply.
I'll give a shot with your skeleton and see how things are going on ...
On Tue, Apr 28, 2009 at 1:59 AM, John Doe jd...@yahoo.com wrote:
From: Julien Philibin jul...@philibin.fr
Hi, I've been trying to find a typical external ACL C program skeleton
for a while, but I wasn't able to find anything very interesting ...
What I would like to do, is to read to different strings and process
them in order to allow/disallow access to a website.
The thing is, after a while I get two processes that use around 10 Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and so
on, everytime I restart squid ...
Personaly, I use fgets/fflush and I did not see any problem (memory leak,
etc) so far...
Something like:
#define INPUTSIZE 4096
char input[INPUTSIZE];
while (fgets(input, sizeof(input), stdin)) {
if ((cp=strchr(input, '\n')) == NULL) {
fprintf(stderr, filter: input too big: %s\n, input);
} else {
*cp = '\0';
}
...
fflush(stderr);
fflush(stdout);
}
Do you use any malloc or functions that malloc... and that would need a free?
Yes I do, but I also free them (the memory usage doesn't change). I
also made a mistake, it is not 10Mb but 1 ...
THe only weird thing is that after a restart (of squid), it looks like
squid doesn't have any control anymore on the externals programs and
they (both of external programs) start to use a lot of CPU...
Maybe it has something to do with stdin that was not flushed correctly
and creates an infinite loop or something ...
Anyways, I'll try your way, and I'll let you know!
thanks again for your reply
JD
[squid-users] External C program
Hi, I've been trying to find a typical external ACL C program skeleton
for a while, but I wasn't able to find anything very interesting ...
What I would like to do, is to read to different strings and process
them in order to allow/disallow access to a website.
This is what I have:
external.c
-
int main(int argc, char* argv[])
{
char source[16];
char dom[2000];
while (scanf(%s, source)){
setbuf( stdout, NULL);
scanf(%s, dom);
/*
Do some work
*/
//Flush stdin
setbuf (stdin, NULL);
if (condition)
printf(ERR\n);
else
printf(OK\n);
}
}
-
The thing is, after a while I get two processes that use around 10 Mb
of memory and 15% of my CPU
Also, if I restart squid, I'll get two more processes running and so
on, everytime I restart squid ...
Furthermore, as I am working on an SQL server, I get around 3k SQL
requests each second ... and the mysqld process is jumping on 70% of
CPU usage ... (And yes, I am the only one that is submitting requests
to the squid server, and no, I don't have any bittorent, emule, or any
other program that would generate such a heavy internet traffic)
Does anyone know where that could come from ?
Thank you all for your help!
Julien
