Re: [squid-users] Will squid core dump with worker threads? Investigating squid crash, 3.5.23

[Jester Purtteman]

> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
> Behalf Of Amos Jeffries
> Sent: Friday, January 13, 2017 9:37 PM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Will squid core dump with worker threads?
> Investigating squid crash, 3.5.23
> 
> On 14/01/2017 10:32 a.m., Jester Purtteman wrote:
> > Hello,
> >
> >
> >
> > I am having period crashes of my squid server, and I am not getting
> > core dump files.  I have set "workers 6" in my squid.conf, and I know
> > that threads can cause trouble from reading the debugging wiki page.
> > I have confirmed permissions on the directory I'm dumping to, so I
> > don't *think* that is the issue.
> 
> Core dumps are done by your OS when programs crash. You may need to
> turn it on explicitly.
> 

Will continue prodding.  I think systemd is doing something funny, because the 
process isn't getting the same ulimits as I am running it with somehow.  I used 
to get dumps and I don't anymore, and systemd is one of the big changes between 
how I have this setup and how it was before.
> 
> >
> > FWIW (core dump to follow, I'll retry without workers and see what
> > happens) I am having squid crashes.  Details I have so far as are as 
> > follows:
> >
> >
> >
> > I am running Squid-3.5.23-R14129 on a stock Ubuntu 16.04 configured with:
> >
> >
> >
> > ./configure --prefix=/usr   --localstatedir=/var
> > --libexecdir=/usr/lib/squid--srcdir=.   --datadir=/usr/share/squid
> > --sysconfdir=/etc/squid   --with-default-user=proxy   --with-logdir=/var/log
> > --with-pidfile=/var/run/squid.pid --enable-linux-netfilter
> > --enable-cache-digests --enable-storeio=ufs,aufs,diskd,rock
> > --enable-async-io=30 --enable-http-violations --enable-zph-qos
> > --with-netfilter-conntrack --with-filedescriptors=65536
> > --with-large-files
> >
> >
> >
> > About once a day it is crashing with the following line as about my
> > only lead in the cache.log:
> >
> >
> >
> > assertion failed: MemBuf.cc:216: "0 <= tailSize && tailSize <= cSize"
> >
> 
> This is . We have
> narrowed it down to something about the collapsed revalidation behaviour
> that became visible after the recent security fix.

If I do not use collapsed forwarding, would it be safe to revert 3.5.22?  
Crashes are happening roughly daily and I don't really want to put a 
"babysitter" script in to keep it running if I have better options.  For right 
now, leaving collapsed forwarding off and not applying that patch seems like a 
better answer.

> 
> >
> > From the possibly interesting-but-who-knows-maybe-unrelated-files,
> > there is one additional detail.  I had this version running on a
> > Ubuntu 14.04 machine until last week, which I had installed GCC-4.9 on
> > (so I could test squid 4.0), and that had ran stable from December
> > 20th to January 5th without a any crashes.  Then something totally
> > outside of squid went horribly off the rails.  Ubuntu dropped support
> > for the 3.x series kernels, so I updated to
> > 4.4 (from the Ubuntu repositories) and that caused
> > /proc/sys/net/bridge to go away.  While testing an unrelated issue, I
> > ran a script that I adapted from
> > http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxBridge which
> contains a dangerous couple lines I had not before contemplated:
> >
> >
> > cd /proc/sys/net/bridge/
> > for i in *
> > do
> >echo 0 > $i
> > done
> > unset i
> >
> > When /proc/sys/net/bridge went away, the change directory failed, then
> > the script proceeded to turn everything in that directory into 0's.
> > OOPS!  I tell this bit so that my fellow admins get a laugh at my
> > expense, and as a cautionary tale.  CHECK the status of that command
> > before you let it do other things!  As it turns out, tproxy works fine
> > without echoing '0' at all those files, but if you want to leave it on
> > the page, may I suggest the following revision to the wiki page:
> >
> 
> Thank you. There is no need for the cd or the * to be without a fixed path. I
> have updated the wiki to prevent this.

Glad my foolish mistakes can make the world better :)

> 
> It it true that TPROXY does not require bridging, and bridging has nothing
> particularly requiring TPROXY. Except that for real transparency they are
> usually both wanted.

I am running a TPROXY system on a bridge, and my script had the snippet in it 
before.  But, it kept working even after the bridging code was migrated to a 
module, and it kept working even when the module was not loaded.  So, what 
effect setting everything in that directory to 0 has is beyond my 
understanding, and (at least appears) to not be necessary in the latest Ubuntu 
at least.  So, I guess that is part of my question, is something silently 
broken that you know of because I am not loading the kernel module for that?

> 
> >
> >
> > In any event, is ther

Re: [squid-users] Customize squid to make it understand malformed requests

[Amos Jeffries]

On 16/01/2017 10:10 p.m., Antony Stone wrote:
> On Monday 16 January 2017 at 09:03:52, Oğuz İsmail Uysal wrote:
> 
>> For a private reason, I want to customize squid version 3.5.12 the way I
>> stated above. For example I have customized it already to make it
>> understand \r\n /\r\n instead of \r\n\r\n as request's end
> 
>> now I want it to remove the characters after a spesific character in request
>> uri, and to remove a spesific character which is placed at the end of all
>> headers (before \r\n).
> 
> Wouldn't this be easier to achieve using content adaptation?
> 

Not if the malformation screws up the HTTP framing syntax, like the
above describes. See my other post about \r\n\r\n being the middle *not*
the end of a request.


To reach ICAP/eCAP Squid has to be able to parse the message and there
are a limited range of frame malformations which are tolerated before
the message is too mangled and simply gets rejected as non-HTTP.

Also, in passing to ICAP the message has to be delivered in correct HTTP
format to the service with Encapsulated header indicating the sizes of
each HTTP frame sub-section. If the malformation screws with the framing
those sizes will be incorrect and ICAP service gets screwed over as well
as Squid.


FWIW: By replacing the end-of-mime terminator with ' \' Ozguy is making
Squid smuggle request messages. The "private reasons" is obviously an
intention to turn a Squid binary into a piece of malware.
 

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Customize squid to make it understand malformed requests

[Amos Jeffries]

On 16/01/2017 9:03 p.m., Oğuz İsmail Uysal wrote:
> For a private reason, I want to customize squid version 3.5.12 the way I

Note: That is a long-ago obsoleted Squid version.

> stated above. For example I have customized it already to make it
> understand \r\n /\r\n instead of \r\n\r\n as request's end, (I know that
> this decreases the reliability of squid but I need.). By the way this

"reliability" has nothing to do with it. You are talking about
redesigning the fundamental definition of how a message is framed. In
direct violation of how the rest of the HTTP talking Internet communicates.
 

DO NOT do that.

> customization was kinda easy,

Of course, changing code is the easy part. Getting it right is much harder.

For example you seem to think that \r\n\r\n is the end of a request.

It is actually the MIME terminator and occurs at what the syntax defines
as the *middle* of an HTTP message frame - between the MIME and payload
sections.

The MIME section is a sub-frame within HTTP messages. It is optional
(either non-existing or empty) and may be used in any of the three major
HTTP message types.

Payload is a rare occurance on GET requests, which I suspect is where
your mistakes is coming from. But there is actually an implicit
Content-Length:0 header in GET requests making the payload section not
visible when looking at traffic.

Beyond that, mime sections are also used by several other protocols that
Squid implements. We re-use code for common protocol parsing so the code
you are changing probably affects _all_ of those protocols.


> now I want it to remove the characters after
> a spesific character in request uri, and to remove a spesific character
> which is placed at the end of all headers (before \r\n).

So what you actually have is your very own non-HTTP custom messaging
syntax. Please stop modifying Squid's HTTP parsing logics.

For a custom protocol (lets call it Foo for now) you should create your
own Foo namespace and one (or more) Foo::Parser classes that do the
message decapsulation in accordance with your protocols syntax.

You can inherit from or use an instance of the Http1::RequestParser if
you want to re-use some of its HTTP logics. But do not re-write the HTTP
logic to do non-HTTP things. Because, as I mentioned above there are
other protocols also re-using that same HTTP logic.


> Which source files
> I need to edit, do anyone know ? Or is there any link in which I can find
> references to source code which are related to request parsing ?
> 

The Squid developers can be contacted through the squid-dev mailing
list. This list is for network administrators to discuss and get help
*operating* Squid properly.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Customize squid to make it understand malformed requests

[Antony Stone]

On Monday 16 January 2017 at 09:03:52, Oğuz İsmail Uysal wrote:

> For a private reason, I want to customize squid version 3.5.12 the way I
> stated above. For example I have customized it already to make it
> understand \r\n /\r\n instead of \r\n\r\n as request's end

> now I want it to remove the characters after a spesific character in request
> uri, and to remove a spesific character which is placed at the end of all
> headers (before \r\n).

Wouldn't this be easier to achieve using content adaptation?

http://wiki.squid-cache.org/Features/eCAP
http://wiki.squid-cache.org/Features/ICAP

Antony.

-- 
"There is no reason for any individual to have a computer in their home."

 - Ken Olsen, President of Digital Equipment Corporation (DEC, later consumed 
by Compaq, later merged with HP)

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Customize squid to make it understand malformed requests

[Oğuz İsmail Uysal]

For a private reason, I want to customize squid version 3.5.12 the way I
stated above. For example I have customized it already to make it
understand \r\n /\r\n instead of \r\n\r\n as request's end, (I know that
this decreases the reliability of squid but I need.). By the way this
customization was kinda easy, now I want it to remove the characters after
a spesific character in request uri, and to remove a spesific character
which is placed at the end of all headers (before \r\n). Which source files
I need to edit, do anyone know ? Or is there any link in which I can find
references to source code which are related to request parsing ?
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users