Re: operator new[] overflow (PR 19351)

2011-01-22 Thread Florian Weimer
* Gabriel Dos Reis: >>> Still, it's certainly an improvement on the current >>> situation and the cost is negligible compared to the call to the >>> allocator.  Since it's a security issue, some form of the patch should >>> go in. >> >> Well, should I resubmit, with the fix for the problem buildin

Re: operator new[] overflow (PR 19351)

2010-12-05 Thread Richard Guenther
On Sun, Dec 5, 2010 at 10:21 PM, Joseph S. Myers wrote: > On Sun, 5 Dec 2010, Richard Guenther wrote: > >> Ah, you're using intrinsics.  I thought of re-using the saturating arithmetic >> and types we have, thus basically do >> >>   size = (unsigned sat int) count * 4; >> >> and defer optimal expa

Re: operator new[] overflow (PR 19351)

2010-12-05 Thread Joseph S. Myers
On Sun, 5 Dec 2010, Richard Guenther wrote: > Ah, you're using intrinsics. I thought of re-using the saturating arithmetic > and types we have, thus basically do > > size = (unsigned sat int) count * 4; > > and defer optimal expansion to an optab. It of course requires saturating > arithmeti

Re: operator new[] overflow (PR 19351)

2010-12-05 Thread Richard Guenther
On Sun, Dec 5, 2010 at 6:49 PM, Chris Lattner wrote: > > On Dec 5, 2010, at 3:19 AM, Richard Guenther wrote: > >>> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm >>> -show-mc-encoding >>>        .section        __TEXT,__text,regular,pure_instructions >>>        .globl  __Z4testl >>

Re: operator new[] overflow (PR 19351)

2010-12-05 Thread Chris Lattner
On Dec 5, 2010, at 9:49 AM, Chris Lattner wrote: > > On Dec 5, 2010, at 3:19 AM, Richard Guenther wrote: > >>> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm >>> -show-mc-encoding >>> .section__TEXT,__text,regular,pure_instructions >>> .globl __Z4testl >>>

Re: operator new[] overflow (PR 19351)

2010-12-05 Thread Chris Lattner
On Dec 5, 2010, at 3:19 AM, Richard Guenther wrote: >> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm >> -show-mc-encoding >>.section__TEXT,__text,regular,pure_instructions >>.globl __Z4testl >>.align 4, 0x90 >> __Z4testl:

Re: operator new[] overflow (PR 19351)

2010-12-05 Thread Richard Guenther
On Sun, Dec 5, 2010 at 8:56 AM, Chris Lattner wrote: > > On Dec 4, 2010, at 5:22 AM, Florian Weimer wrote: > >> * Joe Buck: >> >>> It's wasted code if the multiply instruction detects the overflow. >>> It's true that the cost is small (maybe just one extra instruction >>> and the same number of te

Re: operator new[] overflow (PR 19351)

2010-12-04 Thread Chris Lattner
On Dec 4, 2010, at 5:22 AM, Florian Weimer wrote: > * Joe Buck: > >> It's wasted code if the multiply instruction detects the overflow. >> It's true that the cost is small (maybe just one extra instruction >> and the same number of tests, maybe one more on architectures where you >> have to load

Re: operator new[] overflow (PR 19351)

2010-12-04 Thread Gabriel Dos Reis
On Sat, Dec 4, 2010 at 7:22 AM, Florian Weimer wrote: > * Joe Buck: > >> It's wasted code if the multiply instruction detects the overflow. >> It's true that the cost is small (maybe just one extra instruction >> and the same number of tests, maybe one more on architectures where you >> have to lo

Re: operator new[] overflow (PR 19351)

2010-12-04 Thread Florian Weimer
* Joe Buck: > It's wasted code if the multiply instruction detects the overflow. > It's true that the cost is small (maybe just one extra instruction > and the same number of tests, maybe one more on architectures where you > have to load a large constant), but it is slightly worse code than what

Re: operator new[] overflow (PR 19351)

2010-12-03 Thread Joe Buck
On Thu, Dec 02, 2010 at 02:47:30PM -0800, Gabriel Dos Reis wrote: > On Thu, Dec 2, 2010 at 2:20 PM, Joe Buck wrote: > > On Wed, Dec 01, 2010 at 10:26:58PM -0800, Florian Weimer wrote: > >> * Chris Lattner: > >> > >> > On overflow it just forces the size passed in to operator new to > >> > -1ULL, w

Re: operator new[] overflow (PR 19351)

2010-12-02 Thread Gabriel Dos Reis
On Thu, Dec 2, 2010 at 2:20 PM, Joe Buck wrote: > On Wed, Dec 01, 2010 at 10:26:58PM -0800, Florian Weimer wrote: >> * Chris Lattner: >> >> > On overflow it just forces the size passed in to operator new to >> > -1ULL, which throws bad_alloc. >> >> This is also what my patch tries to implement. >

Re: operator new[] overflow (PR 19351)

2010-12-02 Thread Joe Buck
On Wed, Dec 01, 2010 at 10:26:58PM -0800, Florian Weimer wrote: > * Chris Lattner: > > > On overflow it just forces the size passed in to operator new to > > -1ULL, which throws bad_alloc. > > This is also what my patch tries to implement. Yes, but Chris's code just checks the overflow of the mu

Re: operator new[] overflow (PR 19351)

2010-12-01 Thread Florian Weimer
* Chris Lattner: > On overflow it just forces the size passed in to operator new to > -1ULL, which throws bad_alloc. This is also what my patch tries to implement.

Re: operator new[] overflow (PR 19351)

2010-12-01 Thread Gabriel Dos Reis
On Wed, Dec 1, 2010 at 5:36 PM, Chris Lattner wrote: > > On Nov 30, 2010, at 3:12 PM, Joe Buck wrote: > >> On Tue, Nov 30, 2010 at 01:49:23PM -0800, Gabriel Dos Reis wrote: >>> The existing GCC behaviour is a bit more perverse than the >>> C malloc() case as in >>> >>>       new T[n] >>> >>> there

Re: operator new[] overflow (PR 19351)

2010-12-01 Thread Chris Lattner
On Nov 30, 2010, at 3:12 PM, Joe Buck wrote: > On Tue, Nov 30, 2010 at 01:49:23PM -0800, Gabriel Dos Reis wrote: >> The existing GCC behaviour is a bit more perverse than the >> C malloc() case as in >> >> new T[n] >> >> there is no multiplication that could be credited to careless progra

Re: operator new[] overflow (PR 19351)

2010-11-30 Thread Joe Buck
On Tue, Nov 30, 2010 at 01:49:23PM -0800, Gabriel Dos Reis wrote: > The existing GCC behaviour is a bit more perverse than the > C malloc() case as in > >new T[n] > > there is no multiplication that could be credited to careless programmer. > The multiplication is introduced by GCC. ...

Re: operator new[] overflow (PR 19351)

2010-11-30 Thread Gabriel Dos Reis
On Tue, Nov 30, 2010 at 3:38 PM, Joseph S. Myers wrote: > On Tue, 30 Nov 2010, Florian Weimer wrote: > >> Mozilla seems to receive a report of an exploitable operator new[] >> overflow every couple of months now.  Obviously, this is not good. >> >> What is necessary so that GCC can fix this code g

Re: operator new[] overflow (PR 19351)

2010-11-30 Thread Joseph S. Myers
On Tue, 30 Nov 2010, Florian Weimer wrote: > Mozilla seems to receive a report of an exploitable operator new[] > overflow every couple of months now. Obviously, this is not good. > > What is necessary so that GCC can fix this code generation issue? > I've created a patch, together with a test c

operator new[] overflow (PR 19351)

2010-11-30 Thread Florian Weimer
Mozilla seems to receive a report of an exploitable operator new[] overflow every couple of months now. Obviously, this is not good. What is necessary so that GCC can fix this code generation issue? I've created a patch, together with a test case, but it has not been approved, nor have I been tol