Re: Re[2]: [sqlite] Unaligned access in SQLite on Itanium
> > Unless the on disk format is carried over into memory, there's no > reason not to use a structure alignment which prevents unaligned > access. > > I'm a little surprised your compiler hadn't already padded the struct > out to proper alignment when you built SQLite. > Sorry, I'm a little bit lost here. Maybe I mis-explained something - maybe due to my non-native English. Sorry for that. Let's re-iterate through my problem once again. My compiler does have right padding. By default, almost all compilers will align structure members on natural alignment boundary, which means align a member on the offset which is a multiple of the member data size. In my case it is array of chars that has got mis-aligned. To be precise, it is aligned just right to store 1, 2, 4, 8 size types there, but it is not aligned properly for 16-byte long double. For this case, #pragma pack, which is more or less portable (I really like this 'more or less portable' - crazy world, you've gone too far...), won't work because it allows only to pack - that is, to lower alignment boundaries - it is used when you need to pack structures for network transfer, for example. And in fact, #pragma pack can only make hardware alignment issues only worse. So the only (known so far) portable way to fix the problem is to union the problematic array with long double variable to get it aligned properly - on 16-bytes boundary. On GCC we could also use __attribute__ ((aligned (16))) but it is not portable. Please don't hesitate to ask for clarifications if something is not clear here - I really want to follow it up ASAP. -- Alexei Alexandrov
Re[2]: [sqlite] Unaligned access in SQLite on Itanium
Hello Jay, Monday, March 27, 2006, 12:25:36 PM, you wrote: JS> On 3/27/06, Alexei Alexandrov <[EMAIL PROTECTED]> wrote: >> > >> > Isn't this eliminated with the proper compile settings for data packing? >> > There should be a compile option for sometihng like 'align data on N >> > byte boundaries' >> > >> >> Even if that is possible, I don't want to say to compiler "Please >> align everything on 16 bytes" because this will lead to huge memory >> footprint overhead - every element - even 1 byte-long will take 16 >> bytes in memory. This is not an option. JS> The compiler I used was smarter than that in the 1980's... I agree. Typically it's just structure alignments so, for instance if you had a char followed by a short in the struct, it would get padded out to 16 bytes but, it wouldn't pad unless it had too. 15 chars for instance would have a 1 byte pad. Structs HAVE to be aligned in order to prevent unaligned access unless you're working with something byte centric like a net protocol which requires you to work with no or smaller alignments. The native alignment for my windows apps is 8 bytes. When I tried Visual Studio 2005, the default alignment was 16 bytes (which caused me some real problems till I figured it out). Unless the on disk format is carried over into memory, there's no reason not to use a structure alignment which prevents unaligned access. I'm a little surprised your compiler hadn't already padded the struct out to proper alignment when you built SQLite. -- Best regards, Tegmailto:[EMAIL PROTECTED]
Re: [sqlite] Unaligned access in SQLite on Itanium
On 3/27/06, Alexei Alexandrov <[EMAIL PROTECTED]> wrote: > > > > Isn't this eliminated with the proper compile settings for data packing? > > There should be a compile option for sometihng like 'align data on N > > byte boundaries' > > > > Even if that is possible, I don't want to say to compiler "Please > align everything on 16 bytes" because this will lead to huge memory > footprint overhead - every element - even 1 byte-long will take 16 > bytes in memory. This is not an option. The compiler I used was smarter than that in the 1980's...
Re: [sqlite] Unaligned access in SQLite on Itanium
> > Isn't this eliminated with the proper compile settings for data packing? > There should be a compile option for sometihng like 'align data on N > byte boundaries' > Even if that is possible, I don't want to say to compiler "Please align everything on 16 bytes" because this will lead to huge memory footprint overhead - every element - even 1 byte-long will take 16 bytes in memory. This is not an option. -- Alexei Alexandrov
Re: [sqlite] Unaligned access in SQLite on Itanium
On 3/25/06, Alexei Alexandrov <[EMAIL PROTECTED]> wrote: > Hi, > > I don't know whether it's been already reported or not, so anyway. > There are places in SQLite where unaligned access exception is > generated on Itanium. The unaligned access means that someone tries to > read or write memory crossing 8-bytes boundary. Usually this occur as > a result of pointers casting. Isn't this eliminated with the proper compile settings for data packing? There should be a compile option for sometihng like 'align data on N byte boundaries'
Re: [sqlite] Unaligned access in SQLite on Itanium
"Alexei Alexandrov" <[EMAIL PROTECTED]> wrote: > > > > Well, it's supported by most compilers today, but I try to avoid > > anonymous unions in C code as well. They are fairly standard for C++, > > but not for C. But the approach seems fine to me - the alignment will > > be forced in this case. > > > > Just to make it clear: can I expect this fix in the next SQLite > release? Should I submit a ticket for this? I will provide Linux > ia64/x86_64 testing from my side. > Submit a ticket or it will likely fall through the cracks. -- D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Unaligned access in SQLite on Itanium
> > Well, it's supported by most compilers today, but I try to avoid > anonymous unions in C code as well. They are fairly standard for C++, > but not for C. But the approach seems fine to me - the alignment will > be forced in this case. > Just to make it clear: can I expect this fix in the next SQLite release? Should I submit a ticket for this? I will provide Linux ia64/x86_64 testing from my side. -- Alexei Alexandrov
Re: [sqlite] Unaligned access in SQLite on Itanium
> Perhaps a better fix is this:
>
> struct Mem {
>i64 i;
>double r;
>char *z;
>int n;
>u16 flags;
>u8 type;
>u8 enc;
>void (*xDel)(void *);
>union {
> long double notUsed1;
> char zShort[NBFS];
>};
> };
>
> The compiler would then (hopefully) insert an appropriate
> amount of padding prior to zShort so that it had the same
> alignment restructions as a long double.
>
> This approach uses an anonymous union, which I confess is
> a C construct that I have never in 22 years of C programming
> had the occasion to employ. It seems to work well enough
> using gcc 3.3.4. But down in my gut I have this nagging
> fealing that it will likely break on some compilers.
>
Well, it's supported by most compilers today, but I try to avoid
anonymous unions in C code as well. They are fairly standard for C++,
but not for C. But the approach seems fine to me - the alignment will
be forced in this case.
Re: [sqlite] Unaligned access in SQLite on Itanium
"Alexei Alexandrov" <[EMAIL PROTECTED]> wrote:
> >
> > Perhaps the following definition of Mem would work better:
> >
> > struct Mem {
> >u16 flags;
> >u8 type;
> >u8 enc;
> >char *z;
> >int n;
> >i64 i;
> >double r;
> >char zShort[NBFS];
> >void (*xDel)(void *);
> > };
> >
>
> Not exactly, since 'int' is still 4 bytes on Linux ia64. long/size_t
> is 8 bytes. In fact, I think that the rightest thing here would be to
> put zShort at the beginning of the structure, because in this case it
> would get the same alignment as returned from malloc and the rest of
> fields are strictly typed so that they are aligned by compiler
> properly.
>
> But when I try to put zShort at the beginning, some strange thing
> happens - SQLite doesn't like it. I start SQLite shell, and it says
> immediately (or when I create a simple table):
>
Come to think of it, I there are some places in the code that
assume that zShort[] is at the end of the structure. There
are places that memcpy() the first part of the structure and
ignore zShort[]. So moving zShort[] to any other place in the
structure will not work unless those places are changed. And
those places are there for efficiency reasons so I am reluctant
to change them.
Perhaps a better fix is this:
struct Mem {
i64 i;
double r;
char *z;
int n;
u16 flags;
u8 type;
u8 enc;
void (*xDel)(void *);
union {
long double notUsed1;
char zShort[NBFS];
};
};
The compiler would then (hopefully) insert an appropriate
amount of padding prior to zShort so that it had the same
alignment restructions as a long double.
This approach uses an anonymous union, which I confess is
a C construct that I have never in 22 years of C programming
had the occasion to employ. It seems to work well enough
using gcc 3.3.4. But down in my gut I have this nagging
fealing that it will likely break on some compilers.
--
D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Unaligned access in SQLite on Itanium
>
> The Mem structure does not appear on disk or in any API (except
> as the opaque structure pointer sqlite3_value*) so it can be
> revised as needed to force 16-byte alignment. Perhaps
> the following definition of Mem would work better:
>
> struct Mem {
>u16 flags;
>u8 type;
>u8 enc;
>char *z;
>int n;
>i64 i;
>double r;
>char zShort[NBFS];
>void (*xDel)(void *);
> };
>
> Assuming the entire structure is 16-byte aligned and pointers
> and integers are all 8-bytes and 8-byte aligned, then there
> would be 4 bytes of padding between Mem.enc and Mem.z. This
> would result in zShort appearing on a 16-byte boundary, would
> it not?
>
Not exactly, since 'int' is still 4 bytes on Linux ia64. long/size_t
is 8 bytes. In fact, I think that the rightest thing here would be to
put zShort at the beginning of the structure, because in this case it
would get the same alignment as returned from malloc and the rest of
fields are strictly typed so that they are aligned by compiler
properly.
But when I try to put zShort at the beginning, some strange thing
happens - SQLite doesn't like it. I start SQLite shell, and it says
immediately (or when I create a simple table):
D:\src\3rd-parties\sqlitecomp\bin\windows-x32>sqlite3-test.exe test.db
SQL error: malformed database schema
Can't prepare statement: no such table: test
Reproduced on Windows x86 and Linux ia64.
--
Alexei Alexandrov
Re: [sqlite] Unaligned access in SQLite on Itanium
"Alexei Alexandrov" <[EMAIL PROTECTED]> wrote:
> As far as I understand, this fix will take the problem away only
> because SumCtx is bigger now. But the problem with
> sqlite3_aggregate_context will remain: the pointer returned will not
> be aligned properly if the size of the context is less or equal 32
> bytes. Am I correct?
>
The context should be 8-byte aligned regardless. And after
check-in [3084] there are no long doubles stored in the context
so 8-byte alignment is sufficient - provided of course that you
do not implement your own private aggregate functions that
require a 16-byte aligned context.
The Mem structure does not appear on disk or in any API (except
as the opaque structure pointer sqlite3_value*) so it can be
revised as needed to force 16-byte alignment. Perhaps
the following definition of Mem would work better:
struct Mem {
u16 flags;
u8 type;
u8 enc;
char *z;
int n;
i64 i;
double r;
char zShort[NBFS];
void (*xDel)(void *);
};
Assuming the entire structure is 16-byte aligned and pointers
and integers are all 8-bytes and 8-byte aligned, then there
would be 4 bytes of padding between Mem.enc and Mem.z. This
would result in zShort appearing on a 16-byte boundary, would
it not?
--
D. Richard Hipp <[EMAIL PROTECTED]>
Re: [sqlite] Unaligned access in SQLite on Itanium
As far as I understand, this fix will take the problem away only because SumCtx is bigger now. But the problem with sqlite3_aggregate_context will remain: the pointer returned will not be aligned properly if the size of the context is less or equal 32 bytes. Am I correct? On 3/25/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > "Alexei Alexandrov" <[EMAIL PROTECTED]> wrote: > > Hi, > > > > I don't know whether it's been already reported or not, so anyway. > > There are places in SQLite where unaligned access exception is > > generated on Itanium. The unaligned access means that someone tries to > > read or write memory crossing 8-bytes boundary. Usually this occur as > > a result of pointers casting. > > > > I investigated it a little bit and found that the reason is this line > > in func.c file: > > > > p->sum += sqlite3_value_int64(argv[0]); > > > > This was fixed by check-in [3084] which occurred a few > hours after 3.3.4 was released (2006-Feb-11). > http://www.sqlite.org/cvstrac/chngview?cn=3084 > -- > D. Richard Hipp <[EMAIL PROTECTED]> > > -- Alexei Alexandrov
Re: [sqlite] Unaligned access in SQLite on Itanium
"Alexei Alexandrov" <[EMAIL PROTECTED]> wrote: > Hi, > > I don't know whether it's been already reported or not, so anyway. > There are places in SQLite where unaligned access exception is > generated on Itanium. The unaligned access means that someone tries to > read or write memory crossing 8-bytes boundary. Usually this occur as > a result of pointers casting. > > I investigated it a little bit and found that the reason is this line > in func.c file: > > p->sum += sqlite3_value_int64(argv[0]); > This was fixed by check-in [3084] which occurred a few hours after 3.3.4 was released (2006-Feb-11). http://www.sqlite.org/cvstrac/chngview?cn=3084 -- D. Richard Hipp <[EMAIL PROTECTED]>
[sqlite] Unaligned access in SQLite on Itanium
Hi, I don't know whether it's been already reported or not, so anyway. There are places in SQLite where unaligned access exception is generated on Itanium. The unaligned access means that someone tries to read or write memory crossing 8-bytes boundary. Usually this occur as a result of pointers casting. I investigated it a little bit and found that the reason is this line in func.c file: p->sum += sqlite3_value_int64(argv[0]); It is a part of sumStep function. p->sum is long double, that is 16 bytes size. It means that it must be aligned on 16 bytes boundary, but it is 0x60094248 in my case. The reason it is not aligned is that address returned by sqlite3_aggregate_context is not aligned on 16 bytes boundary which is a must for ia64 systems and which is the alignment of malloc returned pointers. So the reason comes to Mem structure and I can see that zShort member is not forced to be aligned on 16 bytes and in fact it is aligned on 8 bytes. This is where I got stuck because I don't know whether Mem structure is on-disk structure or not. Is it OK to change it? P.S. To catch the unaligned access on Itanium, the easiest way is to say prctl --unaligned=signal ulimit -c unlimited After that unaligned access will cause SIGBUS and core dump will be generated. P.P.S. Unaligned access can lead to serious performance degradations and on some OSes (HP-UX) there isn't default unaligned access handler so it will just crash. -- Alexei Alexandrov
