[Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
Hi all, A quick search indicates that this problem has come up in the past, but I haven't seen any solutions yet. I'm working on the next Persistent release, and one of the changes is that the included sqlite3 C library has been updated (I believe that's the trigger here). I can compile programs against persistent-sqlite, but if there's TH code involved, or I try to runghc the file, I get an error message like: test.hs: /home/ubuntu/.cabal/lib/persistent-sqlite-1.0.0/ghc-7.4.1/HSpersistent-sqlite-1.0.0.o: unknown symbol `stat64' test.hs: test.hs: unable to load package `persistent-sqlite-1.0.0' I'm running GHC 7.4.1 on Ubuntu 12.04 64-bit. Does anyone have insight into what might be causing this? Thanks, Michael ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Wed, Jul 11, 2012 at 10:25 AM, Michael Snoyman wrote: > test.hs: > /home/ubuntu/.cabal/lib/persistent-sqlite-1.0.0/ghc-7.4.1/HSpersistent-sqlite-1.0.0.o: > unknown symbol `stat64' > test.hs: test.hs: unable to load package `persistent-sqlite-1.0.0' > The immediate cause is that some C source file is calling stat() or lstat() without the right #include files; they go through several levels of backward compatibility macros that end in different system calls. Alternately, something is trying to use one of those functions via the FFI instead of System.Posix.File. -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Wed, Jul 11, 2012 at 5:47 PM, Brandon Allbery wrote: > On Wed, Jul 11, 2012 at 10:25 AM, Michael Snoyman > wrote: >> >> test.hs: >> /home/ubuntu/.cabal/lib/persistent-sqlite-1.0.0/ghc-7.4.1/HSpersistent-sqlite-1.0.0.o: >> unknown symbol `stat64' >> test.hs: test.hs: unable to load package `persistent-sqlite-1.0.0' > > > The immediate cause is that some C source file is calling stat() or lstat() > without the right #include files; they go through several levels of backward > compatibility macros that end in different system calls. Alternately, > something is trying to use one of those functions via the FFI instead of > System.Posix.File. > > -- > brandon s allbery allber...@gmail.com > wandering unix systems administrator (available) (412) 475-9364 vm/sms > Hi Brandon, Thanks for the feedback. However, looking at sqlite3.c, I see the necessary #include statements: #include #include #include I'm confident that none of my code is making calls to stat/stat64 via the FFI. In case it makes a difference, this problem also disappears if I compile the library against the system copy of sqlite3 instead of using the C source. Michael ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On 07/11/2012 05:12 PM, Michael Snoyman wrote: > > Thanks for the feedback. However, looking at sqlite3.c, I see the > necessary #include statements: > > #include > #include > #include > > I'm confident that none of my code is making calls to stat/stat64 via > the FFI. In case it makes a difference, this problem also disappears > if I compile the library against the system copy of sqlite3 instead of > using the C source. You may need some extra defines, see the comments in "man stat64". Regards, ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Wed, Jul 11, 2012 at 9:55 PM, Bardur Arantsson wrote:
> On 07/11/2012 05:12 PM, Michael Snoyman wrote:
> >
> > Thanks for the feedback. However, looking at sqlite3.c, I see the
> > necessary #include statements:
> >
> > #include
> > #include
> > #include
> >
> > I'm confident that none of my code is making calls to stat/stat64 via
> > the FFI. In case it makes a difference, this problem also disappears
> > if I compile the library against the system copy of sqlite3 instead of
> > using the C source.
>
> You may need some extra defines, see the comments in "man stat64".
>
> Regards,
>
>
> ___
> Haskell-Cafe mailing list
> Haskell-Cafe@haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
I've come up with a minimal example that demonstrates this problem. The
crux of the matter is the following C code:
#include
#include
#include
#include
typedef int stat_func(const char*, struct stat*);
stat_func *foo = &stat;
void stat_test(void)
{
struct stat buf;
printf("About to stat-test.c\n");
foo("stat-test.c", &buf);
printf("Done\n");
}
As you can see, all of the include statements are present as necessary. The
code compiles just fine with -Wall -Werror. And when you compile the
Haskell code as well, everything works just fine. But if you follow these
steps, you can reproduce the error I saw:
* Unpack the attached tarball
* `cabal install` in that folder
* `runghc main.hs` from the `exe` folder
On my system at least, I get:
main.hs:
/home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
unknown symbol `stat'
main.hs: main.hs: unable to load package `stat-test-0.1.0.0'
One thing I noticed is that I needed to use a function pointer to trigger
the bug. When I called `stat` directly the in stat_test function, gcc
automatically inlined the call, so that the disassessmbled code just showed
a `moveq` (i.e., it's making the system call directly). But using a
function pointer, we're avoiding the inlining. I believe this is why this
issue only came up with the sqlite3 upgrade: previous versions did not use
a function pointer, but rather hard-coded in how to make a stat call.
Does this expose any other possibilities?
Michael
stat-test-0.1.0.0.tar.gz
Description: GNU Zip compressed data
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Thu, Jul 12, 2012 at 6:29 PM, Michael Snoyman wrote:
>
>
> On Wed, Jul 11, 2012 at 9:55 PM, Bardur Arantsson wrote:
>
>> On 07/11/2012 05:12 PM, Michael Snoyman wrote:
>> >
>> > Thanks for the feedback. However, looking at sqlite3.c, I see the
>> > necessary #include statements:
>> >
>> > #include
>> > #include
>> > #include
>> >
>> > I'm confident that none of my code is making calls to stat/stat64 via
>> > the FFI. In case it makes a difference, this problem also disappears
>> > if I compile the library against the system copy of sqlite3 instead of
>> > using the C source.
>>
>> You may need some extra defines, see the comments in "man stat64".
>>
>> Regards,
>>
>>
>> ___
>> Haskell-Cafe mailing list
>> Haskell-Cafe@haskell.org
>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>
>
> I've come up with a minimal example that demonstrates this problem. The
> crux of the matter is the following C code:
>
> #include
> #include
> #include
> #include
>
> typedef int stat_func(const char*, struct stat*);
>
> stat_func *foo = &stat;
>
> void stat_test(void)
> {
> struct stat buf;
>
> printf("About to stat-test.c\n");
> foo("stat-test.c", &buf);
> printf("Done\n");
> }
>
> As you can see, all of the include statements are present as necessary.
> The code compiles just fine with -Wall -Werror. And when you compile the
> Haskell code as well, everything works just fine. But if you follow these
> steps, you can reproduce the error I saw:
>
> * Unpack the attached tarball
> * `cabal install` in that folder
> * `runghc main.hs` from the `exe` folder
>
> On my system at least, I get:
>
> main.hs:
> /home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
> unknown symbol `stat'
> main.hs: main.hs: unable to load package `stat-test-0.1.0.0'
>
> One thing I noticed is that I needed to use a function pointer to trigger
> the bug. When I called `stat` directly the in stat_test function, gcc
> automatically inlined the call, so that the disassessmbled code just showed
> a `moveq` (i.e., it's making the system call directly). But using a
> function pointer, we're avoiding the inlining. I believe this is why this
> issue only came up with the sqlite3 upgrade: previous versions did not use
> a function pointer, but rather hard-coded in how to make a stat call.
>
> Does this expose any other possibilities?
>
> Michael
>
Actually, I just came up with a workaround: declare some local wrappers to
the stat and fstat functions, and use those in place of stat and fstat in
the rest of the code. You can see the change here[1].
Obviously this is a hack, not a real fix. At this point it looks like a GHC
bug to me. Does anything think otherwise? If not, I'll open a ticket.
Michael
[1]
https://github.com/yesodweb/persistent/commit/d7daf0b2fa401fd97ef62e4e74228146d15d8601
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Thu, Jul 12, 2012 at 06:29:41PM +0300, Michael Snoyman wrote:
> I've come up with a minimal example that demonstrates this problem. The
> crux of the matter is the following C code:
>
> #include
> #include
> #include
> #include
>
> typedef int stat_func(const char*, struct stat*);
>
> stat_func *foo = &stat;
>
> void stat_test(void)
> {
> struct stat buf;
>
> printf("About to stat-test.c\n");
> foo("stat-test.c", &buf);
> printf("Done\n");
> }
>
> As you can see, all of the include statements are present as necessary. The
> code compiles just fine with -Wall -Werror. And when you compile the
> Haskell code as well, everything works just fine. But if you follow these
> steps, you can reproduce the error I saw:
>
> * Unpack the attached tarball
> * `cabal install` in that folder
> * `runghc main.hs` from the `exe` folder
>
> On my system at least, I get:
>
> main.hs:
> /home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
> unknown symbol `stat'
> main.hs: main.hs: unable to load package `stat-test-0.1.0.0'
>
> One thing I noticed is that I needed to use a function pointer to trigger
> the bug. When I called `stat` directly the in stat_test function, gcc
> automatically inlined the call, so that the disassessmbled code just showed
> a `moveq` (i.e., it's making the system call directly). But using a
> function pointer, we're avoiding the inlining. I believe this is why this
> issue only came up with the sqlite3 upgrade: previous versions did not use
> a function pointer, but rather hard-coded in how to make a stat call.
>
> Does this expose any other possibilities?
>
> Michael
Are you trying this on a 32 bit system? And when you compiled that C
program, did you try to add
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE
to the compile command? When I define those the resulting object file
from your example correctly references stat64 instead of stat.
pgpTUPJgg88Ia.pgp
Description: PGP signature
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote: > Are you trying this on a 32 bit system? And when you compiled that C > program, did you try to add > > -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE > > to the compile command? When I define those the resulting object file > from your example correctly references stat64 instead of stat. Er sorry, saw your earlier email now. Could this be a mismatch between how your sqlite.so is compiled and how the cbits in persistent-sqlite are compiled, particularly with largefile support? pgpFocJJrnVmJ.pgp Description: PGP signature ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Thu, Jul 12, 2012 at 7:07 PM, Tristan Ravitch wrote:
> On Thu, Jul 12, 2012 at 06:29:41PM +0300, Michael Snoyman wrote:
> > I've come up with a minimal example that demonstrates this problem. The
> > crux of the matter is the following C code:
> >
> > #include
> > #include
> > #include
> > #include
> >
> > typedef int stat_func(const char*, struct stat*);
> >
> > stat_func *foo = &stat;
> >
> > void stat_test(void)
> > {
> > struct stat buf;
> >
> > printf("About to stat-test.c\n");
> > foo("stat-test.c", &buf);
> > printf("Done\n");
> > }
> >
> > As you can see, all of the include statements are present as necessary.
> The
> > code compiles just fine with -Wall -Werror. And when you compile the
> > Haskell code as well, everything works just fine. But if you follow these
> > steps, you can reproduce the error I saw:
> >
> > * Unpack the attached tarball
> > * `cabal install` in that folder
> > * `runghc main.hs` from the `exe` folder
> >
> > On my system at least, I get:
> >
> > main.hs:
> >
> /home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
> > unknown symbol `stat'
> > main.hs: main.hs: unable to load package `stat-test-0.1.0.0'
> >
> > One thing I noticed is that I needed to use a function pointer to trigger
> > the bug. When I called `stat` directly the in stat_test function, gcc
> > automatically inlined the call, so that the disassessmbled code just
> showed
> > a `moveq` (i.e., it's making the system call directly). But using a
> > function pointer, we're avoiding the inlining. I believe this is why this
> > issue only came up with the sqlite3 upgrade: previous versions did not
> use
> > a function pointer, but rather hard-coded in how to make a stat call.
> >
> > Does this expose any other possibilities?
> >
> > Michael
>
> Are you trying this on a 32 bit system? And when you compiled that C
> program, did you try to add
>
> -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE
>
> to the compile command? When I define those the resulting object file
> from your example correctly references stat64 instead of stat.
>
I'm compiling on a 64 bit system. If I add those definitions, the program
uses stat64 instead, but the only difference is that runghc now prints:
main.hs:
/home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
unknown symbol `stat64'
main.hs: main.hs: unable to load package `stat-test-0.1.0.0'
In other words, it's not the symbol that the object file is referencing
(stat vs stat64) that's the problem: runghc is not able to resolve either
one.
Michael
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Jul 12, 2012 7:13 PM, "Tristan Ravitch" wrote: > > On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote: > > Are you trying this on a 32 bit system? And when you compiled that C > > program, did you try to add > > > > -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE > > > > to the compile command? When I define those the resulting object file > > from your example correctly references stat64 instead of stat. > > Er sorry, saw your earlier email now. Could this be a mismatch > between how your sqlite.so is compiled and how the cbits in > persistent-sqlite are compiled, particularly with largefile support? I don't think so. The test case I put together had nothing to do with sqlite. Also, persistent-sqlite will either use sqlite.so _or_ the included sqlite3.c file (based on a compile-time flag). The former works perfectly, only the latter causes problems. Michael ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Thu, Jul 12, 2012 at 07:20:39PM +0300, Michael Snoyman wrote: > On Jul 12, 2012 7:13 PM, "Tristan Ravitch" wrote: > > > > On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote: > > > Are you trying this on a 32 bit system? And when you compiled that C > > > program, did you try to add > > > > > > -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE > > > > > > to the compile command? When I define those the resulting object file > > > from your example correctly references stat64 instead of stat. > > > > Er sorry, saw your earlier email now. Could this be a mismatch > > between how your sqlite.so is compiled and how the cbits in > > persistent-sqlite are compiled, particularly with largefile support? > > I don't think so. The test case I put together had nothing to do with > sqlite. Also, persistent-sqlite will either use sqlite.so _or_ the included > sqlite3.c file (based on a compile-time flag). The former works perfectly, > only the latter causes problems. > > Michael I was looking at the symbols in libc and noticed that it doesn't actually export stat64/stat, so that would explain something at least. I think your idea about the switch to function pointers versus direct calls is probably right - the linker probably does some rewriting of calls to stat into __fxstat and company, but for some reason doesn't handle references to function pointers. I also ran across this stackoverflow post that mentions something similar: http://stackoverflow.com/questions/5478780/c-and-ld-preload-open-and-open64-calls-intercepted-but-not-stat64 So stat64 is actually special and in this libc_nonshared.a library (it is on my system at least). It would be ugly to have to link that manually - not sure what the right answer is, but at least this might explain it. pgpFjqMeqMQZk.pgp Description: PGP signature ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`
On Thu, Jul 12, 2012 at 7:30 PM, Tristan Ravitch wrote: > On Thu, Jul 12, 2012 at 07:20:39PM +0300, Michael Snoyman wrote: > > On Jul 12, 2012 7:13 PM, "Tristan Ravitch" wrote: > > > > > > On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote: > > > > Are you trying this on a 32 bit system? And when you compiled that C > > > > program, did you try to add > > > > > > > > -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE > > > > > > > > to the compile command? When I define those the resulting object > file > > > > from your example correctly references stat64 instead of stat. > > > > > > Er sorry, saw your earlier email now. Could this be a mismatch > > > between how your sqlite.so is compiled and how the cbits in > > > persistent-sqlite are compiled, particularly with largefile support? > > > > I don't think so. The test case I put together had nothing to do with > > sqlite. Also, persistent-sqlite will either use sqlite.so _or_ the > included > > sqlite3.c file (based on a compile-time flag). The former works > perfectly, > > only the latter causes problems. > > > > Michael > > I was looking at the symbols in libc and noticed that it doesn't > actually export stat64/stat, so that would explain something at least. > I think your idea about the switch to function pointers versus direct > calls is probably right - the linker probably does some rewriting of > calls to stat into __fxstat and company, but for some reason doesn't > handle references to function pointers. > > I also ran across this stackoverflow post that mentions something > similar: > > > http://stackoverflow.com/questions/5478780/c-and-ld-preload-open-and-open64-calls-intercepted-but-not-stat64 > > So stat64 is actually special and in this libc_nonshared.a library (it > is on my system at least). It would be ugly to have to link that > manually - not sure what the right answer is, but at least this might > explain it. > That's a great find, and it really explains a lot. It seems then that: * GNU ld has some black magic do know about the stat/stat64 hack. * Compiling via GHC just uses GNU ld, which is able to make the hack work. * Interpreting with GHC doesn't get to take advantage of GNU ld's hack. I've opened a trac ticket[1] for this issue, thank you very much for the help! Michael [1] http://hackage.haskell.org/trac/ghc/ticket/7072 ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
