A macOS static linking mystery
I'm trying to diagnose strange GHC linking behavior which, as far as I can
tell, only occurs on macOS. I feel a bit out of my league debugging this,
so I'm hoping someone knows what is causing this.
When building the Haskell libffi library [1], it will link pass -lffi to
GHC. Typically, most installations of libffi will include both a static
library (libffi.a) and a shared library (libffi.{so,dylib,dll}). My
impression is that when the linker finds both, it will default to the
shared library. This is definitely the case on Linux and Windows, at least.
An exception to this rule is macOS, however. On macOS, building libffi
always appears to default to linking against the static version of libffi,
even when a dynamic version is also available. To reproduce this
phenomenon, check out libffi [1] and run the following commands:
$ brew install libffi # If it is not already installed
$ cabal build ctime
$ otool -L $(cabal list-bin ctime)
The output reveals that the compiled executable does not dynamically link
against libffi.dylib:
$ otool -L $(cabal list-bin ctime)
/Users/rscott/Documents/Hacking/Haskell/libffi/dist-newstyle/build/x86_64-osx/ghc-8.10.7/libffi-examples-0.1/x/ctime/build/ctime/ctime:
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 1281.100.1)
/usr/lib/libcharset.1.dylib (compatibility version 2.0.0,
current version 2.0.0)
This is exceedingly strange, since my Hombrew installation does in fact
provide libffi.dylib:
$ ls -alh ~/Software/homebrew/Cellar/libffi/3.4.2/lib/
total 232
drwxr-xr-x 6 rscott 1340850540 192B Aug 7 09:38 .
drwxr-xr-x 11 rscott 1340850540 352B Aug 7 08:51 ..
-rw-r--r-- 1 rscott 134085054070K Aug 7 08:51 libffi.8.dylib
-r--r--r-- 1 rscott 134085054041K Jun 28 2021 libffi.a
lrwxr-xr-x 1 rscott 134085054014B Jun 28 2021 libffi.dylib ->
libffi.8.dylib
drwxr-xr-x 3 rscott 134085054096B Aug 7 08:51 pkgconfig
What's more, this only seems to happen when building the libffi library in
particular. If I try building another library that has a C extra-libraries
dependency, such as HsOpenSSL, then I can find its dynamic dependencies
(libssl.dylib and libcrypto.dylib) using otool:
$ otool -L
dist-newstyle/build/x86_64-osx/ghc-8.10.7/HsOpenSSL-0.11.7.2/build/test-dsa/test-dsa
dist-newstyle/build/x86_64-osx/ghc-8.10.7/HsOpenSSL-0.11.7.2/build/test-dsa/test-dsa:
/usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility
version 1.1.0, current version 1.1.0)
/usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib
(compatibility version 1.1.0, current version 1.1.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current
version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 1281.100.1)
/usr/lib/libcharset.1.dylib (compatibility version 2.0.0,
current version 2.0.0)
I'm at a bit of a loss trying to figure out why this only happens for
libffi, and only on macOS. To make things even stranger, this only seems to
happen when using GHC. If I try to compile a simple libffi C program using
-lffi, for instance, then otool finds libffi.dylib. Is there something
about GHC's linking behavior that would cause this?
Ryan S.
-
[1] https://github.com/remiturk/libffi
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: [Haskell-cafe] [ANNOUNCE] GHC 9.4.1 is now available
Bruno Damour writes: > Hello, > Thanks for this new release ! > Do you plan to add FreeBSD binaries ? Yes, I have recently been working on the FreeBSD CI infrastructure [1] and hope to have this finished in time for 9.4.2. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/6318 signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
On Mon, Aug 08, 2022 at 07:29:38AM -0400, Ryan Scott wrote: > An exception to this rule is macOS, however. On macOS, building libffi > always appears to default to linking against the static version of libffi, > even when a dynamic version is also available. To reproduce this > phenomenon, check out libffi [1] and run the following commands: > > $ brew install libffi # If it is not already installed > $ cabal build ctime > $ otool -L $(cabal list-bin ctime) What is the output of $ pkg-config --libs libffi on this system? If "cabal" passes any additional flags to "pkg-config" use those as well. On my MacOS laptop I get: $ /usr/local/bin/pkg-config --libs libffi -lffi which does not use the "brew"-installed libffi. Not surprising, since /usr/local/lib/pkgconfig/ has no symlink to the "libffi.pc" file. > This is exceedingly strange, since my Hombrew installation does in fact > provide libffi.dylib: > > $ ls -alh ~/Software/homebrew/Cellar/libffi/3.4.2/lib/ > [...] > drwxr-xr-x 3 rscott 134085054096B Aug 7 08:51 pkgconfig For "pkg-config" to find the HomeBrew "libffi", there would need to be a "libffi.pc" symlink to the one in the "pkgconfig" directory. Perhaps there are additional steps to perform in HomeBrew to activate this "libffi" as a default target for "pkg-config". -- Viktor. ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
On Mon, Aug 08, 2022 at 09:59:48AM -0400, Viktor Dukhovni wrote: > On my MacOS laptop I get: > > $ /usr/local/bin/pkg-config --libs libffi > -lffi > > which does not use the "brew"-installed libffi. Not surprising, since > /usr/local/lib/pkgconfig/ has no symlink to the "libffi.pc" file. When updating "libffi" HomeBrew reports: ==> libffi libffi is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble. For compilers to find libffi you may need to set: export LDFLAGS="-L/usr/local/opt/libffi/lib" export CPPFLAGS="-I/usr/local/opt/libffi/include" If the MacOS libffi works, it is probably safer to use it rather than the HomeBrew version. -- Viktor. ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
ahh, pkgconfig is what i was overlooking! agreed with viktor, using the system provided one is definitely safer On Mon, Aug 8, 2022 at 10:46 AM Viktor Dukhovni wrote: > On Mon, Aug 08, 2022 at 09:59:48AM -0400, Viktor Dukhovni wrote: > > > On my MacOS laptop I get: > > > > $ /usr/local/bin/pkg-config --libs libffi > > -lffi > > > > which does not use the "brew"-installed libffi. Not surprising, since > > /usr/local/lib/pkgconfig/ has no symlink to the "libffi.pc" file. > > When updating "libffi" HomeBrew reports: > > ==> libffi > libffi is keg-only, which means it was not symlinked into /usr/local, > because macOS already provides this software and installing another > version in > parallel can cause all kinds of trouble. > > For compilers to find libffi you may need to set: > export LDFLAGS="-L/usr/local/opt/libffi/lib" > export CPPFLAGS="-I/usr/local/opt/libffi/include" > > If the MacOS libffi works, it is probably safer to use it rather than > the HomeBrew version. > > -- > Viktor. > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
I should clarify that I'm using a borrowed macOS on which I don't have admin privileges, so I'm unable to install pkg-config. As a result, I'm commenting out the pkgconfig-depends: line in libffi.cabal and manually specifying the extra-lib-dirs and include-dirs via a cabal.project.local file. I've configured the project to use both the macOS system libffi (located at /usr/lib/libffi.dylib) as well as the version that Homebrew provides. Both configurations exhibit the strange, apparently-statically-linked behavior. Even stranger, the macOS system libffi _only_ provides a .dylib file, not an .a file. If I examine the x86 code of the compiled executable using `objdump -d`, I can see various libffi definitions (e.g., _ffi_call), so perhaps these definitions are being inlined through some kind of link-time optimization? I'm really not sure how else to explain what's happening. Ryan ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
Any chance this is related to the weird system cache thing for system dylibs that came in with the most recent OS X releases? I don't think those show up in the normal way. On Mon, Aug 8, 2022 at 7:33 PM Ryan Scott wrote: > > I should clarify that I'm using a borrowed macOS on which I don't have admin > privileges, so I'm unable to install pkg-config. As a result, I'm commenting > out the pkgconfig-depends: line in libffi.cabal and manually specifying the > extra-lib-dirs and include-dirs via a cabal.project.local file. > > I've configured the project to use both the macOS system libffi (located at > /usr/lib/libffi.dylib) as well as the version that Homebrew provides. Both > configurations exhibit the strange, apparently-statically-linked behavior. > Even stranger, the macOS system libffi _only_ provides a .dylib file, not an > .a file. If I examine the x86 code of the compiled executable using `objdump > -d`, I can see various libffi definitions (e.g., _ffi_call), so perhaps these > definitions are being inlined through some kind of link-time optimization? > I'm really not sure how else to explain what's happening. > > Ryan > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- brandon s allbery kf8nh allber...@gmail.com ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
Possibly? I'm not familiar with the system cache you're referring to, but it's a more solid lead than what I have currently. Do you know how one would confirm if this is the case or not? Ryan ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
https://developer.apple.com/forums/thread/692383 is what I'm thinking of, but in this case there would still be library references shown by "otool -L", so I guess that's not what you're seeing. On Mon, Aug 8, 2022 at 7:49 PM Ryan Scott wrote: > > Possibly? I'm not familiar with the system cache you're referring to, but > it's a more solid lead than what I have currently. Do you know how one would > confirm if this is the case or not? > > Ryan -- brandon s allbery kf8nh allber...@gmail.com ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Partial type synonyms -- first-class!
I haven't heard back on this so I pushed ahead on the 3 outstanding
problems by just applying more violence to the code base, basically.
1. For wildcard-originating type variables, I added a flag to the
`TauTv` constructor of `MetaInfo`. The knock-on effects of this change
were smaller than I expected:
https://gitlab.haskell.org/cactus/ghc/-/commit/d3165c3f37e9455bc7ca88b884cc370ca39d07f8
2. It turns out I misunderstood the `Role` situation and they are
actually computed via some weird knot tying from the `TyCon` itself.
So on one hand, I don't really need to do anything to "come up with"
roles for these implicit type parameters. On the other hand, I think
this also means that this breaks the surface syntax for role
declarations, since the user would be expected to give roles for type
parameters they never explicitly added to their tysyns...
3. Similar to #1, I started just pushing all the way through GHC a
change to `AnonArgFlag` that adds a third `ImplArg` flag. It is
handled mostly the same way as an `InvisArg`, except `tcInferTyApps`
can use it to insert wildcard arguments. I don't have a full commit
for this yet.
Beside comments on this approach, I would also like to hear from Simon
& Richard if they agree that with this new approach, this could now be
a real language feature that could get upstreamed once all these
implementation wrinkles are smoothed out.
Thanks,
Gergo
On Fri, Aug 5, 2022 at 6:17 PM ÉRDI Gergő wrote:
>
> As I mentioned at the end of the original thread (but probably no one
> was interested in this enough to read that far), after a breakthrough
> idea I have now started working on partial type synonyms that are NOT
> just defined using macro expansion semantics, and indeed can be a
> first-class abstraction. I think if I can iron out the wrinkles
> detailed at the bottom of this message, this would be good for a GHC
> proposal.
>
> 1. Defining partial type synonyms
>
> The idea is to typecheck the type synonym declaration
>
> type Syn a = MyType _ [a] Int
>
> into a type synonym that has some implicit (invisible) type parameters:
>
> type Syn {w1} a = MyType w1 [a] Int
>
> We don't need to come up with any new design for "what this means":
> this means exactly the same as
>
> type Syn w1 a = MyType w1 [a] Int
>
> which is something you can already write today. The only difference is
> how `w1` is given at use sites.
>
> 2. Using partial type synonyms
>
> When a partial type synonym is applied, its argument list is expanded
> by adding wildcard arguments for the invisible parameters. So this use
> site:
>
> Syn Double
>
> is typececked into the type application
>
> Syn {_} Double
>
> Again, we don't need to come up with any new design for "what this
> means": this is the same as writing `Syn _ Double` into the same
> position, it's just the `_` is not written in the surface syntax.
>
> In particular:
>
>* The level of the type variable conjured for the `_` is determined
> fully by where the `Syn Double` occurs, and not at all by the
> definition of `Syn`
>
>* Using the type synonym `Syn` is only valid where a wildcard
> occurrence would be valid. In particular, using `Syn` in a type
> signature causes the signature itself to be regarded as partial
> (`isCompleteHsSig` returns `False` for it).
>
> 3. Implementation
>
> I have a proof-of-concept implementation at
> https://gitlab.haskell.org/cactus/ghc/-/compare/master...cactus%2Fpartial-tysyns?from_project_id=1117
>
> I tried to make it as small a change as I could. Its key points are:
>
>* In the renamer, we allow wildcards in `TySynCtx` context
>
>* In `tcTySynRhs`, we allow wildcards in the right-hand side,
> retrieve them from the result of typechecking, and intersperse
> them with the "normal" type parameters to get a well-scoped
> telescope.
>
>* About a third of the lines changed is just a boring refactoring of
> `isCompleteHsSig` to be able to look up type synonym
> `TyCon`s. This is needed because a type signature referring to
> partial type synonyms will be partial itself once those type
> synonym applications are elaborated.
>
>* When typechecking a type application, implicit arguments get
> filled with the result of `tcAnonWildCardOcc`.
>
> Now, my questions about the problems I've encountered along the way. I
> hope these questions are concrete enough that I can get concrete
> answers to them from this list:
>
> 1. How do I find the wildcard-originating tyvars robustly? Currently,
> I am using `candidateQTyVarsWithBinders` but that picks up "regular"
> out-of-scope tyvars as well. I believe this causes a regression of
> #17567; furthermore, it means if the user writes `type Syn a = Foo b` it
> is handled the same as if they wrote `type Syn a = Foo _`, but it would
> be better to reject it instead.
>
> 2. What should the `Role` of these implicit type parameters be? For
> now, I've went with `Nominal` fo
