On Tue, Dec 18, 2018 at 09:03:41AM -0700, Jeff Law wrote:
> Right. This is the classic example and highlights the ABI concerns. If
> we use the low bit to distinguish between a normal function pointer and
> a pointer to a descriptor and qsort doesn't know about it, then we lose.
>
> One way around this is to make *all* function pointers be some kind of
> descriptor and route all indirect calls through a resolver. THen you
Either way, you are creating a new ABI for calling functions through
function pointers. Because of how rarely GNU C nested functions are used
these days, if we want to do anything I'd think it might be better to use
trampolines, just don't place them on the stack, say have a mmaped page of
trampolines perhaps with some pointer encryption to where they jump to, so
it isn't a way to circumvent non-executable stack, and have some register
and unregister function you'd call to get or release the trampoline.
If more trampolines are needed than currently available, the library could
just mmap another such page. A problem is how it should interact with
longjmp or similar APIs, because then we could leak some trampolines (no
"destructor" for the trampoline would be called. The leaking could be
handled e.g. through remembering the thread and frame pointer for which it
has been allocated and if you ask for a new trampoline with a frame pointer
above the already allocated one, release those entries or reuse them,
instead of allocating a new one. And somehow deal with thread exit.
Jakub
