Re: [HACKERS] problem with msvc linker - cannot build orafce
2015-11-24 11:33 GMT+01:00 Kisung Kim : > 2015-11-24 8:12 GMT+09:00 Chapman Flack : > >> On 11/23/15 15:14, Tom Lane wrote: >> > Lack of PGDLLIMPORT on the extern declaration, no doubt. >> > >> > The fact that we've not heard this before implies that either nobody has >> > ever tried to use orafce on Windows, or it only very recently grew a >> > dependency on session_timezone. >> >> > Actually, we encountered the situation before couple of months. > A client wanted to use orafce on Windows and the same build problem > occurred. > We performed a workaround to edit the PG source to export unresolved > symbols, > which I think of not a good solution. > I found a workaround for session_timezone so master is compileable on msvc again without patching PG source https://github.com/orafce/orafce/commit/9bda5074d44eefebbd002b4720d5833e612428af but can be nice to solve this issue - there can be some interesting variables and not for all the workaround is possible Regards Pavel > > 2015-11-24 8:12 GMT+09:00 Chapman Flack : > >> Has anyone got the stomach to try such a thing and see what happens? >> I don't have MSVC here. >> >> -Chap > > > We have the environment to test your ideas. > Can you explain your ideas with more detail? > > > > > > > (C)Bitnine, Kisung Kim, Ph.D > https://sites.google.com/site/kisungresearch/ > E-mail : ks...@bitnine.co.kr > Office phone : 070-4800-3321 > Mobile phone : 010-7136-0834 > Fax : 02-568-1332 > >
Re: [HACKERS] problem with msvc linker - cannot build orafce
2015-11-23 21:14 GMT+01:00 Tom Lane : > Pavel Stehule writes: > > I am trying to build Orafce and I have problem due access to exported > > variable session_timezone. > > Any idea what can be broken? > > Lack of PGDLLIMPORT on the extern declaration, no doubt. > > The fact that we've not heard this before implies that either nobody has > ever tried to use orafce on Windows, or it only very recently grew a > dependency on session_timezone. > yes, last "official" win build on pgfoundry is for 9.0. Regards Pavel > > regards, tom lane >
Re: [HACKERS] problem with msvc linker - cannot build orafce
On 25 November 2015 at 13:36, Tom Lane wrote: > Craig Ringer writes: > > Actually, if __declspec(dllexport) or a .DEF entry was added in, say, > > 9.4.5, you could probably just: > > > #if PG_VERSION_NUM < 90405 > > extern int* log_min_messages_p; > > #define log_min_messages (*log_min_messages_p) > > #endif > > > after including all PostgreSQL headers. It won't work for inline > functions > > defined in PostgreSQL headers, but should otherwise be OK I think. > > Some of these workarounds look like they would break if we add the missing > PGDLLIMPORT in future releases. That would be nasty :-(. Am I misreading > it? > > I don't think they will, but without testing and more digging I can't be sure. If marking the variable __declspec(dllexport) causes its import table entry to be omitted then yes, that'd break things. I'll try to dig out my Windows VM and prep a few tests once I've delivered on the promised pglogical downstream. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] problem with msvc linker - cannot build orafce
Craig Ringer writes: > Actually, if __declspec(dllexport) or a .DEF entry was added in, say, > 9.4.5, you could probably just: > #if PG_VERSION_NUM < 90405 > extern int* log_min_messages_p; > #define log_min_messages (*log_min_messages_p) > #endif > after including all PostgreSQL headers. It won't work for inline functions > defined in PostgreSQL headers, but should otherwise be OK I think. Some of these workarounds look like they would break if we add the missing PGDLLIMPORT in future releases. That would be nasty :-(. Am I misreading it? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem with msvc linker - cannot build orafce
> > > I don't think that's necessary, per above. You just have to access the > vars via pointer indirection always, so long as *any* Pg version you > support has ever lacked dllexport or DEF entry, so you can't dllimport the > var. > > You could enable direct dllimport if PG_VERSION_NUM shows you're on a new > enough version, but you'd need to use conditionally compiled inlines or > something to switch between the methods of accessing it, so there's not > much point. You just declare > > extern int* log_min_messages_p; > > ... and use that, probably also #define'ing log_min_messages away after > including the Pg headers so that you can't reference it accidentally. > Actually, if __declspec(dllexport) or a .DEF entry was added in, say, 9.4.5, you could probably just: #if PG_VERSION_NUM < 90405 extern int* log_min_messages_p; #define log_min_messages (*log_min_messages_p) #endif after including all PostgreSQL headers. It won't work for inline functions defined in PostgreSQL headers, but should otherwise be OK I think. (again, untested) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: [HACKERS] problem with msvc linker - cannot build orafce
On 24 November 2015 at 07:12, Chapman Flack wrote: > > What I (think I) took away from it was: > > 1. Un-PGDLLIMPORTed references to global *functions* work ok. > Maybe they are thunked and a little less efficient, but they work. > > 2. Un-PGDLLIMPORTed references to global *variables*, not so much. > They used to silently link (at least on some buildfarm critters) > but hold bogus data (maybe a thunk, taken as data?). > > 3. The one concrete *action* taken in the course of that thread was to > tweak the build process to make sure such cases at least *fail* > because that's better than silent bogosity. > Correct on all points. The question that interests me most right now: how, if at all, can the > extension author/maintainer work around this issue when it crops up? > AFAIK the only way to do it is to use the .pdb files to find the address of the variable's storage, then create a pointer-to-var that way, after adjusting for the DLL/EXE's base load address - which on some Windows versions is randomized. You might be able to do it with dbghelp.dll . I haven't looked into it properly. > Obviously, the Right Thing To Do is report it and get the PGDLLIMPORT > added here, but still for years to come the extension will have to cope > with being built against PG distributions that lack it. It's safe to add in a point-release because nobody could've been using it before, so all you have to do is require a certain minimum point release. We all know how easy it is to get users to apply point releases ;) If we annotated extern functions too, we could build on UNIX with -fvisibility=hidden and get immediate linker errors on *nix too. As far as I can tell gcc doesn't have the option to control default visibility separately for functions and data exports. We do that on Windows by generating a .DEF file, since Windows doesn't support it directly either. Doing that on *nix would require using readelf then using --retain-symbols-file at link time. Imagine how popular that'd be. So currently we rely on the buildfarm complaining if things are broken on Windows, but of course that only works for core and in-tree extensions. (Who cares about anything else, right?). See https://gcc.gnu.org/wiki/Visibility for details on visibility. Now, I thought I spotted, somewhere in that long thread, the hint of an > idea that the magic works as long as the *extension* has the variable > declared PGDLLIMPORT, even if it wasn't declared that way when the PG > executable itself was built. Anybody else remember that, or did I > imagine it? > I haven't tested, but I believe you can declare it with an extra level of pointer indirection, without __declspec(dllimport), and chase the pointer. As I understand it (and I'm far from an expert here) the symbol in the PE symbol table points to the address of a pointer to the data. MSVC doesn't know if your "extern" references a variable in the same module (in which case you don't have that indirection) or will be resolved by dynamic linking to point to a pointer to the data. I find this bizarre, given that ELF "just does it"... but PE is pretty weird and a fair bit older, a hacked-up variant of COFF. Anyway, __declspec(dllimport) tells MSVC "I'm going to be linking this from an external DLL, do the pointer chasing for me please". See https://msdn.microsoft.com/en-us/library/aa271769(v=vs.60).aspx "Using __declspec(dllexport) and __declspec(dllimport) on Data". I *think* it's safe to do this even if the var is declared __declspec(dllexport). The dllexport declaration makes sure the export is present without the use of a .DEF file to force it. You can __declspec(dllimport) whether it was __declspec(dllexported)'ed or exported via a .DEF file. We auto-export functions in our DEF files, but not variables. > You *might* get away with creating a separate C file (how about > chamberofhorrors.c?) that, rather revoltingly, *doesn't* include the > proper PostgreSQL .h files, only duplicates the necessary declarations > with PGDLLIMPORT added, and exports some getter/setter methods > to the rest of the extension code. (I like the idea of one > chamberofhorrors > better than scattering such rubbish all over the project.) > I don't think that's necessary, per above. You just have to access the vars via pointer indirection always, so long as *any* Pg version you support has ever lacked dllexport or DEF entry, so you can't dllimport the var. You could enable direct dllimport if PG_VERSION_NUM shows you're on a new enough version, but you'd need to use conditionally compiled inlines or something to switch between the methods of accessing it, so there's not much point. You just declare extern int* log_min_messages_p; ... and use that, probably also #define'ing log_min_messages away after including the Pg headers so that you can't reference it accidentally. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Serv
Re: [HACKERS] problem with msvc linker - cannot build orafce
Dne 24. 11. 2015 15:44 napsal uživatel "Chapman Flack" <
c...@anastigmatix.net>:
>
> On 11/24/2015 05:33 AM, Kisung Kim wrote:
> > 2015-11-24 8:12 GMT+09:00 Chapman Flack :
> >> On 11/23/15 15:14, Tom Lane wrote:
> >>> Lack of PGDLLIMPORT on the extern declaration, no doubt.
> >>
> > Actually, we encountered the situation before couple of months.
> > A client wanted to use orafce on Windows and the same build problem
> > occurred.
> > We performed a workaround to edit the PG source to export unresolved
> > symbols,
> > which I think of not a good solution.
>
> >> Has anyone got the stomach to try such a thing and see what happens?
> >> I don't have MSVC here.
> >
> > We have the environment to test your ideas.
> > Can you explain your ideas with more detail?
>
> Well, the main idea is just this: *IF* it is sufficient to declare
> a variable PGDLLIMPORT only in the code that is importing it (the
> big IF because I don't know whether that is true, but something I
> saw in that long earlier thread seemed to suggest it) ...
>
> Then ... the chief problem that needs to be solved is only that
> MSVC won't allow you to redeclare something with PGDLLIMPORT if
> it is also declared without PGDLLIMPORT in a .h file that you
> include. In other words, you can't simply:
>
> #include
> extern PGDLLIMPORT pg_tz session_timezone; /* the right way now */
>
> because it was already declared the wrong way in pgtime.h.
>
> So one idea is just this:
>
> #define session_timezone decoy_session_timezone;
> #include
> #undef decoy_session_timezone;
>
> extern PGDLLIMPORT pg_tz session_timezone; /* the right way now */
>
> which is not a multiple declaration of the same thing, because
> what got declared the wrong way in pgtime.h is now some other thing
> named decoy_session_timezone. You might need to supply a thing by
> that name, to avoid a linker complaint:
>
> pg_tz decoy_session_timezone; /* never used */
>
> IF the original premise is true, then this technique ought to be
> usable in most cases. It would, however, break in cases where the
> .h file declares macros or inline functions that refer to the
> symbol, because they would all end up referring to the decoy.
>
> My other idea, especially if there were several symbols needing
> to be treated this way, would be to do it all in one dedicated
> .c file, so any of the possible problems with #defining away parts
> of an .h file would be contained in one place, and that file could
> have a simple getter function:
>
> pg_tz getSessionTimezone() { return session_timezone; }
>
> which would be used in the rest of the code instead of referring
> to the global directly. (In that case, of course, the same getter
> function would have to be provided in the non-MSVC case too.)
> A setter function could also be made, if the code needs it.
>
I'll do these checks tomorrow.
Thank you very much, msvc is big unknown for me
regards
Pavel
Re: [HACKERS] problem with msvc linker - cannot build orafce
On 11/24/2015 05:33 AM, Kisung Kim wrote:
> 2015-11-24 8:12 GMT+09:00 Chapman Flack :
>> On 11/23/15 15:14, Tom Lane wrote:
>>> Lack of PGDLLIMPORT on the extern declaration, no doubt.
>>
> Actually, we encountered the situation before couple of months.
> A client wanted to use orafce on Windows and the same build problem
> occurred.
> We performed a workaround to edit the PG source to export unresolved
> symbols,
> which I think of not a good solution.
>> Has anyone got the stomach to try such a thing and see what happens?
>> I don't have MSVC here.
>
> We have the environment to test your ideas.
> Can you explain your ideas with more detail?
Well, the main idea is just this: *IF* it is sufficient to declare
a variable PGDLLIMPORT only in the code that is importing it (the
big IF because I don't know whether that is true, but something I
saw in that long earlier thread seemed to suggest it) ...
Then ... the chief problem that needs to be solved is only that
MSVC won't allow you to redeclare something with PGDLLIMPORT if
it is also declared without PGDLLIMPORT in a .h file that you
include. In other words, you can't simply:
#include
extern PGDLLIMPORT pg_tz session_timezone; /* the right way now */
because it was already declared the wrong way in pgtime.h.
So one idea is just this:
#define session_timezone decoy_session_timezone;
#include
#undef decoy_session_timezone;
extern PGDLLIMPORT pg_tz session_timezone; /* the right way now */
which is not a multiple declaration of the same thing, because
what got declared the wrong way in pgtime.h is now some other thing
named decoy_session_timezone. You might need to supply a thing by
that name, to avoid a linker complaint:
pg_tz decoy_session_timezone; /* never used */
IF the original premise is true, then this technique ought to be
usable in most cases. It would, however, break in cases where the
.h file declares macros or inline functions that refer to the
symbol, because they would all end up referring to the decoy.
My other idea, especially if there were several symbols needing
to be treated this way, would be to do it all in one dedicated
.c file, so any of the possible problems with #defining away parts
of an .h file would be contained in one place, and that file could
have a simple getter function:
pg_tz getSessionTimezone() { return session_timezone; }
which would be used in the rest of the code instead of referring
to the global directly. (In that case, of course, the same getter
function would have to be provided in the non-MSVC case too.)
A setter function could also be made, if the code needs it.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem with msvc linker - cannot build orafce
2015-11-24 8:12 GMT+09:00 Chapman Flack : > On 11/23/15 15:14, Tom Lane wrote: > > Lack of PGDLLIMPORT on the extern declaration, no doubt. > > > > The fact that we've not heard this before implies that either nobody has > > ever tried to use orafce on Windows, or it only very recently grew a > > dependency on session_timezone. > > Actually, we encountered the situation before couple of months. A client wanted to use orafce on Windows and the same build problem occurred. We performed a workaround to edit the PG source to export unresolved symbols, which I think of not a good solution. 2015-11-24 8:12 GMT+09:00 Chapman Flack : > Has anyone got the stomach to try such a thing and see what happens? > I don't have MSVC here. > > -Chap We have the environment to test your ideas. Can you explain your ideas with more detail? (C)Bitnine, Kisung Kim, Ph.D https://sites.google.com/site/kisungresearch/ E-mail : ks...@bitnine.co.kr Office phone : 070-4800-3321 Mobile phone : 010-7136-0834 Fax : 02-568-1332
Re: [HACKERS] problem with msvc linker - cannot build orafce
On 11/23/15 15:14, Tom Lane wrote: > Pavel Stehule writes: >> I am trying to build Orafce and I have problem due access to exported >> variable session_timezone. >> Any idea what can be broken? > > Lack of PGDLLIMPORT on the extern declaration, no doubt. > > The fact that we've not heard this before implies that either nobody has > ever tried to use orafce on Windows, or it only very recently grew a > dependency on session_timezone. Or something changed in the build process. I've had Ken Olson report the same symbol inaccessible when he builds PL/Java with MSVC, and he says it has happened since PG 9.4. I read myself to sleep about a week ago catching up on this old thread: http://www.postgresql.org/message-id/24290.1391303...@sss.pgh.pa.us What I (think I) took away from it was: 1. Un-PGDLLIMPORTed references to global *functions* work ok. Maybe they are thunked and a little less efficient, but they work. 2. Un-PGDLLIMPORTed references to global *variables*, not so much. They used to silently link (at least on some buildfarm critters) but hold bogus data (maybe a thunk, taken as data?). 3. The one concrete *action* taken in the course of that thread was to tweak the build process to make sure such cases at least *fail* because that's better than silent bogosity. So it's possible that (3) is what makes both Orafce and PL/Java seem to have started failing "recently" even though the code in question may be years old (and for most of that time, while linking without complaint, may have been bogus without someone noticing). The question that interests me most right now: how, if at all, can the extension author/maintainer work around this issue when it crops up? Obviously, the Right Thing To Do is report it and get the PGDLLIMPORT added here, but still for years to come the extension will have to cope with being built against PG distributions that lack it. (Never mind the reason, when the extension doesn't build, it's the extension that looks bad.) Now, I thought I spotted, somewhere in that long thread, the hint of an idea that the magic works as long as the *extension* has the variable declared PGDLLIMPORT, even if it wasn't declared that way when the PG executable itself was built. Anybody else remember that, or did I imagine it? The snag seemed to be, MSVC won't tolerate two extern declarations, one PGDLLIMPORT and one without, so you can't get away with including the .h file (which might not have the PGDLLIMPORT) and then declaring it yourself. You *might* get away with creating a separate C file (how about chamberofhorrors.c?) that, rather revoltingly, *doesn't* include the proper PostgreSQL .h files, only duplicates the necessary declarations with PGDLLIMPORT added, and exports some getter/setter methods to the rest of the extension code. (I like the idea of one chamberofhorrors better than scattering such rubbish all over the project.) That might work ok for log_min_messages and client_min_messages, which are just ints. Trickier maybe for session_timezone, which is a pg_tz, so you can't really avoid pgtime.h. That leaves even more revolting options, like #define session_timezone decoy_session_timezone #include #undef session_timezone extern PGDLLIMPORT pg_tz session_timezone pg_tz decoy_session_timezone; /* unused */ Has anyone got the stomach to try such a thing and see what happens? I don't have MSVC here. -Chap -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] problem with msvc linker - cannot build orafce
Pavel Stehule writes: > I am trying to build Orafce and I have problem due access to exported > variable session_timezone. > Any idea what can be broken? Lack of PGDLLIMPORT on the extern declaration, no doubt. The fact that we've not heard this before implies that either nobody has ever tried to use orafce on Windows, or it only very recently grew a dependency on session_timezone. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
