Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 02:48 PM, Jonas Maebe wrote: I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other). Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. Sending us off on wild goose chases does not help. This document http://www.akkadia.org/drepper/tls.pdf explicitly states that the implementation of the thrdeadvar needs to be part of the ABI of the compiler. (Thanks to Vinzent Höfler to point me to this file). -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On Tue, 10 Aug 2010, Michael Schnell wrote: On 07/20/2010 02:48 PM, Jonas Maebe wrote: I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other). Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. Sending us off on wild goose chases does not help. This document http://www.akkadia.org/drepper/tls.pdf explicitly states that the implementation of the thrdeadvar needs to be part of the ABI of the compiler. (Thanks to Vinzent Höfler to point me to this file). This is an extension to C which is specific to GCC; it is in no way an official spec except maybe for GCC (and probably linux-only, at that). Microsoft for example doesn't use ELF; as I understand his remark, Jonas wanted an official Microsoft spec. Other than that, it's of course useful to know. Michael.___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 08/10/2010 10:23 AM, Michael Van Canneyt wrote: This is an extension to C which is specific to GCC; it is in no way an official spec except maybe for GCC (and probably linux-only, at that). Microsoft for example doesn't use ELF; as I understand his remark, Jonas wanted an official Microsoft spec. Other than that, it's of course useful to know. Finally I think that there is a decent chance to modify the compiler and the RTL in a way that threadvars can be used fast and without needing a libc binding, as well reading and writing them and constructing a pointer to them (which seemed impossible to me recently): Linux X86/64: using this document Linux X86/32: using this document (*), Windows X86/32: using the document mentioned in another contribution, based on reverse-engineering of the Microsoft C compiler (*) Linux ARM: tbd, but seemingly quite obvious WinCE ARM. tbd, but I suppose this is not really "on the list". (*) as both archs provide a TCB, the implementation can be done very similar, first constructing the DS-relative pointer to a threadvar and then doing the access: Windows: the pointer to the threadvar is constructed by MOV EAX, FS:Windows_TCB-element_offset MOV EAX, threadvaroffset[EAX] Linux: the pointer to the threadvar is constructed by MOV EAX, GS:Linux_TCB-element_offset MOV EAX, threadvaroffset[EAX] -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Op Tue, 10 Aug 2010, schreef Michael Schnell: Finally I think that there is a decent chance to modify the compiler and the RTL in a way that threadvars can be used fast and without needing a libc binding Yes there is. I worked on this a long time ago, but I didn't finish it. The code for that is still in the compiler, work on that can be continued. However, it won't be usefull without libc independend thread mananger. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 08/10/2010 11:04 AM, Daniël Mantione wrote: However, it won't be usefull without libc independend thread mananger. Of course you are very right here ! Is there any question that I might be able to help with (provided that I did a lot of research regarding porting a heavily multi-threaded / multi-language / multi-arch embedded project). -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/19/2010 11:40 AM, Hans-Peter Diettrich wrote: One requirement in a transformation into OOP is the replacement of all affected procedures by methods, and of many (currently global) reference variables by class members. When e.g. later multiple parser instances are created, one for every unit to compile, the current compiler directive handling should be redesigned. Does somebody have experience with (lists of) procedural variables? Any ideas? While I in fact don't understand the meaning of your post: Regarding OOP, would it not be better to do lists of objects with class procedures instead of lists of procedural variables. So these classes would be children of a parent class and you can do a clean list (TList derivate, array or whatever) of same. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: On 07/19/2010 11:40 AM, Hans-Peter Diettrich wrote: One requirement in a transformation into OOP is the replacement of all affected procedures by methods, and of many (currently global) reference variables by class members. When e.g. later multiple parser instances are created, one for every unit to compile, the current compiler directive handling should be redesigned. Does somebody have experience with (lists of) procedural variables? Any ideas? While I in fact don't understand the meaning of your post: Regarding OOP, would it not be better to do lists of objects with class procedures instead of lists of procedural variables. So these classes would be children of a parent class and you can do a clean list (TList derivate, array or whatever) of same. I also have problems to understand your answer :-( The directive list contains records with the name and other attributes of every directive, and with a reference to the directive handler procedure. These handler procedures need further information, that currently resides in global variables. In an OO approach these variables become class members, so that an additional reference to a concrete object is required in the directive handlers. In so far "class procedure" doesn't help, if you meant that, because Self in a class procedure is the class type, not a class instance. It's also useless to create new classes, because the handlers need access to the elements in a concrete parser/scanner class (instance). DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On Mon, Jul 19, 2010 at 12:29, Hans-Peter Diettrich wrote: > > The directive list contains records with the name and other attributes of > every directive, and with a reference to the directive handler procedure. > > These handler procedures need further information, that currently resides in > global variables. In an OO approach these variables become class members, so > that an additional reference to a concrete object is required in the > directive handlers. > > In so far "class procedure" doesn't help, if you meant that, because Self in > a class procedure is the class type, not a class instance. > > It's also useless to create new classes, because the handlers need access to > the elements in a concrete parser/scanner class (instance). Why don't you separate refactoring in several steps: 1) First, you replace all global variables with class members, and as much as you can (global) procedures with class methods. You also add one global Parser (or whatever you call it) variable, and replace those "problematic" procedures with wrappers. Something like: function f(...) begin Result := Parser._f(...) end; This will result is nicer commit. You can even put wrappers just in front of wrapped methods (in the above example, function TParser._f would go just after function f). You'll probably need to add underscore or some other character at the beginning of wrapped names so you can still reference f from TParser methods. 2) After this, you should have parser which is still fully functional, but uses only one global variable - Parser - in wrapping functions. At that moment, you can analyze further refactoring (promoting Parser to parameter of wrapping functions, or some other solution), and even test them without having to deal with huge commits. Just an idea... ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Zitat von Hans-Peter Diettrich : [...] These handler procedures need further information, that currently resides in global variables. In an OO approach these variables become class members, so that an additional reference to a concrete object is required in the directive handlers. About converting global variables for threads: What are the advantages/disadvantages of using ThreadVar versus member variables? Did someone test the speed differences? Mattias ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 19 Jul 2010, at 11:40, Hans-Peter Diettrich wrote: > One requirement in a transformation into OOP is the replacement of all > affected procedures by methods, and of many (currently global) reference > variables by class members. When e.g. later multiple parser instances are > created, one for every unit to compile, the current compiler directive > handling should be redesigned. > > Does somebody have experience with (lists of) procedural variables? See ncnv.pas, ttypeconvnode.first_call_helper. In fact, for the directives it's even simpler because those methods don't have to be overridable (so you can call them directly instead of having to use wrappers). Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/19/2010 12:52 PM, Mattias Gärtner wrote: About converting global variables for threads: What are the advantages/disadvantages of using ThreadVar versus member variables? I don't see what you are trying to compare here. Threadvars are addressed by a dedicated register (I suppose a segment register with on X86) that is different for each thread ant points to a memory area that is dedicated to each thread. Member variables are addressed using the self pointer of the object and thus they are located on the heap and different for each instance of the Object but common for all threads. I suppose the speed difference is very small but the function is very different. If you want an instances of a class for each thread you need to have each thread create it's own instance (on the heap) and and hold the self-Pointer (the "instance"-variable ) in a threadvar. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Zitat von Michael Schnell : On 07/19/2010 12:52 PM, Mattias Gärtner wrote: About converting global variables for threads: What are the advantages/disadvantages of using ThreadVar versus member variables? I don't see what you are trying to compare here. If I understood the mail thread correct, the goal was to change some global variables, so that each thread has its own. This can be achieved with Threadvar or one object per instance. I don't know how ThreadVar is implemented by the compiler on the various systems. Threadvars are addressed by a dedicated register (I suppose a segment register with on X86) that is different for each thread ant points to a memory area that is dedicated to each thread. Can some compiler guru confirm this? After a quick glance on the rtl sources it seems ThreadVar works under some platforms internally pretty much the same as an object instance: it allocates some memory on the heap. Member variables are addressed using the self pointer of the object and thus they are located on the heap and different for each instance of the Object but common for all threads. I suppose the speed difference is very small but the function is very different. If you want an instances of a class for each thread you need to have each thread create it's own instance (on the heap) and and hold the self-Pointer (the "instance"-variable ) in a threadvar. Mattias ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 19 Jul 2010, at 15:02, Mattias Gärtner wrote: > After a quick glance on the rtl sources it seems ThreadVar works under some > platforms internally pretty much the same as an object instance: it allocates > some memory on the heap. That's correct. FPC has largely platform-independent support for threadvars, which is much more heavy-weight than simply using a segment register (indirect procedure call depending on the thread manager, get thread local storage pointer, index local storage with offset specific to the threadvar). There have been attempts to get the segment register approach to work on Windows, but afaik it always crashed on at least one Windows version. Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/19/2010 03:11 PM, Jonas Maebe wrote: That's correct. FPC has largely platform-independent support for threadvars, which is much more heavy-weight than simply using a segment register (indirect procedure call depending on the thread manager, get thread local storage pointer, index local storage with offset specific to the threadvar). Nonetheless in the end a register is needed to be used that is preloaded with different values for different threads. Otherwise a thread can't know which one it is. Registers is the only per-Thread information that is granted to persist by the OS during preemption. There have been attempts to get the segment register approach to work on Windows, but afaik it always crashed on at least one Windows version. YAK I do know that gcc in Linux on X86 uses a segment register. I don't know how gcc works in Windows, but I suppose even Windows is assumed to restore all register values of a thread after a preemption. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Op Mon, 19 Jul 2010, schreef Michael Schnell: On 07/19/2010 03:11 PM, Jonas Maebe wrote: That's correct. FPC has largely platform-independent support for threadvars, which is much more heavy-weight than simply using a segment register (indirect procedure call depending on the thread manager, get thread local storage pointer, index local storage with offset specific to the threadvar). Nonetheless in the end a register is needed to be used that is preloaded with different values for different threads. Otherwise a thread can't know which one it is. Registers is the only per-Thread information that is granted to persist by the OS during preemption. You could also work with the current thread id as a key to a pointer to the threadvars. However, segment registers are indeed the way libpthread resolves the TLS keys. There have been attempts to get the segment register approach to work on Windows, but afaik it always crashed on at least one Windows version. YAK I do know that gcc in Linux on X86 uses a segment register. I don't know how gcc works in Windows, but I suppose even Windows is assumed to restore all register values of a thread after a preemption. Yes, though I am still puzzled how it works on x86_64; it seems regvars there are also accessed using fs, but x86_64 prevents you from writing to segment registers. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Jonas Maebe schrieb: > > There have been attempts to get the segment register approach to work > on Windows, but afaik it always crashed on at least one Windows > version. The segment register approach is not officially documented and requires anyways an additional code so the gain isn't that big. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi! Yes, though I am still puzzled how it works on x86_64; it seems regvars there are also accessed using fs, but x86_64 prevents you from writing to segment registers. As mentioned here http://en.wikipedia.org/wiki/X86_64#Windows in the 10th point Windows uses the GS segment register to store pointers to the TEB (see http://en.wikipedia.org/wiki/Thread_Environment_Block ) of the current running thread (x86 uses FS). So they should still be writable as the TEB is written to by the Win32 subsystem (of which a part runs in usermode) as part of its bookkeeping and maybe also by user applications to implement the 64 bit equivalent of SEH. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Op Mon, 19 Jul 2010, schreef Sven Barth:
Hi!
Yes, though I am still puzzled how it works on x86_64; it seems regvars
there are also accessed using fs, but x86_64 prevents you from writing
to segment registers.
As mentioned here http://en.wikipedia.org/wiki/X86_64#Windows in the 10th
point Windows uses the GS segment register to store pointers to the TEB (see
http://en.wikipedia.org/wiki/Thread_Environment_Block ) of the current
running thread (x86 uses FS). So they should still be writable as the TEB is
written to by the Win32 subsystem (of which a part runs in usermode) as part
of its bookkeeping and maybe also by user applications to implement the 64
bit equivalent of SEH.
Well, the following works on i386 but doesn't on x86_64, even though
modify_ldt succeeds:
program ldt_test;
{$asmmode intel}
uses baseunix,linux;
var p:pointer;
ud:user_desc;
begin
p:=pointer(fpmmap(nil,4096,3,MAP_PRIVATE+MAP_ANONYMOUS,-1,0));
ud.entry_number:=1;
pointer(ud.base_addr):=p;
ud.limit:=4096;
ud.flags:=UD_SEG_32BIT or UD_CONTENTS_DATA or UD_USEABLE;
modify_ldt(1,@ud,sizeof(ud));
asm
mov ax,15
mov fs,ax
mov fs:[0].dword,29
end;
writeln(Pcardinal(p)^);
end.
If you know how to make it work, I keep myself recommended :)
Daniël___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi! On 19.07.2010 18:50, Daniël Mantione wrote: Well, the following works on i386 but doesn't on x86_64, even though modify_ldt succeeds: (...) If you know how to make it work, I keep myself recommended :) Well... according to this thread http://stackoverflow.com/questions/1180796/how-to-use-gs-in-64-bit-windows-assembly-eg-porting-tls-code it's only possible in kernel mode using a special instruction. The kernel might set up the necessary data and the usermode part of the Win32 subsystem fills in the data later. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Jonas Maebe schrieb: See ncnv.pas, ttypeconvnode.first_call_helper. In fact, for the directives it's even simpler because those methods don't have to be overridable (so you can call them directly instead of having to use wrappers). Thanks, this was my solution #2 :-) DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: I do know that gcc in Linux on X86 uses a segment register. I don't know how gcc works in Windows, but I suppose even Windows is assumed to restore all register values of a thread after a preemption. Windows uses segment registers in exception handling, i.e. it should be possible. But the segment descriptor/selector must point to the correct thread-memory address in the address space, what can not be achieved in application code. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Op Mon, 19 Jul 2010, schreef Hans-Peter Diettrich: Michael Schnell schrieb: I do know that gcc in Linux on X86 uses a segment register. I don't know how gcc works in Windows, but I suppose even Windows is assumed to restore all register values of a thread after a preemption. Windows uses segment registers in exception handling, i.e. it should be possible. But the segment descriptor/selector must point to the correct thread-memory address in the address space, what can not be achieved in application code. Okay, so the Windows kernel sets up the segment registers then by default, that is interresting to know. But Linux does not, yet libpthread is using the segment registers. What I find curious is that you can call modify_ldt sucessfully to setup a segment, but then you cannot use it. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/19/2010 05:16 PM, Daniël Mantione wrote: You could also work with the current thread id as a key to a pointer to the threadvars. Theoretically you can, but in praxis this is not doable for each access to a threadvar, as you would need to request the thread ID from the OS with each access to the threadvar which would be horribly slow. You can't store the thread ID in a global variable nor on the heap (as those obviously are not a threadvars) so you can store it on the stack or in register, which both are dedicated to each thread. As the stackpointer value is not known at runtime within any function, using the stack is not possible.So AFAI can see, without alternate OS-support, dedicating a register as a pointer to the threadvars, is the only viable option a compiler has. AFAIK, in Linux the archs specs define which register is supposed to be used as a theradvar pointer by compilers. There might be some archs that don't define a register but a dedicated memory location, managed by the OS when scheduling threads and processes, to be the threadvar pointer. However, segment registers are indeed the way libpthread resolves the TLS keys. Obviously. Yes, though I am still puzzled how it works on x86_64; it seems regvars there are also accessed using fs, but x86_64 prevents you from writing to segment registers. In fact it is not critical that the threadvar register (with X86/32 Linux) is a segment register, With NIOS it's just one of it's 32 general purpose registers. (I'm not very deep into ARM yet, but I'm quite sure that its similar here). So it might be the same with X86/64. The compiler just needs to make sure that this register is not modified ever and that it's used as a threadvar pointer. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: > > AFAIK, in Linux the archs specs define which register is supposed to be > used as a theradvar pointer by compilers. There might be some archs that > don't define a register but a dedicated memory location, managed by the > OS when scheduling threads and processes, to be the threadvar pointer. Where can be a detailed specification of this found? How does it handle dyn. loaded libs using thread vars? ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/19/2010 08:33 PM, Hans-Peter Diettrich wrote: Windows uses segment registers in exception handling, i.e. it should be possible. But the segment descriptor/selector must point to the correct thread-memory address in the address space, what can not be achieved in application code. If Windows sets the segment registers and the application is not supposed to modify them (and assume them to be restored after preemption), maybe one of them is a pointer to some thread-dedicated memory area, that by can be used (or is used by the OS) to hold a pointer to the threadvars or to hold the threadvars themselves (if the size of the threadvar area is denoted in the exe file and thus known to the OS, which might kindly allocate it when a thread is created). -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 11:16 AM, Florian Klaempfl wrote: Michael Schnell schrieb: AFAIK, in Linux the archs specs define which register is supposed to be used as a theradvar pointer by compilers. There might be some archs that don't define a register but a dedicated memory location, managed by the OS when scheduling threads and processes, to be the threadvar pointer. Where can be a detailed specification of this found? I found these: http://stackoverflow.com/questions/1460483/on-which-platforms-is-thread-local-storage-limited-and-how-much-is-available http://www.linux-mips.org/wiki/NPTL How does it handle dyn. loaded libs using thread vars? The just compiler addresses the threadvars relative to a register instead of the pointer to the global area with normal globals. (Or with X86/32 using another segment register instead of the default DS). So regarding the code it's just another Global area, in realtime the base of same is switching automatically when the thread is scheduled. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi! Am 20.07.2010 11:31, schrieb Michael Schnell: On 07/19/2010 08:33 PM, Hans-Peter Diettrich wrote: Windows uses segment registers in exception handling, i.e. it should be possible. But the segment descriptor/selector must point to the correct thread-memory address in the address space, what can not be achieved in application code. If Windows sets the segment registers and the application is not supposed to modify them (and assume them to be restored after preemption), maybe one of them is a pointer to some thread-dedicated memory area, that by can be used (or is used by the OS) to hold a pointer to the threadvars or to hold the threadvars themselves (if the size of the threadvar area is denoted in the exe file and thus known to the OS, which might kindly allocate it when a thread is created). According to http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation you use a dedicated API to modify the TLS. And according to http://en.wikipedia.org/wiki/Thread_Environment_Block the address of the TLS is indeed accessible/saved in the segment registers. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/19/2010 05:22 PM, Florian Klaempfl wrote: The segment register approach is not officially documented Are we discussing Win32 or Win 64 here ? So, what is officially documented regarding restoring registers after preemption. Obviously all "normal" registers and all FPU registers are restored. CS, DS and SS obviously need to be handled appropriately and are not available. Also ES needs to be restored as same is dedicated to be used for local stuff What more do we have ? FS, GS. Here some documentation would be necessary. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 12:51 PM, Sven Barth wrote: According to http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation you use a dedicated API to modify the TLS. And according to http://en.wikipedia.org/wiki/Thread_Environment_Block the address of the TLS is indeed accessible/saved in the segment registers. OK. Great. A lot more complicated than with Linux, but seemingly decently documented. What about Win 64 ? -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: > On 07/19/2010 05:22 PM, Florian Klaempfl wrote: >> The segment register approach is not officially documented > > Are we discussing Win32 or Win 64 here ? At least Win32: accessing the tlb via fs does not work for all windows with one approach. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 01:42 PM, Florian Klaempfl wrote: At least Win32: accessing the tlb via fs does not work for all windows with one approach. According to Stevens Post using the "Thread Information block" which is addressed by FS is documented and thus needs to be working for 256 TLS-Slots (whatever that means) in NT (and supposedly higher). Who cares about Win9x :) . No idea about Win64. BTW what about GS ? If same is just restored after preemption, this could be used by the compiler to point to the Threadvar area. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: > On 07/20/2010 01:42 PM, Florian Klaempfl wrote: >> At least Win32: accessing the tlb via fs does not work for all windows >> with one approach. > > According to Stevens Post using the "Thread Information block" which is > addressed by FS is documented Where is it *officially" (e.g. MSDN) documented? > and thus needs to be working for 256 > TLS-Slots No. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 01:42 PM, Florian Klaempfl wrote: At least Win32: accessing the tlb via fs does not work for all windows with one approach. The documentation Sven mentions says: FS:[0x2C] 4 Win9x and NT Linear address of the thread-local storage array Do you suppose Vista and Win 7 will not apply ? -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 20 Jul 2010, at 14:03, Michael Schnell wrote: > BTW what about GS ? If same is just restored after preemption, this could be > used by the compiler to point to the Threadvar area. Only if you are guaranteed that no externally called code will modify it. In that case, you FPC itself also save/restore that register before and after calling any potentially non-FPC code, since other applications may make the same assumption. If you don't make that assumption, at the very least you still have to reload it after any call to potentially non-FPC code. Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: > On 07/20/2010 01:42 PM, Florian Klaempfl wrote: >> >> At least Win32: accessing the tlb via fs does not work for all windows >> with one approach. > > The documentation It is probably reverse engineered guesswork but no official documentation. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 02:08 PM, Jonas Maebe wrote: Only if you are guaranteed that no externally called code will modify it. This should be documented in the ABI description and of course could be different for the different calling conventions (Register, STDCALL, ...). If the ABI defines that GS can be modified by the callee FP should of course save it on the stack before calling and restore it afterwards. Seems like "Standard" to me. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 02:07 PM, Florian Klaempfl wrote:
It is probably reverse engineered guesswork but no official documentation.
Hmm. I think M$ wants us to do decent Applications and this includes
multithreaded applications that use threadvars ("__thread" in C).
Thus I can't believe they did not document how a compiler is supposed to
handle threadvars.
Also, I suppose there is a documentation for M$'s own C compiler and for
GNU C how they do it on the appropriate arch, as same should be able to
call ASM functions that define their own threadvars in a way compatible
to the main program written in C. Maybe this is even true for Delphi.
-Michael
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 20 Jul 2010, at 14:22, Michael Schnell wrote: > On 07/20/2010 02:08 PM, Jonas Maebe wrote: >> Only if you are guaranteed that no externally called code will modify it. > This should be documented in the ABI description I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other). Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. Sending us off on wild goose chases does not help. Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 02:48 PM, Jonas Maebe wrote: I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other). At least the "Register" ABI is FPC specific and needs to be documented with the compiler. I hope the others are available near that place. Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. I tried to find it in the FPC docs, but was not successful. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 20 Jul 2010, at 15:00, Michael Schnell wrote: > On 07/20/2010 02:48 PM, Jonas Maebe wrote: >> >> I don't see it there (to the extent that there is an official win32 abi >> document; it seems that information is spread over several msdn pages that >> don't even link to each other). > At least the "Register" ABI is FPC specific and needs to be documented with > the compiler. I hope the others are available near that place. The "register" calling convention is Borland-specific and does not specify anything regarding the segment registers: http://docwiki.embarcadero.com/RADStudio/en/Program_Control >> Please stop speculating that certain things "should be documented" or >> "should work" and post links to the official documents. > I tried to find it in the FPC docs, but was not successful. I know of no single calling convention that says anything about the segment registers. It's an OS ABI issue at best, and at worst undefined behaviour. Jonas___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: BTW what about GS ? If same is just restored after preemption, this could be used by the compiler to point to the Threadvar area. Since it turned out that loading segment registers is very time consuming on newer (i486+) processors, the memory management was changed from segmented (16 bit) into paged (32 bit flat). Segment registers never should be changed by an application, it should be left to the OS process/thread scheduler and thread-API calls. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi! Am 20.07.2010 13:43, schrieb Michael Schnell: On 07/20/2010 12:51 PM, Sven Barth wrote: According to http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation you use a dedicated API to modify the TLS. And according to http://en.wikipedia.org/wiki/Thread_Environment_Block the address of the TLS is indeed accessible/saved in the segment registers. OK. Great. A lot more complicated than with Linux, but seemingly decently documented. What about Win 64 ? The Linux and Windows APIs are more or less similar. The fact that the TLS address is stored inside a segment register shouldn't bother us / an app developer anyhow, because that is defined as "implementation detail" (the TEB is only documented by Wine devs not by Microsoft themselves (only the subsystem independant part of the TEB is)). That said Win 64 uses the same API as Win 32 and the segment register used for the TEB is GS instead of FS. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi!
Am 20.07.2010 14:35, schrieb Michael Schnell:
On 07/20/2010 02:07 PM, Florian Klaempfl wrote:
It is probably reverse engineered guesswork but no official
documentation.
Hmm. I think M$ wants us to do decent Applications and this includes
multithreaded applications that use threadvars ("__thread" in C).
Thus I can't believe they did not document how a compiler is supposed to
handle threadvars.
Also, I suppose there is a documentation for M$'s own C compiler and for
GNU C how they do it on the appropriate arch, as same should be able to
call ASM functions that define their own threadvars in a way compatible
to the main program written in C. Maybe this is even true for Delphi.
For this purpose the TLS-API is to be used. The segment registers that
point to the TLS array are undocumented and part of the internal
bookkeeping of a NT system (and yes, Windows Vista and 7 are NT systems
as well).
But because many applications rely on this undocumented TEB Microsoft
doesn't dare to change anything regarding this. But please no fight
about this topic; I've read the blog and the book of a Microsoft
developer who reports to what lengths Microsoft went to ensure
application compatibility (it's called "The old new thing" and I'm
suggesting it to everyone who wants to know something about Windows
internals in a humorous way).
Btw: if you are really curious you might take a look at the TLS
implementation of ReactOS, which is an NT OS after all:
http://svn.reactos.org/svn/reactos/trunk/reactos/dll/win32/kernel32/thread/tls.c?revision=45018&view=markup
Regards,
Sven
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 03:54 PM, Jonas Maebe wrote: The "register" calling convention is Borland-specific and does not specify anything regarding the segment registers: http://docwiki.embarcadero.com/RADStudio/en/Program_Control Sorry. I did not expect that the ABI documentation is still that bad after all these years. It in fact does not even say what to do with the stack pointer (which is handled differently with STDCALL and CDECL and thus it's not really obvious that it needs to be preserved by the callee). I know of no single calling convention that says anything about the segment registers. It's an OS ABI issue at best, and at worst undefined behaviour. IMHO the ABI documentation should define how all registers are handled :( I think sometimes it might be useful for an ASM programmer to use the segment registers, and so he should be informed whether he needs to save and restore them. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 04:11 PM, Hans-Peter Diettrich wrote: BTW what about GS ? If same is just restored after preemption, this could be used by the compiler to point to the Threadvar area. Since it turned out that loading segment registers is very time consuming on newer (i486+) processors, the memory management was changed from segmented (16 bit) into paged (32 bit flat). Segment registers never should be changed by an application, Not really, The OS needs to set the segment registers, anyway, when returning to a preempted thread or process, otherwise it needs to be clearly documented that the user program is not supposed to use them at all. Assigning some value - e.g.the pointer to the threadvar area - to e.g. GS would not harm performance-wise. it should be left to the OS process/thread scheduler and thread-API calls. If a register is not predefined by the OS (such as GS on WIN32), I think the ABI should inform the ASM programmer about how to use it. Simply stating "all other registers need to be preserved by the callee" would suffice and answer the original question about if the compiler would need to safe GS in case an "external" function might trash it. Moreover of course the ABI should clearly state how an ASM function is supposed to create threadvars, anyway. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/20/2010 08:46 PM, Sven Barth wrote: For this purpose the TLS-API is to be used. I understand that the TLS-API is too slow to be really useful for doing global threadvars. Correct ? The segment registers that point to the TLS array are undocumented and part of the internal bookkeeping of a NT system (and yes, Windows Vista and 7 are NT systems as well). But because many applications rely on this undocumented TEB... I suppose they do so because of what is asked above :) Microsoft doesn't dare to change anything regarding this. If their C compiler uses this, many programs done with same would fail if they would change the Segment register based API. So I do bet they will never change it. And thus I think it's viable for FP to do what the M$ C compiler does. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: Since it turned out that loading segment registers is very time consuming on newer (i486+) processors, the memory management was changed from segmented (16 bit) into paged (32 bit flat). Segment registers never should be changed by an application, Not really, The OS needs to set the segment registers, anyway, when returning to a preempted thread or process, Of course the *entire* CPU state is kept in an stored machine state record. My concern was about the ban of using more than one segment selector (DS=CS=ES=SS), related to one set of segment descriptor tables. The thread specific FS/GS selector most probably selects a tiny segment, with small related tables, that do not affect performance a lot. And since the segment registers are only changed on process/thread switches, the resulting overhead is kept to the absolute minimum. otherwise it needs to be clearly documented that the user program is not supposed to use them at all. Apart from that, I've heard that segment registers have no special effect in 64 bit mode at all? How can these registers ever be used, in 64 bit code? Safe bet: don't touch them ever, use only as documented (for FS/GS). Assigning some value - e.g.the pointer to the threadvar area - to e.g. GS would not harm performance-wise. When an OS does not provide an API to create segments (in a 32 bit flat model), there exist no values (segment selectors) that can be loaded into a segment register at all, by application code. Otherwise it *would* harm performance-wise, to load another segment selector into a segment register, due to the associated overhead in the MMU (address translation logic, from virtual into physical addresses). In a non-segmented environment (64 bit mode?) the segment registers have no special meaning, and consequently no such related overhead. it should be left to the OS process/thread scheduler and thread-API calls. If a register is not predefined by the OS (such as GS on WIN32), I think the ABI should inform the ASM programmer about how to use it. Simply stating "all other registers need to be preserved by the callee" would suffice and answer the original question about if the compiler would need to safe GS in case an "external" function might trash it. Of course this would be nice, but till then it's simply wise to not use such undocumented registers at all, neither in compiler created code, nor in ASM. Moreover of course the ABI should clearly state how an ASM function is supposed to create threadvars, anyway. IMO the OS provides means to allocate an thread-local memory block (Thread Local Storage), whose content *can* be used for threadvars. It's not up to the API or ABI to bother with the "record" layout of that TLS memory block, that's all application/compiler specific. A threadvar simply is a member of that record, and the API specified register contains the (direct or indirect) address of the TLS. IMO also no physical/logical separation exists, between application (main thread) and other thread memory, everything exists in the overall process memory. It's up to the compiler/coder, to prevent unsynchronized access to memory locations that could be updated concurrently by multiple threads. BTW, the definition of "threadvar" is not a very practical one. In practice it should be possible to assign threadvars to *specific* threads, otherwise *every* thread has to allocate an TLS containing *all* the threadvars of the entire application. Furthermore it would be sufficient to have all "threadvars" in the specific TThread-derived classes, eliminating the need for any additional TLS. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/21/2010 04:23 PM, Hans-Peter Diettrich wrote: . And since the segment registers are only changed on process/thread switches, the resulting overhead is kept to the absolute minimum. Same with using a segment registers the base for the threadvar area. This only needs to be initiated when a thread is created. This is a quite rare occasion, as you will use a thread-pool if a threaded application needs to switch between multiple active/passive threads, anyway. Apart from that, I've heard that segment registers have no special effect in 64 bit mode at all? How can these registers ever be used, in 64 bit code? Safe bet: don't touch them ever, use only as documented (for FS/GS). I did not do any research yet regarding X86/64 :(. When an OS does not provide an API to create segments (in a 32 bit flat model), there exist no values (segment selectors) that can be loaded into a segment register at all, by application code. If the OS does not provide an API that defines the use of the registers and the OS manufacturer does not provide a documentation that tells us otherwise, I think we are allowed to presume that these registers are "just registers" that work as documented in the way the hardware documentation describes and thus a user application is free to use them anywhere and the OS guaranteed that they are save regarding preemption. AFAIK, this is commonly done with ES, and so I don't see why it's not true with GS. It seems that on thread start in Windows, FS is predefined with a pointer to some Windows specific thread information, but that does not mean that it's not allowed to assign some other value to FS and that it will not be safe. Of course this would be nice, but till then it's simply wise to not use such undocumented registers at all, neither in compiler created code, nor in ASM. Hmm, is BX explicitly documented ? IMO the OS provides means to allocate an thread-local memory block (Thread Local Storage), whose content *can* be used for threadvars. AFAI understand, in Linux (any architecture) the OS just preserves a count of registers and the (e.g. C) compiler just uses one of them as a pointer to the threadvars. When creating a thread the thread library (e.g. libc) creates the threadvar block and assigns the pointer to the said register. It's not up to the API or ABI to bother with the "record" layout of that TLS memory block, that's all application/compiler specific. The way how threadvars are don _is_ compiler specific. Of course it does make sense if all compilers working for a certain OS use the same paradigm here, so the OS _should_ provide a documentation how they should do it,. In Linux of course the gnu C compiler team takes care of all this. A threadvar simply is a member of that record, and the API specified register contains the (direct or indirect) address of the TLS. I suppose this is the Windows specific way IMO also no physical/logical separation exists, between application (main thread) and other thread memory, everything exists in the overall process memory. It's up to the compiler/coder, to prevent unsynchronized access to memory locations that could be updated concurrently by multiple threads. Of course this is correct. But the compiler only has two options to do this. Either do an OS API call foe each access to a global threadvar (very slow) or dedicate a register a a pointer to the block of global threadvars. I know that e.g. NIOS (that is similar to ARM or MIPS) just uses one of it's 32 general purpose registers for this. X86/32 does not have many registers, so it's a good idea to use a segment register. This is what Linux GNUC is officially documented to do and what the "unsafe" documentation mentioned in this thread states for Windows as well (here in fact maybe as a pointer to a pointer to the threadvar block). BTW, the definition of "threadvar" is not a very practical one. In practice it should be possible to assign threadvars to *specific* threads, otherwise *every* thread has to allocate an TLS containing *all* the threadvars of the entire application. Furthermore it would be sufficient to have all "threadvars" in the specific TThread-derived classes, eliminating the need for any additional TLS. The programmer is free to do this by using just another indirection. You may use a single global threadvar that holds the address of an Object (or record) that a thread allocates on the heap when it starts. The Object or Record now can hold any thread specific information. Many global threadvars only make sense if you start the same code in many threads. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: IMO the OS provides means to allocate an thread-local memory block (Thread Local Storage), whose content *can* be used for threadvars. AFAI understand, in Linux (any architecture) the OS just preserves a count of registers and the (e.g. C) compiler just uses one of them as a pointer to the threadvars. When creating a thread the thread library (e.g. libc) creates the threadvar block and assigns the pointer to the said register. Do you mean segment or general-purpose register(s)? How is that different from WinAPI? It's not up to the API or ABI to bother with the "record" layout of that TLS memory block, that's all application/compiler specific. The way how threadvars are don _is_ compiler specific. Of course it does make sense if all compilers working for a certain OS use the same paradigm here, so the OS _should_ provide a documentation how they should do it,. In Linux of course the gnu C compiler team takes care of all this. When we consider threads created in a DLL/.so, too, then the platform has to establish such conventions, for the entire app<->library ABI. A threadvar simply is a member of that record, and the API specified register contains the (direct or indirect) address of the TLS. I suppose this is the Windows specific way How else should it be done, in a practical way? Registers are not a solution, since: Assume you have 10 units, each of which declares only one threadvar - then you would have to reserve 10 registers for these threadvars, application-wide! IMO also no physical/logical separation exists, between application (main thread) and other thread memory, everything exists in the overall process memory. It's up to the compiler/coder, to prevent unsynchronized access to memory locations that could be updated concurrently by multiple threads. Of course this is correct. But the compiler only has two options to do this. Either do an OS API call foe each access to a global threadvar (very slow) or dedicate a register a a pointer to the block of global threadvars. When we agree that threadvars (etc.) are part of the application/process address space, then they can be accessed like any other variables - by their unique address. In a flat memory model a strict separation would require to shrink the application address space, by the amount required for thread address space. IMO not very likely... BTW, the definition of "threadvar" is not a very practical one. In practice it should be possible to assign threadvars to *specific* threads, otherwise *every* thread has to allocate an TLS containing *all* the threadvars of the entire application. Furthermore it would be sufficient to have all "threadvars" in the specific TThread-derived classes, eliminating the need for any additional TLS. The programmer is free to do this by using just another indirection. You may use a single global threadvar that holds the address of an Object (or record) that a thread allocates on the heap when it starts. The Object or Record now can hold any thread specific information. IMO this duplicates the OS overhead (for the thread memory (TLS) implementation), in the application. A single threadvar then would be sufficient for *all* threads, i.e. the one (register) provided for thread-storage by the platform. Many global threadvars only make sense if you start the same code in many threads. Can you give a more concrete example? I'm not familiar with actual thread programming, so I have no idea about the *usage* of threadvars in concrete code. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/23/2010 02:06 PM, Hans-Peter Diettrich wrote: Do you mean segment or general-purpose register(s)? This depends on the arch. X86 does not really have general-purpose registers, while NIOS, ARM MIPS (and supposedly X86/64) has almost only general-purpose registers. How is that different from WinAPI? The whole discussion here is that X86/32-WinAPI in fact uses a segment register for this purpose but it's nowhere decently documented. Moreover (bus not a problem) I understand that gnu C on X86/32 Linux uses a segment register as a pointer to the threadvar area, while the non-documented WinAPI uses a segment register as a pointer to a thread-dedicated structure that again contains a pointer to the threadvar area. When we consider threads created in a DLL/.so, too, then the platform has to establish such conventions, for the entire app<->library ABI. Not really. The main executable can create a thread and a function called in the thread can internally declare and use a static threadvar. So the ABI needs to document how the (ASM) function is supposed to access the threadvar it internally defines. As it needs to use a register for this, the ABI needs to define that the alee is to use exactly this register and (supposedly) that the callee is not supposed to modify it when it returns. How else should it be done, in a practical way? Registers are not a solution, since: Assume you have 10 units, each of which declares only one threadvar - then you would have to reserve 10 registers for these threadvars, application-wide! Sorry but here you are decently wrong. With most archs all static variables (such as globals and threadvars) always are accessed relative to a register. With X86/32 globals and statics are allocated relative to DS, with X86/32-Linux, threadvars are allocated relative to FS. The archs i know, that don't have segment registers use "address"- or "general purpose" registers for this. So we have two "basic data pointers": one for the normal static vars and one for the threadvars. Both are safe regarding preemption. The basic pointer for normal static vars is initialized at the program start and never changed within the application, the pointer to the threadvars is initialized when a thread starts and - because it is preemption save just like any other register - its never changed when this thread is running. Moreover the point with threadvars is that threads that use the same code see different values in the same threadvar. These threads of course can't use different registers to access them as the run the same code. IMO this duplicates the OS overhead (for the thread memory (TLS) implementation), in the application. A single threadvar then would be sufficient for *all* threads, i.e. the one (register) provided for thread-storage by the platform. See "the point with threadvars...". Can you give a more concrete example? See "the point with threadvars...". I'm not familiar with actual thread programming, so I have no idea about the *usage* of threadvars in concrete code. Using threadvars only make sens for multiple threads running the same code. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Michael Schnell schrieb: Do you mean segment or general-purpose register(s)? This depends on the arch. X86 does not really have general-purpose registers, while NIOS, ARM MIPS (and supposedly X86/64) has almost only general-purpose registers. M68K has distinct address and data registers, x86 has segment and data registers, not to mention other special registers (flags...) in every architecture. How is that different from WinAPI? The whole discussion here is that X86/32-WinAPI in fact uses a segment register for this purpose but it's nowhere decently documented. Moreover (bus not a problem) I understand that gnu C on X86/32 Linux uses a segment register as a pointer to the threadvar area, while the non-documented WinAPI uses a segment register as a pointer to a thread-dedicated structure that again contains a pointer to the threadvar area. I see no difference here. IMO the segment register is used implicitly in thread API calls, with no further use by application code. Every thread has one "slot" reserved in the OS-maintained data structure, where it can store one private value. In the "Using Thread Local Storage" entry in the MSDN library this value is the handle of a memory block, created with LocalAlloc. It could be the handle of an local heap as well, created with HeapCreate, when special handling of threadvars shall be implemented in the RTL. No special segment register seems to be involved in the subsequent use of that allocated memory (flat memory model). When we consider threads created in a DLL/.so, too, then the platform has to establish such conventions, for the entire app<->library ABI. Not really. I found the Windows conventions in "Register Usage" in the MSDN Library, telling which registers are considered (non-)volatile. The main executable can create a thread and a function called in the thread can internally declare and use a static threadvar. So the ABI needs to document how the (ASM) function is supposed to access the threadvar it internally defines. As it needs to use a register for this, the ABI needs to define that the alee is to use exactly this register and (supposedly) that the callee is not supposed to modify it when it returns. Application code can use any non-volatile register for (thread...) communication purposes. When an application and a shared library shall work together, they must use the same convention about the use of such registers. How else should it be done, in a practical way? Registers are not a solution, since: Assume you have 10 units, each of which declares only one threadvar - then you would have to reserve 10 registers for these threadvars, application-wide! Sorry but here you are decently wrong. With most archs all static variables (such as globals and threadvars) always are accessed relative to a register. With X86/32 globals and statics are allocated relative to DS, Here the segment register is somewhat irrelevant, since each maps to the entire 4 GB process address space. All 32 bit "pointers" simply are offsets into that address space, like in any other unsegmented architecture. The only difference may be access rights (CS: executable, read-only). with X86/32-Linux, threadvars are allocated relative to FS. FS and GS may be used for special (OS, SEH and TLS) management, they are useless in application code. The archs i know, that don't have segment registers use "address"- or "general purpose" registers for this. So we have two "basic data pointers": one for the normal static vars and one for the threadvars. Both are safe regarding preemption. The basic pointer for normal static vars is initialized at the program start and never changed within the application, the pointer to the threadvars is initialized when a thread starts and - because it is preemption save just like any other register - its never changed when this thread is running. On Windows such base-pointers should reside in non-volatile registers, otherwise no difference in their use. As long as the base values are immutable, the equivalent absolute addresses can be used instead (in contrast to PIC). Moreover the point with threadvars is that threads that use the same code see different values in the same threadvar. These threads of course can't use different registers to access them as the run the same code. Consider a threadvar as a field of a related thread object (instance). DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
24.07.2010 6:55, Hans-Peter Diettrich: IMO the segment register is used implicitly in thread API calls, with no further use by application code. Exactly the opposite (at least delphi on windows). See delphi's RTL. >In the "Using Thread Local Storage" entry in the MSDN library this >value is the handle of a memory block, created with LocalAlloc. FS and GS may be used for special (OS, SEH and TLS) management, they are useless in application code. fs and gs whould indeed hardly ever be touched by application programmer directly (by hand), however (as it was already metioned several times earlier) delphi (on windows) and gcc (supposedly both windows and linux) use them to implement this "special management" so as exceptions and threadvars can be actually used without explicitely using OS APIs. In such case of course fs/gs will not map to the same flat address segment (as cs, ds, es, ss) common for the whole process and all of its threads. Instead, it will map so a special segment created by OS for a specific thread. This segment might also be accessible as some other offset in the flat address space, or might not, but it isn't the point. Point is that a thread's code basically just needs to do a segment override (add 1-byte opcode modifier on x86 AFAIK) each time it wishes to get into its instance-unique data block, and this is supposed to be very inexpensive compared to OS syscalls. Also note, that fs/gs are not much usefull for anything else in an application (can not do any arithmetics, can not index flat offsets directly, etc etc), so they would otherwise be most probably totally unused (wasted) which is kind of pity. >As long as the base values are immutable, the equivalent absolute >addresses can be used instead (in contrast to PIC). Sure. But, as discussed before, all threads share the same code (i.e. the offset can not be hardcoded). Therefore, you'll need some register to access the instance-specific value (unless you are going to make OS calls every time). Probably on any arcitecture some general-purpose register capable of holding address offsets can be used for that. Such register would then have to be mostly excluded from normal use in the application (in other words - mostly wasted). Now, on x86/x86_64 there is yet additional tricky option: imploy a segment register, which would otherwise be unused. This way no general-purpose register is wasted and no OS syscalls are necessary either. Consider a threadvar as a field of a related thread object (instance). This might be fine most of the time (for end user in particular), however there still might be some non-OO code which it is impossible/impractical to convert into object-oriented model. See FPC RTL. HTH. Nikolai ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Am 24.07.2010 13:42, schrieb Nikolai Zhubr: > use them to implement this "special management" so as exceptions and > threadvars can be actually used without explicitely using OS APIs. Delphi does not use a segment register for threadvar handling but OS calls. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
24.07.2010 17:16, Florian Klämpfl: Am 24.07.2010 13:42, schrieb Nikolai Zhubr: use them to implement this "special management" so as exceptions and threadvars can be actually used without explicitely using OS APIs. Delphi does not use a segment register for threadvar handling but OS calls. For a library - yes, you are right (My memory doesn't serve me good) However, for an ordinary application - no: var a: longint; threadvar b: longint; begin [...] a := 1; movebx,0001 b := a; call @GetTls mov[eax+0004],ebx [...] end. where GetTls in case of an ordinary application (not dll) is just this: MOV EAX,TlsIndex MOV EDX,FS:tlsArray MOV EAX,[EDX+EAX*4] ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Nikolai Zhubr schrieb: IMO the segment register is used implicitly in thread API calls, with no further use by application code. Exactly the opposite (at least delphi on windows). See delphi's RTL. Well, once it's set by the OS, it can be used in hackery ;-) In such case of course fs/gs will not map to the same flat address segment (as cs, ds, es, ss) common for the whole process and all of its threads. I doubt that this address range really is excluded from the 4GB app address space, accessible through the other segment registers. Consider a threadvar as a field of a related thread object (instance). This might be fine most of the time (for end user in particular), however there still might be some non-OO code which it is impossible/impractical to convert into object-oriented model. See FPC RTL. Where exactly should I look? In the meantime I found one possible use for threadvars: when some subroutine can be called from different threads, it may want to retrieve the thread context, e.g. the thread object itself. Right? DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Nikolai Zhubr schrieb: where GetTls in case of an ordinary application (not dll) is just this: MOV EAX,TlsIndex MOV EDX,FS:tlsArray MOV EAX,[EDX+EAX*4] I wonder what TlsIndex here is? Also FS:tlsArray seems to contain an address in the general address space, so that only the tlsArray seems to vary across threads? I had thought that TlsIndex also should vary accross threads. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
24.07.2010 19:12, Hans-Peter Diettrich пишет: [...] I doubt that this address range really is excluded from the 4GB app address space, accessible through the other segment registers. IMHO the question is not how to avoid using FS trick altogether (which is of course possible, right), but on the contrary, how to reliably employ it in order to avoid wasting some more usefull registers and excessive OS calls. Nikolai ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
24.07.2010 19:46, Hans-Peter Diettrich: MOV EAX,TlsIndex MOV EDX,FS:tlsArray MOV EAX,[EDX+EAX*4] I wonder what TlsIndex here is? tlsArray = $2C; IIRC for an application (not a dll) TlsIndex is always 0 (But I might be wrong here though) Also FS:tlsArray seems to contain an address in the general address space, so that only the tlsArray seems to vary across threads? I had thought that TlsIndex also should vary accross threads. I think only FS selector (and/or descriptor) varies across threads. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Nikolai Zhubr schrieb: 24.07.2010 19:46, Hans-Peter Diettrich: MOV EAX,TlsIndex MOV EDX,FS:tlsArray MOV EAX,[EDX+EAX*4] I wonder what TlsIndex here is? tlsArray = $2C; IIRC for an application (not a dll) TlsIndex is always 0 (But I might be wrong here though) The main thread may have an index of zero, but every other thread can have an different index. This can be tested or debugged only with real threads. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 04:55 AM, Hans-Peter Diettrich wrote: I see no difference here. IMO the segment register is used implicitly in thread API calls, with no further use by application code. Every thread has one "slot" reserved in the OS-maintained data structure, where it can store one private value. In the "Using Thread Local Storage" entry in the MSDN library this value is the handle of a memory block, created with LocalAlloc. Delphi (Win32) reads the word at location +4 and addresses the threadvars relative to same. I intend to take a look at FPC Win32 and Linux (32 Bit), gnuc and ;Microsoft C if I find the time to do so, -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 04:55 AM, Hans-Peter Diettrich wrote: With most archs all static variables (such as globals and threadvars) always are accessed relative to a register. With X86/32 globals and statics are allocated relative to DS, Here the segment register is somewhat irrelevant, since each maps to the entire 4 GB process address space. All 32 bit "pointers" simply are offsets into that address space, like in any other unsegmented architecture. The only difference may be access rights (CS: executable, read-only). X86/32 always uses a segment register to access memory, If none is given explicitly DS or SS is used appropriately. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 04:55 AM, Hans-Peter Diettrich wrote: Moreover the point with threadvars is that threads that use the same code see different values in the same threadvar. These threads of course can't use different registers to access them as the run the same code. Consider a threadvar as a field of a related thread object (instance). It is very clear what a threadvar is: the same variable automatically is instantiated dedicated for each thread while a normal (static or global) variable is instantiated once for all threads. Multiple threads that each use different code don't need threadvars as they can be programmed to use individual static/global variables. Inly if the same code is used for multiple threads, threadvars are necessary to allow these threads to identify themselves (e.g a threadvar can store an ID number for the thread so that it can know who it is without doing an OS API call. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 03:16 PM, Florian Klämpfl wrote: Delphi does not use a segment register for threadvar handling but OS calls. sorry but this is not true Turbo Delphi: code: var test integer; threadvar threadtest: integer; test := 1; threadtest := 2; compiles to mov [$...], $1 call @GetTls mov [eax+$34], $2 with @GetTLS resulting in (this is not exactly the original code, which in fact reads a zero form some location to EAX and uses it) mov edx, fs:[4] mov eax, [edx] ret -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 05:46 PM, Hans-Peter Diettrich wrote: I wonder what TlsIndex here is? In a simple test I did, it's 0. BTW: FS is always the same value ($3B), Obviously Windows changes the table entry value, the selector $3B points to. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 06:55 PM, Nikolai Zhubr wrote: I think only FS selector (and/or descriptor) varies across threads. Seemingly not the selector value (the FS content seems to stay constant among the threads), but the table entry it selects. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/25/2010 10:22 AM, Hans-Peter Diettrich wrote: The main thread may have an index of zero, but every other thread can have an different index. This can be tested or debugged only with real threads. Done (with Turbo Delphi): see above. Trying to do tests with FP, gnuC and M$ C. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/26/2010 10:28 AM, Michael Schnell wrote: mov edx, fs:[4] Typo: $2C, instead of $4 -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 05:12 PM, Hans-Peter Diettrich wrote:
In the meantime I found one possible use for threadvars: when some
subroutine can be called from different threads, it may want to
retrieve the thread context, e.g. the thread object itself. Right?
Yep. But of course you can do multiple threads that completely execute
the same code (e.g. a Web Server that handles multiple externals
requests at the same time.
-Michael
Here a simple test program with threadvars;
unit Unit61;
interface
uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls,
Forms,
Dialogs, StdCtrls;
type
TForm61 = class(TForm)
Button1: TButton;
Button2: TButton;
procedure FormCreate(Sender: TObject);
procedure Button1Click(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;
TMyThread = class(TThread)
procedure execute; override;
end;
var
Form61: TForm61;
test: Integer;
teststr: String;
MyThread1, MyThread2 : TMythread;
threadvar threadtest: Integer;
implementation
{$R *.dfm}
function dotest: String;
begin
inc(test);
inc(threadtest);
Result := IntToStr(test) + '/' + IntToStr(threadtest);
end;
procedure TForm61.Button1Click(Sender: TObject);
var s: String;
begin
s := dotest;
teststr := teststr + ' ' + s;
caption := teststr;
end;
procedure TForm61.FormCreate(Sender: TObject);
begin
test := 10;
threadtest := 20;
MyThread1 := TMythread.Create(false);
MyThread2 := TMythread.Create(false);
end;
{ TMyThread }
procedure TMyThread.execute;
var s: String;
begin
s:= dotest;
teststr := teststr + ' .. ' + s;
while (true) do
sleep(1);
end;
end.
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi! Am 26.07.2010 11:07, schrieb Michael Schnell: On 07/26/2010 10:28 AM, Michael Schnell wrote: mov edx, fs:[4] Typo: $2C, instead of $4 I already wondered what "crap" Delphi is doing here :P Btw: would you mind to start a new thread about threadvar implementation, because this discussion currently get's a bit lost in this big "OO rewrite" thread. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/24/2010 06:52 PM, Nikolai Zhubr wrote: IMHO the question is not how to avoid using FS trick altogether (which is of course possible, right), but on the contrary, how to reliably employ it in order to avoid wasting some more usefull registers and excessive OS calls. It's only possible to dedicate another register to that purpose. With X86/32 it does not seem to make much sense, as this arch has so few registers and dedicating one would slow down all programs - and the Win32 FS-"trick" documented in the unofficial aper mentioned above and used in Delphi (and supposedly gnuC and M$ C) is believed to work fine (AFAIK, in Linux GS is different with different threads and directly points to the TLS, but I'm not sure, yet). With X86/64 and other archs (such as ARM) that provide many general purpose registers, I suppose it's a usual way to dedicate one of them as a Thread Local Storage area pointer. Maybe Win64 uses some "Selector" mechanism (similar as ) so that there are no different Selector-Register values in different threads. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
Hi! Am 26.07.2010 11:22, schrieb Michael Schnell: Maybe Win64 uses some "Selector" mechanism (similar as ) so that there are no different Selector-Register values in different threads. Win64 uses GS instead of FS for the TEB and everything else stays the same. In fact FS and GS (and to some extend CS) are the only selectors actively used by the x86_64 platform, but - if I understand that correctly - they can only be set up from ring 0 code (thus: kernel code), which normally happens during context switches so that the data of the current thread's TEB is available again. The usage of the FS and GS selectors in 64bit is also described in AMD's manual: e.g http://support.amd.com/us/Processor_TechDocs/24592.pdf 2.1.2 Segment Registers Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] OO rewrite - technical questions
On 07/26/2010 11:41 AM, Sven Barth wrote: Win64 uses GS instead of FS for the TEB and everything else stays the same. In fact FS and GS (and to some extend CS) are the only selectors actively used by the x86_64 platform, but - if I understand that correctly - they can only be set up from ring 0 code (thus: kernel code), which normally happens during context switches so that the data of the current thread's TEB is available again. OK. Supposedly, Win64 - like Win32 - use the same GS value for all threads and different tables this selector points to. But this is not relevant to the compiler or ASM programmer as it yields the same effect. Supposedly, the compiler does not ever need to write to FS/GS not the the table they select from, so no problem here. But the question remains why - as stated above by the FPC team - trying to use GS in this way, did not seem to work reliably on Win64. -Michael ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
