Re: Shared pain
On Friday, 19 November 2010 at 06:53:40 UTC, Steve Teale wrote: On Thu, 18 Nov 2010 11:26:39 +, Steve Teale wrote: I had D code that provided a basis for creation of Windows services, which I have just tried to get working with the latest D2. No dice. I have made some progress in understanding this. It appears that any D static data structures containing function pointers will now cause a Windows service to crash. [...] I am currently stuck with the same problem. I have code for a windows service written in D1 that works just fine. And I can't get my head around what needs to be changed to make this darn thing work with D2. Steve, did you find a way to make your windows service work with D2? Care to show some example code or to point me to the right direction? All other information regarding windows service and D seem to be D1 only.
Re: Shared pain
On Fri, 19 Nov 2010 01:49:30 -0500, Steve Teale steve.te...@britseyeview.com wrote: As I said before, I don't know if the thread being created by the windows service procedure is properly initializing the D modules of the library/runtime. Try as the first line of ServiceMain to initialize the current thread: auto mythread = thread_attachThis(); see http://www.digitalmars.com/d/2.0/phobos/core_thread.html -Steve Steve, I don't think it gets as far as ServiceMain. Anyway, I tried it there, and at the point just before the first WINAPI call, but it still crashes. I can fix it, by making my statically initialized RegExp objects __gshared or immutable, but the latter involves inserting a slew of tedious casts into my XML parser, since RexExp calls are used all over the place. See my separate post for my thoughts on that. OK, I'm out of ideas, sorry :( It certainly looks like there is an issue with TLS. I wouldn't suggest band-aiding things how you are doing, because some modules in phobos/druntime use TLS also. It's reasonable to assume that if your uses of TLS are failing, those may fail as well. -Steve
Re: Shared pain
A cast to immutable is required when constructing immutable objects. The type
system can't prove that's safe. It's a design limitation to keep complexity low.
http://www.digitalmars.com/d/2.0/phobos/std_regexp.html
Looking at the Regexp docs, find isn't marked const or pure. If that's an
oversite, file it in bugzilla. If it can't be const, then you should find
another approach.
When this stuff crashes for you, is it accessing a different thread's TLS or is
it simply unable to handle TLS at all?
Steve Teale Wrote:
On Thu, 18 Nov 2010 11:26:39 +, Steve Teale wrote:
I had D code that provided a basis for creation of Windows services,
which I have just tried to get working with the latest D2.
No dice.
I have made some progress in understanding this. It appears that any D
static data structures containing function pointers will now cause a
Windows service to crash.
For happiness, these must be forced into the global data segment.
I have demonstrated this within a vestigial service for a couple of cases
- plain old struct containing an int and a function pointer, and for
Regexp objects by using and not using __gshared on the offending objects.
Now that I'm reasonably sure what's happening, I ought to be able to make
these things work by making such static members immutable, and
initializing them in static this().
However, I then run into a slew of complier errors. Lets start with this
simple framework:
import std.stdio;
import std.regexp;
class ThingWithStatic
{
static RegExp rex;
string ns;
static this()
{
rex = RegExp(ab+a);
}
this()
{
ns = abbaabba;
}
int getFromStatic() { return rex.find(ns); }
}
void main()
{
ThingWithStatic tws = new ThingWithStatic();
writefln(%d, tws.getFromStatic());
}
This compiles fine, but if I include this class in a service, and call
getFromStatic(), the service will crash. If I change to:
__gshared RegExp rex;
It compiles fine, and the service does not crash.
If I change to:
static immutable RegExp rex;
Then I get errors:
triv.d(11): Error: cannot implicitly convert expression (opCall(ab
+a,null)) of type std.regexp.RegExp to immutable(RegExp)
triv.d(19): Error: function std.regexp.RegExp.find (string string) is not
callable using argument types (string) immutable
st...@ubuntu:~/scratch$
So then I use a cast:
rex = cast(immutable(RegExp)) RegExp(ab+a);
and get down to:
triv.d(19): Error: function std.regexp.RegExp.find (string string) is not
callable using argument types (string) immutable
as somewhat wierd message in itself. So I use a cast in the call to find
():
return (cast(RegExp) rex).find(ns);
At this point it compiles, and if it's used in the service, the service
does not crash.
BUT - all these complicated casts will be extremely tedious to
administer to code that uses RexExp methods all over the place, so
__gshared is a great temptation. The compiler should try to eliminate
such temptations in the interests of safety.
Also why are these casts necessary. If I am assigning to something that
is the same type and was declared as immutable should not at least the
first cast be done implicitly. In the second case, I'm using a Regexp
object, wherever it may be stored, so why the cast? If find was pure
would the cast still be required?
Thanks
Steve
windows debug [Shared pain]
18.11.2010 21:53, Steve Teale пишет: As you can imagine, this is terrible to debug - printf or similar is bad enough, but for the service I'm pushing entries into the Windows event logger system. [...] But first I think I'll stop and do some carpentry for a couple of days. Then eventually something might dawn on me. Have you tried OutputDebugString()? It's WinAPI function that sends strings to the 'debugger'. DebugView application from SysInternals suite nicely prints those strings, and also times them.
Re: Shared pain
On Fri, 19 Nov 2010 08:25:17 -0500, Jason House wrote: A cast to immutable is required when constructing immutable objects. The type system can't prove that's safe. It's a design limitation to keep complexity low. Jason, But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Languages can't be designed just on theory - some recognition of practicality is also required. As to the thread - I don't think that is under my control. I believe the Windows service dispatcher does its own thing. Thanks for taking the trouble Steve
Re: windows debug [Shared pain]
On Fri, 19 Nov 2010 16:58:26 +0300, Stanislav Blinov wrote: Have you tried OutputDebugString()? It's WinAPI function that sends strings to the 'debugger'. DebugView application from SysInternals suite nicely prints those strings, and also times them. How is this better than the event log. It only takes one line of code, but it's still a pain in the arse. In any case, if it's a service, where does it output the debug string to? Thanks for you suggestion Stanislav, but I'm pretty much worked through it by now - albeit somewhat pissed off ;=) Steve
Re: Shared pain
Steve Teale: Languages can't be designed just on theory - some recognition of practicality is also required. In the case of creation of immutable data structures it looks the opposite to me: DMD is too much pragmatic and there isn't enough theory behind it :-) But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Regarding the creation of immutable data structures, there is a proposal that is probably able to remove some of the pain: the result of strongly pure functions may become implicitly castable to immutable. Bye, bearophile
Re: windows debug [Shared pain]
19.11.2010 19:08, Steve Teale пишет: On Fri, 19 Nov 2010 16:58:26 +0300, Stanislav Blinov wrote: Have you tried OutputDebugString()? It's WinAPI function that sends strings to the 'debugger'. DebugView application from SysInternals suite nicely prints those strings, and also times them. How is this better than the event log. It only takes one line of code, but it's still a pain in the arse. It's better precisely in that it doesn't need event log. Cluttering system log with debug messages is not a thing I personally like :) In any case, if it's a service, where does it output the debug string to? It's not a console output. That function generates special Windows debug event that can be captured by external application (i.e. debugger, but not necessarily). It is very useful with things like services when console is not available and you don't feel like creating log file. SysInternals suite has a DebugView application that captures these debug strings from currently running processes and displays them as a kind of log: they are numbered and timed (see attached screenshot). attachment: debugview.png
Re: Shared pain
On Fri, 19 Nov 2010 11:23:44 -0500, bearophile wrote: Regarding the creation of immutable data structures, there is a proposal that is probably able to remove some of the pain: the result of strongly pure functions may become implicitly castable to immutable. Bye, bearophile BP, I admire your dedication to language theory and purity, but there are many who'd translate that to impracticality and obscurity. I came to D in the first place because I found it refreshingly clear and easy to use after C++ and Java. But now it's getting painful. I've bailed out several times, just keep coming back to see how it is doing. I have a bunch of code that worked with D1 and D2 at one time. I've given up on D1, since it is now obviously legacy, but even without the complexity of supporting both, It's been real hard this visit to get things working again. What's your estimate of how long it will be before D is a stable language? Thanks Steve
Re: Shared pain
On 19-nov-10, at 17:42, Steve Teale wrote: On Fri, 19 Nov 2010 11:23:44 -0500, bearophile wrote: Regarding the creation of immutable data structures, there is a proposal that is probably able to remove some of the pain: the result of strongly pure functions may become implicitly castable to immutable. Bye, bearophile BP, I admire your dedication to language theory and purity, but there are many who'd translate that to impracticality and obscurity. I came to D in the first place because I found it refreshingly clear and easy to use after C++ and Java. But now it's getting painful. I've bailed out several times, just keep coming back to see how it is doing. I have a bunch of code that worked with D1 and D2 at one time. I've given up on D1, since it is now obviously legacy, but even without the complexity of supporting both, It's been real hard this visit to get things working again. What's your estimate of how long it will be before D is a stable language? well D1 is pretty stable I think, if you are interested in stability that is a good choice, has worked well for me. This does not mean that I will not consider D2, but D1 is my main workhorse. Fwzi Thanks Steve
Re: Shared pain
Steve Teale: I admire your dedication to language theory and purity, but there are many who'd translate that to impracticality and obscurity. I like that idea about immutables strong pure functions, but it was not an idea of mine :-) That enhancement request in Bugzilla is not written by me. I came to D in the first place because I found it refreshingly clear and easy to use after C++ and Java. But now it's getting painful. But strongly pure functions are already present in D. So it's morally good to push its usages and implications as far as possible, otherwise we are wasting possibilities. That implicit cast idea is another implication of purity. It's a clean thing, it's not a hack or workaround or a dangerous thing or something like that, it comes from a clean logical reasoning about consequences of purity. So it can't be compared to many other dirty things present in D2. It actually helps to clean the programs a little, because removes the need for one important use case of casts. So I can't agree. There is a large complexity difference between extending the logical implications of an already present feature, and adding a true special case. What's your estimate of how long it will be before D is a stable language? I have no idea, I am not that expert, and probably no one knows. D is developed by a small group of devs, and despite in life you seem to need to be fast everything you do (for example because otherwise others do it before you and you miss the train), D is not a commercial project, so it has less timing pressures. Often commercial software projects produce a worse result, despite having more resources, because of the strict development timing requirements. If you look at the PyPy Python project it's may years out of schedule, it was started by lot of money coming from European Union, but in the end it may deliver something good any way. Open Source projects too need to respect some timing schedules, but in OS you are sometimes able to say it will be done when it will be done, while in most commercial software projects you can't say that. The feature we are describing here probably needs only few lines of code to be implemented in the compiler (but I can't be sure) and it's backward-compatible (because it removes the need of a cast in some cases), so it's a kind of additive change. Bye, bearophile
Re: Shared pain
Steve Teale Wrote: On Fri, 19 Nov 2010 08:25:17 -0500, Jason House wrote: A cast to immutable is required when constructing immutable objects. The type system can't prove that's safe. It's a design limitation to keep complexity low. Jason, But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Languages can't be designed just on theory - some recognition of practicality is also required. I didn't say I liked the need for the cast or even that I think D has the proper design! IIRC, the cast with object creation was in lieu of adding unique tepe qualifier and the extra complexity to get that right. In your example, I believe the object was created once, so a cast really isn't that bad. The bad part to me was the need to cast away immutability with every use. That's why I encouraged a bugzilla entry. I did't check this case, but Phobos had a bad track record with adopting type qualifiers after they were added to D2.
Re: Shared pain
On Fri, 19 Nov 2010 16:21:41 -0500, Jason House wrote: Steve Teale Wrote: On Fri, 19 Nov 2010 08:25:17 -0500, Jason House wrote: A cast to immutable is required when constructing immutable objects. The type system can't prove that's safe. It's a design limitation to keep complexity low. Jason, But design limitations like that will force programmers who are on a tight schedule to just say __gshared, and to hell with it. Languages can't be designed just on theory - some recognition of practicality is also required. I didn't say I liked the need for the cast or even that I think D has the proper design! IIRC, the cast with object creation was in lieu of adding unique tepe qualifier and the extra complexity to get that right. In your example, I believe the object was created once, so a cast really isn't that bad. The bad part to me was the need to cast away immutability with every use. That's why I encouraged a bugzilla entry. I did't check this case, but Phobos had a bad track record with adopting type qualifiers after they were added to D2. I'm a bit calmer now I have got most of my utility modules working again. There were two circumstances where I found the casts particularly irksome. I had at one point an immutable char* that I would be passing to Win32 functions. I was assigning to it the address of a plain static character array, and I had to use a cast to do it. I could not see what harm such an assignment could do. Then I had a slew of cases where I had created immutable RegExp objects, and I had to do (cast(RegExp) rexobj).find(...). I could not see what the cast was protecting here either. I think these are bugs, if you agree I will put them in bugzilla.
Shared pain
I had D code that provided a basis for creation of Windows services,
which I have just tried to get working with the latest D2.
No dice.
The point of failure was in this method
static void StartService()
{
if (!StartServiceCtrlDispatcherA(cast(SERVICE_TABLE_ENTRY *) _sta[0]))
{
...
}
}
Where _sta is an array of SERVICE_TABLE_ENTRY structs. The bits of code
that install and remove the service worked fine, but the service control
dispatcher was clearly not happy about these structs being in TLS, and
the service crashed immediately when I tried to start it.
The class also has a couple of methods with signatures like:
extern (Windows) static export void service_ctrl(uint dwCtrlCode) {...}
That's probably not relevant, but I think it contributed to a host of
errors that finally made be take a big step back.
I re-implemented the thing in the style used for OOP in C.
struct ServiceBase
{
SERVICE_TABLE_ENTRY[2] _sta;
...
}
__gshared ServiceBase __sb;
void initialize(ServiceBase* sb) { ... }
void whatever(ServiceBase* sb ...) { ... }
and used the __sb global within the functions with the Windows signature,
which just became
extern (Windows) export void service_ctrl(uint dwCtrlCode) {...}
Now it's working again, but I want to change it into idiomatically sound
D code.
Style suggestions please!
Steve
Re: Shared pain
I'm not familiar with the API, but are you able to declare your original _sta
as immutable and call it a day? Immutable data should have the same protection
as __gshared but with no implications of bypassing the type system.
If that's not helpful, can you give more details like calling patterns? Is the
function called from one thread or from many? My quick read of the windows
documentation makes it seem like that's true. Probably unimportant, but I'd use
_sta.ptr instead of _sta[0] because the former more clearly implies array
pointer to me, but that may be my own quirky style.
Steve Teale Wrote:
I had D code that provided a basis for creation of Windows services,
which I have just tried to get working with the latest D2.
No dice.
The point of failure was in this method
static void StartService()
{
if (!StartServiceCtrlDispatcherA(cast(SERVICE_TABLE_ENTRY *) _sta[0]))
{
...
}
}
Where _sta is an array of SERVICE_TABLE_ENTRY structs. The bits of code
that install and remove the service worked fine, but the service control
dispatcher was clearly not happy about these structs being in TLS, and
the service crashed immediately when I tried to start it.
The class also has a couple of methods with signatures like:
extern (Windows) static export void service_ctrl(uint dwCtrlCode) {...}
That's probably not relevant, but I think it contributed to a host of
errors that finally made be take a big step back.
I re-implemented the thing in the style used for OOP in C.
struct ServiceBase
{
SERVICE_TABLE_ENTRY[2] _sta;
...
}
__gshared ServiceBase __sb;
void initialize(ServiceBase* sb) { ... }
void whatever(ServiceBase* sb ...) { ... }
and used the __sb global within the functions with the Windows signature,
which just became
extern (Windows) export void service_ctrl(uint dwCtrlCode) {...}
Now it's working again, but I want to change it into idiomatically sound
D code.
Style suggestions please!
Steve
Re: Shared pain
On Thu, 18 Nov 2010 06:26:39 -0500, Steve Teale
steve.te...@britseyeview.com wrote:
I had D code that provided a basis for creation of Windows services,
which I have just tried to get working with the latest D2.
No dice.
The point of failure was in this method
static void StartService()
{
if (!StartServiceCtrlDispatcherA(cast(SERVICE_TABLE_ENTRY *)
_sta[0]))
{
...
}
}
Where _sta is an array of SERVICE_TABLE_ENTRY structs. The bits of code
that install and remove the service worked fine, but the service control
dispatcher was clearly not happy about these structs being in TLS, and
the service crashed immediately when I tried to start it.
I think this has something to do with how services start. From this page:
http://msdn.microsoft.com/en-us/library/ms685990(v=VS.85).aspx
I see that StartServiceCtrlDispatcher creates its own thread to run
ServiceMain, which most likely does not call the D thread initialization
routines (not sure). TLS may not be properly set up inside your thread
that is running ServiceMain, and probably the static this() module
functions have not been called.
The class also has a couple of methods with signatures like:
extern (Windows) static export void service_ctrl(uint dwCtrlCode) {...}
That's probably not relevant, but I think it contributed to a host of
errors that finally made be take a big step back.
I re-implemented the thing in the style used for OOP in C.
struct ServiceBase
{
SERVICE_TABLE_ENTRY[2] _sta;
...
}
__gshared ServiceBase __sb;
void initialize(ServiceBase* sb) { ... }
void whatever(ServiceBase* sb ...) { ... }
and used the __sb global within the functions with the Windows signature,
which just became
extern (Windows) export void service_ctrl(uint dwCtrlCode) {...}
Now it's working again, but I want to change it into idiomatically sound
D code.
Style suggestions please!
I think using a class in D is a good idea. But what you may need is
global functions which forward to your class methods. This is how I would
probably do it.
-Steve
Re: Shared pain
On Thu, 18 Nov 2010 08:33:12 -0500, Steven Schveighoffer wrote: I think using a class in D is a good idea. But what you may need is global functions which forward to your class methods. This is how I would probably do it. -Steve Steve, Thanks. I kind of went that way. Having done the laborious C OOP style thing, I put most of the methods back into the struct, just leaving the extern(Windows) functions at global scope - they referred to the __gshare'd object explicitly - I hate that. Then I hid all of the crap behind a class that sneakily referred to the __gshared struct, but at least now to an external user of the module the interface looks much the same as it did before. Now it works OK with the test case example - a service that just writes entries in the event log. However, the real service I'm trying to update works fine if I run it in debug mode (i.e. basically as just an executable), but if I run it as a service it seems to crash the first time I use a RegExp.find(). As you can imagine, this is terrible to debug - printf or similar is bad enough, but for the service I'm pushing entries into the Windows event logger system. Fortunately my D class for doing that has survived the D2 changes. I'm going to try tweaking the service so it will interact with the desktop, then at least I can revert to printf. But first I think I'll stop and do some carpentry for a couple of days. Then eventually something might dawn on me. Steve
Re: Shared pain
On Thu, 18 Nov 2010 13:53:53 -0500, Steve Teale steve.te...@britseyeview.com wrote: On Thu, 18 Nov 2010 08:33:12 -0500, Steven Schveighoffer wrote: I think using a class in D is a good idea. But what you may need is global functions which forward to your class methods. This is how I would probably do it. -Steve Steve, Thanks. I kind of went that way. Having done the laborious C OOP style thing, I put most of the methods back into the struct, just leaving the extern(Windows) functions at global scope - they referred to the __gshare'd object explicitly - I hate that. Then I hid all of the crap behind a class that sneakily referred to the __gshared struct, but at least now to an external user of the module the interface looks much the same as it did before. Now it works OK with the test case example - a service that just writes entries in the event log. However, the real service I'm trying to update works fine if I run it in debug mode (i.e. basically as just an executable), but if I run it as a service it seems to crash the first time I use a RegExp.find(). As I said before, I don't know if the thread being created by the windows service procedure is properly initializing the D modules of the library/runtime. Try as the first line of ServiceMain to initialize the current thread: auto mythread = thread_attachThis(); see http://www.digitalmars.com/d/2.0/phobos/core_thread.html -Steve
Re: Shared pain
As I said before, I don't know if the thread being created by the windows service procedure is properly initializing the D modules of the library/runtime. Try as the first line of ServiceMain to initialize the current thread: auto mythread = thread_attachThis(); see http://www.digitalmars.com/d/2.0/phobos/core_thread.html -Steve Steve, I don't think it gets as far as ServiceMain. Anyway, I tried it there, and at the point just before the first WINAPI call, but it still crashes. I can fix it, by making my statically initialized RegExp objects __gshared or immutable, but the latter involves inserting a slew of tedious casts into my XML parser, since RexExp calls are used all over the place. See my separate post for my thoughts on that. Thanks Steve
Re: Shared pain
On Thu, 18 Nov 2010 11:26:39 +, Steve Teale wrote:
I had D code that provided a basis for creation of Windows services,
which I have just tried to get working with the latest D2.
No dice.
I have made some progress in understanding this. It appears that any D
static data structures containing function pointers will now cause a
Windows service to crash.
For happiness, these must be forced into the global data segment.
I have demonstrated this within a vestigial service for a couple of cases
- plain old struct containing an int and a function pointer, and for
Regexp objects by using and not using __gshared on the offending objects.
Now that I'm reasonably sure what's happening, I ought to be able to make
these things work by making such static members immutable, and
initializing them in static this().
However, I then run into a slew of complier errors. Lets start with this
simple framework:
import std.stdio;
import std.regexp;
class ThingWithStatic
{
static RegExp rex;
string ns;
static this()
{
rex = RegExp(ab+a);
}
this()
{
ns = abbaabba;
}
int getFromStatic() { return rex.find(ns); }
}
void main()
{
ThingWithStatic tws = new ThingWithStatic();
writefln(%d, tws.getFromStatic());
}
This compiles fine, but if I include this class in a service, and call
getFromStatic(), the service will crash. If I change to:
__gshared RegExp rex;
It compiles fine, and the service does not crash.
If I change to:
static immutable RegExp rex;
Then I get errors:
triv.d(11): Error: cannot implicitly convert expression (opCall(ab
+a,null)) of type std.regexp.RegExp to immutable(RegExp)
triv.d(19): Error: function std.regexp.RegExp.find (string string) is not
callable using argument types (string) immutable
st...@ubuntu:~/scratch$
So then I use a cast:
rex = cast(immutable(RegExp)) RegExp(ab+a);
and get down to:
triv.d(19): Error: function std.regexp.RegExp.find (string string) is not
callable using argument types (string) immutable
as somewhat wierd message in itself. So I use a cast in the call to find
():
return (cast(RegExp) rex).find(ns);
At this point it compiles, and if it's used in the service, the service
does not crash.
BUT - all these complicated casts will be extremely tedious to
administer to code that uses RexExp methods all over the place, so
__gshared is a great temptation. The compiler should try to eliminate
such temptations in the interests of safety.
Also why are these casts necessary. If I am assigning to something that
is the same type and was declared as immutable should not at least the
first cast be done implicitly. In the second case, I'm using a Regexp
object, wherever it may be stored, so why the cast? If find was pure
would the cast still be required?
Thanks
Steve
