Re: [RFC] named pipe message-mode design
On Wed, Mar 4, 2009 at 10:14 PM, Alexandre Julliard wrote: > Luke Kenneth Casson Leighton writes: > >>>> how would you envisage doing client-side SMB named pipes? >>> >>> By doing the I/O through the wineserver. It has all the necessary >>> mechanisms already. >> >> ok - great. whereabouts? which ones? any existing examples? which >> existing code in wineserver utilises the existing mechanisms to which >> you refer? > > Look at the ioctl support. ok - great. will do. i'll come back with further questions when i have worked out what you're hinting at. anyone else who is also interested in seeing a "better" design: i trust that they will also be looking at the ioctl support in wineserver with a view to being in a good position to contribute to the better design. l.
Re: [RFC] named pipe message-mode design
>> sure you can. by redesigning. >> >> >> > Since I deal with that on a daily basis, I'll step in. A great design > is one that does EVERYTHING right the first time. have you heard of incremental improvements? > What you are > proposing goes counter to this and is unacceptable. have you heard of incremental improvements? > Do it right the > first time and you don't have to revisit, revisit and revisit some more. have you heard of incremental improvements? > In other words: AJ is right, you are just looking for the easy way > out. yes. don't like it, pay me money. problem solved. > Not a good idea and others end up cleaning up when the users start > whining. ok. 1) you are confusing a "faulty" implementation with a "technically incorrect" implementation. "revisit more and more and more" has implicity within it the assumption that there will be "something wrong" with The Piper design, such that it will require more and more and more work to make it work. in this regard, you are dead wrong. i expect someone else to RIP OUT the ENTIRE design, replacing it with a quotes better quotes one. replacing it with one that identically fulfils the requirements. and you will have the advantage, in the mean-time, of being able to "move on". "move forward". admittedly with a few whiners but they can be silenced with "if you don't like it, fix it". 2) if you would like me to spend more of my personal time on something that would take more effort, you need to accompany such requests by offers of payment, to compensate me for the additional work. so. moving on: would anyone like to provide details on an alternative design proposal for message-mode named pipes that involves an implementation inside wineserver (using the infrastructure that alexandre hinted could be used - the ioctl infrastructure)? l.
Re: [RFC] named pipe message-mode design
On Thu, Mar 5, 2009 at 3:06 AM, Ben Klein wrote: > 2009/3/5 Luke Kenneth Casson Leighton : >> On Wed, Mar 4, 2009 at 6:10 PM, Alexandre Julliard >> wrote: >>> Luke Kenneth Casson Leighton writes: >>> >>>> i would imagine that inefficient is the _last_ thing on the list of >>>> priorities. "technically correctly fulfilling the semantics" i would >>>> imagine would be the highest priority. >>>> >>>> "efficient" and "nice" can always be done later, yes? >>> >>> No, in many cases efficiency needs to be taken into account in the >>> design phase. You can't just add it later. >> >> sure you can. by redesigning. > > So you're saying that your original design is wasted effort? no, i'm saying that it opens up the doors to the next level for wine - networked msrpc interoperability. > If it CAN > be redesigned to be efficient, nice, sensible and correct, it's worth > doing that from the start, especially in a big project such as Wine. yes it would be nice, wouldn't it. however, if there's a nicer design that would involve _more_ effort on my part rather than less, then offers of money should accompany the requests to implement the better design, to compensate me for the additional time spent. if however the nicer design turns out to involve _less_ effort on my part, i'm very very happy. l.
Re: [RFC] named pipe message-mode design
On Wed, Mar 4, 2009 at 6:10 PM, Alexandre Julliard wrote: > Luke Kenneth Casson Leighton writes: > >> i would imagine that inefficient is the _last_ thing on the list of >> priorities. "technically correctly fulfilling the semantics" i would >> imagine would be the highest priority. >> >> "efficient" and "nice" can always be done later, yes? > > No, in many cases efficiency needs to be taken into account in the > design phase. You can't just add it later. sure you can. by redesigning. and because the functionality is encapsulated, fulfilling the requirements behind a well-documented immutable API, other teams can move forward (even if they're complaining about speed) opening up entirely new areas of functionality and opportunities for Wine, bringing in more interested developers, one of whom _might_ be skilled enough and interested enough to go "that design - it's crap! i can do better!" and thus you get a better, more efficient and much more acceptable ... _incremental_ improvement. so by shutting the door on an idea because it's "inefficient" and "not nice" you risk losing my input, and an opportunity to get the ball rolling. i'm not interested at this stage in making a "nice or efficient" design, i'm interested in getting a "technically correct" design - and a reliable, working implementation. l.
Re: [RFC] named pipe message-mode design
>> how would you envisage doing client-side SMB named pipes? > > By doing the I/O through the wineserver. It has all the necessary > mechanisms already. ok - great. whereabouts? which ones? any existing examples? which existing code in wineserver utilises the existing mechanisms to which you refer? the reason i ask is because such mechanisms could also be used, if they are filedescriptor-based, to do communication with samba franky unix-domain-sockets. but, also, i need to know, to evaluate the mechanisms you mention, and work out how to use them. l. p.s. if they're asynchronous mechanisms, i can't stand asychronous designs, i had it up to ^here^ spending nearly 18 months implementing a single-process asynchronous nmbd (which is basically a DNS server and client, rolled into one). i didn't know any better so i cheerfully did it. so i'd be happy to help design an asynchronous message-mode namedpipe design but would in no way want to be part of its actual implementation. p.p.s. that's why i did a synchronous design for "the piper".
Re: [RFC] named pipe message-mode design
On Wed, Mar 4, 2009 at 2:23 PM, Alexandre Julliard wrote: > Luke Kenneth Casson Leighton writes: > >> so - what do people think? would you agree that a user-space pipe >> "proxy" is an effective solution? > > No, you are on the wrong track. That solution is ugly, inefficient, and i would imagine that inefficient is the _last_ thing on the list of priorities. "technically correctly fulfilling the semantics" i would imagine would be the highest priority. "efficient" and "nice" can always be done later, yes?
Re: [RFC] named pipe message-mode design
On Wed, Mar 4, 2009 at 2:23 PM, Alexandre Julliard wrote: > Luke Kenneth Casson Leighton writes: > >> so - what do people think? would you agree that a user-space pipe >> "proxy" is an effective solution? > > No, you are on the wrong track. That solution is ugly, inefficient, and > it doesn't help anything since the wineserver constraints that you are > trying to avoid obviously apply just as well to your proxy process. how would you envisage doing client-side SMB named pipes?
[RFC] named pipe message-mode design
the requirements for message-mode named pipes semantics on top of unix / wine brings some... interesting limitations on how it can be implemented, and i believe that i have finally come up with something that would fit the requirements: the socketpair equivalent of "double-buffering". as the implementation of this idea would involve changes to (doubling-up) of the wineserver namedpipe state machine, and after the significant amount of time and resources spent (resulting in loss of personal financial income somewhere in excess of £3,000) i'm happy to commit further time _with_ active cooperation and engagement from wine developers, in adding this ... rather strategically important and complex functionality. so - here's a description of the idea and its background: your input into poking at it with a stick and discussing how it can be achieved greatly appreciated. background --- the first (successful) attempt at adding "technically correct" named pipes message-mode semantics brought up several issues: 1) due to multiple threads being able to read from the same pipe, receiving of messages cannot be atomically done by reading a "header" indicating the length of the message (unless there is a per-pipe mutex, which is a bit too much to be adding and would slow things down). one thread would read the header, indicating the beginning of the message; the other thread would read the beginning of the pipe data and _treat_ it as the header. 2) the data cannot be read inside wineserver itself due to the requirement to ensure that wineserver never blocks on read (which would be fatal). write really isn't ok, either, due to asynchronicity possibilities on EPIPE etc. would require a loop, thus delaying further responses, or it would require a state machine (messy). 3) breaking individual messages into separate socketpair() pipes is "technically correct" but quickly results in wineserver running out of filedescriptors! 4) changing the filedescriptor from BLOCKING to O_NONBLOCK in order to do a poll() inside wineserver (in order to do a read()) is mutually exclusively incompatible with having a userspace thread perform a blocking read operation on the same socket, waiting for a new message. logical deduction of design -- so, it was made clear by the socketpair()-queue experiment that messaging semantics _can_ be achieved - without any read/write operations being performed by wineserver itself, and so the next logical step is to limit the number of socketpairs used to just two. i.e. four filedescriptors per named pipe: two for the server-side (CreateNamedPipe), and two for the client-side (NtCreateFile). the use of this "double-buffering" will, i believe, make it possible for the "blocking on read" semantics to be achievable. effectively, the "first" socketpair replaces the "tail" of the socketpair()-queue experiment. however, in order to make _that_ work (i.e. to preserve the message boundaries), writes to the "first" socketpair will require a header (4-byte length) to be prefixed to the message data. BUT - and this is the really important bit - when the data is transferred from the first socketpair to the second, the header is NOT sent, for the recipient to read. otherwise, exactly as is done in the current socketpair-queue experiment: the data structures are locked, marked with the "available data", unlocked, and _then_ the write to the secondary socketpair is done. on a read, the data structures are locked, a wineserver message sent requesting that the amount read is to be subtracted from "available data", and then unlocked. when the "available data" reaches zero, that's the signal indicating that it's time to try reading from the first socketpair to fetch the next message (again, stripping its header). the piper -- you may have noticed that there's nothing mentioned so far about _how_ the data is to be transferred between the pairs of socketpairs. we've already established that it cannot be wineserver that does the transfer (because of blocking / nonblocking mutually exclusive issues and asynchronous issues, already outlined) this is actually good news in disguise. conclusion: a *SEPARATE APPLICATION* must be responsible for ferrying the data between the two socketpairs. for all named pipes. a diagram outlining this all together is here: http://lkcl.net/namedpipes/wine-piper-design.pdf exactly how the client and server "signal" to the piper that a new message is required is yet to be determined, but the simplest way it could be achieved would be by instead of sending you send: <2-byte command> where in the case of "i want piper to wake up!" you send <0x0001> <0x> (no more data) and in the case of "i want to write a message" you send <0x0002> <0x> a ping, effectively. The Good News here's the good bit about using the piper to transfer data in userspace: you don't _have_ to h
rpcrt4 <-> freedce conversion guide: functions and structures with direct equivalence
i've started documenting and reviewing the functions and structures in wine's rpcrt4 implementation and providing direct equivalents, including the prototypes of each, in freedce. http://lkcl.net/software/msrpc_to_dcerpc i figured that if i was going to be saying "the two are equivalent" i might best actually be telling the truth :) so far, it looks like i'm about... 40% accurate: certainly 95% of the "public" functions (the ones with IDL files specifying them) are dead-on, with NDRCContext* and NDRSContext* being difficult to find equivalents (will look tomorrow), Mep* functions being _way_ out (10of), Ndr* presently difficult to track down (will look tomorrow), RpcServer* being 95% dead-on, RpcBinding* being 100% on, Uuid* is going to be dead-on, RpcAsync* being _way_ out, etc. etc. i left the Ndr marshalling functions (conformant, simple, etc) for now, will get to those later. these are just the public functions i'm evaluating (quantity 283 in rpcrt4.def) - there are behind-the-scenes implementations of the second-and-beyond layer APIs which are... well... is it actually worth evaluating them? that wasn't a rhetorical question. is it a worthwhile exercise to point out the equivalent functions, those that can be identified, between "internal" functions as found in rpcrt4, and those in freedce? e.g. struct connection_ops will have a direct equivalence somewhere in freedce (i'd start investigating ncklib/transport/cn/cninit.c, with rpc_prot_call_epv_t and other friends being likely candidates). the internal infrastructure in freedce is just... _incredible_. everything's subdivided down into clean interfaces behind vector tables, such that it's an incredibly trivial task to add a new transport or a new authentication module. i dropped in uxd_naf (unix domain sockets network address family) really easily - the interface for adding network address families (towers, aka sockaddr types) is clearly laid out. the transports are subdivided into dg (datagram) and cn (connection) so if you want to add a datagram transport, you know which set of 25 functions to implement, and likewise for a connection-orientated transport. remember - people have been dropping dce/rpc on top of ncacn_http, direct onto netbios datagrams and netbios sessions, decnet, osi networking, _goodness_ knows what else, and dropping in kerberos authentication, nt domains authentication - the works. you would do well to consider taking at the very least the transport infrastructure and the authentication infrastructure from freedce and just dropping it in. i've already added ncacn_np - on top of _real_ NamedPipes - for the Win32 port of freedce. l.
Re: rpcrt4: Check for null endpoint in RpcServerUseProtseqEpExW
come on folks, pick up the ball. acknowledge that you're aware that
there is an authoritative source of information on this
horrendously-complex topic, in the form of nothing less than a fully
functioning reference implementation.
so this:
> > Ok, I'll try to find out.
> > Rob, could you suggest a good documentation source on RPC? I'm not
> > experienced in it.
>Sure, although this particular aspect isn't well documented.
simply isn't true.
it's _authoritatively_ documented, so concretely documented you could
break it into a million pieces to sink a million people who walked
into your turf, if you were in da mafia or summink :)
so - carrying on from where i hinted at you should investigate, the
"real" work is done here:
/*
* Call the protocol service to do the real work of creating the
* socket(s), setting them up right, and adding them (via calls
* to rpc__network_add_desc).
*/
(*net_epv->network_use_protseq)
(pseq_id, max_calls, rpc_addr, endpoint, status);
so where is that function? well, it's not _a_ function, because
net_epv is a pointer to a vector table [array of
higher-order-functions]. so the design of freedce is _incredibly_
well done, and allows for new transports to be dropped in at will.
and new authentication schemes. etc.
so, where is _an_ example of a function that's pointed to by
network_use_protseq?
a quick grep through the code turns up this:
ncklib/transport/cn/cninit.c:rpc__cn_network_use_protseq,
having a quick look at that file actually shows _two_ connection-based
(cn) protseqs - one is tcp and the other is named pipes (which i'd
added in 2005). both of those are in cninit.c
to answer the specific question about allocation (NULL or not NULL)
you have to chunder through quite a bit of convoluted lookups to find
out where _this_ goes to:
(*naf_epv->naf_addr_alloc) (
_yes_ as you can see, NULL is an acceptable endpoint parameter and
rob is correct in that it means "please allocate me aaabsolutely
aanything you like, per transport".
you can tell this from what happens inside the code. in TCP/IP terms,
NULL means "give me an socket on 0.0.0.0 with _any_ port number".
same in UDP terms. in ncacn_np terms, i don't honestly know - you'll
have to do some experiments to check - but you have two choices:
1) error
2) a random pipe name e.g. \pipe\0x0NNN.0x
grep naf_addr_alloc shows one file - ncklib/include/comnaf.h which
shows promise, showing you what you want, but the trail goes dead
there, until you actually _look_ at comnaf.h and you can see these:
/*
* Declarations of the Network Address Family Extension Service init
* routines.
*/
PRIVATE void rpc__unix_init _DCE_PROTOTYPE_ ((
rpc_naf_epv_p_t * /*naf_epv*/ ,
unsigned32 * /*status*/
));
PRIVATE void rpc__ip_init _DCE_PROTOTYPE_ ((
rpc_naf_epv_p_t * /*naf_epv*/ ,
unsigned32 * /*status*/
));
PRIVATE void rpc__dnet_init _DCE_PROTOTYPE_ ((
rpc_naf_epv_p_t * /*naf_epv*/ ,
unsigned32 * /*status*/
));
PRIVATE void rpc__osi_init _DCE_PROTOTYPE_ ((
rpc_naf_epv_p_t * /*naf_epv*/ ,
unsigned32 * /*status*/
));
which are for unix domain sockets (which i think i added as an extra
NAF in 2004), TCP/IP and UDP/IP, DecNet and goodness knows what OSI is
i _so_ don't want to know :)
_again_ it's hard to track down rpc__ip_init() and friends but jumping
straight to ls ncklib/naf/*.c (because duh the header file
ncklib/include/comnaf.h is the clue) you can see e.g. ipnaf.c
hooraaay, there's the vector table, rpc_naf_epv_t, with the static
function addr_alloc as the first member, so in the IP cases, TCP/IP
and UDP/IP, addr_alloc in ipnaf.c is what's called by that
(*naf_epv->naf_addr_alloc) () call.
addr_alloc calls addr_set_endpoint. addr_set_endpoint does this:
if (endpoint == NULL || strlen ((char *) endpoint) == 0)
{
ip_addr->sa.sin_port = 0;
*status = rpc_s_ok;
return;
}
so - hooray! you now _authoritatively_ have your answer, thanks to
the reference implementation.
when NULL is used as the endpoint, you're expected to be given _any_
available address (which in IP terms means "any port").
now you have to do the same thing for ncacn_np. a random and globally
unique pipe name will be required.
remember also do do exactly what the reference implementation says,
when the endpoint _isn't_ NULL, and strip off leading slashes.
l.
On Sat, Feb 7, 2009 at 4:09 PM, Luke Kenneth Casson Leighton
wrote:
> see implementation in dual BSD-licensed _and_ LGPL-licensed freedce
> reference implementation, on which MSRPC was originally blatantly
> copied
[RFC] Named Pipes emulation and interoperability layer
http://lkcl.net/namedpipes/namedpipes-emulation.txt this specification is intended to be implemented as a library, similar to a systems library that provides for example TCP/IP in userspace. thus it could easily be utilised by samba, samba tng, samba 4, apache runtime or in fact any POSIX-compliant system that intends to offer interoperable "Windows NT Named Pipe Emulation". the purpose of the document is to provide a guide to creating emulated implementation of Windows NT Named Pipes. implementors could use this to communicate between clients and servers inside their applications (such as the Wine application), with the added benefit that additional applications (such as the Samba application) that also conformed to the specification could interoperate. For example, the creation of a NamedPipe inside a Samba service could be talked to by a Wine client, and vice-versa. ultimately the specification _could_ potentially be used as a guide to writing a kernel-level implementation of Windows NT Named Pipes emulation. Note that this specification does NOT include "transfer of data over a network", as that task is carried out by SMB clients and SMB servers utilising SMBopenX, SMBclose, SMBreadX, SMBwriteX and SMBtrans2 subcommands 0x1, 0x26 and 0x53 to represent SetNPHS, WaitNamedPipe and TransactNamedPipe. Note that this specification does NOT have the API of the Win32 functions it is intended to emulate but is instead based very much on the design of the SMB protocols (much simplified), and thus provides an exact topological mapping of all SMB operations necessary to provide full Named Pipes functionality. Missing from the specification (but mandatory) at this time is PeekNamedPipe and GetNamedPipeHandleState, because it is clear that the functionality inside of Samba is entirely missing support for such (see call_trans2qpipeinfo in samba 3 smbd/trans2.c source code). Research WILL be required at a later stage in the form of running Win32 test applications, along with examination of the resultant network traffic, in order to determine the necessary support inside Samba for these two vital Named Pipe operations. From the design of Wine, and the stub code in smbd/trans2.c it is likely that these two functions will simply be direct mappings from NtQueryFileInformation #defines onto Trans2_QueryFileInfo, with blindingly obvious parameters and data structures to anyone familiar with both dlls/ntdll/file.c in Wine, and smbd/trans2.c in Samba. l.
Re: http://wiki.winehq.org/NamedPipes - documenting the samba / wine NamedPipes integration
ok, written up as a specification - http://lkcl.net/namedpipes/namedpipes-emulation.txt: please review and provide feedback. kai, volker, you may wish to forward this to the samba technical lists. l.
Re: http://wiki.winehq.org/NamedPipes - documenting the samba / wine NamedPipes integration
On Mon, Feb 16, 2009 at 8:22 AM, Kai Blin wrote: > On Sunday 15 February 2009 19:47:13 Luke Kenneth Casson Leighton wrote: >> i've been updating this page with relevant information that is part >> requirements specification, part documentation. of particular >> relevance is the lack of support (in both tng and samba 3) for >> transferring SetNamedPipeHandleState semantics, although the "stub" >> functionality inside tng (smbd/pipe-proxy.c) at least records the NP >> message-mode flags, and then _completely_ ignores them ha ha :) >> >> exactly how this is to be implemented is unclear, and to be absolutely >> honest, by far the simplest-sounding approach may be to actually >> compile samba up as a win32 app! > > That'd force you to run Wine as root (or using > POSIX capabilities) to be able to bind to < 1024 ports. strictly speaking, just wineserver. and, in order to secure the unix domain sockets, given that you're sending a security context and effectively telling the listener to "seteuid" to that security context (both in NT terminology and also ultimately _really_ telling listeners to do seteuid once you've converted the NT SID to uid,gid using the winbindd built-in SURS mechanism) *deep breath :) * in order to secure the unix domain sockets, you have to run wineserver as root, anyway. side-note: _but_ (spelling this out for the benefit of juan and other wine developers), you only need to do that [run wineserver as root] if you actually start doing nt security contexts. so, if you start off with a named-pipes sockaddr_un emulation design that is neeaarly interoperable with samba, then people can at least experiment, test it, privately run wineserver as root, but public releases always _never_ send that security context etc. etc. > Also, if the box is > also running Samba, we're at the old clash of who owns ports 137-139 and 445. yehh, i knoww. >> given that samba tng has _already_ successfully been ported to win32, >> it has a head start here. and in that regard, ReactOS has it made, >> done and dusted! > > Yes, but ReactOS doesn't run any other smbd, while I'd argue that's relatively > common on Unix machines these days. *thinks*... ok, so.. dropping the "but", the sentence makes more sense to my literal-minded brain: you're saying that because reactos doesn't have posix compliance, and so no smbd to speak of, it's in a better position to choose. which leaves its users in an unfavourable position. the best thing there would be to give steven's samba4 Win32 port a bit higher priority, yes? >> the alternative is... gaahd, i dunno... messy, to say the least, and >> *shudder* an mmap'd tdb is sounding _really_ attractive. > > I'm not sure why you'd want to make assumptions about the database backend the > samba server is using. ah you misunderstand *thinks a bit more* where are you coming from... ok i get it. answer 1: if you have the six namedpipe functions rolled out into a dynamically loadable module (same as what you call VFS layers) then it's up to the dso to choose the implementation of namedpipes. so samba3 / samba 4 uses sockaddr_un, with its own unique dedicated dso, *shrug* you're a user / sysadmin, you want to interoperate with wine? change some config files, replace one component - the namedpipes-emulation-dso, bang, you're done. and in that wine-samba npemu dso, it _happens_ to use an mmap'd tdb, or it _happens_ to use anything-it-likes, and that has _nothing_ to do with samba's internal use or otherwise of tdb, yes? answer 2: i like the sockaddr_un idea much better than an mmap'd tdb anyway :) >> tell me if this sounds reasonable. basically, a mini-wineserver >> protocol - a mini-SMB protocol - is required. commands to "send", >> "receive", "setnamedpipestate", "waitnamedpipestate" and "close". all >> required. >> > [...] >> does this sound like a reasonable proposition - a mini SMB/wineserver >> protocol - for use for inter-communication between wine and samba in >> order to exchange named pipe traffic? > > Is there any reason we can't use plain old RPC for talking to Samba? i can think of several, they're not exactly "absolutely you can't" reasons, more of "yukk" reasons reasons _for_ the idea of using RPC: 1) writing an IDL file npemul.idl would be _great_! int open_npemp([in] NET_USER_INFO_3 *ctx, [in] UNICODE_STRING *pipe_name, [in] int messagemode, ..., [out, handle] *pipehnd); int set_npemuhs([in, handle] *pipehnd, int new_messagemode, ...); int write_npemu([in, handle] *pipehnd, [in, out] *length, [in
Re: http://wiki.winehq.org/NamedPipes - documenting the samba / wine NamedPipes integration
On Sun, Feb 15, 2009 at 9:43 PM, Juan Lang wrote: > Hi Luke, > >> does this sound like a reasonable proposition - a mini SMB/wineserver >> protocol - for use for inter-communication between wine and samba in >> order to exchange named pipe traffic? > > Well, I think we have to crawl before we can walk. :) that's why i said "mini" smb / wineserver protocol, comprising: [2-byte command] auth, waitnphs, setnphs, read, write [parameters] [variable-data] which is incredibly incredibly simple, and would take ... what... 200 lines of code, to write? setting the "auth" data blob to NULL _is_ "crawl before walk". > I think it's more > important that we have a working local named pipe implementation > before we start thinking about SMBs from Wine to a remote machine. follow the chain, here... :) not least, to make sure it sounds reasonable! well, the more i thought about how awful the "multiple socket-pair queue" design is, and the more i think about what's on the MSDN "CreateNamedPipe" page, the more i believe that a per-file-handle Mutex is utterly utterly essential. and if you're going to do that [per-file-handle mutex], then you _can_ use that to emulate blocking-read semantics correctly instead of read(), select(), poll() etc. _that_ means that you can do whatever you like - send headers down the socket, send lengths down the socket, do whatever-you-like-down-the-socket because there will be only one reader. _that_ means you can get away with sending quote commands quotes. the combination of sending "commands" and exclusivity means you don't have to have the multiple socketpair queue and, while you're at it, it so happens that you can replace the socketpair() with a sockaddr_un pair (hooray!) so, you get to write a simplified design that meets the requirements _and_ is well on the way to meeting the other requirements - being able to interoperate with samba. > Usually, having a specific app in mind really helps drive an > implementation, too. i like to think ahead. saves a lot of time and effort in the long run. l.
[patch] review requested: message-mode named pipes
http://bugs.winehq.org/attachment.cgi?id=19477 * functionality of non-message-mode (byte type) is preserved. * putting too many messages into the queue runs wineserver out of file descriptors, but the solutions to that are too much work for this revision, to be included, so - tough. * research indicates that message-mode is supposed to block when resources are low, but that would require a per-file mutex. again - too much work for this revision. l.
Re: [patch but not quite ready] NamedPipes messagemode
On Sat, Feb 14, 2009 at 7:02 PM, Luke Kenneth Casson Leighton wrote: > http://bugs.winehq.org/show_bug.cgi?id=17195 > > with only one significant bug - some memory corruption - the semantics > are now correct in the namedpipe messagemode patch. found it. as suspected, utilising the same critical section fd_cache_section as in ntdll/file/server.c this issue went away. it wasn't memory corruption at all, it was the use of the wineserver socket to send filedescriptors using ioctls: server_get_named_pipe_fd() was picking up a filedescriptor that was intended for server_get_unix_fd() and vice-versa. so - fixed! done! working! ... sort-of :) the one remaining "fly in the ointment", which i'm going to ignore, is demonstrated by threadread and threadwrite tests: running unix out of socketpair file handles. 20 writes by 5 threads are queued up by e.g. threadwrite, resulting in a whopping two _hundred_ filedescriptors being handed out. reading this: http://msdn.microsoft.com/en-us/library/aa365150.aspx describes how use of a NamedPipe should block on write: "The write operation will block until the data is read from the pipe so that the additional buffer quota can be released. Therefore, if your specified buffer size is too small, the system will grow the buffer as needed, but the downside is that the operation will block. If the operation is overlapped, a system thread is blocked; otherwise, the application thread is blocked." ignoring the bollocks about "buffer sizes", and "buffer quotas", which give an indication of how microsoft internally implements NamedPipes, it's clear then that blocking the application is perfectly acceptable. exactly how this is to be achieved is yet to be determined, but as i keep mentioning, it's highly likely that a per-file-handle mutex is required. quick tests: 1) test1, shortread, send2recv2: pass. 2) reducing BUF_SIZE in the tests code to 128 works, threadread and threadwrite: success 3) removing the interim use of ncacn_ip_tcp in SVCCTL and returning to ncacn_np: success. 4) going back to running a simple python2.7.exe "multiprocessing" tests, which use message-mode named pipes, the whole damn _point_ of this exercise: success 5) running MSYS: fail. bugger. a veery cursory look at the output, it looks like it's a non-message-mode pipe. arg. no surprise there, then. *sigh*. will let you know how i get on. l.
[patch but not quite ready] NamedPipes messagemode
http://bugs.winehq.org/show_bug.cgi?id=17195 with only one significant bug - some memory corruption - the semantics are now correct in the namedpipe messagemode patch. i drew a diagram outlining the data structures / design http://bugs.winehq.org/attachment.cgi?id=19449 because it's so horrendous that even i want to throw up. assistance in finding the memory corruption would be really helpful. an assert is thrown after more than 4 short reads are performed on the same namedpipe, with this strnge error: ead: file.c:631: server_get_named_pipe_fd: Assertion `wine_server_ptr_handle(fd_handle) == handle' failed. ead: server.c:628: server_get_unix_fd: Assertion `wine_server_ptr_handle(fd_handle) == handle' failed. wine: Assertion failed at address 0xf7ca8d7f (thread 000f), starting debugger... err:ntdll:RtlpWaitForCriticalSection section 0x7bc99ee0 "server.c: fd_cache_section" wait timed out in thread 000f, blocked by 0014, retrying (60 sec) i'll try to write a simplified test because that's the "threadread" test on which this occurs. i can't quite believe that all 5 tests can be passed [by reducing the number of short reads to less than 4]. it's been a bit of a "drunken walk" development style, i'm afraid :) notes: * in order to preserve message-mode semantics correctly, each message gets ITS OWN socketpair. the socketpair is used EXCLUSIVELY for that one message and is NOT re-used by the other end. so if it's client writing to server, one message, that's one socketpair. if it's server writing to client, one message, that's another socketpair. if client writes 20 messages to server, that's 20 new socketpairs, if at same time server writes 25 messages to client, and nobody's done any reading yet, that's a total of 45 outstanding socketpairs - 90 filedescriptors. this is not a hoax. it's the only way i could think of that preserves messagemode semantics correctly. * for each end - client and server - to be able to "block", waiting at any time, there MUST be at LEAST one socketpair in each of the queues. so, initially, that's what's done: one pair of socketpairs are created, one for the client, one for the server. on the FIRST write (to server, or to client), this initial "empty" socketpair must be written to; subsequent writes (to server or to client, each) must have a NEW socketpair created. * on a read, a socketpair must "hang about" - MUST not be shut down - until there is a new one created (which happens only on a new write). in this way, multiple threads will always have a filedescriptor to block on. * when a read has read all available data, the socketpair is marked as "finished" but is NOT shut down. * on each write, if a new socketpair is created, then and ONLY then can "old" marked socketpairs be shut down. * shutdown of read-sockets, on which [potentially many] reader[s] are waiting will result in an EPIPE. this is an ACCEPTABLE error IFF no data has been read yet, i.e. at the start of a new message. what happens, basically, is that the "old" socketpair - even though it had been used already by the previous reader - is what the thread "blocks" on (because you have to block on _something_). and on the _very_ first message, of course, you block on the specially-prepared initial socketpair, the one that had to be created and marked as "as-yet unwritten to". * you can see at line 872 of ntdlls/file.c (after patch is applied) that oh whoopsie, EPIPE occurred, but we've not read any data at all, just grab another filedescriptor from wineserver and try again. in this way, wineserver hands out the next available message from the filedescriptor queue. if yet _another_ EPIPE occurs, then all that means is that another thread "got there first", wow big deal, around you go again. * indicating that data has been read is performed using a special wineserver message "NAMED_PIPE_SUB_AVAILDATA" using server_set_named_pipe_info(). this horrible hack passes information to wineserver telling it to subtract the amount of data read in userspace from the filedescriptor data structure (fdqueue) representing that socketpair. * indicating that data _will_ be available is performed again using server_set_named_pipe_info(), this time using a message "NAMED_PIPE_SET_AVAILDATA". the set is performed BEFORE the actual write, otherwise a reader will "unblock" and potentially try to set the available data to negative!!! this would be baaad. * a critical section is performed around all uses of server_set_named_pipe_info() and server_get_named_pipe_fd(). this is NOT the same critical section as server_get_handle_fd(). i don't know if this makes any difference. things that cannot be done: * you cannot do the read or the write inside wineserver (server_named_pipe_read() will be deleted, later). the reason is complex to describe, but boils down to "blocking in userspace on the socket in order to wait for a message" and "non-blocking read on the same socket in order to get the data" are unfortunately
http://wiki.winehq.org/NamedPipes - documenting the samba / wine NamedPipes integration
i've been updating this page with relevant information that is part requirements specification, part documentation. of particular relevance is the lack of support (in both tng and samba 3) for transferring SetNamedPipeHandleState semantics, although the "stub" functionality inside tng (smbd/pipe-proxy.c) at least records the NP message-mode flags, and then _completely_ ignores them ha ha :) exactly how this is to be implemented is unclear, and to be absolutely honest, by far the simplest-sounding approach may be to actually compile samba up as a win32 app! under such circumstances, it would be incredibly easy to then call the wine-emulated NamedPipe win32 functions CreateNamedPipe, SetNamedPipeHandleState etc. etc. given that samba tng has _already_ successfully been ported to win32, it has a head start here. and in that regard, ReactOS has it made, done and dusted! the alternative is... gaahd, i dunno... messy, to say the least, and *shudder* an mmap'd tdb is sounding _really_ attractive. tell me if this sounds reasonable. basically, a mini-wineserver protocol - a mini-SMB protocol - is required. commands to "send", "receive", "setnamedpipestate", "waitnamedpipestate" and "close". all required. * samba 3 / tng create or access named pipes as emulations over unix domain sockets, including sending a security context. in samba 3, that's called a SamInfo3, but it looks suspiciously like a NETLOGON NET_USER_INFO_3) and in tng it's a NET_USER_INFO_3... so... yep, it looks like just the name has been changed. * a program wants to create or access a named pipe; it asks wineserver (via a new function) to create or open up a unix domain socket. the locations will be the same. i.e. tng / samba3's unix domain socket pipe locations same as wineserver's. * sending / receiving of the security context blob is done in the dlls NOT in wineserver, so as to avoid blocking wineserver. * read, write, snphs, wnphs are all done with a 2-byte command prefix, followed by parameters, followed by data. exactly the same as is done in SMB and in wineserver. ... you know... if SMB wasn't so damn complicated i'd say bugger it, just send the SMBs to Wine, for _it_ to sort out!! in fact, with cliffsd's auto-generated SMB parsing code, that might even still be a valid option, given that the relevant code for parsing the SMBs is public domain. what do people think? does this sound like a reasonable proposition - a mini SMB/wineserver protocol - for use for inter-communication between wine and samba in order to exchange named pipe traffic? l.
Re: [patch but not quite ready] NamedPipes messagemode
> 5) running MSYS: fail. found it: rxvt.exe is looking for libX11.dll (!!) which is nothing to do with NamedPipes. running c:/msys/bin/sh.exe --login -i is: success.
Re: [patch but not quite ready] NamedPipes messagemode
On Sat, Feb 14, 2009 at 7:02 PM, Luke Kenneth Casson Leighton wrote: > http://bugs.winehq.org/show_bug.cgi?id=17195 > > with only one significant bug - some memory corruption - the semantics > are now correct in the namedpipe messagemode patch. found it. as suspected, utilising the same critical section fd_cache_section as in ntdll/file/server.c this issue went away. it wasn't memory corruption at all, it was the use of the wineserver socket to send filedescriptors using ioctls: server_get_named_pipe_fd() was picking up a filedescriptor that was intended for server_get_unix_fd() and vice-versa. so - fixed! working! ... sort-of :) the one remaining "fly in the ointment", which i'm going to ignore, is demonstrated by threadread and threadwrite tests: running unix out of socketpair file handles. 20 writes by 5 threads are queued up by e.g. threadwrite, resulting in a whopping two _hundred_ filedescriptors being handed out. reading this: http://msdn.microsoft.com/en-us/library/aa365150.aspx describes how use of a NamedPipe should block on write: "The write operation will block until the data is read from the pipe so that the additional buffer quota can be released. Therefore, if your specified buffer size is too small, the system will grow the buffer as needed, but the downside is that the operation will block. If the operation is overlapped, a system thread is blocked; otherwise, the application thread is blocked." ignoring the bollocks about "buffer sizes", and "buffer quotas", which give an indication of how microsoft internally implements NamedPipes, it's clear then that blocking the application is perfectly acceptable. exactly how this is to be achieved is yet to be determined, but as i keep mentioning, it's highly likely that a per-file-handle mutex is required. quick tests: 1) test1, shortread, send2recv2: pass. 2) reducing BUF_SIZE in the tests code to 128 works, threadread and threadwrite: success 3) removing the interim use of ncacn_ip_tcp in SVCCTL and returning to ncacn_np: success. 4) going back to running a simple python2.7.exe "multiprocessing" tests, which use message-mode named pipes, the whole damn _point_ of this exercise: success 5) running MSYS: fail. bugger. a veery cursory look at the output, it looks like it's a non-message-mode pipe. arg. no surprise there, then. *sigh*. will let you know how i get on. l.
[patch but not quite ready] NamedPipes messagemode
http://bugs.winehq.org/show_bug.cgi?id=17195 with only one significant bug - some memory corruption - the semantics are now correct in the namedpipe messagemode patch. i drew a diagram outlining the data structures / design http://bugs.winehq.org/attachment.cgi?id=19449 because it's so horrendous that even i want to throw up. assistance in finding the memory corruption would be really helpful. an assert is thrown after more than 4 short reads are performed on the same namedpipe, with this strnge error: ead: file.c:631: server_get_named_pipe_fd: Assertion `wine_server_ptr_handle(fd_handle) == handle' failed. ead: server.c:628: server_get_unix_fd: Assertion `wine_server_ptr_handle(fd_handle) == handle' failed. wine: Assertion failed at address 0xf7ca8d7f (thread 000f), starting debugger... err:ntdll:RtlpWaitForCriticalSection section 0x7bc99ee0 "server.c: fd_cache_section" wait timed out in thread 000f, blocked by 0014, retrying (60 sec) i'll try to write a simplified test because that's the "threadread" test on which this occurs. i can't quite believe that all 5 tests can be passed [by reducing the number of short reads to less than 4]. it's been a bit of a "drunken walk" development style, i'm afraid :) notes: * in order to preserve message-mode semantics correctly, each message gets ITS OWN socketpair. the socketpair is used EXCLUSIVELY for that one message and is NOT re-used by the other end. so if it's client writing to server, one message, that's one socketpair. if it's server writing to client, one message, that's another socketpair. if client writes 20 messages to server, that's 20 new socketpairs, if at same time server writes 25 messages to client, and nobody's done any reading yet, that's a total of 45 outstanding socketpairs - 90 filedescriptors. this is not a hoax. it's the only way i could think of that preserves messagemode semantics correctly. * for each end - client and server - to be able to "block", waiting at any time, there MUST be at LEAST one socketpair in each of the queues. so, initially, that's what's done: one pair of socketpairs are created, one for the client, one for the server. on the FIRST write (to server, or to client), this initial "empty" socketpair must be written to; subsequent writes (to server or to client, each) must have a NEW socketpair created. * on a read, a socketpair must "hang about" - MUST not be shut down - until there is a new one created (which happens only on a new write). in this way, multiple threads will always have a filedescriptor to block on. * when a read has read all available data, the socketpair is marked as "finished" but is NOT shut down. * on each write, if a new socketpair is created, then and ONLY then can "old" marked socketpairs be shut down. * shutdown of read-sockets, on which [potentially many] reader[s] are waiting will result in an EPIPE. this is an ACCEPTABLE error IFF no data has been read yet, i.e. at the start of a new message. what happens, basically, is that the "old" socketpair - even though it had been used already by the previous reader - is what the thread "blocks" on (because you have to block on _something_). and on the _very_ first message, of course, you block on the specially-prepared initial socketpair, the one that had to be created and marked as "as-yet unwritten to". * you can see at line 872 of ntdlls/file.c (after patch is applied) that oh whoopsie, EPIPE occurred, but we've not read any data at all, just grab another filedescriptor from wineserver and try again. in this way, wineserver hands out the next available message from the filedescriptor queue. if yet _another_ EPIPE occurs, then all that means is that another thread "got there first", wow big deal, around you go again. * indicating that data has been read is performed using a special wineserver message "NAMED_PIPE_SUB_AVAILDATA" using server_set_named_pipe_info(). this horrible hack passes information to wineserver telling it to subtract the amount of data read in userspace from the filedescriptor data structure (fdqueue) representing that socketpair. * indicating that data _will_ be available is performed again using server_set_named_pipe_info(), this time using a message "NAMED_PIPE_SET_AVAILDATA". the set is performed BEFORE the actual write, otherwise a reader will "unblock" and potentially try to set the available data to negative!!! this would be baaad. * a critical section is performed around all uses of server_set_named_pipe_info() and server_get_named_pipe_fd(). this is NOT the same critical section as server_get_handle_fd(). i don't know if this makes any difference. things that cannot be done: * you cannot do the read or the write inside wineserver (server_named_pipe_read() will be deleted, later). the reason is complex to describe, but boils down to "blocking in userspace on the socket in order to wait for a message" and "non-blocking read on the same socket in order to get the data" are unfortunately
Re: rpcrt4: Check for null endpoint in RpcServerUseProtseqEpExW
see implementation in dual BSD-licensed _and_ LGPL-licensed freedce reference implementation, on which MSRPC was originally blatantly copied (thanks to the BSD license on the 1.0 reference implementation). e.g. online annotated copy here: http://www.sfr-fresh.com/linux/misc/freedce-1.1.0.7.tar.gz:a/freedce-1.1.0.7/ncklib/com/comnet.c line 2470. anyone who is doing wine dce/rpc development _really_ needs to be looking at the reference implementation, side-by-side, line-for-line, in virtually every aspect except for exception handling, threading, binary-interoperability with the typelibraries and a few other areas. l.
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/smbsrv/ndl/
holy cow. opensolaris appears to have its own smb server / idl compiler: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/tools/ndrgen/ CDDL licensed, unfortunately (listed as incompatible with GPL. darn). but... dang :)
Re: [patch] rpcrt4_conn_np_read needs to expect and accept ERROR_MORE_DATA
On Fri, Feb 6, 2009 at 12:35 PM, Juan Lang wrote: > Hi Luke, > > @@ -382,7 +382,14 @@ static int rpcrt4_conn_np_read(RpcConnection *Connection, > Style nit: it would be simpler to do: > DWORD bytes_read; > ret = ReadFile(npc->pipe, buf, bytes_left, &bytes_read, NULL); > +if (!ret && GetLastError() == ERROR_MORE_DATA) > +ret = TRUE; > if (!ret || !bytes_read) > break; > > Don't you think? ... oh yeah :) well, i wanted to know what was going on, so i added the TRACE. rob, this is clearly much better. oh - the nice thing that you should find is that given that the current wine NtReadFile() implementation never returns ERROR_MORE_DATA, is that all checks that you add to treat ERROR_MORE_DATA as "not actually an error but actually to be expected" should have _zero_ impact. until NamedPipes messagemode gets fixed, of course. l.
Re: http://wiki.winehq.org/NamedPipes
> Funny enough, the Samba project codenamed "franky" > (http://wiki.samba.org/index.php/Franky) will be doing all of this. It's not > quite there yet, but after some nice work done in the Samba3 smbd by Volker > Lendecke and some more work in the Samba4 "samba" daemon by Stefan > Metzmacher, Jelmer Vernooij and some overall effort by other folks from the > Samba Team, franky is coming along quite well. http://wiki.samba.org/index.php/Franky/Notes_20080508 how veery iiinteresting. i see that the justification for adding proper named pipe break-out isn't to do with inter-project cooperation, at all - it's to provide a samba3 - samba4 migration path, utilising components from each. by fortunate coincidence, it so happens that the exact same reasons and near-exact same architecture as required for samba3-samba4 interoperability is exactly what's required for every other project that also wants to use samba as a service not as _the_ be-all-and-end-all service. just make sure you add an SMB SetNamedPipeHandleState function to the list of callbacks / capabilities, not just createpipe, readpipe, writepipe, transactnamedpipe and closepipe. in samba, the _only_ operation you will ever see on-wire, is SetNamedPipeHandleState to 0x3 (PIPE_MESSAGE_READ | PIPE_MESSAGE_WRITE). but other systems will want to use franky-breakout read, franky-breakout write, franky-breakout transactnamedpipe in NON-message-mode. and also make sure that the namedpipestate flag on the initial SMBcreateX is also passed over to the franky-breakout-create-pipe function. oh, and a franky-breakout-get-namedpipe-state too. which probably maps to an SMB IPC$ ioctl somewhere. needs analysis. rob, you'll want to put these on your list of things to keep an eye on: 1) franky-create-pipe receives PIPE_MESSAGE_MODE flags on/off 2) franky-set-namedpipe_handle_state function exists 3) franky-get-namedpipe-handle-state function has to exist somewhere. all of them will be necessary for the "existing" wine apps to work correctly over-the-wire because that's what wine presently only supports: non-message-moded named pipes. l.
Re: [patch] rpcrt4_conn_np_read needs to expect and accept ERROR_MORE_DATA
On Fri, Feb 6, 2009 at 10:29 AM, Rob Shearman wrote: > 2009/2/5 Luke Kenneth Casson Leighton : >> http://bugs.winehq.org/show_bug.cgi?id=17263 > > The patch looks acceptable in theory. Please email the patch to > wine-patches (or wine-devel) and I'll review the implementation of it. ok!
Re: ideas and questions for implementation of MessageMode in Named Pipes
On Thu, Feb 5, 2009 at 6:38 PM, Juan Lang wrote: >> and no, you _can't_ do "everything in wineserver", because you _still_ >> need a mechanism to be able to tell the client (ntdll) to "block". > > Well, if there are two fds still, but each end is either a client > (reading or writing) and the other is the wineserver, I believe you > could. that was "the plan" > That is, the server would only write complete messages to a > fd, presumably prepended with the length. that would be fine, if it wasn't for one thing: i've seen NT send _eight megabytes_ of data down a named pipe, on a spoolss service. i tried it out, once, with rpcclient. the limit (on NT 4.0) was around 8 mb of PDUs. now, what i _can't_ tell you is whether that eight megabyte limit was in the RPC subsystem (or if it was in the SMB-NP subsystem), with the MSRPC infrastructure breaking it down into PDU-sized chunks [of MTU-size e.g. 1500] and i certainly can't tell you whether those PDUs were put into a single 8mb buffer that was then handed to a single NtWriteFile() call on a Named Pipe, or if it was done on a per-pdu basis but, ultimately, i would be _very_ wary of getting wineserver to malloc exactly the amount of memory that it was told was being written down the pipe. not least is the issue that any such large mallocs or writes would put a _significant_ latency onto the response time of other wineserver calls (which is something i wanted to raise as a issue, separately. another time) so, i'm doing it as a loop, and... *thinks*... i like the queue of "struct fd"s more than i like the double-socket idea, the more i think about it. > A reading client would then > block if there weren't any data to read, and there wouldn't be any > data until a complete message were written there by wineserver. yes. this would be _very_ nice. ... except... what do you do when there is another message being written? NtWriteFile (puts the complete message into the fd) NtWriteFile (what do you do now? you can't block the writer) NtReadFile (gets the first complete message) NtReadFile (what do you do now?) this is why having a queue of "struct fd"s, one per message, could end up being the only viable solution. >> which is why i asked if there was a way for wineserver to tell a >> client wine thread/process to "go to sleep" [and didn't get an >> answer]. > > Right. There isn't any such mechanism, at least as far as I know. bugger :) mrrm... i encountered async_queue - that looks suspiciously like it's a wake-up mechanism. >> _somewhere_ there has to be a "break" between "messages". >> >> and you can't stop people from sending data down the pipe (in NtWritePipe). >> >> therefore, logically, you have to have a "firewall" between "send" and >> "receive". and, because you _also_ need "blocking on read" >> characteristics (across multiple processes and threads), the most >> sensible way to implement that is with a unix filedescriptor on which >> all of the clients (ntdll) can block. > > Yep, agreed. This discussion is more about what the structure of the > file descriptors is. But in general, my opinion here doesn't carry > much weight, Alexandre's does ;-) And he's unlikely to voice an > opinion except on patches. ha ha :) anyone who helps crystallise ideas is valuable. >> message-mode namedpipes falls into that category, which is why i'm doing it. > > And we appreciate your attention to it. Really, we do :) thanks :) >> so - once i have _a_ working implementation, then please do not be >> offended, or surprised, if, when asked to make further enhancements, >> or if asked to fit specific criteria (such as doing things a "slightly >> different way"), i decline to do so. > > Yep, that's been my assumption all along. That's why I've nudged you > with respect to tests a couple of times. Thanks for putting some in > the bug, by the way. no problem. there will need to be more - a lot more. e.g. PeekNamedPipe actually does an optional read, and _then_ tells you what's left. wine's implementation, because nobody ever really _does_ that "optional read", they only do the "peek". > Since I assumed it was unlikely you'd have the > patience to work with AJ to get your patches accepted, I thought > having some test cases, along with an implementation that isn't > fundamentally flawed, would help point us in the right direction. ... well... now i'm... deeply impressed (with your foresight). > Pointing out that handles can be shared between processes was a hint > about one fundamental flaw that can trip you up. mrmmrmmm *grumble-about-how-nt-works* yeah that was appreciated, although it won't be until i do some tests using dup on handles, or simply blatantly passing the 32-bit handle pointer value between processes (over a socket or a file), that i'll know what is and isn't possible. > I'll try to look at > what you're doing now and again to point out things that just can't > work. he he. appreciate it. > For the most
Re: ideas and questions for implementation of MessageMode in Named Pipes
On Thu, Feb 5, 2009 at 4:08 PM, Juan Lang wrote: >> i think i've finally come up with an idea that i believe will work: >> double-socketing. > (snip) >> it'll be ok (and desirable) to allow multiple "readers" of the "read" >> socket. what you _don't_ want is more than one reader trying to >> indicate "please start sending a new message" whilst there are other >> reader(s) still grabbing the previous message. >> >> so i believe that a critical section (copying the style of the code >> around server_get_unix_fd) each around "please start a new message" >> and "please send some more read-data" would be sufficient. > > Out of curiosity, why is this better than a single socket with the > length of each message prepended? a length prepended is still needed. what i had implemented so far (and demonstrated that it's flawed, thanks to the "threadread" test) is: * writes send length-prepended to data * wineserver-function get_named_pipe_info * wineserver-function read_named_pipe then: * in the client (ntdll) instead of using read() you use server_read_named_pipe() iff it's a pipe. BUT, before you do that, you do a poll() on the unix_handle (obtained using server_get_unix_fd()). this is your "blocking mode". so, although you tell the _server_ to get the data for you, you _still_ have to do "block on socket". and, because you can't use read() to do the "blocking", you have to use poll() instead. * in the wineserver-function, read_named_pipe MUST NOT EVER block on reads, but it is still being asked to perform _a_ read, and so there is some code that sets O_NONBLOCK, followed by a recv using MSG_PEEK to double-check that there's data, and _then_ a read is used to actually obtain the data. the problem here is that setting O_NONBLOCK in wineserver, in order to not end up permanently hanging wineserver, is *interfered with* by the requirement to have "blocking mode" in the client (ntdll). so there are two conflicting requirements, by using the same filedescriptor. and no, you _can't_ do "everything in wineserver", because you _still_ need a mechanism to be able to tell the client (ntdll) to "block". which is why i asked if there was a way for wineserver to tell a client wine thread/process to "go to sleep" [and didn't get an answer]. > One difficulty I'm having trouble getting my head around is, isn't the > data removed from a socket once it's ready by any process? or a thread - yes. fortunately, read() and write() to/from sockets are at least atomic (whew). > Or does it > remain for each process to read independently? no, thank god. the data is removed. except if you use recv() with MSG_PEEK, of course, but then the data is _guaranteed_ not to be removed. ever. > I guess I'm not that > familiar with how reading from a socket works. I'd always assumed > that the former was true, and that therefore the only correct approach > would be to buffer message-mode named pipe data in the server. "relax, luther - it's much worse than you think" [Mission Impossible I] :) > This > is ugly (and slow), well... it's unfortunate, but that's the way it's going to have to be. if i was told "there's these absolutely fantastic guaranteed-message-size, guaranteed-message-order characteristics you can get from message-mode named pipes, but they're a bit slower than normal pipes" when developing an application i'd go "great! i don't care if it's a bit slower, the features make my life a _lot_ easier". _somewhere_ there has to be a "break" between "messages". and you can't stop people from sending data down the pipe (in NtWritePipe). therefore, logically, you have to have a "firewall" between "send" and "receive". and, because you _also_ need "blocking on read" characteristics (across multiple processes and threads), the most sensible way to implement that is with a unix filedescriptor on which all of the clients (ntdll) can block. _but_.. if you also allow any _other_ process to send data down the _same_ unix filedescriptor (from NtWritePipe), you've just gone and screwed with the message-boundaries. ... actually, it may turn out to be the case that you literally need one filedescriptor _per message_. not joking about, or anything, but it may end up being the case that a queue of "struct fd*" pointers is required (in wineserver). in non-message-mode, that would simply be "queue length of 1". in message-mode, you'd go "oh, dear, we got an EPIPE error when trying to read, that means that someone else got that message, wow big deal, let's grab another unix filedescriptor from wineserver and try again". if grabbing another filedescriptor fails, THEN you go "oh, whoops, let's return STATUS_PIPE_DISCONNECTED". by having one filedescriptor per message, you are cast-iron guaranteeing 100% that individual messages will not interfere with each other. and, the neat thing is, you wouldn't need any "buffering". you'd still need to indicate the length (and the easiest way to do that is _still_ to se
[patch] rpcrt4_conn_np_read needs to expect and accept ERROR_MORE_DATA
http://bugs.winehq.org/show_bug.cgi?id=17263
hooray! very simple fix, here - allow ERROR_MORE_DATA status code as
an acceptable error.
... and _check it_, afterwards.
ideally, the loop to read a complete PDU should be vaguely like this:
do {
RpcConnection conn->ops->read(data, &len)
total_read += len;
if GetLastError() == ERROR_MORE_DATA
if total_read == total_packet_size
return FALSE; /* not ok to return ERROR_MORE_DATA at end of PDU!
} while (total_read != total_packet_size);
i.e. you should _expect_ there to be an ERROR_MORE_DATA status code on
all but the _very_ last bit of the PDU. but, you can't do that in
rpcrt4_conn_np_read() because that function is just a "data getter",
it doesn't know when it's being used to grab the last bit of the PDU.
this will prove to be important for catching problems, both internally
and also when SMBs are used. _yes_ the SMB system also returns
ERROR_MORE_DATA. actually, if NT STATUS codes are enabled, it's
STATUS_MORE_PROCESSING_REQUIRED which _do_ go over-the-wire if the smb
server negotiates their use, but that's another story.
l.
Re: ideas and questions for implementation of MessageMode in Named Pipes
On Thu, Feb 5, 2009 at 1:34 AM, Juan Lang wrote: > Hi Luke, hi juan, thanks for responding. >> how about: ripping out the use of unix-pipes altogether, and replacing >> them with tdb (trivial database) in a mmap'd file? the nice thing > The usual response around here is, show us some patches, and we'll let > you know ;-) yehh, ha ha :) i was kinda hoping to not have to go through five different implementations! i think i've finally come up with an idea that i believe will work: double-socketing. very simple: you write down one filedescriptor, and read from a different one. wineserver proxies the data from one to the other, whenever it's requested. here's the important bit: wineserver ONLY allows ONE lot of data into the "read" socket at any one time. it'll be ok (and desirable) to allow multiple "readers" of the "read" socket. what you _don't_ want is more than one reader trying to indicate "please start sending a new message" whilst there are other reader(s) still grabbing the previous message. so i believe that a critical section (copying the style of the code around server_get_unix_fd) each around "please start a new message" and "please send some more read-data" would be sufficient. what do you think? also the advantage of double-socketing is that when it comes to doing SMB named pipes there will be a clean "way in and out" for SMBreadX, SMBwriteX and SMBtransactNamedPipe data over the IPC$ share. the double-sockets will still be needed (one for reading, one for writing) - a _third_ socket will be needed, to communicate with the smb server (via the break-out mechanism). > Some test cases showing us where we're deficient would be a great start, btw. all there, in #17195. five so far. * test1 (the first i wrote) just does send recv send recv. this one works (in existing wine) * send2recv2 does send send recv recv and it's this one that shows _immediately_ that there's a problem with wine's messagemode code. * shortread demonstrates the "standard" way to be able to read blocks of data when the message is _way_ beyond acceptable incoming buffer sizes (e.g. in smbs it was often the case that spoolss would result in SMBwriteX SMBwriteX SMBwriteX SMBtransactnamedpipe SMBreadX SMBreadX SMBreadX transferring _whopping_ great sized messages). again, wine keels over on this one as it doesn't know what to do at the message boundaries. * threadread demonstrates several messages being queued followed by a number of threads performing simultaneous reads (deliberately using shortreads where the individual reads are fixed sizes). i only have this doing 1 thread at the moment because the data needs to be "reconstructed" properly otherwise (even on nt). * threadwrite demonstrates several messages being written simultaneously, with a single process grabbing them all. this one _doesn't_ have to be limited in the number of threads that perform writes, which indicates in the internal design of NT that the writes are atomic.
Re: messagemode named pipes investigation: blocking-reads block writers indefinitely
> main process: blocking-read on namedpipe. > 5 threads: write to same named pipe > > the writes NEVER return (this is with xp). > > so that WOULD indicate that there IS a per-pipe mutex (and that there > are bugs in nt!) unbelievable. turns out it's an msvcrt bug (in windows nt). if you use _beginthread() to create the 5 threads (or even just the one), the reader can block the writer(s). however if you use CreateThread(), everything's hunky-dory. anyone want to record _that_ one as a future interoperability bug-test, be my guest! ha ha :) no, but seriously: if someone tells me it's a worthwhile thing to have as an interoperability test, i'll write one. l.
messagemode named pipes investigation: blocking-reads block writers indefinitely
fascinatingly strange observation came out of writing the latest test (threadwrites) to do namedpipe interoperability testing. main process: blocking-read on namedpipe. 5 threads: write to same named pipe the writes NEVER return (this is with xp). so that WOULD indicate that there IS a per-pipe mutex (and that there are bugs in nt!) l.
Re: ideas and questions for implementation of MessageMode in Named Pipes
mwhaaahahah, i just came up with a _horrible_ idea :) how about: ripping out the use of unix-pipes altogether, and replacing them with tdb (trivial database) in a mmap'd file? the nice thing about tdb is that it's LGPL'd. the messages could be saved and transferred via shared memory; tdb is multi-readable / multi-writeable, and it would save a bitch-awful job of implementing a shared memory sockets emulation layer (which i _have_ seen done, once). l.
ideas and questions for implementation of MessageMode in Named Pipes
ok, alexandre: i tried moving named_pipe_read into wineserver - it's not possible to do completely, as you cannot do blocking-reads in wineserver but you still need blocking-read characteristics in the client (kernel32, ntdll). if you start messing with the fd, setting or clearing ioctl O_NONBLOCK in one, it interferes with the other. there's a race-condition risk. so, i have some ideas and questions. 1) is it possible for wineserver to tell a thread to wait indefinitely? is there a communication channel where wineserver can tell "anyone who's interested" to go away, go to sleep, i'll tell you when to wake up"? what i want is to replace the fd blocking-read with something that still blocks, but doesn't interfere with the file descriptor. 2) would a _second_ filedescriptor fit the required criteria, as described in 1), above? in other words, if the message-lengths (4 bytes) were sent on a secondary control fd - and nothing _but_ message-lengths were sent on that fd - my intuition tells me that that could work, even without having any infrastructure in wineserver. in fact, i get the feeling that it would be _better_ - and a lot simpler - than having any infrastructure in wineserver. the reasoning goes roughly something like this: * 5 threads are waiting - blocking - on a message-mode read named pipe. all of them are blocking to read from the "control" channel, in userspace (kernel32/ntdll). * client 1 does a message-mode write, and in userspace (kernel32/ntdll) 4 bytes indicating the length of the message are sent, followed by N bytes on the unix_fd (exactly as is done at the moment). * one and ONLY one of the 5 threads, thanks to STANDARD linux kernel scheduling, happens to be awake and gets to read the 4-byte length. all the other 4 stay asleep. damnit, that's not right, is it. you want the other 4 to wake up once ONE of them has read the 4-byte length, and for ALL of them to move on to blocking on the "message" pipe - the actual content. so, i suppose what you could do is have a per-read-thread filedescriptor (!!!) which _would_ have to involve arbitration by wineserver, so that when a write is performed, ALL of the clients get notified "wake up now, start trying to read". is there _anything_ in wine which already exists that does what is required, as illustrated, above? answers and information greatly appreciated so as to be able to improve this free software project's effectiveness. l.
Re: [patch] review requested - NamedPipe Message mode (was Reliable Datagram / Stream behaviour of NT Pipes)
> so, adding server_named_pipe_read() avoids this issue by doing locking > (like server_get_unix_fd()) does, i see) $ git commit -a -m '#17185 - server-based read_named_pipe. does blocking in client and non-blocking reads (using recv MSG_PEEK) in the server. messy as hell.' says it all, really :) bit of a marathon session that one. break-time. more in a couple of days. anyone want to take over, email me, you can have what i've got so far. blocking checks client-side (using select) are made on the unix-pipe fd to ensure that there's _actually_ data that wineserver can get at. after reading the message-length (done only once per message duh), read() is replaced by server_named_pipe_read() - still in the for (;;;) loop. alexandre, would it be reasonable to put a CriticalSection around the [optional message-length-reading and] for-loop? i really can't see any other way. what i've done is protect the server from blocking on a read(), by making sure that recv(MSG_PEEK) - actually select() might be better - checks the length. that means however that you still, in kernel32, in ReadFile(), still have to do the for() loop, but preceding each read by a blocking wait (i clear O_NONBLOCK and do an infinite/blocking select()) it all screams "CriticalSection" protection - on a per-handle basis. anyone got any better ideas? let me know, i'll come back to this in a couple of days. l.
Re: [patch] review requested - NamedPipe Message mode (was Reliable Datagram / Stream behaviour of NT Pipes)
On Sun, Feb 1, 2009 at 9:29 AM, Alexandre Julliard wrote:
> Luke Kenneth Casson Leighton writes:
>
>>> You can't do that stuff on the client side. You either have to do all
>>> pipe I/O in the server, or add named pipe support in the kernel. The
>>> latter is harder, but would be much more useful.
>>
>> well, not entirely knowing the difference, i'm guessing that i'm
>> adding named pipe support in the kernel. starting from
>> SetNamedPipeHandleState and going from there.
>
> I mean the Linux kernel.
ohhh, ok. yeah, you're right, it would be _incredibly_ useful to
have message-moding / named-pipes in the linux kernel. i do wonder
though if the linux developers would actually appreciate its
significance - it's "not uuunix", after all. mind you, they _did_
grok the change-notify stuff that was inspired by samba's
requirements, so maybe.
... but anyway, i'll leave that one to you [dear wine developers] to
deal with, as the main linux kernel developers have no respect for
anything i say. it'll have to come from you. i'll help all i can in
designing and reviewing such a feature, but the actual proposal - and
liasing with the linux kernel developers - will have to come from you.
also, for running wine on non-linux-based (or
non-in-kernel-message-mode-supporting unixen) you'd still need
"emulation" anyway.
>> comparing what i've got so far in the improvements to ReadFile,
>> WriteFile, NtFsControlIo etc. with the way that it is believed that
>> this _should_ be done, what is right and what is wrong?
>
> It's all wrong, because this can't be done on the client side (kernel32
> and ntdll are the client side, as opposed to the wineserver side).
ok, understood.
> The
> client side is the equivalent of user space, with the wineserver being
> the equivalent of kernel space. You can't do this stuff in user space
> (hint: concurrency).
lemme think about that, see if i can write a multi-process or
multi-threaded test that makes what i've got here blow up. any help
there greatly appreciated. multiple connections to named pipes are ...
a bit... odd! the designers forgot to add in the equivalent of
listen(), accept() and friends.
question: the existing code just uses server_get_unix_fd() followed
by read-and-write on the underlying unix domain socket / unix-pipe
(whatever). obviously, to emulate message-mode, a read is now
potentially _two_ read operations: one of 4 bytes to get the
length-field, and the other read(s) will get the data.
i take it that you're concerned that there will be a race condition
where two threads reading from the same handle might do _two_ 4-byte
reads, one correct and the other reading "data" as if it was a
length-field?
so, adding server_named_pipe_read() avoids this issue by doing locking
(like server_get_unix_fd()) does, i see)
well... you see... there could well be a problem with the existing
code, on concurrency, because reads and writes can be broken down into
unix-specific default buffer-reading/writing sizes (both read and
write do a for (;;) { ... read/write } loop
oh - update: http://bugs.winehq.org/attachment.cgi?id=19145
this one passes all three tests.
the rather unfortunate thing is: it has knock-on ramifications for any
code *inside* wine that uses NamedPipes, relies on
SetNamedPipeHandleState() working correctly!
so, although the tests are passed, the bigger "test" - running wine -
it's a bit... mair :) how the heck does wine work _without_
messagemode on named pipes, anyway?? surely there's something keeling
over, somewhere.
which brings me on to another quite important question: are there any
work-arounds that have been added into rpcrt4 or any other location
which says "yes i do messagemode" and then goes "but actually, we know
it doesn't work so we do xyz instead"?
btw, i'm dead impressed with the state of the functionality of svcctl,
and its idl file. if you've got svcctl working over ncacn_np then
that's a _really_ good sign that the MSRPC interoperability is up to
scratch (svcctl has those looovely variable-sized extensible
structs...) - and a damn good indication that you stand a really good
chance of being able to make this "networked", replacing the
stub-services in samba (the ones that i kept getting told "have no
meaning" to a unix system). in combination with the break-out
mechanism in samba, you will _finally_ be able to run "true" nt
services.
that'd be absolutely... wild.
you'll still need to use the built-in \\pipe\netlogon, \\pipe\lsarpc
and \\pipe\netlogon, for the time being, and if ... oh. i'm sure i've
got a netlogon.idl and a samr.idl file somewhere, reverse-engineered.
but hey, you can always get th
Re: [patch] review requested - NamedPipe Message mode (was Reliable Datagram / Stream behaviour of NT Pipes)
> You can't do that stuff on the client side. You either have to do all > pipe I/O in the server, or add named pipe support in the kernel. The > latter is harder, but would be much more useful. well, not entirely knowing the difference, i'm guessing that i'm adding named pipe support in the kernel. starting from SetNamedPipeHandleState and going from there. comparing what i've got so far in the improvements to ReadFile, WriteFile, NtFsControlIo etc. with the way that it is believed that this _should_ be done, what is right and what is wrong? l.
[patch] review requested - NamedPipe Message mode (was Reliable Datagram / Stream behaviour of NT Pipes)
http://bugs.winehq.org/attachment.cgi?id=19132 ok this is the beginnings of an implementation of messagemode, it passes the test1 and send2recv2 tests and fails the shortread test as part of PeekNamedPipe(). i'm not exactly sure why, yet. i get the distinct feeling that there's confusion over who writes into the wineserver data structures what the "availdata" is. msrpc.exe says "set availdata to 9", msrpcd.exe says "set it to 0, we did a write!" and it's all just o dear. also i have double-copies of server_set_named_pipe_info(), one in kernel32 and another in ntdll because i am using it in both places, i think SetNamedPipeHandleState() should be calling an NtFsControlIo rather than using server_set_named_pipe_info() but for the life of me i _really_ don't want to be finding out what the heck that function is :) hmm, maybe i can find out by copying ntdll.dll over to wine and doing a WINEDEBUG=trace+ntdll,kernel32 to see what functions get called. anyway the primary purpose of submitting the patch is to ask people if i'm along the right lines. anything seriously objectionable, etc. etc. l.
Re: http://wiki.winehq.org/NamedPipes
> Funny enough, the Samba project codenamed "franky" > (http://wiki.samba.org/index.php/Franky) will be doing all of this. It's not > quite there yet, but after some nice work done in the Samba3 smbd by Volker > Lendecke and some more work in the Samba4 "samba" daemon by Stefan > Metzmacher, Jelmer Vernooij and some overall effort by other folks from the > Samba Team, franky is coming along quite well. kai, thinking about this some more (hi volker, nice to see you're still working on this stuff!) i just want to emphasise a few things: 1) i really am absolutely delighted that a break-out mechanism has been added to samba, for many many reasons 2) you will also need to add a break-out mechanism for the entire IPC$ channel, along the same architecture lines, for exactly the same reasons. wine and other systems have their _own_ implementations of NetUserGetInfo and other functions; in combination with the code in cliffsd (which is again public domain and was written from taking the IPC spec document and using it as an IDL file!) it will be possible for wine add IPC$ functions to WineServer with minimal coding effort on the part of the Wine Developers. the way that i implemented this (or maybe planned to - looking at the code i don't see it. odd. oh well!) in samba tng was to create a daemon called "lanmand", which received an authentication context "blob" (effectively a marshalled NET_USER_GET_INFO_3 structure aka a SAMR_USER_INFO_21), and the functions where the "break-out" occurred were on SMB_TransactNamedPipe, SMB_ReadX, SMB_WriteX and SMB_Close on anything that's an IPC$. think about it: the wine team have to get notifications to wineserver of events such as WaitNamedPipeState, SetNamedPipeState etc. etc. and the only way to do _that_ properly is if they receive proper notifications. it's made a little more complex unfortunately in that _some_ of the data needs to go to the LANMAN break-out mechanism, and _some_ of the data needs to go into the DCE/RPC IPC$ stuff. ohhh, no it isn't: it's actually very simple: i forgot - there's a PIPE named LANMAN, isn't there? :) so, actually, it might be simpler than i thought, in that WineServer could simply have its own LANMAN pipe service (bypassing the one that's hard-coded in smbd). ... you'll work it out :) 3) the strategic importance of the "break-out" mechanism cannot be underestimated. by not implementing a break-out mechanism for all these years, samba has effectively been operating under a "proprietary implementation" shadow, made all the more worse by the fact that the source code is GPL'd and so it _looks_ like it's free software. free software projects, as the trolltech code base and other examples showed by being effectively a "one-way-push" instead of a "collaborating encouraging community project", are not just about "freeing the code", when it comes to multi-man-decade efforts such as samba, wine, reactos etc. it's your _responsibility_ to ensure that there are interfaces where other projects can interoperate. otherwise, you are doing _exactly_ what microsoft has been doing - shutting the door in people's faces by not allowing them to interoperate along clearly-defined interface boundaries. and that's the primary reason why i was so pissed off with what was going on in samba. you weren't just shutting out samba tng by not adding in the break-out mechanism, you were shutting _everybody_ out - wine, reactos, embedded NAS storage devices, sun microsystems, _everybody_ who wanted to use smbd's expertise as a service that could be improved on _every_ single one of them was shut out - and thus you've been _pretending_ to serve the free software community ... up until now, when you are now, finally, _actually_ serving the free software community. which is why, and i cannot say this enough times, nor emphasise it enough, that it's such a big damn deal to have this interoperability interface. so - keep it up, and ing well done! oh - did andrew bartlett complete that authentication plugin interoperability interface? because that's another critical / crucial area that needs to be .so dynamically-loadable (a la pam) so that it can be "broken out of", as well. wine and reactos will need that at some point (actually, they'll probably need it at exactly the same time as implementing a \\PIPE\LANMAN and \\PIPE\) l.
Re: Reliable Datagram / Stream behaviour of NT Pipes
juan, hi, >> Some nice C test cases >> would go a long way toward a correct implementation. > > ok, then that's where i'll start. i've got a qemu'd xp so i can > actually test that they work properly on nt. > > i've just dragged the tng source code out of cvs, first time in ages, > so there's plenty there to work from, really quite quickly. http://bugs.winehq.org/attachment.cgi?id=19121 ok - i documented how to use the test code, in the bugreport. i'll extend it later, to cover the cross-process things you mentioned. short reads etc. one test i'm looking forward to adding is passing a 4-byte handle as a "number" over to another process and trying to use that :) are there any tests of any kind, for NamedPipes? i don't mind writing some weird test cases if it would help. opening pipes read-only and writing to them, that sort of thing. ohh i nearly forgot about transactnamedpipe - got to add that, too - it'll again be absolutely essential for samba-integration. l.
Re: http://wiki.winehq.org/NamedPipes
On Sat, Jan 31, 2009 at 9:50 AM, Kai Blin wrote: > On Friday 30 January 2009 19:38:20 Luke Kenneth Casson Leighton wrote: > >> samba having "break-out" for named pipes traffic is one of those >> absolutely _essential_ things that should have fucking well gone into >> the samba source code a _long_ time ago. around 1998 would have been >> good. > > Funny enough, the Samba project codenamed "franky" > (http://wiki.samba.org/index.php/Franky) will be doing all of this. It's not HOOORAY! time for a celebration. nearly a decade waiting for that stuff. that's really good news - thank you, kai.
Re: Reliable Datagram / Stream behaviour of NT Pipes
On Fri, Jan 30, 2009 at 6:41 PM, Juan Lang wrote: > Hi Luke, hi juan :) >> the issue with the implementation of nt named pipes on top of _just_ >> streams (unix sockets) is this: two packets sent get blatted into one >> read. > > You are correct. We've known about that bug for ages, but never got > around to fixing it. See e.g. Mike McCormack's interview and his > comments on message mode: > http://www.winehq.org/interview/7 o - message mode _that's_ what it's called :) dang, that got dragged out of long-term storage. i remember seeing that in smb packets. >> ... i'm veery tempted to say "can i have a go at implementing this?" >> but... i know it'll be ... how-to-say... eeenteresting :) >> >> should i try? > > Yes, by all means. this'll be interesting ha ha >> would an extra server message need to be added (for read), so that the >> server can keep track of how much data has been read back (of the >> current datagram)? > > Possibly. This is tricky too, because a named pipe handle can be > shared across processes. eek. > You really want named pipe reads to block > until a complete datagram is available. What to do with short reads > is something you'd have to figure out too. yep. > Some nice C test cases > would go a long way toward a correct implementation. ok, then that's where i'll start. i've got a qemu'd xp so i can actually test that they work properly on nt. i've just dragged the tng source code out of cvs, first time in ages, so there's plenty there to work from, really quite quickly. yaay! :)
Re: http://wiki.winehq.org/NamedPipes
p.s. the win32 port of samba tng was originally done for wine - and reactos's - benefit. and because it'd just be utterly one-in-the-eye to be able to turn something like a P.O.S windows98 box into a Primary Domain Controller ha ha :)
http://wiki.winehq.org/NamedPipes
rob, hi, regarding the above wiki page - there is no "obstacle". you can use any of the code that i wrote for samba (about 100,000 lines of code), i released all of it into the public domain, on dec 16th, 2005. there's enough there so that you can do an entirely "hands off" approach, getting packets in and out using (see last post!) surprise,surprise, emulated named pipes. what i did for tng was to create a "packaging" system which sent security contexts (complete security descriptors and other identifiying information) over a unix socket. it was completely transparent as far as smbd was concerned, which just "handed it around" to the appropriate daemons. i asked jeremy to add in simple support for this into main samba several times. the last time (dec 2005) once _again_ he fucked about, but did actually agree that it was a good idea. samba having "break-out" for named pipes traffic is one of those absolutely _essential_ things that should have fucking well gone into the samba source code a _long_ time ago. around 1998 would have been good. _several_ projects would enormously benefitted - including samba - if that had happened. sun microsystems, who bought the [well known to be a complete piece-of-crap] port of nt to unix, AFPS, for $USD 4m and were utterly appalled by the poor quality of the implementation of the smb server gave serious consideration to dumping it, keeping the MSRPC services, and riding off the back of samba .. via _guess what_! a named pipe "break-out" interface. which, of course, doesn't exist. so they couldn't do it. but gordon did discuss it with me, saying things like, "y'know, if samba had a way in to its named pipes interface..." and guess what? in order for you to do named pipes via samba, you need a named pipe "break-out" interface. guess how luke howard implemented XAD (which he sold to novell a couple of years ago)? he implemented a patch to samba which provided a named pipe "break-out" interface. that patch was released back in about 2002 and has never been incorporated into samba. the protocol for a break-out interface is very simple: you need four functions ( and they can be done in as little as about 300 lines of c-code, as a patch to samba) 1) open named pipe connection 2) read pipe 3) write pipe 4) close pipe if you implement that in a dynamically-loadable .so, style, as a vector table (see e.g. apache2 module interface for example) then you can dump a .so into a directory which samba can pick up. any data that comes in from SMB will then "activate" the .so, call the open_named_pipe function in the vector table, and you're away. you release the code of the .so under the LGPL, that gets "GPL'd" by samba, license problem is solved. so, with a small patch to samba, the means to take control of the direction of wine is within _your_ hands. because the dynamically loadable .so conforms to the "break-out" interface and is loaded _by_ samba, on demand, and the code in it is part of Wine and communicates using Wine protocols - entirely under your control. ... you know... it would actually be a lot easier to integrate wine with samba tng, because the infrastructure to communicate by emulated NamedPipes already exists. take a look at the following functions: ncalrpc_l_connect ncalrpc_l_close_socket receive_msrpc msrpc_send all of which are in lib/msrpc-client.c - you'll see in there that there are corresponding functions for the samba tng win32 port - CreateNamedPipe, ReadFile, WriteFile, WaitNamedPipeHandleState, CloseHandle etc. which will help give you some _really_ big clue-handles to understand the corresponding "emulating" functions using unix sockets. you _should_ also recognise the ncalrpc_srv_get_user_creds function and its equivalent client-side "sender" ncalrpc_l_authenticate. these two are a matched pair of functions which marshal and unmarshal an NT security context [based on a NET_USER_INFO_3 structure - recognise that one? :) it's a stub in wine - now you have some code that gives it purpose! ] basically after a connect (ncalrpc_l_connect) the client sends the user creds _immediately_, and the server picks them up, and saves them in its internal context (for later passing around etc). also they end up getting used to do seteuid() but as the entire protocol was used by root _only_ it was perfect safe to do this. the unix sockets were protected 600 with the directory they were in being protected 755 because some unixen _still_ bloody well let you access a unix domain socket even if its permissions were 600, all owned by root, all services running _as_ root, so one daemon being a client passing over a security context that told _another_ daemon running as root to seteuid to a user's uid was no big hairy deal. except that andrew thought that i was some sort of fucking 17-year-old incompetent student, and he didn't like the thought of me actually being a capable unix systems programmer. thus, this code could be justified as never go
Reliable Datagram / Stream behaviour of NT Pipes
folks, hi, if you recall, i implemented nt named pipes in samba tng, and also implemented the use of nt named pipes as a transport in freedce, for when i ported freedce to win32. so i have quite a bit of experience when it comes to nt named pipes. i've encountered some bugs (thanks to python again :) ) in the wine implementation of nt named pipes, and have raised it as a bug http://bugs.winehq.org/show_bug.cgi?id=17195 but it started getting complicated so i thought i'd best raise it here. the behaviour of nt named pipes is a cross between datagrams and streams - it's the best of both worlds: * absolute inviolate message sizes (datagrams) * absolute inviolate message order (streams). the issue with the implementation of nt named pipes on top of _just_ streams (unix sockets) is this: two packets sent get blatted into one read. you _cannot_ allow this to happen - it screws applications up _royally_, as applications _critically_ depend on the number of reads and writes being interleaved, thanks to the characteristics of NamedPipes. standard unix applications / protocols (SMTP, etc) have no choice but to do manual synchronisation built-in to the protocol, because standard unix streams don't _have_ this "free feature". so, in samba tng, the emulated-implementation of nt named pipes (on top of unix sockets) i was forced to send a (32-bit) length field actually _in_ the data stream, just like is done in NetBIOS. the NetBIOS protocol (which was implemented by IBM, not microsoft) sends a 16-bit length field followed by that exact number of bytes. again, NetBIOS implements a "reliable datagram-stream" protocol on top of standard unix streams (tcp stream, this time) inexplicably, unix simply... is ... entirely... missing this concept. so, i'm sorry to have to tell you this, but you are absolutely forced to emulate it (by sending length, data, length data), just like everyone else ever has. so, send (write) is simple: write out the length field, write out the data. done. dead simple. read is... tricky. you _may_ have to keep track of having read the length, already. or, of how _much_ has been read already, of the "current packet being read". *walks through source code a bit...* ... i'm veery tempted to say "can i have a go at implementing this?" but... i know it'll be ... how-to-say... eeenteresting :) should i try? would an extra server message need to be added (for read), so that the server can keep track of how much data has been read back (of the current datagram)? l.
Re: report on obscure issues: running out of file handles with python under wine
On Fri, Jan 30, 2009 at 3:41 AM, Vitaliy Margolen wrote: > Luke Kenneth Casson Leighton wrote: >> also note it's file descriptors not file handles > There are no such thing as "handle" in *NIX world. The only thing matter is > a file descriptor. But it's for files, directories, devices, etc. > > Wine takes care of all the handles internally (one of the main tasks for > wineserver). Wineserver has a really big handle limit: > #define MAX_HANDLE_ENTRIES 0x00ff ahh, then... not being funny or anything, but this might be a problem. bug-for-bug compatibility would imply having to bring that limit down to match what nt does. ... actually is there a second location where there are handle / filedescriptor limits? the reports i read seemed to indicate that there was a filedescriptor limit of 2048 and a handle limit that was far in excess of that (just like wine) > I really doubt you overrunning that. i'm not, and neither is the application - that's the point. ... but on windows, the application _is_ hitting the limit (of 2048) and is gracefully throwing errors. l.
Re: report on obscure issues: running out of file handles with python under wine
On Thu, Jan 29, 2009 at 2:02 PM, Michael Stefaniuc wrote: > Hi! > > Luke Kenneth Casson Leighton wrote: >> well the saga continues with using python under wine - i thought i'd >> let you know the analysis and clues so far, in case it rings any bells >> with anyone, now or in the future. as you're probably aware, there's >> a limit in nt of 2048 file descriptors which has caused a large number > Well the "default" number of file handles one gets on Linux is 1024. > Though that can be easily bumped up. Are you sure you do not run into > your host limit? hiya michael, good point - although the "success" instance was under wine (when it should not have been successful, according to the exact same test with the exact same app running under xp). also note it's file descriptors not file handles, and to emphasise this, the selectmodule.c in python i had to increase the #define FD_SETMAX to 2048 in order to cope with the massive number of file descriptors - both under wine and also native xp in a qemu VM. funny. it's always the bumbling-along not knowing what's really going on that finds the errors. if you quotes know what you're doing quotes then you always do things quotes right quotes and the errors are never encountered
report on obscure issues: running out of file handles with python under wine
hiya folks, well the saga continues with using python under wine - i thought i'd let you know the analysis and clues so far, in case it rings any bells with anyone, now or in the future. as you're probably aware, there's a limit in nt of 2048 file descriptors which has caused a large number of free software projects quite a lot of grief , e.g. this: http://bugs.mysql.com/bug.php?id=24509. it turns out that python is no different, and i'm just in the process of investigating whether compiling every single c-based module as a builtin saves me enough file handles to be able to not hit the 2048 handles limit. it's really quite stunning. even something as simple as running a single small test script can result in 1,400 file handles being used up. by the time you get to things like the popen-replacement (see msdn KB: Q190351 for a simple code example). anyway - what prompted me to write was that i ran the python.exe with msvcr80 on wine, and the popen-like code segfaulted. under win32 naitive (an actual XP qemu) it failed - gracefully. but, weirdly, python.exe _without_ msvcr80 did _not_ segfault, but, again, same code on XP qemu'd failed gracefully. so, matrix: wine xp-qemu'd vc80 segfault error vc71 okerror in other words, i'm beginning to suspect that the reason for that msvcr80 segfault i was investigating last week is because of running out of file handles, and wine doesn't _have_ that 2048 file descriptor limit that msvcr80 is assuming actually exists. either that, or when you _do_ reach that limit, it results in corruption that an app running msvcr71 doesn't notice. or something. the reason i haven't raised this as a bugreport is because i really don't exactly know what's going on - there are just rather a lot of empirical observations that are ... "out there" :) so i apologise for this being a bit long, and not very specific, and if ever i get anything specific i'll of course raise it as a bugreqeuest, but i didn't want to leave this undocumented. l.
Re: cmd.exe.manifest
> ahh... correction... err... actually, builtin msvcrt _doesn't_ go > according to plan! > > the data is returned (echo hello) but the python process hangs - it > never sees the results come back. correction: the reason for that was that my test case had "cmd /k echo test" not "cmd /c echo test". so i needed to type "exit" return. so, we conclude that the child process creation code using msvcrt is fine but with msvcr80 is buggered. i've found a test example (KB Q190351) which does a reaaasonable job - reading from the console of the child doesn't work. so the code from Q190351 would be a _good_ one to use. or write a test case based on Modules/posixmodule.c.
Re: cmd.exe.manifest
On Thu, Jan 22, 2009 at 8:14 PM, Luke Kenneth Casson Leighton wrote: >>> So the problem lies elsewhere... do you have any more information on >>> the debug output? >> >> hiya reece, thanks for responding. yes, i do - lemme get back to you >> with it: i'm in the middle of a ... actually, i _do_ have it: >> >> >> >> it's a trace+all so is about 15mb (sorry!) >> >> i'm doing a rebuild back to msvcrt to see if the problem goes away. > > http://lkcl.net/subprocess.wine.trace.msvcrt.native sorry that's ... oh bugger it i'll log in to my server and correct it :) > and, once that's uploaded, i'll also do one: > > http://lkcl.net/subprocess.wine.trace.msvcrt.builtin > anyway, as you can see, open_osfhandle and CreateProcess all go > swimmingly well - it's just that when you get msvcr80 involved it all > goes pearshaped. ahh... correction... err... actually, builtin msvcrt _doesn't_ go according to plan! the data is returned (echo hello) but the python process hangs - it never sees the results come back. ahh -it i'm going to have to write a c test case, aren't i.
Re: cmd.exe.manifest
>> So the problem lies elsewhere... do you have any more information on >> the debug output? > > hiya reece, thanks for responding. yes, i do - lemme get back to you > with it: i'm in the middle of a ... actually, i _do_ have it: > > > > it's a trace+all so is about 15mb (sorry!) > > i'm doing a rebuild back to msvcrt to see if the problem goes away. http://lkcl.net/subprocess.wine.trace.msvcrt.native and, once that's uploaded, i'll also do one: http://lkcl.net/subprocess.wine.trace.msvcrt.builtin but give me a few mins to do that one, it'll be there in a bit. here's the python application that did that: import subprocess p = subprocess.Popen(["cmd", "/k" "echo", "hello"], stdout=subprocess.PIPE) (stdout, stderr) = p.communicate() print repr(stdout) print repr(stderr) nice and short, isn't it? :) the code that goes with that - in the python subprocess module - is a bit of a lairy hairy bundle of fun. anyway, as you can see, open_osfhandle and CreateProcess all go swimmingly well - it's just that when you get msvcr80 involved it all goes pearshaped. l.
Re: cmd.exe.manifest
On Thu, Jan 22, 2009 at 6:59 PM, Reece Dunn wrote: > 2009/1/22 Luke Kenneth Casson Leighton : >> 0010:trace:file:RtlGetFullPathName_U >> (L"C:\\windows\\system32\\cmd.exe.manifest" 520 0xff9aaaf4 (nil)) >> attr= sharing=0001 disp=1 options=0010 ea=(nil).0x >> 0009:trace:seh:start_debugger Starting debugger "winedbg --auto 8 100" >> >> >> ? >> >> ahh fer 's sake :) >> >> this is after various complicated ways to create a subprocess with >> which to communicate (stdin / stdout) involving CreatePipe, >> open_osfhandle, - but it's being done from an app that's compiled with >> msvcr80. >> >> 0009: close_handle() = 0 >> 0010:trace:heap:RtlAllocateHeap (0x11,0002,0038): returning >> 0x114e58 >> 0010:trace:actctx:get_manifest_in_module looking for res #0001 in >> module 0x7ec1 L"C:\\windows\\system32\\cmd.exe" >> 0010:trace:heap:RtlFreeHeap (0x11,0002,0x114e58): returning TRUE >> 0010:trace:resource:LdrFindResource_U module 0x7ec1 type #0018 >> name #0001 lang level 3 >> 0010:trace:resource:find_entry_by_id root 0x7ec322e4 dir 0x7ec322e4 id >> 0018 not found >> 0010:trace:actctx:get_manifest_in_associated_manifest looking for >> manifest associated with (null) id 1 >> 0010:trace:heap:RtlAllocateHeap (0x11,0002,0060): returning >> 0x114e58 >> 0009:trace:heap:RtlFreeHeap (0x11,0002,0x1361e0): returning TRUE >> 0010:trace:file:RtlDosPathNameToNtPathName_U >> (L"C:\\windows\\system32\\cmd.exe.manifest",0xffdc6170,(nil),(nil)) >> 0009:trace:process:CreateProcessW started process pid 000f tid 0010 >> 0010:trace:file:RtlGetFullPathName_U >> (L"C:\\windows\\system32\\cmd.exe.manifest" 520 0xffdc5f14 (nil)) >> >> >> ... but hang on... some notes somewhere after a google search for >> cmd.exe.manifest, apparently there isn't supposed to _be_ a manifest >> for cmd.exe - so that it can't be "themed". > > cmd.exe *may* have a manifest, it's just that it won't specify common > controls v6 (which enables the theming code in >= XP). For example, > patch.exe and install.exe in cygwin have .manifest files to tell Vista > "I am *not* an installer!". > >> ... wossgoinon?! :) > > When Wine (or Windows >= XP) loads a process (exe or dll), it looks > for a .manifest file in the folder where the exe is located. If it > does not find one there, it looks for a RT_MANIFEST resource in the > processes resource block. Failing that, it assumes that the process > was not built with a manifest. NOTE: I'm not sure on the ordering of > the check for a manifest file (i.e. which takes precedence -- external > file or embedded resource). > > In the trace log above, you can see that Wine is checking for an > embedded manifest resource first, then the external .manifest file. > > So the problem lies elsewhere... do you have any more information on > the debug output? hiya reece, thanks for responding. yes, i do - lemme get back to you with it: i'm in the middle of a ... actually, i _do_ have it: http://lkcl.net/subprocess.wine.trace it's a trace+all so is about 15mb (sorry!) i'm doing a rebuild back to msvcrt to see if the problem goes away. reproducing this in c-code will be quite a bit of work - it's not like the other tests i did (the msvcrt ones) - it'll be about... 150 to 200 lines of c code, doing a CreateProcess and other tricks, and would take about a day to go through the python code, replicating the python subsytems being used _without_ the python. and this is getting pretty draining - it's been a _lot_ of work. l.
cmd.exe.manifest
0010:trace:file:RtlGetFullPathName_U (L"C:\\windows\\system32\\cmd.exe.manifest" 520 0xff9aaaf4 (nil)) attr= sharing=0001 disp=1 options=0010 ea=(nil).0x 0009:trace:seh:start_debugger Starting debugger "winedbg --auto 8 100" ? ahh fer 's sake :) this is after various complicated ways to create a subprocess with which to communicate (stdin / stdout) involving CreatePipe, open_osfhandle, - but it's being done from an app that's compiled with msvcr80. 0009: close_handle() = 0 0010:trace:heap:RtlAllocateHeap (0x11,0002,0038): returning 0x114e58 0010:trace:actctx:get_manifest_in_module looking for res #0001 in module 0x7ec1 L"C:\\windows\\system32\\cmd.exe" 0010:trace:heap:RtlFreeHeap (0x11,0002,0x114e58): returning TRUE 0010:trace:resource:LdrFindResource_U module 0x7ec1 type #0018 name #0001 lang level 3 0010:trace:resource:find_entry_by_id root 0x7ec322e4 dir 0x7ec322e4 id 0018 not found 0010:trace:actctx:get_manifest_in_associated_manifest looking for manifest associated with (null) id 1 0010:trace:heap:RtlAllocateHeap (0x11,0002,0060): returning 0x114e58 0009:trace:heap:RtlFreeHeap (0x11,0002,0x1361e0): returning TRUE 0010:trace:file:RtlDosPathNameToNtPathName_U (L"C:\\windows\\system32\\cmd.exe.manifest",0xffdc6170,(nil),(nil)) 0009:trace:process:CreateProcessW started process pid 000f tid 0010 0010:trace:file:RtlGetFullPathName_U (L"C:\\windows\\system32\\cmd.exe.manifest" 520 0xffdc5f14 (nil)) ... but hang on... some notes somewhere after a google search for cmd.exe.manifest, apparently there isn't supposed to _be_ a manifest for cmd.exe - so that it can't be "themed". ... wossgoinon?! :) l.
http://bugs.winehq.org/show_bug.cgi?id=17003 - lseek with SEEK_END being ignored after fread
this one's a loovely obscure one - fread makes lseek be ignored :) l.
stdin and stdout getting connected to the wrong things!
http://bugs.winehq.org/show_bug.cgi?id=16968 this one's _really_ intriguing. i've been observing that occasionally, running python.exe e.g from /bin/sh.exe interactively shows... nothing. it's like, "dude, where's my car^H^H^Hinteractive python prompt?>>>" and the python developers are _really_ getting heavy at me for moving Py_Initialize() around and doing all sorts of weird shit like _assuming_ that the isatty(stdin) test has succeeded n stuff. well, i think what's happening - and this popen() test gives some clues - is that stdin and stdout are being connected to the xterm from which /usr/bin/wine was started... for a _brief_ period of time. after some unspecified time, they are no longer connected to the starting-process's stdin or stdout, but are connected to the correct places. but in many instances, that's far too late - a process should _never_ start up with its stdin and stdout connected to the wrong place in the first place! l.
libmsi.a import library from wine, and header files available (entirely free software), available for python-win32 builds under msys+wine
as part of building python2.5.2 under msys under wine on linux using mingw, i thought i'd try building _msi.pyd just for kicks. of course, that required having an msi.lib import library, and associated header files. so, purely as an experiment, i've documented the process by which it is possible to take wine 1.1.13 (current development release) source code, modify the header files for use with mingw, create a typelibrary (using dlltool) and then, surprise-surprise, _msi.pyd successfully builds. i say successfully builds: but then, running from _outside_ of a wineconsole cmd, doing this: /usr/local/bin/wine ./python.exe -c 'import _msi' succeeds. but if you do this: /usr/local/bin/wineconsole cmd c:/python2.5/bin/python.exde -c 'import _msi' you get an "access violation - no access to memory" blah blah to be honest, i don't care about that: this message is to let people know that it _is_ possible, and, if anyone is interested in e.g. adding -lmsi -lcabinet support to MinGW, here's where you can get the necessary crud: http://lkcl.net/msi.tgz if the wine team have any objections, if they believe this is a bad idea, please do say so :) l. p.s. if anyone would like to add a regression test to python called "test_msi.py" - or if they know of one that exists, i'd love to hear from you and try it out. outside of a wine cmd.exe of course :)
Re: python regression tests - a monster hammer to keep wine "in check" :)
On Sun, Jan 18, 2009 at 4:52 PM, Hin-Tak Leung wrote: > --- On Sun, 18/1/09, Luke Kenneth Casson Leighton wrote: > >> the regression test test_file.py has succeeded for years >> under >> proprietary native win32 platforms using the proprietary >> msvc compiler >> to build python.exe and python2N.dll for win32 platforms. >> >> this particular test is therefore about testing and >> ensuring that >> \r\n handling SPECIFICALLY in O_TEXT mode is >> correct. >> >> it has nothing to do with O_BINARY mode. > > I already wrote that the default translation behavior (if not explicitly > specified) is both compiler and runtime dependent. nope, it's not. > The fact that msvc-compiled python working in native win32 is rather > different from mingw-compiled python working in wine, isn't it? nope, not in the slightest bit. >It differs both in the compiler used no, it doesn't. i'm using exactly the same compiler that is used by rouman petrov, as described in http://bugs.python.org/issue3871 this is not "compile under linux and link with -lwine". this is "compile using mingw32.exe running under msys to EXACTLY emulate the EXACT same compile as if running under win32". > and the runtime used. is msvcrt and is, on wine, endeavouring to be _exactly_ the same. therefore, this is testing msvcrt. therefore, the bugs are in wine's implementation of msvcrt. specifically, the O_TEXT handling. and then some. ftell, fseek with SEEK_END after an fread - there's tons of "little" bugs. > From my experience if my memory serves (in the above bug report), Borland's > compiler by-default generate code which is asymetrical - the reading code > won't read what the writing code writes, in either windows or wine or both; > that was very surprising to me. well, i'm not using borland's compiler, i'm using mingw32, running it under msys+wine, so it is "exactly as if compiling on a windows system". l.
Re: python regression tests - a monster hammer to keep wine "in check" :)
On Sun, Jan 18, 2009 at 1:52 AM, Hin-Tak Leung wrote:
> --- On Sat, 17/1/09, Luke Kenneth Casson Leighton wrote:
>
>> > but the behaviour of msvcrt wrt crlf is _definitely_
>> not right, as it
>> > stands, and as a result it completely screws any
>> possibility for
>> > running python.exe under wine. completely. you
>> can't have files that
>> > you write to, that are different from when they are
>> read back.
>>
>> ok this turns out to be a _really_ interesting
>> edge-case :)
>>
>> l...@gonzalez:~/src/python2.5-2.5.2$ ./python.exe
>> Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34)
>> [gcc-mingw32] on win32
>> Type "help", "copyright",
>> "credits" or "license" for more
>> information.
>> >>> f = open("tst", "w")
>> >>> f.write("hell\n")
>> >>> f.write("back\n")
>> >>> f.close()
>> >>>
>> [80]+ Stopped ./python.exe
>> l...@gonzalez:~/src/python2.5-2.5.2$ more tst
>> hell
>> back
>> l...@gonzalez:~/src/python2.5-2.5.2$ %
>> ./python.exe
>> f = open("tst", "r")
>> >>> x = f.readline(5)
>> >>> x
>> 'hell\r' <--- wrongg
>> >>> f.seek(0)
>> >>> f.readline()
>> 'hell\n' < ah _ha_. if you
>> don't specify the length to be read
>> >>> f.seek(0)
>> >>> f.readline(4)
>> 'hell'
>> >>> f.readline()
>> '\n'
>> >>>
>>
>> will raise this as a bug, now that there's a known test
>> case for it.
>
> But that's as much a problem as mingw, as wine, as usage of python...
no it isn't.
> I am about to go back to fix a bug with ghostscript's ijs interface when
> compiled with non-MSVC: IJS is a binary protocol (for communicating with
> non-postscript printer drivers), but if you don't explicitly declare "wb" and
> "rb", and precise behavior - whether to do \n to \r\n translation - is
> dependent on both compilers (mingw, msvc, borland) and C runtime (wine or
> native windows). So if you want it to work correctly, you explicitly declare
> everything.
that _isn't_ the case here.
> I think python supports a wb/rb argument to open()?
yes it does but that is not what is being tested, here.
> It is probably your usage of python is wrong here.
no, it isn't.
the regression test test_file.py has succeeded for years under
proprietary native win32 platforms using the proprietary msvc compiler
to build python.exe and python2N.dll for win32 platforms.
this particular test is therefore about testing and ensuring that
\r\n handling SPECIFICALLY in O_TEXT mode is correct.
it has nothing to do with O_BINARY mode.
l.
Re: python regression tests - a monster hammer to keep wine "in check" :)
l...@gonzalez:~/src/python2.5-2.5.2$ ./python.exe Lib/test/test_file.py -v testIteration (__main__.OtherFileTests) ... ok -- Ran 1 test in 0.008s OK yaay! so, obviously, the regression tests in python aren't _that_ good, as they haven't spotted the seek position off-by-one bug. On Sat, Jan 17, 2009 at 8:40 PM, Luke Kenneth Casson Leighton wrote: > spiffo, dan, jolly good show :) > > that fixed both #16982 _and_ #16970 - the character reading bit - with > the exception that the ftell() position, which was wrong _before_ this > patch, is still also wrong, as you suspected. > > l. > > On Sat, Jan 17, 2009 at 7:15 PM, Dan Kegel wrote: >> Luke wrote: >>> [MSVCRT__filbuf() needs to strip CR's.] >> >> Here's a quickie patch that kind of gets there, >> but fails an ftell test. My in-laws arrive in ten >> minutes, so I probably can't go any further on >> it this weekend. >> >> There are probably other simplifications that >> could be made beyond this, I didn't rip out >> all the scattered places that were involved >> in \r removal. >> - Dan >> >
Re: python regression tests - a monster hammer to keep wine "in check" :)
spiffo, dan, jolly good show :) that fixed both #16982 _and_ #16970 - the character reading bit - with the exception that the ftell() position, which was wrong _before_ this patch, is still also wrong, as you suspected. l. On Sat, Jan 17, 2009 at 7:15 PM, Dan Kegel wrote: > Luke wrote: >> [MSVCRT__filbuf() needs to strip CR's.] > > Here's a quickie patch that kind of gets there, > but fails an ftell test. My in-laws arrive in ten > minutes, so I probably can't go any further on > it this weekend. > > There are probably other simplifications that > could be made beyond this, I didn't rip out > all the scattered places that were involved > in \r removal. > - Dan >
fgets followed by fread gives wrong results
http://bugs.winehq.org/show_bug.cgi?id=16982 test case included, along with output shown from native msvcrt.dll run. this is _different_ from #16790 (but inter-related, as fgets is made up from lots of fgetc()s. the file pos (from ftell) is off by one, compared to msvcrt.dll On Sat, Jan 17, 2009 at 7:53 PM, Luke Kenneth Casson Leighton wrote: > give it a bit more time - there's another one (involving fgets > followed by fread) which i'm investigating, i'll have that one nailed > as a test case in a bit, am just about to add a test case to #16790
Re: python regression tests - a monster hammer to keep wine "in check" :)
give it a bit more time - there's another one (involving fgets followed by fread) which i'm investigating, i'll have that one nailed as a test case in a bit, am just about to add a test case to #16790 On Sat, Jan 17, 2009 at 7:15 PM, Dan Kegel wrote: > Luke wrote: >> [MSVCRT__filbuf() needs to strip CR's.] > > Here's a quickie patch that kind of gets there, > but fails an ftell test. My in-laws arrive in ten > minutes, so I probably can't go any further on > it this weekend. > > There are probably other simplifications that > could be made beyond this, I didn't rip out > all the scattered places that were involved > in \r removal. > - Dan >
Re: python regression tests - a monster hammer to keep wine "in check" :)
got it.
in stdio.h:
#if !defined _MT (which it isn't...)
__CRT_INLINE int __cdecl __MINGW_NOTHROW getc (FILE* __F)
{
return (--__F->_cnt >= 0)
? (int) (unsigned char) *__F->_ptr++
: _filbuf (__F);
}
#else /* Use library functions. */
_CRTIMP int __cdecl __MINGW_NOTHROW getc (FILE*);
#endif
consequently, MSVCRT__filbuf() is getting called _directly_. and in
native MSVCRT.DLL, guess which function performs CRLF skipping, and
_guess_ which function _doesn't_ perform CRLF skipping in
dlls/msvcrt/file.c ?
:)
On Sat, Jan 17, 2009 at 10:58 AM, Luke Kenneth Casson Leighton
wrote:
>> but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it
>> stands, and as a result it completely screws any possibility for
>> running python.exe under wine. completely. you can't have files that
>> you write to, that are different from when they are read back.
>
> ok this turns out to be a _really_ interesting edge-case :)
>
> l...@gonzalez:~/src/python2.5-2.5.2$ ./python.exe
> Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34) [gcc-mingw32] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> f = open("tst", "w")
>>>> f.write("hell\n")
>>>> f.write("back\n")
>>>> f.close()
>>>>
> [80]+ Stopped ./python.exe
> l...@gonzalez:~/src/python2.5-2.5.2$ more tst
> hell
> back
> l...@gonzalez:~/src/python2.5-2.5.2$ %
> ./python.exe
> f = open("tst", "r")
>>>> x = f.readline(5)
>>>> x
> 'hell\r' <--- wrongg
>>>> f.seek(0)
>>>> f.readline()
> 'hell\n' < ah _ha_. if you don't specify the length to be read
>>>> f.seek(0)
>>>> f.readline(4)
> 'hell'
>>>> f.readline()
> '\n'
>>>>
>
> will raise this as a bug, now that there's a known test case for it.
>
Re: python regression tests - a monster hammer to keep wine "in check" :)
> but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it
> stands, and as a result it completely screws any possibility for
> running python.exe under wine. completely. you can't have files that
> you write to, that are different from when they are read back.
ok this turns out to be a _really_ interesting edge-case :)
l...@gonzalez:~/src/python2.5-2.5.2$ ./python.exe
Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34) [gcc-mingw32] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("tst", "w")
>>> f.write("hell\n")
>>> f.write("back\n")
>>> f.close()
>>>
[80]+ Stopped ./python.exe
l...@gonzalez:~/src/python2.5-2.5.2$ more tst
hell
back
l...@gonzalez:~/src/python2.5-2.5.2$ %
./python.exe
f = open("tst", "r")
>>> x = f.readline(5)
>>> x
'hell\r' <--- wrongg
>>> f.seek(0)
>>> f.readline()
'hell\n' < ah _ha_. if you don't specify the length to be read
>>> f.seek(0)
>>> f.readline(4)
'hell'
>>> f.readline()
'\n'
>>>
will raise this as a bug, now that there's a known test case for it.
python regression tests - a monster hammer to keep wine "in check" :)
folks, hi, i've started running the python regression tests under wine (using the build of python i've only just created) and have started filling in bugreports for the tests that fail, but some things occurred to me: 1) why am i the only person filling out the bugreports? :) 2) surely these regression tests would work equally as well under the python2.5.2.exe that you can get as part of the pre-prepared python win32 install that comes from python.org and activestate.com, you don't _need_ my wine-built version of python.exe 3) why am i considering rewriting the tests as c-code when they can just as well be left as python code? one of the bugs that's _really_ really important is: crlf conversion. the code in wine's msvcrt is completely borked. i demonstrated this by switching over to "native" msvcrt.dll, and nearly _all_ of the 12 tests that failed, all but the test_os.py ones miraculously succeeded. the majority of tests fail due to file corruption over writing \n and getting back \r\n, which is clearly bogus. _yes_ i'm aware of the difference between O_BINARY and O_TEXT, and it would seem that there's either a default setting which is wrong, or there's some runtime detection going on, or... just... _something_ :) but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it stands, and as a result it completely screws any possibility for running python.exe under wine. completely. you can't have files that you write to, that are different from when they are read back. l.
Re: success: compilation of python2.5.2 under wine+msys+mingw32
On Fri, Jan 16, 2009 at 4:57 PM, Dan Kegel wrote: > Awesome. Thanks for the tmpfile bug report. > If you can feed us any more similar reports, > please do! no problem. there's plenty - i'll try to track them down to concrete examples. l.
success: compilation of python2.5.2 under wine+msys+mingw32
hi folks, i promised i'd get python built under wine - and i'm happy to report that this goal has been successfully achieved. http://bugs.python.org/issue4954 if anyone's interested. what has been achieved is that a python.exe, libpython2.5.dll, an implib libpython2.5.dll.a and a python2.5.def have all been successfully created using an entirely free software toolchain, along with the dynamic libraries (such as pyexpat.pyd - python for win32's convention is .pyd not .so). running the regression testing has been... interesting :) several bugs in wine have been detected, along with faults in the header-files that need to be corrected. tmpfile() is faulty; MAX_LONG and MIN_INT #defines cause problems; the default file format CRLF instead of LF causes the builtin regression test to fail; _ctypes _utterly_ borks on c structure creation and manipulation but is using assembly-stuff (a copy of libffi is included in python) - there are a few more, but out of 350 tests, the majority of them succeed. amazingly. anyway, the point of this message is primarily to let people know that this successful compile shows just how far along wine is coming, and it's like a really big hairy deal that python can now be compiled for win32 platforms, for native use on windows or on wine, _without_ having to have a proprietary operating system or a proprietary compiler to do it. now it's possible to port things like pygtk2, pyqt4, pywebkit-gtk and numpy to win32 etc. (because of the header files and the implibs) - again, _without_ requiring proprietary technology. which is absolutely fantastic. thank you to everyone on the wine team. l.
Re: running msys under wine
On Sat, Jan 10, 2009 at 12:05 AM, David Laight wrote: > On Fri, Jan 09, 2009 at 06:39:27PM +0000, Luke Kenneth Casson Leighton wrote: >> >> further up the strace files, i'm looking at the biggest offender and >> it looks like it's reading files one byte at a time. > > Certainly a normal unix shell has to read input (from stdin) byte by byte > since it never knows when it might have to fork a process passing stdin > through - and the child needs the correct file offset. ohhh, ok :) it just looked... mad!
Re: running msys under wine
> If it starts up the wine environment every time this might take some time. > > Try to keep a "notepad" open in the background so the wine environment stays > running. hiya marcus, good advice - sounds reasonable. tried it: wineserver is the top-running process, stays with the same process id. so i don't believe it makes a difference: we're still looking at 20-30 seconds per line from configure. what i did was utterly utterly nastily cheat: i #ifdef'd most of the lines from python's configure script, for a win32 build! this would normally be an absolute recipe for a horror-story disaster, but, given that there's a preconfigured hard-coded pcbuild.h for visual studio to use, to replace what configure normally does _anyway_, it's a one-off acceptable thing. this cuts the ./configure time from 2 hours to 10 minutes... :) l.
running msys under wine
http://bugs.winehq.org/show_bug.cgi?id=13606
folks, hi,
am running a configure script under wine, and it's _literally_ one to
two seconds per sh.exe instance. i started running ./configure over
two hours ago and there are about 100 lines of output so far, with a
further 200 to go i am better off leaving this running overnight. as
this is development work i am doing, not "building something that's
already known to work" i have to call a halt to the development,
because it is crazy to expect to have to wait four to six hours if i
make a single change to configure.in or Makefile.pre.in - god help me
if i have to run autoconf (fortunately i can do that from linux).
... so i did some investigation:
strace -o log.txt -ff wine cmd
followed by running c:\msys\msys.bat
ignoring the fact that sh.exe crashed on me (which it doesn't do when
you don't use strace) i was able to obtain some system call usage when
firing up sh.exe, and it's _horrendous_.
X11 keyboard files, X11 locale files, iconv, windows fonts, unix fonts
- all to fire up c:/msys/bin/sh.exe ?
further up the strace files, i'm looking at the biggest offender and
it looks like it's reading files one byte at a time. reconstructing
some of the sentences i can make out "remember that we get here even
under command.com\r\n". ohhh i remember where i've seen that - that's
in msys.bat. why is it being read one byte at a time?? there are
_thousands_ of lines like this:
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 USR2 ALRM CHLD IO], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
read(13, "t", 1)= 1
rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 USR2 ALRM CHLD IO], [], 8) = 0
write(3, "7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) = 64
read(6, "\r\0\0\300\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) = 64
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 USR2 ALRM CHLD IO], [], 8) = 0
write(3, "4\0\0\0\0\0\0\0\0\0\0\0?\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) = 64
read(6, "$\0\0\300\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) = 64
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 USR2 ALRM CHLD IO], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
checking earlier in the same strace log file (the one that loaded
msys.bat one byte at a time) _again_ i see font-loading (every single
one on my system).
there are _forty_ instances of libraries being loaded - per process!
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = 3
open("/usr/lib/libwine.so.1", O_RDONLY) = 3
open("/usr/lib/wine/wine-pthread", O_RDONLY) = 3
open("/usr/lib/libwine.so.1", O_RDONLY) = 3
open("/usr/lib/wine/ntdll.dll.so", O_RDONLY) = 3
open("/usr/lib/wine/kernel32.dll.so", O_RDONLY) = 7
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 7
open("/usr/lib/wine/cmd.exe.so", O_RDONLY) = 10
open("/usr/lib/wine/shell32.dll.so", O_RDONLY) = 10
open("/usr/lib/wine/shlwapi.dll.so", O_RDONLY) = 10
open("/usr/lib/wine/user32.dll.so", O_RDONLY) = 10
open("/usr/lib/wine/gdi32.dll.so", O_RDONLY) = 10
open("/usr/lib/wine/advapi32.dll.so", O_RDONLY) = 10
open("/usr/lib/wine/comctl32.dll.so", O_RDONLY) = 10
open("/usr/lib/wine/krnl386.exe16", O_RDONLY|O_LARGEFILE) = 10
open("/usr/lib/wine/system.drv16", O_RDONLY|O_LARGEFILE) = 10
open("/usr/lib/wine/gdi.exe16", O_RDONLY|O_LARGEFILE) = 10
open("/usr/lib32/libfreetype.so.6", O_RDONLY) = 10
open("/usr/lib32/libz.so.1", O_RDONLY) = 10
open("/usr/lib32/libexpat.so.1", O_RDONLY) = 10
open("/usr/lib/wine/user.exe16", O_RDONLY|O_LARGEFILE) = 10
open("/usr/lib/wine/winex11.drv.so", O_RDONLY) = 10
open("/usr/lib32/libSM.so.6", O_RDONLY) = 10
open("/usr/lib32/libICE.so.6", O_RDONLY) = 10
open("/usr/lib32/libXxf86vm.so.1", O_RDONLY) = 10
open("/usr/lib32/libXext.so.6", O_RDONLY) = 10
open("/usr/lib32/libX11.so.6", O_RDONLY) = 10
open("/usr/lib32/libXau.so.6", O_RDONLY) = 10
open("/usr/lib32/libxcb-xlib.so.0", O_RDONLY) = 10
open("/usr/lib32/libxcb.so.1", O_RDONLY) = 10
open("/usr/lib32/libXdmcp.so.6", O_RDONLY) = 10
open("/usr/lib/wine/imm32.dll.so", O_RDONLY) = 10
open("/usr/lib32/libXinerama.so.1", O_RDONLY) = 11
open("/usr/lib32/libXrender.so.1", O_RDONLY) = 11
open("/usr/lib32/libXrandr.so.2", O_RDONLY) = 11
open("/usr/lib32/libXcomposite.so.1", O_RDONLY) = 11
open("/usr/lib32/libXfixes.so.3", O_RDONLY) = 11
open("/usr/lib32/libXcursor.so.1", O_RDONLY) = 11
open("/usr/lib/wine/uxtheme.dll.so", O_RDONLY) = 11
open("/usr/lib/wine/ole32.dll.so", O_RDONLY) = 11
open("/usr/lib/wine/rpcrt4.dll.so", O_RDONLY) = 11
open("/usr/lib/wine/iphlpapi.dll.so", O_RDONLY) = 11
open("/usr/lib/wine/start.exe.so", O_RDONLY|O_LARGEFILE) = 7
surely, on a console-based command shell, all of the fonts and the X11
libraries can be skipped?
l.
Re: compiling python2.5 on linux under wine
On Sat, Jan 3, 2009 at 9:22 PM, Luke Kenneth Casson Leighton
wrote:
> hey, has anyone investigated compiling python2.5 using winegcc, under wine?
some people might find this kind of thing amusing. it's considered in
very obtuse circles to be "progress"... :)
l...@gonzalez:/mnt/src/python2.5-2.5.2/Lib$ ../build/python -v
Could not find platform independent libraries
Could not find platform dependent libraries
Consider setting $PYTHONHOME to [:]
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
'import site' failed; traceback:
ImportError: No module named site
Python 2.5.2 (r252:60911, Jan 7 2009, 20:33:53) [gcc] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
[]
[]
[]
import sre_compile # from Z:\mnt\src\python2.5-2.5.2\Lib\sre_compile.py
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
# wrote Z:\mnt\src\python2.5-2.5.2\Lib\sre_compile.pyc
import _sre # builtin
import sre_constants # from Z:\mnt\src\python2.5-2.5.2\Lib\sre_constants.py
# wrote Z:\mnt\src\python2.5-2.5.2\Lib\sre_constants.pyc
import sre_parse # from Z:\mnt\src\python2.5-2.5.2\Lib\sre_parse.py
# wrote Z:\mnt\src\python2.5-2.5.2\Lib\sre_parse.pyc
Traceback (most recent call last):
File "", line 1, in
File "site.py", line 415, in
main()
File "site.py", line 406, in main
aliasmbcs()
File "site.py", line 356, in aliasmbcs
import locale, codecs
File "Z:\mnt\src\python2.5-2.5.2\Lib\locale.py", line 167, in
import re, operator
File "Z:\mnt\src\python2.5-2.5.2\Lib\re.py", line 223, in
_pattern_type = type(sre_compile.compile("", 0))
File "Z:\mnt\src\python2.5-2.5.2\Lib\sre_compile.py", line 530, in compile
groupindex, indexgroup
OverflowError: signed integer is less than minimum
>>>
Re: libwine.so "incompatible"???
frick. with -lwine -m32 at link time: ... sre.o Modules/_codecsmodule.o Modules/zipimport.o Modules/symtablemodule.o Modules/xxsubtype.o -lpthread -ldl -lutil -lwine -m32 -lm; \ fi collect2: ld terminated with signal 11 [Segmentation fault] /usr/bin/ld: i386:x86-64 architecture of input file `Modules/_typesmodule.o' is incompatible with i386 output /usr/bin/ld: i386:x86-64 architecture of input file `Modules/getbuildinfo.o' is incompatible with i386 output ... ... ... with CFLAGS="$CFLAGS -m32" gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I../Include -fPIC -I/usr/include/wine/windows -DPy_BUILD_CORE -o Modules/python.o ../Modules/python.c In file included from ../Include/Python.h:57, from ../Modules/python.c:3: ../Include/pyport.h:761:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)." make: *** [Modules/python.o] Error 1 hmmm time to start compiling under the 32-bit dchroot environment... On Sat, Jan 3, 2009 at 12:40 PM, Luke Kenneth Casson Leighton wrote: > http://www.winehq.org/pipermail/wine-devel/2007-January/053744.html > > hold that thought :) > > On Sat, Jan 3, 2009 at 12:38 PM, Luke Kenneth Casson Leighton > wrote: >> rob, anyone - any ideas? >> >>gcc -pthread -shared -o libpython2.5.so >> Modules/_typesmodule.o Modules/getbuildinfo.o Parser/acceler.o >> Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o >> Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o >> Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o >> Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o >> Objects/bufferobject.o Objects/cellobject.o Objects/classobject.o >> Objects/cobject.o Objects/codeobject.o Objects/complexobject.o >> Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o >> Objects/genobject.o Objects/fileobject.o Objects/floatobject.o >> Objects/frameobject.o Objects/funcobject.o Objects/intobject.o >> Objects/iterobject.o Objects/listobject.o Objects/longobject.o >> Objects/dictobject.o Objects/methodobject.o Objects/moduleobject.o >> Objects/object.o Objects/obmalloc.o Objects/rangeobject.o >> Objects/setobject.o Objects/sliceobject.o Objects/stringobject.o >> Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o >> Objects/weakrefobject.o Objects/unicodeobject.o Objects/unicodectype.o >> Python/Python-ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o >> Python/ceval.o Python/compile.o Python/codecs.o Python/errors.o >> Python/frozen.o Python/frozenmain.o Python/future.o Python/getargs.o >> Python/getcompiler.o Python/getcopyright.o Python/getmtime.o >> Python/getplatform.o Python/getversion.o Python/graminit.o >> Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o >> Python/mystrtoul.o Python/mysnprintf.o Python/pyarena.o Python/pyfpe.o >> Python/pystate.o Python/pythonrun.o Python/structmember.o >> Python/symtable.o Python/sysmodule.o Python/traceback.o >> Python/getopt.o Python/pystrtod.o Python/dynload_shlib.o >> Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o >> Modules/gcmodule.o Modules/threadmodule.o Modules/signalmodule.o >> Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o >> Modules/_sre.o Modules/_codecsmodule.o Modules/zipimport.o >> Modules/symtablemodule.o Modules/xxsubtype.o -lpthread -ldl -lutil >> -lwine -lm; \ >>fi >> /usr/bin/ld: skipping incompatible >> /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libwine.so when >> searching for -lwine >> /usr/bin/ld: skipping incompatible /usr/lib/../lib/libwine.so when >> searching for -lwine >> /usr/bin/ld: skipping incompatible >> /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../libwine.so when searching >> for -lwine >> /usr/bin/ld: skipping incompatible /usr/bin/../lib/libwine.so when >> searching for -lwine >> /usr/bin/ld: skipping incompatible /usr/lib64/libwine.so when >> searching for -lwine >> /usr/bin/ld: skipping incompatible /usr/lib/libwine.so when searching for >> -lwine >> /usr/bin/ld: cannot find -lwine >> collect2: ld returned 1 exit status >> >
Re: libwine.so "incompatible"???
http://www.winehq.org/pipermail/wine-devel/2007-January/053744.html hold that thought :) On Sat, Jan 3, 2009 at 12:38 PM, Luke Kenneth Casson Leighton wrote: > rob, anyone - any ideas? > >gcc -pthread -shared -o libpython2.5.so > Modules/_typesmodule.o Modules/getbuildinfo.o Parser/acceler.o > Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o > Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o > Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o > Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o > Objects/bufferobject.o Objects/cellobject.o Objects/classobject.o > Objects/cobject.o Objects/codeobject.o Objects/complexobject.o > Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o > Objects/genobject.o Objects/fileobject.o Objects/floatobject.o > Objects/frameobject.o Objects/funcobject.o Objects/intobject.o > Objects/iterobject.o Objects/listobject.o Objects/longobject.o > Objects/dictobject.o Objects/methodobject.o Objects/moduleobject.o > Objects/object.o Objects/obmalloc.o Objects/rangeobject.o > Objects/setobject.o Objects/sliceobject.o Objects/stringobject.o > Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o > Objects/weakrefobject.o Objects/unicodeobject.o Objects/unicodectype.o > Python/Python-ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o > Python/ceval.o Python/compile.o Python/codecs.o Python/errors.o > Python/frozen.o Python/frozenmain.o Python/future.o Python/getargs.o > Python/getcompiler.o Python/getcopyright.o Python/getmtime.o > Python/getplatform.o Python/getversion.o Python/graminit.o > Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o > Python/mystrtoul.o Python/mysnprintf.o Python/pyarena.o Python/pyfpe.o > Python/pystate.o Python/pythonrun.o Python/structmember.o > Python/symtable.o Python/sysmodule.o Python/traceback.o > Python/getopt.o Python/pystrtod.o Python/dynload_shlib.o > Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o > Modules/gcmodule.o Modules/threadmodule.o Modules/signalmodule.o > Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o > Modules/_sre.o Modules/_codecsmodule.o Modules/zipimport.o > Modules/symtablemodule.o Modules/xxsubtype.o -lpthread -ldl -lutil > -lwine -lm; \ >fi > /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libwine.so when > searching for -lwine > /usr/bin/ld: skipping incompatible /usr/lib/../lib/libwine.so when > searching for -lwine > /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../libwine.so when searching > for -lwine > /usr/bin/ld: skipping incompatible /usr/bin/../lib/libwine.so when > searching for -lwine > /usr/bin/ld: skipping incompatible /usr/lib64/libwine.so when > searching for -lwine > /usr/bin/ld: skipping incompatible /usr/lib/libwine.so when searching for > -lwine > /usr/bin/ld: cannot find -lwine > collect2: ld returned 1 exit status >
libwine.so "incompatible"???
rob, anyone - any ideas? gcc -pthread -shared -o libpython2.5.so Modules/_typesmodule.o Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o Objects/bufferobject.o Objects/cellobject.o Objects/classobject.o Objects/cobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/intobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/methodobject.o Objects/moduleobject.o Objects/object.o Objects/obmalloc.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/stringobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/weakrefobject.o Objects/unicodeobject.o Objects/unicodectype.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o Python/errors.o Python/frozen.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getmtime.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/pyarena.o Python/pyfpe.o Python/pystate.o Python/pythonrun.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/traceback.o Python/getopt.o Python/pystrtod.o Python/dynload_shlib.o Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/threadmodule.o Modules/signalmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/zipimport.o Modules/symtablemodule.o Modules/xxsubtype.o -lpthread -ldl -lutil -lwine -lm; \ fi /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libwine.so when searching for -lwine /usr/bin/ld: skipping incompatible /usr/lib/../lib/libwine.so when searching for -lwine /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../libwine.so when searching for -lwine /usr/bin/ld: skipping incompatible /usr/bin/../lib/libwine.so when searching for -lwine /usr/bin/ld: skipping incompatible /usr/lib64/libwine.so when searching for -lwine /usr/bin/ld: skipping incompatible /usr/lib/libwine.so when searching for -lwine /usr/bin/ld: cannot find -lwine collect2: ld returned 1 exit status
Re: question on how to do language bindings to MSHTML's COM interface
> There are a number of parts to your question: > 1. Is it OK for the application to be a winelib one (i.e. invoked via > or linked to wine, rather than being "native")? i'm on the first step - getting python compiled. so i'm in the middle of an experiment to compile python with wine, and it looks like i _have_ to make it an "invoked via wine" one. in fact, stunningly, that actually succeeded! using winegcc, a script called python and a dll called python.exe.so were created. of course, make install only copied over the script not the .so duh. the mistake i've made however is in not compiling with -DMS_WINDOWS. the issue that i have is that there _is_ no Makefile or Makefile.in for win32 python, there _is_ no configure script: it's all damn well done with visual studio .proj files on a hard-coded basis argh. i had the same difficulty i ran into when cross-compiling with a mingw32 cross-compiler. anyway - the first step is going reasonably well. i'm running into a bit of bother with python modules (.so extension) - i'll have to investigate the .proj files a bit more because the only things being built are _winreg.o, _md5.o and _not_ _winreg.so or _winreg.dll or wine equivalent. whoops. but - this is the first step. the next step would be to, i assume, add mshtml as a module? or perhaps see if someone's already done it, somewhere, with windows.
Re: question on how to do language bindings to MSHTML's COM interface
quick question: is there any progress on implementing COM, in wine, or is it necessary to grab dcom98.exe and to find the headers and the .libs from visual studio? and have you seen these - IDL files defined by wez, for DCOM? http://freedce.cvs.sourceforge.net/viewvc/freedce/freedce/dcom/ l. p.s. nice to hear from you again, rob :) p.p.s. below is "background" behind where i'm coming from with the python mshtml "thing". >> so there are some _really_ weird esoteric "sub-questions" involved, such as: >> "what are the chances of porting pywin32 - specifically pywin32's COM >> bindings - to linux, thanks to widl and friends"? >> see http://sourceforge.net/projects/pywin32 >> many thanks for any advice and information. > > We already build a typelib for mshtml using widl and pywin32 can use > that to make bindings for use at runtime, so it "should" work, but I > fail to see the need for it at the moment. you have to jump through a series of hoops to get there - being honest i'm not that good at expressing some of my leaps of multi-hoop-jumping but i'll try to give some pointers. 1) making python apps that rely on features of the win32 platform possible under unix as well. winreg and dcom are two that _really_ spring to mind. win32 developers are writing applications that use e.g. the registry. i tried compiling pywin32 on linux and _instantly_ ran into a difficulty: import _winreg. so, i would have to go into the python 2.5 / 2.6 tarball and look at a way of adding --enable-wine=yes and see what happens. 2) "ActiveScripting" of web pages - DOM manipulation through other programming languages. this is an _incredibly_ powerful concept that microsoft must have had for what... ten to fifteen years, and nobody really noticed its significance?? :) the reason for _that_ is because the main available programming languages were utterly dire (Visual Basic). to give you an idea of how powerful scripted DOM manipulation is, i've implemented a project called Pyjamas-Desktop - http://pyjd.org. it's a Desktop Widget Set UI framework. thanks to webkit-glib and pywebkitgtk, is actually _more_ feature-capable than PyQT4 and PyGTK2 put together! i'd just... _really_ like to see Pyjamas-Desktop be capable of running on top of MSHTML, on windows. especially because then it would be possible to _just_ install python (and pyjd) and, thanks to I.E. being de-facto already _on_ windows, pyjd would just... _work_. i wouldn't have to ask people to download a 30mb bundle of GTK.DLL, CAIRO.DLL, PANGO.DLL, GDK.DLL, WEBKIT.DLL, ICU38.DLL plus supporting infrastructure. the "dll hell" burden, for gtk appplications on win32, is _immense_. btw whilst i use pyjamas-desktop as an example, anything that has _remotely_ a subset of the features that pyjd covers would of course benefit from the same features [that python bindings to mshtml.dll would bring]. pyjd is a bit of a "path-finder" as it absolutely _hammers_ the underlying DOM model technology. [i had enormous difficulty finding suitable technology: pykde didn't work; pygtk2 was useless; pyqt4 was only slightly better - in the end i had to go for webkit.]
Re: question on how to do language bindings to MSHTML's COM interface
On Fri, Jan 2, 2009 at 9:17 PM, Rob Shearman wrote: > 2009/1/1 Luke Kenneth Casson Leighton : >> folks, hi, >> i have a rather intriguing issue i'd like to look at, which takes >> quite a bit of explaining as to why i'd like to go about it - if >> people are interested i can answer that, but for now i'll leave it at >> this: >> how, under linux, would i go about making an application that made >> _use_ of MSHTML.DLL's functionality, via its COM / DCE/RPC / MSRPC >> interface? >> in other words, if you are familiar with gecko / XUL, how would i go >> about making a gecko / XUL style of application, using MSHTML as the >> basis, and, once that was achieved, would it stand a chance of >> successfully running under vanilla windows, using MS's own version of >> MSHTML? > > There are a number of parts to your question: > 1. Is it OK for the application to be a winelib one (i.e. invoked via > or linked to wine, rather than being "native")? i'd much prefer native, as then two platforms would be supported rather than one. > 2. What language are you writing the application in? as little c as possible to get as quickly as possible into python. > 3. Pretty much all interaction with mshtml happens through COM > interfaces, so which interfaces are you interested in? i don't know (and would appreciate some pointers!) - i can define it as follows: everything that will give me sufficient complete control of the DOM such that i can duplicate - 100% - the XUL / Gecko project - but through MSHTML _not_ XUL / Gecko (i.e. ignore the fact that mshtml uses XUL / Gecko underneath). if you've seen Hulahop, that's what i'm ultimately after - running on both linux and windows. also, if you've seen WebKit, i'm the person who's responsible for webkit now having glib / gobject bindings to its DOM model, and i also added python bindings to _those_ bindings, such that pywebkitgtk can now be used to manipulate the DOM model. basically, i want to be able to do platform-independent "scripting" on top of MSHTML, in exactly the same way that pywebkitgtk allows platform-independent python scripting on top of webkit, and in exactly the same way that hulahop allows platform-independent python scripting on top of Gecko / XUL. that's the goal. >> yes i _am_ fully aware that, underneath, according to the wiki page on >> mshtml's implementation, mshtml underneath uses Gecko - but i don't >> _want_ to use the Gecko / XUL interface (mostly because it already >> exists! i don't like challenges that have already been done :) >> specifically, i'd _really_ like to have python bindings to the COM >> bindings to MSHTML - all under linux. >> so there are some _really_ weird esoteric "sub-questions" involved, such as: >> "what are the chances of porting pywin32 - specifically pywin32's COM >> bindings - to linux, thanks to widl and friends"? >> see http://sourceforge.net/projects/pywin32 >> many thanks for any advice and information. > > We already build a typelib for mshtml using widl and pywin32 can use > that to make bindings for use at runtime, _great_. > so it "should" work, :) > but I fail to see the need for it at the moment. now buried in that statement is a question and a half :) as you know, i don't really take on under-ambitious projects ha ha :) > At some point I'd like to see a Python output generator for widl so > that it can directly generate Python code that will communicate to a > remote process using DCE/RPC, that would be _fantastic_. it's something i've wanted to see happen for about six years. however, the lessons from my involvement in big free software projects are that i will do significant work either for my own personal benefit or if someone pays me money. i've had enough people spongeing off of my expertise, time and energy. so - if you find anyone who needs python dce/rpc, who's willing to pay for it, you know where to reach me :) > but that comes second place to getting > the C generator as close to perfect as possible at the moment. tsk, tsk - rob, rob, it's been _years_ since wine began the process of duplicating FreeDCE, a BSD-licensed _freely available_ reference implementation of DCE/RPC, which took only a few weeks of effort, back in about 2000/2001, to turn it into MSRPC (adding support for [range] and other minor details). as you're aware, i began the task of porting FreeDCE to win32 back in 2005. it was successful: however i went a bit too far with the removal of the dcethreads library (which was done with #define macros) so that is easily reverted. ncacn_ip_udp worked absolutely fine, and without
REALLY EXCITING! wine iexplore.exe http://pyjs.org/examples/
aww folks - bless you :) if all of these worked, then it means that mshtml is coming along _really_ well! i went through the kitchen sink example, and it passed - everything - with flying colours. the library unit test - passes everything! even the SVG canvas (in the addons) works! i was _just_ about to get _really_ excited, when i ran the JSONRPC example, but aw, i got this: fixme:mshtml:nsUploadChannel_SetUploadStream Unsupported aContentType argument: "application/x-www-form-urlencoded" darn, darn :) if that had worked first time, it would have been _unbelievable_ - and _so_ exciting. i liked especially the way that internet explorer is detected as Mozilla-compatible. to support IE's javascript features _would_ perhaps bit a _little_ too much :) but the real purpose of this message is to say WELL ING DONE! l.
question on how to do language bindings to MSHTML's COM interface
folks, hi, i have a rather intriguing issue i'd like to look at, which takes quite a bit of explaining as to why i'd like to go about it - if people are interested i can answer that, but for now i'll leave it at this: how, under linux, would i go about making an application that made _use_ of MSHTML.DLL's functionality, via its COM / DCE/RPC / MSRPC interface? in other words, if you are familiar with gecko / XUL, how would i go about making a gecko / XUL style of application, using MSHTML as the basis, and, once that was achieved, would it stand a chance of successfully running under vanilla windows, using MS's own version of MSHTML? yes i _am_ fully aware that, underneath, according to the wiki page on mshtml's implementation, mshtml underneath uses Gecko - but i don't _want_ to use the Gecko / XUL interface (mostly because it already exists! i don't like challenges that have already been done :) specifically, i'd _really_ like to have python bindings to the COM bindings to MSHTML - all under linux. so there are some _really_ weird esoteric "sub-questions" involved, such as: "what are the chances of porting pywin32 - specifically pywin32's COM bindings - to linux, thanks to widl and friends"? see http://sourceforge.net/projects/pywin32 many thanks for any advice and information. l.
Re: Race in rpc__cn_network_receiver()?
On Thu, Apr 20, 2006 at 03:04:21PM +1000, Luke Howard wrote: luke, you are a star: thank you for helping out with this. just for people's information (in case it's not been mentioned here - i must admit i forgot to notify [EMAIL PROTECTED]) about 6-8 months ago i did a coding-spree on freedce, with a view to achieving two, maybe three, major things: 1) porting to win32 (if you use pthreads-win32) 2) using posix final draft (7?) threads not posix draft 4. 3) proving to the people on wine and reactos that they have a hell of a lot of work to do (measured in man-decades) by not using the open group's DCE/RPC reference implementation. here's the gotchas that i encountered, and had to give up: - udp worked, but tcp did not - reactos had some problems at the time with the code behind ipconfig.exe (NetTransportEnum i think it was) - pthreads-win32 was too strictly posix compliant (!) and there is a cancellation exception or something which makes it fail in the dcethreads exception library tests, where linux does not fail (!) maybe this is related to what lukeh is talking about. *shrug*. anyway, the upshot is that freedce/latest-cvs also doesn't work on TCP, and thank you so much people for helping track down why. it's only been ten years so far: hopefully some time within the next decade, freedce will be used for some real MSRPC-related free software projects. l > >In rpc__cn_network_receiver(), rpc__socket_close() is called without an > >exception handler in place. According to IEEE Std 1003.1 (2004), close() > >is a cancellation point, see: > > > >http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_09.html > > > >Given this, a receiver thread could receive a cancel whilst closing the > >socket, which will cause to entire program to abort. (We have seen this > >occur when the RPC server is under load.) > > > >A suggested fix is to move the call to rpc__socket_close() inside the > >TRY/CATCH_ALL block that removes any pending cancels. > > It seems like the real fix is to disable cancellation within > rpc__socket_close(), because in other places the OSF code appears to > assume that this is not a cancellation point. > > Ditto for open(), socket(), etc (but not I/O routines that may block). > > I should note that the Linux dcethreads library does actually disable > cancellation for these calls, however in our own DCE thread library > we opted not to override any system calls to avoid changing semantics > when used by non-DCE code. > > -- Luke > > -- > -- -- lkcl.net - mad free software computer person, visionary and poet. --
Re: freedce-win32 - progress!
On Fri, Sep 30, 2005 at 08:41:17AM +0200, Jakob Eriksson wrote: > Luke Kenneth Casson Leighton wrote: > > >i am making the "amateur" version of progress: i just had > >echo_server.exe run for the first time on win32: echo_client.exe > >has been running successfully since this morning. > > > > > > > That's really great! yeh :) winsock is a pain in the nuts, i can't ctrl-c echo_server.exe because all signals are blocked (in the demo code, there's some #ifdef AIX stuff i just copied that), tcp/ip is going ape, rpcd.exe the endpoint mapper has no file locking, but it's progress. i've made a decision _not_ to focus on fixing the tcp/ip stuff - ncacn_ip_tcp - and instead to jump straight in to creating a ncacn_np and then a ncalrpc transport. i figured that my expertise is _not_ in fixing friggin winsock: if by the time i've done ncacn_np nobody's picked this up then i'll come back to it. ncacn_np is more important for reactos than ncacn_ip_tcp. l.
freedce-win32 - progress!
i am making the "amateur" version of progress: i just had echo_server.exe run for the first time on win32: echo_client.exe has been running successfully since this morning. rpcd.exe - the endpoint mapper - just crashed :) hmmm... recv_state_timer is indicating Ping timeout, sending "Orphan timeout", and then bombing out... hmmm... this despite echo_server.exe happily running. i don't really know enough to say what's going on here - other than to consider quite cheerfully disabling the ping rpc timer code with a _huge_ manic grin on my face :) i have had to disable TCP (ncacn_ip_tcp) for now, and will try that next. i've refrained from notifying wine-devel of status reports until i actually had something working. for your benefit: * i am using pthreads-win32, from sourceware.org, with pthread_self.c patched to set sp->implicit = 0 instead of 1. this gets the dcethreads test programs to work: whether this is a good thing remains to be seen... * i have removed the awfulness that is linux-specific (system call wrapping) which previously made dcethreads non-portable to win32: instead, you now must #define sys_pthread_create pthd4_create because dcethreads provides pthd4_xxx. * the dcethreads exception emulation library still uses setjmp and longjmp and to my surprise and horror it actually works. * i have tracked down an implementation of gettimeofday, and liberally sprinkled it across the code, statically, instead of providing a nice library routine * i have provided a win32_winsock.c "wrapper" set of code to divorce the #including of and from everything else. no, it is _not_ appropriate to put #include in the freedce header files. there are too many conflicts for that to be suitable. HANDLE, handle_t, UUID, uuid_t rpc_binding_t ain't the half of it. * i could not be arsed to write yet another version of uuid_create() and the uuid_create() function in freedce conforms to the RFC - grabs networking MAC address etc. etc. therefore, rather than rip wine code from UuidCreate(), i decided to call UuidCreate() - via a wrapper function. this _could_ therefore get interesting, if someone decides to make rpcrt4.dll utilise uuid_create() ... :) * i have NOT done any proper file locking stuff - yet - in rpcd.exe. there's a database that really needs to be locked: i aim to grab APR's apr_open(), apr_flock() etc. etc. and utilise those because i _so_ cannot be bothered to reinvent the wheel. wine-devel team: you now have a partially-working and therefore proof-of-concept port of freedce to win32. there are many next steps that can be taken from here. -- -- http://lkcl.net";>http://lkcl.net --
Re: freedce/win32 - making progress.
hello peter, thank you v. much for the advice. yes i tried -no-undefined, it got a little further but no banana :) will look at that patch, let you know if it helps On Wed, Sep 21, 2005 at 01:44:12PM +0200, Peter Ekberg wrote: > Hi! > > * Luke Kenneth Casson Leighton writes: > > okay, i'm getting somewhere, and there's a key part that i would > > appreciate some advice about: if there is anyone who knows how to > > do cross-compiling of dlls using libtool, mingw32 (in automake > > Makefile.am's) where the dlls need to link against _other_ libraries, > > i would love to hear from you. > > > > if you know of a better place to ask questions such as these, i would > > also appreciate some advice on where i should ask them. > > Perhaps libtool@gnu.org, but please be more detailed in what > you're trying to do and what you want to happen. > > > [please respond cc direct because although i am subscribed > > to the wine-devel list i have "receive postings" switched off > > because i haven't sufficient time any more to sort or delete > > hundreds of messages per day like i used to have. > > ] > > > > there are about eight libraries containing a function > > rpc__module_init_func() and these need to be LoadLibrary'd and > > GetProcAddress'd and called... > > > > ... but in endeavouring to create dlls, i am getting mad warnings from > > libtool about a) static libraries containing unresolved functions > > therefore libtool says it _could_ link against the library you > > requested, but only if you made a dynamic version and b) unresolved > > symbols for functions in other libraries, and it's all a bit of > > a vicious circle and i'd rather not bash my head against a brick > > wall for days on this. > > You could try the first patch in this post: > http://lists.gnu.org/archive/html/libtool/2005-09/msg00063.html > > It just might help... > > It could also be that it's as simple as saying -no-undefined > to libtool, that option is needed on windows. > > But as always it's hard to tell without all the gory details. > > > if this issue cannot be resolved then there are a number of nasty > > options remaining: > > > > 1) statically link binaries against the libraries, with hard-coded > > module init functions (which is what i'm doing now and i link against > > libnaf_ip.la, libcom.la, libprot_ncacn.la, libprot_ncadg.la, the list > > goes on and you have to do that for EVERY client AND server program) > > > > 2) abandon automake, work out how to manually create dlls. yes i know > > about .def files but libtool appears unconvinced. > > > > neither of these two is attractive options. > > > > anyway: what i have working so far can be found at the freedce-devel > > sf.net mailing list archives, and as of today, rpcd (the endpoint > mapper > > on port 135) works. > > > > and also when i add a NamedPipes transport (ncacn_np) it will > > be on \PIPE\epmapper, too. that'll be a fun job. no, really! > > > > tomorrow's job is to link the echo_server and test that, and if that > > works, try the echo_client too. > > > > if those work, then the free software community will have its own > > DCE/RPC reference implementation running on win32. > > > > much damn good may it do, i am sure. > > > > l. > > > > p.s. thank goodness for pthreads-win32. > > > > Cheers, > Peter -- -- http://lkcl.net";>http://lkcl.net --
freedce/win32 - making progress.
okay, i'm getting somewhere, and there's a key part that i would appreciate some advice about: if there is anyone who knows how to do cross-compiling of dlls using libtool, mingw32 (in automake Makefile.am's) where the dlls need to link against _other_ libraries, i would love to hear from you. if you know of a better place to ask questions such as these, i would also appreciate some advice on where i should ask them. [please respond cc direct because although i am subscribed to the wine-devel list i have "receive postings" switched off because i haven't sufficient time any more to sort or delete hundreds of messages per day like i used to have. ] there are about eight libraries containing a function rpc__module_init_func() and these need to be LoadLibrary'd and GetProcAddress'd and called... ... but in endeavouring to create dlls, i am getting mad warnings from libtool about a) static libraries containing unresolved functions therefore libtool says it _could_ link against the library you requested, but only if you made a dynamic version and b) unresolved symbols for functions in other libraries, and it's all a bit of a vicious circle and i'd rather not bash my head against a brick wall for days on this. if this issue cannot be resolved then there are a number of nasty options remaining: 1) statically link binaries against the libraries, with hard-coded module init functions (which is what i'm doing now and i link against libnaf_ip.la, libcom.la, libprot_ncacn.la, libprot_ncadg.la, the list goes on and you have to do that for EVERY client AND server program) 2) abandon automake, work out how to manually create dlls. yes i know about .def files but libtool appears unconvinced. neither of these two is attractive options. anyway: what i have working so far can be found at the freedce-devel sf.net mailing list archives, and as of today, rpcd (the endpoint mapper on port 135) works. and also when i add a NamedPipes transport (ncacn_np) it will be on \PIPE\epmapper, too. that'll be a fun job. no, really! tomorrow's job is to link the echo_server and test that, and if that works, try the echo_client too. if those work, then the free software community will have its own DCE/RPC reference implementation running on win32. much damn good may it do, i am sure. l. p.s. thank goodness for pthreads-win32. -- -- http://lkcl.net";>http://lkcl.net --
scared! freedce-win32 in a cvs repository near you
right. 48 hours later, i've got a first version of freedce-win32 to try out, which i'm actually a bit frightened of running. just in case it works :) thank goodness for pthreads-win32, otherwise i'd be up small creek. major things todo: in ipnaf.c (actually ipnaf_linux.c or even better ipnaf_win32.c) it's necessary to write an IP-interface-enumeration function. this function presumably will call GetIpAddrTable and go over all the addresses including netmasks and broadcast addresses returning information that the rpc runtime can then "listen" on - one by one. no, freedce doesn't go listening on 0.0.0.0 it binds to all "towers" whatever they're called. because i've commented this code out (stubbed) i am not expecting freedce-win32 to work straight off. once this is working, putting it in place of rpcrt4 should be relatively straightforward (and mostly very very mind numbingly boring) - a matter of "back-converting" and following the lead of dceport.h - microsoft's "wrapper" header file that allows application developers to port DCE/RPC applications to MSRPC. i say "lead" because it's woefully inadequate, representing about 10% of the functions and structs that actually need "converting", because dceport.h contains the "publicly accessible" functions. once _that's_ done, then there will be two remaining tasks: 1) check that pthreads-win32 works under wine (!) and if so, job done, otherwise it's _yet_ another emulation layer to write, but at least the source code of pthreads-win32 represents a damn good starting point 2) mash freedce's typelib output to be compatible with MSRPC binaries. this latter is no small task (but at least it could be considered). the data structures have been _totally_ mashed, by microsoft. gotta run. later. l. -- -- http://lkcl.net";>http://lkcl.net --
freedce-win32 - winsock joy
progress being made. am about 10-15% the way through a compile, having previously got dcethreads to use pthread-win32 by judicious ripping out of large amounts of code. fortunately, elrond has already win32'd tng, so there is stacks of code in tng's lib/util_sock.c to rip off :) -- -- http://lkcl.net";>http://lkcl.net --
found it! the "dce porting" header file
http://examples.oreilly.de/english_examples/dce/dce_nt/dceport.h.aug95 o joy o joy. also, i found _this_!!! http://sourceware.org/pthreads-win32/ which is an implementation of pthreads on win32. _also_ i found - was reminded - by judicious searching on google, that of course entegrity.com have "PC-DCE" which is of course a _commercial_ product that has done exactly the work required: making dce (1.1? 1.2.2?) compile on Win32. including exceptions, dealing with mapping the names, blah blahhh. now. anyone got any brainy ideas on how to reverse dceport.h - in particular the thingy. wotsit. TRY/FINALLY/CATCH stuff? what i'm going to try to do is take a shot at simply compiling dcethreads - on win32. if that _works_ (the tests pass) it'll be great - it will prove two things: 1) that pthreads-win32 works 2) that the dce exception handling will fly, on win32, and _that_ means pretty much zero (urk! let's hope!) work. for now. my main goal: "GetItWorking(tm)". i couldn't care less about binary compatibility for now: i want "services" - i want srvsvcd, i want winregd, i want lsarpcd, i want samrd - all compiled, all working, using TCP/IP. i can then work on rewriting freedce's ntlmsspauth.so to be a DLL (starting off as my favourite - a stub with username and password of "test"). i can then work on writing an "ncalrpc" plugin transport (which is very straightforward: a matter of filling in about 25 functions in a vector table). and also on a ncacn_np plugin transport. "working" means that other people will then be able to focus on issues like binary compatibility. guys (wine team) i have to say this. you were utterly insane to not use freedce as your starting point - and i am still bewildered by you not even looking at its source code (which is topologically identical in almost all areas, to what you've created) or the opengroup's documentation. i say that with the greatest of respect for what you've achieved without freedce. l. -- -- http://lkcl.net";>http://lkcl.net --
Re: subauth.h
On Sun, Sep 11, 2005 at 11:07:23AM +0900, Mike McCormack wrote: > > Luke Kenneth Casson Leighton wrote: > >oh maaan, that's really sad: i know what the stuff in subauth.h is all > >about, agh! > > Looking at the title of the post, I mistakenly thought that you posted a > patch for a moment... ha, sorry mike :) not yet. many windowisms to learn, which i am hoping to avoid by someone else picking up the initiative. i'm focussing on getting samba tng into reasonable shape - compiling everything under mingw32 is the first step - such that it can be plugged into an implementation of msv1_0.dll. however i am not holding out forever waiting for someone else to start msv1_0.dll. so i'm trying to track down "LsaAp"s to see if anyone - outside of microsoft - has ever written one of these. l. -- -- http://lkcl.net";>http://lkcl.net --
lib/netapi32/access.c and friends
these are standard smbclient-related and/or rpcclient-related functions. NetUserGetInfo grabs the information from a NET_USER_INFO_3 structure which is "cached" from the MSV1_0.DLL access token for example (it's a really long story). NetShareEnum() is a LANMAN function, whoopeee what fun. in nt, there are _stacks_ of functions that tie pretty much directly into samba source code. l. -- -- http://lkcl.net";>http://lkcl.net --
subauth.h
oh maaan, that's really sad: i know what the stuff in subauth.h is all about, agh! it is incredibly similar to the MSRPC "NETLOGON" stuff that's implemented in cli_nt_login_interactive, cli_nt_login_network and cli_nt_login_generic in rpc_client/cli_login.c joy. l. -- -- http://lkcl.net";>http://lkcl.net --
Re: proper nt-style authentication (reactos, wine, samba tng)
On Sat, Sep 03, 2005 at 01:09:47AM +0100, Luke Kenneth Casson Leighton wrote: >i trust that you are aware that NTLM authentication has >been provided for quite some time to external services, >in a number of ways. as you helped design one of those >methods (the winbind_auth_crap :) and guided tim in its >implementation, i am puzzled by the fact that it was not >mentioned in your message. > >there is also an additional method which has been >developed in samba 3, called ntlm_auth, which has even >been utilised by a SOC Google student. > >both these methods avoid reimplementation of NTLMSSP: >what therefore are you proposing that is, if at all, >different from this? i don't understand: perhaps you >could explain? the summary: ntlm_auth and winbindd are appropriate for Wine. due to the fact that samba cannot run on a Win32 platform, neither of those two programs are appropriate. it is therefore much more appropriate to implement the same stuff that's in NT _for_ ReactOS, and the closest bit of code to that is cli_nt_login_interactive (or similar), as MSRPC-based APIs. l.
Re: proper nt-style authentication (reactos, wine, samba tng)
ng people to some appropriate documentation on the ntlm_auth API, in order to evaluate whether it would be suitable (possibly even for reactos: i won't know until i see the API) okay. what else. oh, yes. the use of ntlm_auth and/or winbindd to "outsource" authentication is, i believe, a "temporary" measure, that allows the parallel development and maturisation of Wine / ReactOS specific code (in the case of ReactOS, that's MSV1_0.DLL) _without_ having to pull in a shed-load of code. the API is basically this: send (in unicode) a plaintext password, a domain name and a username, and receive a yes/no answer along with a NET_USER_INFO_3 structure, from which the session key and group sids can be "pulled". there's also the second function LsaLogonUserEx which is i believe "send NT+LM hashes, plus unicode domain name, unicode username, and receive..." but it would take a little reverse-engineering to double-check. so yes, in conclusion, i believe that the use of ntlm_auth and/or winbind _may_ be appropriate for Wine, is temporarily appropriate for proof-of-concept in ReactOS, but for final code in ReactOS, definitely not. that's the gist: hope this helps clarify things. l. On Fri, Sep 02, 2005 at 11:25:36PM +1000, Andrew Bartlett wrote: > On Fri, 2005-09-02 at 01:39 +0100, Luke Kenneth Casson Leighton wrote: > > I will leave the rest of this mail well aside, but I just wanted to > clarify exactly how long we have been providing NTLM authentication > services to external projects: > > > 2) write a lovely insecure method of "outsourcing" the username, > > domain and password to an external server - Samba TNG - which performs > > the authentication on your behalf and gets back "real" data. > > > > this could be done simply with a TCP connection, throw the data > > in-the-clear over to a simple temporary shim service blah blah, > > bob's your uncle. > > Like, say the winbind_auth_crap (thank Mr Potter for the name) function > in Samba's winbindd client interface, used successfully by external > projects (Squid in particular) since Samba 2.2? > > Or better still (avoiding reimplementing NTLMSSP) by calling ntlm_auth > (shipped with Samba 3.0 since release)? Oh wait, we hooked up a Google > SOC student to do just that, and it's working well! :-) > > Andrew Bartlett > > -- > Andrew Bartletthttp://samba.org/~abartlet/ > Samba Developer, SuSE Labs, Novell Inc.http://suse.de > Authentication Developer, Samba Team http://samba.org > Student Network Administrator, Hawker College http://hawkerc.net -- -- http://lkcl.net";>http://lkcl.net --
Re: proper nt-style authentication (reactos, wine, samba tng)
hi, rob, thanks for responding. On Fri, Sep 02, 2005 at 12:05:24AM -0500, Robert Shearman wrote: > Luke Kenneth Casson Leighton wrote: > >that you - the wine team - continue to reinvent an non-interoperable > >version of MSRPC, for binary-level "DCOM" interoperabiltiy ONLY, > >demonstrates quite how just as bloody stupid you are being. that _can_ > >be taken as a compliment, as i genuinely i mean it with the greatest of > >respect. > > > > > > Ok, I want to condense your huge message into a numbered list of things > you say we need: whee :) > 1. Type marshaling. > 2. IDL compiler. > 3. RPC Server. more specifically, rpc server "infrastructure" from which to develop, create, compile run and support RPC servic_es_. this you gain in the quickest possible way from FreeDCE and in the grungiest possible way from either samba or samba tng. > 4. Authentication. > 5. Services (lsa, netlogon, samr, etc). yep. > Ok, so we are we at with Wine and ReactOS? > 1: We need to implement this anyway because, like you crudely put it, we > are in the crazy business of getting real code like InstallShield and > Office 2003 to work. muhahahahah. hey, Security-Enhanced ReactOS, anyone? :) > 2. We can either use MIDL and accept the problems that go with it (like > not generating code that will compile with gcc 4.0) or we can finish > implementing proxy support in widl. okay. the amount of effort required to modify dceidl to support binary-interoperability with MIDL i do not believe to be very large, but the key bit which _will_ take some expertise (i.e. there are better people than myself to do the job) is the adding of Win32 threading model support to FreeDCE. going down THIS path will GET you code that will compile with gcc 4.0 (and mingw32) AND it would be binary interoperable with MIDL. AND it would actually work, without having to reimplement type marshalling, which if you look at FreeDCE's marshalling/unmarshalling code is SEVENTY THOUSAND LINES of immensely complex code. > 3. We have support for named pipes in the RPC server, what is the "RPC server"? i do not know of such a beast :) > but Wine doesn't > support remote named pipes and AFAIK ReactOS doesn't either. This is not > a problem that should be solved by the RPC runtime. no. it has nothing to do with the RPC runtime. you write a plugin that implements thirty or so functions (which have names like open, write, close etc. funnily enough :) and then register it with the RPC subsystem (yeh, okay, probably the rpc runtime :) e.g. the named pipe transport is called ncacn_np - network computing architecture CoNnection-based named pipes. and then MSRPC clients and MSRPC services can simply utilise that transport automatically or by name. > 4. Kai and Juan are working on delegating NTLM authentication to Samba. ah. brilliant. that's exactly what i outlined as "stage 1" and it should be done by implementing MSV1_0.DLL, for use by LSASS, see LsaLogonUser in the ntoskrnl code, it's a stub at the moment, in the reactos code. > We still need to tie this in to the RPC server though. i do not know what you mean by the "RPC server". no such entity actually exists afaik. > That should be a trivial task in comparison. yes, which is why it is mentioned about 2/3 of the way through my document, as "stage 1". > Not sure how this will fit in on ReactOS. in NTOSKRNL the LSASS implementation, which then has MSV1_0.DLL register with it. i mentioned it in my document. see the two occurrences of LsaLogonUser in reactos code. > 5. Wine isn't really interested in having those types of servers, tough luck for them. nt domain services are a necessary and integral part of supporting win32, even in a stand-alone environment. wine has _bypassed_ the MSRPC bits by instead of utilising MSRPC to split client from server they have "linked" the client-side code with its server-side code, cutting out the MSRPC runtime altogether. due to the incredible way that MSRPC (actually DCE/RPC) is designed, it is perfectly feasible to do this sort of thing, and actually have your code work (just with no networking and no distributed capability) anyway. if the wine team cannot be convinced of the necessity of adding MSRPC into the mix then wine will pretty much remain in the dark ages of interoperability along the lines of "win95 with networking removed". > but it > would be nice for the client code for those to work. I'm not sure that > porting them from Samba would be fruitful as they would fundamentally > need to tie into the registry. yes? :) and? :) i have no problem with that. i'd _love_ to see a registry service implemented in s
proper nt-style authentication (reactos, wine, samba tng)
[please note: due to its cross-discipline and cross-project nature, this message is going out to SEVERAL related project mailing lists. when replying, please take EXTRA caution not least because some of these lists are subscriber-only. also, please note: i _would_ post this also to the samba mailing lists but due to the fascist censorship in place since the 15th dec 2004, i am unable to do so. this is their decision and it is their loss. i am not asking you to respect that decision i am simply making you aware of it.] hi, out the woodwork i pop - not necessarily ready to chew anything because i know just how much work's involved, but what i did want to do was say "hi, i'm still here" and do a brain-dump of how authentication ideally needs to be implemented in reactos. at 2,600 words and 16k, this message is quite long and so i have placed a copy at http://lkcl.net/software/reports/reactos-auth.txt, just in case it doesn't make it past various mailing-list limits (i'll find out in a couple of minutes... :) it breaks down into a number of sections: 1) i describe the timescales and ways to cut them, along with some warnings and stuff. 2) amonst other stuff i outline some background as to why i am posting this to so many lists. 3) i outline a project plan and the dependencies and "optional" steps, 4) i describe a recommended implementation roadmap starting with the "minimum" requirements, and expand on some technical info and references i found, which would help with some of the "nice-to-have" steps. so. first. please do not be scared by how much work is involved, and how much code there is. it all hinges one ONE function and that one function... you are _not_ going to believe how much code that one function drags in, kicking and screaming, behind it. please also please i beg you DO NOT consider going "bleedin 'ellfire, that's so much frakkin work we can't POSSIBLY do it the way you describe, we MUST do it our own way, starting with what we know, love and have already started tackling, and can't possibly back out of what we've already done". should you choose to exercise the "NIH" option, i frakkin guarantee you that you will waste about five to eight years of your (collective) lives reinventing the wheel. "The Plan" outlined here will shave that down to about 12 to 18 months, utilising some SIX HUNDRED THOUSAND lines of pre-existing code, and later in this message i also outline a prioritisation of the necessary work to "cut down" the time to maybe about ... mmm... three or so months, by leaving out some of the "nice-to-have" stuff. actually... _all_ of the nice-to-have stuff :) that will at least "GetItWorking(tm)" and the rest of the bits can be considered at leisure once people go "god that's awful, we can't possibly leave it like that" and hopefully hopefully things will actually progress from there. remember - please: this stuff is sufficiently complicated such that you really can't afford the niceties such as "It Must Be Perfect (tm)" before it can be accepted. i've seen that shit before, and it's a baad mxxxer path to go down, especially with such complex and heavily interdependant reverse-engineering projects as reactos, wine, samba and freedce. anyway. fyi - before i begin, i should mention a couple of things: 1) this message also goes out the the apache devel mailing list (specifically the APR one) because the NT style authentication thing is the sort of thing that really _should_ be in a (special?) APR util library, along with "NT Named Pipes" - see http://en.wikipedia.org/wiki/NamedPipes one of the reasons why NT-style NamedPipes is _not_ in an APR util is because it is believed that NT-style NamedPipes do not fit the "least common denominator". by having the infrastructure outlined below, it is possible to move things "up one level" to the point where unix _has_ that denominononmination. also, if APR still has support for that god-awful program-running program called Win95, it would, with not much extra effort, be possible to port some reactos components to Win95 (!) such that ThePlan outlined here would make Win95 have proper NT-style authentication (!) now there's a horrible thought that will give MS and free software people nightmares both: upgrading Win95 by installing free software components at it. muhahahahah ahem. 2) i have filled in a number of pages on wikipedia.org. see http://en.wikipedia.org/wiki/User:lkcl. please take the time to review these pages, PLEASE help me with my totally biased "POV" (point-of-view) comments, by either editing the pages direct or by adding comments on the "talk" pages as appropriate. or alternatively dumping me in the nearest pond if ever you meet me. wikipedia is supposed to be encyclopedia-ic and some of my comments are anything but that. help!!! 3) please due to the quite broad distribution of this
Re: NPTL and Wine threads
On Tue, Feb 01, 2005 at 06:56:57AM -0800, Dan Kegel wrote: > Luke Kenneth Casson Leighton wrote: > >>>there is some code in FreeDCE which expects to be able to jump out > >>>of a cancellation handler > >> > >>Then FreeDCE should be fixed to be POSIX-compliant, I think. > >>Or is there something subtle going on here? > > > > > > the behaviour of LinuxThreads is different from NPTL. > > > > therefore, given that this cancellation thing matters for dce > > applications (the runtime relies on it being possible) i thought > > you might wish to be aware of this subtle difference in case > > Wine MSRPC or other applications also rely on it. > > > > that's all - nothing more. > > Right. Nothing subtle, then. I guess FreeDCE needs some > work to run on NPTL or other POSIX-compliant threads packages, *sigh*. yes. depends on your version of POSIX (final or draft 4). loic is working on keeping dcethreads "current", at least. there are approx 350 occurrences of pthread_somethingorother in FreeDCE, so it's not a trivial task, more's the pity. > as it's using Linuxthreads behavior that is an extension of POSIX threads. > (Surely FreeDCE doesn't *need* to longjmp out of a cancellation handler; it's the dcethreads (posix draft 4) emulation library that handles that, from what i can gather. > it can't be that hard to fix.) Thanks for the heads up. ack.
Re: MAPI32.dll.so - how to use in linux programs?
On Tue, Feb 01, 2005 at 11:20:12PM +0900, Mike McCormack wrote: > > Luke Kenneth Casson Leighton wrote: > > > [hm, if crossover office runs ms outlook, that would mean that > > codeweavers enhanced the mapi code, hm...] > > That's incorrect. We allow Office 2000/XP to install and it installs > it's own version of MAPI. ah, thanks for the correction.
Re: MAPI32.dll.so - how to use in linux programs?
On Tue, Feb 01, 2005 at 04:45:41AM -0800, Jon Griffiths wrote: > Hi Luke, > > >i noticed that Wine has a mapi32.dll.so. > > The current MAPI code is very, very far from complete. I have been > implementing it in a bottom up manner (i.e. starting with the lower > level/utility functions and working up toward constructing the higher > order functions/objects). For example, there is no table support at > present (I have an IMAPITable implemenation of sorts, but its not > ready to be committed yet, pending cleanup, tests and thread safety > checks). Providing the message stores etc must then be built on top > of these objects. > > I believe Crossover Office runs MS Outlook so you may have more > success using the native dlls at this moment in time. ah ha, you're going to like this: i'm implementing exchange 5.5 server so the native dlls don't help :) [hm, if crossover office runs ms outlook, that would mean that codeweavers enhanced the mapi code, hm...] what i am looking to do is to actually _implement_ a MAPI database, such that tables can be read etc. i haven't a clue what i'm doing, so am just copying the data from off-the-wire. but it will look very strangely familiar to anyone who has been working for some time with MAPI. i spent two hours yesterday mapping the data structures in mapidefs.h into emsabp.idl. they're identical (idl file sent to wine-devel list yesterday). > For the functions that are complete there is documentation in the > source: "make htmlpages" from the base Wine dir will format these > into readable html starting at documentation/html/index.html. thank you. > If your tests will rely on an exchange server being present it is > unlikely that you will be able to put them in the main wine tree, so > a stand alone program is likely the best solution. ha ha - i am _replacing_ exchange 5.5 server :) > As for sample programs, a quick google for "mapi sample program" and > you will get a bunch of links, including some freely downloadable MS > samples. ... i am kinda looking for something with a Makefile, compiles and links against Wine directly - under linux not windows. i am inclined to rip bits of code out of Wine (parts of header files so far) until i can get to grips with it. the thing is that by the time i am done, there will be some code that should just... integrate in a _very_ obvious manner into Wine's MAPI32.dll implementation, providing an interface to Nspi (emsabp.idl) and EcMapi (emsmdb.idl). ... it's just that at the moment i haven't really got a handle on this stuff: all i can say is i appear to be implementing MAPI tables and MAPI properties etc. but don't know what those really are. l.
Re: NPTL and Wine threads
On Mon, Jan 31, 2005 at 08:28:53PM -0800, Dan Kegel wrote: > Luke wrote: > > it's come to my attention that NPTL cannot cope with jumping out > > of a cancellation handler. > > Tough noogies, as it were. See it'll bother me later but for now it doesn't :) > > there is some code in FreeDCE which expects to be able to jump out > > of a cancellation handler > > Then FreeDCE should be fixed to be POSIX-compliant, I think. > Or is there something subtle going on here? the behaviour of LinuxThreads is different from NPTL. therefore, given that this cancellation thing matters for dce applications (the runtime relies on it being possible) i thought you might wish to be aware of this subtle difference in case Wine MSRPC or other applications also rely on it. that's all - nothing more. l.
Re: emsabp.idl - virtually identical data structures to mapidefs.h
On Mon, Jan 31, 2005 at 11:12:25PM +, Luke Kenneth Casson Leighton wrote: > hi, > > here's an IDL file i'm working on for exchange 5.5 server. p.s. when compiled with the FreeDCE dceidl compiler, it results in Nspi* functions that can actually be used to contact a remote exchange 5.5 server and obtain MAPI row information.
emsabp.idl - virtually identical data structures to mapidefs.h
hi,
here's an IDL file i'm working on for exchange 5.5 server.
as you can see it's virtually identical, structure-for-structure,
to those listed in mapidefs.h.
at about struct MAPIERROR, the similarity stops.
thoughts, anyone?
l.
--
--
http://lkcl.net";>http://lkcl.net
--
[ uuid(f5cc5a18-4264-101a-8c59-08002b2f8426),
version(56.0),
implicit_handle(handle_t rpc_binding)
] interface emsabp
{
/* this is mostly identical to wine/include/mapidefs.h,
* up until MAPIERROR, at which point it looks completely
* unfamiliar and alien.
*
* the functions in the IMAPITable only look _vaguely_
* familiar but are like, utterly different.
* really odd. same data structures. different functions.
* oh well.
*/
#define PR_ENTRYID0x0fff0102
#define PR_OBJECTID 0xfffd0003
#define PR_DISPLAY_NAME 0x3001001e
#define PT_CONTAINER_FLAGS0x3603 /* PCF_ISCONTAINER |
PCF_HASCHILDCONTAINER */
#define PR_DEPTH 0x30050003
#define PR_PARENTENTRYID 0xfffc0102
typedef unsigned short WCHAR;
typedef struct {
long element_1;
long element_2;
long element_3;
long element_4;
long element_5;
long element_6;
long element_7;
long element_8;
long element_9;
} EMS_MAPI_UNIDENTIFIED;
typedef struct {
char ab[16];
} MAPIUID;
typedef [context_handle] void *emsabp_hnd_t;
long NspiBind(
[in] handle_t element_11,
[in] long element_12,
[in, ref] EMS_MAPI_UNIDENTIFIED *element_13,
[in,out, unique] MAPIUID *element_14,
[out] emsabp_hnd_t *element_15
);
long NspiUnbind(
[in,out] emsabp_hnd_t *element_16,
[in] long element_17
);
long NspiUpdateStat(
[in] emsabp_hnd_t element_18,
[in] long element_19,
[in,out, ref] EMS_MAPI_UNIDENTIFIED *element_20,
[in,out, unique] long *element_21
);
/* typedef [ptr, string] unsigned long *EMS_SPropTagArray;*/
/* this is probably an SPropTagArray. although it doesn't match
* up with the definition in wine/includes/mapidefs.h, it also
* is the data structure i've had the most difficulty with, matching
* it to on-wire format.
*/
typedef struct {
[unique, length_is(cValues), size_is(cValues)] long *aulPropTag;
long cValues;
/*long element_24;*/
} EMS_SPropTagArray;
typedef [unique] EMS_SPropTagArray *EMS_LPSPropTagArray ;
typedef struct {
long cb;
[size_is(cb), ptr] char *lpb;
} EMS_SBinary;
typedef struct {
long dwLowDateTime;
long dwHighDateTime;
} EMS_FILETIME;
typedef struct {
long cValues;
[size_is(cValues), ptr] short *lpi;
} EMS_SShortArray;
typedef struct {
long cValues;
[size_is(cValues), ptr] long *lpl;
} EMS_MULTIVALUE_LONG_STRUCT;
typedef struct {
long cValues;
[size_is(cValues), ptr] long *lppszA; /* this doesn't look right: should be
a LPSTR */
} EMS_SLPSTRArray;
typedef struct {
long cValues;
[size_is(cValues), ptr] EMS_SBinary *lpbin;
} EMS_SBinaryArray;
typedef struct {
long cValues;
[size_is(cValues), ptr] long *lpguid; /* this doesn't look right: it should
be a GUID* */
} EMS_SGuidArray;
typedef struct {
long cValues;
[size_is(cValues), ptr] long *lpi; /* this doesn't look right: it should be
a LPWSTR* */
} EMS_MULTIVALUE_UNICODE_STRUCT;
typedef struct {
long cValues;
[size_is(cValues), ptr] EMS_FILETIME *lpft;
} EMS_SDateTimeArray;
typedef [ptr, string] unsigned short *WSTRING;
typedef [ptr, string] char *STRING;
#define PT_UNSPECIFIED 0x
#define PT_NULL 0x0001
#define PT_I2 0002
#define PT_LONG 0003
#define PT_R4 0004
#define PT_DOUBLE 0x0005
#define PT_CURRENCY 0x0006
#define PT_APPTIME 0x0007
/*#define PT_CLSID 0x0008 */
#define PT_ERROR 0x000a
/*
means in a response package, that the given attribute contains no value,
or not exists.
-> So it won't be contained in the enumeration of the attrib.
values array.
*/
#define PT_BOOLEAN 0x000b
#define PT_OBJECT 0x000d
#define PT_I8 0x0014
#define PT_STRING8 0x001e
#define PT_UNICODE 0x001f
#define PT_SYSTIME 0x0040
#define PT_CLSID 0x0048
#define PT_BINARY 0x0102 /*(the corresponding ???HDR entry is 4 bytes longer in
this case!) 1??? */
/* MV means MultiValued*/
#define PT_MV_I2 0x1002
#define PT_MV_LONG 0x1003
#define PT_MV_R4 0x1004
#define PT_MV_DOUBLE 0x1005
#define PT_MV_CURRENCY 0x1006
#define PT_MV_APPTIME 0x1007
#define PT_MV_I8 0x1014
#define PT_MV_STRING8 0x101e
#define PT_MV_TSTRING 0x101e
#define PT_MV_UNICODE 0x101f
#define PT_MV_SYSTIME 0x1040
#define PT_MV_CLSID 0x1048
#define PT_MV_BINARY 0x1102
typedef [switch_type(long)] union {
[case(PT_I2)] short i;
[case(PT_LONG)] long l;
[case(PT_BOOLEAN)] short b;
[case(PT_STRING8)] STRING lpszA;
[case(PT_BINARY)] EMS_SBinary bin;
[case(PT_UNICODE)] WSTRING lpszW;
[case(PT_CLSID), ptr] MAPIUID *lpguid;
[case(PT_SYSTIME)] EMS_FILETIM
