Ricky, I'm not sure even where to begin, because your proposed solution is simply direct embedding of one or more scripting language systems into the address space of the viewer. This ignores every single advantage of process-separated plugins and replaces them with a catalog of problems that could fill a book.
Direct embedding is what everyone does when they want a simplistic sequential scripting facility for configuring or testing a host application. We haven't proposed it ourselves not because we've had a brain lapse and can't see the simplest solution, but because the simplest solution is a complete disaster when misapplied as a general scripting facility for user-written extensions. I regularly embed Lua into host programs your way, but using that approach here would have so many disadvantages and so few advantages that I have avoided it intentionally. I'll try to summarize some key issues for you, although this will leave much unsaid. Let's call the conventional approach Direct Language Embedding, or *DLE* for brevity. - Unlike process-based plugins, DLE normally provides no concurrent execution. This means that if you want your plugins to execute concurrently you will have to implement either multitasking or threading in the viewer, both of which involve shared-state multiprogramming. - Shared-state multiprogramming is hard and error-prone because of race conditions, non-determinacy, atomicity and locking, transactional integrity, as well as deadlock, livelock, and various other highly unfriendly critters. I'm not going to justify this further here as it would take too long, but I did my research in this area, and anyone who thinks it's not hard needs to study the topic another 10 years. A reliable solution requires operating system quality design, so this job really needs to be left to the operating system, - Because shared-state multiprogramming is hard, the execution engine for client-side scripts would be large, complex, and very hard to get right. It would be literally a multi-year effort to make it work correctly, and it would always be hanging by a thread because full-coverage testing is very difficult owing to state complexity and timing dependencies. - In contrast to DLE, process-based plugins would not require any multiprogramming in the viewer at all. The C++ host side can be completely sequential in its processing of messages arriving from plugins, while the plugins themselves would run with full concurrency courtesy of the operating system. This makes for simple, reliable, and determinist viewer code. - DLE would add the entire code and data space of each embedded language runtime into the viewer's address space. This is bloat with a capital 'B', and because virtually every language has a different embedding mechanism into the hosting C++ code, there will be very little code reuse. Application stability is inversely proportional to application size, so bloating the viewer is asking for trouble, not to mention large maintenance costs. - Because of the preceding issue, in practice DLE will never allow more than one or two different language runtimes to be integrated into the viewer, and hence a language-agnostic approach to client-side scripting is effectively barred. - Without language agnosticism, a small proportion of users will be favored by the chosen language or runtime while a larger proportion will be disadvantaged and their primary skills will not be harnessed. This is a waste of people's skills, time and productivity. - Without language agnosticism, you are stuck with the facilities provided by the chosen languages or runtime. If another language has a great feature that is highly appropriate for a particular plugin task, too bad, you cannot use it. This is not very empowering, and it's an unnecessary restriction. - With DLE, the language runtimes would be directly *linked* with the viewer code, and hence licensing restrictions would come into play. This would add a whole new rat's nest of problems and effectively make embedding of many useful languages impossible, whereas the license barrier provided by sockets avoids this issue entirely. - Because all plugins run in the address space of the viewer in DLE, the opportunity for programs to interfere with each other or with the viewer is large. The viewer's stability drops to that of its least stable plugin. - With DLE, security is a nightmare. Every 3rd party plugin can potentially alter something in the viewer unless it runs sandboxed, but if it runs sandboxed then it cannot provide much (or any) local interfacing, so the power of such plugins is weak. Fast C/C++ plugins are almost out of the question for this reason. - Even if low inherent power is accepted so that sandboxing can provide security, this sandboxing can only ever be relatively weak *software sandboxing* because the plugin code and data lie within the address space of the host application. This contrasts markedly with plugins implemented as system processes, which are backed by the much stronger guarantees of isolation provided by the system's hardware MMU, even if plugins are written in bare-metal C/C++. - Because DLE runs plugins as part of the host application, the standard system tools for working with processes cannot easily be used. Even basic facilities such as killing a script would have to be implemented from scratch within the multiprogramming environment, instead of using the existing operating system kill command. The same applies to profiling, performance monitoring, debugging, optimizing, and so on --- all are much more difficult within the host application, or not even viable. (An example of this problem is provided by SL's Mono subsystem, which *still* has no means of unloading sim-side Mono assemblies after script termination, simply because the unload facility was never written.) - DLE requires that user programming of plugins be done in an environment of special calls to the host application so that plugins can cooperate properly with the host's code, and this has to be done very carefully to avoid damaging the host. It is very distant from the simplicity and safety of programming external processes, which can use stock textbook styles and the normal libraries for the language. DLE really needs client-side plugin programmers to be relative experts, while process-based plugins are safe enough for a much larger and less expert audience to program. This list of negatives could easily grow a lot longer, but to bring it to a close I'll just mention that DLE has some advantages too, but they are remarkably few in number. The most important one is that the interface to viewer facilities can be very thin and hence fast. For example, if the purpose of a plugin is to render fast GL graphics then DLE is a good approach, albeit for single plugins only. In addition, as long as no concurrent execution is required, implementing DLE can be done in under a day (at least for Lua), plus a few weeks for defining API functions. It's really that simple. However, the vast majority of plugin applications do not require bare metal speed, and what's more, concurrent execution of plugins is a mandatory requirement. This makes the tradeoff of pros versus cons weigh massively against the DLE approach for our application. Morgaine. PS. With regards to "Networking code in every plugin just to connect to the client", networking is made available by the operating system to every process through system calls or system subroutines, ie. the thinnest interface possible. There is no bloat or overhead involved. Particular languages sometimes pretty up the system interface a little, but these bindings do not normally introduce any significant overhead. Throughput and latency of socket communications is not a significant issue either --- I've measured them in an environment which emulated this design pattern, and the level of performance might even suit some rendering tasks. ====================================== On Mon, Mar 8, 2010 at 2:19 AM, Ricky <kf6...@gmail.com> wrote: > So far, barring any LL concepts, we have (as far as I know so far!) two > designs of plugin system: > 1: Socket-based plugins - as suggested by Morgaine. > 2: D-Bus or similar existing IPC tool. > 3: C++ Dynamically Shared Objects - my suggestion. > > Morgaine's design has a couple advantages that I can think of: Namely, a > distinct lawyer-approved separation of code allowing the plugins to be > released under different licenses, and a fairly stable target to design > around, meaning the socket API wouldn't typically change a lot across > releases, plugins also could be built into other external packages that > interface with the socket API to provide massive amounts of integration. > However, it's disadvantages come along: Networking code in every plugin > just to connect to the client, separate processes that have to be launched > by something - presumably the client code, but not always... > > I'm not too sure about the D-Bus design, there hasn't been much discussion, > other than it seems a lot like Morgaine's suggestion to me, just an > available existent re-usable tool. That can be a good thing, as we wouldn't > have to worry about the protocol, it would be already done. > > The last option is one I've seen references to across the 'net. All that > would be written in the client software is one or more Interface Objects > (those familiar with polymorphic design are very familiar with these,) and > then a list or two of these for different tasks. For instance, one list > could be for those that have registered themselves as Dynamic Script > interpreters. These Interfaces would then be implemented by all plugins, > depending on what tasks the plugin was spec'd to handle. Plugins would then > be located in the plugins directory in the main install, and/or in a plugins > directory under each profile. They would be loaded automatically upon > startup of the client, and their functions would be called upon event. The > event could be as simple as the last step in the render/update loops, or as > detailed as when the user clicks or touches a keyboard button. They also > are given access to various commands they can call in the client. The more > the merrier. > The advantages are that these marry right up to the client giving > full-speed access, along with the ability to get into the nitty-gritty of > the client's underpinnings. Also an easier way to code for a lot of > programmers (like myself) who, while having a passing familiarity with > networking code, are not terribly interested in writing plugins as clients > to the client.... > Disadvantages are a potential lock-in to the client's license, although I'm > not sure about that, and the API could be easily changed while patching some > other part of the client code. > > Each has it's pro's and con's. And my listing of such is by no means > comprehensive! I want to see this hashed out and the best over-all design > get implemented: Client capabilities would soar fast, and be done in a very > modular way improving SL massively. > > Ricky > Cron Stardust > > _______________________________________________ > Policies and (un)subscribe information available here: > http://wiki.secondlife.com/wiki/OpenSource-Dev > Please read the policies before posting to keep unmoderated posting > privileges >
_______________________________________________ Policies and (un)subscribe information available here: http://wiki.secondlife.com/wiki/OpenSource-Dev Please read the policies before posting to keep unmoderated posting privileges