Re: Suggestions on an XML-RPC Service using modperl?
Ciao! I'm on the modper digest list so pardon the delayed response to your input. Aaron == Aaron E Ross [EMAIL PROTECTED] writes: Aaron I've been working with a mod_perl based XML-RPC/SOAP Aaron service for a few months now and I thought I'd share some Aaron quick thoughts. Thanks! I'll take 'um. Aaron As long as you are _sure_ that you won't be writing data, Aaron in-memory will be fast and easy to code. if you use Aaron objects you can pretty easily build an interface Aaron encouraging programmers to avoid modifying shared data. Yes, I refuse to do *any* data arbitration between the localized portions of the data (if any?) and the remote or official copies of it. Copying it to a local store is meant to remove the access latency and availability of the data, although I have not yet determined there to be latency issues. There are however availability issues surround these databases. I agree, an OO interface to it will facilitate its use and certainly simplify any data format changes that are almost certain to take place over the life of the service itself. Aaron Aim for what you may need later, 5+ will be easy as long as Aaron you have some memory. I'm lobbying for excessive memory as we speak! Good point. Aaron I have found the Cache::Cache modules really easy to use, Aaron well written, documented and supported. MLDBM::Sync Aaron provides some locking, but if you really need concurrent Aaron access I would highly recommend BerkeleyDB, Aaron http://sleepycat.com/, nb this is _not_ DB_File. Mr. Turner also mentioned the Cache::* modules for this. I'm not really interested in locking the data in any way whatsoever. It's strictly read-only so I see no reason to have to manage read locks. Therefore concurrent access to an in-memory data store between all the modperl processes should be no problem, correct? Aaron You haven't explained the cacheing logic at all.. perhaps Aaron you don't need a cache? maybe just an object that reads Aaron from the data stores on startup? This is a good idea. It would eliminate the interim step of loading the data onto the local machines prior to bringing it into memory. I'm working on data access issues this week and want to be able to describe the data better as well as gain query access to it. Then I should be able to benchmark access and availability, my two primary concerns with the data. Aaron If you do need a cacheing layer that updates on some event Aaron or expiration, remember to seperate the cacheing logic and Aaron the storage as much as possible to be able to tune and to Aaron scale up later on. I'll keep this in mind. Aaron Why not write a simple object? I try to avoid tie's, b/c Aaron they are too magical for my taste, but i don't think Aaron there is any inherent overhead. The object description of the data seems to be a good way to go. I believe this will end up being the API I present to the handler for data access. I agree. Aaron I would recommend using SOAP::Lite for both XML-RPC and Aaron SOAP. While the code is unreadable, the author is Aaron responsive and helpful, and the switch from XML-RPC to SOAP Aaron couldn't be easier. You got the unreadable part right (c: Sorry. I've had occasion to use Randy Ray's RPC-XML module in the past and it functioned very well. He describes it as a reference implementation of the XML-RPC specification and does not attest to its effeciency or speed. I'll do some comparison shopping here with SOAP::Lite and see what comes out. Aaron Hope this helps, Aaron Thanks for your insight into this matter. I really appreciate your input. Peace.
Re: Suggestions on an XML-RPC Service using modperl?
Ciao! Jon == Jon Robison [EMAIL PROTECTED] writes: Jon As far as the cacheing goes, we have had extremely good luck Jon with IPC::ShareLite used to share info across mod_perl Jon processes. Have you compared this module to the Cache::* modules? I believe the Cache::SharedMemoryCache module provides a shared memory implementation, for example. I have had a few suggestions offered surrounding the use of the Cache::* modules which is why I ask. Thanks for your thoughts. Peace.
Re: Suggestions on an XML-RPC Service using modperl?
Ciao! I'm on the digest for this list, so pardon my delay in responding to everyone's comments. Chip == Chip Turner [EMAIL PROTECTED] writes: Chip Bruce W. Hoylman [EMAIL PROTECTED] writes: Ciao! I would like some input on an intranet web service I am currently in the process of designing, the core of which will be modperl on UN*X. Chip Excellent choice. This works quite well. Of course, like Chip others on this list, I might be a bit biased. I'm sort of sneaking this in the back door. Much of the development surrounding these new objectives is centered in the J2EE architecture. I'm taking a 'don't say much' approach until it's working and cranking out the data. Then I'll be able to meet any advances on architecture choices. It's not that perl/Apache is taboo here, just not looked upon as an implementation solution of choice. However it won't be the first time I have faced these pundits. Chip How often does the data change? How is it stored on the back Chip end? You may not need to cache anything if, say, you have a Chip decent SQL database on the backend. Caching never hurts, but Chip it isn't always necessary. The Cache::* modules may be of use Chip for this, though, should you still need it. You also might Chip want to consider not sharing the data in each process; the Chip complexity gained vs the memory lost by storing it in each Chip process may be a workable tradeoff. I probably would try it Chip first with no cache, then a per-process on-demand cache, then Chip finally a shared cache, in that order. I still have to determine the dynamic characteristics of the data in question. This certainly needs to be taken into account in any cacheing or local storage of the data. The remote data is Oracle-ized in some 7.x database instances. When I speak of cacheing I'm referring to taking remote data sources and making the local for some period of time, after which they must be refreshed and localized again. I don't believe the service can withstand the cost of remote database access and still be as streamlined as the requirements specify. However I will be going through a series of benchmarks to see just which data management approach is appropriate. I also am keen to the unavailability of these external, remote data sources. They seem to be 'down' at the most inopportune moments. In summation, data throughput and availability are the two most compelling reasons for considering a localized, and I emphasize READ-ONLY capture of the remote stores (there are two). Chip Five hits/second should be absolutely no problem. If you Chip expect slow clients, a mod_proxy in front of things Chip (http://perl.apache.org/guide) can help. This is a very interesting section of the guide. I recommend it to everyone simply for the sake knowing alternative configurations that include a modperl component. Chip There are a lot of options, but really, I would hold off on Chip deciding complicated caching schemes until you know what Chip throughput you get without them. Even then, I'd avoid Chip disk-based cache systems, instead preferring Cache::* if it Chip must be shared, or just global variables if it doesn't need to Chip be. This is a good point. I'm working on securing access to the data stores this week, so I should be able to measure throughput soon. Loading the in-memory stores direct from the data sources is a good idea, if necessary. This precludes any local storage of the data which precludes any refresh to disk logic, although I still will need to flush/fill memory on a regular basis. Thus I don't want to utilize per-process data stores, but rather a common shared memory represenetation that all the modperl processes can draw from Chip Can you be more specific about what the data looks like, where Chip it resides, and how expensive loading it is? I wouldn't worry Chip about optimization yet, unless you know beyond the shadow of a Chip doubt speed will be a problem. My hunch is you can do maybe Chip 50-100 hits/second on decent Intel hardware via the Frontier Chip modules, so I don't think performance will be a problem. This Chip is unverified, though; I really need to benchmark it sometime. Chip Maybe others have pushed Frontier to its speed limits? I'll be able to provide some numbers and the actual data representation later on this week. I agree on performance numbers. I'm getting awesome throuput now acording to ApacheBench, but that's only with my skeletal handler. As it I fill it out, I plan on benchmarking each significant piece of functionality for comparison purposes. Should be fun. Chip Chip Thanks so much for your insightful comments. I appreciate the time you took to respond to my queries. Peace.
Re: Suggestions on an XML-RPC Service using modperl?
Even then, I'd avoid disk-based cache systems, instead preferring Cache::* if it must be shared, or just global variables if it doesn't need to be. Cache::FileCache is disk-based, and it is the fastest of the Cache:: options for most data sets. There was a thread a little while back about data sharing that showed the top performers to be Cache::Mmap and IPC::MM. Cache::Cache and MLDBM::Sync should be more than fast enough for all but the most highly optimized systems. - Perrin
Re: Suggestions on an XML-RPC Service using modperl?
As far as the cacheing goes, we have had extremely good luck with IPC::ShareLite used to share info across mod_perl processes. IPC::ShareLite is not as fast as some of the other options, especially when dealing with a large data set. The disk-based options tend to be faster. - Perrin
Re: Suggestions on an XML-RPC Service using modperl?
As far as the cacheing goes, we have had extremely good luck with IPC::ShareLite used to share info across mod_perl processes. --Jon R. Chip Turner wrote: Bruce W. Hoylman [EMAIL PROTECTED] writes: Ciao! I would like some input on an intranet web service I am currently in the process of designing, the core of which will be modperl on UN*X. Excellent choice. This works quite well. Of course, like others on this list, I might be a bit biased. The service itself is to access a couple of back end data stores given parameters received in an XML-RPC request, then return the results in an XML-RPC formated response. The data from the back end sources will be loaded into memory at service initialization, for fast access. The data is small enough and memory plentiful enough to allow this. How often does the data change? How is it stored on the back end? You may not need to cache anything if, say, you have a decent SQL database on the backend. Caching never hurts, but it isn't always necessary. The Cache::* modules may be of use for this, though, should you still need it. You also might want to consider not sharing the data in each process; the complexity gained vs the memory lost by storing it in each process may be a workable tradeoff. I probably would try it first with no cache, then a per-process on-demand cache, then finally a shared cache, in that order. That's pretty much it in terms of the high level data flow. It has to be relatively fast, OTO 5+ requests/sec. as a relative volumetric. This should be quite easy. I don't have the necessary setup handy to benchmark it, but I imagine you can easily achieve performance at that level using Frontier::RPC inside a mod_perl handler. We typically use custom code for interfacing the handler, but IIRC the Frontier module comes with a mod_perl handler that, if not enturely suitable, is easily modified to your needs. I'm going to use modperl due to the embedded perl interpreter characteristics it provides, allowing initialization overhead to be incurred at startup. I also wish to use an in-memory, read-only hash structure shared across all modperl processes for access to the cached back end data, rather than making expensive calls to these stores for each request. Again, throughput is critical. Five hits/second should be absolutely no problem. If you expect slow clients, a mod_proxy in front of things (http://perl.apache.org/guide) can help. I would like your thoughts on the cache management concept of the service. I'm looking at MLDBM::Sync as the mechanism for managing the filesystem representation of the in-memory hash content. What to manage the in-memory structure itself in terms of accessing its content? Is a Tie structure too expensive? I want to end up with a single structure accessible to all of the modperl processes, loaded at service Chip -- Chip Turner [EMAIL PROTECTED] Red Hat Network
Suggestions on an XML-RPC Service using modperl?
Ciao! I would like some input on an intranet web service I am currently in the process of designing, the core of which will be modperl on UN*X. The service itself is to access a couple of back end data stores given parameters received in an XML-RPC request, then return the results in an XML-RPC formated response. The data from the back end sources will be loaded into memory at service initialization, for fast access. The data is small enough and memory plentiful enough to allow this. That's pretty much it in terms of the high level data flow. It has to be relatively fast, OTO 5+ requests/sec. as a relative volumetric. I'm going to use modperl due to the embedded perl interpreter characteristics it provides, allowing initialization overhead to be incurred at startup. I also wish to use an in-memory, read-only hash structure shared across all modperl processes for access to the cached back end data, rather than making expensive calls to these stores for each request. Again, throughput is critical. I would like your thoughts on the cache management concept of the service. I'm looking at MLDBM::Sync as the mechanism for managing the filesystem representation of the in-memory hash content. What to manage the in-memory structure itself in terms of accessing its content? Is a Tie structure too expensive? I want to end up with a single structure accessible to all of the modperl processes, loaded at service startup. This service will ultimately be registered within a UDDI/SOAP framework, FYI. However this will not be in the first incarnation of the service itself. Thoughts and comments welcome. Obviously this is an early brainstorm (more like a drizzle) but I hope to get a few stimulating comments from this most excellent resource, the list. Peace.