Re: web development in D programming
bioinfornatics wrote: I am a little disapointed, so if you have many request for a web page this lib http://arsdnet.net/dcode/ is usable or not ? I am developing an implementation of the FastCGI protocol (http://www.fastcgi.com) in D2 with the main goal go have multiplexing support. However the development is slow due to the lack of time. I believe a good http server implementation would be appreciated by the community, but FastCGI server will make it possible to create a highly- scalable web applications using some of the well-known web servers (I use Lighttpd for an example). Your code is usable, but not production-ready.
Re: web development in D programming
On 09.11.2011 12:58, Dejan Lekic wrote: bioinfornatics wrote: I am a little disapointed, so if you have many request for a web page this lib http://arsdnet.net/dcode/ is usable or not ? I am developing an implementation of the FastCGI protocol (http://www.fastcgi.com) in D2 with the main goal go have multiplexing support. However the development is slow due to the lack of time. I believe a good http server implementation would be appreciated by the community, but FastCGI server will make it possible to create a highly- scalable web applications using some of the well-known web servers (I use Lighttpd for an example). Your code is usable, but not production-ready. Does Lighttpd and nginx support multiplexing? I couldn't get them to multiplex in my tests.
Re: web development in D programming
I am a little disapointed, so if you have many request for a web page this lib http://arsdnet.net/dcode/ is usable or not ? If they are any other method to do a web application in D2 programming do not hesitate and tell to me which lib used. There's also a project named Serenity http://www.digitalmars.com/d/archives/digitalmars/D/Serenity_web_framework_-_early_feedback_wanted_125473.html But it hasn't been updated in months nor is it mature. Adam uses his code in production and also provides FastCGI support.
Re: web development in D programming
SCGI is a lot easier to implement than FASTCGI and is well-supported too from webservers. I've just developed a SCGI web-service in D over lighttpd. Il giorno mer, 09/11/2011 alle 11.58 +, Dejan Lekic ha scritto: bioinfornatics wrote: I am a little disapointed, so if you have many request for a web page this lib http://arsdnet.net/dcode/ is usable or not ? I am developing an implementation of the FastCGI protocol (http://www.fastcgi.com) in D2 with the main goal go have multiplexing support. However the development is slow due to the lack of time. I believe a good http server implementation would be appreciated by the community, but FastCGI server will make it possible to create a highly- scalable web applications using some of the well-known web servers (I use Lighttpd for an example). Your code is usable, but not production-ready.
Re: web development in D
joe wrote: I currently do most of my web development in PHP, with some work in Ruby with RoR. Right now I'm starting to think about building my own stack for web dev (I already use my own MVC framework and libs in PHP), but I'd really like to move to something faster and more powerful. Java or ASP.NET are two obvious choices, but I'm currently researching other alternatives (like D). So first I was wondering if there has been any web development using D, and if there are many (or any) libraries/frameworks/other resources already available for this? As mentioned Adam D. Ruppe has nice libs for web development, he managed to make a living out of it too so he must be doing something right :) There is one other web framework for D2 called serenity, I'm not sure how far it is but you can check it out here: https://github.com/mrmonday/serenity Perhaps this is unnecessary to mention, but would you choose ASP.NET, I'd recommend against webforms and look into ASP.NET mvc. If haven't used the latter myself but I didn't like webforms too much and heard good things about the mvc framework. Good luck!
Re: web development in D
Vladimir Panteleev wrote: Regular CGI still has the overhead of starting a new process for every request. That's a very small cost - between 1 and 5 milliseconds, depending on your hardware. Next to the network latency involved in making a request, that's literally nothing - it's less than random network variation, so it has no impact on the end user experience. It also disappears quickly if your program does any processing, since it's like a line: y = mx + b. The startup cost is b, and the speed of the language is m. If x is even slightly big, reducing m means you'll get better end results than reducing b. (unless b is *huge*... like network latency. Adding an Expires: header to your responses gives the biggest gains of all.) My work application runs about 40% faster on CGI than the fastest comparable product we've found in mod_php, on average, with certain CPU intensive portions being up to 5x faster. That said, an embedded webserver /is/ even faster, but it's also harder to use. The end result depends on your app. Only actually benchmarking it can tell for sure what matters. My cgi.d provides the same interface regardless of how it is called - switching to built in httpd is easy if needed. I've had great experience with using an HTML/jQuery frontend with a D backend From what I've seen in my benchmarks, you would probably get a bigger usability boost by ditching jQuery than by ditching CGI. Even when cached, jQuery takes a while for the browser to parse - comparable to starting up a native process on the server.
Re: web development in D
I've never done any CGI stuff before, but can't you employ some kind of messaging mechanism with an already running process? Yes. I find it's quite useful when you want instant notifications on something, like a web chat application, but it could do some speed boosts too. I'm working on a post showing some of my benchmarks too on built-in vs cgi. Some of them are over 20x faster with a built in webserver! (I wonder if this is cgi's fault or if Apache is just not optimized for it... I'm changing the i/o code too so we can make sure it's not just stdio being slow.)
Re: web development in D
Vladimir Panteleev wrote: PHP without CGI is likely to be faster than D with CGI. Nope. In my last post, I did cgi/D vs builtin/D and we saw a 10 ms difference (running on my home computer. I did the same thing on the live company server just now and saw only a 2 ms difference - proper configuration and hardware helps cut it down too.) I mentioned a 40% speedup on comparable products D vs PHP. Let's list some more. One of the company websites is run by Wordpress. 246 ms per request. Another uses phpBB3. 73 ms per request. One of them uses a PHP graph library to make a quick line graph. 136 ms per request. Let's contrast to an assortment of D sites I've written for them, all on that same server. Dynamic homepage: 11 ms per request Scheduling and conferencing app: 14 ms per request Gradient generator outputting small pngs: 4 ms per request Most the D apps would be too hard to rewrite in PHP to do a direct comparison, so we'll have to settle for the 10x difference we see between these and the popular PHP packages... But the gradient generator is simple. I found a PHP program on the web that does the same thing with basic options, utilizing the GD library. I downloaded and benchmarked it 19 ms per request, creating identical images. (average of 2000 requests) Recall the D program, running on CGI, did the very same thing in 4 ms per request, about 5x faster. We can ignore the network, embrace CGI, and still *easily* crush PHP performance with anything more complicated than hello world.
Re: web development in D
On 22/05/2011 13:09, Lutger Blijdestijn wrote: joe wrote: I currently do most of my web development in PHP, with some work in Ruby with RoR. Right now I'm starting to think about building my own stack for web dev (I already use my own MVC framework and libs in PHP), but I'd really like to move to something faster and more powerful. Java or ASP.NET are two obvious choices, but I'm currently researching other alternatives (like D). So first I was wondering if there has been any web development using D, and if there are many (or any) libraries/frameworks/other resources already available for this? As mentioned Adam D. Ruppe has nice libs for web development, he managed to make a living out of it too so he must be doing something right :) There is one other web framework for D2 called serenity, I'm not sure how far it is but you can check it out here: https://github.com/mrmonday/serenity Perhaps this is unnecessary to mention, but would you choose ASP.NET, I'd recommend against webforms and look into ASP.NET mvc. If haven't used the latter myself but I didn't like webforms too much and heard good things about the mvc framework. Good luck! Thanks for mentioning it! :D I held off mentioning it due to its immaturity in comparison to Adam's stuff, I'm gradually working on it. The latest thing I've added is an ORM to remove about 100 lines of code for simple stuff (still a work in progress). As a result I'm probably going to be reworking the overall design (It started off being MVC, it's not going to be now). -- Robert http://octarineparrot.com/
Re: web development in D
On 22/05/2011 19:11, Adam D. Ruppe wrote: I've never done any CGI stuff before, but can't you employ some kind of messaging mechanism with an already running process? Yes. I find it's quite useful when you want instant notifications on something, like a web chat application, but it could do some speed boosts too. I'm working on a post showing some of my benchmarks too on built-in vs cgi. Some of them are over 20x faster with a built in webserver! (I wonder if this is cgi's fault or if Apache is just not optimized for it... I'm changing the i/o code too so we can make sure it's not just stdio being slow.) Have you tried FastCGI? It's essentially CGI but with a long running process rather than re-spawning for each request (and it still has the advantage that it can be re-spawned by the web server). I'm using it for serenity, albeit the FCGX interface rather than the interface that emulates CGI. -- Robert http://octarineparrot.com/
Re: web development in D
Robert Clipsham wrote: Have you tried FastCGI? I haven't. I read about it but concluded it would actually be just about as difficult to implement as a minimalist http server, so I never went into it. (http itself is hard to get all the corner cases right, but if you live behind something like Apache, you can get by without a full implementation. But in both cases, you have to write a fast network layer, and that's the harder part.) Besides though, there's a penalty between D in CGI and D with long lived processes but D in CGI beats PHP a lot of the time, so it still wins. I wonder how D/cgi stacks up against a long lived Python or Ruby process. I doubt they'd do better than embedded PHP, but I haven't tried them so I'm not sure.
Re: web development in D
Very interesting benchmarks!
Re: web development in D
On Sun, 22 May 2011 20:10:07 +0300, Adam D. Ruppe destructiona...@gmail.com wrote: Vladimir Panteleev wrote: Regular CGI still has the overhead of starting a new process for every request. That's a very small cost - between 1 and 5 milliseconds, depending on your hardware. Next to the network latency involved in making a request, that's literally nothing - it's less than random network variation, so it has no impact on the end user experience. Yes, of course - the difference is negligible if the server load is low. But if your website is getting enough hits to generate more requests than the server can process, technology choice matters a lot. It also disappears quickly if your program does any processing, since it's like a line: y = mx + b. The startup cost is b, and the speed of the language is m. Yes, I should have added that I meant for very simple requests. The speed advantage of a compiled language will always overtake most initialization overheads for non-trivial requests. The end result depends on your app. Only actually benchmarking it can tell for sure what matters. My cgi.d provides the same interface regardless of how it is called - switching to built in httpd is easy if needed. Is there any reason you didn't go for FastCGI or SCGI? The main advantage I can see is that CGI is much simpler for edit-compile-test iterations. I've had great experience with using an HTML/jQuery frontend with a D backend From what I've seen in my benchmarks, you would probably get a bigger usability boost by ditching jQuery than by ditching CGI. Even when cached, jQuery takes a while for the browser to parse - comparable to starting up a native process on the server. The difference here is that HTML, CSS and jQuery will only be loaded once. I was talking about AJAX web-apps. -- Best regards, Vladimirmailto:vladi...@thecybershadow.net
Re: web development in D
Vladimir Panteleev wrote: But if your website is getting enough hits to generate more requests than the server can process, technology choice matters a lot. Yeah. I've never had that happen, so I don't really know. If it happens, it's easy enough to change later. (it was a two line change just now to switch it to built-in webserver - none of the actual app code needs to change at all) Is there any reason you didn't go for FastCGI or SCGI? The biggest reason is they are harder to implement, and it's premature optimization. I didn't want to spend a lot of extra time writing code that would never be needed! Secondary benefits are simplicity and reliability. A CGI process can segfault without affecting anyone else. It can go completely wild on memory, and it doesn't matter because it's short-lived anyway. It can deployed to any server with ease - just copy the binary in there. Worst case, you add a couple lines to .htaccess. Apache (or IIS) also handles logging and other details for a CGI program. It's just simpler in a lot of ways.
Re: web development in D
On Sun, 22 May 2011 22:41:12 +0300, Adam D. Ruppe destructiona...@gmail.com wrote: We can ignore the network, embrace CGI, and still *easily* crush PHP performance with anything more complicated than hello world. Interesting. Thanks. -- Best regards, Vladimirmailto:vladi...@thecybershadow.net
Re: web development in D
On 22/05/2011 22:40, Adam D. Ruppe wrote: Vladimir Panteleev wrote: But if your website is getting enough hits to generate more requests than the server can process, technology choice matters a lot. Yeah. I've never had that happen, so I don't really know. If it happens, it's easy enough to change later. (it was a two line change just now to switch it to built-in webserver - none of the actual app code needs to change at all) Is there any reason you didn't go for FastCGI or SCGI? The biggest reason is they are harder to implement, and it's premature optimization. I didn't want to spend a lot of extra time writing code that would never be needed! FastCGI has an interface available that emulates CGI - that's not exactly harder to implement! ;) Secondary benefits are simplicity and reliability. A CGI process can segfault without affecting anyone else. It can go completely wild on memory, and it doesn't matter because it's short-lived anyway. It can deployed to any server with ease - just copy the binary in there. Worst case, you add a couple lines to .htaccess. Same for FCGI, for the most part, except it's a long running process. You can set it up to periodically re-spawn if you like, or it will automatically re-spawn when you exit (error or otherwise). Apache (or IIS) also handles logging and other details for a CGI program. Same for FCGI. It's just simpler in a lot of ways. That's debatable! :D -- Robert http://octarineparrot.com/
Re: web development in D
Robert Clipsham wrote: FastCGI has an interface available that emulates CGI - that's not exactly harder to implement What scared me was that the data comes in on a socket... so wouldn't that come with the same threading problems my simple http server has? (Where one slow request means everyone else has to wait) I guess I'll have to try it! [a little later] So I changed cgi.d to include support for fast cgi by linking in one of the C libraries. That wasn't hard. It was kinda painful to get the C library and apache module working though... Anyway, I recompiled my same in-development work application to use fcgi, so let's run the benchmarks on it and see what we got. Same page as I used in the last post. 7 ms per request under regular cgi it ran in about 20 ms. My embedded httpd served it up in 5 ms per request. Turning up the concurrent requests to 50 (I'm using the program ab for these benchmarks btw) shows me that Apache spawned more copies of the process automatically. Excellent. Got 150 ms with fcgi. 35 ms with embedded. 370 ms with plain cgi. (I wonder why the plain cgi shows weirdness here) With 100, my embedded one still wins by a lot in 90% of requests, but in the other ones, it gets brutally slow - 600 ms for 5% and 9 full seconds for 1% of them. Ouch! fcgi came in at 250 for most requests, with the longest one being a little over a second. Getting the generated Javascript file: -c 50, CGI: 300 ms -c 50, FCGI: 50 ms (nice!) -c 50, embedded: 6 ms for 95%, but one took 3 seconds. Getting the static stylesheet, still -c 50: CGI: 150 ms FCGI: 50 ms embedded: 4ms -c 1 CGI: 12 ms FCGI: 1 ms embedded: 1 ms Keep in mind, this program was *already* beating comparable mod_php applications, even in plain CGI. Now, it's just a curbstomp. Overall, that's pretty ok, there is a consistent boost over plain CGI. My simple embedded server seems to do consistently better though, until it hits the concurrency wall. (I'd be willing to bet a big part of the reason is that the embedded one uses slices for a lot of stuff. The other ones copy data at least once.) Not bad at all, but I'm not sure it's good enough for me to switch to it in general. Regardless, this proves something I've been saying though: my cgi.d adapts with ease to different environments. I'm running the same program in these benchmarks. Only difference is compiling with -version=fastcgi or -version=embedded_httpd. The app code is completely unchanged.
Re: web development in D
On Mon, 23 May 2011 03:49:49 +0300, Adam D. Ruppe destructiona...@gmail.com wrote: What scared me was that the data comes in on a socket... so wouldn't that come with the same threading problems my simple http server has? (Where one slow request means everyone else has to wait) Asynchronous networking is all the rage now. lighttpd is completely asynchronous and VERY fast. Have you heard about NodeJS? It uses asynchronous IO as the primary (only?) way to do IO. As closures are first-class JavaScript language features, async IO works pretty well there. I dream about creating something like this in D as well one day. Anyway, the idea is that nothing should block your event loop for more than a millisecond or so. If something takes longer, then either you need to look into asynchronizing it (file access?), or putting it in a separate, isolated thread. -- Best regards, Vladimirmailto:vladi...@thecybershadow.net
Re: web development in D
Vladimir Panteleev wrote: Have you heard about NodeJS? Yeah. It's actually not too much different than the way my network manager code works in D (which the embedded http server is made on) although their implementation is much better than mine - I just use select() in the event loop. The key to using it is to always return from your function as soon as possible. It will then add to your buffer and call the function again as new stuff comes in. It actually works quite well for it's simplicity. The problem with using it for the cgi apps is they aren't really written to work in that style.
Re: web development in D
Have a look at the recent thread titled 'How To Dynamic Web Rendering?' Adam Ruppe has created a package for web development with D and it seems to work like a charm.
Re: web development in D
Yea, the Dynamic Web Rendering thread just went over this, so shouldn't be hard to find that thread. It went into discussing my method and got a little long, but the summary is: You can write web apps in D using the standard CGI interface or a long lived process (embedded http server or whatever.) My code works best with standard CGI, cgi.d in here: http://arsdnet.net/dcode/ I also have an httpd.d (requires netman.d) that lets the cgi interface work with an embedded server, but it's single threaded so that can be a problem. It's not production ready. Regular cgi though works with other existing web servers, is easy to use, and actually performs quite well. (You'll find a lot of people complaining on the internet that CGI is slow, but check the date on those articles and see if they are using Perl or native compiled binaries. Usually they are very old and using an interpreter - that's why it is slow. A native binary over cgi is quite fast.) The stuff you build on top of cgi can be whatever you want. I personally use my dom.d and web.d modules for templating and some automatic code generation, but you can spin your own libs too.
Re: web development in D
joe j...@studiofortress.com wrote in message news:ir8frr$ait$1...@digitalmars.com... Second, from your own experience using D do think it would make a good or bad choice for a web development language? I'd really say that a good choice for a web development language is whatever you find to be a good choice for a language. Personally, I love D. It's my favorite language, and out of all the languages I've used (many over the years) it's really the only one I'm overall happy with (but then, I'm notoriously picky ;) ). So I find D to be an excellent choice for a web development language since I personally find D to be an excellect language. OTOH, if someone happened to absolutely hate D, then D probably wouldn't be a very good choice for them for a web development language. There is one other factor to consider, though: You need to think about what server or servers it will need to run on: If, for example, you need it to be able to run on pretty much any shared web host out there, then you're pretty much stuck with either PHP or something like Haxe that compiles down to PHP. And that defintiely won't help you with performance, and it'll only help alleviate *some* of PHP's probelms, not all of them. That's the unfortunate state of shared web hosting :/ If, OTOH, you only need it to run on one specific server and you don't have complete control over it (for example, if it's a shared web host, or if it's run by some separate IT department thet you're not part of), then naturally you'll need to see what the server supports and make your choice from those. But keep in mind one thing I've learned the hard way: Just because they say they support CGI does *not* necessarily mean they support CGI from a natively compiled langauge (which D is). So make sure to get clarification on that. And, of course, if you have total control over the server, then naturally you can choose whatever you want. Oh, but unless you do have direct access to the server and total control over it, I would recommend against ASP.NET. Not because I really have anything against ASP.NET, Windows or MS, per se. It's just that ASP.NET typically requires a Windows server, and shared Windows hosts usually don't provide any way to set up things like file permissions and internal URL rewriting. And not being able to do those things can often be a problem.
