On Nov 8, 2007 12:23 PM, mojo.talantikite <[EMAIL PROTECTED]> wrote: > > I'm in the very early stages of founding a startup, and currently > we're in talks about what type of technology to be using. As there > aren't too many people on board now, and none besides myself that are > technical, there is a pretty blank slate for development as of now. > After playing around with Erlang for the past couple of days I'm > starting to think about the pros/cons of using Erlang. > > Anyways, I have questions pertaining to how Erlang/ErlyWeb addresses > certain needs and issues of scaling. I thought it might be more useful > for a greater number of people if instead of speaking solely about our > needs for our web app, we use an example people might be familiar with > and get more from: YouTube. Here are some notes on a Google Tech Talk > that was given, which also contains a link to the tech talk video: > http://kylecordes.com/2007/07/12/youtube-scalability/ > > Here are some questions that come to mind when thinking about what > they faced: > > 1. Their main reason for choosing Python as their language is because > development speed and productivity was (and is) very important for > various reasons (one important one was probably showing prototypes and > their progresses to potential investors). Erlang seems to be a very > productive environment for some applications, but how does it stand up > for web app development? Using Python, PHP, or Ruby has the added > benefit of having lots of developers to draw from their experiences -- > how would Erlang compare?
Erlang definitely has fewer developers using it for webapps that PHP, Python or Ruby. My subjective take is that, having done PHP programming and played with RoR, ErlyWeb can be just as productive if not more and its code is very concise and expressive. I rarely feel that I'm writing repetitive code that doesn't directly solve the problem domain. Plus, I prefer Erlang's functional style to imperative one. I think FP makes your code more readable and debuggable. Erlang has fewer libraries than more popular languages, which may or may not affect you depending on your app's needs, but I think that the fewer libraries isn't as big an issue as some people say. Erlang has a variety of ways for talking to other lanugages and I'd rather write most of my code in Erlang in occasionally outsource some functionality to another language than lock my whole app to a language I don't like as much just because of the existence of a couple of libraries for it. Btw, Erlang also has some tools that other languages don't have -- most notably, elegant concurrency and Mnesia. > > 2. Serving content was of course a big issue. They moved to lighttpd > from Apache for their videos and got a lot of performance increases > (they use CDNs for their most popular content). Yaws seems great for > dynamic content, but is there anything better for the static content? Use lighttpd for static content and Yaws (or MochiWeb) for dynamic content. > > 3. Another major issue YouTube ran into was serving their > thumbnails. Each video can have about 4 thumbnails and each page, > while having only one video playing, could have 40-60 thumbnails per > page. Thumbnails, unlike video, aren't distributed across hundreds of > machines (due to their size). So, you have lots of requests per > second and tons of disk seeks. Apache didn't really work out, so they > then went to a modified version of lighttpd that put the disk reads > into worker threads -- but they ran into issues with that as well. > They finally went to Google's BTFE. How would Erlang look at this > problem -- would Yaws help at all? I'm not too familiar with this problem, but if it's static data, it's generally better to serve if from lighttpd than Yaws. > > 4. They use MySql to store metadata, but as the site got huge, they > ran into some issues with it. They went from having one database to > doing db replication. That caused issues eventually, one of them > being that spreading read load (being asynchronous) caused the > database to serve outdated data from time to time. They then went to > partitioning the database into shards. Overall, though, they're > running into issues due to the fact that they're trying to do more and > more stuff with the data. For example, recommendation systems and > data mining gets really hard on 100s of millions of views per day, so > they are needing some solutions that can do parallel queries and > handle distributed computation. Would Mnesia be able to help handle > these issues, or anything else from the Erlang world that could deal > with these issues? Mnesia is more suited for storing live session data than large amounts of persistent data. And Mnesia isn't really suited for data mining -- just for simple queries. However, Erlang helps in that it allows you to connect to multiple databases using its great connection pooling capabilities (a process is spawned for each database connection, and a dispatcher process is responsible for routing queries to connection processes). If I were building YouTube in Erlang, I would consider doing all replication and partitioning work in the application code rather than relying on the database engine. > > So, those are a few general questions I'm sure all of us newly looking > at Erlang for web development might ask. Looking forward to hearing > some answers and opinions -- thanks in advance. Hope this helps, Yariv --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "erlyweb" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/erlyweb?hl=en -~----------~----~----~----~------~----~------~--~---
