I'm in the very early stages of founding a startup, and currently we're in talks about what type of technology to be using. As there aren't too many people on board now, and none besides myself that are technical, there is a pretty blank slate for development as of now. After playing around with Erlang for the past couple of days I'm starting to think about the pros/cons of using Erlang.
Anyways, I have questions pertaining to how Erlang/ErlyWeb addresses certain needs and issues of scaling. I thought it might be more useful for a greater number of people if instead of speaking solely about our needs for our web app, we use an example people might be familiar with and get more from: YouTube. Here are some notes on a Google Tech Talk that was given, which also contains a link to the tech talk video: http://kylecordes.com/2007/07/12/youtube-scalability/ Here are some questions that come to mind when thinking about what they faced: 1. Their main reason for choosing Python as their language is because development speed and productivity was (and is) very important for various reasons (one important one was probably showing prototypes and their progresses to potential investors). Erlang seems to be a very productive environment for some applications, but how does it stand up for web app development? Using Python, PHP, or Ruby has the added benefit of having lots of developers to draw from their experiences -- how would Erlang compare? 2. Serving content was of course a big issue. They moved to lighttpd from Apache for their videos and got a lot of performance increases (they use CDNs for their most popular content). Yaws seems great for dynamic content, but is there anything better for the static content? 3. Another major issue YouTube ran into was serving their thumbnails. Each video can have about 4 thumbnails and each page, while having only one video playing, could have 40-60 thumbnails per page. Thumbnails, unlike video, aren't distributed across hundreds of machines (due to their size). So, you have lots of requests per second and tons of disk seeks. Apache didn't really work out, so they then went to a modified version of lighttpd that put the disk reads into worker threads -- but they ran into issues with that as well. They finally went to Google's BTFE. How would Erlang look at this problem -- would Yaws help at all? 4. They use MySql to store metadata, but as the site got huge, they ran into some issues with it. They went from having one database to doing db replication. That caused issues eventually, one of them being that spreading read load (being asynchronous) caused the database to serve outdated data from time to time. They then went to partitioning the database into shards. Overall, though, they're running into issues due to the fact that they're trying to do more and more stuff with the data. For example, recommendation systems and data mining gets really hard on 100s of millions of views per day, so they are needing some solutions that can do parallel queries and handle distributed computation. Would Mnesia be able to help handle these issues, or anything else from the Erlang world that could deal with these issues? So, those are a few general questions I'm sure all of us newly looking at Erlang for web development might ask. Looking forward to hearing some answers and opinions -- thanks in advance. best, Mojo Talantikite --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "erlyweb" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/erlyweb?hl=en -~----------~----~----~----~------~----~------~--~---
