Welcome back Darren! Seems like we have a lot in common, I tend to leave "no's and not's" quite often!
Since your experience was based on older CS 2.x system, I wonder how different (better or worse) it is now? Are we closer to GoDaddy hybrid solution? -----Original Message----- From: Frank Zhang [mailto:frank.zh...@citrix.com] Sent: Wednesday, August 21, 2013 6:59 PM To: dev@cloudstack.apache.org Subject: RE: Introduction Welcome back! > -----Original Message----- > From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] > Sent: Wednesday, August 21, 2013 12:17 PM > To: dev@cloudstack.apache.org > Subject: Introduction > > All, > > I want to introduce myself to the Apache CloudStack community. I've > have had a long love/hate relationship with CloudStack over the years, > but recently I've decided to fully dedicate myself to working on this > platform (again). > > Just a little about me, well okay, this might be a little long. I've > been a Linux user for a good amount of time. I've ran Linux as my > primary OS for over 15 years. Both personally and professionally, my > desktops/laptops have always ran Linux. As soon as I discovered Linux I also > fell in love with virtualization. > I've ran the gamut in terms of virtualization technologies. Type 1, > Type 2, containers, etc. I have a particular fondness for Xen though. > I've been a Xen user since the 2.x days. > > Couple years after I got into Linux I discovered programming. After > so many failed "./configure && make" I figured I'd should probably > learn C/C++ to figure out why the heck nothing ever "make'd" right. I > worked for awhile running a lab for developers that were writing stuff > in Java. Again I learned Java to figure out why the heck their stuff > kept failing to build and run. After I learned Java I took the > professional route of leaving the sysadmin stuff and started programming > distributed systems. > > As means to an end I developed a system that would take code builds, > deploy them to Xen thin provisioned VMs. So a developer could go to a > portal, click some stuff and get a clean VM that optionally had a code > build on it. We would then running automated integration tests against it. > This was around 2006. > About a year or 2 later I discovered EC2 and was like "holy crap, > that's exactly what I want." Then I found eucalpytus and realized > there was a whole industry around this stuff. IaaS is basically the > trifecta for me. I get to combine my love of Linux, virtualization, > and programming distributed system. Basically since then I've been > absolutely obsessed with IaaS. > > Around 2010 I got a job at Go Daddy specifically to build their public > Cloud product. Before I even got there they had already decided to go > with vmops/cloud.com. I'm pretty sure they were already cloud.com when I > started. > At Go Daddy with a core team of about 5 guys (1 UI guys, 2 java guys, > 2 sysengs) we built three generations of clouds. The first generation > of cloud was basically CloudStack 2.2++. It was CloudStack 2.2 with a > new UI, billing system, storage subsystem, and a tons of operational > tools. That was the initial public launch of our Cloud. > > The second generation was "CloudStack like." Basically we had a lot > of problems with CloudStack at the time. First, it was just a > complete pain to operationalize. Trying to hand off CloudStack to our > ops and support teams was practically impossible. Basically our level > 1 and 2 support would be very efficient at escalating all issues to level 3 > (my team). > Typically the sysadmins would see something is stuck, they'd look at a > log (that made no sense to them at all), see a 100 line stack trace > and then they'd just toss the issue to the developers. Also, their > was no visibility to the general health of CloudStack. If you asked ops, > "How's production doing right now?" > Their answer would be, "Well I can ping the UI and the java processes > are still running." In developer terms that is kinda like saying, > "Well it compiles so it must work fine." > > Secondly, we had a hard time extending the platform. We always seemed > to be a little ahead of the curve in how we wanted to implement the cloud. > How we wanted to run storage and networking didn't match at all what > CloudStack was at that time. So we embarked on improving it. It was > just way too difficult. Basically we felt like 70% of our dev time > was just fighting the framework. So we made the choice to abandon the > platform. We started rewriting the the entire platform. We did this > piece meal over time. We'd redo a subsystem, put that in production, > and then move onto the next subsystem. This ended up being very > tricky because we had mixture of CloudStack and our own stuff. We > also always tried to do no downtime deployments. So we would almost > never lock people out of the UI and most deployments were during > business hours. We basically finished most of this work around summer > last year. I gave a talk at OSCON 2012 [1] regarding what we built. > A couple months later, solely based on business direction, we decided > to stop selling the public cloud (it really wasn't shutdown until > about May this year. I'm proud to say it ran for almost 8 months with live > customers, some quite large, with no incident and no dedicated ops team). > > After we shutdown the public cloud, we turned our effort to building a > private cloud to run Go Daddy on top of it. This marked the third > generation of the cloud. At this point the IaaS system was 100% our code. > The problem was it inherited the CloudStack DB schema and with it a > lot of cruft in dealing with being compatible with CloudStack. So we > started a clean up of the code, which ended up being a huge refactor. > Our goal was to be able to support all of Go Daddy running on it. > That meant we needed a very flexible and extensible platform to > support a very heterogeneous environment. (We had a very scalable L3 > network design that I've yet to see any IaaS system be able to support > too.) Additionally it needed to plug into all other systems in Go > Daddy for monitor, tracking, and network management (IPAM, etc). This was by > far my favorite cloud to build. > Really, really complex. The goal was to build a system that could > deploy over 1 million VMs in less than a day. (I think we did it too, > unfortunately I quit before I could fully prove it. We got the system > to the point of deploying 1000 real VMs in 1 minute. I just never had the > time to fully run the 1m test). > > So I've done a lot with cloud provisioning systems. I was given an > incredible opportunity at Go Daddy that most people don't get. I was > allowed to build an IaaS system from scratch and given millions and > millions in hardware to build and test it out and run real production > work loads on it. I very much liked my cloud I built. Unfortunately, > despite my best efforts, it was all proprietary. I've decided to > abandon all the proprietary stuff I've done and try to do my best to > help out the open source IaaS landscape. After looking at all the > stacks out there, I honestly believe CloudStack is the best one. I reached > out to Citrix and they have been so gracious as to employee me full time to > work on CloudStack. > > So expect to hear a lot from me. I'm initially very interested at > improving the core architecture of CloudStack. Just a couple random > things. 1) I'm here to code. Talking is great, collaboration and > information sharing is important, but I'm not here just to talk. I'm > here to actually get stuff done. 2) I've only ran production clouds. > None of this dev/test/poc "optimize for innovation" crap. I want a > system I can run in production. So stability, upgrades, making > sysadmins happy, is very important to me. 3) I'm very pragmatic and > practical about how I implement stuff. I always have the 2 year > vision, but I'm very interested in what I can deliver in the next 3-6 > months. I enjoy delivering solutions far more than coding. 4) I'm really bad > at typing emails and make tons of typos and forget important words like "not" > or "no" all the time. > > I really look forward to gaining the trust and respect of this > community and, more importantly, improving CloudStack. I'm based out > of Phoenix, AZ area, if anybody happens to be in this fabulous part of > the country hit me up and we'll get together. > > Darren Shepherd > > [1] You can see my crappy slides for my OSCON talk at > http://www.oscon.com/oscon2012/public/schedule/detail/24035 . They > were really bad, basically I created them the night before. Did I > mention I'm good at procrastination too. Also that has the only known > picture of me on the internet.
