On 03/08/2014 09:23, Joost Roeleveld wrote: > On Saturday 02 August 2014 16:53:26 James wrote: >> Alan McKinnon <alan.mckinnon <at> gmail.com> writes: >>> Well, we've found 2 projects that at least in part seek to achieve our >>> general goals - chronos and Martin's new project. >>> Why don't we both fool around with them for a bit and get a sense of >>> what it will take to add features etc? Then we can meet back here and >>> discuss. Always better to build on an existing foundation >> >> Mesos looks promising for a variety of (Apache) reasons. Some key >> technologies folks may want google about that are related: >> >> Quincy (fair schedular) >> Chronos (scheduler) >> Hadoop (scheduler) > > Hadoop not a scheduler. It's a framework for a Big Data clustered database. > >> HDFS (clusterd file system) > > Unless it's changed recently, not suitable for anything else then Hadoop and > contains a single point of failure. > >> http://gpo.zugaina.org/sys-cluster/apache-hadoop-common >> >> Zookeeper (Fault tolerance) >> SPARK ( optimized for interative jobs where a datase is resued in many >> parallel operations (advanced math/science and many other apps.) >> https://spark.apache.org/ >> >> Dryad Torque Mpiche2 MPI >> Globus tookit >> >> mesos_tech_report.pdf >> >> It looks as though Amazon, google, facebook and many others >> large in the Cluster/Cloud arena are using Mesos......? >> >> So let's all post what we find, particularly in overlays. > > Unless you are dealing with Big Data projects, like Google, Facebook, Amazon, > big banks,... you don't have much use for those projects.
My wife works in BigData for real, she and Joost speak the same language, I don't :-) She reckons Big Data is like teenage sex - everyone says they are doing it and no-one really does ;-D > Mesos looks like a nice project, just like Hadoop and related are also nice. > But for most people, they are as usefull as using Exalytics. A bit OT, but it might be worthwhile for interested persons to get good ebuilds going for these projects. Someone will use it on Gentoo, and it will add value to the project. Much like gems and other business-oriented packages benefit > > A scheduler should not have a large set of dependencies that you wouldn't use > otherwise. That makes Chronos a non-option to me. > > Martin's project looks promising, but doesn't store the schedules internally. > For repeating schedules, like what Alan was describing, you need to put those > into scripts and start those from an existing cron. Sounds like a small feature-add. If Martin did his groundwork correctly[1] then the core logic will work and it's just a case of adding some persistence and loading the data back in on demand > Of the 2, I think improving Martin's project is the most likely option for me > as it doesn't have additional dependencies and seems to be easily implemented. Don't forget Martins is the guy who does eix. Street cred? check Knows Gentoo? check [1] I only say it this way as I haven't evaluated his code at all yet so have no idea how far Martin has taken it -- Alan McKinnon alan.mckin...@gmail.com
