many: we have a processing pipeline with ~10 stages (one C++ program per stage usually), batch processing (almost-)all pairs of files in the dataset. the dataset contains >10K files at the moment, so a couple of hundreds of millions of program executions would be my definition for "many" in this case :-)
I'll start with few machines with deploy scripts and a small subset of the dataset just to get the hang of it. It's a bit difficult to comprehend the stack, with all the possible options and combinations, though. If I have a main Python script that generates all the processing pipeline commands (that can be simply executed via shell), should I use a specific framework (like Hydra)? Or maybe use raw mesos? Or maybe I should write my own framework? On Wed, Jul 23, 2014 at 2:25 PM, Tomas Barton <barton.to...@gmail.com> wrote: > Define many :) If you want to use some provisioning tools like Puppet, > Chef, Ansible... there are quite a few modules to do this job: > > http://mesosphere.io/learn/#tools > > If you have only a few machines, you might be fine with deploy scripts. > > An example of MPI framework is here: > > https://github.com/mesosphere/mesos-hydra > > > > > On 23 July 2014 12:26, Itamar Ostricher <ita...@yowza3d.com> wrote: > >> Thanks Tomas. >> >> ldconfig didn't change anything. make still failed. >> >> But the Debian packaged installed like a charm, so I'm good :-) >> Now I just need to figure out how to use it... >> (going to start with [1], unless anyone chimes in with a better >> recommended starting point for a mesos-newbie who is trying to set up a >> cluster of GCE instances in order to distribute execution of *many* C++ >> programs working on a large dataset that is currently stored in Google >> Cloud Storage.) >> >> [1] http://mesos.apache.org/documentation/latest/deploy-scripts/ >> >> >> On Wed, Jul 23, 2014 at 11:55 AM, Tomas Barton <barton.to...@gmail.com> >> wrote: >> >>> Hi, >>> >>> that's quite strange. Try to run >>> >>> ldconfig >>> >>> and then again make. >>> >>> You can find binary packages for Debian here: >>> http://mesosphere.io/downloads/ >>> >>> Tomas >>> >>> >>> On 23 July 2014 10:09, Itamar Ostricher <ita...@yowza3d.com> wrote: >>> >>>> Hi, >>>> >>>> I'm trying to do a clean build of mesos for the 0.19.0 tarball. >>>> I was following the instructions from >>>> http://mesos.apache.org/gettingstarted/ step by step. Got to running >>>> `make`, which ran for quite a while, and exited with errors (see the end of >>>> the output below). >>>> >>>> Extra env info: I'm trying to do this build on a 64-bit Debian GCE >>>> instance: >>>> itamar@mesos-test-1:/tmp/mesos-0.19.0/build$ uname -a >>>> Linux mesos-test-1 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux >>>> >>>> Assistance will be much appreciated! >>>> Alternatively, I don't mind using precompiled binaries, if anyone can >>>> point me in the direction of such binaries for the GCE environment I >>>> described :-) >>>> >>>> tail of make output: >>>> ---------------------------- >>>> >>>> libtool: link: warning: >>>> `/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/libgflags.la' seems >>>> to be moved >>>> *** Warning: Linking the shared library libmesos.la against the >>>> *** static library ../3rdparty/leveldb/libleveldb.a is not portable! >>>> libtool: link: warning: >>>> `/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/libgflags.la' seems >>>> to be moved >>>> libtool: link: g++ -fPIC -DPIC -shared -nostdlib >>>> /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o >>>> /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o -Wl,--whole-archive >>>> ./.libs/libmesos_no_3rdparty.a ../3rdparty/libprocess/.libs/libprocess.a >>>> ./.libs/libjava.a -Wl,--no-whole-archive >>>> ../3rdparty/libprocess/3rdparty/protobuf-2.5.0/src/.libs/libprotobuf.a >>>> ../3rdparty/libprocess/3rdparty/glog-0.3.3/.libs/libglog.a >>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib >>>> ../3rdparty/leveldb/libleveldb.a >>>> ../3rdparty/zookeeper-3.4.5/src/c/.libs/libzookeeper_mt.a >>>> /tmp/mesos-0.19.0/build/3rdparty/libprocess/3rdparty/glog-0.3.3/.libs/libglog.a >>>> /usr/lib/libgflags.so -lpthread >>>> /tmp/mesos-0.19.0/build/3rdparty/libprocess/3rdparty/libev-4.15/.libs/libev.a >>>> -lsasl2 /usr/lib/x86_64-linux-gnu/libcurl-nss.so -lz -lrt >>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.7 >>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu >>>> -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu >>>> -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. -lstdc++ -lm >>>> -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o >>>> /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o >>>> -pthread -Wl,-soname -Wl,libmesos-0.19.0.so -o .libs/ >>>> libmesos-0.19.0.so >>>> libtool: link: (cd ".libs" && rm -f "libmesos.so" && ln -s " >>>> libmesos-0.19.0.so" "libmesos.so") >>>> libtool: link: ( cd ".libs" && rm -f "libmesos.la" && ln -s "../ >>>> libmesos.la" "libmesos.la" ) >>>> g++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" >>>> -DPACKAGE_VERSION=\"0.19.0\" -DPACKAGE_STRING=\"mesos\ 0.19.0\" >>>> -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" >>>> -DVERSION=\"0.19.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 >>>> -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 >>>> -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 >>>> -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_PTHREAD=1 -DMESOS_HAS_JAVA=1 >>>> -DHAVE_PYTHON=\"2.7\" -DMESOS_HAS_PYTHON=1 -DHAVE_LIBZ=1 -DHAVE_LIBCURL=1 >>>> -DHAVE_LIBSASL2=1 -I. -I../../src -Wall -Werror >>>> -DLIBDIR=\"/usr/local/lib\" -DPKGLIBEXECDIR=\"/usr/local/libexec/mesos\" >>>> -DPKGDATADIR=\"/usr/local/share/mesos\" -I../../include >>>> -I../../3rdparty/libprocess/include >>>> -I../../3rdparty/libprocess/3rdparty/stout/include -I../include >>>> -I../3rdparty/libprocess/3rdparty/boost-1.53.0 >>>> -I../3rdparty/libprocess/3rdparty/protobuf-2.5.0/src >>>> -I../3rdparty/libprocess/3rdparty/picojson-4f93734 >>>> -I../3rdparty/libprocess/3rdparty/glog-0.3.3/src >>>> -I../3rdparty/leveldb/include -I../3rdparty/zookeeper-3.4.5/src/c/include >>>> -I../3rdparty/zookeeper-3.4.5/src/c/generated -pthread -g -g2 -O2 -MT >>>> local/mesos_local-main.o -MD -MP -MF local/.deps/mesos_local-main.Tpo -c -o >>>> local/mesos_local-main.o `test -f 'local/main.cpp' || echo >>>> '../../src/'`local/main.cpp >>>> mv -f local/.deps/mesos_local-main.Tpo local/.deps/mesos_local-main.Po >>>> /bin/bash ../libtool --tag=CXX --mode=link g++ -pthread -g -g2 -O2 >>>> -o mesos-local local/mesos_local-main.o libmesos.la -lsasl2 -lcurl -lz >>>> -lrt >>>> libtool: link: g++ -pthread -g -g2 -O2 -o .libs/mesos-local >>>> local/mesos_local-main.o ./.libs/libmesos.so /usr/lib/libgflags.so >>>> -lpthread -lsasl2 /usr/lib/x86_64-linux-gnu/libcurl-nss.so -lz -lrt >>>> -pthread >>>> ./.libs/libmesos.so: error: undefined reference to 'dlopen' >>>> ./.libs/libmesos.so: error: undefined reference to 'dlsym' >>>> ./.libs/libmesos.so: error: undefined reference to 'dlerror' >>>> collect2: error: ld returned 1 exit status >>>> make[2]: *** [mesos-local] Error 1 >>>> make[2]: Leaving directory `/tmp/mesos-0.19.0/build/src' >>>> make[1]: *** [all] Error 2 >>>> make[1]: Leaving directory `/tmp/mesos-0.19.0/build/src' >>>> make: *** [all-recursive] Error 1 >>>> >>> >>> >> >
