Thanks for jumping in here. Any improvements to the wiki are greatly appreciated.
-Jeff On Thu, Nov 13, 2014 at 2:07 AM, Edward J. Yoon <[email protected]> wrote: > Done! > > On Thu, Nov 13, 2014 at 6:05 AM, Stratos Dimopoulos > <[email protected]> wrote: >> Thanks Edward! >> My wiki login name is StratosDimopoulos - I'll edit the page and add a link >> to the blog as soon as I have edit permission. >> >> cheers, >> Stratos >> >> On Wed, Nov 12, 2014 at 1:06 AM, Edward J. Yoon <[email protected]> >> wrote: >> >>> Awesome! Please feel free to link to your blog post on >>> GettingStartedMesos wiki page, or update it. If you need edit >>> permission let us know your wiki ID. >>> >>> On Sat, Nov 8, 2014 at 6:51 PM, Stratos Dimopoulos >>> <[email protected]> wrote: >>> > Hi Jeff, >>> > >>> > Just wanted to let you know that your patch worked on my setup. Since >>> > lately I am doing a lot of installations I was keeping some notes and >>> now I >>> > polished them and published a blog post ( >>> > https://strat0sphere.wordpress.com/2014/11/08/apache-hama-on-mesos/) on >>> > which I am including some errors I encountered on the process as a >>> > non-expert mesos/hama user and the configuration I did to make it work. >>> > Also, if you think it would be useful to update the documentation ( >>> > http://wiki.apache.org/hama/GettingStartedMesos) on the Hama website, >>> > mainly the hama-site.xml file, please let me know and I'll be happy doing >>> > it as my first tiny contribution to the project. >>> > >>> > I also noticed that all the links provided on the website are for the >>> older >>> > 0.6.4 Hama version, which doesn't support Mesos. Not sure if this is >>> > intentional or just a mistake. >>> > >>> > Again, I really appreciated your help and thanks a ton for doing the fix >>> in >>> > such a timely manner! >>> > >>> > cheers, >>> > Stratos >>> > >>> > On Mon, Oct 27, 2014 at 12:35 AM, Stratos Dimopoulos < >>> > [email protected]> wrote: >>> > >>> >> Great thank you! Looking forward for the patch! >>> >> On Oct 26, 2014 10:30 PM, "Jeff Fenchel" <[email protected]> wrote: >>> >> >>> >>> Stratos, >>> >>> >>> >>> Thanks for sending me the stack trace. I was able to replicate your >>> >>> error. So, it definitely appears to be a bug in Hama. I will open up a >>> >>> ticket and see if I can put together a patch for this sometime this >>> >>> week. In the mean time, Hama without mesos seems to work fine with >>> >>> cdh5. >>> >>> >>> >>> -Jeff >>> >>> >>> >>> On Fri, Oct 24, 2014 at 8:20 PM, Stratos Dimopoulos >>> >>> <[email protected]> wrote: >>> >>> > And when you added the list's user email you had a typo which I copy >>> >>> pasted >>> >>> > and the email never reached the list - So I am just forwarding again >>> to >>> >>> the >>> >>> > correct address this time! :-) >>> >>> > >>> >>> > >>> >>> > On Thu, Oct 23, 2014 at 5:36 PM, Stratos Dimopoulos >>> >>> > <[email protected]> wrote: >>> >>> >> >>> >>> >> Yes Jeff correct - this is the way I compile. Here is the stack >>> trace >>> >>> >> close to the error on the executor - I don't think the log file on >>> the >>> >>> BSP >>> >>> >> master would help you at all but let me know if you need to send you >>> >>> >> anything else: >>> >>> >> >>> >>> >> I1022 17:25:31.828336 22440 exec.cpp:132] Version: 0.20.0 >>> >>> >> I1022 17:25:31.838382 22454 exec.cpp:206] Executor registered on >>> slave >>> >>> >> 20141019-011259-1257767434-5050-4023-1 >>> >>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Executor registered with >>> the >>> >>> >> slave >>> >>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Launching task : Task_0 >>> >>> >> 14/10/22 17:25:32 INFO bsp.GroomServer: groom start >>> >>> >> 14/10/22 17:25:32 INFO bsp.GroomServer: >>> >>> >> >>> >>> >>> /mnt/mesos-work-dir/slaves/20141019-011259-1257767434-5050-4023-1/frameworks/20141019-011259-1257767434-5050-4023-0031/executors/executor_Task_0/runs/d69f21af-9180-4c0c-9263-82038ae81937/hama-0.7.0-SNAPSHOT/bsp/local >>> >>> >> 14/10/22 17:25:32 WARN util.NativeCodeLoader: Unable to load >>> >>> native-hadoop >>> >>> >> library for your platform... using builtin-java classes where >>> >>> applicable >>> >>> >> 14/10/22 17:25:33 ERROR bsp.MesosExecutor: Caught exception, >>> committing >>> >>> >> suicide. >>> >>> >> java.util.ServiceConfigurationError: >>> org.apache.hadoop.fs.FileSystem: >>> >>> >> Provider org.apache.hadoop.fs.LocalFileSystem not found >>> >>> >> at java.util.ServiceLoader.fail(ServiceLoader.java:231) >>> >>> >> at >>> java.util.ServiceLoader.access$300(ServiceLoader.java:181) >>> >>> >> at >>> >>> >> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365) >>> >>> >> at java.util.ServiceLoader$1.next(ServiceLoader.java:445) >>> >>> >> at >>> >>> >> >>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2364) >>> >>> >> at >>> >>> >> >>> >>> >>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2375) >>> >>> >> at >>> >>> >> >>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392) >>> >>> >> at >>> >>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) >>> >>> >> at >>> >>> >> >>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431) >>> >>> >> at >>> >>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413) >>> >>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) >>> >>> >> at >>> >>> org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339) >>> >>> >> at >>> >>> >> >>> org.apache.hama.bsp.GroomServer.deleteLocalFiles(GroomServer.java:483) >>> >>> >> at >>> >>> >> org.apache.hama.bsp.GroomServer.initialize(GroomServer.java:321) >>> >>> >> at org.apache.hama.bsp.GroomServer.run(GroomServer.java:860) >>> >>> >> at >>> >>> org.apache.hama.bsp.MesosExecutor$1.run(MesosExecutor.java:92) >>> >>> >> >>> >>> >> thanks, >>> >>> >> Stratos >>> >>> >> >>> >>> >> On Thu, Oct 23, 2014 at 2:51 PM, Jeff Fenchel <[email protected]> >>> >>> wrote: >>> >>> >>> >>> >>> >>> Fyi... Sorry I forgot to add your email to list. Please add >>> >>> >>> [email protected] to your reply. >>> >>> >>> >>> >>> >>> Thanks, >>> >>> >>> Jeff >>> >>> >>> >>> >>> >>> ---------- Forwarded message ---------- >>> >>> >>> From: Jeff Fenchel <[email protected]> >>> >>> >>> Date: Thursday, October 23, 2014 >>> >>> >>> Subject: Using hama with Mesos 0.20 and CDH5.1.2 >>> >>> >>> To: [email protected] >>> >>> >>> >>> >>> >>> >>> >>> >>> Hey Stratos, >>> >>> >>> >>> >>> >>> Thanks for giving Hama on Mesos a try. I do not think it has ever >>> been >>> >>> >>> tested with hadoop 2, but I see no reason why it shouldn't work. >>> So, >>> >>> I am >>> >>> >>> >>> >>> >>> guessing you compiled with something like the following? >>> >>> >>> >>> >>> >>> mvn clean install -Phadoop2 -Dhadoop.version=2.3.0-cdh5.1.2 >>> >>> >>> -Dmesos.version=0.20.0 -DskipTests >>> >>> >>> >>> >>> >>> If so, can you please send us the complete stack trace? It does >>> look >>> >>> >>> like this might be a bug in hama. >>> >>> >>> >>> >>> >>> Thanks, >>> >>> >>> -Jeff >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Oct 23, 2014 at 1:13 AM, Stratos Dimopoulos >>> >>> >>> <[email protected]> wrote: >>> >>> >>> > Hi, >>> >>> >>> > >>> >>> >>> > I cloned Hama from Git and installed in a Eucalyptus cloud with >>> >>> Mesos >>> >>> >>> > 0.20 >>> >>> >>> > and CDH5.1.2 HDFS. It is also configured with zookeeper but my >>> >>> >>> > zookeeper at >>> >>> >>> > the moment just runs in one master. I followed the instructions >>> >>> here: >>> >>> >>> > http://wiki.apache.org/hama/GettingStartedMesos - that are for >>> an >>> >>> older >>> >>> >>> > version and additionally to that I added some more properties to >>> >>> >>> > overcome >>> >>> >>> > some runtime problems and until Mesos was able to recognize Hama >>> as >>> >>> a >>> >>> >>> > framework. Overall the properties I have set are the following: >>> >>> >>> > >>> >>> >>> > bsp.master.address >>> >>> >>> > bsp.master.port >>> >>> >>> > bsp.master.TaskWorkerManager.class >>> >>> >>> > fs.default.name (I don't think it needs to be here but in an >>> >>> effort to >>> >>> >>> > fix >>> >>> >>> > the issues I had) >>> >>> >>> > hama.mesos.executor.uri >>> >>> >>> > bsp.tasks.maximum.total >>> >>> >>> > hama.mesos.master >>> >>> >>> > bsp.child.java.opts >>> >>> >>> > bsp.system.dir >>> >>> >>> > bsp.local.dir >>> >>> >>> > bsp.tmp.dir >>> >>> >>> > bsp.disk.queue.dir >>> >>> >>> > hama.zookeeper.quorum >>> >>> >>> > hama.zookeeper.property.clientPort >>> >>> >>> > >>> >>> >>> > When I run the example the executor is running but the task >>> fails. >>> >>> The >>> >>> >>> > error output on the executors log is: ERROR bsp.MesosExecutor: >>> >>> Caught >>> >>> >>> > exception, committing suicide. >>> >>> >>> > java.util.ServiceConfigurationError: >>> >>> org.apache.hadoop.fs.FileSystem: >>> >>> >>> > Provider org.apache.hadoop.fs.LocalFileSystem not found >>> >>> >>> > >>> >>> >>> > I did some research on this error message but I cannot find >>> >>> something >>> >>> >>> > that >>> >>> >>> > really helps. It seems to me as a configuration issue but I don't >>> >>> know >>> >>> >>> > how >>> >>> >>> > to solve it. >>> >>> >>> > Also note that I have verified that my HDFS works properly. I am >>> >>> >>> > actually >>> >>> >>> > already running Spark and Hadoop over Mesos without problems. >>> >>> >>> > >>> >>> >>> > I hope somebody can help! >>> >>> >>> > >>> >>> >>> > thanks, >>> >>> >>> > Stratos >>> >>> >>> >>> >>> >> >>> >>> > >>> >>> >>> >> >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> CEO at DataSayer Co., Ltd. >>> > > > > -- > Best Regards, Edward J. Yoon > CEO at DataSayer Co., Ltd.
