Hi Arun, On Mesos the easiest way to debug your job is to go to the Mesos UI which is usually on port 5050. If you are using the amaterasu-vagrant machine, the address would be 192.168.33.11:5050.
As for the rest of your questions: 1. Currently using the Mesos cluster is the only way to run Amaterasu jobs. Next version will also support YARN. We are thinking about creating a stand alone deployment for Amaterasu but currently it's not in high priority. As for the buildHomeDir, that is a part of our dev workflow and is very handy. We have another gradle action called buildDitribution which packages Amaterasu, and this is how we plan to release versions going forward. 2. a. You are correct, in the current version pipelines are monoliths. We do plan on making pipelines composable in version 0.3 which is when we also plan on supporting binaries for actions b. To add dependencies, you can use the deps folder, you can see an example in the https://github.com/shintoio/amaterasu-job-sample in the error-handling branch. For Scala (jar) dependencies you use the jars.yml file and for python we support anaconda dependencies using a file called paython.yml. 3. that should be the SDK :) Cheers, Yaniv On Sun, Oct 1, 2017 at 12:05 AM, Arun Manivannan <a...@arunma.com> wrote: > Hi Yaniv, > > That was spot on. Yes. That was the issue and I am able to complete the > demo job successfully !! I wonder where I should have looked into to have > figured out this issue myself. :-( > > Probably, I should have listened carefully to your presentation but I have > a few really basic questions. > > 1. Is the general way to test a deployment is to run the buildHomeDir (for > dev) and run it on vagrant with mesos? Is there a way to bypass mesos and > run my spark jobs on local mode and python jobs on my local machine? We > discussed this earlier - I would like to use amaterasu for running some > quick integration tests and it would be easier to test it on my local > machine than to have a VM running. > > 2. Probably, I am not seeing this right. On the amaterasu-jobs, I notice > that all the jobs are in the same repo. As you know, most often our jobs > in the pipeline isn't in a single repo. I also notice that the > SparkRunnerHelper interprets the file.scala/other file arguments that is > passed into it and binds the context variables. However, once we are out > of the repository, all we would have is binaries. > > a. Is it a requirement at the moment to have all the Driver source > files in a single repo i.e. the job repo ? > b. If that's the case, then how do I add external dependencies to > the component spark jobs? > > 3. I see that using AmaContext would enable handshake between jobs in the > pipeline. I realise then, we must have the sdk/amaterasu library in the > classpath of the Spark job. Which one would that be? > > I am terribly sorry if these questions doesn't make much sense in the > context of the project. I would just like to know if I had misunderstood > the purpose of the project. I absolutely realise that the project is just > incubating and it's too much to ask for all the bells and whistles on day > 1. > > Best Regards, > Arun > > > > On Sat, Sep 30, 2017 at 8:00 PM Yaniv Rodenski <ya...@shinto.io> wrote: > > > Hi Arun, > > > > I think you are hitting a bug that we’ve fixed but was in a pending PR, > > I've just merged the PR. > > Try to do a git pull and run again, let us know if it solves the problem. > > > > Cheers, > > Yaniv > > > > On Sat, 30 Sep 2017 at 7:36 pm, Arun Manivannan <a...@arunma.com> wrote: > > > > > Hi, > > > > > > I am trying to make an initial run on Amaterasu with > > > https://github.com/arunma/amaterasu-v2-demo (just an unmodified fork > of > > > https://github.com/shintoio/amaterasu-v2-demo). Seems like the spark > > job > > > fails with error (as I see from the logs). Not surprisingly, I am > > unable > > > so see a json on the /tmp/test1. > > > > > > I am not familiar with Mesos. Tried to check for clues on > /var/log/mesos > > on > > > the vagrant box with no luck. > > > > > > I am just running a single node mesos on vagrant ( > > > https://github.com/shintoio/amaterasu-vagrant). Greatly appreciate if > > you > > > could help me with some hints. > > > > > > Earlier I ran a `./gradlew buildHomeDir` and modified the Vagrantfile > to > > > point to my local build directory of Amaterasu. > > > > > > Cheers, > > > Arun > > > > > > > > > > > > [vagrant@node1 ama]$ ./ama-start.sh --repo=" > > > https://github.com/arunma/amaterasu-v2-demo.git"; --branch="master" > > > --env="test" --report="code" > > > serving amaterasu from /ama/lib on user supplied port > > > ./ama-start.sh: line 29: popd: directory stack empty > > > > > > > > > /\ > > > / \ /\ > > > / /\ / \ > > > _ _ / / / /\ \ > > > /_\ _ __ __ _ | |_ ___ _ _ __(_( _(_(_ )_) > > > / _ \ | ' \ / _` || _|/ -_)| '_|/ _` |(_-<| || | > > > /_/ \_\|_|_|_|\__,_| \__|\___||_| \__,_|/__/ \_,_| > > > > > > Continuously deployed data pipelines > > > Version 0.2.0-incubating > > > > > > > > > repo: https://github.com/arunma/amaterasu-v2-demo.git > > > java -cp ./bin/leader-0.2.0-incubating-all.jar > > > -Djava.library.path=/usr/lib > > > org.apache.amaterasu.leader.mesos.JobLauncher --home . --repo > > > https://github.com/arunma/amaterasu-v2-demo.git --branch master --env > > test > > > --report code > > > 2017-09-30 09:22:46.184:INFO::main: Logging initialized @688ms > > > 2017-09-30 09:22:46.262:INFO:oejs.Server:main: jetty-9.2.z-SNAPSHOT > > > 2017-09-30 09:22:46.300:INFO:oejsh.ContextHandler:main: Started > > > o.e.j.s.ServletContextHandler@385e9564{/,file:/ama/dist/,AVAILABLE} > > > 2017-09-30 09:22:46.317:INFO:oejs.ServerConnector:main: Started > > > ServerConnector@1dac5ef{HTTP/1.1}{0.0.0.0:8000} > > > 2017-09-30 09:22:46.317:INFO:oejs.Server:main: Started @822ms > > > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > > > SLF4J: Defaulting to no-operation (NOP) logger implementation > > > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > > further > > > details. > > > I0930 09:22:46.591053 5481 sched.cpp:232] Version: 1.4.0 > > > I0930 09:22:46.592723 5514 sched.cpp:336] New master detected at > > > master@192.168.33.11:5050 > > > I0930 09:22:46.593138 5514 sched.cpp:352] No credentials provided. > > > Attempting to register without authentication > > > I0930 09:22:46.596042 5511 sched.cpp:759] Framework registered with > > > a82be91b-d3fb-4ccd-9dc3-145f60ac8316-0000 > > > ===> Executor 0000000002-6bfdbb03-88b7-44af-9cd0-2a12ddda7b62 > registered > > > ===> a provider for group spark was created > > > ===> launching task: 0000000002 > > > ===> ================= started action start ================= > > > ===> val data = 1 to 1000 > > > ===> val rdd = sc.parallelize(data) > > > ===> val odd = rdd.filter(n => n%2 != 0).toDF("number") > > > ===> ================= finished action start ================= > > > ===> complete task: 0000000002 > > > ===> launching task: 0000000003 > > > ===> Executor 0000000003-6f7eb9e9-8977-4d47-906e-e0802bb7ffe8 > registered > > > ===> a provider for group spark was created > > > ===> launching task: 0000000003 > > > ===> Executor 0000000003-605f3b31-e15e-482d-8c93-737f1605e9f6 > registered > > > ===> a provider for group spark was created > > > ===> launching task: 0000000003 > > > ===> moving to err action null > > > 2017-09-30 09:24:07.355:INFO:oejs.ServerConnector:Thread-59: Stopped > > > ServerConnector@1dac5ef{HTTP/1.1}{0.0.0.0:8000} > > > 2017-09-30 09:24:07.359:INFO:oejsh.ContextHandler:Thread-59: Stopped > > > o.e.j.s.ServletContextHandler@385e9564{/,file:/ama/dist/,UNAVAILABLE} > > > I0930 09:24:07.361192 5512 sched.cpp:2021] Asked to stop the driver > > > I0930 09:24:07.361409 5512 sched.cpp:1203] Stopping framework > > > a82be91b-d3fb-4ccd-9dc3-145f60ac8316-0000 > > > kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... > > or > > > kill -l [sigspec] > > > > > > > > > W00t amaterasu job is finished!!! > > > > > > -- Yaniv Rodenski
