The execute() call on the Environment blocks. The future will hence not be
done until the execution is finished...
On Tue, Nov 25, 2014 at 7:00 PM, Flavio Pompermaier <pomperma...@okkam.it>
wrote:
> Sounds good to me..how do you check for completion from java code?
>
> On Tue, Nov 25, 2014 at 6:56 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi!
>>
>> 1) The Remote Executor will automatically transfer the jar, if needed.
>>
>> 2) Background execution is not supported out of the box. I would go for a
>> Java ExecutorService with a FutureTask to kick of tasks in a background
>> thread and allow to check for completion.
>>
>> Stephan
>>
>>
>> On Tue, Nov 25, 2014 at 6:41 PM, Flavio Pompermaier <pomperma...@okkam.it
>> > wrote:
>>
>>> Do I have to upload the jar from my application to the Flink Job manager
>>> every time?
>>> Do I have to wait the job to finish? I'd like to start the job
>>> execution, get an id of it and then poll for its status..is that possible?
>>>
>>> On Tue, Nov 25, 2014 at 6:04 PM, Robert Metzger <rmetz...@apache.org>
>>> wrote:
>>>
>>>> Cool.
>>>>
>>>> So you have basically two options:
>>>> a) use the bin/flink run tool.
>>>> This tool is meant for users to submit a job once. To use that, upload
>>>> the jar to any location in the file system (not HDFS).
>>>> use ./bin/flink run <pathToJar> -c classNameOfJobYouWantToRun
>>>> <JobArguments>
>>>> to run the job.
>>>>
>>>> b) use the RemoteExecutor.
>>>> For using the remove Executor, you don't need to put your jar file
>>>> anywhere in your cluster.
>>>> The only thing you need is the jar file somewhere were the Java
>>>> Application can access it.
>>>> Inside this Java Application, you have something like:
>>>>
>>>> runJobOne(ExecutionEnvironment ee) {
>>>> ee.readFile( ... );
>>>> ...
>>>> ee.execute("job 1");
>>>> }
>>>>
>>>> runJobTwo(Exe ..) {
>>>> ...
>>>> }
>>>>
>>>>
>>>> main() {
>>>> ExecutionEnvironment ee = new Remote execution environment ..
>>>>
>>>> if(something) {
>>>> runJobOne(ee);
>>>> } else if(something else) {
>>>> runJobTwo(ee);
>>>> } ...
>>>> }
>>>>
>>>>
>>>> The object returned by the ExecutionEnvironment.execute() call also
>>>> contains information about the final status of the program (failed etc.).
>>>>
>>>> I hope that helps.
>>>>
>>>> On Tue, Nov 25, 2014 at 5:30 PM, Flavio Pompermaier <
>>>> pomperma...@okkam.it> wrote:
>>>>
>>>>> See inline
>>>>>
>>>>> On Tue, Nov 25, 2014 at 3:37 PM, Robert Metzger <rmetz...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hey,
>>>>>>
>>>>>> maybe we need to go a step back because I did not yet fully
>>>>>> understand what you want to do.
>>>>>>
>>>>>> My understanding so far is the following:
>>>>>> - You have a set of jobs that you've written for Flink
>>>>>>
>>>>>
>>>>> Yes, and they are all in the same jar (that I want to put in the
>>>>> cluster somehow)
>>>>>
>>>>> - You have a cluster with Flink running
>>>>>>
>>>>>
>>>>> Yes!
>>>>>
>>>>>
>>>>>> - You have an external client, which is a Java Application that is
>>>>>> controlling when and how the different jobs are launched. The client is
>>>>>> running basically 24/7 or started by a cronjob.
>>>>>>
>>>>>
>>>>> I have a Java application somewhere that triggers the execution of one
>>>>> of the available jobs in the jar (so I need to pass also the necessary
>>>>> arguments required by each job) and then monitor if the job has been put
>>>>> into a running state and its status (running/failed/finished and
>>>>> percentage
>>>>> would be awesome).
>>>>> I don't think RemoteExecutor is enough..am I wrong?
>>>>>
>>>>>
>>>>>> Correct me if these assumptions are wrong. If they are true, the
>>>>>> RemoteExecutor is probably what you are looking for. Otherwise, we have
>>>>>> to
>>>>>> find another solution.
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 25, 2014 at 2:56 PM, Flavio Pompermaier <
>>>>>> pomperma...@okkam.it> wrote:
>>>>>>
>>>>>>> Hi Robert,
>>>>>>> I tried to look at the RemoteExecutor but I can't understand what
>>>>>>> are the exact steps to:
>>>>>>> 1 - (upload if necessary and) register a jar containing multiple
>>>>>>> main methods (one for each job)
>>>>>>> 2 - start the execution of a job from a client
>>>>>>> 3 - monitor the execution of the job
>>>>>>>
>>>>>>> Could you give me the exact java commands/snippets to do that?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Nov 23, 2014 at 8:26 PM, Robert Metzger <rmetz...@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> +1 for providing some utilities/tools for application developers.
>>>>>>>> This could include something like an application registry. I also
>>>>>>>> think that almost every user needs something to parse command line
>>>>>>>> arguments (including default values and comprehensive error messages).
>>>>>>>> We should also see if we can document and properly expose the
>>>>>>>> FileSystem abstraction to Flink app programmers. Users sometimes need
>>>>>>>> to do
>>>>>>>> manipulate files directly.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regarding your second question:
>>>>>>>> For deploying a jar on your cluster, you can use the "bin/flink run
>>>>>>>> <JAR FILE>" command.
>>>>>>>> For starting a Job from an external client you can use the
>>>>>>>> RemoteExecutionEnvironment (you need to know the JobManager address for
>>>>>>>> that). Here is some documentation on that:
>>>>>>>> http://flink.incubator.apache.org/docs/0.7-incubating/cluster_execution.html#remote-environment
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Nov 22, 2014 at 9:06 PM, Flavio Pompermaier <
>>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>>
>>>>>>>>> That was exactly what I was looking for. In my case it is not a
>>>>>>>>> problem to use hadoop version because I work on Hadoop. Don't you
>>>>>>>>> think it
>>>>>>>>> could be useful to add a Flink ProgramDriver so that you can use it
>>>>>>>>> both
>>>>>>>>> for hadoop and native-flink jobs?
>>>>>>>>>
>>>>>>>>> Now that I understood how to bundle together a bunch of jobs, my
>>>>>>>>> next objective will be to deploy the jar on the cluster (similarity
>>>>>>>>> to what
>>>>>>>>> tge webclient does) and then start the jobs from my external client
>>>>>>>>> (which
>>>>>>>>> in theory just need to know the jar name and the parameters to pass to
>>>>>>>>> every job it wants to call). Do you have an example of that?
>>>>>>>>> On Nov 22, 2014 6:11 PM, "Kostas Tzoumas" <ktzou...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Are you looking for something like
>>>>>>>>>> https://hadoop.apache.org/docs/r1.1.1/api/org/apache/hadoop/util/ProgramDriver.html
>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> You should be able to use the Hadoop ProgramDriver directly, see
>>>>>>>>>> for example here:
>>>>>>>>>> https://github.com/ktzoumas/incubator-flink/blob/tez_support/flink-addons/flink-tez/src/main/java/org/apache/flink/tez/examples/ExampleDriver.java
>>>>>>>>>>
>>>>>>>>>> If you don't want to introduce a Hadoop dependency in your
>>>>>>>>>> project, you can just copy-paste ProgramDriver, it does not have any
>>>>>>>>>> dependencies to Hadoop classes. That class just accumulates
>>>>>>>>>> <String,Class>
>>>>>>>>>> pairs (simplifying a bit) and calls the main method of the
>>>>>>>>>> corresponding
>>>>>>>>>> class.
>>>>>>>>>>
>>>>>>>>>> On Sat, Nov 22, 2014 at 5:34 PM, Stephan Ewen <se...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Not sure I get exactly what this is, but packaging multiple
>>>>>>>>>>> examples in one program is well possible. You can have arbitrary
>>>>>>>>>>> control
>>>>>>>>>>> flow in the main() method.
>>>>>>>>>>>
>>>>>>>>>>> Should be well possible to do something like that hadoop
>>>>>>>>>>> examples setup...
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Nov 21, 2014 at 7:02 PM, Flavio Pompermaier <
>>>>>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> That was something I used to do with hadoop and it's
>>>>>>>>>>>> comfortable when testing stuff (so it is not so important).
>>>>>>>>>>>> For an example see what happens when you run the old "hadoop
>>>>>>>>>>>> jar hadoop-mapreduce-examples.jar" command..it "drives" you to the
>>>>>>>>>>>> correct
>>>>>>>>>>>> invokation of that job.
>>>>>>>>>>>> However, the important thing is that I'd like to keep existing
>>>>>>>>>>>> related jobs somewhere (like a repository of jobs), deploy them
>>>>>>>>>>>> and then be
>>>>>>>>>>>> able to start the one I need from an external program.
>>>>>>>>>>>>
>>>>>>>>>>>> Could this be done with RemoteExecutor? Or is there any WS to
>>>>>>>>>>>> manage the job execution? That would be very useful..
>>>>>>>>>>>> Is the Client interface the only one that allow something
>>>>>>>>>>>> similar right now?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Nov 21, 2014 at 6:19 PM, Stephan Ewen <se...@apache.org
>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I am not sure exactly what you need there. In Flink you can
>>>>>>>>>>>>> write more than one program in the same program ;-) You can
>>>>>>>>>>>>> define complex
>>>>>>>>>>>>> flows and execute arbitrarily at intermediate points:
>>>>>>>>>>>>>
>>>>>>>>>>>>> main() {
>>>>>>>>>>>>> ExecutionEnvironment env = ...;
>>>>>>>>>>>>>
>>>>>>>>>>>>> env.readSomething().map().join(...).and().so().on();
>>>>>>>>>>>>> env.execute();
>>>>>>>>>>>>>
>>>>>>>>>>>>> env.readTheNextThing().do()Something();
>>>>>>>>>>>>> env.execute();
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can also just "save" a program and keep it for later
>>>>>>>>>>>>> execution:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Plan plan = env.createProgramPlan();
>>>>>>>>>>>>>
>>>>>>>>>>>>> at a later point you can start that plan: new
>>>>>>>>>>>>> RemoteExecutor(master, 6123).execute(plan);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Stephan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Nov 21, 2014 at 5:49 PM, Flavio Pompermaier <
>>>>>>>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any help on this? :(
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 21, 2014 at 9:33 AM, Flavio Pompermaier <
>>>>>>>>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>> I forgot to ask you if there's a Flink utility to simulate
>>>>>>>>>>>>>>> the Hadoop ProgramDriver class that acts somehow like a
>>>>>>>>>>>>>>> registry of jobs.
>>>>>>>>>>>>>>> Is there something similar?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Flavio
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
