Re: Running Flink jobs directly from Eclipse
I exported it in an environment variable before starting Flink:
flink-0.10.1/bin/yarn-session.sh -n 64 -s 4 -jm 4096 -tm 4096
2016-04-06 15:36 GMT+02:00 Serhiy Boychenko <serhiy.boyche...@cern.ch>:
> What about YARN(and HDFS) configuration? I put yarn-site.xml directly into
> classpath? Or I can set the variables in the execution environment? I will
> give it a try tomorrow morning, will report back and if successful blog
> about it ofc J
>
>
>
> *From:* Christophe Salperwyck [mailto:christophe.salperw...@gmail.com]
> *Sent:* 06 April 2016 13:41
> *To:* user@flink.apache.org
> *Subject:* Re: Running Flink jobs directly from Eclipse
>
>
>
> For me it was taking the local jar and uploading it into the cluster.
>
>
>
> 2016-04-06 13:16 GMT+02:00 Shannon Carey <sca...@expedia.com>:
>
> Thanks for the info! It is a bit difficult to tell based on the
> documentation whether or not you need to put your jar onto the Flink master
> node and run the flink command from there in order to get a job running.
> The documentation on
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/yarn_setup.html
> isn't
> very explicit about where you can run the flink command from, and doesn't
> mention that you can run the job programmatically instead of using the CLI.
>
>
>
> *From: *Christophe Salperwyck <christophe.salperw...@gmail.com>
> *Date: *Wednesday, April 6, 2016 at 1:24 PM
> *To: *<user@flink.apache.org>
> *Subject: *Re: Running Flink jobs directly from Eclipse
>
>
>
> From my side I was starting the YARN session from the cluster:
>
> flink-0.10.1/bin/yarn-session.sh -n 64 -s 4 -jm 4096 -tm 4096
>
>
>
> Then getting the IP/port from the WebUI and then from Eclipse:
>
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment("xx.xx.xx.xx", 40631,
> "target/FlinkTest-0.0.1-SNAPSHOT-jar-with-dependencies.jar");
>
>
>
> The JAR need to be compiled before.
>
>
>
> Hope it helps!
>
> Christophe
>
>
>
> 2016-04-06 9:25 GMT+02:00 Serhiy Boychenko <serhiy.boyche...@cern.ch>:
>
> Cheerz,
>
>
>
> I have been working last few month on the comparison of different data
> processing engines and recently came across Apache Flink. After reading
> different academic papers on comparison of Flink with other data processing
> I would definitely give it a shot. The only issue I am currently having is
> that I am unable to submit Flink jobs directly from Eclipse (to YARN
> cluster). I am wondering if you got any guildelines how I could do the
> submission not from the client but from Eclipse directly? (I was unable to
> find anything related, with the exception of setting up Eclipse for working
> on Flink core)
>
>
>
> Best regards,
>
> Serhiy.
>
>
>
>
>
>
>
RE: Running Flink jobs directly from Eclipse
What about YARN(and HDFS) configuration? I put yarn-site.xml directly into
classpath? Or I can set the variables in the execution environment? I will give
it a try tomorrow morning, will report back and if successful blog about it ofc
☺
From: Christophe Salperwyck [mailto:christophe.salperw...@gmail.com]
Sent: 06 April 2016 13:41
To: user@flink.apache.org
Subject: Re: Running Flink jobs directly from Eclipse
For me it was taking the local jar and uploading it into the cluster.
2016-04-06 13:16 GMT+02:00 Shannon Carey
<sca...@expedia.com<mailto:sca...@expedia.com>>:
Thanks for the info! It is a bit difficult to tell based on the documentation
whether or not you need to put your jar onto the Flink master node and run the
flink command from there in order to get a job running. The documentation on
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/yarn_setup.html
isn't very explicit about where you can run the flink command from, and
doesn't mention that you can run the job programmatically instead of using the
CLI.
From: Christophe Salperwyck
<christophe.salperw...@gmail.com<mailto:christophe.salperw...@gmail.com>>
Date: Wednesday, April 6, 2016 at 1:24 PM
To: <user@flink.apache.org<mailto:user@flink.apache.org>>
Subject: Re: Running Flink jobs directly from Eclipse
From my side I was starting the YARN session from the cluster:
flink-0.10.1/bin/yarn-session.sh -n 64 -s 4 -jm 4096 -tm 4096
Then getting the IP/port from the WebUI and then from Eclipse:
ExecutionEnvironment env =
ExecutionEnvironment.createRemoteEnvironment("xx.xx.xx.xx", 40631,
"target/FlinkTest-0.0.1-SNAPSHOT-jar-with-dependencies.jar");
The JAR need to be compiled before.
Hope it helps!
Christophe
2016-04-06 9:25 GMT+02:00 Serhiy Boychenko
<serhiy.boyche...@cern.ch<mailto:serhiy.boyche...@cern.ch>>:
Cheerz,
I have been working last few month on the comparison of different data
processing engines and recently came across Apache Flink. After reading
different academic papers on comparison of Flink with other data processing I
would definitely give it a shot. The only issue I am currently having is that I
am unable to submit Flink jobs directly from Eclipse (to YARN cluster). I am
wondering if you got any guildelines how I could do the submission not from the
client but from Eclipse directly? (I was unable to find anything related, with
the exception of setting up Eclipse for working on Flink core)
Best regards,
Serhiy.
Re: Running Flink jobs directly from Eclipse
For me it was taking the local jar and uploading it into the cluster.
2016-04-06 13:16 GMT+02:00 Shannon Carey <sca...@expedia.com>:
> Thanks for the info! It is a bit difficult to tell based on the
> documentation whether or not you need to put your jar onto the Flink master
> node and run the flink command from there in order to get a job running.
> The documentation on
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/yarn_setup.html
> isn't
> very explicit about where you can run the flink command from, and doesn't
> mention that you can run the job programmatically instead of using the CLI.
>
> From: Christophe Salperwyck <christophe.salperw...@gmail.com>
> Date: Wednesday, April 6, 2016 at 1:24 PM
> To: <user@flink.apache.org>
> Subject: Re: Running Flink jobs directly from Eclipse
>
> From my side I was starting the YARN session from the cluster:
> flink-0.10.1/bin/yarn-session.sh -n 64 -s 4 -jm 4096 -tm 4096
>
> Then getting the IP/port from the WebUI and then from Eclipse:
> ExecutionEnvironment env =
> ExecutionEnvironment.createRemoteEnvironment("xx.xx.xx.xx", 40631,
> "target/FlinkTest-0.0.1-SNAPSHOT-jar-with-dependencies.jar");
>
> The JAR need to be compiled before.
>
> Hope it helps!
> Christophe
>
> 2016-04-06 9:25 GMT+02:00 Serhiy Boychenko <serhiy.boyche...@cern.ch>:
>
>> Cheerz,
>>
>>
>>
>> I have been working last few month on the comparison of different data
>> processing engines and recently came across Apache Flink. After reading
>> different academic papers on comparison of Flink with other data processing
>> I would definitely give it a shot. The only issue I am currently having is
>> that I am unable to submit Flink jobs directly from Eclipse (to YARN
>> cluster). I am wondering if you got any guildelines how I could do the
>> submission not from the client but from Eclipse directly? (I was unable to
>> find anything related, with the exception of setting up Eclipse for working
>> on Flink core)
>>
>>
>>
>> Best regards,
>>
>> Serhiy.
>>
>>
>>
>
>
Re: Running Flink jobs directly from Eclipse
Thanks for the info! It is a bit difficult to tell based on the documentation
whether or not you need to put your jar onto the Flink master node and run the
flink command from there in order to get a job running. The documentation on
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/yarn_setup.html
isn't very explicit about where you can run the flink command from, and
doesn't mention that you can run the job programmatically instead of using the
CLI.
From: Christophe Salperwyck
<christophe.salperw...@gmail.com<mailto:christophe.salperw...@gmail.com>>
Date: Wednesday, April 6, 2016 at 1:24 PM
To: <user@flink.apache.org<mailto:user@flink.apache.org>>
Subject: Re: Running Flink jobs directly from Eclipse
From my side I was starting the YARN session from the cluster:
flink-0.10.1/bin/yarn-session.sh -n 64 -s 4 -jm 4096 -tm 4096
Then getting the IP/port from the WebUI and then from Eclipse:
ExecutionEnvironment env =
ExecutionEnvironment.createRemoteEnvironment("xx.xx.xx.xx", 40631,
"target/FlinkTest-0.0.1-SNAPSHOT-jar-with-dependencies.jar");
The JAR need to be compiled before.
Hope it helps!
Christophe
2016-04-06 9:25 GMT+02:00 Serhiy Boychenko
<serhiy.boyche...@cern.ch<mailto:serhiy.boyche...@cern.ch>>:
Cheerz,
I have been working last few month on the comparison of different data
processing engines and recently came across Apache Flink. After reading
different academic papers on comparison of Flink with other data processing I
would definitely give it a shot. The only issue I am currently having is that I
am unable to submit Flink jobs directly from Eclipse (to YARN cluster). I am
wondering if you got any guildelines how I could do the submission not from the
client but from Eclipse directly? (I was unable to find anything related, with
the exception of setting up Eclipse for working on Flink core)
Best regards,
Serhiy.
Re: Running Flink jobs directly from Eclipse
>From my side I was starting the YARN session from the cluster:
flink-0.10.1/bin/yarn-session.sh -n 64 -s 4 -jm 4096 -tm 4096
Then getting the IP/port from the WebUI and then from Eclipse:
ExecutionEnvironment env =
ExecutionEnvironment.createRemoteEnvironment("xx.xx.xx.xx", 40631,
"target/FlinkTest-0.0.1-SNAPSHOT-jar-with-dependencies.jar");
The JAR need to be compiled before.
Hope it helps!
Christophe
2016-04-06 9:25 GMT+02:00 Serhiy Boychenko :
> Cheerz,
>
>
>
> I have been working last few month on the comparison of different data
> processing engines and recently came across Apache Flink. After reading
> different academic papers on comparison of Flink with other data processing
> I would definitely give it a shot. The only issue I am currently having is
> that I am unable to submit Flink jobs directly from Eclipse (to YARN
> cluster). I am wondering if you got any guildelines how I could do the
> submission not from the client but from Eclipse directly? (I was unable to
> find anything related, with the exception of setting up Eclipse for working
> on Flink core)
>
>
>
> Best regards,
>
> Serhiy.
>
>
>
Running Flink jobs directly from Eclipse
Cheerz, I have been working last few month on the comparison of different data processing engines and recently came across Apache Flink. After reading different academic papers on comparison of Flink with other data processing I would definitely give it a shot. The only issue I am currently having is that I am unable to submit Flink jobs directly from Eclipse (to YARN cluster). I am wondering if you got any guildelines how I could do the submission not from the client but from Eclipse directly? (I was unable to find anything related, with the exception of setting up Eclipse for working on Flink core) Best regards, Serhiy.
