I don't use submit: I start my standalone cluster and connect to it remotely.
Is that a bad practice?
I'd like to be able to it dynamically as the system knows whether it needs more
or less resources based on its own context
> On Jul 15, 2016, at 10:55 AM, Mich Talebzadeh <mich.talebza...@gmail.com>
> wrote:
>
> Hi,
>
> You can also do all this at env or submit time with spark-submit which I
> believe makes it more flexible than coding in.
>
> Example
>
> ${SPARK_HOME}/bin/spark-submit \
> --packages com.databricks:spark-csv_2.11:1.3.0 \
> --driver-memory 2G \
> --num-executors 2 \
> --executor-cores 3 \
> --executor-memory 2G \
> --master spark://50.140.197.217:7077
> <http://50.140.197.217:7077/> \
> --conf "spark.scheduler.mode=FAIR" \
> --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps" \
> --jars
> /home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar \
> --class "${FILE_NAME}" \
> --conf "spark.ui.port=${SP}" \
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss,
> damage or destruction of data or any other property which may arise from
> relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
> On 15 July 2016 at 13:48, Jean Georges Perrin <j...@jgp.net
> <mailto:j...@jgp.net>> wrote:
> Merci Nihed, this is one of the tests I did :( still not working
>
>
>
>> On Jul 15, 2016, at 8:41 AM, nihed mbarek <nihe...@gmail.com
>> <mailto:nihe...@gmail.com>> wrote:
>>
>> can you try with :
>> SparkConf conf = new SparkConf().setAppName("NC Eatery
>> app").set("spark.executor.memory", "4g")
>> .setMaster("spark://10.0.100.120:7077 <>");
>> if (restId == 0) {
>> conf = conf.set("spark.executor.cores", "22");
>> } else {
>> conf = conf.set("spark.executor.cores", "2");
>> }
>> JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
>>
>> On Fri, Jul 15, 2016 at 2:31 PM, Jean Georges Perrin <j...@jgp.net
>> <mailto:j...@jgp.net>> wrote:
>> Hi,
>>
>> Configuration: standalone cluster, Java, Spark 1.6.2, 24 cores
>>
>> My process uses all the cores of my server (good), but I am trying to limit
>> it so I can actually submit a second job.
>>
>> I tried
>>
>> SparkConf conf = new SparkConf().setAppName("NC Eatery
>> app").set("spark.executor.memory", "4g")
>> .setMaster("spark://10.0.100.120:7077 <>");
>> if (restId == 0) {
>> conf = conf.set("spark.executor.cores", "22");
>> } else {
>> conf = conf.set("spark.executor.cores", "2");
>> }
>> JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
>>
>> and
>>
>> SparkConf conf = new SparkConf().setAppName("NC Eatery
>> app").set("spark.executor.memory", "4g")
>> .setMaster("spark://10.0.100.120:7077 <>");
>> if (restId == 0) {
>> conf.set("spark.executor.cores", "22");
>> } else {
>> conf.set("spark.executor.cores", "2");
>> }
>> JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
>>
>> but it does not seem to take it. Any hint?
>>
>> jg
>>
>>
>>
>>
>>
>> --
>>
>> M'BAREK Med Nihed,
>> Fedora Ambassador, TUNISIA, Northern Africa
>> http://www.nihed.com <http://www.nihed.com/>
>>
>> <http://tn.linkedin.com/in/nihed>
>>
>
>
