Maybe your application is overriding the master variable when it creates its
SparkContext. I see you are still passing “yarn-client” as an argument later to
it in your command.
> On Jun 17, 2018, at 11:53 AM, Raymond Xie <xie3208...@gmail.com> wrote:
>
> Thank you Subhash.
>
> Here is the new command:
> spark-submit --master local[*] --class retail_db.GetRevenuePerOrder --conf
> spark.ui.port=12678 spark2practice_2.11-0.1.jar yarn-client
> /public/retail_db/order_items /home/rxie/output/revenueperorder
>
> Still seeing the same issue here.
> 2018-06-17 11:51:25 INFO RMProxy:98 - Connecting to ResourceManager at
> /0.0.0.0:8032
> 2018-06-17 11:51:27 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:28 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:29 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:30 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:31 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:32 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:33 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:34 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:35 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2018-06-17 11:51:36 INFO Client:871 - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is
>
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
>
>
>
> ------------------------------------------------
> Sincerely yours,
>
>
> Raymond
>
> On Sun, Jun 17, 2018 at 2:36 PM, Subhash Sriram <subhash.sri...@gmail.com>
> wrote:
> Hi Raymond,
>
> If you set your master to local[*] instead of yarn-client, it should run on
> your local machine.
>
> Thanks,
> Subhash
>
> Sent from my iPhone
>
> On Jun 17, 2018, at 2:32 PM, Raymond Xie <xie3208...@gmail.com> wrote:
>
>> Hello,
>>
>> I am wondering how can I run spark job in my environment which is a single
>> Ubuntu host with no hadoop installed? if I run my job like below, I will end
>> up with infinite loop at the end. Thank you very much.
>>
>> rxie@ubuntu:~/data$ spark-submit --class retail_db.GetRevenuePerOrder --conf
>> spark.ui.port=12678 spark2practice_2.11-0.1.jar yarn-client
>> /public/retail_db/order_items /home/rxie/output/revenueperorder
>> 2018-06-17 11:19:36 WARN Utils:66 - Your hostname, ubuntu resolves to a
>> loopback address: 127.0.1.1; using 192.168.112.141 instead (on interface
>> ens33)
>> 2018-06-17 11:19:36 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind
>> to another address
>> 2018-06-17 11:19:37 WARN NativeCodeLoader:62 - Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> 2018-06-17 11:19:38 INFO SparkContext:54 - Running Spark version 2.3.1
>> 2018-06-17 11:19:38 WARN SparkConf:66 - spark.master yarn-client is
>> deprecated in Spark 2.0+, please instead use "yarn" with specified deploy
>> mode.
>> 2018-06-17 11:19:38 INFO SparkContext:54 - Submitted application: Get
>> Revenue Per Order
>> 2018-06-17 11:19:38 INFO SecurityManager:54 - Changing view acls to: rxie
>> 2018-06-17 11:19:38 INFO SecurityManager:54 - Changing modify acls to: rxie
>> 2018-06-17 11:19:38 INFO SecurityManager:54 - Changing view acls groups to:
>> 2018-06-17 11:19:38 INFO SecurityManager:54 - Changing modify acls groups
>> to:
>> 2018-06-17 11:19:38 INFO SecurityManager:54 - SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(rxie); groups with view permissions: Set(); users with modify
>> permissions: Set(rxie); groups with modify permissions: Set()
>> 2018-06-17 11:19:39 INFO Utils:54 - Successfully started service
>> 'sparkDriver' on port 44709.
>> 2018-06-17 11:19:39 INFO SparkEnv:54 - Registering MapOutputTracker
>> 2018-06-17 11:19:39 INFO SparkEnv:54 - Registering BlockManagerMaster
>> 2018-06-17 11:19:39 INFO BlockManagerMasterEndpoint:54 - Using
>> org.apache.spark.storage.DefaultTopologyMapper for getting topology
>> information
>> 2018-06-17 11:19:39 INFO BlockManagerMasterEndpoint:54 -
>> BlockManagerMasterEndpoint up
>> 2018-06-17 11:19:39 INFO DiskBlockManager:54 - Created local directory at
>> /tmp/blockmgr-69a8a12d-0881-4454-96ab-6a45d5c58bfe
>> 2018-06-17 11:19:39 INFO MemoryStore:54 - MemoryStore started with capacity
>> 413.9 MB
>> 2018-06-17 11:19:39 INFO SparkEnv:54 - Registering OutputCommitCoordinator
>> 2018-06-17 11:19:40 INFO log:192 - Logging initialized @7035ms
>> 2018-06-17 11:19:40 INFO Server:346 - jetty-9.3.z-SNAPSHOT
>> 2018-06-17 11:19:40 INFO Server:414 - Started @7383ms
>> 2018-06-17 11:19:40 INFO AbstractConnector:278 - Started
>> ServerConnector@51ad75c2{HTTP/1.1,[http/1.1]}{0.0.0.0:12678}
>> 2018-06-17 11:19:40 INFO Utils:54 - Successfully started service 'SparkUI'
>> on port 12678.
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@50b8ae8d{/jobs,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@60afd40d{/jobs/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@28a2a3e7{/jobs/job,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@10b3df93{/jobs/job/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@ea27e34{/stages,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@33a2499c{/stages/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@e72dba7{/stages/stage,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@3c321bdb{/stages/stage/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@24855019{/stages/pool,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@3abd581e{/stages/pool/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@4d4d8fcf{/storage,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@610db97e{/storage/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@6f0628de{/storage/rdd,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@3fabf088{/storage/rdd/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@1e392345{/environment,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@12f3afb5{/environment/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@4ced35ed{/executors,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@2c22a348{/executors/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@7bd69e82{/executors/threadDump,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@74d7184a{/executors/threadDump/json,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@51b01960{/static,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@6ca320ab{/,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@50d68830{/api,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@6754ef00{/jobs/job/kill,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO ContextHandler:781 - Started
>> o.s.j.s.ServletContextHandler@619bd14c{/stages/stage/kill,null,AVAILABLE,@Spark}
>> 2018-06-17 11:19:40 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started
>> at http://192.168.112.141:12678
>> 2018-06-17 11:19:41 INFO SparkContext:54 - Added JAR
>> file:/home/rxie/data/spark2practice_2.11-0.1.jar at
>> spark://192.168.112.141:44709/jars/spark2practice_2.11-0.1.jar with
>> timestamp 1529259581058
>> 2018-06-17 11:19:44 INFO RMProxy:98 - Connecting to ResourceManager at
>> /0.0.0.0:8032
>> 2018-06-17 11:19:46 INFO Client:871 - Retrying connect to server:
>> 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
>> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
>> MILLISECONDS)
>> Start the infinite loop here
>>
>> ------------------------------------------------
>> Sincerely yours,
>>
>>
>> Raymond
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
