That looks normal. Do you have access to your YARN ResourceManager WebUI? It should have a Scheduler tab on the left, on it you could find out about YARN Application Queue. From that you can find out about application pending, memory used and so on, perhaps that would help to investigate this further.
_____________________________
From: Will Du <will...@gmail.com>
Sent: Thursday, December 3, 2015 4:13 PM
Subject: Re: zeppelin job is running all the time
To: <users@zeppelin.incubator.apache.org>
All is what’s by default Properties name
value args master yarn-client
spark.app.name Zeppelin spark.cores.max
spark.executor.memory 512m spark.home
/usr/hdp/current/spark-client/ spark.yarn.jar
hdfs:///apps/zeppelin/zeppelin-spark-0.5.5-SNAPSHOT.jar
zeppelin.dep.localrepo local-repo zeppelin.pyspark.python
python zeppelin.spark.concurrentSQL false
zeppelin.spark.maxResult 1000 zeppelin.spark.useHiveContext
true
On Dec 3, 2015, at 11:51 AM, Felix Cheung <
felixcheun...@hotmail.com> wrote:
Could you send us your configuration of the Spark
Interpreter in Zeppelin?
It can see how both jobs can be long lived in Spark &
hive/Tez but they should not block one or another.
_____________________________
From: Will Du < will...@gmail.com>
Sent: Thursday, December 3, 2015 2:15 AM
Subject: Re: zeppelin job is running all the time
To: < users@zeppelin.incubator.apache.org>
When I run spark-shell and hive cml, everything is good.
One more thing I find is that when I run the hive CML, there is single hive
job running (seen from hue job browser) all the time until I exit from hive
cml. No matter how many HQL I submit, it is still
single job there. I think it is because of using Tez. Is Tez having conflict
with zeppelin?
On Dec 3, 2015, at 2:15 AM, Felix
Cheung < felixcheun...@hotmail.com> wrote:
I don't know
enough about HDP, but there should be a way to check user queue in YARN?
Spark job shouldn't affect Hive job though. Have you tried running spark-shell
(--master yarn-client) and a Hive job at the same time?
From: will...@gmail.com
Subject: Re: zeppelin job is running all the time
Date: Tue, 1 Dec 2015 22:56:55 -0500
To: users@zeppelin.incubator.apache.org
Do I have to stop zeppelin to make hive job running?
I am thinking to have one notebook have both spark and hive coding running.
In addition, I am quite sure the
first job in the picture is hive job. The 2nd one is zeppelin running spark. I
wonder how to unblock hive job.
On Dec 1,
2015, at 10:45 PM, Jeff Zhang < zjf...@gmail.com> wrote:
>>>
the status of hive query becomes from ACCEPT to RUNNING
The status you see on the hue is the yarn app status which is
used for spark. It should be running forever until you shutdown it in zeppelin
On Wed, Dec 2, 2015 at 11:42 AM, Will
Du <will...@gmail.com> wrote:
The latest
HDP sandbox 2.3.2
On Dec 1, 2015, at 10:38 PM, Jeff Zhang <
zjf...@gmail.com> wrote:
Which version of HDP do you use ?
On Wed, Dec 2,
2015 at 11:23 AM, Will Du <will...@gmail.com>
wrote:
I have assigned an dedicate yarn queue to spark, the status of
hive query becomes from ACCEPT to RUNNING. However, it sees running forever
still. Everything is else is in default config of HDP sandbox. Do I need to set
something else? The browser I saw status is from hue.
Thanks,
wd
On Nov 30, 2015, at 12:40 AM, Rick Moritz <
rah...@gmail.com> wrote:
To
explain the previous reply: the SparkContext created by Zeppelin is persistent
and independent of whether it's currently processing a paragraph or not.
Therefore the Zeppelin job will claim all ressources assigned to it until
Zeppelin is stopped.
On Mon, Nov 30, 2015 at 4:20 AM, Jeff Zhang <zjf...@gmail.com> wrote:
I assume this is yarn app Job Browser. How many executors do you specify
for your zeppelin yarn app ? It seems your zeppelin yarn app consume all the
resources so that it block other applications.
On Mon, Nov 30, 2015 at 11:10 AM, Will Du <will...@gmail.com> wrote:
Hi folks,
I am running a simple scala word count from zeppelin in HDP sandbox. The
job is successful with expected result. But the job of zeppelin is shown in job
status of hue forever. It seems block the my hive job. Does anyone know why?
thanks,
wd
<PastedGraphic-1.tiff>
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
