it seems that the job block whene we call newAPIHadoopRDD to get data from Hbase. it may be the issue !! is there another api to load date from hbase ?
<https://mailtrack.io/> Sent with Mailtrack <https://chrome.google.com/webstore/detail/mailtrack-for-gmail-inbox/ndnaehgpjlnokgebbaldlmgkapkpjkkb?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality> <#> 2017-10-11 14:45 GMT+02:00 Sebastian Piu <sebastian....@gmail.com>: > We do have this issue randomly too, so interested in hearing if someone > was able to get to the bottom of it > > On Wed, 11 Oct 2017, 13:40 amine_901, <cherifimohamedam...@gmail.com> > wrote: > >> We encounter a problem on a Spark job 1.6(on yarn) that never ends, whene >> several jobs launched simultaneously. >> We found that by launching the job spark in yarn-client mode we do not >> have >> this problem, unlike launching it in yarn-cluster mode. >> it could be a trail to find the cause. >> >> we changed the code to add a sparkContext.stop () >> Indeed, the SparkContext was created (val sparkContext = >> createSparkContext) >> but not stopped. this solution has allowed us to decrease the number of >> jobs >> that remains blocked but nevertheless we still have some jobs blocked. >> >> by analyzing the logs we have found this log that repeats without >> stopping: >> /17/09/29 11:04:37 DEBUG SparkEventPublisher: Enqueue >> SparkListenerExecutorMetricsUpdate(1,WrappedArray()) >> 17/09/29 11:04:41 DEBUG ApplicationMaster: Sending progress >> 17/09/29 11:04:41 DEBUG ApplicationMaster: Number of pending allocations >> is >> 0. Sleeping for 5000. / >> >> Does someone have an idea about this issue ? >> Thank you in advance >> >> >> >> -- >> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> -- CHERIFI Mohamed Amine Développeur Big data/Data scientist 07 81 65 17 03
