Thanks a lot, we miss that setting, will do the experiments right away, :)

best


AL

> On Jan 25, 2017, at 6:17 PM, Paul Brenner <pbren...@placeiq.com> wrote:
> 
> 
> Did you try setting your interpreter to “Isolated” mode? Is it currently in 
> “shared” mode? 
> 
> If you haven’t played with this setting before then: 
> 1. Open the interpreters page
> 2. Find your interpreter and click the edit button in the top right corner
> 3. Beneath the word “option” at the top left there should be a drop down that 
> says “shared”, “scoped”, or “isolated”.
> 4. Set that drop down to isolated and then scroll down and click save.
> 
> By doing that, we get a separate YARN application for each notebook running 
> on the same interpreter.
> 
>  <http://www.placeiq.com/> <http://www.placeiq.com/> 
> <http://www.placeiq.com/>        Paul Brenner     
> <https://twitter.com/placeiq> <https://twitter.com/placeiq> 
> <https://twitter.com/placeiq>       <https://www.facebook.com/PlaceIQ> 
> <https://www.facebook.com/PlaceIQ>   
> <https://www.linkedin.com/company/placeiq> 
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> (217) 390-3033  
> 
>  
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
>  
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>  
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>  
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>  
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>  
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
>  
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>  
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>  
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>  
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>  
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>  
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
>  
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
>  
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>
>  
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
> 
> On Wed, Jan 25, 2017 at 9:12 PM Alec Lee <Alec Lee  <mailto:Alec Lee 
> <alec.in...@gmail.com>>> wrote:
> Hello, Paul 
> 
> Thank you so much for prompt reply. Our understanding is each interpreter 
> takes one application on YARN, if multiple notebooks share same interpreter 
> will run through same YARN application. 
> 
> As you mentioned this “… which will kill any other YARN application 
> associated with that interpreter.” , we seem never see more than one YARN 
> application in YARN UI, the case we had, I use one interpreter but wrote many 
> notebooks, but no matter how many notebooks spark jobs, I only see one YARN 
> application. So I am curious how to make single interpreter being associated 
> with more than YARN applications.
> 
> 
> Thanks
> 
> 
> AL
> 
> 
> 
> 
> 
>> On Jan 25, 2017, at 5:39 PM, Paul Brenner <pbren...@placeiq.com 
>> <mailto:pbren...@placeiq.com>> wrote:
>> 
>> 
>> Alec,
>> 
>> The way we use zeppelin at our company is to set our interpreters to 
>> “isolated”. That way each notebook gets it’s own application on yarn. 
>> 
>> This mostly works well. The one downside is that if you stop a notebook 
>> (e.g. by calling sys.exit in a cell of a spark notebook) it does stop the 
>> YARN application most of the time but you can’t restart that notebook until 
>> you have restarted the interpreter… which will kill any other YARN 
>> application associated with that interpreter.
>> 
>> So our full setup is that we give each user an interpreter (which is good 
>> because we can set each user’s interpreter to have their username in 
>> spark.yarn.queue) and set each user’s interpreter to isolated.
>> 
>> Honestly I still don’t understand what scoped does… maybe that would work as 
>> well?
>> 
>>  <http://www.placeiq.com/> <http://www.placeiq.com/> 
>> <http://www.placeiq.com/>       Paul Brenner     
>> <https://twitter.com/placeiq> <https://twitter.com/placeiq> 
>> <https://twitter.com/placeiq>       <https://www.facebook.com/PlaceIQ> 
>> <https://www.facebook.com/PlaceIQ>   
>> <https://www.linkedin.com/company/placeiq> 
>> <https://www.linkedin.com/company/placeiq>
>> DATA SCIENTIST
>> (217) 390-3033  
>> 
>>  
>> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
>>  
>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>  
>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>  
>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>  
>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>  
>> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
>>  
>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>  
>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>  
>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>  
>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>  
>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>  
>> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
>>  
>> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
>>  
>> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>
>>  
>> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>> 
>> On Wed, Jan 25, 2017 at 8:20 PM Alec Lee <Alec Lee  
>> <mailto:Alec%20Lee%20<alec.in...@gmail.com>>> wrote:
>> Hi, all 
>> 
>> 
>> Currently we are exploring feature of zeppelin, now the situation we are 
>> using YARN to manage spark jobs. In terms of the experiments, we conclude 
>> that one interpreter is corresponding to An application in YARN cluster, 
>> that means all the notebooks from zeppelin with same interpreter go through 
>> single Application in YARN. Also we found if out code shuts down the 
>> application in YARN, then any notebooks fail to run after this point - error 
>> like this “can’t call a stop spark context …..” . The only solution for this 
>> is to restart the interpreter. How to get around this without restart the 
>> interpreter? 
>> 
>> 
>> 
>> Thanks 
>> 
>> 
>> AL
>> 

Reply via email to