Re: Jupyter notebook on Dataproc versus GKE

[Holden Karau]

I’ve used Argo for K8s scheduling, for awhile it’s also what Kubeflow used
underneath for scheduling.

On Tue, Sep 6, 2022 at 10:01 AM Mich Talebzadeh 
wrote:

> Thank you all.
>
> Has anyone used Argo for k8s scheduler by any chance?
>
> On Tue, 6 Sep 2022 at 13:41, Bjørn Jørgensen 
> wrote:
>
>> "*JupyterLab is the next-generation user interface for Project Jupyter
>> offering all the familiar building blocks of the classic Jupyter Notebook
>> (notebook, terminal, text editor, file browser, rich outputs, etc.) in a
>> flexible and powerful user interface.*"
>> https://github.com/jupyterlab/jupyterlab
>>
>> You will find them both at https://jupyter.org
>>
>> man. 5. sep. 2022 kl. 23:40 skrev Mich Talebzadeh <
>> mich.talebza...@gmail.com>:
>>
>>> Thanks Bjorn,
>>>
>>> What are the differences and the functionality Jupyerlab brings in on
>>> top of Jupyter notebook?
>>>
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Mon, 5 Sept 2022 at 20:58, Bjørn Jørgensen 
>>> wrote:
>>>
 Jupyter notebook is replaced with jupyterlab :)

 man. 5. sep. 2022 kl. 21:10 skrev Holden Karau :

>
>
> On Mon, Sep 5, 2022 at 9:00 AM Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> Thanks for that.
>>
>> How do you rate the performance of Jupyter W/Spark on K8s compared to
>> the same on  a cluster of VMs (example Dataproc).
>>
>> Also somehow a related question (may be naive as well). For example,
>> Google offers a lot of standard ML libraries for example built into a 
>> data
>> warehouse like BigQuery. What does the Jupyter notebook offer that others
>> don't?
>>
> Jupyter notebook doesn’t offer any particular set of libraries,
> although you can add your own to the container etc.
>
>>
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>> for any loss, damage or destruction of data or any other property which 
>> may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Mon, 5 Sept 2022 at 12:47, Holden Karau 
>> wrote:
>>
>>> I’ve run Jupyter w/Spark on K8s, haven’t tried it with Dataproc
>>> personally.
>>>
>>> The Spark K8s pod scheduler is now more pluggable for Yunikorn and
>>> Volcano can be used with less effort.
>>>
>>> On Mon, Sep 5, 2022 at 7:44 AM Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>

 Hi,


 Has anyone got experience of running Jupyter on dataproc versus
 Jupyter notebook on GKE (k8).


 I have not looked at this for a while but my understanding is that
 Spark on GKE/k8 is not yet performed. This is classic Spark with
 Python/Pyspark.


 Also I would like to know the state of spark with Volcano. Has
 progress made on that front.


 Regards,


 Mich


view my Linkedin profile
 


  https://en.everybodywiki.com/Mich_Talebzadeh



 *Disclaimer:* Use it at your own risk. Any and all responsibility
 for any loss, damage or destruction of data or any other property 
 which may
 arise from relying on this email's technical content is explicitly
 disclaimed. The author will in no case be liable for any monetary 
 damages
 arising from such loss, damage or destruction.



>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: 

Re: Jupyter notebook on Dataproc versus GKE

[Mich Talebzadeh]

Thank you all.

Has anyone used Argo for k8s scheduler by any chance?

On Tue, 6 Sep 2022 at 13:41, Bjørn Jørgensen 
wrote:

> "*JupyterLab is the next-generation user interface for Project Jupyter
> offering all the familiar building blocks of the classic Jupyter Notebook
> (notebook, terminal, text editor, file browser, rich outputs, etc.) in a
> flexible and powerful user interface.*"
> https://github.com/jupyterlab/jupyterlab
>
> You will find them both at https://jupyter.org
>
> man. 5. sep. 2022 kl. 23:40 skrev Mich Talebzadeh <
> mich.talebza...@gmail.com>:
>
>> Thanks Bjorn,
>>
>> What are the differences and the functionality Jupyerlab brings in on top
>> of Jupyter notebook?
>>
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Mon, 5 Sept 2022 at 20:58, Bjørn Jørgensen 
>> wrote:
>>
>>> Jupyter notebook is replaced with jupyterlab :)
>>>
>>> man. 5. sep. 2022 kl. 21:10 skrev Holden Karau :
>>>


 On Mon, Sep 5, 2022 at 9:00 AM Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> Thanks for that.
>
> How do you rate the performance of Jupyter W/Spark on K8s compared to
> the same on  a cluster of VMs (example Dataproc).
>
> Also somehow a related question (may be naive as well). For example,
> Google offers a lot of standard ML libraries for example built into a data
> warehouse like BigQuery. What does the Jupyter notebook offer that others
> don't?
>
 Jupyter notebook doesn’t offer any particular set of libraries,
 although you can add your own to the container etc.

>
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any loss, damage or destruction of data or any other property which may
> arise from relying on this email's technical content is explicitly
> disclaimed. The author will in no case be liable for any monetary damages
> arising from such loss, damage or destruction.
>
>
>
>
> On Mon, 5 Sept 2022 at 12:47, Holden Karau 
> wrote:
>
>> I’ve run Jupyter w/Spark on K8s, haven’t tried it with Dataproc
>> personally.
>>
>> The Spark K8s pod scheduler is now more pluggable for Yunikorn and
>> Volcano can be used with less effort.
>>
>> On Mon, Sep 5, 2022 at 7:44 AM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>> Has anyone got experience of running Jupyter on dataproc versus
>>> Jupyter notebook on GKE (k8).
>>>
>>>
>>> I have not looked at this for a while but my understanding is that
>>> Spark on GKE/k8 is not yet performed. This is classic Spark with
>>> Python/Pyspark.
>>>
>>>
>>> Also I would like to know the state of spark with Volcano. Has
>>> progress made on that front.
>>>
>>>
>>> Regards,
>>>
>>>
>>> Mich
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>> for any loss, damage or destruction of data or any other property which 
>>> may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary 
>>> damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
> --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4
>>> ,
>>> 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4
> ,
> 6010 Ålesund
> 

[ANNOUNCE] Apache Kyuubi (Incubating) released 1.6.0-incubating

[Nicholas Jiang]

Hi all,


The Apache Kyuubi (Incubating) community is pleased to announce that

Apache Kyuubi (Incubating) 1.6.0-incubating has been released!




Apache Kyuubi (Incubating) is a distributed multi-tenant JDBC server for

large-scale data processing and analytics, build on top of multiple compule

engines include Apache Spark, Apache Flink, Apache Hive, Trino.


Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface

for end-users to manipulate large-scale data with pre-programmed and

extensible Spark/Flink SQL engines.


We are aiming to make Kyuubi an "out-of-the-box" tool for data warehouses
and data lakes.


This "out-of-the-box" model minimizes the barriers and costs for end-users
to use Spark, Flink at the client side.


At the server-side, Kyuubi server and engine's multi-tenant architecture

provides the administrators a way to achieve computing resource isolation,

data security, high availability, high client concurrency, etc.


The full release notes and download links are available at:
Release Notes: https://kyuubi.apache.org/release/1.6.0-incubating.html


To learn more about Apache Kyuubi (Incubating), please see
https://kyuubi.apache.org/


Kyuubi Resources:
- Issue: https://github.com/apache/incubator-kyuubi/issues
- Mailing list: d...@kyuubi.apache.org


We would like to thank all contributors of the Kyuubi community and Incubating
community who made this release possible!



Thanks,

On behalf of Apache Kyuubi (Incubating) community

Re: Jupyter notebook on Dataproc versus GKE

[Bjørn Jørgensen]

"*JupyterLab is the next-generation user interface for Project Jupyter
offering all the familiar building blocks of the classic Jupyter Notebook
(notebook, terminal, text editor, file browser, rich outputs, etc.) in a
flexible and powerful user interface.*"
https://github.com/jupyterlab/jupyterlab

You will find them both at https://jupyter.org

man. 5. sep. 2022 kl. 23:40 skrev Mich Talebzadeh :

> Thanks Bjorn,
>
> What are the differences and the functionality Jupyerlab brings in on top
> of Jupyter notebook?
>
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 5 Sept 2022 at 20:58, Bjørn Jørgensen 
> wrote:
>
>> Jupyter notebook is replaced with jupyterlab :)
>>
>> man. 5. sep. 2022 kl. 21:10 skrev Holden Karau :
>>
>>>
>>>
>>> On Mon, Sep 5, 2022 at 9:00 AM Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 Thanks for that.

 How do you rate the performance of Jupyter W/Spark on K8s compared to
 the same on  a cluster of VMs (example Dataproc).

 Also somehow a related question (may be naive as well). For example,
 Google offers a lot of standard ML libraries for example built into a data
 warehouse like BigQuery. What does the Jupyter notebook offer that others
 don't?

>>> Jupyter notebook doesn’t offer any particular set of libraries, although
>>> you can add your own to the container etc.
>>>



view my Linkedin profile
 


  https://en.everybodywiki.com/Mich_Talebzadeh



 *Disclaimer:* Use it at your own risk. Any and all responsibility for
 any loss, damage or destruction of data or any other property which may
 arise from relying on this email's technical content is explicitly
 disclaimed. The author will in no case be liable for any monetary damages
 arising from such loss, damage or destruction.




 On Mon, 5 Sept 2022 at 12:47, Holden Karau 
 wrote:

> I’ve run Jupyter w/Spark on K8s, haven’t tried it with Dataproc
> personally.
>
> The Spark K8s pod scheduler is now more pluggable for Yunikorn and
> Volcano can be used with less effort.
>
> On Mon, Sep 5, 2022 at 7:44 AM Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>>
>> Hi,
>>
>>
>> Has anyone got experience of running Jupyter on dataproc versus
>> Jupyter notebook on GKE (k8).
>>
>>
>> I have not looked at this for a while but my understanding is that
>> Spark on GKE/k8 is not yet performed. This is classic Spark with
>> Python/Pyspark.
>>
>>
>> Also I would like to know the state of spark with Volcano. Has
>> progress made on that front.
>>
>>
>> Regards,
>>
>>
>> Mich
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>> for any loss, damage or destruction of data or any other property which 
>> may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
 --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>

-- 
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge

+47 480 94 297

Re: Error in Spark in Jupyter Notebook

[Sean Owen]

That just says a task failed - no real info there. YOu have to look at
Spark logs from the UI to see why.

On Tue, Sep 6, 2022 at 7:07 AM Mamata Shee 
wrote:

> Hello,
>
> I'm using spark in Jupyter Notebook, but when performing some queries
> getting the below error, can you please tell me what is the actual reason
> for this or any suggestions to make it work?
>
> *Error:*
> [image: image.png]
>
> Thank you
>
> 
>
> CONFIDENTIALITY NOTICE:
> The contents of this email message and any attachments are intended solely
> for the addressee(s)  and may contain confidential and/or privileged
> information and may be legally protected from disclosure. Please do not
> share it with others. If you are not the intended recipient of this
> message, please immediately notify the sender by reply email and destroy
> this message and any attachments. XenonStack monitor's email traffic data
> and the content of email for the purposes of security and confidentiality.
> If you are not the intended recipient, you are hereby notified that any
> use, disseminate ,copying, or storage of this message or its attachments is
> strictly prohibited. Before opening any mail and attachments please check
> them for viruses. Xenonstack does not accept any liability for virus
> infected mails.
>

Error in Spark in Jupyter Notebook

[Mamata Shee]

Hello,

I'm using spark in Jupyter Notebook, but when performing some queries
getting the below error, can you please tell me what is the actual reason
for this or any suggestions to make it work?

*Error:*
[image: image.png]

Thank you

-- 
 



CONFIDENTIALITY NOTICE:
The contents of 
this email message and any attachments are intended solely for the 
addressee(s)  and may contain confidential and/or privileged information 
and may be legally protected from disclosure. Please do not share it with 
others. If you are not the intended recipient of this message, please 
immediately notify the sender by reply email and destroy this message and 
any attachments. XenonStack monitor's email traffic data and the content of 
email for the purposes of security and confidentiality. If you are not the 
intended recipient, you are hereby notified that any use, disseminate 
,copying, or storage of this message or its attachments is strictly 
prohibited. Before opening any mail and attachments please check them for 
viruses. Xenonstack does not accept any liability for virus infected mails.