hi, community I updated the design documentation, https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> added the interactive development of machine learning algorithms and the overall design of submitting Note packages to YARN for job execution.
I have completed the development of the pre-research part of the code, and established JIRA: https://issues.apache.org/jira/browse/ZEPPELIN-3856 <https://issues.apache.org/jira/browse/ZEPPELIN-3856> welcome comments, thank you! > 在 2018年10月30日，下午10:11，liuxun <neliu...@163.com> 写道： > > HI, community > > > I updated the design document and submitted it to the hadoop community. > https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# > > <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> > > If you have any questions, you can ask them in the email or directly in the > document. Thank you! > > I developed the submarine interpreter module in zeppelin. Below is the > interface for the system to run. > > 1. zeppelin submarine interpreter properties > <submarine-interpreter.png> > 1.1 System administrators can configure multiple interpreters, and different > interpreters can have different resource configurations. > 1.2 Allow different users to use different interpreters for resource > allocation and management. > > 2. zeppelin submarine Tensorflow interpreter > <submarine-tensorflow.png> > 2.1 Users use their own interpreter, First through the %submarine.tensorflow > paragraph, To write python code for tensorflow. > 2.2 After the user writes the python code of tensorflow, Click the [RUN] > button and zeppelin will upload the python code to the specified HDFS > directory. The submarine is loaded into the docker container when it is > running. > > 3. zeppelin submarine interpreter > <submarine-command.png> > 3.1 The user sets the call parameter value of the tensorflow python code, > Then enter the job run command. > 3.2 The zeppelin submarine interpreter first checks that all parameters of > the submarine run are set completely. After the check is passed, the > implementation of YARN is performed via submarine.jar. > 3.3 The progress and log information of the submarine will be displayed in > zeppelin's note. > 3.4 You can enter submarine's other commands in the submarine interpreter, > view all the jobs in the submarine, etc. > > >> 在 2018年10月30日，下午6:40，liuxun <neliu...@163.com <mailto:neliu...@163.com>> 写道： >> >> >> HI, >> >> >> I updated the design document and submitted it to the hadoop community. >> >> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# >> >> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#> >> >> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit# >> >> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#>> >> >> If you have any questions, you can ask them in the email or directly in >> the document. Thank you! >> >> I developed the submarine interpreter module in zeppelin. Below is the >> interface for the system to run. >> >> zeppelin submarine interpreter properties >> >> System administrators can configure multiple interpreters, and different >> interpreters can have different resource configurations. >> Allow different users to use different interpreters for resource allocation >> and management. >> zeppelin submarine Tensorflow interpreter >> >> Users use their own interpreter, First through the %submarine.tensorflow >> paragraph, To write python code for tensorflow. >> After the user writes the python code of tensorflow, Click the [RUN] button >> and zeppelin will upload the python code to the specified HDFS directory. >> The submarine is loaded into the docker container when it is running. >> >> zeppelin submarine interpreter >> >> The user sets the call parameter value of the tensorflow python code, Then >> enter the job run command. >> The zeppelin submarine interpreter first checks that all parameters of the >> submarine run are set completely. After the check is passed, the >> implementation of YARN is performed via submarine.jar. >> The progress and log information of the submarine will be displayed in >> zeppelin's note. >> You can enter submarine's other commands in the submarine interpreter, view >> all the jobs in the submarine, etc. >> >> >> >>> 在 2018年10月21日，上午6:26，Felix Cheung <felixcheun...@hotmail.com >>> <mailto:felixcheun...@hotmail.com>> 写道： >>> >>> Very cool! >>> >>> >>> ________________________________ >>> From: Jeff Zhang <zjf...@gmail.com <mailto:zjf...@gmail.com>> >>> Sent: Friday, October 19, 2018 7:14 AM >>> To: d...@zeppelin.apache.org <mailto:d...@zeppelin.apache.org> >>> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) >>> interpreter >>> >>> Thanks xun. This would be a great addon for zeppelin to support deep >>> learning. I will check the design later. >>> >>> >>> >>> liu xun <neliu...@163.com <mailto:neliu...@163.com>>于2018年10月19日周五 下午3:56写道： >>> >>>> Hi, >>>> Hadoop Submarine is the latest machine learning framework subproject in >>>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow, >>>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a >>>> full-featured system framework for machine learning algorithm development, >>>> distributed model training, model management, and model publishing, >>>> combined with hadoop's intrinsic data storage and data processing >>>> capabilities to enable data scientists to Good mining and the value of the >>>> data. >>>> >>>> >>>> I was involved in the development of the hadoop submarine project. So I >>>> plan to add the interpreter module of hadoop submarine to zeppelin. >>>> Let zeppeline increase the development of deep learning. This is my design >>>> document, let's see if there is any opinion, you can put it directly in the >>>> document, thank you! >>>> >>>> >>>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd >>>> >>>> <https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd> >>>> >>>> >>>> >> >
