[GitHub] [zeppelin] liuxunorg commented on issue #3333: [ZEPPELIN-4053]. Implement impersonation via c native api

2019-03-12 Thread GitBox 
liuxunorg commented on issue #: [ZEPPELIN-4053]. Implement impersonation 
via c native api
URL: https://github.com/apache/zeppelin/pull/#issuecomment-472296791
 
 
   For security, YARN uses the yarn user group to execute 
`/etc/yarn/sbin/Linux-amd64-64/container-executor`, which requires special 
settings for the `container-executor` file.
   
   1. Running zeppelin with root can switch users, But this is not safe enough.
   
   2. So need to create a zeppelin user group in the operating system (for 
example: zeppelin-group). The user running the zeppelin service belongs to this 
group zeppelin-group.
   After performing the following operations on the file using the root 
account, the execution user of the zeppelin service can also switch users.
   
   ```
   chown root:${zeppelin-group} zeppelin/bin/execute-as-user
   chmod 6050 zeppelin/bin/execute-as-user
   ```
   
   So need to add some instructions for use.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu opened a new pull request #3333: [ZEPPELIN-4053]. Implement impersonation via c native api

2019-03-12 Thread GitBox 
zjffdu opened a new pull request #: [ZEPPELIN-4053]. Implement 
impersonation via c native api
URL: https://github.com/apache/zeppelin/pull/
 
 
   ### What is this PR for?
   
   This PR use a native c code to implement impersonation. The previous 
approach use ssh which require user to enable password-less for each user. 
While this approach is much easier for user. zeppelin admin only needs to 
compile this c code it to enable impersonation.
   
   ### What type of PR is it?
   [Improvement | Feature | Documentation ]
   
   ### Todos
   * [ ] - Task
   
   ### What is the Jira issue?
   * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
   * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. 
[ZEPPELIN-533]
   
   ### How should this be tested?
   * Tested manually on shell, spark and python interpreter
   
   ### Screenshots (if appropriate)
   
   
![image](https://user-images.githubusercontent.com/164491/54253044-3e9f3200-4588-11e9-809a-7abae961641b.png)
   
   ### Questions:
   * Does the licenses files need update? No
   * Is there breaking changes for older versions? No
   * Does this needs documentation? No
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (ZEPPELIN-4065) Display ipython process launch output when ipython fails to launch

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4065:


 Summary: Display ipython process launch output when ipython fails 
to launch
 Key: ZEPPELIN-4065
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4065
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang


Currently, user have to change log4j.properties to change the log level to 
debug to see this python output, this is not convenient for users, it would be 
better to display the launch output in frontend by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [zeppelin] zjffdu opened a new pull request #3332: ZEPPELIN-4062. Don't wait ipython kernel if python process failed

2019-03-12 Thread GitBox 
zjffdu opened a new pull request #3332: ZEPPELIN-4062. Don't wait ipython 
kernel if python process failed
URL: https://github.com/apache/zeppelin/pull/3332
 
 
   ### What is this PR for?
   This PR will exist wait ipython process loop if the python process is 
failed. This can save time to exit the waiting loop earlier, otherwise it would 
wait until timeout (30 seconds by default) 
   
   ### What type of PR is it?
   [Improvement]
   
   ### Todos
   * [ ] - Task
   
   ### What is the Jira issue?
   * https://jira.apache.org/jira/browse/ZEPPELIN-4062
   
   ### How should this be tested?
   * Manually tested.
   
   Before this PR
   ```
INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
WARN [2019-03-13 10:06:30,810] ({Exec Default Executor} 
IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python 
Process
   org.apache.commons.exec.ExecuteException: Process exited with an error: 1 
(Exit value: 1)
   at 
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
   at 
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
   at 
org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
   at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1}
   ```
   
   After this PR
   ```
   INFO [2019-03-13 10:28:12,805] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:28:12,907] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:28:13,011] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:28:13,116] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:28:13,220] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
INFO [2019-03-13 10:28:13,325] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
WARN [2019-03-13 10:28:13,347] ({Exec Default Executor} 
IPythonInterpreter.java[onProcessFailed]:423) - Exception happens in Python 
Process
   org.apache.commons.exec.ExecuteException: Process exited with an error: 1 
(Exit value: 1)
   at 
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
   at 
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
   at 
org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
   at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:28:13,430] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to 
be started
WARN [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} 
PythonInterpreter.java[open]:98) - Fail to open IPythonInterpreter
   java.lang.RuntimeException: Fail to open IPythonInterpreter
   at 
org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:152)
   at 
org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:93)
   at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
   at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:598)
   at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpr

[jira] [Created] (ZEPPELIN-4064) Add Appveyor to support Windows environment

2019-03-12 Thread Jongyoul Lee (JIRA) 
Jongyoul Lee created ZEPPELIN-4064:
--

 Summary: Add Appveyor to support Windows environment
 Key: ZEPPELIN-4064
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4064
 Project: Zeppelin
  Issue Type: Improvement
Affects Versions: 0.8.1
Reporter: Jongyoul Lee


Some issues related to instability in Windows environment were reported after 
releasing 0.8.1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-4063) Don't include noteId for constructing Interpreter GroupId when under isolated per user mode

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4063:


 Summary: Don't include noteId for constructing Interpreter GroupId 
when under isolated per user mode
 Key: ZEPPELIN-4063
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4063
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang


{code}
INFO [2019-03-13 10:05:52,162] ({qtp640363654-19} 
InterpreterSetting.java[getOrCreateInterpreterGroup]:416) - Create 
InterpreterGroup with groupId: python-user1- for user: user1 and note: 2E6SSZK4P
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-4062) Keep waiting ipython kernel when python process failed

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4062:


 Summary: Keep waiting ipython kernel when python process failed
 Key: ZEPPELIN-4062
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4062
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang


{code}
 INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 WARN [2019-03-13 10:06:30,810] ({Exec Default Executor} 
IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python 
Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit 
value: 1)
at 
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at 
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
 INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:31,119] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:31,223] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:31,329] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
 INFO [2019-03-13 10:06:31,433] ({FIFOScheduler-Worker-1} 
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to 
be started
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-4061) `mvn clean` didn't clear the spark interpreter jar under folder interpreter/spark

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4061:


 Summary: `mvn clean` didn't clear the spark interpreter jar under 
folder interpreter/spark
 Key: ZEPPELIN-4061
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4061
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang


{code}
(python2.7) ➜  zeppelin git:(ZEPPELIN-4053) ✗ ll interpreter/spark
total 110840
drwxr-xr-x  5 jzhang  staff   160B Mar 13 09:55 META-INF
drwxr-xr-x  3 jzhang  staff96B Mar 13 09:55 R
-rw-r--r--  1 jzhang  staff   8.8K Mar 13 10:00 interpreter-setting.json
drwxr-xr-x  4 jzhang  staff   128B Mar 13 10:00 pyspark
drwxr-xr-x  4 jzhang  staff   128B Mar 13 09:55 python
-rw-r--r--  1 jzhang  staff20M Mar 13 09:55 
spark-interpreter-0.8.2-SNAPSHOT.jar
-rw-r--r--  1 jzhang  staff33M Mar 13 10:00 
spark-interpreter-0.9.0-SNAPSHOT.jar
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [zeppelin] asfgit closed pull request #3325: [ZEPPELIN-4030] spark-scala-2.10 won't be built under some profile

2019-03-12 Thread GitBox 
asfgit closed pull request #3325: [ZEPPELIN-4030] spark-scala-2.10 won't be 
built under some profile
URL: https://github.com/apache/zeppelin/pull/3325
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] asfgit closed pull request #3326: [ZEPPELIN-4028]. Pop up error message if user try to create duplicated note

2019-03-12 Thread GitBox 
asfgit closed pull request #3326: [ZEPPELIN-4028]. Pop up error message if user 
try to create duplicated note
URL: https://github.com/apache/zeppelin/pull/3326
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] Leemoonsoo commented on a change in pull request #3329: [Website] fix some broken links. Update some link pointing old docs.

2019-03-12 Thread GitBox 
Leemoonsoo commented on a change in pull request #3329: [Website] fix some 
broken links. Update some link pointing old docs.
URL: https://github.com/apache/zeppelin/pull/3329#discussion_r264851493
 
 

 ##
 File path: technologies_deployments.md
 ##
 @@ -49,15 +49,15 @@ limitations under the License.
 
   Python is supported with Matplotlib, Conda, Pandas SQL and 
PySpark integrations.
 
-USE NOW 
+USE NOW 
   
 
   
 
 
   
 See more details in Zeppelin supports 20+ different interpreters.
-LEARN MORE 

+LEARN MORE 
 
 Review comment:
   supported_interpreters.html is in root directory of `gh-pages` branch, 
instead of docs directory of master. (part of website, not part of zeppelin 
documentation). So the path is correct.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Call for Presentations (CFP) is now open for ApacheCon North America in Las Vegas, September 9-13th

2019-03-12 Thread Felix Cheung 
See the Big Data track or the Machine Learning track!

We’re delighted to announce that the Call for Presentations (CFP) is now
open for ApacheCon North America in Las Vegas, September 9-13th! As the
official conference series of the ASF, ApacheCon North America will
feature over a dozen Apache project summits, including Cassandra,
Cloudstack, Tomcat, Traffic Control, and more. We’re looking for talks
in a wide variety of categories -- anything related to ASF projects and
the Apache development process. The CFP closes at midnight on May 26th.
In addition, the ASF will be celebrating its 20th Anniversary during the
event. For more details and to submit a proposal for the CFP, visit
https://apachecon.com/acna19/ . Registration will be opening soon.

[GitHub] [zeppelin] zjffdu commented on issue #3330: [ZEPPELIN-4054] Fix LdapGroupRealm path in shiro doc

2019-03-12 Thread GitBox 
zjffdu commented on issue #3330: [ZEPPELIN-4054] Fix LdapGroupRealm path in 
shiro doc
URL: https://github.com/apache/zeppelin/pull/3330#issuecomment-472048992
 
 
   LGTM, thanks for the contribution @cquptEthan 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (ZEPPELIN-4060) Upgrade note to include note permission

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4060:


 Summary: Upgrade note to include note permission
 Key: ZEPPELIN-4060
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4060
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang


This is followup of ZEPPELIN-3985



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-4059) Link of 0.8.1 doc is missing on apache site

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4059:


 Summary: Link of 0.8.1 doc is missing on apache site 
 Key: ZEPPELIN-4059
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4059
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [zeppelin] zjffdu commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
zjffdu commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine 
Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#issuecomment-471988787
 
 
   @liuxunorg Regarding integration, tensorflow can be installed in travis. 
(See 
https://github.com/apache/zeppelin/blob/master/testing/install_external_dependencies.sh)
   And We can also set up a mini yarn cluster (See 
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter-integration/src/test/java/org/apache/zeppelin/integration/MiniHadoopCluster.java)
   For docker, I am not sure whether travis can install it, need to investigate 
more. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Hadoop submarine interpreter development report

2019-03-12 Thread Xun Liu 
Hello, everyone

The hadoop community developed the submarine machine learning system, submarine 
has now become the same level of project as hadoop and hdfs in hadoop.
The hadoop community hopes to provide machine learning services such as 
tensorflow, mxnet, pytorch, caffe, etc. through the submarine project.

Hadoop submarine project official website: https://hadoop.apache.org/submarine 
 

Hadoop submarine issue: https://issues.apache.org/jira/browse/SUBMARINE 
 

The hadoop submarine community found zeppelin to be a very good visual 
development platform.
Therefore, I hope to use zeppelin to carry out the whole life cycle of Hadoop 
submarine algorithm development, 
model offline training, model release, model online prediction, and model 
incremental update.

After 3 months of development, We completed the development of the hadoop 
submarine (machine learning) interpreter.

And it will be used online for one month in our company.
It is now possible to write tensorflow algorithms in zeppelin.
The tensorflow algorithm is submitted to YARN for offline training in zeppelin.

Hadoop submarine interpreter design doc: 
https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd
 

 
Zeppelin add Hadoop Submarine (machine learning) interpreter: 
https://issues.apache.org/jira/browse/ZEPPELIN-3856 
 
PR : https://github.com/apache/zeppelin/pull/3331 


submarine-integration-zeppelin doc： 
https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/tree/master/submarine-integration-zeppelin
 

Youtebe Submarine channel：https:// 
www.youtube.com/channel/UC4JBt8Y8VJ0BW0IM9YpdCyQ
 


welcome to submit their own ideas and opinions.

Thanks!


Xun Liu
March 12, 2019

[GitHub] [zeppelin] liuxunorg commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
liuxunorg commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine 
Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#issuecomment-471957740
 
 
   I also thought about the way to provide integration testing.
   However, machine learning requires strict environmental requirements.
   For example, need to install `docker`, `tensorflow`, and `YARN` in test 
environment.
   This requires a process of gradual improvement.
   Therefore, only documents and operation videos can be provided at present.
   
   Doc: 
https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/tree/master/submarine-integration-zeppelin
   Youtebe：https://www.youtube.com/channel/UC4JBt8Y8VJ0BW0IM9YpdCyQ


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r264623929
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/PySubmarineInterpreter.java
 ##
 @@ -0,0 +1,84 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.thrift.ParagraphInfo;
+import org.apache.zeppelin.python.IPythonInterpreter;
+import org.apache.zeppelin.python.PythonInterpreter;
+import org.apache.zeppelin.submarine.commons.SubmarineConstants;
+import org.apache.zeppelin.submarine.job.SubmarineJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Properties;
+
+public class PySubmarineInterpreter extends PythonInterpreter {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PySubmarineInterpreter.class);
+
+  private SubmarineInterpreter submarineInterpreter = null;
+  private SubmarineContext submarineContext = null;
+
+  public PySubmarineInterpreter(Properties property) {
+super(property);
+submarineContext = SubmarineContext.getInstance();
+  }
+
+  @Override
+  public InterpreterResult interpret(String st, InterpreterContext context)
+  throws InterpreterException {
+// algorithm & checkpoint path support replaces ${username} with real user 
name
+String algorithmPath = properties.getProperty(
 
 Review comment:
   Now our users are using `python` to develop machine learning algorithms. 
Because submarine now offers online services, And have a certain amount of 
code, So I hope I can merge this phase code first. Subsequent improvements to 
ipython support.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r264619167
 
 

 ##
 File path: submarine/pom.xml
 ##
 @@ -0,0 +1,159 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+
+  
+zeppelin-interpreter-parent
+org.apache.zeppelin
+0.9.0-SNAPSHOT
+../zeppelin-interpreter-parent/pom.xml
+  
+
+  org.apache.zeppelin
+  zeppelin-submarine
+  jar
+  0.9.0-SNAPSHOT
+  Zeppelin: Submarine interpreter
+
+  
+
+submarine
+2.7.3
 
 Review comment:
   Here Hadoop is used to upload algorithm files to HDFS, Hadoop3.1+ needs to 
be deployed locally like a spark. I have instructions in the 
`interpreter/submarine.md` documentation.
   Submarine is developing a client that does not rely on hadoop, There is only 
one JAR file package. I will update it when it is released.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
zjffdu commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine 
Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#issuecomment-471936228
 
 
   @liuxunorg Is it possible to add integration test for this interpreter ? 
E.g. Run one deep learning task via SubmarineInterpreter ? I understand it may 
be not possible because currently submarine is not mature enough, just want to 
bring this out if it is possible. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
zjffdu commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r264600429
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/PySubmarineInterpreter.java
 ##
 @@ -0,0 +1,84 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.thrift.ParagraphInfo;
+import org.apache.zeppelin.python.IPythonInterpreter;
+import org.apache.zeppelin.python.PythonInterpreter;
+import org.apache.zeppelin.submarine.commons.SubmarineConstants;
+import org.apache.zeppelin.submarine.job.SubmarineJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Properties;
+
+public class PySubmarineInterpreter extends PythonInterpreter {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PySubmarineInterpreter.class);
+
+  private SubmarineInterpreter submarineInterpreter = null;
+  private SubmarineContext submarineContext = null;
+
+  public PySubmarineInterpreter(Properties property) {
+super(property);
+submarineContext = SubmarineContext.getInstance();
+  }
+
+  @Override
+  public InterpreterResult interpret(String st, InterpreterContext context)
+  throws InterpreterException {
+// algorithm & checkpoint path support replaces ${username} with real user 
name
+String algorithmPath = properties.getProperty(
 
 Review comment:
   Should this piece of code be applied in IPySubmarineInterpreter ? IIUC, 
IPySubmarineInterpreter should behave the same as PySubmarineInterpreter except 
more ipython features


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
zjffdu commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r264599725
 
 

 ##
 File path: submarine/pom.xml
 ##
 @@ -0,0 +1,159 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+
+  
+zeppelin-interpreter-parent
+org.apache.zeppelin
+0.9.0-SNAPSHOT
+../zeppelin-interpreter-parent/pom.xml
+  
+
+  org.apache.zeppelin
+  zeppelin-submarine
+  jar
+  0.9.0-SNAPSHOT
+  Zeppelin: Submarine interpreter
+
+  
+
+submarine
+2.7.3
 
 Review comment:
   Do I need to change it 3.x if I want to use submarine in hadoop 3 ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox 
zjffdu commented on issue #3331: [Zeppelin-4049] Hadoop Submarine (Machine 
Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#issuecomment-471933878
 
 
   Thanks @liuxunorg  It is a very interested interpreter for zeppelin in deep 
learning area. Since it is a very PR, could you send one mail in user mail list 
to gather more attention on this feature ? I hope more people could try it and 
gave feedback. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (ZEPPELIN-4058) Merge method NotebookAuthorization #hasReadAuthorization and NotebookAuthorization #isReader

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4058:


 Summary: Merge method NotebookAuthorization #hasReadAuthorization 
and NotebookAuthorization #isReader
 Key: ZEPPELIN-4058
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4058
 Project: Zeppelin
  Issue Type: Improvement
Reporter: Jeff Zhang


These 2 methods do similar things and is very hard to figure out what's the 
difference, so should merge them into one method



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-4057) No commit when deleting note in GitNotebookRepo

2019-03-12 Thread Jeff Zhang (JIRA) 
Jeff Zhang created ZEPPELIN-4057:


 Summary: No commit when deleting note in GitNotebookRepo
 Key: ZEPPELIN-4057
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4057
 Project: Zeppelin
  Issue Type: New Feature
Reporter: Jeff Zhang


That's means once note can not roll back if it is deleted and never committed 
before



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [zeppelin] zjffdu commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file

2019-03-12 Thread GitBox 
zjffdu commented on a change in pull request #3320: [ZEPPELIN-3977]. Create 
shell script for converting old note file to new file
URL: https://github.com/apache/zeppelin/pull/3320#discussion_r264550663
 
 

 ##
 File path: docs/setup/operation/upgrading.md
 ##
 @@ -35,6 +35,12 @@ So, copying `notebook` and `conf` directory should be 
enough.
 
 ## Migration Guide
 
+### Upgrading from Zeppelin 0.8 to 0.9
+
+ - From 0.9, we change the notes file name structure 
([ZEPPELIN-2619](https://issues.apache.org/jira/browse/ZEPPELIN-2619)). So when 
you upgrading zeppelin to 0.9, you need to upgrade note file. Here's steps you 
need to follow:
 
 Review comment:
   @felixcheung I have fixed by adding commit message during note message.
   Here's the what I see when I upgrade the built-in zeppelin tutorial notes. 
But there's no commit message when deleting note. This is a bug of 
`GitNotebookRepo`
   
   ```
   commit 7b1e12fe720fcaee916b7be7494e587d766703cd (HEAD -> master)
   Author: Jeff Zhang 
   Date:   Tue Mar 12 15:36:46 2019 +0800
   
   Upgrade note 'Untitled Note 1' from old format to new format
   
   commit db9de85f65896f02f8ed112d84ca0ba89d09625b
   Author: Jeff Zhang 
   Date:   Tue Mar 12 15:36:46 2019 +0800
   
   Upgrade note 'Basic Features (Spark)' from old format to new format
   
   commit cf0c44f3c6ecf17ed846ce953c4d281a9623b4b5
   Author: Jeff Zhang 
   Date:   Tue Mar 12 15:36:45 2019 +0800
   
   Upgrade note 'Using Pig for querying data' from old format to new format
   
   commit b4fb0f512b0f5862983a308c792ea1ab6bd2351f
   Author: Jeff Zhang 
   Date:   Tue Mar 12 15:36:45 2019 +0800
   
   Upgrade note 'Matplotlib (Python • PySpark)' from old format to new 
format
   
   commit 6f1d7608adb793f5efa58b9916eb5ff36002f236
   Author: Jeff Zhang 
   Date:   Tue Mar 12 15:36:45 2019 +0800
   
   Upgrade note 'R (SparkR)' from old format to new format
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] asfgit closed pull request #3327: [ZEPPELIN-4048]. Rename SecurityService to AuthenticationService

2019-03-12 Thread GitBox 
asfgit closed pull request #3327: [ZEPPELIN-4048]. Rename SecurityService to 
AuthenticationService
URL: https://github.com/apache/zeppelin/pull/3327
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] asfgit closed pull request #3328: [ZEPPELIN-4051]. Commit note is broken

2019-03-12 Thread GitBox 
asfgit closed pull request #3328: [ZEPPELIN-4051]. Commit note is broken
URL: https://github.com/apache/zeppelin/pull/3328
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu commented on issue #3255: [ZEPPELIN-3095] fix UI when paragraphs run sequential (in v0.8)

2019-03-12 Thread GitBox 
zjffdu commented on issue #3255: [ZEPPELIN-3095] fix UI when paragraphs run 
sequential (in v0.8)
URL: https://github.com/apache/zeppelin/pull/3255#issuecomment-471884700
 
 
   I don't think this is critical issue for branch-0.8, and am not so familiar 
with frontend, so I am not so confident to merge into branch-0.8. If anyone who 
is familiar with frontend can help review, that would be great.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] zjffdu commented on issue #3327: [ZEPPELIN-4048]. Rename SecurityService to AuthenticationService

2019-03-12 Thread GitBox 
zjffdu commented on issue #3327: [ZEPPELIN-4048]. Rename SecurityService to 
AuthenticationService
URL: https://github.com/apache/zeppelin/pull/3327#issuecomment-471884063
 
 
   Thanks for review @felixcheung Authorization will be done in #3316


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services