[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
Yes， we need to support scala 2.11 for spark 2.4 first. 
And please also update travis.yml to build it with spark-2.4 profile


---

[GitHub] zeppelin issue #3034: [WIP] ZEPPELIN-3552. Support Scala 2.12 of SparkInterp...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/zeppelin/pull/3034
  
> Spark 2.4 will officially support Scala 2.12, so it will be great if 
Zeppelin will support it together with Spark. And also, there are some libs 
that are Scala 2.12 only

The default distribution will still be with Scala 2.11 for Spark 2.4 if I 
am not mistaken. It is nice to support it but Spark 2.4 with 2.11 should be 
supported first as a higher priority. I can work on 2.4.0 with 2.12 support 
further after this one got merged.


---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon closed the pull request at:

https://github.com/apache/zeppelin/pull/3206


---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

GitHub user HyukjinKwon reopened a pull request:

https://github.com/apache/zeppelin/pull/3206

[WIP][ZEPPELIN-3810] Support Spark 2.4

### What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see 
SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, 
`loopPostInit` became inaccessible.

See:
 - 
https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - 
https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain 
the behaviour. Some functions that are commonly existing at both Scala 2.11.8 
and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.


2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. 
Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError: 

jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at 
scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to 
`2.14.3`.


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810

### How should this be tested?

Verified manually against Spark 2.4.0 RC3

### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon 
Date:   2018-10-17T14:41:29Z

Support Spark 2.4




---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

GitHub user HyukjinKwon reopened a pull request:

https://github.com/apache/zeppelin/pull/3206

[WIP][ZEPPELIN-3810] Support Spark 2.4

### What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see 
SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, 
`loopPostInit` became inaccessible.

See:
 - 
https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - 
https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain 
the behaviour. Some functions that are commonly existing at both Scala 2.11.8 
and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.


2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. 
Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError: 

jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at 
scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to 
`2.14.3`.


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810

### How should this be tested?

Verified manually against Spark 2.4.0 RC3

### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon 
Date:   2018-10-17T14:41:29Z

Support Spark 2.4




---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon closed the pull request at:

https://github.com/apache/zeppelin/pull/3206


---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon closed the pull request at:

https://github.com/apache/zeppelin/pull/3206


---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

GitHub user HyukjinKwon reopened a pull request:

https://github.com/apache/zeppelin/pull/3206

[WIP][ZEPPELIN-3810] Support Spark 2.4

### What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see 
SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, 
`loopPostInit` became inaccessible.

See:
 - 
https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - 
https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain 
the behaviour. Some functions that are commonly existing at both Scala 2.11.8 
and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.


2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. 
Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError: 

jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at 
scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to 
`2.14.3`.


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810

### How should this be tested?

Verified manually against Spark 2.4.0 RC3

### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon 
Date:   2018-10-17T14:41:29Z

Support Spark 2.4




---

[jira] [Created] (ZEPPELIN-3816) after moderate usage, can no longer use Spark2

[Charles Hedrick (JIRA)]

Charles Hedrick created ZEPPELIN-3816:
-

 Summary: after moderate usage, can no longer use Spark2
 Key: ZEPPELIN-3816
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3816
 Project: Zeppelin
  Issue Type: Bug
  Components: zeppelin-interpreter
 Environment: h3. spark2 %spark2, %spark2.sql, %spark2.dep, 
%spark2.pyspark, %spark2.r 
 spark ui  edit  restart  remove
h5. Option
The interpreter will be instantiated Per User  in isolated  process. 
 
User Impersonate
Connect to existing process
Set permission
 
h5. Properties
||name||value||
|SPARK_HOME|/usr/hdp/current/spark2-client/|
|args| |
|master|local[*]|
|spark.app.name|Zeppelin|
|spark.cores.max| |
|spark.executor.memory| |
|zeppelin.R.cmd|R|
|zeppelin.R.image.width|100%|
|zeppelin.R.knitr|true|
|zeppelin.R.render.options|out.format = 'html', comment = NA, echo = FALSE, 
results = 'asis', message = F, warning = F|
|zeppelin.dep.additionalRemoteRepository|spark-packages,http://dl.bintray.com/spark-packages/maven,false;|
|zeppelin.dep.localrepo|local-repo|
|zeppelin.interpreter.localRepo|/usr/hdp/current/zeppelin-server/local-repo/2DRMGSB7A|
|zeppelin.interpreter.output.limit|102400|
|zeppelin.pyspark.python|/usr/local/bin/zsparkpy|
|zeppelin.spark.concurrentSQL|false|
|zeppelin.spark.importImplicit|true|
|zeppelin.spark.maxResult|1000|
|zeppelin.spark.printREPLOutput|true|
|zeppelin.spark.sql.stacktrace|false|
|zeppelin.spark.useHiveContext|true|
Reporter: Charles Hedrick
 Fix For: 0.7.3


This is Zeppelin installed as part of HDP 2.6.3.0-235

We have a Zeppelin system being used by a large class. Everything except MD is 
configured to run with user impersonation, isolated. Users primarily use spark2.

After a while the system becomes unusable. I've been restarting once a day, but 
today even that wasn't enough. Once the problem occurs we get this kind of 
error:

Restarting my interpreter doesn't help, and indeed I believe this happens to 
all users.

Livy2 still works.

Our system is kerberized. Users get Kerberos credentials when they login 
automatically (via PAM).

ERROR [2018-10-17 16:04:55,608] (\{Thread-2817} 
RemoteInterpreterEventPoller.java[run]:113) - Can't get RemoteInterpreterEvent
 org.apache.thrift.transport.TTransportException
 at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
 at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
 at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
 at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
 at 
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getEvent(RemoteInterpreterService.java:429)
 at 
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getEvent(RemoteInterpreterService.java:417)
 at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:110)
 ERROR [2018-10-17 16:04:55,620] (\{Thread-2819} 
JobProgressPoller.java[run]:54) - Can not get or update progress
 org.apache.zeppelin.interpreter.InterpreterException: 
org.apache.thrift.transport.TTransportException: java.net.SocketException: 
Connection reset

at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:500)
 at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:121)
 at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:333)
 at 
org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:51)
 Caused by: org.apache.thrift.transport.TTransportException: 
java.net.SocketException: Connection reset
 at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
 at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
 at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
 at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
 at 
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getProgress(RemoteInterpreterService.java:313)
 at 
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getProgress(RemoteInterpreterService.java:298)
 at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:497)
 ... 3 more

Caused by: java.net.SocketException: Connection reset

        at java.net.SocketInputStream.read(SocketInputStream.java:209)

        at java.net.SocketInputStream.read(SocketInputStream.java:141

[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
oops. I haven't. Will check that too while I am here.

BTW, my understanding is that we need this one as well since Spark still 
can be compiled against Scala 2.11.x, am I in the right way?


---

[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
Thanks @HyukjinKwon Have you checked this PR 
(https://github.com/apache/zeppelin/pull/3034) for supporting scala 2.12


---

[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
This is a WIP. We should wait for Spark 2.4.0.

cc @zjffdu and @felixcheung


---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/zeppelin/pull/3206

[WIP][ZEPPELIN-3810] Support Spark 2.4

### What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see 
SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, 
`loopPostInit` became inaccessible.

See:
 - 
https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - 
https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain 
the behaviour. Some functions that are commonly existing at both Scala 2.11.8 
and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.


2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. 
Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError: 

jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at 
scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to 
`2.14.3`.


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810

### How should this be tested?

Verified manually against Spark 2.4.0 RC3

### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon 
Date:   2018-10-17T14:41:29Z

Support Spark 2.4




---

[GitHub] zeppelin issue #3013: [ZEPPELIN-3511] remove old button "Download Data as CS...

[jongyoul]

Github user jongyoul commented on the issue:

https://github.com/apache/zeppelin/pull/3013
  
Sure ~

On Wed, Oct 17, 2018 at 3:22 PM Felix Cheung 
wrote:

> ok, let's go ahead then?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>


-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net



---

[jira] [Created] (ZEPPELIN-3815) No interpreter group name in spark interpreter log file

[Jeff Zhang (JIRA)]

Jeff Zhang created ZEPPELIN-3815:


 Summary: No interpreter group name in spark interpreter log file
 Key: ZEPPELIN-3815
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3815
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] zeppelin pull request #3205: [ZEPPELIN-3814] Add apply button to table setti...

[egorklimov]

GitHub user egorklimov opened a pull request:

https://github.com/apache/zeppelin/pull/3205

[ZEPPELIN-3814] Add apply button to table settings

### What is this PR for?
Now changes in table settings applies only after page refreshing which 
isn't convenient
* Settings menu before PR:

![screenshot-1](https://user-images.githubusercontent.com/6136993/47079792-f49b6580-d20e-11e8-837e-98ba5443f3b3.png)
* Menu with new button:

![screen](https://user-images.githubusercontent.com/6136993/47079810-05e47200-d20f-11e8-91f0-38e1a57067e0.gif)

### What type of PR is it?
Improvement

### What is the Jira issue?
* issue - https://issues.apache.org/jira/browse/ZEPPELIN-3814

### How should this be tested?
* CI pass
* Manually tested

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/TinkoffCreditSystems/zeppelin DW-17956

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3205.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3205


commit e9b5d11813aa15b0eedb5387d67594bd7195aed6
Author: George Klimov 
Date:   2018-10-10T13:13:26Z

button added




---

[jira] [Created] (ZEPPELIN-3814) Add apply button to table settings

[George Klimov (JIRA)]

George Klimov created ZEPPELIN-3814:
---

 Summary: Add apply button to table settings
 Key: ZEPPELIN-3814
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3814
 Project: Zeppelin
  Issue Type: Improvement
  Components: GUI
Reporter: George Klimov
Assignee: George Klimov
 Attachments: screenshot-1.png

Now changes in table settings applies only after page refreshing which isn't 
conveniently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ZEPPELIN-3813) Sync Button per Notebook

[minhyeok kweun (JIRA)]

minhyeok kweun created ZEPPELIN-3813:


 Summary: Sync Button per Notebook
 Key: ZEPPELIN-3813
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3813
 Project: Zeppelin
  Issue Type: Improvement
  Components: GUI
Reporter: minhyeok kweun


My suggestion is to add a refresh button per notebook to sync, which may 
improve user experience of Zeppelin. In Zeppelin, currently we have a *refresh 
button* at the front page to sync notebooks to an external storage like HDFS. I 
found this function very essential especially to support multi-instance 
Zeppelin with a shared HDFS notebook storage (also with a load balancer). Since 
you should allow a user to access the same version of his/her notebooks at any 
instance. However in this kind of multi-user environment, if a user click the 
button, not only his/her notebooks but all others on that instance will be 
affected. In my opinion It may cause some lag or some inconveniences in using 
Zeppelin. So I thought it may get better if we have a button per notebook with 
which we can refresh only the specific notebook.

 

(I beg your understanding since my English is not perfect and I don't know 
exactly how Zeppelin works internally, please correct me if I'm wrong)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)