[
https://issues.apache.org/jira/browse/LIVY-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mnr Bsf updated LIVY-1010:
--------------------------
Description:
It will be good to keep the Apache Livy project up to date with the latest
Spark version (3.5.4).
Tried Apache Livy 0.8 for Spark 3.5.4 and did not work as Spark version 3.0.0
with Java 11, 17.
Java 17, I was not able to start an Apache Livy session
{code:java}
Exception in thread "main" java.util.concurrent.ExecutionException:
javax.security.sasl.SaslException: Client closed before SASL negotiation
finished {code}
Java 11, I was able to start an Apache Livy session but running alivy-server |
25/02/05 17:08:27 INFO LineBufferedStream: Type --help for more information.
livy-server | 25/02/05 17:08:27 WARN LivySparkUtils$: Current Spark (3,5) is
not verified in Livy, please use it carefully livy-server | 25/02/05 17:08:27
WARN LivySparkUtils$: Spark version (3,5) is greater then the maximum version
(3,0) supported by Livy, will choose Scala version 2.12 instead, please specify
manually if it is the expected Scala version you want livy-server | Exception
in thread "main" java.lang.ExceptionInInitializerError livy-server | at
org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) livy-server |
at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
livy-server | at
org.apache.livy.server.LivyServer.start(LivyServer.scala:106) livy-server |
at org.apache.livy.server.LivyServer$.main(LivyServer.scala:431) livy-server |
at org.apache.livy.server.LivyServer.main(LivyServer.scala) livy-server |
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
livy-server | at
java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) livy-server |
at java.base/java.lang.String.substring(String.java:1874) livy-server | at
org.apache.hadoop.util.Shell.<clinit>(Shell.java:52) livy-server | ... 9 more
was:
It will be good to keep the Apache Livy project up to date with the latest
Spark version (3.5.4).
Tried Apache Livy 0.8 for Spark 3.5.4 and did not work with Java 8, 11, 17.
Java 17, I was not able to start an Apache Livy session
{code:java}
Exception in thread "main" java.util.concurrent.ExecutionException:
javax.security.sasl.SaslException: Client closed before SASL negotiation
finished {code}
Java 8, 11, I was not able to start an Apache Livy session
{code:java}
livy-server | 25/02/05 17:08:27 INFO LineBufferedStream: Type --help for more
information.
livy-server | 25/02/05 17:08:27 WARN LivySparkUtils$: Current Spark (3,5) is
not verified in Livy, please use it carefully
livy-server | 25/02/05 17:08:27 WARN LivySparkUtils$: Spark version (3,5) is
greater then the maximum version (3,0) supported by Livy, will choose Scala
version 2.12 instead, please specify manually if it is the expected Scala
version you want
livy-server | Exception in thread "main" java.lang.ExceptionInInitializerError
livy-server | at
org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
livy-server | at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
livy-server | at org.apache.livy.server.LivyServer.start(LivyServer.scala:106)
livy-server | at org.apache.livy.server.LivyServer$.main(LivyServer.scala:431)
livy-server | at org.apache.livy.server.LivyServer.main(LivyServer.scala)
livy-server | Caused by: java.lang.StringIndexOutOfBoundsException: begin 0,
end 3, length 2
livy-server | at
java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
livy-server | at java.base/java.lang.String.substring(String.java:1874)
livy-server | at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)
livy-server | ... 9 more {code}
> Add support for Spark 3.5.4
> ---------------------------
>
> Key: LIVY-1010
> URL: https://issues.apache.org/jira/browse/LIVY-1010
> Project: Livy
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Mnr Bsf
> Priority: Major
> Fix For: 0.8.0
>
>
> It will be good to keep the Apache Livy project up to date with the latest
> Spark version (3.5.4).
> Tried Apache Livy 0.8 for Spark 3.5.4 and did not work as Spark version 3.0.0
> with Java 11, 17.
> Java 17, I was not able to start an Apache Livy session
> {code:java}
> Exception in thread "main" java.util.concurrent.ExecutionException:
> javax.security.sasl.SaslException: Client closed before SASL negotiation
> finished {code}
> Java 11, I was able to start an Apache Livy session but running alivy-server
> | 25/02/05 17:08:27 INFO LineBufferedStream: Type --help for more
> information. livy-server | 25/02/05 17:08:27 WARN LivySparkUtils$: Current
> Spark (3,5) is not verified in Livy, please use it carefully livy-server |
> 25/02/05 17:08:27 WARN LivySparkUtils$: Spark version (3,5) is greater then
> the maximum version (3,0) supported by Livy, will choose Scala version 2.12
> instead, please specify manually if it is the expected Scala version you want
> livy-server | Exception in thread "main"
> java.lang.ExceptionInInitializerError livy-server | at
> org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) livy-server
> | at
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
> livy-server | at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> livy-server | at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> livy-server | at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
> livy-server | at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
> livy-server | at
> org.apache.livy.server.LivyServer.start(LivyServer.scala:106) livy-server |
> at org.apache.livy.server.LivyServer$.main(LivyServer.scala:431) livy-server
> | at org.apache.livy.server.LivyServer.main(LivyServer.scala) livy-server |
> Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length
> 2 livy-server | at
> java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) livy-server
> | at java.base/java.lang.String.substring(String.java:1874) livy-server |
> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52) livy-server | ... 9
> more
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
