[jira] [Commented] (SPARK-12312) JDBC connection to Kerberos secured databases fails on remote executors
[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710898#comment-16710898 ] Alan commented on SPARK-12312: -- I agree! Can we please get this implemented as soon as possible? This prevents us from being compliant with our internal security policies. > JDBC connection to Kerberos secured databases fails on remote executors > --- > > Key: SPARK-12312 > URL: https://issues.apache.org/jira/browse/SPARK-12312 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.2 >Reporter: nabacg >Priority: Minor > > When loading DataFrames from JDBC datasource with Kerberos authentication, > remote executors (yarn-client/cluster etc. modes) fail to establish a > connection due to lack of Kerberos ticket or ability to generate it. > This is a real issue when trying to ingest data from kerberized data sources > (SQL Server, Oracle) in enterprise environment where exposing simple > authentication access is not an option due to IT policy issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24421) Accessing sun.misc.Cleaner in JDK11
[
https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682951#comment-16682951
]
Alan edited comment on SPARK-24421 at 11/11/18 6:13 PM:
Sean: In the proposed release note text you say that access to the internal JDK
classes "is no longer possible in Java 9 and later, because of the new module
encapsulation system". I don't think this is quite right as the main issue you
ran into is that the JDK's internal cleaner mechanism was refactored and moved
from sun.misc to jdk.internal.ref. Also the comment about using add-opens may
need update too as java.lang remains open to code on the class path in JDK
9/10/11. I suspect the the comments in this issue about add-opens meant to say
jdk.internal.ref instead (although you just don't want to go there as directly
using anything in that package may break at any time).
was (Author: bateman):
Sean - In the proposed release note text you say that access to the internal
JDK classes "is no longer possible in Java 9 and later, because of the new
module encapsulation system". I don't think this is quite right as the main
issue you ran into is that the JDK's internal cleaner mechanism was refactored
and moved from sun.misc to jdk.internal.ref. Also the comment about using
`--add-opens` may need update too as java.lang remains open to code on the
class path in JDK 9/10/11. I suspect the the comments in this issue about
`--add-opens` meant to say jdk.internal.ref instead (although you just don't
want to go there as directly using anything in that package may break at any
time).
> Accessing sun.misc.Cleaner in JDK11
> ---
>
> Key: SPARK-24421
> URL: https://issues.apache.org/jira/browse/SPARK-24421
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: DB Tsai
>Priority: Major
> Labels: release-notes
>
> Many internal APIs such as unsafe are encapsulated in JDK9+, see
> http://openjdk.java.net/jeps/260 for detail.
> To use Unsafe, we need to add *jdk.unsupported* to our code’s module
> declaration:
> {code:java}
> module java9unsafe {
> requires jdk.unsupported;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24421) Accessing sun.misc.Cleaner in JDK11
[
https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682951#comment-16682951
]
Alan commented on SPARK-24421:
--
Sean - In the proposed release note text you say that access to the internal
JDK classes "is no longer possible in Java 9 and later, because of the new
module encapsulation system". I don't think this is quite right as the main
issue you ran into is that the JDK's internal cleaner mechanism was refactored
and moved from sun.misc to jdk.internal.ref. Also the comment about using
`--add-opens` may need update too as java.lang remains open to code on the
class path in JDK 9/10/11. I suspect the the comments in this issue about
`--add-opens` meant to say jdk.internal.ref instead (although you just don't
want to go there as directly using anything in that package may break at any
time).
> Accessing sun.misc.Cleaner in JDK11
> ---
>
> Key: SPARK-24421
> URL: https://issues.apache.org/jira/browse/SPARK-24421
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: DB Tsai
>Priority: Major
> Labels: release-notes
>
> Many internal APIs such as unsafe are encapsulated in JDK9+, see
> http://openjdk.java.net/jeps/260 for detail.
> To use Unsafe, we need to add *jdk.unsupported* to our code’s module
> declaration:
> {code:java}
> module java9unsafe {
> requires jdk.unsupported;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24421) Accessing sun.misc.Cleaner in JDK11
[
https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681151#comment-16681151
]
Alan commented on SPARK-24421:
--
If I understanding correctly, the high-level need is
-XX:MaxDirectMemorySize=unlimited but without specifying a command line option.
Do you specify any other arguments? Maybe you could include an arg file with
all options?
As regards the hack then it looks like it involves the non-public constructor
needed for JNI NewDirectMemoryBuffer and then patching the cleaner field. Ugh,
that it way too fragile as the JDK internals can change at any time, also
hacking into buffer fields will break once java.base is fully encapsulated.
> Accessing sun.misc.Cleaner in JDK11
> ---
>
> Key: SPARK-24421
> URL: https://issues.apache.org/jira/browse/SPARK-24421
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: DB Tsai
>Priority: Major
>
> Many internal APIs such as unsafe are encapsulated in JDK9+, see
> http://openjdk.java.net/jeps/260 for detail.
> To use Unsafe, we need to add *jdk.unsupported* to our code’s module
> declaration:
> {code:java}
> module java9unsafe {
> requires jdk.unsupported;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24421) sun.misc.Unsafe in JDK11
[
https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680355#comment-16680355
]
Alan commented on SPARK-24421:
--
The comment that sun.misc.Unsafe is private and not accessible in JDK 9 or
newer releases is not correct. When compiling or running code on the class path
then sun.misc.Unsafe works as it did in JDK 8 and older releases. Yes, you need
to use core reflection to get the Unsafe instance but this is no different to
JDK 8 and older. If developing a module then the module should `requires
jdk.unsupported` as per the description. The jdk.unsupported module opens the
sun.misc package so you can use reflection to get at the Unsafe instances in
the same way as code on the class path.
As regards freeing the memory underlying of a reachable direct buffer then this
is always a very dangerous as further access to the buffer will lead to a crash
and security issues. So anything doing this needs to be really careful and
immediately discard all references to the Buffer object. There is no need to
hack private fields to get at Cleaner objects with JDK 9 or newer, instead look
at the Unsafe invokeCleaner method which will do what you want. The comment
(from Kris Mo?) suggests that sun.misc.Cleaner still exists in JDK 9 - that
isn't so, it was removed in JDK 9 as part of clearing out sun.misc. So I
suspect Kris may be looking at a JDK 8 or older build instead. The Unsafe
invokeCleaner API works the same in the OpenJDK builds from jdk.net as it does
with Oracle JDK builds, there are no differences.
> sun.misc.Unsafe in JDK11
>
>
> Key: SPARK-24421
> URL: https://issues.apache.org/jira/browse/SPARK-24421
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 2.3.0
>Reporter: DB Tsai
>Priority: Major
>
> Many internal APIs such as unsafe are encapsulated in JDK9+, see
> http://openjdk.java.net/jeps/260 for detail.
> To use Unsafe, we need to add *jdk.unsupported* to our code’s module
> declaration:
> {code:java}
> module java9unsafe {
> requires jdk.unsupported;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
