[
https://issues.apache.org/jira/browse/RANGER-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553871#comment-17553871
]
Zhou Yifan commented on RANGER-3787:
------------------------------------
PR: https://github.com/apache/ranger/pull/151
> Non-daemon threads started by ElasticSearchAuditDestination cause Spark
> application hanging
> -------------------------------------------------------------------------------------------
>
> Key: RANGER-3787
> URL: https://issues.apache.org/jira/browse/RANGER-3787
> Project: Ranger
> Issue Type: Improvement
> Components: audit
> Affects Versions: 3.0.0
> Reporter: Zhou Yifan
> Priority: Major
>
> When using [kyuubi-spark-authz
> plugin|https://github.com/apache/incubator-kyuubi/tree/master/extensions/spark/kyuubi-spark-authz]
> (which extends RangerBasePlugin) to save audit log to ElasticSearch in Spark
> application, I found that if Spark application was submitted in local or
> client mode, it hanged forever even after main thread exited. But if I turn
> off `xasecure.audit.destination.elasticsearch`, Spark application exits
> normally.
>
> Here is my `ranger-spark-audit.xml`:
> {code:java}
> <configuration>
> <property>
> <name>xasecure.audit.destination.elasticsearch</name>
> <value>enabled</value>
> </property> <property>
> <name>xasecure.audit.destination.elasticsearch.urls</name>
> <value>es-master-1,es-master-2,es-master-3</value>
> </property> <property>
> <name>xasecure.audit.destination.elasticsearch.port</name>
> <value>9200</value>
> </property>
> </configuration> {code}
> In `jstack` output, I found 2 kinds of non-daemon threads started by
> ElasticSearch RestHighLevelClient:
> {code:java}
> "I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303
> runnable [0x00007000054bc000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
> - locked <0x00000007808a7b48> (a
> java.util.Collections$UnmodifiableSet)
> - locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
> at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5
> os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
> - locked <0x0000000780832db0> (a
> java.util.Collections$UnmodifiableSet)
> - locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> at java.lang.Thread.run(Thread.java:750) {code}
>
> h3. TIPS
> Build kyuubi-spark-authz plugin:
> {code:java}
> git clone [email protected]:zhouyifan279/incubator-kyuubi.git
> cd incubator-kyuubi
> ./build/mvn clean package -pl :kyuubi-spark-authz_2.12 -Pspark-3.1-ranger-3.0
> -DskipTests {code}
> Setup plugin in Spark:
> # Dowload and untar spark distribution:
> [https://www.apache.org/dyn/closer.lua/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.2.tgz]
> # Put
> `incubator-kyuubi/extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_2.12-1.6.0-SNAPSHOT-1-with-ranger-3.0.jar`
> under `spark-3.1.3-bin-hadoop3.2/jars`
> # Put `ranger-spark-security.xml` and `ranger-spark-audit.xml` under
> `spark-3.1.3-bin-hadoop3.2/conf`
> Run Spark application:
> {code:java}
> ./spark-3.1.3-bin-hadoop3.2/bin/spark-submit --master local --class
> org.apache.spark.examples.sql.hive.SparkHiveExample --conf
> spark.sql.extensions=spark.sql.extensions
> org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension
> examples/jars/spark-examples_2.12-3.1.3.jar{code}
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
