[
https://issues.apache.org/jira/browse/SPARK-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044931#comment-17044931
]
NITISH SHARMA edited comment on SPARK-18879 at 2/25/20 9:59 PM:
----------------------------------------------------------------
I have a similar requirement where i am looking to update LAST_ACCESS_TIME in
TBLS of Hive metastore whenever any table is accessed through spark. I set this
below property in hive-site.xml and hive honors it and updates the
LAST_ACCESS_TIME everytime it is accessed.
<property>
<name>hive.exec.pre.hooks</name>
<value>org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec</value>
</property>
However, the same thing i want to achieve using pyspark/spark-shell but its not
honoring this property of hive hooks. Is there an alternate approach of
achieving this - 'Update of LAST_ACCESS_TIME in hive metastore on access using
spark'.
I passed the property like this -
spark-sql -e 'set
spark.hadoop.hive.exec.post.hooks=org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;select
* from db.table;'
as well as i put the same property in /etc/spark/conf/hive-site.xml location.
was (Author: nitishcse412):
I have a similar requirement where i am looking to update LAST_ACCESS_TIME in
TBLS of Hive metastore whenever any table is accessed through spark. I set this
below property in hive-site.xml and hive honors it and updates the
LAST_ACCESS_TIME everytime it is accessed.
<property>
<name>hive.exec.pre.hooks</name>
<value>org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec</value>
</property>
However, the same thing i want to achieve using pyspark/spark-shell but its not
honoring this property of hive hooks. Is there an alternate approach of
achieving this - 'Update of LAST_ACCESS_TIME in hive metastore on access using
spark'.
> Spark SQL support for Hive hooks regressed
> ------------------------------------------
>
> Key: SPARK-18879
> URL: https://issues.apache.org/jira/browse/SPARK-18879
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0, 2.0.2
> Reporter: Atul Payapilly
> Priority: Major
>
> As per the stack trace from this post:
> http://ihorbobak.com/index.php/2015/05/08/113/
> run on Spark 1.3.1
> hive.exec.pre.hooks Class not found:org.apache.hadoop.hive.ql.hooks.ATSHook
> FAILED: Hive Internal Error:
> java.lang.ClassNotFoundException(org.apache.hadoop.hive.ql.hooks.ATSHook)
> java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.hooks.ATSHook
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59)
> at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1172)
> at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1156)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1206)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:318)
> at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:290)
> at
> org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33)
> at
> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54)
> at
> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54)
> at
> org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64)
> at
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099)
> at
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099)
> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:147)
> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
> at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:101)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:164)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> It looks like Spark used to rely on the Hive Driver for execution and
> supported hive hooks. The current code path does not rely on the Hive Driver
> and support for Hive hooks regressed. This is problematic, for example, there
> is no way to tell which partitions were updated as part of a query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
