[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20864 @gatorsmile, I triggered tests for PRs where there's committer's command for it. I don't know why or who started to block the tests in Jenkins for what reason. If the author is willing to update the PR, the Jenkins is a blocker again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20864 @HyukjinKwon We also do not need to trigger the test for this PR. This fix does not look good based on the above comment from @liufengdb --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93061/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20864 **[Test build #93061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93061/testReport)** for PR 20864 at commit [`b177bf4`](https://github.com/apache/spark/commit/b177bf441a04c0a700e33d7e40c1b2408c3c0c3b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20864 **[Test build #93061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93061/testReport)** for PR 20864 at commit [`b177bf4`](https://github.com/apache/spark/commit/b177bf441a04c0a700e33d7e40c1b2408c3c0c3b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20864 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user liufengdb commented on the issue: https://github.com/apache/spark/pull/20864 I thought the directory is also created from this line: https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L143. For this one, we need to think about whether we can remove all the temp directories creation, because the statements are executed by spark sql and it has nothing about the Hive in the thrift server. You are right that HiveClientImpl (the Hive inside spark sql) will also produce such temp directories. However, it seems like the following line alone is sufficient to add the jar to the class loader: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L836. So I doubt we still need the `runSqlHive(s"ADD JAR $path")` to download the jar to a temp directory. Overall, I think we need an overall design to remove the Hive legacy in both the thrift server and Spark SQL. Adding more temp fixes will make such a design harder. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user zuotingbing commented on the issue: https://github.com/apache/spark/pull/20864 @felixcheung Thanks. The test result is 0 failed, what is meaning of > This patch fails due to an unknown error code, -9 ? what should i do to fix the test fails? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88537/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20864 **[Test build #88537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88537/testReport)** for PR 20864 at commit [`b177bf4`](https://github.com/apache/spark/commit/b177bf441a04c0a700e33d7e40c1b2408c3c0c3b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20864 **[Test build #88537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88537/testReport)** for PR 20864 at commit [`b177bf4`](https://github.com/apache/spark/commit/b177bf441a04c0a700e33d7e40c1b2408c3c0c3b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20864 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user zuotingbing commented on the issue: https://github.com/apache/spark/pull/20864 `add jar 'hdfs:///xxx.jar'` or create UDF will download jars to the resource dir: `CREATE FUNCTION xxx AS 'xxx' USING JAR 'hdfs:///xxx.jar' SELECT xxx()` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user liufengdb commented on the issue: https://github.com/apache/spark/pull/20864 @samartinucci @zuotingbing a high-level question: is it necessary to create these temp directories when the hive thrift server starts? It sounds some legacy from Hive and we can skip creating them in the first place. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user zuotingbing commented on the issue:
https://github.com/apache/spark/pull/20864
i take a look at [https://github.com/apache/spark/pull/18666]. i find it
can not clean all the ***_resources directories. because when we start
HiveThriftServer2, we created two resource directories:
`8/03/21 11:23:33 INFO **SessionState: Created local directory:
/data1/zdh/spark/hive/tmp/616f66c9-fa4e-4a0c-a63a-10ff97e5019c_resources**
18/03/21 11:23:33 INFO SessionState: Created HDFS directory:
/spark-tmp/scratchdir/root/616f66c9-fa4e-4a0c-a63a-10ff97e5019c
18/03/21 11:23:33 INFO SessionState: Created local directory:
/data1/zdh/spark/hive/tmp/616f66c9-fa4e-4a0c-a63a-10ff97e5019c
18/03/21 11:23:33 INFO SessionState: Created HDFS directory:
/spark-tmp/scratchdir/root/616f66c9-fa4e-4a0c-a63a-10ff97e5019c/_tmp_space.db
18/03/21 11:23:33 INFO HiveClientImpl: Warehouse location for Hive client
(version 1.2.2) is file:/media/A/gitspace/spark/dist/sbin/spark-warehouse
18/03/21 11:23:33 INFO HiveMetaStore: 0: get_database: default
18/03/21 11:23:33 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_database: default
18/03/21 11:23:33 INFO StateStoreCoordinatorRef: Registered
StateStoreCoordinator endpoint
18/03/21 11:23:33 INFO HiveUtils: Initializing execution hive, version 1.2.1
18/03/21 11:23:34 INFO HiveMetaStore: 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/03/21 11:23:34 INFO ObjectStore: ObjectStore, initialize called
18/03/21 11:23:34 INFO Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/03/21 11:23:34 INFO Persistence: Property datanucleus.cache.level2
unknown - will be ignored
18/03/21 11:23:36 INFO ObjectStore: Setting MetaStore object pin classes
with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/03/21 11:23:36 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
18/03/21 11:23:36 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so
does not have its own datastore table.
18/03/21 11:23:37 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
18/03/21 11:23:37 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so
does not have its own datastore table.
18/03/21 11:23:37 INFO MetaStoreDirectSql: Using direct SQL, underlying DB
is DERBY
18/03/21 11:23:37 INFO ObjectStore: Initialized ObjectStore
18/03/21 11:23:37 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 1.2.0
18/03/21 11:23:38 WARN ObjectStore: Failed to get database default,
returning NoSuchObjectException
18/03/21 11:23:38 INFO HiveMetaStore: Added admin role in metastore
18/03/21 11:23:38 INFO HiveMetaStore: Added public role in metastore
18/03/21 11:23:38 INFO HiveMetaStore: No user is added in admin role, since
config is empty
18/03/21 11:23:38 INFO HiveMetaStore: 0: get_all_databases
18/03/21 11:23:38 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_all_databases
18/03/21 11:23:38 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/03/21 11:23:38 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_functions: db=default pat=*
18/03/21 11:23:38 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
"embedded-only" so does not have its own datastore table.
18/03/21 11:23:38 INFO **SessionState: Created local directory:
/data1/zdh/spark/hive/tmp/16aa5bb9-33e4-43e6-8bdb-8e0318ab175e_resources**
18/03/21 11:23:38 INFO SessionState: Created HDFS directory:
/spark-tmp/scratchdir/root/16aa5bb9-33e4-43e6-8bdb-8e0318ab175e
18/03/21 11:23:38 INFO SessionState: Created local directory:
/data1/zdh/spark/hive/tmp/16aa5bb9-33e4-43e6-8bdb-8e0318ab175e
18/03/21 11:23:38 INFO SessionState: Created HDFS directory:
/spark-tmp/scratchdir/root/16aa5bb9-33e4-43e6-8bdb-8e0318ab175e/_tmp_space.db
18/03/21 11:23:38 INFO HiveClientImpl: Warehouse location for Hive client
(version 1.2.2) is file:/media/A/gitspace/spark/dist/sbin/spark-warehouse`
but when stop just remove only one resource directory which is current:
`public void close() throws IOException {
registry.clear();
if (txnMgr != null) txnMgr.closeTxnManager();
JavaUtils.closeClassLoadersTo(conf.getClassLoader(), parentLoader);
**File resourceDir =
new
File(getConf().getVar(HiveConf.ConfVars.DOWNLOADED_RESOURCES_DIR))**;
LOG.debug("Removing resource dir "
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user samartinucci commented on the issue: https://github.com/apache/spark/pull/20864 Seems to be related to: https://github.com/apache/spark/pull/18666 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user zuotingbing commented on the issue: https://github.com/apache/spark/pull/20864 @liufengdb would you please take a review. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20864: [SPARK-23745][SQL]Remove the directories of the “hive....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20864 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
