Hi Raymond, I cross checked hive/conf/hive-site.xml and spark2/conf/hive-site.xml Same value is being shown by Ambari Hive config. Seems correct value here:
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/apps/hive/warehouse</value>
</property>
Problem :
Spark trying to create a local directory under the home directory of hive user
(/home/hive/).
Why is it referring the local file system and from where?
Thanks,
Nirmal
From: Raymond Honderdors <[email protected]>
Sent: 19 June 2019 11:18
To: Nirmal Kumar <[email protected]>
Cc: user <[email protected]>
Subject: Re: Unable to run simple spark-sql
Hi Nirmal,
i came across the following article
"https://stackoverflow.com/questions/47497003/why-is-hive-creating-tables-in-the-local-file-system<https://secure-web.cisco.com/1eJXDpPVEl4WoA0ZWL4WJdfrYSsbn4TuKCqHt_IFHMsP29j7xLbCNNBf3Mvmm39OoR8qKeyuLZrkovYLX3CFWIyaUVQ2G3sCCFB9XdWPy_cd2sZrbiLq-hrsZ6rfmMFYZgd27mWYvc49jRUsx6YpUM1JNWdfOidNCVet4LOLJO3VV9kODNw0hhJAirwm0dpxceiGNfGSV_lJIDJvrPt-NG_SiqzFt9HGrOFCJCnCYJHbTlMGKh3LDbkFAvqhDvG8kYkmAU6eMvMUAkjSVQZGjP2uZg0fL1U-AwYPbfU1FsqKyd171Ctt3cFHwGgks1IxkBU-PhKMe4lwFoOI3KuMARwQOGuH2obX4ZJsgeZlZFQw/https%3A%2F%2Fstackoverflow.com%2Fquestions%2F47497003%2Fwhy-is-hive-creating-tables-in-the-local-file-system>"
(and an updated ref link :
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration<https://secure-web.cisco.com/1lHF1a_dGhv0gGAUGVVJizv-j46GpuInCGeNUEhVAkSIeRS8079OhMBRiqwAoRNS9SXkMo_hZuQnvKuiKbSfXjmbSZwpbPTMrDdKaDOB0shFSn5B_9Xn99nORdhBXNdRB0otIq_iqx3_jNdvgWkxzmlQnLnI6-wE26x8ToJYq06GIN-NEi5K9ZvIvCGRt7xNQJaVsXmTpNNKJp0v5bJ8WfTVWt2sOpR1N8W1on7ZrJCKHl9mH8QTJNdRYWEYfF4HkMn5V8U_wGEOsTcx8RDOc7kZHisS_ZUrEwDPKA0PAk35HLJtQ-26-XF1teKiEh8oKB4U_3aMoMcC41nkdckQ7ig/https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHive%2FAdminManual%2BMetastore%2BAdministration>)
you should check "hive.metastore.warehouse.dir" in hive config files
On Tue, Jun 18, 2019 at 8:09 PM Nirmal Kumar
<[email protected]<mailto:[email protected]>> wrote:
Just an update on the thread that it's kerberized.
I'm trying to execute the query with a different user xyz not hive.
Because seems like some permission issue the user xyz trying creating directory
in /home/hive directory
Do i need some impersonation setting?
Thanks,
Nirmal
Get Outlook for
Android<https://aka.ms/ghei36<https://secure-web.cisco.com/11z28bxN4NP4Z9g1qxRqPBXzLZShxonyI1ilwAlTV7-TyszSMWOzoSN6NKJr6jGA4169JJxYBOz8iEGs9x3uOAc9izmc36tkqKjjhkgHCJ9-BCf39p4n1xVDehS9j-LVMqvQ3E_0WFBUJS6iHhuj9iAwq_hgac83c0r_VYMtzPCsVC2dyLoiN2QaLQ4UjFMm8nv8ylOR-3ZpolBGGxEe0aKtWOm5o5iWnpTgF1uDzcAD0pDjikQCBS4FpMeXZL1T-LSQcoieAbZxNKH3_TO9PVC_CX_oedg3tlnuUaVFE3pq3DR5Ofx5YcuuGN43d3WGKK_2c8a6ZE74bdDI0IMDusQ/https%3A%2F%2Faka.ms%2Fghei36>>
________________________________
From: Nirmal Kumar
Sent: Tuesday, June 18, 2019 5:56:06 PM
To: Raymond Honderdors; Nirmal Kumar
Cc: user
Subject: RE: Unable to run simple spark-sql
Hi Raymond,
Permission on hdfs is 777
drwxrwxrwx - impadmin hdfs 0 2019-06-13 16:09
/home/hive/spark-warehouse
But it’s pointing to a local file system:
Exception in thread "main" java.lang.IllegalStateException: Cannot create
staging directory
'file:/home/hive/spark-warehouse/testdb.db/employee_orc/.hive-staging_hive_2019-06-18_16-08-21_448_1691186175028734135-1'
Thanks,
-Nirmal
From: Raymond Honderdors
<[email protected]<mailto:[email protected]>>
Sent: 18 June 2019 17:52
To: Nirmal Kumar
<[email protected]<mailto:[email protected]>.invalid>
Cc: user <[email protected]<mailto:[email protected]>>
Subject: Re: Unable to run simple spark-sql
Hi
Can you check the permission of the user running spark
On the hdfs folder where it tries to create the table
On Tue, Jun 18, 2019, 15:05 Nirmal Kumar
<[email protected]<mailto:[email protected]>.invalid<mailto:[email protected]<mailto:[email protected]>>>
wrote:
Hi List,
I tried running the following sample Java code using Spark2 version 2.0.0 on
YARN (HDP-2.5.0.0)
public class SparkSQLTest {
public static void main(String[] args) {
SparkSession sparkSession = SparkSession.builder().master("yarn")
.config("spark.sql.warehouse.dir", "/apps/hive/warehouse")
.config("hive.metastore.uris", "thrift://xxxxxxxxx:9083")
.config("spark.driver.extraJavaOptions", "-Dhdp.version=2.5.0.0-1245")
.config("spark.yarn.am<http://secure-web.cisco.com/1Oej1V5I6Wn0ZMShpetHkWPNxZdYa42iHw6_YDz46cXrzRZOTD-0yqiAfVff322KBoIQfnSgjLpj9GtmBrQwgIqXmFHEzE79eR4q3Zdg1BpR8CYu3QnPelloxvrqvvJpm4nRuUhubMBchyxfV_vobG5zekcHMWiCaQIx8q44CSJ9n6UvG5zk6KBIzQ4w1YDqbPzQ0EUiC6sLguFVlQg0pCtFjzMvWFzgcS8qtLdQt-yWQ8Lq7pvIAySuh6zGGVFIH9Ux2AmMgFIlwNR70p0WnrfeegNn7J52a4ebtyI_E6JVCsB2_QbBpecBtugmK9ILk/http%3A%2F%2Fspark.yarn.am><http://secure-web.cisco.com/1beuiC-aaBQJ0jgI7vONgZiTP5gCokYFEbllyW3ShZVdpQaIuYfuuEuS8iwzhqvwBE8C_E_bBe_7isO-HyPEVX6ZgJajKrQ6oWvTeBQCMjTHVCVImERG2S9qSHrH_mDzf656vrBFxAT1MYZhTZYzXl_3hyZ4BH-XCbKjXrCDyR1OR3tYqqDc7if9NJ1gqHWPwg84tho0__fut2d8y4XxMoMTQNnJzx5367QL6lYV5CFZj055coSLihVVYrh5jBID5jJF40PsrWSvdW7gJ_P6IAN9jTpHFJD7ZrokjlyS7WBAx5Mtnd2KxvNc2O6kKcxk2/http%3A%2F%2Fspark.yarn.am>.extraJavaOptions",
"-Dhdp.version=2.5.0.0-1245")
.config("spark.yarn.jars",
"hdfs:///tmp/lib/spark2/*").enableHiveSupport().getOrCreate();
sparkSession.sql("insert into testdb.employee_orc select * from
testdb.employee where empid<5");
}
}
I get the following error pointing to a local file system
(file:/home/hive/spark-warehouse) wondering from where its being picked:
16:08:21.321 [dispatcher-event-loop-7] INFO
org.apache.spark.storage.BlockManagerInfo - Added broadcast_0_piece0 in memory
on
192.168.218.92:40831<http://secure-web.cisco.com/1eI1c9QpJbbj4N_Pk9ycydcqDmF85PC5WwISSc4b9_qEOVqfTBBxfEPZY1xDNjJwOx_7Q9FU5iB0hIP6oTpNq69sQ_T27J-d1qyGafuFL_4TJpabeoUh_sOK3KNO0gc2Gb23wPb47w4qBtqjEq5ZIIdgCEWRO2rMbxvaRd3m3hZnSt0LLcLOh9L4aI_tIWiXvHaCjOUKZe6GTEapTyDenGbrcDPI8MDnBJpsf1qmyk6bnXDtZ0TKjCu4XVDipC9-iyfvXcrMPpKabMheF5xBuKddtfYz8sPK7F-2dgG8vahJAHOXgbDqXbqPVrCKS7HIxM-6zHGb69A6LhLt7kg7qjw/http%3A%2F%2F192.168.218.92%3A40831><http://secure-web.cisco.com/18zd_gzhF2N4NeZyolJRHaQMm3mYmE7J-u5p8lbMjuy7lxIZN8zgUUzR8pAzFfMxMiTknORj-329_qyn9tpyQcLejfGKtMK8lhr24CVjsWQVC_YXrT8Ie0c3rifE3KxpJ2y2k58cNtAr0je4JPtzOp6x1HuSmOHLU6CXb80FNn2yi0-PBSRKBHYDJVGU9TlTto9wpY5gkO3U-u7BLR69hXgrqotcSHjzbipPVbI1-HcKKcTbYaEFEqUkM7yy9XJiBfxeqYYJyvstG-5JMJ8Vu8R9DU7gRE0VWMYDNKWPF9KAk_ky4jPHMYHf_DEJimDFI9l0OCyJlELPQs0iw1M6d5g/http%3A%2F%2F192.168.218.92%3A40831>
(size: 30.6 KB, free: 4.0 GB)
16:08:21.322 [main] DEBUG org.apache.spark.storage.BlockManagerMaster - Updated
info of block broadcast_0_piece0
16:08:21.323 [main] DEBUG org.apache.spark.storage.BlockManager - Told master
about block broadcast_0_piece0
16:08:21.323 [main] DEBUG org.apache.spark.storage.BlockManager - Put block
broadcast_0_piece0 locally took 4 ms
16:08:21.323 [main] DEBUG org.apache.spark.storage.BlockManager - Putting block
broadcast_0_piece0 without replication took 4 ms
16:08:21.326 [main] INFO org.apache.spark.SparkContext - Created broadcast 0
from sql at SparkSQLTest.java:33
16:08:21.449 [main] DEBUG
org.apache.spark.sql.hive.execution.InsertIntoHiveTable - Created staging dir =
file:/home/hive/spark-warehouse/testdb.db/employee_orc/.hive-staging_hive_2019-06-18_16-08-21_448_1691186175028734135-1
for path = file:/home/hive/spark-warehouse/testdb.db/employee_orc
16:08:21.451 [main] INFO org.apache.hadoop.hive.common.FileUtils - Creating
directory if it doesn't exist:
file:/home/hive/spark-warehouse/testdb.db/employee_orc/.hive-staging_hive_2019-06-18_16-08-21_448_1691186175028734135-1
Exception in thread "main" java.lang.IllegalStateException: Cannot create
staging directory
'file:/home/hive/spark-warehouse/testdb.db/employee_orc/.hive-staging_hive_2019-06-18_16-08-21_448_1691186175028734135-1'
at
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.getStagingDir(InsertIntoHiveTable.scala:83)
at
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.getExternalScratchDir(InsertIntoHiveTable.scala:97)
at
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.getExternalTmpPath(InsertIntoHiveTable.scala:105)
at
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:148)
at
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:142)
at
org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:313)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:186)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
at com.xxxx.xxx.xxx.xxx.xxxx.SparkSQLTest.main(SparkSQLTest.java:33)
16:08:21.454 [pool-8-thread-1] INFO org.apache.spark.SparkContext - Invoking
stop() from shutdown hook
16:08:21.455 [pool-8-thread-1] DEBUG
org.spark_project.jetty.util.component.AbstractLifeCycle - stopping
org.spark_project.jetty.server.Server@620aa4ea<mailto:org.spark_project.jetty.server.Server@620aa4ea<mailto:org.spark_project.jetty.server.Server@620aa4ea>>
16:08:21.455 [pool-8-thread-1] DEBUG org.spark_project.jetty.server.Server -
Graceful shutdown
org.spark_project.jetty.server.Server@620aa4ea<mailto:org.spark_project.jetty.server.Server@620aa4ea<mailto:org.spark_project.jetty.server.Server@620aa4ea>>
by
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary,
privileged or otherwise protected by law. The message is intended solely for
the named addressee. If received in error, please destroy and notify the
sender. Any use of this email is prohibited when received in error. Impetus
does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors,
virus, interception or interference.
---------------------------------------------------------------------
To unsubscribe e-mail:
[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
________________________________
NOTE: This message may contain information that is confidential, proprietary,
privileged or otherwise protected by law. The message is intended solely for
the named addressee. If received in error, please destroy and notify the
sender. Any use of this email is prohibited when received in error. Impetus
does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors,
virus, interception or interference.
--
[http://secure-web.cisco.com/1J8E_pPd0U8-41M82KchUeTYEL1zVUiIgzVTYJrvDPIdEu0_gxkqLZZF0C3-gOOkGJPABfZXDxMLmdtIs0RSF37S3sMH-7JeRciRJ5CGuxrOaBIGnRCDZgoQtd4XIaP6MNEUqrKzkY5j1nkkS2GzGq4ZSL4r1uZBtCc02Z8wLNH73la-OJ7MfLteiMYr7Em7XVWSIMSF8eoJFGjDDyjtbDGvdk-hVlK9Mdxp0O6zHqJ__39obcMoS5PIULCpwgqu2n6QJks_h-YEQXGA7mLJtcbHnWVeZLIGdd7ZqzjyOCctjiCaWTjAmEUPRmEy2_icatpb_jCdp_q1VMmpQxBmHSw/http%3A%2F%2Fwww.sizmek.com%2Fmedia%2Ffiler_public%2F19%2Fc6%2F19c6b884-b374-4adb-8ef1-6d08226bc1ba%2Femail-signature-logo.png]
Raymond Honderdors
R&D Tech Lead / Open Source evangelist
[email protected]<mailto:[email protected]>
w: +972732535698
Herzliya
[https://secure-web.cisco.com/1MZCah12N9eVeV70hk81tZGUfeO3SPwkKwT-9Rb4Dk6oJRx7r3_Rlq5P5W7x3fpGXvWYA9lLOhFYecMIHhqKv3YKAMS2LINXgDHG3i8UyY6llcg0YjTX3YQn1Zxb2NMAyw9a06C-2Jd4_IIqMq4e_rFAQvKC4BvDA9o0vxbHiIWBv8LIsYF3a6mtEMzb7z9z-2e9CA1fOe7eCkBt2NSeqYdnfMxNY6Kn1LAwrdoWzv-Besq9scTlx6s5uletOkwOdyhbpFKQEHppGctP2VQyVK9u87FoJM38m1D-ixnQwmKy3hfwRxFK3nuWr2seM5w0Nq7xrI2rkkM90UqSfKZYXlPMWxHFPpVcuGDRtTaUPhEs/https%3A%2F%2Finfo.sizmek.com%2Frs%2F303-WWQ-966%2Fimages%2FSAS_Signature_5.gif]<https://secure-web.cisco.com/1fwToZsgXAv2WVmExBuodAeT9UroAUGo6KLmHj2G7dSabl16faT1FnM_NzJmcEo7CsVq72taZ-efARSJN8GWMI5OoxY9Q7dsLWRb9yIiUqzg6cYF5r8Kxb0xK095FX8aDM6UAmoUSuAzO36iS3wFT_1wqQlEh3ydko7HcAMBF_pJ9L6fCE1Df-V0dcjcAULFxFlrMVYu0e9t2seCXxX98RhawqglzWyzUqZXsfiPkAjNbgt3yhgpFD_zzKGZKqd1AXmHIQhtMUFDpQcG0bp8HDQPdyV1fZi8fy8w05hliR9EzRJdux54gu3gv2auAE_ppejdqEZhceSfBbYdgt1-w25xt7DIgjHioAO47SCGmE4A/https%3A%2F%2Finfo.sizmek.com%2Fsizmek-advertising-suite>
________________________________
NOTE: This message may contain information that is confidential, proprietary,
privileged or otherwise protected by law. The message is intended solely for
the named addressee. If received in error, please destroy and notify the
sender. Any use of this email is prohibited when received in error. Impetus
does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors,
virus, interception or interference.
<<attachment: winmail.dat>>
--------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
