Hi Romain, Thank you for your response.
Adding Kerberos support might be as simple as https://issues.cloudera.org/browse/LIVY-44 ? I.e. add Livy --principal and --keytab parameters to be passed to spark-submit. As a workaround I just did kinit (using hues' keytab) and then launched Livy Server. It probably will work as long as kerberos ticket doesn't expire. That's it would be great to have support for --principal and --keytab parameters for spark-submit as explined in http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_sg_yarn_long_jobs.html The only problem I have currently is the above error stack in my previous email: The Spark session could not be created in the cluster: > at org.apache.hadoop.security.*UserGroupInformation.doAs*( > UserGroupInformation.java:1671) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:160) >> AFAIK Hive impersonation should be turned off when using Sentry Yep, exactly. That's what I did. It is disabled now. But looks like on other hand, Spark or Spark Notebook want to have that enabled? It tries to do org.apache.hadoop.security.UserGroupInformation.doAs() hence the error. So Sentry isn't compatible with Spark in kerberized clusters? Is any workaround for this problem? -- Ruslan Dautkhanov On Mon, Jan 18, 2016 at 3:52 PM, Romain Rigaux <rom...@cloudera.com> wrote: > Livy does not support any Kerberos yet > https://issues.cloudera.org/browse/LIVY-3 > > Are you focusing instead about HS2 + Kerberos with Sentry? > > AFAIK Hive impersonation should be turned off when using Sentry: > http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/sg_sentry_service_config.html > > On Sun, Jan 17, 2016 at 10:04 PM, Ruslan Dautkhanov <dautkha...@gmail.com> > wrote: > >> Getting following error stack >> >> The Spark session could not be created in the cluster: >>> at org.apache.hadoop.security.*UserGroupInformation.doAs* >>> (UserGroupInformation.java:1671) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:160) >>> at >>> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ) >>> at org.*apache.hadoop.hive.metastore.HiveMetaStoreClient* >>> .open(HiveMetaStoreClient.java:466) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:234) >>> at >>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) >>> ... 35 more >> >> >> My understanding that hive.server2.enable.impersonation and >> hive.server2.enable.doAs should be enabled to make >> UserGroupInformation.doAs() work? >> >> When I try to enable these parameters, Cloudera Manager shows error >> >> Hive Impersonation is enabled for Hive Server2 role 'HiveServer2 >>> (hostname)'. >>> Hive Impersonation should be disabled to enable Hive authorization using >>> Sentry >> >> >> So Spark-Hive conflicts with Sentry!? >> >> Environment: Hue 3.9 Spark Notebooks + Livy Server (built from master). >> CDH 5.5. >> >> This is a kerberized cluster with Sentry. >> >> I was using hue's keytab as hue user is normally (by default in CDH) is >> allowed to impersonate to other users. >> So very convenient for Spark Notebooks. >> >> Any information to help solve this will be highly appreciated. >> >> >> -- >> Ruslan Dautkhanov >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Hue-Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to hue-user+unsubscr...@cloudera.org. >> > >
