Re: Spark thrift service and Hive impersonation.

[Steve Loughran]

On 5 Oct 2015, at 22:51, Jagat Singh 
> wrote:

Hello Steve,

Thanks for confirmation.

Is there any work planned work on this.


Not that I'm aware of, though somebody may be doing it.

SparkSQL is not hive. It uses some of the libraries -the thriftserver stuff is 
subclassed to get thrift support, and so, transitively ODBC support. The 
sql/hive module uses the hive JAR to parse SQL and to get at data. And in Spark 
1.5, it's based off Hive 1.2.1, so can access hive metadata created by the 
current hive releases. But it's a different SQL engine underneath


Thanks,

Jagat Singh



On Wed, Sep 30, 2015 at 9:37 PM, Vinay Shukla 
> wrote:
Steve is right,
 The Spark thing server does not profs page end user identity downstream yet.



On Wednesday, September 30, 2015, Steve Loughran 
> wrote:

On 30 Sep 2015, at 03:24, Mohammed Guller  wrote:

Does each user needs to start own thrift server to use it?

No. One of the benefits of the Spark Thrift Server is that it allows multiple 
users to share a single SparkContext.

Most likely, you have file permissions issue.



I don't think the spark hive thrift server does the multi-user stuff (yet)

Mohammed

From: Jagat Singh [mailto:jagatsi...@gmail.com]
Sent: Tuesday, September 29, 2015 5:30 PM
To: SparkUser
Subject: Spark thrift service and Hive impersonation.

Hi,

I have started the Spark thrift service using spark user.

Does each user needs to start own thrift server to use it?

Using beeline i am able to connect to server and execute show tables;

However when we try to execute some real query it runs as spark user and HDFS 
permissions does not allow them to be read.

The query fails with error

0: jdbc:hive2://localhost:1> select count(*) from mytable;
Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table 
mytable. java.security.AccessControlException: Permission denied: user=spark, 
access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)


And in thrift server we get log.


In the hive-site.xml we have impersonation enabled.

   
  hive.server2.enable.doAs
  true



  hive.server2.enable.impersonation
  true


Is there any other configuration to be done for it to work like normal hive 
thrift server.

Thanks

Re: Spark thrift service and Hive impersonation.

[Jagat Singh]

Hello Steve,

Thanks for confirmation.

Is there any work planned work on this.

Thanks,

Jagat Singh



On Wed, Sep 30, 2015 at 9:37 PM, Vinay Shukla  wrote:

> Steve is right,
>  The Spark thing server does not profs page end user identity downstream
> yet.
>
>
>
> On Wednesday, September 30, 2015, Steve Loughran 
> wrote:
>
>>
>> On 30 Sep 2015, at 03:24, Mohammed Guller  wrote:
>>
>> Does each user needs to start own thrift server to use it?
>>
>>
>>
>> No. One of the benefits of the Spark Thrift Server is that it allows
>> multiple users to share a single SparkContext.
>>
>>
>>
>> Most likely, you have file permissions issue.
>>
>>
>>
>>
>> I don't think the spark hive thrift server does the multi-user stuff (yet)
>>
>> Mohammed
>>
>>
>>
>> *From:* Jagat Singh [mailto:jagatsi...@gmail.com]
>> *Sent:* Tuesday, September 29, 2015 5:30 PM
>> *To:* SparkUser
>> *Subject:* Spark thrift service and Hive impersonation.
>>
>>
>>
>> Hi,
>>
>>
>>
>> I have started the Spark thrift service using spark user.
>>
>>
>>
>> Does each user needs to start own thrift server to use it?
>>
>>
>>
>> Using beeline i am able to connect to server and execute show tables;
>>
>>
>>
>> However when we try to execute some real query it runs as spark user and
>> HDFS permissions does not allow them to be read.
>>
>>
>>
>> The query fails with error
>>
>>
>>
>> 0: jdbc:hive2://localhost:1> select count(*) from mytable;
>>
>> Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
>> table mytable. java.security.AccessControlException: Permission denied:
>> user=spark, access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
>>
>>
>>
>>
>>
>> And in thrift server we get log.
>>
>>
>>
>>
>>
>> In the hive-site.xml we have impersonation enabled.
>>
>>
>>
>>
>>
>>   hive.server2.enable.doAs
>>
>>   true
>>
>> 
>>
>>
>>
>> 
>>
>>   hive.server2.enable.impersonation
>>
>>   true
>>
>> 
>>
>>
>>
>> Is there any other configuration to be done for it to work like normal
>> hive thrift server.
>>
>>
>>
>> Thanks
>>
>>
>>

Re: Spark thrift service and Hive impersonation.

[Steve Loughran]

On 30 Sep 2015, at 03:24, Mohammed Guller 
> wrote:

Does each user needs to start own thrift server to use it?

No. One of the benefits of the Spark Thrift Server is that it allows multiple 
users to share a single SparkContext.

Most likely, you have file permissions issue.



I don't think the spark hive thrift server does the multi-user stuff (yet)

Mohammed

From: Jagat Singh [mailto:jagatsi...@gmail.com]
Sent: Tuesday, September 29, 2015 5:30 PM
To: SparkUser
Subject: Spark thrift service and Hive impersonation.

Hi,

I have started the Spark thrift service using spark user.

Does each user needs to start own thrift server to use it?

Using beeline i am able to connect to server and execute show tables;

However when we try to execute some real query it runs as spark user and HDFS 
permissions does not allow them to be read.

The query fails with error

0: jdbc:hive2://localhost:1> select count(*) from mytable;
Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table 
mytable. java.security.AccessControlException: Permission denied: user=spark, 
access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)


And in thrift server we get log.


In the hive-site.xml we have impersonation enabled.

   
  hive.server2.enable.doAs
  true



  hive.server2.enable.impersonation
  true


Is there any other configuration to be done for it to work like normal hive 
thrift server.

Thanks

Re: Spark thrift service and Hive impersonation.

[Vinay Shukla]

Steve is right,
 The Spark thing server does not profs page end user identity downstream
yet.



On Wednesday, September 30, 2015, Steve Loughran 
wrote:

>
> On 30 Sep 2015, at 03:24, Mohammed Guller  > wrote:
>
> Does each user needs to start own thrift server to use it?
>
>
>
> No. One of the benefits of the Spark Thrift Server is that it allows
> multiple users to share a single SparkContext.
>
>
>
> Most likely, you have file permissions issue.
>
>
>
>
> I don't think the spark hive thrift server does the multi-user stuff (yet)
>
> Mohammed
>
>
>
> *From:* Jagat Singh [mailto:jagatsi...@gmail.com
> ]
> *Sent:* Tuesday, September 29, 2015 5:30 PM
> *To:* SparkUser
> *Subject:* Spark thrift service and Hive impersonation.
>
>
>
> Hi,
>
>
>
> I have started the Spark thrift service using spark user.
>
>
>
> Does each user needs to start own thrift server to use it?
>
>
>
> Using beeline i am able to connect to server and execute show tables;
>
>
>
> However when we try to execute some real query it runs as spark user and
> HDFS permissions does not allow them to be read.
>
>
>
> The query fails with error
>
>
>
> 0: jdbc:hive2://localhost:1> select count(*) from mytable;
>
> Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
> table mytable. java.security.AccessControlException: Permission denied:
> user=spark, access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
>
>
>
>
>
> And in thrift server we get log.
>
>
>
>
>
> In the hive-site.xml we have impersonation enabled.
>
>
>
>
>
>   hive.server2.enable.doAs
>
>   true
>
> 
>
>
>
> 
>
>   hive.server2.enable.impersonation
>
>   true
>
> 
>
>
>
> Is there any other configuration to be done for it to work like normal
> hive thrift server.
>
>
>
> Thanks
>
>
>

RE: Spark thrift service and Hive impersonation.

[Mohammed Guller]

When a user issues a connect command from Beeline, it asks for username and 
password. What happens if you give spark as the user name?

Also, it looks like permission for "/data/mytable” is drwxr-x—x

Have you tried changing the permission to allow other users to read?

Mohammed

From: Jagat Singh [mailto:jagatsi...@gmail.com]
Sent: Tuesday, September 29, 2015 6:32 PM
To: Mohammed Guller
Cc: SparkUser
Subject: Re: Spark thrift service and Hive impersonation.

Hi,

Thanks for your reply.

If you see the log message

Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table 
mytable. java.security.AccessControlException: Permission denied: user=spark, 
access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x

Spark is trying to read as spark user , using which we started thrift server.

Since spark user does not have actual read access we get the error.

However the beeline is used by end user not spark user and throws error.

Thanks,

Jagat Singh



On Wed, Sep 30, 2015 at 11:24 AM, Mohammed Guller 
<moham...@glassbeam.com<mailto:moham...@glassbeam.com>> wrote:
Does each user needs to start own thrift server to use it?

No. One of the benefits of the Spark Thrift Server is that it allows multiple 
users to share a single SparkContext.

Most likely, you have file permissions issue.

Mohammed

From: Jagat Singh [mailto:jagatsi...@gmail.com<mailto:jagatsi...@gmail.com>]
Sent: Tuesday, September 29, 2015 5:30 PM
To: SparkUser
Subject: Spark thrift service and Hive impersonation.

Hi,

I have started the Spark thrift service using spark user.

Does each user needs to start own thrift server to use it?

Using beeline i am able to connect to server and execute show tables;

However when we try to execute some real query it runs as spark user and HDFS 
permissions does not allow them to be read.

The query fails with error

0: jdbc:hive2://localhost:1> select count(*) from mytable;
Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table 
mytable. java.security.AccessControlException: Permission denied: user=spark, 
access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)


And in thrift server we get log.


In the hive-site.xml we have impersonation enabled.

   
  hive.server2.enable.doAs
  true



  hive.server2.enable.impersonation
  true


Is there any other configuration to be done for it to work like normal hive 
thrift server.

Thanks

RE: Spark thrift service and Hive impersonation.

[Mohammed Guller]

Does each user needs to start own thrift server to use it?

No. One of the benefits of the Spark Thrift Server is that it allows multiple 
users to share a single SparkContext.

Most likely, you have file permissions issue.

Mohammed

From: Jagat Singh [mailto:jagatsi...@gmail.com]
Sent: Tuesday, September 29, 2015 5:30 PM
To: SparkUser
Subject: Spark thrift service and Hive impersonation.

Hi,

I have started the Spark thrift service using spark user.

Does each user needs to start own thrift server to use it?

Using beeline i am able to connect to server and execute show tables;

However when we try to execute some real query it runs as spark user and HDFS 
permissions does not allow them to be read.

The query fails with error

0: jdbc:hive2://localhost:1> select count(*) from mytable;
Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table 
mytable. java.security.AccessControlException: Permission denied: user=spark, 
access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)


And in thrift server we get log.


In the hive-site.xml we have impersonation enabled.

   
  hive.server2.enable.doAs
  true



  hive.server2.enable.impersonation
  true


Is there any other configuration to be done for it to work like normal hive 
thrift server.

Thanks

Re: Spark thrift service and Hive impersonation.

[Jagat Singh]

Hi,

Thanks for your reply.

If you see the log message

Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
table mytable. java.security.AccessControlException: Permission denied:
user=spark, access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x

Spark is trying to read as spark user , using which we started thrift
server.

Since spark user does not have actual read access we get the error.

However the beeline is used by end user not spark user and throws error.

Thanks,

Jagat Singh



On Wed, Sep 30, 2015 at 11:24 AM, Mohammed Guller 
wrote:

> Does each user needs to start own thrift server to use it?
>
>
>
> No. One of the benefits of the Spark Thrift Server is that it allows
> multiple users to share a single SparkContext.
>
>
>
> Most likely, you have file permissions issue.
>
>
>
> Mohammed
>
>
>
> *From:* Jagat Singh [mailto:jagatsi...@gmail.com]
> *Sent:* Tuesday, September 29, 2015 5:30 PM
> *To:* SparkUser
> *Subject:* Spark thrift service and Hive impersonation.
>
>
>
> Hi,
>
>
>
> I have started the Spark thrift service using spark user.
>
>
>
> Does each user needs to start own thrift server to use it?
>
>
>
> Using beeline i am able to connect to server and execute show tables;
>
>
>
> However when we try to execute some real query it runs as spark user and
> HDFS permissions does not allow them to be read.
>
>
>
> The query fails with error
>
>
>
> 0: jdbc:hive2://localhost:1> select count(*) from mytable;
>
> Error: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch
> table mytable. java.security.AccessControlException: Permission denied:
> user=spark, access=READ, inode="/data/mytable":tdcprdr:tdcprdr:drwxr-x--x
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
>
>
>
>
>
> And in thrift server we get log.
>
>
>
>
>
> In the hive-site.xml we have impersonation enabled.
>
>
>
>
>
>   hive.server2.enable.doAs
>
>   true
>
> 
>
>
>
> 
>
>   hive.server2.enable.impersonation
>
>   true
>
> 
>
>
>
> Is there any other configuration to be done for it to work like normal
> hive thrift server.
>
>
>
> Thanks
>