[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2019-12-05 Thread t oo (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989251#comment-16989251
 ] 

t oo edited comment on SPARK-5159 at 12/5/19 11:31 PM:
---

[~yumwang] does removal of hive fork solve this one?

 


was (Author: toopt4):
[~yumwang] does removal of hive fork soove this one?

 

> Thrift server does not respect hive.server2.enable.doAs=true
> 
>
> Key: SPARK-5159
> URL: https://issues.apache.org/jira/browse/SPARK-5159
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.2.0
>Reporter: Andrew Ray
>Priority: Major
> Attachments: spark_thrift_server_log.txt
>
>
> I'm currently testing the spark sql thrift server on a kerberos secured 
> cluster in YARN mode. Currently any user can access any table regardless of 
> HDFS permissions as all data is read as the hive user. In HiveServer2 the 
> property hive.server2.enable.doAs=true causes all access to be done as the 
> submitting user. We should do the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2017-06-04 Thread Dapeng Sun (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036478#comment-16036478
 ] 

Dapeng Sun edited comment on SPARK-5159 at 6/5/17 2:44 AM:
---

I tested it with version 2.1.1, the issue is still existed


was (Author: dapengsun):
I tested it with version 2.1.1, the issue is still existed,

> Thrift server does not respect hive.server2.enable.doAs=true
> 
>
> Key: SPARK-5159
> URL: https://issues.apache.org/jira/browse/SPARK-5159
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.2.0
>Reporter: Andrew Ray
> Attachments: spark_thrift_server_log.txt
>
>
> I'm currently testing the spark sql thrift server on a kerberos secured 
> cluster in YARN mode. Currently any user can access any table regardless of 
> HDFS permissions as all data is read as the hive user. In HiveServer2 the 
> property hive.server2.enable.doAs=true causes all access to be done as the 
> submitting user. We should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2017-02-28 Thread Shridhar Ramachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889677#comment-15889677
 ] 

Shridhar Ramachandran edited comment on SPARK-5159 at 3/1/17 7:29 AM:
--

I am facing this issue as well, on both 1.6 and 2.0. Some solutions have 
indicated setting hive.metastore.execute.setugi to true on the metastore as 
well as the thrift server, but this did not help.


was (Author: shridharama):
I have faced this issue as well, on both 1.6 and 2.0. Some solutions have 
indicated setting hive.metastore.execute.setugi to true on the metastore as 
well as the thrift server, but this did not help.

> Thrift server does not respect hive.server2.enable.doAs=true
> 
>
> Key: SPARK-5159
> URL: https://issues.apache.org/jira/browse/SPARK-5159
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.2.0
>Reporter: Andrew Ray
> Attachments: spark_thrift_server_log.txt
>
>
> I'm currently testing the spark sql thrift server on a kerberos secured 
> cluster in YARN mode. Currently any user can access any table regardless of 
> HDFS permissions as all data is read as the hive user. In HiveServer2 the 
> property hive.server2.enable.doAs=true causes all access to be done as the 
> submitting user. We should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2016-01-18 Thread Ma Xiaoyu (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104911#comment-15104911
 ] 

Ma Xiaoyu edited comment on SPARK-5159 at 1/18/16 8:16 AM:
---

Sorry and I realised that I messed up my PR with SPARK-6910.
My code is shadowed inside and not getting merged.
If needed, I might resubmit it with only the change of doAs part. That one is 
just trying to make doAs work.


was (Author: ilovesoup):
Sorry and I realised that I messed up my PR with SPARK-6910.
My code is shadowed inside.
If needed, I might resubmit it with only the change of doAs part. That one is 
just trying to make doAs work.

> Thrift server does not respect hive.server2.enable.doAs=true
> 
>
> Key: SPARK-5159
> URL: https://issues.apache.org/jira/browse/SPARK-5159
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.2.0
>Reporter: Andrew Ray
> Attachments: spark_thrift_server_log.txt
>
>
> I'm currently testing the spark sql thrift server on a kerberos secured 
> cluster in YARN mode. Currently any user can access any table regardless of 
> HDFS permissions as all data is read as the hive user. In HiveServer2 the 
> property hive.server2.enable.doAs=true causes all access to be done as the 
> submitting user. We should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2016-01-18 Thread Ma Xiaoyu (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104911#comment-15104911
 ] 

Ma Xiaoyu edited comment on SPARK-5159 at 1/18/16 8:17 AM:
---

Sorry and I realised that I messed up my PR with SPARK-6910.
My change is shadowed inside and not getting merged.
If needed, I might resubmit it with only the change of doAs part. That one is 
just trying to make doAs work.


was (Author: ilovesoup):
Sorry and I realised that I messed up my PR with SPARK-6910.
My code is shadowed inside and not getting merged.
If needed, I might resubmit it with only the change of doAs part. That one is 
just trying to make doAs work.

> Thrift server does not respect hive.server2.enable.doAs=true
> 
>
> Key: SPARK-5159
> URL: https://issues.apache.org/jira/browse/SPARK-5159
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.2.0
>Reporter: Andrew Ray
> Attachments: spark_thrift_server_log.txt
>
>
> I'm currently testing the spark sql thrift server on a kerberos secured 
> cluster in YARN mode. Currently any user can access any table regardless of 
> HDFS permissions as all data is read as the hive user. In HiveServer2 the 
> property hive.server2.enable.doAs=true causes all access to be done as the 
> submitting user. We should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2016-01-15 Thread Zhan Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102183#comment-15102183
 ] 

Zhan Zhang edited comment on SPARK-5159 at 1/15/16 5:50 PM:


What happen if an user have a valid visit to a table, which will be saved in 
catalog. Another user then also can visit the table as it is cached in local 
hivecatalog, even if the latter does not have the access to the table meta 
data, right? To make the impersonate to work, all the information has to be 
tagged by user, right?


was (Author: zzhan):
What happen if an user have a valid visit to a table, which will be saved in 
catalog. Another user then also can visit the table as it is cached in local 
hivecatalog, even if the latter does not have the access to the table, right? 
To make the impersonate to really work, all the information has to be tagged by 
user, right?

> Thrift server does not respect hive.server2.enable.doAs=true
> 
>
> Key: SPARK-5159
> URL: https://issues.apache.org/jira/browse/SPARK-5159
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.2.0
>Reporter: Andrew Ray
> Attachments: spark_thrift_server_log.txt
>
>
> I'm currently testing the spark sql thrift server on a kerberos secured 
> cluster in YARN mode. Currently any user can access any table regardless of 
> HDFS permissions as all data is read as the hive user. In HiveServer2 the 
> property hive.server2.enable.doAs=true causes all access to be done as the 
> submitting user. We should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2015-02-11 Thread Tao Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315951#comment-14315951
 ] 

Tao Wang edited comment on SPARK-5159 at 2/12/15 4:15 AM:
--

I have tested this on branch 1.2, below are results:
1.When set hive.server2.enable.doAs=true, I use user `hdfs` to connect 
ThriftServer, then do some operation, the audit log in NameNode shows like this:
bq.2015-02-11 18:07:50,568 | INFO  | IPC Server handler 62 on 25000 | 
allowed=true ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)  
 ip=/9.91.11.204 cmd=getfileinfo 
src=/user/sparkhive/warehouse/yarn.db/child   dst=nullperm=null | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)
2015-02-11 18:07:50,577 | INFO  | IPC Server handler 16 on 25000 | allowed=true 
ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)   
ip=/9.91.11.204 cmd=mkdirs  src=/user/sparkhive/warehouse/yarn.db/child   
dst=nullperm=hdfs:hadoop:rwxr-xr-x | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)

and ThriftServer's log shows like this:
bq.2015-02-11 18:07:50,471 | INFO  | [pool-9-thread-2] | ugi=hdfs  
ip=unknown-ip-addr  cmd=create_table: Table(tableName:child, dbName:yarn, 
owner:hdfs, createTime:1423649270, lastAccessTime:0, retention:0, 
sd:StorageDescriptor(cols:[FieldSchema(name:name, type:string, comment:null), 
FieldSchema(name:age, type:int, comment:null)], location:null, 
inputFormat:org.apache.hadoop.mapred.TextInputFormat, 
outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, 
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, 
serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, 
parameters:{serialization.format=,, field.delim=,}), bucketCols:[], 
sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], 
skewedColValues:[], skewedColValueLocationMaps:{}), 
storedAsSubDirectories:false), partitionKeys:[], parameters:{}, 
viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE) | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:305)

2. When set hive.server2.enable.doAs=false, NameNode's log show like this:
bq.2015-02-11 18:00:05,599 | INFO  | IPC Server handler 32 on 25000 | 
allowed=true ugi=spark/had...@hadoop.com (auth:KERBEROS) ip=/9.91.11.204 
cmd=getfileinfo src=/user/sparkhive/warehouse/yarn.db   dst=null  perm=null 
| 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)
2015-02-11 18:00:05,607 | INFO  | IPC Server handler 24 on 25000 | allowed=true 
ugi=spark/had...@hadoop.com (auth:KERBEROS) ip=/9.91.11.204 cmd=mkdirs  
src=/user/sparkhive/warehouse/yarn.db   dst=null  
perm=spark:hadoop:rwxr-xr-x | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)

ThriftServer's log shows like this:
bq.2015-02-11 18:00:05,437 | INFO  | [pool-9-thread-2] | 
ugi=spark/had...@hadoop.com   ip=unknown-ip-addr  cmd=create_database: 
Database(name:yarn, description:null, locationUri:null, parameters:null, 
ownerName:spark, ownerType:USER) | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:305)
2015-02-11 18:00:05,437 | INFO  | [pool-9-thread-2] | 2: get_database: yarn | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:623)
2015-02-11 18:00:05,438 | INFO  | [pool-9-thread-2] | 
ugi=spark/had...@hadoop.com   ip=unknown-ip-addr  cmd=get_database: 
yarn   | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:305)

I am not an expert on Hive or `doAs` feature. But it met my expect from my 
point.

P.S. spark/had...@hadoop.com is the principle for HiveServer2 to access HDFS.


was (Author: wangtaothetonic):
I have tested this on branch 1.2, below are results:
1.When set hive.server2.enable.doAs=false, I use user `hdfs` to connect 
ThriftServer, then do some operation, the audit log in NameNode shows like this:
bq.2015-02-11 18:07:50,568 | INFO  | IPC Server handler 62 on 25000 | 
allowed=true ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)  
 ip=/9.91.11.204 cmd=getfileinfo 
src=/user/sparkhive/warehouse/yarn.db/child   dst=nullperm=null | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)
2015-02-11 18:07:50,577 | INFO  | IPC Server handler 16 on 25000 | allowed=true 
ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)   
ip=/9.91.11.204 cmd=mkdirs  src=/user/sparkhive/warehouse/yarn.db/child   
dst=nullperm=hdfs:hadoop:rwxr-xr-x | 
org.apac

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2015-02-11 Thread Tao Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315951#comment-14315951
 ] 

Tao Wang edited comment on SPARK-5159 at 2/11/15 10:18 AM:
---

I have tested this on branch 1.2, below are results:
1.When set hive.server2.enable.doAs=false, I use user `hdfs` to connect 
ThriftServer, then do some operation, the audit log in NameNode shows like this:
bq.2015-02-11 18:07:50,568 | INFO  | IPC Server handler 62 on 25000 | 
allowed=true ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)  
 ip=/9.91.11.204 cmd=getfileinfo 
src=/user/sparkhive/warehouse/yarn.db/child   dst=nullperm=null | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)
2015-02-11 18:07:50,577 | INFO  | IPC Server handler 16 on 25000 | allowed=true 
ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)   
ip=/9.91.11.204 cmd=mkdirs  src=/user/sparkhive/warehouse/yarn.db/child   
dst=nullperm=hdfs:hadoop:rwxr-xr-x | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)

and ThriftServer's log shows like this:
bq.2015-02-11 18:07:50,471 | INFO  | [pool-9-thread-2] | ugi=hdfs  
ip=unknown-ip-addr  cmd=create_table: Table(tableName:child, dbName:yarn, 
owner:hdfs, createTime:1423649270, lastAccessTime:0, retention:0, 
sd:StorageDescriptor(cols:[FieldSchema(name:name, type:string, comment:null), 
FieldSchema(name:age, type:int, comment:null)], location:null, 
inputFormat:org.apache.hadoop.mapred.TextInputFormat, 
outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, 
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, 
serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, 
parameters:{serialization.format=,, field.delim=,}), bucketCols:[], 
sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], 
skewedColValues:[], skewedColValueLocationMaps:{}), 
storedAsSubDirectories:false), partitionKeys:[], parameters:{}, 
viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE) | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:305)

2. When set hive.server2.enable.doAs=true, NameNode's log show like this:
bq.2015-02-11 18:00:05,599 | INFO  | IPC Server handler 32 on 25000 | 
allowed=true ugi=spark/had...@hadoop.com (auth:KERBEROS) ip=/9.91.11.204 
cmd=getfileinfo src=/user/sparkhive/warehouse/yarn.db   dst=null  perm=null 
| 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)
2015-02-11 18:00:05,607 | INFO  | IPC Server handler 24 on 25000 | allowed=true 
ugi=spark/had...@hadoop.com (auth:KERBEROS) ip=/9.91.11.204 cmd=mkdirs  
src=/user/sparkhive/warehouse/yarn.db   dst=null  
perm=spark:hadoop:rwxr-xr-x | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)

ThriftServer's log shows like this:
bq.2015-02-11 18:00:05,437 | INFO  | [pool-9-thread-2] | 
ugi=spark/had...@hadoop.com   ip=unknown-ip-addr  cmd=create_database: 
Database(name:yarn, description:null, locationUri:null, parameters:null, 
ownerName:spark, ownerType:USER) | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:305)
2015-02-11 18:00:05,437 | INFO  | [pool-9-thread-2] | 2: get_database: yarn | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logInfo(HiveMetaStore.java:623)
2015-02-11 18:00:05,438 | INFO  | [pool-9-thread-2] | 
ugi=spark/had...@hadoop.com   ip=unknown-ip-addr  cmd=get_database: 
yarn   | 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.logAuditEvent(HiveMetaStore.java:305)

I am not an expert on Hive or `doAs` feature. But it met my expect from my 
point.

P.S. spark/had...@hadoop.com is the principle for HiveServer2 to access HDFS.


was (Author: wangtaothetonic):
I have tested this on branch 1.2, below are results:
1.When set hive.server2.enable.doAs=false, I use user `hdfs` to connect 
ThriftServer, then do some operation, the audit log in NameNode shows like this:
bq.
2015-02-11 18:07:50,568 | INFO  | IPC Server handler 62 on 25000 | allowed=true 
ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)   
ip=/9.91.11.204 cmd=getfileinfo src=/user/sparkhive/warehouse/yarn.db/child   
dst=nullperm=null | 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:7950)
2015-02-11 18:07:50,577 | INFO  | IPC Server handler 16 on 25000 | allowed=true 
ugi=hdfs (auth:PROXY) via spark/had...@hadoop.com (auth:KERBEROS)   
ip=/9.91.11.204 cmd=mkdirs  src=/user/sparkhive/warehouse/yarn.db/child   
dst=nullperm=hdfs:hadoop:rwxr-xr-x | 
org.a