Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
> On Sept. 21, 2018, 2:11 p.m., kalyan kumar kalvagadda wrote: > > Arjun, > > > > Can you use an example to explain the optimization this chnage is going to > > make? > > Arjun Mishra wrote: > Kalyan, will give you an example. ReadEntity has an attribute called > accessedColumns. Say a table has 10 partitions and 7 columns. If we run a > select on that table Hive will build the ReadEntity inputs for us as > 1. 1 table input - Database.table. Will have 7 accessedColumns > 2. 10 partition inputs - Each partition input will have 7 accessedColumns > > Because of this we will build an inputHeirarchyList of size 77 (1*7 + > 10*7) like Server->Database->Table->Column. This inputHeirarchyList will have > duplicates. When sentry is checking for authorization it will be checking > over duplicates which is unnecessary Making it a Set of List will eliminate chances of any duplicates making the authorization check much faster - Arjun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/#review208857 --- On Sept. 26, 2018, 5:11 a.m., Arjun Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68787/ > --- > > (Updated Sept. 26, 2018, 5:11 a.m.) > > > Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. > > > Bugs: SENTRY-2406 > https://issues.apache.org/jira/browse/SENTRY-2406 > > > Repository: sentry > > > Description > --- > > When building input or output hierarchy list, we iterate over all ReadEntity > inputs received from Hive. The inputs particularly have accessed columns that > can repeat for other ReadEntity objects. This happens definitively when a > table has partitions. We should in general protect Sentry from not having to > authorize over a list of DBModelAuthorizable objects when it has already been > done. > > > Diffs > - > > > sentry-binding/sentry-binding-hive-common/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBinding.java > 6a1556fa7 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/DefaultSentryValidator.java > f0764767f > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java > da1956b8e > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBinding.java > 5beda9f80 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBindingBase.java > 0909656be > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/util/SentryAuthorizerUtil.java > dd6936c19 > > sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindingHookBase.java > PRE-CREATION > > sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindings.java > 3bbf6fb19 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/AccessURI.java > dd8104d72 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Column.java > 305fd1f5a > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Database.java > 0d94805c7 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Server.java > 33e735ef2 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Table.java > b161d0291 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/View.java > f6d84997e > > > Diff: https://reviews.apache.org/r/68787/diff/3/ > > > Testing > --- > > > Thanks, > > Arjun Mishra > >
Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
> On Sept. 21, 2018, 2:11 p.m., kalyan kumar kalvagadda wrote: > > Arjun, > > > > Can you use an example to explain the optimization this chnage is going to > > make? Kalyan, will give you an example. ReadEntity has an attribute called accessedColumns. Say a table has 10 partitions and 7 columns. If we run a select on that table Hive will build the ReadEntity inputs for us as 1. 1 table input - Database.table. Will have 7 accessedColumns 2. 10 partition inputs - Each partition input will have 7 accessedColumns Because of this we will build an inputHeirarchyList of size 77 (1*7 + 10*7) like Server->Database->Table->Column. This inputHeirarchyList will have duplicates. When sentry is checking for authorization it will be checking over duplicates which is unnecessary - Arjun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/#review208857 --- On Sept. 26, 2018, 5:11 a.m., Arjun Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68787/ > --- > > (Updated Sept. 26, 2018, 5:11 a.m.) > > > Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. > > > Bugs: SENTRY-2406 > https://issues.apache.org/jira/browse/SENTRY-2406 > > > Repository: sentry > > > Description > --- > > When building input or output hierarchy list, we iterate over all ReadEntity > inputs received from Hive. The inputs particularly have accessed columns that > can repeat for other ReadEntity objects. This happens definitively when a > table has partitions. We should in general protect Sentry from not having to > authorize over a list of DBModelAuthorizable objects when it has already been > done. > > > Diffs > - > > > sentry-binding/sentry-binding-hive-common/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBinding.java > 6a1556fa7 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/DefaultSentryValidator.java > f0764767f > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java > da1956b8e > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBinding.java > 5beda9f80 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBindingBase.java > 0909656be > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/util/SentryAuthorizerUtil.java > dd6936c19 > > sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindingHookBase.java > PRE-CREATION > > sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindings.java > 3bbf6fb19 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/AccessURI.java > dd8104d72 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Column.java > 305fd1f5a > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Database.java > 0d94805c7 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Server.java > 33e735ef2 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Table.java > b161d0291 > > sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/View.java > f6d84997e > > > Diff: https://reviews.apache.org/r/68787/diff/3/ > > > Testing > --- > > > Thanks, > > Arjun Mishra > >
Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/ --- (Updated Sept. 26, 2018, 5:11 a.m.) Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. Changes --- Added new unit test Bugs: SENTRY-2406 https://issues.apache.org/jira/browse/SENTRY-2406 Repository: sentry Description --- When building input or output hierarchy list, we iterate over all ReadEntity inputs received from Hive. The inputs particularly have accessed columns that can repeat for other ReadEntity objects. This happens definitively when a table has partitions. We should in general protect Sentry from not having to authorize over a list of DBModelAuthorizable objects when it has already been done. Diffs (updated) - sentry-binding/sentry-binding-hive-common/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBinding.java 6a1556fa7 sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/DefaultSentryValidator.java f0764767f sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java da1956b8e sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBinding.java 5beda9f80 sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBindingBase.java 0909656be sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/util/SentryAuthorizerUtil.java dd6936c19 sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindingHookBase.java PRE-CREATION sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindings.java 3bbf6fb19 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/AccessURI.java dd8104d72 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Column.java 305fd1f5a sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Database.java 0d94805c7 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Server.java 33e735ef2 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Table.java b161d0291 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/View.java f6d84997e Diff: https://reviews.apache.org/r/68787/diff/3/ Changes: https://reviews.apache.org/r/68787/diff/2-3/ Testing --- Thanks, Arjun Mishra
Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/ --- (Updated Sept. 21, 2018, 5:05 p.m.) Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. Changes --- Implementing a set of list to remove duplicates Bugs: SENTRY-2406 https://issues.apache.org/jira/browse/SENTRY-2406 Repository: sentry Description --- When building input or output hierarchy list, we iterate over all ReadEntity inputs received from Hive. The inputs particularly have accessed columns that can repeat for other ReadEntity objects. This happens definitively when a table has partitions. We should in general protect Sentry from not having to authorize over a list of DBModelAuthorizable objects when it has already been done. Diffs (updated) - sentry-binding/sentry-binding-hive-common/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBinding.java 6a1556fa7 sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/DefaultSentryValidator.java f0764767f sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java da1956b8e sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBinding.java 5beda9f80 sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBindingBase.java 0909656be sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/util/SentryAuthorizerUtil.java dd6936c19 sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestHiveAuthzBindings.java 3bbf6fb19 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/AccessURI.java dd8104d72 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Column.java 305fd1f5a sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Database.java 0d94805c7 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Server.java 33e735ef2 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/Table.java b161d0291 sentry-core/sentry-core-model-db/src/main/java/org/apache/sentry/core/model/db/View.java f6d84997e Diff: https://reviews.apache.org/r/68787/diff/2/ Changes: https://reviews.apache.org/r/68787/diff/1-2/ Testing --- Thanks, Arjun Mishra
Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/#review208857 --- Arjun, Can you use an example to explain the optimization this chnage is going to make? - kalyan kumar kalvagadda On Sept. 20, 2018, 7:32 p.m., Arjun Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68787/ > --- > > (Updated Sept. 20, 2018, 7:32 p.m.) > > > Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. > > > Bugs: SENTRY-2406 > https://issues.apache.org/jira/browse/SENTRY-2406 > > > Repository: sentry > > > Description > --- > > When building input or output hierarchy list, we iterate over all ReadEntity > inputs received from Hive. The inputs particularly have accessed columns that > can repeat for other ReadEntity objects. This happens definitively when a > table has partitions. We should in general protect Sentry from not having to > authorize over a list of DBModelAuthorizable objects when it has already been > done. > > > Diffs > - > > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java > da1956b8e > > > Diff: https://reviews.apache.org/r/68787/diff/1/ > > > Testing > --- > > > Thanks, > > Arjun Mishra > >
Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
> On Sept. 20, 2018, 9:44 p.m., Na Li wrote: > > can you add unit test? I thought the existing end to end tests should cover it. Let me try - Arjun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/#review208815 --- On Sept. 20, 2018, 7:32 p.m., Arjun Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68787/ > --- > > (Updated Sept. 20, 2018, 7:32 p.m.) > > > Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. > > > Bugs: SENTRY-2406 > https://issues.apache.org/jira/browse/SENTRY-2406 > > > Repository: sentry > > > Description > --- > > When building input or output hierarchy list, we iterate over all ReadEntity > inputs received from Hive. The inputs particularly have accessed columns that > can repeat for other ReadEntity objects. This happens definitively when a > table has partitions. We should in general protect Sentry from not having to > authorize over a list of DBModelAuthorizable objects when it has already been > done. > > > Diffs > - > > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java > da1956b8e > > > Diff: https://reviews.apache.org/r/68787/diff/1/ > > > Testing > --- > > > Thanks, > > Arjun Mishra > >
Re: Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/#review208815 --- can you add unit test? - Na Li On Sept. 20, 2018, 7:32 p.m., Arjun Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68787/ > --- > > (Updated Sept. 20, 2018, 7:32 p.m.) > > > Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. > > > Bugs: SENTRY-2406 > https://issues.apache.org/jira/browse/SENTRY-2406 > > > Repository: sentry > > > Description > --- > > When building input or output hierarchy list, we iterate over all ReadEntity > inputs received from Hive. The inputs particularly have accessed columns that > can repeat for other ReadEntity objects. This happens definitively when a > table has partitions. We should in general protect Sentry from not having to > authorize over a list of DBModelAuthorizable objects when it has already been > done. > > > Diffs > - > > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java > da1956b8e > > > Diff: https://reviews.apache.org/r/68787/diff/1/ > > > Testing > --- > > > Thanks, > > Arjun Mishra > >
Review Request 68787: SENTRY-2406: Make sure inputHierarchy and outputHierarchy have unique values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68787/ --- Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. Bugs: SENTRY-2406 https://issues.apache.org/jira/browse/SENTRY-2406 Repository: sentry Description --- When building input or output hierarchy list, we iterate over all ReadEntity inputs received from Hive. The inputs particularly have accessed columns that can repeat for other ReadEntity objects. This happens definitively when a table has partitions. We should in general protect Sentry from not having to authorize over a list of DBModelAuthorizable objects when it has already been done. Diffs - sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzBindingHookBase.java da1956b8e Diff: https://reviews.apache.org/r/68787/diff/1/ Testing --- Thanks, Arjun Mishra
