----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73239/ -----------------------------------------------------------
(Updated March 25, 2021, midnight) Review request for atlas, Madhan Neethiraj and Sarath Subramanian. Changes ------- Updates include: - Addressed review comments. - Radically changed implementation approach. - Earlier unit tests are now supplanted with new ones. Bugs: ATLAS-4204 https://issues.apache.org/jira/browse/ATLAS-4204 Repository: atlas Description (updated) ------- **Background** Please see JIRA. Terms and abbreviations used: - Database objects: Database, table, views, etc. - DDL: Data Definition Langugage. Database parlance. Operations performed on a database server that result in creation of objects that hold data. - DML: Data Manipulation Language. Database parlance. Operations performed on a database server that result in manipulation (add, update, delete) of data. These are performed using database objects that are already created. Hive hook is implemented by: - _HiveHook_ - _HiveMetastoreImpl_ - _BaseHiveEvent_ has 2 methods: - getHiveMetastoreEntities: Called by HMS events. - getHiveEntities: Called by HS2 events. **Approach** - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL operation should be skipped by a hook. This flag is set as a configuration parameters: atlas.hive.hook.ignore.ddl.operations - Continue using HMS event processing unchanged. - New: _ActiveEntityFilter_: Initializes entity filter based on the skipDDL setting. - New: _LineageOnlyFilter_: Retains entities that participate in lineage. - New: _PassthroughFilter_: Does not do any filtering. Diffs (updated) ----- addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 79e87c79d addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTable.java e164370ba addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java 6961fa7c2 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRenameCol.java 756a6089e addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 7c269ce53 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java bf5f5620e addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java 659417d82 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java 63fc89424 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropDatabase.java 20019d2ef addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropTable.java 440c08aff addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/LineageOnlyFilter.java PRE-CREATION addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java PRE-CREATION addons/hive-bridge/src/test/java/org/apache/atlas/hive/ActiveEntityFilterTest.java PRE-CREATION addons/hive-bridge/src/test/resources/atlas-application.properties 898b69c99 addons/hive-bridge/src/test/resources/json/hs2-alter-view-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-alter-view.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-create-db.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-create-process.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-create-table.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-drop-db-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-drop-db.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-drop-table-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-drop-table.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json PRE-CREATION addons/hive-bridge/src/test/resources/json/hs2-table-rename.json PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 Diff: https://reviews.apache.org/r/73239/diff/3/ Changes: https://reviews.apache.org/r/73239/diff/2-3/ Testing (updated) ------- **Functional tests** - Used specific queries to exercise the affected code paths. - Verified messages going out of the hooks, onces posted on ATLAS_HOOK and the entities created within Atlas. **Test data** ``` create database cadb02; use cadb02; create external table hh6(col1 int) location '/tmp/external/hh6.csv'; ALTER TABLE hh6 RENAME TO hh6_renamed; create view hh6_renamed_view as select * from hh6_renamed; ALTER VIEW hh6_renamed_view RENAME TO hh6_renamed_view2; create external table bb1(col1 int) location '/tmp/bb1.csv'; ALTER TABLE bb1 CHANGE COLUMN col1 col11 string; ALTER TABLE bb1 SET SERDEPROPERTIES ('field.delim' = '|'); ALTER TABLE bb1 ADD COLUMNS (dept STRING COMMENT 'Department name'); ALTER TABLE bb1 RENAME TO cc1; ``` **Volume test** +----------+-------------+--------------+ | Hook | Baseline | New | +----------+-------------+--------------+ | HMS | 31 KB. | 31 KB | +----------+-------------+--------------+ | HS2 | 53 KB. | 12 KB | +----------+-------------+--------------+ **Unit test** New tests added. Thanks, Ashutosh Mestry