-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73239/
-----------------------------------------------------------
(Updated March 22, 2021, 5:23 p.m.)
Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-4204
https://issues.apache.org/jira/browse/ATLAS-4204
Repository: atlas
Description
-------
**Background**
Please see JIRA.
Terms and abbreviations used:
- Database objects: Database, table, views, etc.
- DDL: Data Definition Langugage. Database parlance. Operations performed on a
database server that result in creation of objects that hold data.
- DML: Data Manipulation Language. Database parlance. Operations performed on a
database server that result in manipulation (add, update, delete) of data.
These are performed using database objects that are already created.
Hive hook is implemented by:
- _HiveHook_
- _HiveMetastoreImpl_
- _BaseHiveEvent_ has 2 methods:
- getHiveMetastoreEntities: Called by HMS events.
- getHiveEntities: Called by HS2 events.
**Approach**
- Introduce a flag _skipDDL_ indicating if entities generated as part of DDL
operation should be skipped by a hook. This flag is set as a configuration
parameters: atlas.hive.hook.ignore.ddl.operations
- Skip creation of Atlas's DDL entities when HS2 event is used.
- Continue using HMS event processing unchanged.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
128647147
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
79e87c79d
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
f01419ce6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java
6961fa7c2
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRenameCol.java
756a6089e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
7c269ce53
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
bf5f5620e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java
63fc89424
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropDatabase.java
20019d2ef
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/DropTable.java
440c08aff
addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
7a11ec3b3
addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/AlterDatabaseTest.java
PRE-CREATION
addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/CreateDBTest.java
PRE-CREATION
addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/CreateTableTest.java
PRE-CREATION
addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/DropDatabaseTest.java
PRE-CREATION
addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/DropTableTest.java
PRE-CREATION
addons/hive-bridge/src/test/java/org/apache/atlas/hive/events/HiveHookEventFixtures.java
PRE-CREATION
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
52684e75a
addons/hive-bridge/src/test/resources/json/alter-db-hs2.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/alter-db.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/alter-table-hs2.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/alter-table-rename.json
PRE-CREATION
addons/hive-bridge/src/test/resources/json/alter-table.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/create-db-hs2.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/create-db-table-col-hms.json
PRE-CREATION
addons/hive-bridge/src/test/resources/json/create-db-table-col.json
PRE-CREATION
addons/hive-bridge/src/test/resources/json/create-db.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/create-table-hs2.json PRE-CREATION
addons/hive-bridge/src/test/resources/json/create-table.json PRE-CREATION
common/src/main/java/org/apache/atlas/utils/PathExtractorContext.java
ce688e4e4
Diff: https://reviews.apache.org/r/73239/diff/2/
Changes: https://reviews.apache.org/r/73239/diff/1-2/
Testing (updated)
-------
**Functional tests**
- Used specific queries to exercise the affected code paths.
- Verified messages going out of the hooks, onces posted on ATLAS_HOOK and the
entities created within Atlas.
**Volume test**
Pending.
**Unit test**
New tests added.
File Attachments
----------------
HMS Create Table
https://reviews.apache.org/media/uploaded/files/2021/03/18/748a849a-c8f8-4869-bcb0-ee4ee50215d8__hms-create-table.json
HS2 Create Table
https://reviews.apache.org/media/uploaded/files/2021/03/18/3f27aac6-a741-4758-9d6a-db67b05a63eb__hs2-create-table.json
HS2 Create Database
https://reviews.apache.org/media/uploaded/files/2021/03/18/e0472b5d-3367-42d8-834a-7cfafa13aae6__hs2-create-db.json
HMS Create Database
https://reviews.apache.org/media/uploaded/files/2021/03/18/bc34e035-f805-4ddd-abf3-a2b833ce3af0__hms-create-db.json
Thanks,
Ashutosh Mestry
