> On Aug. 12, 2020, 1:45 a.m., Madhan Neethiraj wrote: > > For some DDLs like 'create view', HMS doesn't have all details. So, it is > > critical that we process DDLs in HiveServer2. > > > > Consider adding a new message-type ENTITY_RENAME, and for such messages > > Atlas should ignore if source entity is not found.
I have updated patch with your suggested changes kindly review. - chaitali ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72756/#review221551 ----------------------------------------------------------- On Sept. 14, 2020, 11:36 a.m., chaitali wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72756/ > ----------------------------------------------------------- > > (Updated Sept. 14, 2020, 11:36 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3918 > https://issues.apache.org/jira/browse/ATLAS-3918 > > > Repository: atlas > > > Description > ------- > > Issue : Messages are being sent for DDL operations one from HMS and > HiveServer2 which causes duplicate messages hence it throws exception: > > message > {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":1594802485264,"message":{"type":"ENTITY_PARTIAL_UPDATE_V2","user":"hrt_qa","entityId":{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.t1@cm"}},"entity":{"entity":{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"MANAGED_TABLE","temporary":false,"lastAccessTime":1594802469000,"aliases":["t1"],"createTime":1594802469000,"qualifiedName":"default.t2@cm","name":"t2","comment":null,"parameters":{"last_modified_time":"1594802481","rawDataSize":"0","numRows":"0","transient_lastDdlTime":"1594802481","bucketing_version":"2","last_modified_by":"hrt_qa","numFilesErasureCoded":"0","totalSize":"0","transactional_properties":"default","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}}","numFiles":"0","transactional":"true"},"retention":0},"guid":"-21782793831367970","isIncomplete":fal se,"provenanceType":0,"version":0,"proxy":false}}}} > > This patch fix it by providing a property "DDL_MESSAGES_ENABLE" in > configuration when enabled the HiveServer2 will send ddl messages and if > disabled then it will only send non ddl messages. > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > 6961fa7c2 > > intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java > 5b5fa04e2 > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > f02c05fff > > > Diff: https://reviews.apache.org/r/72756/diff/4/ > > > Testing > ------- > > Tested by creating and renaming the hive_table hence ddl messages are not > sent from hiveserver2 after the property is set to false and so no exception > is thrown: > > create table t1(id int); > > > alter table t1 rename to t2; > > > Thanks, > > chaitali > >
