Commit to Atlas.
Hello, I would like to contribute to Apache Atlas. Can you please add me as a contributor in Apache Atlas? My JIRA account name is lma . Thanks, Le
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
> On May 8, 2019, 12:40 a.m., Sarath Subramanian wrote: > > DDL queries needs to be captured for the following events as well. Please > > review. > > > > AlterDatabase > > AlterTableRenameCol Hello Sarath, thanks for reviewing it. AlterDatabase re-uses getHiveEntities() method in CreateDatabase and same for AlterTableRenameCol, it calls getNotificationMessages() form its super class which is inside createTable.And changes are already made for these two classes. - Le - Le --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/#review215111 --- On May 7, 2019, 6:35 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70605/ > --- > > (Updated May 7, 2019, 6:35 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3197 > https://issues.apache.org/jira/browse/ATLAS-3197 > > > Repository: atlas > > > Description > --- > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > d3d83490f > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > de8de63b6 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java > b01f61f76 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > b5ce09332 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 24de30f5a > addons/models/1000-Hadoop/1030-hive_model.json ee6945a77 > > > Diff: https://reviews.apache.org/r/70605/diff/3/ > > > Testing > --- > > 1. table : > > CREATE TABLE IF NOT EXISTS testDDL (d int, n string); > > > ALTER TABLE testDDL RENAME TO renameTestDDL; > > ALTER TABLE renameTestDDL CHANGE n changedCol string; > > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); > > DROP TABLE renameTestDDL; > > > 2. db > > create database testdb; > > ALTER DATABASE database SET OWNER USER changedOwner; > > DROP DATABSE testdb; > > > File Attachments > > > CREATE TABLE IF NOT EXISTS testDDL.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png > ALTER TABLE testDDL RENAME TO renameTestDDL; .png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png > ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png > ALTER TABLE renameTestDDL CHANGE n changedCol string;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png > ALTER TABLE renameTestDDL ADD COLUMNS.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png > DROP TABLE renameTestDDL;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png > ddl - relationship.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png > ALTER DATABASE testdb SET OWNER USER changedOwner;.png > > https
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/ --- (Updated May 8, 2019, 11:22 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Thank you all for reviewing it. Addressed comments. Re-run end-2-end tests and unit tests. it works good. Bugs: ATLAS-3197 https://issues.apache.org/jira/browse/ATLAS-3197 Repository: atlas Description --- This patch introduced a model called Hive_table_ddl and hive_db_ddl which are used to track commands that are executed either on a table entity or bd entity. DDL entities will be added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity will be created, it will append to the existing DDLQueries field if a table/db has already been created. Once a table/db is deleted, all these relationship attribute will be deleted as well. Diffs (updated) - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java d3d83490f addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java de8de63b6 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java b01f61f76 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java b5ce09332 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 24de30f5a addons/models/1000-Hadoop/1030-hive_model.json ee6945a77 Diff: https://reviews.apache.org/r/70605/diff/4/ Changes: https://reviews.apache.org/r/70605/diff/3-4/ Testing --- 1. table : CREATE TABLE IF NOT EXISTS testDDL (d int, n string); ALTER TABLE testDDL RENAME TO renameTestDDL; ALTER TABLE renameTestDDL CHANGE n changedCol string; ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); DROP TABLE renameTestDDL; 2. db create database testdb; ALTER DATABASE database SET OWNER USER changedOwner; DROP DATABSE testdb; File Attachments CREATE TABLE IF NOT EXISTS testDDL.png https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png ALTER TABLE testDDL RENAME TO renameTestDDL; .png https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png ALTER TABLE renameTestDDL CHANGE n changedCol string;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png ALTER TABLE renameTestDDL ADD COLUMNS.png https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png DROP TABLE renameTestDDL;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png ddl - relationship.png https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png ALTER DATABASE testdb SET OWNER USER changedOwner;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/388e7ffa-6691-4c2b-bcd9-d8755dc13e3e__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner.png ALTER DATABASE testdb SET OWNER USER changedOwner; - db entity.png https://reviews.apache.org/media/uploaded/files/2019/05/07/fd22b6a5-6cb9-40eb-aa10-5389f6f74792__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner_-_db_entity.png drop database testdb;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/7c889d62-66c8-4c26-bf94-257bd01cdb70__drop_database_testdb.png drop database testdb;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/4163043b-c2f1-42ee-9e5c-cfffbc589fb3__drop_database_testdb.png Thanks, Le Ma
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/ --- (Updated May 9, 2019, 10:30 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Address Aadarsh's comment and add corresponding test. Bugs: ATLAS-3197 https://issues.apache.org/jira/browse/ATLAS-3197 Repository: atlas Description --- This patch introduced a model called Hive_table_ddl and hive_db_ddl which are used to track commands that are executed either on a table entity or bd entity. DDL entities will be added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity will be created, it will append to the existing DDLQueries field if a table/db has already been created. Once a table/db is deleted, all these relationship attribute will be deleted as well. Diffs (updated) - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java d3d83490f addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java de8de63b6 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java b01f61f76 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java d1511c0d9 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java b5ce09332 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 24de30f5a addons/models/1000-Hadoop/1030-hive_model.json ee6945a77 Diff: https://reviews.apache.org/r/70605/diff/5/ Changes: https://reviews.apache.org/r/70605/diff/4-5/ Testing --- 1. table : CREATE TABLE IF NOT EXISTS testDDL (d int, n string); ALTER TABLE testDDL RENAME TO renameTestDDL; ALTER TABLE renameTestDDL CHANGE n changedCol string; ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); DROP TABLE renameTestDDL; 2. db create database testdb; ALTER DATABASE database SET OWNER USER changedOwner; DROP DATABSE testdb; File Attachments CREATE TABLE IF NOT EXISTS testDDL.png https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png ALTER TABLE testDDL RENAME TO renameTestDDL; .png https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png ALTER TABLE renameTestDDL CHANGE n changedCol string;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png ALTER TABLE renameTestDDL ADD COLUMNS.png https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png DROP TABLE renameTestDDL;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png ddl - relationship.png https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png ALTER DATABASE testdb SET OWNER USER changedOwner;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/388e7ffa-6691-4c2b-bcd9-d8755dc13e3e__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner.png ALTER DATABASE testdb SET OWNER USER changedOwner; - db entity.png https://reviews.apache.org/media/uploaded/files/2019/05/07/fd22b6a5-6cb9-40eb-aa10-5389f6f74792__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner_-_db_entity.png drop database testdb;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/7c889d62-66c8-4c26-bf94-257bd01cdb70__drop_database_testdb.png drop database testdb;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/4163043b-c2f1-42ee-9e5c-cfffbc589fb3__drop_database_testdb.png Thanks, Le Ma
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/ --- (Updated May 13, 2019, 5:38 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Handle HMS for process/processExe/DDL Bugs: ATLAS-3197 https://issues.apache.org/jira/browse/ATLAS-3197 Repository: atlas Description --- This patch introduced a model called Hive_table_ddl and hive_db_ddl which are used to track commands that are executed either on a table entity or bd entity. DDL entities will be added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity will be created, it will append to the existing DDLQueries field if a table/db has already been created. Once a table/db is deleted, all these relationship attribute will be deleted as well. Diffs (updated) - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java 67ea5275a addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java d1511c0d9 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java ef4e6874d addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 28c600e28 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveMetastoreHookIT.java fdd65eb33 Diff: https://reviews.apache.org/r/70605/diff/6/ Changes: https://reviews.apache.org/r/70605/diff/5-6/ Testing --- 1. table : CREATE TABLE IF NOT EXISTS testDDL (d int, n string); ALTER TABLE testDDL RENAME TO renameTestDDL; ALTER TABLE renameTestDDL CHANGE n changedCol string; ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); DROP TABLE renameTestDDL; 2. db create database testdb; ALTER DATABASE database SET OWNER USER changedOwner; DROP DATABSE testdb; File Attachments CREATE TABLE IF NOT EXISTS testDDL.png https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png ALTER TABLE testDDL RENAME TO renameTestDDL; .png https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png ALTER TABLE renameTestDDL CHANGE n changedCol string;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png ALTER TABLE renameTestDDL ADD COLUMNS.png https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png DROP TABLE renameTestDDL;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png ddl - relationship.png https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png ALTER DATABASE testdb SET OWNER USER changedOwner;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/388e7ffa-6691-4c2b-bcd9-d8755dc13e3e__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner.png ALTER DATABASE testdb SET OWNER USER changedOwner; - db entity.png https://reviews.apache.org/media/uploaded/files/2019/05/07/fd22b6a5-6cb9-40eb-aa10-5389f6f74792__ALTER_DATABASE_testdb_SET_OWNER_USER_changedOwner_-_db_entity.png drop database testdb;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/7c889d62-66c8-4c26-bf94-257bd01cdb70__drop_database_testdb.png drop database testdb;.png https://reviews.apache.org/media/uploaded/files/2019/05/07/4163043b-c2f1-42ee-9e5c-cfffbc589fb3__drop_database_testdb.png Thanks, Le Ma
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
> On May 13, 2019, 5:46 p.m., Madhan Neethiraj wrote: > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > > Lines 120 (patched) > > <https://reviews.apache.org/r/70605/diff/6/?file=2144655#file2144655line120> > > > > Consider adding this condition in line #117 above. Hello Madhan, thanks for viewing it. If we move the HMS check above #117, none of the table enities will be created for HMS, because table entity will be added to AtlasEntitiesWithExtInfo ret inside the toTableEntity function. Will conbine the hookcheck with #122. Thanks. - Le --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/#review215222 ----------- On May 13, 2019, 5:38 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70605/ > --- > > (Updated May 13, 2019, 5:38 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3197 > https://issues.apache.org/jira/browse/ATLAS-3197 > > > Repository: atlas > > > Description > --- > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > 67ea5275a > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java > d1511c0d9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > ef4e6874d > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 28c600e28 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveMetastoreHookIT.java > fdd65eb33 > > > Diff: https://reviews.apache.org/r/70605/diff/6/ > > > Testing > --- > > 1. table : > > CREATE TABLE IF NOT EXISTS testDDL (d int, n string); > > > ALTER TABLE testDDL RENAME TO renameTestDDL; > > ALTER TABLE renameTestDDL CHANGE n changedCol string; > > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); > > DROP TABLE renameTestDDL; > > > 2. db > > create database testdb; > > ALTER DATABASE database SET OWNER USER changedOwner; > > DROP DATABSE testdb; > > > File Attachments > > > CREATE TABLE IF NOT EXISTS testDDL.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png > ALTER TABLE testDDL RENAME TO renameTestDDL; .png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png > ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png > ALTER TABLE renameTestDDL CHANGE n changedCol string;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png > ALTER TABLE renameTestDDL ADD COLUMNS.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png > DROP TABLE renameTestDDL;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png > ddl - relationship.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png > ALTER
Re: Review Request 70605: ATLAS-3197 Add QueryText for hive_table and hive_db
> On May 13, 2019, 6:06 p.m., Ashutosh Mestry wrote: > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > > Line 205 (original), 205 (patched) > > <https://reviews.apache.org/r/70605/diff/6/?file=2144656#file2144656line205> > > > > This is a disabled test. Should this be enabled? Hello Ashutosh, thanks for reviewing this patch. i've ran this test. It seems it has not been fixed yet. It shows some errors on my local env. It would be better to leave it there for now. Thanks - Le --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70605/#review215224 ------- On May 13, 2019, 5:38 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70605/ > --- > > (Updated May 13, 2019, 5:38 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3197 > https://issues.apache.org/jira/browse/ATLAS-3197 > > > Repository: atlas > > > Description > --- > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterTableRename.java > 67ea5275a > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java > d1511c0d9 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateTable.java > ef4e6874d > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java > 28c600e28 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveMetastoreHookIT.java > fdd65eb33 > > > Diff: https://reviews.apache.org/r/70605/diff/6/ > > > Testing > --- > > 1. table : > > CREATE TABLE IF NOT EXISTS testDDL (d int, n string); > > > ALTER TABLE testDDL RENAME TO renameTestDDL; > > ALTER TABLE renameTestDDL CHANGE n changedCol string; > > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int); > > DROP TABLE renameTestDDL; > > > 2. db > > create database testdb; > > ALTER DATABASE database SET OWNER USER changedOwner; > > DROP DATABSE testdb; > > > File Attachments > > > CREATE TABLE IF NOT EXISTS testDDL.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/5047d461-f288-4776-a843-556e45217fef__CREATE_TABLE_IF_NOT_EXISTS_testDDL.png > ALTER TABLE testDDL RENAME TO renameTestDDL; .png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/373a656d-8ab3-4710-b9a1-3eac20adc947__ALTER_TABLE_testDDL_RENAME_TO_renameTestDDL_.png > ALTER TABLE renameTestDDL CHANGE n changedCol string; - ddl.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/c9f57d67-c16b-4142-a6b4-9d372e7fd765__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string_-_ddl.png > ALTER TABLE renameTestDDL CHANGE n changedCol string;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/42612eb6-83c2-4825-8601-5a1629c9de6f__ALTER_TABLE_renameTestDDL_CHANGE_n_changedCol_string.png > ALTER TABLE renameTestDDL ADD COLUMNS(addedCol int);.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/f5d3522d-17b6-4227-8072-6fc07d88d515__ALTER_TABLE_renameTestDDL_ADD_COLUMNSaddedCol_int.png > ALTER TABLE renameTestDDL ADD COLUMNS.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/73b9046a-2667-4049-9e11-c5c2b375e9ca__ALTER_TABLE_renameTestDDL_ADD_COLUMNS.png > DROP TABLE renameTestDDL;.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/ebfa01f2-e886-4f94-aa75-ff45bb519ca0__DROP_TABLE_renameTestDDL.png > ddl - relationship.png > > https://reviews.apache.org/media/uploaded/files/2019/05/07/7b30282b-6419-4791-b107-f938918e1cfd__ddl_-_relationship.png > ALTER DATABASE testdb SET OWNER USER changedOwner;.png > > https://reviews.apache.org/m
Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70759/#review215618 --- addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java Lines 107 (patched) <https://reviews.apache.org/r/70759/#comment302374> Hello Lina, thanks for doing this. There is one case you might need to consider - CreateImpalaProcess will be called by cases like QUERY/ALTERVIEW_AS/CREATETABLE_AS_SELECT/CREATEVIEW. However, ddl might not need to capture QUERY case. -Thanks addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java Lines 93 (patched) <https://reviews.apache.org/r/70759/#comment302375> Please align '=' to match the code-style. and please check other places. -Thanks. - Le Ma On May 31, 2019, 4:02 a.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70759/ > --- > > (Updated May 31, 2019, 4:02 a.m.) > > > Review request for atlas, madhan and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3197 introduced a new feature on Hive integration. It introduced new > model called Hive_table_ddl and hive_db_ddl which are used to track commands > that are executed either on a table entity or bd entity. DDL entities will be > added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL > and hive_tb_to_ddl. Every time when a new command is executed, a new ddl > entity will be created, it will append to the existing ddlQueries field if a > table/db has already been created. Once a table/db is deleted, all these > relationship attributes will be deleted as well. > > We need to support this new feature on Impala integration. > > > Diffs > - > > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > d241b6a > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java > b7506a4 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java > 10ce448 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > b8cbf6b > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > 7f9a534 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > 6156208 > addons/models/-Area0/0010-base_model.json d14b227 > addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 > addons/models/1000-Hadoop/1090-impala_model.json 24eb5db > > > Diff: https://reviews.apache.org/r/70759/diff/3/ > > > Testing > --- > > add check for ddl entities in integration tests > > > Thanks, > > Na Li > >
Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70759/#review215628 --- addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java Lines 646 (patched) <https://reviews.apache.org/r/70759/#comment302386> Hello Lina, thanks for the fix. If serviceType is going to be added there. The model joson file might need to do the corresponding by adding a new attribute serviceType just like queryText and execTime. I did not see the change in the json file. If you have that change, please ignore this comment. -Thanks - Le Ma On May 31, 2019, 11:24 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70759/ > --- > > (Updated May 31, 2019, 11:24 p.m.) > > > Review request for atlas, madhan and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3197 introduced a new feature on Hive integration. It introduced new > model called Hive_table_ddl and hive_db_ddl which are used to track commands > that are executed either on a table entity or bd entity. DDL entities will be > added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL > and hive_tb_to_ddl. Every time when a new command is executed, a new ddl > entity will be created, it will append to the existing ddlQueries field if a > table/db has already been created. Once a table/db is deleted, all these > relationship attributes will be deleted as well. > > We need to support this new feature on Impala integration. > > > Diffs > - > > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > d241b6a > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java > b7506a4 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java > 10ce448 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > b8cbf6b > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > 7f9a534 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > 6156208 > addons/models/-Area0/0010-base_model.json d14b227 > addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 > > > Diff: https://reviews.apache.org/r/70759/diff/4/ > > > Testing > --- > > add check for ddl entities in integration tests > > > Thanks, > > Na Li > >
Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70759/#review215661 --- Ship it! Ship It! - Le Ma On June 3, 2019, 7:08 p.m., Na Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70759/ > --- > > (Updated June 3, 2019, 7:08 p.m.) > > > Review request for atlas, madhan and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3197 introduced a new feature on Hive integration. It introduced new > model called Hive_table_ddl and hive_db_ddl which are used to track commands > that are executed either on a table entity or bd entity. DDL entities will be > added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL > and hive_tb_to_ddl. Every time when a new command is executed, a new ddl > entity will be created, it will append to the existing ddlQueries field if a > table/db has already been created. Once a table/db is deleted, all these > relationship attributes will be deleted as well. > > We need to support this new feature on Impala integration. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 7b4b893 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java > d241b6a > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java > b7506a4 > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java > 10ce448 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > b8cbf6b > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > 7f9a534 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > 6156208 > addons/models/-Area0/0010-base_model.json d14b227 > addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 > > > Diff: https://reviews.apache.org/r/70759/diff/5/ > > > Testing > --- > > add check for ddl entities in integration tests > > > Thanks, > > Na Li > >
Re: [VOTE] Release Apache Atlas version 1.2.0 - rc3
+ 1 for Apache Atlas 1.2.0 # 3 - built successfully from the source tar file - ran quick_start_v1, data successfully shows up - validate basic search functionality - add/delete classification, results are expected. thanks, Le Ramesh Mani 于2019年6月4日周二 上午8:44写道： > +1 for Apache Atlas 1.2.0 rc3 > > - Built successfully from the source tar file > > https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz > - Verified the PGP and SHA > > Thanks, > Ramesh > > On Mon, Jun 3, 2019 at 5:10 AM Binit Gutka < > binit.gu...@freestoneinfotech.com> wrote: > > > +1 Release Apache Atlas 1.2.0 release candidate #3 > > > > - Verified PGP signature. > > - Built successfully from the source tar file > > - Ran quick start and validated basic functionality. > > - Tested Atlas UI packaging in embedded-hbase mode. > > - Performed few basic search. > > - Lineage Export functionality tested. > > - Lineage Pan, Zoom, Filter, Search and settings functionality tested. > > > > Thanks. > > > > On Mon, Jun 3, 2019 at 1:41 PM Nixon Rodrigues < > > nixon.rodrig...@freestoneinfotech.com> wrote: > > > > > Atlas team, > > > > > > Apache Atlas 1.2.0 release candidate #3 is now available for a vote > > within > > > dev community. > > > > > > Following commits went into branch-1.0 since the last release candidate > > > (rc2): > > > > > > ATLAS-3180:- New Bulk retrieval API with servicetype parameter. > > > ATLAS-3227:- Bulk delete does not delete the indexes properly. > > > > > > Links to the release artifacts are given below. Please review and vote. > > > > > > The vote will be open for at least 72 hours or until necessary votes > are > > > reached. > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and reason why) > > > > > > > > > Thanks, > > > Nixon Rodrigues > > > > > > > > > List of issues addressed in this release: > > > > > > > > > https://issues.apache.org/jira/browse/ATLAS-3201?jql=project%20%3D%20ATLAS%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%201.2.0%20ORDER%20BY%20key%20DESC > > > > > > Git tag for the release: > > > https://github.com/apache/atlas/releases/tag/release-1.2.0-rc3 > > > > > > Sources for the release: > > > > > > > > > https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz > > > > > > Source release verification: > > > PGP Signature: > > > > > > > > > https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz.asc > > > MD5 Hash: > > > > > > > > > https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz.md5 > > > SHA512 Hash: > > > > > > > > > https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz.sha512 > > > > > > Keys to verify the signature of the release artifacts are available at: > > > https://dist.apache.org/repos/dist/dev/atlas/KEYS > > > > > > > > > -- > > *Thanks,* > > *Binit.* > > >
Re: Review Request 70913: ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70913/#review216285 --- intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java Lines 168 (patched) <https://reviews.apache.org/r/70913/#comment303430> Please make sure you've run pre-commit job before committing the code. - Le Ma On July 1, 2019, 6:10 p.m., Merryle Wang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70913/ > --- > > (Updated July 1, 2019, 6:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes > > > Diffs > - > > addons/models/1000-Hadoop/1030-hive_model.json > 8901aa4aa86fb2802a9e9b1e65c2ff8aad8855ad > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > d9ae9e36773146fad652a1e28fc8822bae5c8557 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java > 254eee4c9d18ab87a8862ce479fc2274227c7c29 > intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java > 3c53c02b48747515217c9327c98209a48ee84237 > > > Diff: https://reviews.apache.org/r/70913/diff/7/ > > > Testing > --- > > Added unit test > > > Thanks, > > Merryle Wang > >
Re: Review Request 71127: ATLAS-3343: Ordering of dynAttr evaluation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71127/#review216815 --- Hello Merryle, the toplogical sort algo is very nice done. I just have some questions about the in/outvertex and the bimap. Is the mapping from attr to index and index to attr needed here? Instead, you can use Map> adjList, a queue and a inDegree array/map, and a result list to help you finish the sort: 1. traverse the input the attr list, maintain a key-value (attr and list of outVertex), for every outVertex we've met, incrase the inDegree array/map. 2. Once the traverse of the input attr arry is done, initialize a queue to maintain the starting point of your graph, travese the inDegree map, add any attributs which have 0 as its inDegree. 3. while(!queue.isEmpty()), poll out the attr inside the queue, add it to the result list, 4. traverse it outVertex lists, for every outVertex, decrease its inDegree, and at the same time, check if the ingree is zore, if so, offer that attr into the queue. 5. Once done while-loop the queue, your result list will be the desire list, no sorting is needed. If you have questions, i am happy to discuss. Thanks, Le - Le Ma On July 19, 2019, 6:49 p.m., Merryle Wang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71127/ > --- > > (Updated July 19, 2019, 6:49 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3343: Ordering of dynAttr evaluation > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > 23eaa0a2e88fd348d2347314170726ebb5cb4393 > > > Diff: https://reviews.apache.org/r/71127/diff/1/ > > > Testing > --- > > Created dummy dynamic attributes to test the correct ordering. > > > Thanks, > > Merryle Wang > >
Re: Review Request 71127: ATLAS-3343: Ordering of dynAttr evaluation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71127/#review216838 --- Looks good to me. Nice done! - Le Ma On July 23, 2019, 11:22 p.m., Merryle Wang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71127/ > --- > > (Updated July 23, 2019, 11:22 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3343: Ordering of dynAttr evaluation > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > 23eaa0a2e88fd348d2347314170726ebb5cb4393 > intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java > c114bdf7888519cdd396fc8501bbcd92b00ef0bb > > > Diff: https://reviews.apache.org/r/71127/diff/2/ > > > Testing > --- > > Created dummy dynamic attributes to test the correct ordering. > > > Thanks, > > Merryle Wang > >
Run tests/pre-commit job before committing the patch
Hello Atlas team and community, Hope you all are doing great. It is a honor to be part of this awesome product. And thanks to the efforts from community, Atlas is growing fantastic, and we are getting more contribution. In order to help Atlas grow, improve the efficiency of development and the quality of code, I would suggest that all of us could run Pre-commit job or local unit tests fist, once it passes all tests, patches could go into the branch. And I would also suggest that all of us could mention either pre-commit job or local tests has succeed while we send our code out for review Or when we create a pull request. This would really improve the development of Atlas by decreasing time to triage and debugging some bugs which could have been found out If units are ran. Thanks again, Le
Re: Review Request 71620: ATLAS-3461: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71620/#review218312 --- addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java Line 33 (original), 33 (patched) <https://reviews.apache.org/r/71620/#comment305971> Hey Sid, thanks for the patch. I am not sure about the regex there "^[ ]*", does this mean the beginning of the line can have multiple white spaces? I saw you added this square brackets at diff2. '\bcreate\b' means exact match word "create", then if there are spaces before "create", will it still match? thanks, Le - Le Ma On Oct. 21, 2019, 9:01 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71620/ > --- > > (Updated Oct. 21, 2019, 9:01 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3461 > https://issues.apache.org/jira/browse/ATLAS-3461 > > > Repository: atlas > > > Description > --- > > Now even if the Impala queries contains comment/ extra spaces, Atlas will > create the impala process approproately. > Even "create external table" will create Impala process at Atlas and user can > see lineage. > > > Diffs > - > > > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java > b9dd894f8 > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > dc4d1333e > > addons/impala-bridge/src/test/resources/impalaAlterViewAsSelectWithCommentSpaces.json > PRE-CREATION > > addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectWithCommentSpaces.json > PRE-CREATION > > addons/impala-bridge/src/test/resources/impalaCreateViewWithCommentSpaces.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/71620/diff/3/ > > > Testing > --- > > 1. Unit Tests added to ImpalaLineageToolIT and without fix it was failing. > Now after the fix it works. > 2. Validated manually by running below mentioned query from Impala:JDBC, Hue > and Impala Shell > > a) create table testMon4 as select * from t1; > > b) creAte external table /*test*/ testMo3 aS -- temp fff > /* temp */ -- sdfrnfkjef select > select * from t1; > > > c) Create /*dd*/ table testMon4 aS select * from t1; > > d) alter view testMonView2 as select * from t1; > > e) create view testMonView4 as select * from t1; > > f) alter view testMonView4 as select * from t1; > > g) create EXTERNAL TABLE IF NOT EXISTS thusday_ext1 >(STUD_ID int, >DEPT_ID int, >NAME string >) >ROW FORMAT DELIMITED >FIELDS TERMINATED BY ',' >STORED AS TEXTFILE >LOCATION '/warehouse/tablespace/external/hive/db1.db/thusday_ext1'; > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71649/#review218343 --- intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java Lines 60 (patched) <https://reviews.apache.org/r/71649/#comment306014> "==" will comapre the reference of real objects in Java, please consider using "equals" intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java Lines 106 (patched) <https://reviews.apache.org/r/71649/#comment306015> instead of using a variable i, Why not using for (int i = 0; i < xx.size(); i++), it would be cleaner and more intitive, only when i is the laste index, you can stop appending "," intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java Lines 180 (patched) <https://reviews.apache.org/r/71649/#comment306016> same commet as above. intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java Lines 195 (patched) <https://reviews.apache.org/r/71649/#comment306017> Should we consider compare its attributes inside this equal funciton? Use name as the only comparison might cause issues. intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java Lines 207 (patched) <https://reviews.apache.org/r/71649/#comment306018> Consider using the result of toString() to get hashcode. intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java Lines 211 (patched) <https://reviews.apache.org/r/71649/#comment306019> Why not merging two toString() functions? create the stringbuilder inside, and return sb.toString() ? - Le Ma On Oct. 22, 2019, 9:08 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71649/ > --- > > (Updated Oct. 22, 2019, 9:08 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3486 > https://issues.apache.org/jira/browse/ATLAS-3486 > > > Repository: atlas > > > Description > --- > > This change is the first part of the bigger task defined as part of > ATLAS-3485. > This adds the data model needed for supporting namespaces and namespace > attributes > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > e10965b870c4a300b41e93ee046b5f6d6b722728 > > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/71649/diff/1/ > > > Testing > --- > > Added unit tests > > > Thanks, > > Aadarsh Jajodia > >
Re: Review Request 71674: ATLAS-3491 Regression : Search on parent tag with includeSubclassifications:True doesn't return entities associated to its child tag
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71674/ --- (Updated Oct. 24, 2019, 10:19 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Pre-commit job succeed. Bugs: ATLAS-3491 https://issues.apache.org/jira/browse/ATLAS-3491 Repository: atlas Description --- Create tag parent_tag Create child_tag with parent tag as parent_tag Associate child_tag to entity e1. Perform search with parent_tag with includeSubclassifications set to True. Expected result is e1 but no entities are returned. No exceptions seen in application logs Diffs - repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java f9989ab86 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 015aadec2 webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 16260bc1b Diff: https://reviews.apache.org/r/71674/diff/2/ Testing (updated) --- - Added tests, passed. - will run pre-commit job:https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1492/ - Pre-commit job succeed. Thanks, Le Ma
Re: Review Request 71612: ATLAS-3457 Support Multi-Classification search without attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71612/ --- (Updated Oct. 25, 2019, 5:34 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Rebase codebase and resolve comments Bugs: ATLAS-3457 https://issues.apache.org/jira/browse/ATLAS-3457 Repository: atlas Description --- support multi-classification Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 7a2aae2e9 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/GraphIndexQueryBuilder.java 3f58acb9c repository/src/main/java/org/apache/atlas/discovery/MultiClassificationSearchProcessor.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java b56d8e83a webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 908ae7033 webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71612/diff/5/ Changes: https://reviews.apache.org/r/71612/diff/4-5/ Testing --- - Search related tests passed. - added unit tests, it passed. - Pre-commit job passed. https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1471/ Thanks, Le Ma
Review Request 71689: ATLAS-3483 Support adding/deleting labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71689/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3483 https://issues.apache.org/jira/browse/ATLAS-3483 Repository: atlas Description --- The current implementation of label only exposes setLabel endpoint to users. setLabel will replace the existing labels with labels passed in with function. It would be nice to have the enhancement to support add labels to existing label and, delete label. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java a986520a9 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 17b4a5b9d repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java f07cff1a8 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 19d4cef3d repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java d95c12762 webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java e5f7d3339 Diff: https://reviews.apache.org/r/71689/diff/1/ Testing --- - Added tests, passed. - will run pre-commit job. Thanks, Le Ma
Re: Review Request 71689: ATLAS-3483 Support adding/deleting labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71689/ --- (Updated Oct. 28, 2019, 9:23 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- resolve comments Bugs: ATLAS-3483 https://issues.apache.org/jira/browse/ATLAS-3483 Repository: atlas Description --- The current implementation of label only exposes setLabel endpoint to users. setLabel will replace the existing labels with labels passed in with function. It would be nice to have the enhancement to support add labels to existing label and, delete label. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java a986520a9 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 17b4a5b9d repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java f07cff1a8 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 19d4cef3d repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java d95c12762 webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java e5f7d3339 Diff: https://reviews.apache.org/r/71689/diff/2/ Changes: https://reviews.apache.org/r/71689/diff/1-2/ Testing --- - Added tests, passed. - will run pre-commit job. Thanks, Le Ma
Re: Review Request 71650: ATLAS-3482 Enhance basic search to enable search on entity system attributes
] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:32 h [INFO] Finished at: 2019-10-23T13:37:53-07:00 [INFO] Thanks, Le Ma
Re: Review Request 71689: ATLAS-3483 Support adding/deleting labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71689/ --- (Updated Oct. 29, 2019, 5:30 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- resolve comments Bugs: ATLAS-3483 https://issues.apache.org/jira/browse/ATLAS-3483 Repository: atlas Description --- The current implementation of label only exposes setLabel endpoint to users. setLabel will replace the existing labels with labels passed in with function. It would be nice to have the enhancement to support add labels to existing label and, delete label. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java a986520a9 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 17b4a5b9d repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java f07cff1a8 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 19d4cef3d repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java d95c12762 webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java e5f7d3339 Diff: https://reviews.apache.org/r/71689/diff/3/ Changes: https://reviews.apache.org/r/71689/diff/2-3/ Testing --- - Added tests, passed. - will run pre-commit job. https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1503/console Thanks, Le Ma
Re: Review Request 71694: ATLAS-3498: Update basic-search to enable search using labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71694/#review218438 --- repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java Line 109 (original), 110 (patched) <https://reviews.apache.org/r/71694/#comment306159> This line of code (canApplyIndexFilter()) will throw NPE when enityType is missing, labels are present. -Thanks. repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java Lines 270 (patched) <https://reviews.apache.org/r/71694/#comment306160> No need to check this/sortBy when entityType is not present. #277 validateAttributes() at SearchContext.java guarantee searchParameter is valide when sortBy is a valid attribute of given enity. Please consider simplify the code here. When entityType is present, if sortBy is present, it is ready for ueing. No extra check is needed. repository/src/main/java/org/apache/atlas/discovery/SearchContext.java Line 259 (original), 259 (patched) <https://reviews.apache.org/r/71694/#comment306158> With this change, entitySearchProcessor can be created when entityType is not present, but inside enitySearchProssor, these are code that has the prerequisite that entityType is present, otherwise, NPE will be thrown. If we are not intending to support label-search without enity type, it is not necessay to add hasLables() here. -Thanks repository/src/main/java/org/apache/atlas/discovery/SearchContext.java Lines 361 (patched) <https://reviews.apache.org/r/71694/#comment306157> Please try to move this function above #262 with other protected/public function. -Thanks repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java Lines 727 (patched) <https://reviews.apache.org/r/71694/#comment306156> - Class attrName is not used in the isValid function. - isValid check if the incoming val is Collection, but later cast it into Set. It might cause exception. - vertexLabels.containsAll(attrLabels) will check inclusion relation, if we intend to check equality? we need to add a size check. If not, please ignore this comment. Thanks. webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java Line 43 (original), 44 (patched) <https://reviews.apache.org/r/71694/#comment306163> - Please consieder add more tests inside TestEntitiesREST.java, especially - when entityType is not present and labels are. - e1.labels: key val, search entity only with val(this depends on the predict, if it is exact match or inclusion) - webapp/src/test/resources/hive-db-50-tables.zip is incldued in this patch, if not needed, please consider removing it. Thanks. - Le Ma On Oct. 29, 2019, 7:24 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71694/ > --- > > (Updated Oct. 29, 2019, 7:24 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer > Shaikh, and Sidharth Mishra. > > > Bugs: ATLAS-3498 > https://issues.apache.org/jira/browse/ATLAS-3498 > > > Repository: atlas > > > Description > --- > > Update basic-search to enable search using labels > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 8f0e5912d > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > 03eb92bcc > > repository/src/main/java/org/apache/atlas/discovery/GraphIndexQueryBuilder.java > 3f58acb9c > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 7ad32bdb9 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 173165a28 > repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java > bb1e9f633 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 908ae7033 > webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java > 808f62354 > webapp/src/test/resources/hive-db-50-tables.zip > fe29f5726e4c180bee42084576d43f0447453be8 > webapp/src/test/resources/json/search-parameters/combination-filters.json > dc52d33d1 > webapp/src/test/resources/json/search-parameters/entity-filters.json > 93d2d7d36 > webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 > > > Diff: https://reviews.apache.org/r/71694/
Re: Review Request 71689: ATLAS-3483 Support adding/deleting labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71689/ --- (Updated Oct. 29, 2019, 9:13 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Pre-commit job succeed. Bugs: ATLAS-3483 https://issues.apache.org/jira/browse/ATLAS-3483 Repository: atlas Description --- The current implementation of label only exposes setLabel endpoint to users. setLabel will replace the existing labels with labels passed in with function. It would be nice to have the enhancement to support add labels to existing label and, delete label. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java a986520a9 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 17b4a5b9d repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java f07cff1a8 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 19d4cef3d repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java d95c12762 webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java e5f7d3339 Diff: https://reviews.apache.org/r/71689/diff/3/ Testing (updated) --- - Added tests, passed. - Pre-commit job succeed. https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1507/ Thanks, Le Ma
Re: Review Request 71689: ATLAS-3483 Support adding/deleting labels
> On Oct. 29, 2019, 10:47 p.m., Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > > Lines 343 (patched) > > <https://reviews.apache.org/r/71689/diff/3/?file=2170797#file2170797line343> > > > > Refactor: invert if Hello Ashutosh, could you please elaborate about how and why we need to intervt if. Thanks - Le --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71689/#review218444 ------- On Oct. 29, 2019, 9:13 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71689/ > --- > > (Updated Oct. 29, 2019, 9:13 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3483 > https://issues.apache.org/jira/browse/ATLAS-3483 > > > Repository: atlas > > > Description > --- > > The current implementation of label only exposes setLabel endpoint to users. > setLabel will replace the existing labels with labels passed in with > function. It would be nice to have the enhancement to support add labels to > existing label and, delete label. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > a986520a9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17b4a5b9d > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > f07cff1a8 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 19d4cef3d > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java > d95c12762 > webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java e5f7d3339 > > > Diff: https://reviews.apache.org/r/71689/diff/3/ > > > Testing > --- > > - Added tests, passed. > - Pre-commit job succeed. > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1507/ > > > Thanks, > > Le Ma > >
Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 server-api/src/main/java/org/apache/atlas/listener/EntityChangeListener.java e24f582c8 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/1/ Testing --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Will run pre-commit job. File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Re: Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- (Updated Nov. 4, 2019, 6:58 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Resolve comments Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs (updated) - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 server-api/src/main/java/org/apache/atlas/listener/EntityChangeListener.java e24f582c8 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/2/ Changes: https://reviews.apache.org/r/71710/diff/1-2/ Testing --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Will run pre-commit job. File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Re: Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- (Updated Nov. 4, 2019, 8:57 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Resolve comments Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs (updated) - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/3/ Changes: https://reviews.apache.org/r/71710/diff/2-3/ Testing --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Will run pre-commit job. File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Re: Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- (Updated Nov. 5, 2019, 12:21 a.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Include front-end change Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs (updated) - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 dashboardv2/public/js/utils/Enums.js c7316d542 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/5/ Changes: https://reviews.apache.org/r/71710/diff/4-5/ Testing --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Will run pre-commit job. File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Re: Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- (Updated Nov. 5, 2019, 12:26 a.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- include ui change again Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs (updated) - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 dashboardv2/public/js/utils/Enums.js c7316d542 dashboardv3/public/js/utils/Enums.js c7316d542 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/6/ Changes: https://reviews.apache.org/r/71710/diff/5-6/ Testing --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Will run pre-commit job. File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Re: Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- (Updated Nov. 5, 2019, 7:11 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Resolve comments Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs (updated) - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 dashboardv2/public/js/utils/Enums.js c7316d542 dashboardv3/public/js/utils/Enums.js c7316d542 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 173165a28 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java aba988d25 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/7/ Changes: https://reviews.apache.org/r/71710/diff/6-7/ Testing --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Will run pre-commit job. File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Re: Review Request 71710: ATLAS-3497 Add audit entry for adding/removing labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71710/ --- (Updated Nov. 5, 2019, 9:40 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Pre-commit job succeed. Bugs: ATLAS-3497 https://issues.apache.org/jira/browse/ATLAS-3497 Repository: atlas Description --- https://issues.apache.org/jira/browse/ATLAS-3483 added adding/removing labels, it would be good to add audit entry for these operations: classification added / classification deleted. Diffs - client/client-v1/src/main/java/org/apache/atlas/EntityAuditEvent.java fcd6a62b2 dashboardv2/public/js/utils/Enums.js c7316d542 dashboardv3/public/js/utils/Enums.js c7316d542 intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 106c7978d intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 649f11f71 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 20624da54 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 1f0cc8626 repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 173165a28 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java c910d9eeb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2c2fc59b9 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java aba988d25 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java b2211fcaa Diff: https://reviews.apache.org/r/71710/diff/7/ Testing (updated) --- - Manual testing on cluster. 1. labels "fordelete" "frompostman" are added by REST call - /addLabels PUT via Postman. screenshot1. audits are as expected. see pic2. 2. delete label "fordelete" by REST call - /removeLabels DELETE via Postman. audits are as expected. see pic2. - Pre-commit job succeed. https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1527/ File Attachments Screen Shot 2019-11-01 at 1.38.22 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/a38995b5-5bf9-4e2a-a7f0-884409c312cf__Screen_Shot_2019-11-01_at_1.38.22_PM.png Screen Shot 2019-11-01 at 1.39.37 PM.png https://reviews.apache.org/media/uploaded/files/2019/11/01/987ae5aa-39f6-4d61-87f9-87d2d4439dac__Screen_Shot_2019-11-01_at_1.39.37_PM.png Thanks, Le Ma
Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3482 https://issues.apache.org/jira/browse/ATLAS-3482 Repository: atlas Description --- Introduce ALL_ENTITY_TYPE to support search on system attributes across all entity types. System attributes will be passed in as normal entity attributes. Diffs - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 8f0e5912d intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java ed1e5ded2 intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 417194202 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 03eb92bcc repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 152ade8d4 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java b56d8e83a webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71736/diff/1/ Testing --- - Added new unit tests, passed. - will run pre-commit job. Thanks, Le Ma
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/ --- (Updated Nov. 9, 2019, 1:12 a.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Update Patch Bugs: ATLAS-3482 https://issues.apache.org/jira/browse/ATLAS-3482 Repository: atlas Description --- Introduce ALL_ENTITY_TYPE to support search on system attributes across all entity types. System attributes will be passed in as normal entity attributes. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 8f0e5912d intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java e10965b87 intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java ed1e5ded2 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 03eb92bcc repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 152ade8d4 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 7c551304b webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71736/diff/2/ Changes: https://reviews.apache.org/r/71736/diff/1-2/ Testing --- - Added new unit tests, passed. - will run pre-commit job. Thanks, Le Ma
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/ --- (Updated Nov. 14, 2019, 11:57 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Resolve comments. Bugs: ATLAS-3482 https://issues.apache.org/jira/browse/ATLAS-3482 Repository: atlas Description --- Introduce ALL_ENTITY_TYPE to support search on system attributes across all entity types. System attributes will be passed in as normal entity attributes. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 8f0e5912d intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java e10965b87 intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java ed1e5ded2 intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 417194202 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 03eb92bcc repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 152ade8d4 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java b56d8e83a repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 7c551304b webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71736/diff/3/ Changes: https://reviews.apache.org/r/71736/diff/2-3/ Testing --- - Added new unit tests, passed. - will run pre-commit job. Thanks, Le Ma
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/ --- (Updated Nov. 18, 2019, 6:27 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Resolve comments Bugs: ATLAS-3482 https://issues.apache.org/jira/browse/ATLAS-3482 Repository: atlas Description --- Introduce ALL_ENTITY_TYPE to support search on system attributes across all entity types. System attributes will be passed in as normal entity attributes. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 8f0e5912d intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java e10965b87 intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 417194202 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 8b4fd1c3b intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION intg/src/test/java/org/apache/atlas/TestUtilsV2.java 530d5cda4 intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java d3f4b57ad intg/src/test/java/org/apache/atlas/model/ModelTestUtil.java 5df952546 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 742970390 intg/src/test/java/org/apache/atlas/type/TestAtlasRelationshipType.java 107539598 intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java 476bc3300 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 03eb92bcc repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 152ade8d4 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java b56d8e83a webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71736/diff/4/ Changes: https://reviews.apache.org/r/71736/diff/3-4/ Testing --- - Added new unit tests, passed. - will run pre-commit job. Thanks, Le Ma
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/ --- (Updated Nov. 18, 2019, 10:30 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Pre-commit job succeeds. Bugs: ATLAS-3482 https://issues.apache.org/jira/browse/ATLAS-3482 Repository: atlas Description --- Introduce ALL_ENTITY_TYPE to support search on system attributes across all entity types. System attributes will be passed in as normal entity attributes. Diffs - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 8f0e5912d intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java e10965b87 intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 417194202 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 8b4fd1c3b intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION intg/src/test/java/org/apache/atlas/TestUtilsV2.java 530d5cda4 intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java d3f4b57ad intg/src/test/java/org/apache/atlas/model/ModelTestUtil.java 5df952546 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 742970390 intg/src/test/java/org/apache/atlas/type/TestAtlasRelationshipType.java 107539598 intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java 476bc3300 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 03eb92bcc repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 152ade8d4 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java b56d8e83a webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71736/diff/4/ Testing (updated) --- - Added new unit tests, passed. - pre-commit job succeeds. https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1540/ Thanks, Le Ma
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/ --- (Updated Nov. 19, 2019, 8:04 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan Neethiraj, and Sarath Subramanian. Changes --- Resolve comments Bugs: ATLAS-3482 https://issues.apache.org/jira/browse/ATLAS-3482 Repository: atlas Description --- Introduce ALL_ENTITY_TYPE to support search on system attributes across all entity types. System attributes will be passed in as normal entity attributes. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 8f0e5912d intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java e10965b87 intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java 417194202 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 8b4fd1c3b intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION intg/src/test/java/org/apache/atlas/TestUtilsV2.java 530d5cda4 intg/src/test/java/org/apache/atlas/type/TestAtlasRelationshipType.java 107539598 intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java 476bc3300 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 672f38132 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 03eb92bcc repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 152ade8d4 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 7ad32bdb9 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java b56d8e83a webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java cd8f8981c Diff: https://reviews.apache.org/r/71736/diff/5/ Changes: https://reviews.apache.org/r/71736/diff/4-5/ Testing --- - Added new unit tests, passed. - pre-commit job succeeds. https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1540/ Thanks, Le Ma
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
> On Nov. 19, 2019, 9:20 a.m., Sarath Subramanian wrote: > > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > > Line 59 (original), 61 (patched) > > <https://reviews.apache.org/r/71736/diff/4/?file=2173823#file2173823line61> > > > > avoid unnecessary whitespace/newline changes > > > > line 61-75 > > line 83-90 Hello Sarath, thanks for reviewing the code. I've reverted the changes for #83 - #90. The change for line 61 - 75 is to make variables with same scope together for readability. I suggest we keep it. Thanks. - Le --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/#review218667 ------- On Nov. 19, 2019, 8:04 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71736/ > --- > > (Updated Nov. 19, 2019, 8:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3482 > https://issues.apache.org/jira/browse/ATLAS-3482 > > > Repository: atlas > > > Description > --- > > Introduce ALL_ENTITY_TYPE to support search on system attributes across all > entity types. System attributes will be passed in as normal entity attributes. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 8f0e5912d > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > e10965b87 > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 417194202 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 8b4fd1c3b > intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 530d5cda4 > intg/src/test/java/org/apache/atlas/type/TestAtlasRelationshipType.java > 107539598 > intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java > 476bc3300 > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 672f38132 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > 03eb92bcc > > repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java > 152ade8d4 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 7ad32bdb9 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > b56d8e83a > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b > webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java > cd8f8981c > > > Diff: https://reviews.apache.org/r/71736/diff/5/ > > > Testing > --- > > - Added new unit tests, passed. > - pre-commit job succeeds. > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1540/ > > > Thanks, > > Le Ma > >
Re: Review Request 71796: ATLAS-3530: Retrieve entityDefinition for '_ALL_ENTITY_TYPES' in /api/atlas/v2/types/entitydef/name/{name}
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71796/#review218739 --- Ship it! Ship It! - Le Ma On None, Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71796/ > --- > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer > Shaikh, and Sidharth Mishra. > > > Bugs: ATLAS-3530 > https://issues.apache.org/jira/browse/ATLAS-3530 > > > Repository: atlas > > > Description > --- > > '_ALL_ENTITY_TYPES' is a new system type with all system attributes of an > entity in ATLAS-3482. > > A new TypesREST endpoint to retrieve definition of '_ALL_ENTITY_TYPES' is > necessary to render system attributes in UI. > > GET /api/atlas/v2/types/entitydef/name/ALL_ENTITY_TYPES should return > entityDef. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 557ef74a9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java > 2e2ab1a66 > > repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java > 51dd16b85 > > > Diff: https://reviews.apache.org/r/71796/diff/1/ > > > Testing > --- > > Precommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/ > > Added unit test to check if 'ALL_ENTITY_TYPES' definition is returned > correctly. > > > Thanks, > > Sarath Subramanian > >
Review Request 71980: ATLAS-3580 Search on __customAttributes doesn't return right results.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71980/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3580 https://issues.apache.org/jira/browse/ATLAS-3580 Repository: atlas Description --- Search on __customAttributes doesn't return right results. Diffs - repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 76d5a017d webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 Diff: https://reviews.apache.org/r/71980/diff/1/ Testing --- - added unit tests, works. Thanks, Le Ma
Re: Review Request 71980: ATLAS-3580 Search on __customAttributes doesn't return right results.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71980/ --- (Updated Jan. 9, 2020, 9:03 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Madhan Neethiraj, and Sarath Subramanian. Changes --- Add description. Bugs: ATLAS-3580 https://issues.apache.org/jira/browse/ATLAS-3580 Repository: atlas Description (updated) --- Search on __customAttributes doesn't return right results. when the value of the customAttributes contains special character, like ":", the in-memory filter doesn't work for pipeSeparated attributes, like classificationNames, customAttributes. The results will be the results that are not filtered by the attributes. It is a super set of the desired results. Diffs - repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 76d5a017d webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java afc4c50d7 Diff: https://reviews.apache.org/r/71980/diff/1/ Testing --- - added unit tests, works. Thanks, Le Ma
[jira] [Assigned] (ATLAS-989) Modify import-hive to use entity state
[ https://issues.apache.org/jira/browse/ATLAS-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma reassigned ATLAS-989: --- Assignee: Le Ma > Modify import-hive to use entity state > -- > > Key: ATLAS-989 > URL: https://issues.apache.org/jira/browse/ATLAS-989 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Le Ma >Priority: Critical > > Import hive currently uses DSL queries to check if entity exists and DSL > query doesn't use entity state. This will not work with soft deletes in which > case DSL returns deleted entity as well. Instead of DSl query, import hive > can use get entity by unique attribute which filters deleted entities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma reassigned ATLAS-2933:
Assignee: Le Ma
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ATLAS-2928) Namespace entity relationship-attribute names to avoid conflict with entity attributes
[ https://issues.apache.org/jira/browse/ATLAS-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma reassigned ATLAS-2928: Assignee: Le Ma > Namespace entity relationship-attribute names to avoid conflict with entity > attributes > -- > > Key: ATLAS-2928 > URL: https://issues.apache.org/jira/browse/ATLAS-2928 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.0.0, 1.1.0 >Reporter: Madhan Neethiraj >Assignee: Le Ma >Priority: Critical > Fix For: 1.2.0, 2.0.0 > > > Apache Atlas 1.0 introduced relationships as first-class types. A > relationship effectively 'injects' an attribute on the entities at each end. > For example, assignment of a glossary term to an entity is captured by > relationship AtlasGlossarySemanticAssignment (in > -Area0/0011-glossary_model.json). This relationship adds following > attributes: > - Referceable.meanings: set > - AtlasGlossaryTerm.assignedEntities: set > This works fine as long as the entity-types involved in the relationship, or > their sub-types don't have an attribute with the same name as the attribute > injected by the relationships. Once such instance was reported by [~bolke], > about relationship attribute DataSet.schema introduced by > avro_schema_associatedEntities - which conflicts with spark_table.schema, > since spark_table type is a sub-type of DataSet. > Such name conflicts can be avoided by prefixing relationship-attribute names > with a prefix like "r:". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ATLAS-989) Modify import-hive to use entity state
[ https://issues.apache.org/jira/browse/ATLAS-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma resolved ATLAS-989. - Resolution: Invalid This Jira is invalid because Atlas has already used attributes to find entities by calling getEntityByAttribute() before creating a new one, which goes through api/atlas/v2/entity/uniqueAttribute/type/. Thanks. > Modify import-hive to use entity state > -- > > Key: ATLAS-989 > URL: https://issues.apache.org/jira/browse/ATLAS-989 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Le Ma >Priority: Critical > > Import hive currently uses DSL queries to check if entity exists and DSL > query doesn't use entity state. This will not work with soft deletes in which > case DSL returns deleted entity as well. Instead of DSl query, import hive > can use get entity by unique attribute which filters deleted entities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: ATLAS-2933.patch
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
Le Ma created ATLAS-3071:
Summary: Add Functionalities to Collect Notification
Metrics/Entity Lifecyle
Key: ATLAS-3071
URL: https://issues.apache.org/jira/browse/ATLAS-3071
Project: Atlas
Issue Type: New Feature
Reporter: Le Ma
Assignee: Le Ma
Extend api/atlas/admin/metrics to provide metrics for notifications/entity
lifycycle:
# notification
processed
# notification failed
today/thisHour/pastHour/total { # entity created
# entity updated
# entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: (was: ATLAS-2933.patch)
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: ATLAS-2933.patch
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: ATLAS-2933.patch
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: (was: ATLAS-2933.patch)
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801062#comment-16801062
]
Le Ma commented on ATLAS-2933:
--
Link for pre-commit build -
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/997/
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: (was: ATLAS-2933.patch)
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-2933) Empty array attributes are returned as null instead of an empty list
[
https://issues.apache.org/jira/browse/ATLAS-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-2933:
-
Attachment: ATLAS-2933.patch
> Empty array attributes are returned as null instead of an empty list
>
>
> Key: ATLAS-2933
> URL: https://issues.apache.org/jira/browse/ATLAS-2933
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Cade Parker
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-2933.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
>
> Given the following entity def:
> {code:java}
> {
> "entityDefs": [
>{
> "name": "ExampleType",
> "superTypes": [],
> "typeVersion": "0.1",
> "attributeDefs": [
>{
> "name": "things",
> "typeName": "array",
> "cardinality": "SINGLE",
> "isIndexable": false,
> "isOptional": false,
> "isUnique": false
>}
> ]
>}
> ]
> }
> {code}
> I can POST a new entity with an empty array for the things attribute
> successfully:
> {code:java}
> POST /api/atlas/v2/entity
> {
> "entity":{
> "typeName":"ExampleType",
> "attributes":{
>"things": []
> },
> "guid":"-thing",
> "version":1
> }
> }
> {code}
> But when i GET the entity by guid, the attribute is null instead of the empty
> list:
> {code:java}
> GET /api/atlas/v2/entity/guid/d5e33187-3293-4206-bcb8-807dc87c1440
> {
> "referredEntities": {},
> "entity": {
> "typeName": "ExampleType",
> "attributes": {
>"things": null
> },
> "guid": "d5e33187-3293-4206-bcb8-807dc87c1440",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1540335741854,
> "updateTime": 1540335741854,
> "version": 1
> }
> }
> {code}
> Atlas should preserve the original value of the create request as an empty
> list. Returning null here causes further confusion, because the attribute is
> defined with `isOptional: false` so any consumer of the entity would
> reasonably expect a non-null value.
> I tested this behavior against Atlas versions 1.0.0 and 1.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071
>
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: (was: ATLAS-3071)
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804368#comment-16804368
]
Le Ma commented on ATLAS-3071:
--
Pre commit link :
[https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1005/]
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
>Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: MetricsDataModel.json
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071.patch
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071.patch
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: (was: ATLAS-3071.patch)
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: (was: ATLAS-3071.patch)
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071.patch
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071.patch
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: (was: ATLAS-3071.patch)
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
> Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: ATLAS-3114.patch > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: ATLAS-3114.patch, AtlasClientV2Test.java, Error.txt, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819439#comment-16819439 ] Le Ma commented on ATLAS-3114: -- [~ayushmnnit] Hello, I've modified the attached java. It works on my side. Instance/db/table/col are created properly. I ran end2end tests multiple times and I did not run into the issue mentioned here except " mandatory attribute value missing ... " My testing in on Atlas master branch, using my local models. Changes included: - Added mandatory attr for table and col during creation. - some misc changes. Please let us know if you have other questions. Thanks, Le > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: ATLAS-3114.patch, AtlasClientV2Test.java, Error.txt, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826553#comment-16826553 ] Le Ma commented on ATLAS-3114: -- Hello [~sreenivasulu_nallap...@intuit.com] Sorry for late reply. I am using Atlas built-in rdbms model (from master branch). You could check the diff between your model and the default model to see if that's the reason causing your issue. If not, i would recommend you to try the way that i used in the patch file to see if it works. Thanks, Le > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: ATLAS-3114.patch, AtlasClientV2Test.java, Error.txt, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: 2010-rdbms_model.json > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma resolved ATLAS-3114. -- Resolution: Fixed > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834209#comment-16834209 ] Le Ma commented on ATLAS-3114: -- Hello [~sreenivasulu_nallap...@intuit.com] Attached is the model that i used. Please take a look. I am going to close this item. But if you have further questions, please feel free to comment, i am happy to help :) Thanks, Le > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3197) Add QueryText for hive_table and hive_db
Le Ma created ATLAS-3197: Summary: Add QueryText for hive_table and hive_db Key: ATLAS-3197 URL: https://issues.apache.org/jira/browse/ATLAS-3197 Project: Atlas Issue Type: New Feature Reporter: Le Ma Assignee: Le Ma -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: ATLAS-3197.patch > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: ATLAS-3197.patch > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: (was: ATLAS-3197.patch) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Description: This patch introduced a model called Hive_table_ddl and hive_db_ddl which are used to track commands that are executed either on a table entity or bd entity. DDL entities will be added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity will be created, it will append to the existing ddlQueries field if a table/db has already been created. Once a table/db is deleted, all these relationship attributes will be deleted as well. (was: This patch introduced a model called Hive_table_ddl and hive_db_ddl which are used to track commands that are executed either on a table entity or bd entity. DDL entities will be added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity will be created, it will append to the existing DDLQueries field if a table/db has already been created. Once a table/db is deleted, all these relationship attribute will be deleted as well.) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - External issue URL: https://reviews.apache.org/r/70605/ > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Description: This patch introduced a model called Hive_table_ddl and hive_db_ddl which are used to track commands that are executed either on a table entity or bd entity. DDL entities will be added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity will be created, it will append to the existing DDLQueries field if a table/db has already been created. Once a table/db is deleted, all these relationship attribute will be deleted as well. > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing DDLQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attribute will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: (was: ATLAS-3197.patch) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: (was: ATLAS-3197.patch) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: Screen Shot 2019-05-07 at 4.32.26 PM.png Screen Shot 2019-05-07 at 4.32.00 PM.png Screen Shot 2019-05-07 at 4.31.38 PM.png Screen Shot 2019-05-07 at 4.31.26 PM.png > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, Screen Shot 2019-05-07 at 4.31.26 PM.png, > Screen Shot 2019-05-07 at 4.31.38 PM.png, Screen Shot 2019-05-07 at 4.32.00 > PM.png, Screen Shot 2019-05-07 at 4.32.26 PM.png, > image-2019-05-07-09-28-21-892.png, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835195#comment-16835195 ] Le Ma commented on ATLAS-3114: -- Hello [~sreenivasulu_nallap...@intuit.com] , Thanks for the update. I've attached the screenshots I took after I ran the attached patch file. I will re-upload the model I am using again. Thanks, Le > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, image-2019-05-07-09-28-21-892.png, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: (was: Screen Shot 2019-05-07 at 4.32.00 PM.png) > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, image-2019-05-07-09-28-21-892.png, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: (was: Screen Shot 2019-05-07 at 4.32.26 PM.png) > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, image-2019-05-07-09-28-21-892.png, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: (was: Screen Shot 2019-05-07 at 4.31.26 PM.png) > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, image-2019-05-07-09-28-21-892.png, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: (was: Screen Shot 2019-05-07 at 4.31.38 PM.png) > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, image-2019-05-07-09-28-21-892.png, > model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: Screen Shot 2019-05-07 at 4.32.26 PM.png Screen Shot 2019-05-07 at 4.32.00 PM.png Screen Shot 2019-05-07 at 4.31.38 PM.png > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, Screen Shot 2019-05-07 at 4.31.38 PM.png, > Screen Shot 2019-05-07 at 4.32.00 PM.png, Screen Shot 2019-05-07 at 4.32.26 > PM.png, image-2019-05-07-09-28-21-892.png, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: 2010-rdbms_model.json > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, Screen Shot 2019-05-07 at 4.31.38 PM.png, > Screen Shot 2019-05-07 at 4.32.00 PM.png, Screen Shot 2019-05-07 at 4.32.26 > PM.png, image-2019-05-07-09-28-21-892.png, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3114) Issue with concurrent bulk inserts for entities
[ https://issues.apache.org/jira/browse/ATLAS-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3114: - Attachment: (was: 2010-rdbms_model.json) > Issue with concurrent bulk inserts for entities > --- > > Key: ATLAS-3114 > URL: https://issues.apache.org/jira/browse/ATLAS-3114 > Project: Atlas > Issue Type: Bug >Reporter: Ayush Nigam >Assignee: chaitali borole >Priority: Major > Attachments: 2010-rdbms_model.json, ATLAS-3114.patch, > AtlasClientV2Test.java, Error.txt, Screen Shot 2019-05-07 at 4.31.38 PM.png, > Screen Shot 2019-05-07 at 4.32.00 PM.png, Screen Shot 2019-05-07 at 4.32.26 > PM.png, image-2019-05-07-09-28-21-892.png, model.json > > > We have a model with tables having attribute 'columns' in which we are > attaching list of object ids for all columns once these are created. We are > using clientV2 java APIs. > We are doing bulk operation for columns and parallelizing the tables. > Sometimes the issue is that bulk creation for columns is successful,i.e. > atlas don't throw any exception but we get some columns as created,some as > updated,whereas as none of the columns existed before.Even it misses out some > entities while creating.Some are created and some are just silently missed > without throwing an exception. > So to sum up issue is there for concurrent bulk create/update calls.It works > for concurrent single entity create/update calls. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: (was: ATLAS-3197.patch) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: (was: ATLAS-3197.patch) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: trunk > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: ATLAS-3197.patch > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: trunk > > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: ATLAS-3197.patch > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: trunk > > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: (was: ATLAS-3197.patch) > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: trunk > > Attachments: ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Le Ma updated ATLAS-3197: - Attachment: ATLAS-3197-2.patch > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature > Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: trunk > > Attachments: ATLAS-3197-2.patch, ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3197) Add QueryText for hive_table and hive_db
[ https://issues.apache.org/jira/browse/ATLAS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836785#comment-16836785 ] Le Ma commented on ATLAS-3197: -- ATLAS-3197-2.patch is the enhancement patch from what is committed in the code base. It includes handling CTAS table creation and corresponding tests. A minor change in the committed test added since last commit. > Add QueryText for hive_table and hive_db > > > Key: ATLAS-3197 > URL: https://issues.apache.org/jira/browse/ATLAS-3197 > Project: Atlas > Issue Type: New Feature >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: trunk > > Attachments: ATLAS-3197-2.patch, ATLAS-3197.patch > > > This patch introduced a model called Hive_table_ddl and hive_db_ddl which are > used to track commands that are executed either on a table entity or bd > entity. DDL entities will be added to a table/db entity through > relationshipAttribute call HIVE_DB_TO_DDL and hive_tb_to_ddl. Every time when > a new command is executed, a new ddl entity will be created, it will append > to the existing ddlQueries field if a table/db has already been created. Once > a table/db is deleted, all these relationship attributes will be deleted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-3204) HMS hook fails to process create_table event due to ATLAS-3197
[
https://issues.apache.org/jira/browse/ATLAS-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma reassigned ATLAS-3204:
Assignee: Le Ma (was: Sarath Subramanian)
> HMS hook fails to process create_table event due to ATLAS-3197
> --
>
> Key: ATLAS-3204
> URL: https://issues.apache.org/jira/browse/ATLAS-3204
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3204.001.patch
>
>
> ATLAS-3197 creates Hive DDL entity even in HMS hook when processing
> create_table event, where the HiveContext is null. That causes
> NullPointerException.
> The fix should generate AtlasEntity based on its in HiveHook or HMS hook.
> {code}
> 2019-05-09 23:15:37,936 ERROR - [main:] ~
> HiveMetastoreHook.handleEvent(org.apache.hadoop.hive.metastore.events.CreateTableEvent@1850f2da):
> failed to process operation {} (HiveMetastoreHookImpl$HiveMetastoreHook:169)
> java.lang.NullPointerException
> at
> org.apache.atlas.hive.hook.events.BaseHiveEvent.createHiveDDLEntity(BaseHiveEvent.java:692)
> at
> org.apache.atlas.hive.hook.events.BaseHiveEvent.createHiveDDLEntity(BaseHiveEvent.java:671)
> at
> org.apache.atlas.hive.hook.events.CreateTable.processTable(CreateTable.java:153)
> at
> org.apache.atlas.hive.hook.events.CreateTable.getHiveMetastoreEntities(CreateTable.java:77)
> at
> org.apache.atlas.hive.hook.events.CreateTable.getNotificationMessages(CreateTable.java:52)
> at
> org.apache.atlas.hive.hook.HiveMetastoreHookImpl$HiveMetastoreHook.handleEvent(HiveMetastoreHookImpl.java:166)
> at
> org.apache.atlas.hive.hook.HiveMetastoreHookImpl.onCreateTable(HiveMetastoreHookImpl.java:77)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$20.notify(MetaStoreListenerNotifier.java:92)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:267)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:329)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:367)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1999)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.create_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2867)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:121)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:837)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:822)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy29.createTable(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:921)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937)
> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954)
> at org.apache.hadoop.hive.ql.ex
[jira] [Created] (ATLAS-3207) Run pre-commit job once a patch is submitted on jira
Le Ma created ATLAS-3207: Summary: Run pre-commit job once a patch is submitted on jira Key: ATLAS-3207 URL: https://issues.apache.org/jira/browse/ATLAS-3207 Project: Atlas Issue Type: New Feature Reporter: Le Ma It would be nice to have the automation that could run pre-commit job once a patch is submitted to Jira. By adding this, it could be easier for contributors to validate their patch and once is this part of committing process, we could commit more bug-free code. I've heard that Hive use this approach for commit. Thankd -- This message was sent by Atlassian JIRA (v7.6.3#76005)
