[jira] [Created] (ATLAS-4502) Add cassandra and elasticsearch search profile for Atlas
Sidharth Kumar Mishra created ATLAS-4502: Summary: Add cassandra and elasticsearch search profile for Atlas Key: ATLAS-4502 URL: https://issues.apache.org/jira/browse/ATLAS-4502 Project: Atlas Issue Type: New Feature Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (ATLAS-4492) Atlas to skip external temporary table created in hive
[
https://issues.apache.org/jira/browse/ATLAS-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharth Kumar Mishra resolved ATLAS-4492.
--
Resolution: Fixed
> Atlas to skip external temporary table created in hive
> --
>
> Key: ATLAS-4492
> URL: https://issues.apache.org/jira/browse/ATLAS-4492
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
>
> As per the current design, skip temp tables feature in Atlas will skip
> creating temporary tables only if that temp table is {*}not external{*}. *New
> requirement is to skip creating external temp tables also.*
> Introducing new config property *skip.all.temp.tables* which can be used to
> skip all temp tables {*}irrespective of external or not{*}.
> With this fix in Atlas two options will be available.
> atlas.hook.hive.skip.temp.tables => which works same as earlier, skipping
> non-external temporary table creation and it is enabled in Atlas by default.
> atlas.hook.hive.skip.all.temp.tables => which skips all temporary tables
> irrespective of external or not
> To skip all the temporary tables user has to configure the property as below.
> atlas.hook.hive.skip.all.temp.tables=true
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (ATLAS-4508) Though a particular tag is blocked propagation , tag is still propagated
Sidharth Kumar Mishra created ATLAS-4508:
Summary: Though a particular tag is blocked propagation , tag is
still propagated
Key: ATLAS-4508
URL: https://issues.apache.org/jira/browse/ATLAS-4508
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Sidharth Kumar Mishra
Assignee: Sidharth Kumar Mishra
If we do the operation like Add classification and remove concurrently or Add
classification and block the same classification concurrently then most likely
due to deferred action we are hitting issue at Atlas and the Add/Remove/Block
is not working as expected. Example of Add and block happening concurrently
where the block is not working as expected:
{code:java}
2021-12-12 10:59:36,145 INFO -
[etp1330247343-177:hrt_qa:POST/api/atlas/v2/entity/bulk/classification] ~
Request from authenticated user: hrt_qa,
URL=/api/atlas/v2/entity/bulk/classification
(AtlasAuthenticationFilter$KerberosFilterChainWrapper:800)
2021-12-12 10:59:36,170 INFO - [etp1330247343-177 -
15c26538-375e-4352-b27e-34caf1d4a6f7:hrt_qa:POST/api/atlas/v2/entity/bulk/classification]
~ updateClassificationText: tag_zxpzn: tag_emufo_0 tag_qxnez tag_crqpn_0
tag_xldqo tag_xjdig tag_ksgxh tag_dpzcq tag_vszhd tag_zxpzn tag_crqpn_1
tag_crqpn_2 tag_hzjqk tag_tafen date Sun Dec 12 10:55:20 UTC 2021
hive_principal_type USER boolean false string nJzsc byte -21 double
1.2633179028830385E38 arr_string dYfsL mKlVp iCOwp ssfum iAaGF kcJWp dWxpm
VFsqP hYbVs short 24586 arr_int 476972899 729046663 1927464331 499332336
233258034 -670732418 -1304702956 205121029 float 1.54E38 int 1233345212 long
7716583360736583680 tag_ywwov date Thu Jan 01 00:00:00 UTC 1970 boolean false
byte 0 double 0.0 short 0 float 0.0 int 0 long 0 (EntityGraphMapper:2769)
2021-12-12 10:59:36,325 INFO - [etp1330247343-177 -
15c26538-375e-4352-b27e-34caf1d4a6f7:hrt_qa:POST/api/atlas/v2/entity/bulk/classification]
~
{"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"32961799-d7e5-4aa1-a7b1-9172920f430b","createdBy":"hrt_qa","createdTime":1639306776164,"updatedTime":1639306776164,"parameters":{"relationshipGuid":null,"entityGuid":"b20848ce-4e8d-41d9-abdb-d0a846166e9e","classificationVertexId":"40972512"},"attemptCount":0,"status":"PENDING"}
(TaskExecutor$TaskLogger:170)
2021-12-12 10:59:36,368 INFO - [etp1330247343-570:] ~ Logged into Atlas as =
hrt_qa (AtlasAuthenticationFilter$KerberosFilterChainWrapper:787)
.
2021-12-12 10:59:37,068 INFO - [etp1330247343-570 -
1b14bb3f-face-40f3-b4a4-cbac281ed685:hrt_qa:PUT/api/atlas/v2/relationship] ~
{"type":"CLASSIFICATION_PROPAGATION_RELATIONSHIP_UPDATE","guid":"9b36fa74-46e8-4bd4-9abc-2865354d0d46","createdBy":"hrt_qa","createdTime":1639306776679,"updatedTime":1639306776679,"parameters":{"relationshipEdgeId":"osjyc-okbog-4e1h-3v7k","relationshipObject":"{\"typeName\":\"dataset_process_inputs\",\"guid\":\"a2f03e60-d677-473b-ada5-dd77b21dc149\",\"provenanceType\":0,\"end1\":{\"guid\":\"3e9b2e57-1e32-4692-ac7a-a617ea0c4648\",\"typeName\":\"hive_process\",\"uniqueAttributes\":{\"qualifiedName\":\"default.table_xluwf_2@cm:1639305904000\"}},\"end2\":{\"guid\":\"b20848ce-4e8d-41d9-abdb-d0a846166e9e\",\"typeName\":\"hive_table\",\"uniqueAttributes\":{\"qualifiedName\":\"default.table_xluwf_1@cm\"}},\"label\":\"__Process.inputs\",\"propagateTags\":\"TWO_TO_ONE\",\"status\":\"ACTIVE\",\"createdBy\":\"hrt_qa\",\"updatedBy\":\"hrt_qa\",\"createTime\":1639305904397,\"updateTime\":1639305904397,\"version\":0,\"propagatedClassifications\":[{\"typeName\":\"tag_qxnez\",\"entityGuid\":\"b20848ce-4e8d-41d9-abdb-d0a846166e9e\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"tag_crqpn_1\",\"entityGuid\":\"b20848ce-4e8d-41d9-abdb-d0a846166e9e\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"tag_ksgxh\",\"entityGuid\":\"b20848ce-4e8d-41d9-abdb-d0a846166e9e\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"tag_tafen\",\"attributes\":{\"short\":24586,\"string\":\"nJzsc\",\"boolean\":false,\"double\":1.2633179028830385E+38,\"float\":1.54E+38,\"arr_string\":[\"dYfsL\",\"mKlVp\",\"iCOwp\",\"ssfum\",\"iAaGF\",\"kcJWp\",\"dWxpm\",\"VFsqP\",\"hYbVs\"],\"long\":7716583360736583680,\"arr_int\":[476972899,729046663,1927464331,499332336,233258034,-670732418,-1304702956,205121029],\"int\":1233345212,\"hive_principal_type\":\"USER\",\"date\":1639306520089,\"byte\":-21},\"entityGuid\":\"b20848ce-4e8d-41d9-abdb-d0a846166e9e\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"tag_crqpn_0\",\"entityGuid\":\"b20848ce-4e8d-41d9-abdb-d0a846166e9e\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"tag_ywwo
[jira] [Assigned] (ATLAS-3985) Not able to assign classification with mandatory attribute of type "array"
[
https://issues.apache.org/jira/browse/ATLAS-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharth Kumar Mishra reassigned ATLAS-3985:
Assignee: Sidharth Kumar Mishra
> Not able to assign classification with mandatory attribute of type
> "array"
> ---
>
> Key: ATLAS-3985
> URL: https://issues.apache.org/jira/browse/ATLAS-3985
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-webui
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: Screenshot 2020-10-07 at 1.40.43 PM.png, Screenshot
> 2020-10-07 at 1.41.17 PM.png
>
>
> Create a classification that has mandatory attribute of type
> "*array*" via REST api
> {code:java|title=Create classification with payload:}
> {
> "enumDefs": [],
> "structDefs": [],
> "classificationDefs": [
> {
> "category": "CLASSIFICATION",
> "guid": "8996315b-b5fe-48c6-b9d0-4668fhkfd21af",
> "createdBy": "hrt_qa",
> "updatedBy": "hrt_qa",
> "createTime": 1602057743077,
> "updateTime": 1602057743077,
> "version": 1,
> "name": "read_auth_test_tag1",
> "description": "read_auth_test_tag1",
> "typeVersion": "1.0",
> "attributeDefs": [
> {
> "name": "type_long_min",
> "typeName": "long",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": false,
> "isIndexable": false
> },
> {
> "name":"type_arr_list",
> "typeName":"array",
> "isOptional":false,
> "cardinality":"LIST",
> "valuesMinCount":1,
> "valuesMaxCount":2147483647,
> "isUnique":false,
> "isIndexable":false
> },
> {
> "name":"type_set",
> "typeName":"array",
> "isOptional":false,
> "cardinality":"SET",
> "valuesMinCount":1,
> "valuesMaxCount":2147483647,
> "isUnique":false,
> "isIndexable":false
> }
> ],
> "superTypes": [],
> "entityTypes": [],
> "subTypes": []
> }
> ],
> "entityDefs": [],
> "relationshipDefs": [],
> "businessMetadataDefs": []
> }
> {code}
> Assign the same to an entity.
> The assignment fails as the attributes of type "array" are not
> visible on UI
> {code:java|title=error message}
> {"errorCode":"ATLAS-400-00-01A",
> "errorMessage":"invalid parameters: read_auth_test_tag1.type_arr_list:
> mandatory attribute value missing in type read_auth_test_tag1"}
> {code}
>
> The assignment fails via both UI and rest api.
> In UI creation of attribute of such a type is not allowed but via rest api it
> is possible.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (ATLAS-4548) Set Empty attribute value for queryStr and queryId in case of null value when atlas.hook.hive.hive_process.populate.deprecated.attributes=true
Sidharth Kumar Mishra created ATLAS-4548:
Summary: Set Empty attribute value for queryStr and queryId in
case of null value when
atlas.hook.hive.hive_process.populate.deprecated.attributes=true
Key: ATLAS-4548
URL: https://issues.apache.org/jira/browse/ATLAS-4548
Project: Atlas
Issue Type: Bug
Reporter: Sidharth Kumar Mishra
Assignee: Sidharth Kumar Mishra
As part of https://issues.apache.org/jira/browse/ATLAS-3606 by default we have
made the userName, queryId and queryText for hive_process deprecated and we
don't get these attribute values for some of the queries from Hive context. For
example creating external table query below we don't get queryText and queryId
from hive context and we can't populate either. If you mark the flag
"atlas.hook.hive.hive_process.populate.deprecated.attributes=true" then it will
cause problems in this case as hive context has null value and Atlas expects
some value to be present.
Repro steps:
--
First set "atlas.hook.hive.hive_process.populate.deprecated.attributes=true".
Then "beeline -e 'create external table csaba_table_3(msg string);'"
Check Atlas log:
{noformat}
2022-01-26 12:26:44,950 ERROR - [NotificationHookConsumer thread-0:] ~ graph
rollback due to exception AtlasBaseException:Invalid instance creation/updation
parameters passed : hive_process.queryText: mandatory attribute value missing
in type hive_process (GraphTransactionInterceptor:202)
2022-01-26 12:26:44,951 WARN - [NotificationHookConsumer thread-0:] ~ Max
retries exceeded for message
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgCreationTime":164324951,"spooled":false,"message":{"type":"ENTITY_CREATE_V2","user":"hive","entities":{"referredEntities":{"-30336865853165805":{"typeName":"hive_storagedesc","attributes":{"qualifiedName":"default.csaba_table_3@cm_storage","storedAsSubDirectories":false,"location":"hdfs://ns1/warehouse/tablespace/external/hive/csaba_table_3","compressed":false,"inputFormat":"org.apache.hadoop.mapred.TextInputFormat","parameters":{},"outputFormat":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serdeInfo":{"typeName":"hive_serde","attributes":{"serializationLib":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe","name":null,"parameters":{"serialization.format":"1"}}},"numBuckets":-1},"guid":"-30336865853165805","isIncomplete":false,"provenanceType":0,"version":0,"relationshipAttributes":{"table":{"guid":"-30336865853165804","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.csaba_table_3@cm"},"relationshipType":"hive_table_storagedesc"}},"proxy":false},"-30336865853165806":{"typeName":"hive_column","attributes":{"owner":"hrt_qa","qualifiedName":"default.csaba_table_3.msg@cm","name":"msg","comment":null,"position":0,"type":"string"},"guid":"-30336865853165806","isIncomplete":false,"provenanceType":0,"version":0,"relationshipAttributes":{"table":{"guid":"-30336865853165804","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"default.csaba_table_3@cm"},"relationshipType":"hive_table_columns"}},"proxy":false},"-30336865853165807":{"typeName":"hdfs_path","attributes":{"nameServiceId":"ns1","path":"hdfs://ns1/warehouse/tablespace/external/hive/csaba_table_3","qualifiedName":"hdfs://ns1/warehouse/tablespace/external/hive/csaba_table_3@cm","clusterName":"cm","name":"/warehouse/tablespace/external/hive/csaba_table_3"},"guid":"-30336865853165807","isIncomplete":false,"provenanceType":0,"version":0,"proxy":false}},"entities":[{"typeName":"hive_table","attributes":{"owner":"hrt_qa","tableType":"EXTERNAL_TABLE","temporary":false,"lastAccessTime":164323000,"createTime":164323000,"qualifiedName":"default.csaba_table_3@cm","name":"csaba_table_3","comment":null,"parameters":{"totalSize":"0","EXTERNAL":"TRUE","numRows":"0","rawDataSize":"0","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"msg\":\"true\"}}","numFiles":"0","transient_lastDdlTime":"164323","bucketing_version":"2","numFilesErasureCoded":"0"},"retention":0},"guid":"-30336865853165804","isIncomplete":false,"provenanceType":0,"version":0,"relationshipAttributes":{"sd":{"guid":"-30336865853165805","typeName":"hive_storagedesc","uniqueAttributes":{"qualifiedName":"default.csaba_table_3@cm_storage"},"relationshipType":"hive_table_storagedesc"},"columns":[{"guid":"-30336865853165806","typeName":"hive_column","uniqueAttributes":{"qualifiedName":"default.csaba_table_3.msg@cm"},"relationshipType":"hive_table_columns"}],"partitionKeys":[],"db":{"typeName":"hive_db","uniqueAttributes":{"qualifiedName":"default@cm"},"relationshipType":"hive_table_db"}},"proxy":false},{"typeName":"hive_process","attributes":{"recentQueries":[null],"qualifiedName":"default.csaba_table_3@cm:164323000","clusterName":"cm","name":"default.csaba_tabl
[jira] [Commented] (ATLAS-4712) atlas-hive image shows "No such file or directory error"
[
https://issues.apache.org/jira/browse/ATLAS-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17635969#comment-17635969
]
Sidharth Kumar Mishra commented on ATLAS-4712:
--
+1 for Patch. Thanks [~madhan]
> atlas-hive image shows "No such file or directory error"
> -
>
> Key: ATLAS-4712
> URL: https://issues.apache.org/jira/browse/ATLAS-4712
> Project: Atlas
> Issue Type: Bug
>Reporter: Snehal Ambavkar
>Assignee: Madhan Neethiraj
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: ATLAS-4712.patch
>
>
> Running docker-compose
> {code:java}
> docker-compose -f docker-compose.atlas-base.yml -f docker-compose.atlas.yml
> -f docker-compose.atlas-hadoop.yml -f docker-compose.atlas-hbase.yml -f
> docker-compose.atlas-kafka.yml -f docker-compose.atlas-hive.yml up -d
> {code}
> shows with:
> {code:java}
> /home/atlas/scripts/atlas-hive-setup.sh: line 44: cd:
> /opt/atlas/atlas-hive-plugin: No such file or directory
> /home/atlas/scripts/atlas-hive-setup.sh: line 45: ./enable-hive-plugin.sh: No
> such file or directory {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (ATLAS-3461) Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
Sidharth Kumar Mishra created ATLAS-3461: Summary: Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces Key: ATLAS-3461 URL: https://issues.apache.org/jira/browse/ATLAS-3461 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra Some examples of such queries are at Impala: creAted external table /*test*/ impala1 aS – temp fff /* temp */ – sdfrnfkjef select select * from t1; Create /*dd*/ table impala2 aS select * from t1; >From JDBC: If we add more spaces in between create and table as shown below and use jdbc to execute the query then the impala process is not getting created at Atlas. create table testSun2 as select * from t1; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3477) No way to purge the soft deleted entity from Atlas
Sidharth Kumar Mishra created ATLAS-3477: Summary: No way to purge the soft deleted entity from Atlas Key: ATLAS-3477 URL: https://issues.apache.org/jira/browse/ATLAS-3477 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3477) No way to purge the soft deleted entity from Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra reassigned ATLAS-3477: Assignee: Sidharth Kumar Mishra > No way to purge the soft deleted entity from Atlas > -- > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: Bug >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) No way to purge the soft deleted entity from Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Issue Type: New Feature (was: Bug) > No way to purge the soft deleted entity from Atlas > -- > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) Introduce purging of entity in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Summary: Introduce purging of entity in Atlas (was: No way to purge the soft deleted entity from Atlas) > Introduce purging of entity in Atlas > > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) Introduce purging of entity in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Description: In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. There are two cases which needs to be addressed as part of this problem. # There are active entity which has to be purged in case the entity is dropped. # There might be existing soft deleted entities and which has to be purge deleted In both the above cases we also need to think about the auditing and its impact. was: In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. > Introduce purging of entity in Atlas > > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. > There are two cases which needs to be addressed as part of this problem. > # There are active entity which has to be purged in case the entity is > dropped. > # There might be existing soft deleted entities and which has to be purge > deleted > In both the above cases we also need to think about the auditing and its > impact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) Introduce purging of entity in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Description: In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. There are two cases which needs to be addressed as part of this problem. # There is an active entity which has to be purged in case the entity is dropped. # There might be existing soft deleted entities and which has to be purge deleted if user desires In both the above cases we also need to think about the auditing and its impact. was: In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. There are two cases which needs to be addressed as part of this problem. # There are active entity which has to be purged in case the entity is dropped. # There might be existing soft deleted entities and which has to be purge deleted In both the above cases we also need to think about the auditing and its impact. > Introduce purging of entity in Atlas > > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. > There are two cases which needs to be addressed as part of this problem. > # There is an active entity which has to be purged in case the entity is > dropped. > # There might be existing soft deleted entities and which has to be purge > deleted if user desires > In both the above cases we also need to think about the auditing and its > impact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3461) Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
[ https://issues.apache.org/jira/browse/ATLAS-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3461: - Attachment: ATLAS-3461-3.patch > Impala process and lineage is not getting created appropriately if query > contains variation like comment or extra spaces > > > Key: ATLAS-3461 > URL: https://issues.apache.org/jira/browse/ATLAS-3461 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3461-3.patch > > > Some examples of such queries are at Impala: > creAted external table /*test*/ impala1 aS – temp fff > /* temp */ – sdfrnfkjef select > select * from t1; > Create /*dd*/ table impala2 aS select * from t1; > > From JDBC: > If we add more spaces in between create and table as shown below and use jdbc > to execute the query then the impala process is not getting created at Atlas. > create table testSun2 as select * from t1; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3461) Impala process and lineage is not getting created appropriately if query contains variation like comment or extra spaces
[ https://issues.apache.org/jira/browse/ATLAS-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3461: - Attachment: (was: ATLAS-3461-1.patch) > Impala process and lineage is not getting created appropriately if query > contains variation like comment or extra spaces > > > Key: ATLAS-3461 > URL: https://issues.apache.org/jira/browse/ATLAS-3461 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3461-3.patch > > > Some examples of such queries are at Impala: > creAted external table /*test*/ impala1 aS – temp fff > /* temp */ – sdfrnfkjef select > select * from t1; > Create /*dd*/ table impala2 aS select * from t1; > > From JDBC: > If we add more spaces in between create and table as shown below and use jdbc > to execute the query then the impala process is not getting created at Atlas. > create table testSun2 as select * from t1; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) Introduce purging of entity in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Description: In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. As part of this issue solution we should be able to purge the entities which are already soft deleted. This will be useful to users who is interested to purge some list of soft deleted entities from Atlas. Due to governance compliances, only admin can perform purge of entities. Purge should have separate Audit similar to delete entity. was: In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there is no way to do cleanup if user wants to delete those permanently. Right now we can change the configuration atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as desired. But what happens to existing soft deleted Entities. There is no way in that case. There are two cases which needs to be addressed as part of this problem. # There is an active entity which has to be purged in case the entity is dropped. # There might be existing soft deleted entities and which has to be purge deleted if user desires In both the above cases we also need to think about the auditing and its impact. > Introduce purging of entity in Atlas > > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. > As part of this issue solution we should be able to purge the entities which > are already soft deleted. This will be useful to users who is interested to > purge some list of soft deleted entities from Atlas. Due to governance > compliances, only admin can perform purge of entities. > Purge should have separate Audit similar to delete entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) Introduce purging of entity in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Attachment: ATLAS-3477.patch > Introduce purging of entity in Atlas > > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-3477.patch > > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. > As part of this issue solution we should be able to purge the entities which > are already soft deleted. This will be useful to users who is interested to > purge some list of soft deleted entities from Atlas. Due to governance > compliances, only admin can perform purge of entities. > Purge should have separate Audit similar to delete entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3477) Introduce purging of entity in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3477: - Attachment: (was: ATLAS-3477.patch) > Introduce purging of entity in Atlas > > > Key: ATLAS-3477 > URL: https://issues.apache.org/jira/browse/ATLAS-3477 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3477.patch > > > In case of soft delete at Atlas i.e. the entities marked as 'DELETED' there > is no way to do cleanup if user wants to delete those permanently. Right now > we can change the configuration > atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1. > HardDeleteHandlerV1 and make the new delete of entity to be cleaned up as > desired. But what happens to existing soft deleted Entities. There is no way > in that case. > As part of this issue solution we should be able to purge the entities which > are already soft deleted. This will be useful to users who is interested to > purge some list of soft deleted entities from Atlas. Due to governance > compliances, only admin can perform purge of entities. > Purge should have separate Audit similar to delete entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3518) Create Audit Framework for Atlas Purge Entity
Sidharth Kumar Mishra created ATLAS-3518: Summary: Create Audit Framework for Atlas Purge Entity Key: ATLAS-3518 URL: https://issues.apache.org/jira/browse/ATLAS-3518 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3518) Create Audit Framework for Atlas Purge Entity
[ https://issues.apache.org/jira/browse/ATLAS-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3518: - Issue Type: New Feature (was: Bug) > Create Audit Framework for Atlas Purge Entity > - > > Key: ATLAS-3518 > URL: https://issues.apache.org/jira/browse/ATLAS-3518 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3522) Add new REST API to purge an entity at Atlas using typeName/uniqueAttributes
Sidharth Kumar Mishra created ATLAS-3522: Summary: Add new REST API to purge an entity at Atlas using typeName/uniqueAttributes Key: ATLAS-3522 URL: https://issues.apache.org/jira/browse/ATLAS-3522 Project: Atlas Issue Type: New Feature Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3522) Add new REST API to purge an entity at Atlas using typeName/uniqueAttributes
[ https://issues.apache.org/jira/browse/ATLAS-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3522: - Description: Add new method that take List as parameter for purge. Such method can be useful in purging entities using typeName/uniqueAttributes. > Add new REST API to purge an entity at Atlas using typeName/uniqueAttributes > > > Key: ATLAS-3522 > URL: https://issues.apache.org/jira/browse/ATLAS-3522 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > Add new method that take List as parameter for purge. Such > method can be useful in purging entities using typeName/uniqueAttributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3524) Add new privilege - admin-purge at ranger
Sidharth Kumar Mishra created ATLAS-3524: Summary: Add new privilege - admin-purge at ranger Key: ATLAS-3524 URL: https://issues.apache.org/jira/browse/ATLAS-3524 Project: Atlas Issue Type: New Feature Components: atlas-core Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra For Atlas Admin Purge we need to add new privilege - admin-purge at ranger and change the atlas purge to verifyAccess accordingly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-2985) Fix delete handlers and relationship store
[ https://issues.apache.org/jira/browse/ATLAS-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987220#comment-16987220 ] Sidharth Kumar Mishra commented on ATLAS-2985: -- Hi [~alollo], This is been resolved as part of ATLAS-3477. There is one more follow up feature for auditing the purge which is being tracked as part of ATLAS-3518. > Fix delete handlers and relationship store > -- > > Key: ATLAS-2985 > URL: https://issues.apache.org/jira/browse/ATLAS-2985 > Project: Atlas > Issue Type: Bug >Reporter: Graham Wallis >Assignee: Graham Wallis >Priority: Critical > Attachments: ATLAS-2985-2018-11-30.patch > > > The problems this fixes are as follows: > * following a soft delete, a hard delete is not possible because the entity > or relationship will have status DELETED, and prior to this Jira and patch > the abstract delete handler skips the delete. > * during a soft delete the modified-time, modified-by and version number > attributes on the entity or relationship should be updated, but > updateRelationship does not update system attributes - this Jira and patch > updates the system attributes that will need to be updated during a delete or > other update operations. > * also fixes a minor copy-paste debug logging error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3564) New AWS S3 model addition
Sidharth Kumar Mishra created ATLAS-3564: Summary: New AWS S3 model addition Key: ATLAS-3564 URL: https://issues.apache.org/jira/browse/ATLAS-3564 Project: Atlas Issue Type: New Feature Components: atlas-core Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3500) Implement S3 Bulk Extraction
[ https://issues.apache.org/jira/browse/ATLAS-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra reassigned ATLAS-3500: Assignee: Sidharth Kumar Mishra (was: Sridhar) > Implement S3 Bulk Extraction > > > Key: ATLAS-3500 > URL: https://issues.apache.org/jira/browse/ATLAS-3500 > Project: Atlas > Issue Type: Improvement >Reporter: Sridhar >Assignee: Sidharth Kumar Mishra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3605) Test case failure due to addition of two fields displayName and userDescription to base model
Sidharth Kumar Mishra created ATLAS-3605: Summary: Test case failure due to addition of two fields displayName and userDescription to base model Key: ATLAS-3605 URL: https://issues.apache.org/jira/browse/ATLAS-3605 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra As Part of ATLAS_3559 we have added two new fields and duw to that there are test case failures at EntityJerseyResourceIT.java. The failure details: [https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1616/console] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3518) Create Audit Framework for Atlas Purge Entity
[ https://issues.apache.org/jira/browse/ATLAS-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3518: - Attachment: (was: ATLAS-3518.patch) > Create Audit Framework for Atlas Purge Entity > - > > Key: ATLAS-3518 > URL: https://issues.apache.org/jira/browse/ATLAS-3518 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3518-16.patch > > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3518) Create Audit Framework for Atlas Purge Entity
[ https://issues.apache.org/jira/browse/ATLAS-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3518: - Attachment: ATLAS-3518-16.patch > Create Audit Framework for Atlas Purge Entity > - > > Key: ATLAS-3518 > URL: https://issues.apache.org/jira/browse/ATLAS-3518 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3518-16.patch > > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3518) Create Audit Framework for Atlas Purge Entity
[ https://issues.apache.org/jira/browse/ATLAS-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046931#comment-17046931 ] Sidharth Kumar Mishra commented on ATLAS-3518: -- The distro (TestMetaDate.py) test case failure is not related to my changes and been confirmed > Create Audit Framework for Atlas Purge Entity > - > > Key: ATLAS-3518 > URL: https://issues.apache.org/jira/browse/ATLAS-3518 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3518-16.patch > > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3684) Atlas glossary get is taking more time when number of terms increase
Sidharth Kumar Mishra created ATLAS-3684: Summary: Atlas glossary get is taking more time when number of terms increase Key: ATLAS-3684 URL: https://issues.apache.org/jira/browse/ATLAS-3684 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra If we create a glossary Item and create around 3k terms inside it, get glossary takes around 25-30 seconds which is a lot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3707) Atlas Purge entity is generating both purge and delete audit
Sidharth Kumar Mishra created ATLAS-3707: Summary: Atlas Purge entity is generating both purge and delete audit Key: ATLAS-3707 URL: https://issues.apache.org/jira/browse/ATLAS-3707 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra It should generate only purge audit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3719) Tag and Type search is not doing index search
Sidharth Kumar Mishra created ATLAS-3719: Summary: Tag and Type search is not doing index search Key: ATLAS-3719 URL: https://issues.apache.org/jira/browse/ATLAS-3719 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3726) Add result count to Atlas Admin Audit and enable filters on count for get audit
Sidharth Kumar Mishra created ATLAS-3726: Summary: Add result count to Atlas Admin Audit and enable filters on count for get audit Key: ATLAS-3726 URL: https://issues.apache.org/jira/browse/ATLAS-3726 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3727) Add get purge result summary REST for admin audit record
Sidharth Kumar Mishra created ATLAS-3727: Summary: Add get purge result summary REST for admin audit record Key: ATLAS-3727 URL: https://issues.apache.org/jira/browse/ATLAS-3727 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Sidharth Kumar Mishra Right now purge admin audit result stores purged entities guids. We need a separate REST call to get more details summary of the entities purged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3727) Add get purge result summary REST for admin audit record
[ https://issues.apache.org/jira/browse/ATLAS-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra reassigned ATLAS-3727: Assignee: Sidharth Kumar Mishra > Add get purge result summary REST for admin audit record > > > Key: ATLAS-3727 > URL: https://issues.apache.org/jira/browse/ATLAS-3727 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > Right now purge admin audit result stores purged entities guids. We need a > separate REST call to get more details summary of the entities purged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3727) Add get purge result summary REST for admin audit record
[ https://issues.apache.org/jira/browse/ATLAS-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3727: - Issue Type: New Feature (was: Bug) > Add get purge result summary REST for admin audit record > > > Key: ATLAS-3727 > URL: https://issues.apache.org/jira/browse/ATLAS-3727 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > Right now purge admin audit result stores purged entities guids. We need a > separate REST call to get more details summary of the entities purged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3727) Add get result details REST for admin audit record
[ https://issues.apache.org/jira/browse/ATLAS-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3727: - Summary: Add get result details REST for admin audit record (was: Add get purge result summary REST for admin audit record) > Add get result details REST for admin audit record > -- > > Key: ATLAS-3727 > URL: https://issues.apache.org/jira/browse/ATLAS-3727 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > Right now purge admin audit result stores purged entities guids. We need a > separate REST call to get more details summary of the entities purged. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3729) Add audit operation as a parameter to v2/entity/{guid}/audit
Sidharth Kumar Mishra created ATLAS-3729:
Summary: Add audit operation as a parameter to
v2/entity/{guid}/audit
Key: ATLAS-3729
URL: https://issues.apache.org/jira/browse/ATLAS-3729
Project: Atlas
Issue Type: New Feature
Reporter: Sidharth Kumar Mishra
Assignee: Sidharth Kumar Mishra
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3729) Add audit operation as a parameter to v2/entity/{guid}/audit
[
https://issues.apache.org/jira/browse/ATLAS-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082563#comment-17082563
]
Sidharth Kumar Mishra commented on ATLAS-3729:
--
We should be able to filter the entity audit records based on audit operations
which can be used by GUI in case of purge entity.
> Add audit operation as a parameter to v2/entity/{guid}/audit
>
>
> Key: ATLAS-3729
> URL: https://issues.apache.org/jira/browse/ATLAS-3729
> Project: Atlas
> Issue Type: New Feature
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3730) In case Hard delete and Purge of an Entity we should make the entity status as purged instead of deleted
Sidharth Kumar Mishra created ATLAS-3730: Summary: In case Hard delete and Purge of an Entity we should make the entity status as purged instead of deleted Key: ATLAS-3730 URL: https://issues.apache.org/jira/browse/ATLAS-3730 Project: Atlas Issue Type: New Feature Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra Right now if we do hard delete or purge of an entity the status for the entity is still deleted. Due to this it's hard to know if any entity is really purged from the header which we keep. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3730) In case Hard delete or Purge of an Entity we should make the entity status as purged instead of deleted at classification
[ https://issues.apache.org/jira/browse/ATLAS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3730: - Summary: In case Hard delete or Purge of an Entity we should make the entity status as purged instead of deleted at classification (was: In case Hard delete and Purge of an Entity we should make the entity status as purged instead of deleted) > In case Hard delete or Purge of an Entity we should make the entity status as > purged instead of deleted at classification > - > > Key: ATLAS-3730 > URL: https://issues.apache.org/jira/browse/ATLAS-3730 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-3730.patch > > > Right now if we do hard delete or purge of an entity the status for the > entity is still deleted. Due to this it's hard to know if any entity is > really purged from the header which we keep. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3730) In case Hard delete or Purge of an Entity we should make the entity status as purged instead of deleted at classification
[ https://issues.apache.org/jira/browse/ATLAS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3730: - Description: Right now if we do hard delete or purge of an entity the status for the entity is still deleted at classification. Due to this it's hard to know if any entity is really purged from classification (was: Right now if we do hard delete or purge of an entity the status for the entity is still deleted. Due to this it's hard to know if any entity is really purged from the header which we keep.) > In case Hard delete or Purge of an Entity we should make the entity status as > purged instead of deleted at classification > - > > Key: ATLAS-3730 > URL: https://issues.apache.org/jira/browse/ATLAS-3730 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-3730.patch > > > Right now if we do hard delete or purge of an entity the status for the > entity is still deleted at classification. Due to this it's hard to know if > any entity is really purged from classification -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3747) Atlas Admin Purge API should take list of guids as query param instead of body
[ https://issues.apache.org/jira/browse/ATLAS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra reassigned ATLAS-3747: Assignee: Sidharth Kumar Mishra > Atlas Admin Purge API should take list of guids as query param instead of body > -- > > Key: ATLAS-3747 > URL: https://issues.apache.org/jira/browse/ATLAS-3747 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3747) Atlas Admin Purge API should take list of guids as query param instead of body
Sidharth Kumar Mishra created ATLAS-3747: Summary: Atlas Admin Purge API should take list of guids as query param instead of body Key: ATLAS-3747 URL: https://issues.apache.org/jira/browse/ATLAS-3747 Project: Atlas Issue Type: New Feature Reporter: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3747) Atlas Admin Purge API should take list of guids as body with operation Put instead of Delete
[ https://issues.apache.org/jira/browse/ATLAS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3747: - Summary: Atlas Admin Purge API should take list of guids as body with operation Put instead of Delete (was: Atlas Admin Purge API should take list of guids as query param instead of body) > Atlas Admin Purge API should take list of guids as body with operation Put > instead of Delete > > > Key: ATLAS-3747 > URL: https://issues.apache.org/jira/browse/ATLAS-3747 > Project: Atlas > Issue Type: New Feature >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-3747.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3781) update authorization to admin-purge for admin entity purge Rest
Sidharth Kumar Mishra created ATLAS-3781: Summary: update authorization to admin-purge for admin entity purge Rest Key: ATLAS-3781 URL: https://issues.apache.org/jira/browse/ATLAS-3781 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3781) update authorization to admin-purge for admin entity purge Rest
[ https://issues.apache.org/jira/browse/ATLAS-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra resolved ATLAS-3781. -- Resolution: Duplicate > update authorization to admin-purge for admin entity purge Rest > --- > > Key: ATLAS-3781 > URL: https://issues.apache.org/jira/browse/ATLAS-3781 > Project: Atlas > Issue Type: Bug >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3781) update authorization to admin-purge for admin entity purge Rest
[ https://issues.apache.org/jira/browse/ATLAS-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100215#comment-17100215 ] Sidharth Kumar Mishra commented on ATLAS-3781: -- Duplicate to https://issues.apache.org/jira/browse/ATLAS-3575 > update authorization to admin-purge for admin entity purge Rest > --- > > Key: ATLAS-3781 > URL: https://issues.apache.org/jira/browse/ATLAS-3781 > Project: Atlas > Issue Type: Bug >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3785) Modify the aws s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional
Sidharth Kumar Mishra created ATLAS-3785: Summary: Modify the aws s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional Key: ATLAS-3785 URL: https://issues.apache.org/jira/browse/ATLAS-3785 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra Modify the aws s3 v2 model to make relationship def end points as legacy attributes and objectPrefix as optional -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3806) Classifications information missing in notification events during entity update/delete
Sidharth Kumar Mishra created ATLAS-3806: Summary: Classifications information missing in notification events during entity update/delete Key: ATLAS-3806 URL: https://issues.apache.org/jira/browse/ATLAS-3806 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3806) Classifications information missing in notification events during entity create/update
[ https://issues.apache.org/jira/browse/ATLAS-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3806: - Summary: Classifications information missing in notification events during entity create/update (was: Classifications information missing in notification events during entity create/update/delete) > Classifications information missing in notification events during entity > create/update > -- > > Key: ATLAS-3806 > URL: https://issues.apache.org/jira/browse/ATLAS-3806 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-3806.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3806) Classifications information missing in notification events during entity create/update/delete
[ https://issues.apache.org/jira/browse/ATLAS-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3806: - Summary: Classifications information missing in notification events during entity create/update/delete (was: Classifications information missing in notification events during entity update/delete) > Classifications information missing in notification events during entity > create/update/delete > - > > Key: ATLAS-3806 > URL: https://issues.apache.org/jira/browse/ATLAS-3806 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-3806.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3877) Purged entity audit throws 404 Entity not found error
[ https://issues.apache.org/jira/browse/ATLAS-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra reassigned ATLAS-3877: Assignee: Sidharth Kumar Mishra > Purged entity audit throws 404 Entity not found error > - > > Key: ATLAS-3877 > URL: https://issues.apache.org/jira/browse/ATLAS-3877 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sharmadha S >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > Attachments: rc2.png > > > 1. Create an hdfs_path entity. > 2. Delete the entity using DELETE: /api/atlas/v2/entity/guid/. The > entity is soft deleted > 3. Purge the entity using PUT : /api/atlas/admin/purge with body [""] > 4. Go to admin -> Administration -> Audits > 5. Click on the GUID of purged entity. > The purged entity details window opens but Atlas throws a 404 error "Given > entity GUID is invalid/not found" . Attached screenshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3921) Need migration path from AWS s3 v1 to v2 entities
Sidharth Kumar Mishra created ATLAS-3921: Summary: Need migration path from AWS s3 v1 to v2 entities Key: ATLAS-3921 URL: https://issues.apache.org/jira/browse/ATLAS-3921 Project: Atlas Issue Type: Bug Reporter: Sidharth Kumar Mishra Assignee: Sidharth Kumar Mishra We have introduced the new AWS S3 V2 entity at Atlas. After upgrade the service like Hive will create the new V2 entities at Atlas, but the old v1 Aws S3 entity will not be migrated to new one. This will cause missing relationship and/or lineage. We should have a migration path from the old v1 entity to new v2 entity . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3921) Need migration path from AWS s3 v1 to v2 entities
[ https://issues.apache.org/jira/browse/ATLAS-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated ATLAS-3921: - Issue Type: Improvement (was: Bug) > Need migration path from AWS s3 v1 to v2 entities > - > > Key: ATLAS-3921 > URL: https://issues.apache.org/jira/browse/ATLAS-3921 > Project: Atlas > Issue Type: Improvement >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > > We have introduced the new AWS S3 V2 entity at Atlas. After upgrade the > service like Hive will create the new V2 entities at Atlas, but the old v1 > Aws S3 entity will not be migrated to new one. This will cause missing > relationship and/or lineage. We should have a migration path from the old v1 > entity to new v2 entity . -- This message was sent by Atlassian Jira (v8.3.4#803005)
