[jira] [Created] (ATLAS-4844) Atlas - Upgrade Common Configuration2 to 2.10.1

2024-04-01 Thread Priyanshi Shah (Jira) 
Priyanshi Shah created ATLAS-4844:
-

 Summary: Atlas - Upgrade Common Configuration2 to 2.10.1
 Key: ATLAS-4844
 URL: https://issues.apache.org/jira/browse/ATLAS-4844
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 3.0.0
Reporter: Priyanshi Shah
Assignee: Priyanshi Shah






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (ATLAS-4844) Atlas - Upgrade Common Configuration2 to 2.10.1

2024-04-01 Thread Priyanshi Shah (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyanshi Shah updated ATLAS-4844:
--
Description: Upgrading common configurations2 from 2.8.0 to 2.10.1

> Atlas - Upgrade Common Configuration2 to 2.10.1
> ---
>
> Key: ATLAS-4844
> URL: https://issues.apache.org/jira/browse/ATLAS-4844
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 3.0.0
>Reporter: Priyanshi Shah
>Assignee: Priyanshi Shah
>Priority: Major
>
> Upgrading common configurations2 from 2.8.0 to 2.10.1



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value

2024-04-01 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832770#comment-17832770
 ] 

ASF subversion and git services commented on ATLAS-4838:


Commit f4f9d623301d49426470ff38d6a4a60ce5fc014d in atlas's branch 
refs/heads/master from priyanshi-freestoneinfotech
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=f4f9d6233 ]

ATLAS-4838 : Export/Import : changeMarker is not set to entity's lastupdatetime 
or its closer timestamp value

Signed-off-by: Pinal Shah 


> Export/Import : changeMarker is not set to entity's lastupdatetime or its 
> closer timestamp value
> 
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Sharmadha S
>Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4838-V2.patch, ATLAS-4838.patch
>
>
> When an hive table entity is exported using fetchType incremental with 
> changeMarker 0 , after exporting , the changeMarker in the export response is 
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and 
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ 
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table', 
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null, 
> fetchType=incremental): found 1 entities: options: 
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
>  (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid = 
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source: 
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin, 
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928 
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 =>  Tuesday, February 6, 2024 
> 6:43:03.928 PM
>  
> But lastUpdatetime of iceberg table is 
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM  
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM 
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it 
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next 
> export



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value

2024-04-01 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832773#comment-17832773
 ] 

ASF subversion and git services commented on ATLAS-4838:


Commit 89fdbe333aba5bb771142206de76221cc362f8f0 in atlas's branch 
refs/heads/branch-2.0 from priyanshi-freestoneinfotech
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=89fdbe333 ]

ATLAS-4838 : Export/Import : changeMarker is not set to entity's lastupdatetime 
or its closer timestamp value

Signed-off-by: Pinal Shah 


> Export/Import : changeMarker is not set to entity's lastupdatetime or its 
> closer timestamp value
> 
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Sharmadha S
>Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4838-V2.patch, ATLAS-4838.patch
>
>
> When an hive table entity is exported using fetchType incremental with 
> changeMarker 0 , after exporting , the changeMarker in the export response is 
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and 
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ 
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table', 
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null, 
> fetchType=incremental): found 1 entities: options: 
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
>  (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid = 
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source: 
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin, 
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928 
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 =>  Tuesday, February 6, 2024 
> 6:43:03.928 PM
>  
> But lastUpdatetime of iceberg table is 
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM  
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM 
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it 
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next 
> export



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs

2024-04-01 Thread Priyanshi Shah (Jira) 
Priyanshi Shah created ATLAS-4845:
-

 Summary: Atlas Import is failing with fetchType: Incremental if 
there are no changes between two consecutive runs
 Key: ATLAS-4845
 URL: https://issues.apache.org/jira/browse/ATLAS-4845
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Reporter: Priyanshi Shah
Assignee: Priyanshi Shah






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected

2024-04-01 Thread Priyanshi Shah (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyanshi Shah updated ATLAS-4842:
--
Attachment: (was: 0001-Hive-Incremental.patch)

> Export/Import: fetchType as "incremental" does full export instead of 
> connected
> ---
>
> Key: ATLAS-4842
> URL: https://issues.apache.org/jira/browse/ATLAS-4842
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Sheetal Shah
>Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4842-V2.patch
>
>
> Create two hive_tables in same database:
>  # create table hive1(id int);
>  # create table hive1_ctas as select * from hive1;
>  # create table hive2(name string);
> Perform incremental export of hive1 with changeMarker as 0 and skipLineage 
> value as false.
> Import the zip.
> Expectation is only the requested hive_table hive1 and its related entities 
> should be exported, but it does full export and hive2 is also imported.
>     



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (ATLAS-4842) Export/Import: fetchType as "incremental" does full export instead of connected

2024-04-01 Thread Priyanshi Shah (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyanshi Shah updated ATLAS-4842:
--
Attachment: (was: ATLAS-4842-V2.patch)

> Export/Import: fetchType as "incremental" does full export instead of 
> connected
> ---
>
> Key: ATLAS-4842
> URL: https://issues.apache.org/jira/browse/ATLAS-4842
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Sheetal Shah
>Assignee: Priyanshi Shah
>Priority: Major
>
> Create two hive_tables in same database:
>  # create table hive1(id int);
>  # create table hive1_ctas as select * from hive1;
>  # create table hive2(name string);
> Perform incremental export of hive1 with changeMarker as 0 and skipLineage 
> value as false.
> Import the zip.
> Expectation is only the requested hive_table hive1 and its related entities 
> should be exported, but it does full export and hive2 is also imported.
>     



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (ATLAS-4838) Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value

2024-04-01 Thread Priyanshi Shah (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyanshi Shah updated ATLAS-4838:
--
Attachment: (was: ATLAS-4838.patch)

> Export/Import : changeMarker is not set to entity's lastupdatetime or its 
> closer timestamp value
> 
>
> Key: ATLAS-4838
> URL: https://issues.apache.org/jira/browse/ATLAS-4838
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Sharmadha S
>Assignee: Priyanshi Shah
>Priority: Major
> Attachments: ATLAS-4838-V2.patch
>
>
> When an hive table entity is exported using fetchType incremental with 
> changeMarker 0 , after exporting , the changeMarker in the export response is 
> not set to recent timestamp.
> For example , on exporting iceberg table with fetchType incremental and 
> changeMarker as 0 :
> {code:java}
> 2024-02-08 13:28:43,838 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ 
> export(item=AtlasObjectId{guid='null', typeName='iceberg_table', 
> uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null, 
> fetchType=incremental): found 1 entities: options: 
> {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0}
>  (StartEntityFetchByExportRequest:123)
> 2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid = 
> f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
> 2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source: 
> cm, target: , operation: EXPORT (ExportImportAuditService:168)
> 2024-02-08 13:28:48,699 INFO  - [etp891461509-20 - 
> db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin, 
> from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928 
> (ExportService:107) {code}
> changeMarker from response is 1707225183928 =>  Tuesday, February 6, 2024 
> 6:43:03.928 PM
>  
> But lastUpdatetime of iceberg table is 
> {code:java}
> "createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM  
> "updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM 
> {code}
> Value is set to 2 days back. This is not then an incremental export , as it 
> will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next 
> export



--
This message was sent by Atlassian Jira
(v8.20.10#820010)