[
https://issues.apache.org/jira/browse/ATLAS-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17956247#comment-17956247
]
Aditya Gupta commented on ATLAS-5034:
-------------------------------------
Can refer the doc. It includes findings, analysis, UI snippets wrt to OZONE
code changes:
https://docs.google.com/document/d/1SjAHScw1hBlinu6tQF7FFIcY-qbV3mJcM5GSX2MUUZA/edit?usp=sharing
> Entities not created in Atlas when hive table created on top a OFS/O3FS
> bucket/volume
> -------------------------------------------------------------------------------------
>
> Key: ATLAS-5034
> URL: https://issues.apache.org/jira/browse/ATLAS-5034
> Project: Atlas
> Issue Type: Bug
> Reporter: Paresh Devalia
> Assignee: Aditya Gupta
> Priority: Major
> Attachments: Screenshot 2024-08-22 at 10.35.16 AM.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Facing issues when trying to create hive tables on top of ozone ofs
> buckets/volumes
> Steps to reproduce
> {code:java}
> # ozone sh volume create /volume1
> # ozone sh bucket create /volume1/bucket1
> # ozone sh bucket create /volume1/bucket1.bucket1.bucket1
> # beeline
> 0: jdbc:hive2://altas-hive> CREATE EXTERNAL TABLE table_volume(id int) row
> format delimited fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1';
> INFO : Compiling
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
> CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
> terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44);
> Time taken: 0.053 seconds
> INFO : Executing
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
> CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
> terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44);
> Time taken: 0.417 seconds
> INFO : OK
> No rows affected (0.544 seconds)
>
> 0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_bucket_dotted(id int)
> row format delimited fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1';
> INFO : Compiling
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
> CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
> terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa);
> Time taken: 0.182 seconds
> INFO : Executing
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
> CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
> terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa);
> Time taken: 0.465 seconds
> INFO : OK
> No rows affected (1.155 seconds)
>
> 0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_key_bucket_dotted(id
> int) row format delimited fields terminated by ' ' stored as textfile
> location 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2';
> INFO : Compiling
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
> CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
> fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421);
> Time taken: 0.026 seconds
> INFO : Executing
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
> CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
> fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421);
> Time taken: 0.226 seconds
> INFO : OK
> No rows affected (0.342 seconds)
>
> 0: jdbc:hive2://atlas-hive> show tables;
> INFO : Compiling
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
> type:string, comment:from deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515);
> Time taken: 0.045 seconds
> INFO : Executing
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515);
> Time taken: 0.026 seconds
> INFO : OK
> +--------------------------+
> | tab_name |
> +--------------------------+
> | sample_07 |
> | sample_08 |
> | table_bucket_dotted |
> | table_key |
> | table_key_bucket_dotted |
> | table_volume |
> | web_logs |
> +--------------------------+
> In Atlas, key based tables are created properly (eg table_key,
> table_key_bucket_dotted). However, bucket/volume based tables do not show up
> in Atlas
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
