[
https://issues.apache.org/jira/browse/ATLAS-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paresh Devalia updated ATLAS-5034:
----------------------------------
Description:
Facing issues when trying to create hive tables on top of ozone ofs
buckets/volumes
Steps to reproduce
{code:java}
# ozone sh volume create /volume1
# ozone sh bucket create /volume1/bucket1
# ozone sh bucket create /volume1/bucket1.bucket1.bucket1
# beeline
0: jdbc:hive2://altas-hive> CREATE EXTERNAL TABLE table_volume(id int) row
format delimited fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1';
INFO : Compiling
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44); Time
taken: 0.053 seconds
INFO : Executing
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44); Time
taken: 0.417 seconds
INFO : OK
No rows affected (0.544 seconds)
0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_bucket_dotted(id int)
row format delimited fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1';
INFO : Compiling
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa); Time
taken: 0.182 seconds
INFO : Executing
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa); Time
taken: 0.465 seconds
INFO : OK
No rows affected (1.155 seconds)
0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_key_bucket_dotted(id
int) row format delimited fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2';
INFO : Compiling
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421); Time
taken: 0.026 seconds
INFO : Executing
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421); Time
taken: 0.226 seconds
INFO : OK
No rows affected (0.342 seconds)
0: jdbc:hive2://atlas-hive> show tables;
INFO : Compiling
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515): show
tables
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515); Time
taken: 0.045 seconds
INFO : Executing
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515); Time
taken: 0.026 seconds
INFO : OK
+--------------------------+
| tab_name |
+--------------------------+
| sample_07 |
| sample_08 |
| table_bucket_dotted |
| table_key |
| table_key_bucket_dotted |
| table_volume |
| web_logs |
+--------------------------+
In Atlas, key based tables are created properly (eg table_key,
table_key_bucket_dotted). However, bucket/volume based tables do not show up in
Atlas
{code}
was:
Facing issues when trying to create hive tables on top of ozone ofs
buckets/volumes
Steps to reproduce
{code:java}
# ozone sh volume create /volume1
# ozone sh bucket create /volume1/bucket1
# ozone sh bucket create /volume1/bucket1.bucket1.bucket1
# beeline
0: jdbc:hive2://altas-hive> CREATE EXTERNAL TABLE table_volume(id int) row
format delimited fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1';
INFO : Compiling
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44); Time
taken: 0.053 seconds
INFO : Executing
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44); Time
taken: 0.417 seconds
INFO : OK
No rows affected (0.544 seconds)
0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_bucket_dotted(id int)
row format delimited fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1';
INFO : Compiling
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa); Time
taken: 0.182 seconds
INFO : Executing
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa); Time
taken: 0.465 seconds
INFO : OK
No rows affected (1.155 seconds)
0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_key_bucket_dotted(id
int) row format delimited fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2';
INFO : Compiling
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421); Time
taken: 0.026 seconds
INFO : Executing
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
fields terminated by ' ' stored as textfile location
'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421); Time
taken: 0.226 seconds
INFO : OK
No rows affected (0.342 seconds)
0: jdbc:hive2://atlas-hive> show tables;
INFO : Compiling
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515): show
tables
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515); Time
taken: 0.045 seconds
INFO : Executing
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515); Time
taken: 0.026 seconds
INFO : OK
+--------------------------+
| tab_name |
+--------------------------+
| sample_07 |
| sample_08 |
| table_bucket_dotted |
| table_key |
| table_key_bucket_dotted |
| table_volume |
| web_logs |
+--------------------------+
In Atlas, key based tables are created properly (eg table_key,
table_key_bucket_dotted). However, bucket/volume based tables do not show up in
Atlas
{code}
> Entities not created in Atlas when hive table created on top a
> OFS/O3FSbucket/volume
> ------------------------------------------------------------------------------------
>
> Key: ATLAS-5034
> URL: https://issues.apache.org/jira/browse/ATLAS-5034
> Project: Atlas
> Issue Type: Bug
> Reporter: Paresh Devalia
> Assignee: Aditya Gupta
> Priority: Major
> Attachments: Screenshot 2024-08-22 at 10.35.16 AM.png
>
>
> Facing issues when trying to create hive tables on top of ozone ofs
> buckets/volumes
> Steps to reproduce
> {code:java}
> # ozone sh volume create /volume1
> # ozone sh bucket create /volume1/bucket1
> # ozone sh bucket create /volume1/bucket1.bucket1.bucket1
> # beeline
> 0: jdbc:hive2://altas-hive> CREATE EXTERNAL TABLE table_volume(id int) row
> format delimited fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1';
> INFO : Compiling
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
> CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
> terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44);
> Time taken: 0.053 seconds
> INFO : Executing
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44):
> CREATE EXTERNAL TABLE table_volume(id int) row format delimited fields
> terminated by ' ' stored as textfile location 'ofs://ozone1724268703/volume1'
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822174025_98caa1a2-f814-4d83-9f9a-31b157a49a44);
> Time taken: 0.417 seconds
> INFO : OK
> No rows affected (0.544 seconds)
>
> 0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_bucket_dotted(id int)
> row format delimited fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1';
> INFO : Compiling
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
> CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
> terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa);
> Time taken: 0.182 seconds
> INFO : Executing
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa):
> CREATE EXTERNAL TABLE table_bucket_dotted(id int) row format delimited fields
> terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1'
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822172811_7868d63e-ff31-462b-bcce-7dd6fe9d49fa);
> Time taken: 0.465 seconds
> INFO : OK
> No rows affected (1.155 seconds)
>
> 0: jdbc:hive2://atlas-hive> CREATE EXTERNAL TABLE table_key_bucket_dotted(id
> int) row format delimited fields terminated by ' ' stored as textfile
> location 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2';
> INFO : Compiling
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
> CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
> fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421);
> Time taken: 0.026 seconds
> INFO : Executing
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421):
> CREATE EXTERNAL TABLE table_key_bucket_dotted(id int) row format delimited
> fields terminated by ' ' stored as textfile location
> 'ofs://ozone1724268703/volume1/bucket1.bucket1.bucket1/key2'
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822173256_3da9c7ff-76b8-4116-8b5b-929428b77421);
> Time taken: 0.226 seconds
> INFO : OK
> No rows affected (0.342 seconds)
>
> 0: jdbc:hive2://atlas-hive> show tables;
> INFO : Compiling
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
> type:string, comment:from deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515);
> Time taken: 0.045 seconds
> INFO : Executing
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20240822174219_dc6e9ada-4c30-4e82-84cb-d2c0ec071515);
> Time taken: 0.026 seconds
> INFO : OK
> +--------------------------+
> | tab_name |
> +--------------------------+
> | sample_07 |
> | sample_08 |
> | table_bucket_dotted |
> | table_key |
> | table_key_bucket_dotted |
> | table_volume |
> | web_logs |
> +--------------------------+
> In Atlas, key based tables are created properly (eg table_key,
> table_key_bucket_dotted). However, bucket/volume based tables do not show up
> in Atlas
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
