[
https://issues.apache.org/jira/browse/ATLAS-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Radhika Kundam updated ATLAS-4293:
----------------------------------
Description:
*Scenario-1:* Create external table pointing to external storage handler
hbase/kafka
In this case there is no entity created for the external hbase path, though
there is atlas entity for hbase table.Also no lineage present on atlas for this
table to external path.
Query for Hive Table with external HBase storage handler:
{code:java}
create table default.tbl29_hbs(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" = ":key,cf:name",
"hbase.table.name" = "default:hbs_tbl29"
);{code}
Below are the Alter table scenarios which are not propagated to Atlas
*Scenario-2:* SET SERDE
{code:java}
CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int,
d int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 61 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected
[org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-3:* SET FILEFORMAT TEXTFILE
{code:java}
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e
int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.788 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected
[org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found
[org.apache.hadoop.mapred.SequenceFileInputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-4:* SET FILEFORMAT SEQUENCEFILE
{code:java}
CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d
int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.796 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected
[org.apache.hadoop.mapred.SequenceFileOutputFormat] but found
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-5:* SET FILEFORMAT PARQUET
{code:java}
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d
int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.803 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected
[org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch
expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but
found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
mismatch expected
[org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found
[org.apache.hadoop.mapred.SequenceFileInputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-6:* ALTER TABLE using CLUSTERED BY
{code:java}
create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID
Int,FirstName String,Designation String, Salary Int,Department String)
clustered by (department) into 3 buckets stored as orc TBLPROPERTIES
('transactional'='true')
ALTER TABLE db2k9eooxraj5.cluster_by_k9eooxraj5 CLUSTERED BY (viewTime, userid)
SORTED BY (page_url) INTO 48 BUCKETS
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 61.25 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
bucketCols mismatch: lists don't have the same size expected [2] but found
[0], numBuckets mismatch expected [48] but found [-1]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
was:
*Scenario-1:* Create external table pointing to external storage handler
hbase/kafka
In this case there is no entity created for the external hbase path, though
there is atlas entity for hbase table.Also no lineage present on atlas for this
table to external path.
Query for Hive Table with external HBase storage handler:
create table default.tbl29_hbs(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" = ":key,cf:name",
"hbase.table.name" = "default:hbs_tbl29"
);
Below are the Alter table scenarios which are not propagated to Atlas
*Scenario-2:* SET SERDE
{code:java}
CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int,
d int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 61 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected
[org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-3:* SET FILEFORMAT TEXTFILE
{code:java}
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e
int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.788 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected
[org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found
[org.apache.hadoop.mapred.SequenceFileInputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-4:* SET FILEFORMAT SEQUENCEFILE
{code:java}
CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d
int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.796 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected
[org.apache.hadoop.mapred.SequenceFileOutputFormat] but found
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-5:* SET FILEFORMAT PARQUET
{code:java}
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d
int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.803 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected
[org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch
expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but
found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
mismatch expected
[org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found
[org.apache.hadoop.mapred.SequenceFileInputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
*Scenario-6:* ALTER TABLE using CLUSTERED BY
{code:java}
create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID
Int,FirstName String,Designation String, Salary Int,Department String)
clustered by (department) into 3 buckets stored as orc TBLPROPERTIES
('transactional'='true')
ALTER TABLE db2k9eooxraj5.cluster_by_k9eooxraj5 CLUSTERED BY (viewTime, userid)
SORTED BY (page_url) INTO 48 BUCKETS
Assert Error:
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 61.25 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
bucketCols mismatch: lists don't have the same size expected [2] but found
[0], numBuckets mismatch expected [48] but found [-1]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
> Table changes are not propagated to Atlas in Create Table with External
> Storage Handler and few scenarios of Alter table
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-4293
> URL: https://issues.apache.org/jira/browse/ATLAS-4293
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 2.1.0
> Reporter: Radhika Kundam
> Assignee: Radhika Kundam
> Priority: Major
>
> *Scenario-1:* Create external table pointing to external storage handler
> hbase/kafka
> In this case there is no entity created for the external hbase path, though
> there is atlas entity for hbase table.Also no lineage present on atlas for
> this table to external path.
> Query for Hive Table with external HBase storage handler:
> {code:java}
> create table default.tbl29_hbs(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = ":key,cf:name",
> "hbase.table.name" = "default:hbs_tbl29"
> );{code}
>
> Below are the Alter table scenarios which are not propagated to Atlas
> *Scenario-2:* SET SERDE
> {code:java}
> CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c
> int, d int, e int, f int) ROW FORMAT
> SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as
> SequenceFile
> ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE
> 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
>
> Assert Error:
>
> alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
> Time elapsed: 61 sec <<< FAILURE!
> java.lang.AssertionError: The following asserts failed:
> serdeInfo serializationLib mismatch expected
> [org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found
> [org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
> at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
> at
> org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
>
> *Scenario-3:* SET FILEFORMAT TEXTFILE
> {code:java}
> CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int,
> e int, f int) ROW FORMAT
> SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as
> SequenceFile
> ALTER TABLE db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
> Assert Error:
>
> alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
> Time elapsed: 60.788 sec <<< FAILURE!
> java.lang.AssertionError: The following asserts failed:
> outputFormat mismatch expected
> [org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found
> [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
> mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found
> [org.apache.hadoop.mapred.SequenceFileInputFormat]
> at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
> at
> org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
>
> *Scenario-4:* SET FILEFORMAT SEQUENCEFILE
> {code:java}
> CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d
> int, e int, f int) ROW FORMAT
> SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as
> SequenceFile
> ALTER TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT
> SEQUENCEFILE
>
> Assert Error:
>
> alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
> Time elapsed: 60.796 sec <<< FAILURE!
> java.lang.AssertionError: The following asserts failed:
> outputFormat mismatch expected
> [org.apache.hadoop.mapred.SequenceFileOutputFormat] but found
> [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
> at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
> at
> org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
> *Scenario-5:* SET FILEFORMAT PARQUET
> {code:java}
> CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d
> int, e int, f int) ROW FORMAT
> SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as
> SequenceFile
> ALTER TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
>
> Assert Error:
>
> alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
> Time elapsed: 60.803 sec <<< FAILURE!
> java.lang.AssertionError: The following asserts failed:
> serdeInfo serializationLib mismatch expected
> [org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found
> [org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch
> expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but
> found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat],
> inputFormat mismatch expected
> [org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found
> [org.apache.hadoop.mapred.SequenceFileInputFormat]
> at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
> at
> org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
> *Scenario-6:* ALTER TABLE using CLUSTERED BY
> {code:java}
> create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID
> Int,FirstName String,Designation String, Salary Int,Department String)
> clustered by (department) into 3 buckets stored as orc TBLPROPERTIES
> ('transactional'='true')
> ALTER TABLE db2k9eooxraj5.cluster_by_k9eooxraj5 CLUSTERED BY (viewTime,
> userid) SORTED BY (page_url) INTO 48 BUCKETS
>
> Assert Error:
>
> alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
> Time elapsed: 61.25 sec <<< FAILURE!
> java.lang.AssertionError: The following asserts failed:
> bucketCols mismatch: lists don't have the same size expected [2] but found
> [0], numBuckets mismatch expected [48] but found [-1]
> at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
> at
> org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
