SHREELEKHYA GAMPA created CARBONDATA-4282: ---------------------------------------------
Summary: Issues with table having complex columns related to long string, SI, local dictionary Key: CARBONDATA-4282 URL: https://issues.apache.org/jira/browse/CARBONDATA-4282 Project: CarbonData Issue Type: Bug Reporter: SHREELEKHYA GAMPA *1. Insert/load fails after alter add complex column if table contains long string columns.* [Steps] :- DROP TABLE IF EXISTS alter_com; CREATE TABLE alter_com(intfield int,EDUCATED string ,rankk string ) STORED AS carbondata TBLPROPERTIES('inverted_index'='intField','sort_columns'='intField','TABLE_BLOCKSIZE'= '256 MB','TABLE_BLOCKLET_SIZE'='8','SORT_SCOPE'='no_sort','COLUMN_META_CACHE'='rankk','carbon.column.compressor'='gzip','long_string_columns'='rankk','table_page_size_inmb'='1'); insert into alter_com values(1,'cse','xi'); select * from alter_com limit 1; ALTER TABLE alter_com ADD COLUMNS(map1 Map<short,int>, map2 Map<long,double>, map3 Map<decimal(3,2),string>, map4 Map<char(5),varchar(50)>, map5 Map<boolean,date>,map6 Map<string,timestamp>,map7 map<string,array<string>>, map8 map<string,struct<d:int, s:struct<im:string>>>); ALTER TABLE alter_com SET TBLPROPERTIES('long_string_columns'='EDUCATED'); insert into alter_com values(1,'ece','x', map(1,2),map(3,2.34), map(1.23,'hello'),map('abc','def'), map(true,'2017-02-01'),map('time','2018-02-01 02:00:00.0'),map('ph',array(1,2)), map('a',named_struct('d',23,'s',named_struct('im','sh')))); [Expected Result] :- insert/load should be success after alter add map column ,if table contains long string columns *2. create index on array of complex column (map/struct) throws null pointer exception instead of correct error message.* [Steps] :- drop table if exists strarmap1; create table strarmap1(id int,name string,str struct<intval:int,tm:timestamp,map1:map<int,string>>,arr array<map<string,timestamp>>) stored as carbondata tblproperties('inverted_index'='name','sort_columns'='name','TABLE_BLOCKSIZE'= '256 MB','TABLE_BLOCKLET_SIZE'='8','CACHE_LEVEL'='BLOCKLET'); load data inpath 'hdfs://hacluster/chetan/strarmap1.csv' into table strarmap1 options('fileheader'='id,name,str,arr','COMPLEX_DELIMITER_LEVEL_3'='#','COMPLEX_DELIMITER_LEVEL_2'='$','COMPLEX_DELIMITER_LEVEL_1'='&','BAD_RECORDS_ACTION'='FORCE'); CREATE INDEX index2 ON TABLE strarmap1 (arr) as 'carbondata' properties('sort_scope'='global_sort','global_sort_partitions'='3'); [Expected Result] :- create index on array of map(string,timestamp) should thrown correct validation error message. [Actual Issue]:- create index on array of map(string,timestamp) throws null pointer exception instead of correct error message *3. alter table property local dictionary inlcude/exclude with newly added map column is failing.* [Steps] :- drop table if exists strarmap1; create table strarmap1(id int,name string,str struct<intval:int,tm:timestamp,map1:map<int,string>>,arr array<map<string,timestamp>>) stored as carbondata tblproperties('inverted_index'='name','sort_columns'='name','local_dictionary_enable'='false','local_dictionary_include'='map1','local_dictionary_exclude'='str,arr','local_dictionary_threshold'='1000'); ALTER TABLE strarmap1 ADD COLUMNS(map1 Map<short,int>, map2 Map<long,double>, map3 Map<decimal(3,2),string>, map4 Map<char(5),varchar(50)>, map5 Map<boolean,date>,map6 Map<string,timestamp>,map7 map<string,array<string>>, map8 map<string,struct<d:int, s:struct<im:string>>>); ALTER TABLE strarmap1 SET TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='true','local_dictionary_include'='map4','local_dictionary_threshold'='1000'); [Expected Result] :- alter table property local dictionary inlcude/exclude with newly added map column should be success [Actual Issue]:- alter table property local dictionary inlcude/exclude with newly added map column is failing -- This message was sent by Atlassian Jira (v8.3.4#803005)