[jira] [Created] (CARBONDATA-4207) MV data getting lost
suyash yadav created CARBONDATA-4207: Summary: MV data getting lost Key: CARBONDATA-4207 URL: https://issues.apache.org/jira/browse/CARBONDATA-4207 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Reporter: suyash yadav Fix For: 2.0.1 Hi Team, We have observed one more issue, We had created one table and a timeseries MV on it. We had loaded almost 15 hours of data into it and then when we were loading 16th hour data the loading failed because of some reason but it caused MV to go empty. Our mv has now zero rows. Could you please let us know if there is any bug or this is how it is supposed to work. Because our MV did not have any avg function so ideally the loading to MV should have been incremental , and in that case MV should not have got impacted if the subsequent hour loading to main table failed. Please have a look into this issue. And let us know what information you need. scala> spark.sql("insert into Flow_TS_2day_stats_04062021 select start_time,end_time,source_ip_address,destintion_ip_address,appname,protocol_id,source_tos,src_as,dst_as,source_mask,destination_mask,dst_tos,input_pkt,output_pkt,input_byt,output_byt,source_port,destination_port,in_interface,out_interface from Flow_TS_1day_stats_24052021 where start_time>='2021-03-04 07:00:00' and start_time< '2021-03-04 09:00:00'").show() [1:38|https://carbondataworkspace.slack.com/archives/D01GLHKSAFL/p1623226096008700] scala> spark.sql("insert into Flow_TS_2day_stats_04062021 select start_time,end_time,source_ip_address,destintion_ip_address,appname,protocol_id,source_tos,src_as,dst_as,source_mask,destination_mask,dst_tos,input_pkt,output_pkt,input_byt,output_byt,source_port,destination_port,in_interface,out_interface from Flow_TS_1day_stats_24052021 where start_time>='2021-03-04 15:00:00' and start_time< '2021-03-04 16:00:00'").show() 21/06/06 14:25:33 AUDIT audit: \{"time":"June 6, 2021 2:25:33 PM IST","username":"root","opName":"INSERT INTO","opId":"4069819623887063","opStatus":"START"} 21/06/06 14:44:14 AUDIT audit: \{"time":"June 6, 2021 2:44:14 PM IST","username":"root","opName":"INSERT INTO","opId":"4070940294400824","opStatus":"START"} 21/06/06 16:06:05 AUDIT audit: \{"time":"June 6, 2021 4:06:05 PM IST","username":"root","opName":"INSERT INTO","opId":"4070940294400824","opStatus":"SUCCESS","opTime":"4911240 ms","table":"default.Interface_Level_Agg_10min_MV_04062021","extraInfo":{"SegmentId":"6","DataSize":"4.52GB","IndexSize":"108.27KB"}} 21/06/06 16:06:09 AUDIT audit: \{"time":"June 6, 2021 4:06:09 PM IST","username":"root","opName":"INSERT INTO","opId":"4069819623887063","opStatus":"SUCCESS","opTime":"6036073 ms","table":"default.flow_ts_2day_stats_04062021","extraInfo":{"SegmentId":"6","DataSize":"12.37GB","IndexSize":"262.43KB"}}[^Stack_Trace] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4205) MINOR compaction getting triggered by it self while inserting data to a table
suyash yadav created CARBONDATA-4205: Summary: MINOR compaction getting triggered by it self while inserting data to a table Key: CARBONDATA-4205 URL: https://issues.apache.org/jira/browse/CARBONDATA-4205 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Environment: apache carbondata 2.0.1, hadoop 2.7.2, spark 2.4.5 Reporter: suyash yadav Hi Team we have created a table and also created a timeseries MV on it. Later we tried to insert a some data from other table to this newly created table but we have observed that while inserting ...MINOR compaction on the MV is getting triggered by it self. It doesn't happen for all the insert but whnever we insert 6 to 7th hour data and then 14 to 15 hour datathe MINOR compaction gets triggered. Could you tell us why the MINOR compaction is getting triggered by it self. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4182) Performance issue when multiple load happeneing to same table for same interval with 2 MVs.
suyash yadav created CARBONDATA-4182: Summary: Performance issue when multiple load happeneing to same table for same interval with 2 MVs. Key: CARBONDATA-4182 URL: https://issues.apache.org/jira/browse/CARBONDATA-4182 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.1.0 Environment: Apache carbon 2.1.0 Reporter: suyash yadav Hi Team, We need your help to resolve one of the performance issue that we are facing. Please see below details about the table structure and schema implemented at our end: 1.We have 25 tables and 2 MVS created for these tables for hour and day granularity. 2.One table can have more than 1 can for same interval and whenver multiple csv are there as an input to these tables then sequential loading will take place. 3.for different tables data loading is parallal but whenver 2 csvs are there for same table then for that table sequential load will happen by one and then other csv respectivly. 4.We have observed for 1minute of csv to be loaded into the table it is taking approximalty 345 to 60 minutes which is creating a huge backlog. We need your help to resolve this performanec issue as there will be no use of 15 minutes of data will take more than 15 minutes to load. Users are not going to wait and will get fed up by this slowness. Kindly advice. Regards Suyash Yadav -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4177) performence issue with Query
suyash yadav created CARBONDATA-4177: Summary: performence issue with Query Key: CARBONDATA-4177 URL: https://issues.apache.org/jira/browse/CARBONDATA-4177 Project: CarbonData Issue Type: Bug Components: core Affects Versions: 2.0.1 Reporter: suyash yadav Fix For: 2.0.1 Hi Team,Hi Team, We are working on a POC using carbondata 2.0.1 and have come across parformance issue.Below are the details: 1.Table creation query: == spark.sql("create table Flow_TS_1day_stats_16042021(start_time timestamp,end_time timestamp,source_ip_address string,destintion_ip_address string,appname string,protocol_name string,source_tos smallint,in_interface smallint,out_interface smallint,src_as bigint,dst_as bigint,source_mask smallint,destination_mask smallint, dst_tos smallint,input_pkt bigint,input_byt bigint,output_pkt bigint,output_byt bigint,source_port int,destination_port int) stored as carbondata TBLPROPERTIES ('local_dictionary_enable'='false')").show() TWO MVs are there on this table, Below are the queries for those MVs :=== 1. Network MV spark.sql("create materialized view Network_Level_Agg_10min_MV_with_ip_15042021_again as select timeseries(end_time,'ten_minute') as end_time,source_ip_address, destintion_ip_address,appname,protocol_name,source_port,destination_port,source_tos,src_as,dst_as,sum(input_pkt) as input_pkt,sum(input_byt) as input_byt,sum(output_pkt) as output_pkt,sum(output_byt) as output_byt from Flow_TS_1day_stats_15042021_again group by timeseries(end_time,'ten_minute'),source_ip_address,destintion_ip_address, appname,protocol_name,source_port,destination_port,source_tos,src_as,dst_as order by input_pkt,input_byt,output_pkt,output_byt desc").show(false) 2. Interfae MV: ==Interface :== spark.sql("create materialized view Interface_Level_Agg_10min_MV_16042021 as select timeseries(end_time,'ten_minute') as end_time, source_ip_address,destintion_ip_address,appname,protocol_name,source_port,destination_port,source_tos,src_as,dst_as,in_interface,out_interface,sum(input_pkt) as input_pkt,sum(input_byt) as input_byt,sum(output_pkt) as output_pkt,sum(output_byt) as output_byt from Flow_TS_1day_stats_16042021 group by timeseries(end_time,'ten_minute'), source_ip_address,destintion_ip_address,appname,protocol_name,source_port,destination_port,source_tos,src_as,dst_as,in_interface,out_interface order by input_pkt,input_byt,output_pkt,output_byt desc").show(false) +*We are firing below query for fethcing data which is taking almost 10 seconds:*+ *Select appname,input_byt from Flow_TS_1day_stats_16042021 where end_time >= '2021-03-02 00:00:00' and end_time < '2021-03-03 00:00:00' group by appname,input_byt order by input_byt desc LIMIT 10* The above query is only fetching 10 records but it is taking almost 10 seconds to complete. Could you please review above schemas and help us to understand how can we get some improvement in the qury execution time. We are expectingt he response should be in subseconds. Table Name : RAW Table (1 Day - 300K/Sec)#Records : 2592000 RegardsSuyash Yadav -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4165) Carbondata summing up two values of same timestamp.
suyash yadav created CARBONDATA-4165: Summary: Carbondata summing up two values of same timestamp. Key: CARBONDATA-4165 URL: https://issues.apache.org/jira/browse/CARBONDATA-4165 Project: CarbonData Issue Type: Wish Components: core Affects Versions: 2.0.1 Environment: apache carbondata 2.0.1, apache spark 2.4.5 hadoop 2.7.2 Reporter: suyash yadav Fix For: 2.0.1 Hi Team, We have seen a behaviour while using Carbondata 2.0.1 that if we get 2 values for same timestamp then it tries to sum both the values and put it as one value. Instead we need that it should discard previous value and use the latest one. Please let us know if there is any functionality already available in carbondata to handle duplicate values by it self or if there is any plan to implement such a functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4150) Information about indexed datamap
suyash yadav created CARBONDATA-4150: Summary: Information about indexed datamap Key: CARBONDATA-4150 URL: https://issues.apache.org/jira/browse/CARBONDATA-4150 Project: CarbonData Issue Type: Wish Components: core Affects Versions: 2.0.1 Environment: apache 2.0.1 spark 2.4.5 hadoop 2.7.2 Reporter: suyash yadav Fix For: 2.0.1 Hi Team, We would like to know detailed information about indexed datamap and possible use cases for this datamap. So please help us in getting answer to below queries:- 1) What is an indexed datamap and related use cases. 2) how it is to be used, 3) any reference documents -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4132) Numer of records not matching in MVs
suyash yadav created CARBONDATA-4132: Summary: Numer of records not matching in MVs Key: CARBONDATA-4132 URL: https://issues.apache.org/jira/browse/CARBONDATA-4132 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Environment: Apache carbondata 2.0.1 Reporter: suyash yadav Fix For: 2.0.1 Hi Team, We are working on a POC where we need to insert 300k records/second in a table where we have already created Timeeries MVs with Minute,Hour,Day granularity. As per our the Minute based MV should contain 300K records till the insertion of next minute data. Also the hour and Day based MVs should contain 300K records till the arrival of next hour and next day data respectively. But The count of records in MV is not coming out as per our expectation.It is always more than our expectation. But the strange thing is, When we drop the MV and create the MV after inserting the data in the table then the count if reocrds comes correct.So it is clear there is no problem with MV definition and the data. Kindly help us in resolving this issue on priority.Please find more details below: Table definition: === spark.sql("create table Flow_Raw_TS(export_ms bigint,exporter_ip string,pkt_seq_num bigint,flow_seq_num int,src_ip string,dst_ip string,protocol_id smallint,src_tos smallint,dst_tos smallint,raw_src_tos smallint,raw_dst_tos smallint,src_mask smallint,dst_mask smallint,tcp_bits int,src_port int,in_if_id bigint,in_if_entity_id bigint,in_if_enabled boolean,dst_port int,out_if_id bigint,out_if_entity_id bigint,out_if_enabled boolean,direction smallint,in_octets bigint,out_octets bigint,in_packets bigint,out_packets bigint,next_hop_ip string,bgp_src_as_num bigint,bgp_dst_as_num bigint,bgp_next_hop_ip string,end_ms timestamp,start_ms timestamp,app_id string,app_name string,src_ip_group string,dst_ip_group string,policy_qos_classification_hierarchy string,policy_qos_queue_id bigint,worker_id int,day bigint ) stored as carbondata TBLPROPERTIES ('local_dictionary_enable'='false') MV definition: == +*Minute based*+ spark.sql("create materialized view Flow_Raw_TS_agg_001_min as select timeseries(end_ms,'minute') as end_ms,src_ip,dst_ip,app_name,in_if_id,src_tos,src_ip_group,dst_ip_group,protocol_id,bgp_src_as_num, bgp_dst_as_num,policy_qos_classification_hierarchy, policy_qos_queue_id,sum(in_octets) as octects, sum(in_packets) as packets, sum(out_packets) as out_packets, sum(out_octets) as out_octects FROM Flow_Raw_TS group by timeseries(end_ms,'minute'),src_ip,dst_ip,app_name,in_if_id,src_tos,src_ip_group, dst_ip_group,protocol_id,bgp_src_as_num,bgp_dst_as_num,policy_qos_classification_hierarchy, policy_qos_queue_id").show() +*Hour Based*+ val startTime = System.nanoTime spark.sql("create materialized view Flow_Raw_TS_agg_001_hour as select timeseries(end_ms,'hour') as end_ms,app_name,sum(in_octets) as octects, sum(in_packets) as packets, sum(out_packets) as out_packets, sum(out_octets) as out_octects, in_if_id,src_tos,src_ip_group, dst_ip_group,protocol_id,src_ip, dst_ip,bgp_src_as_num, bgp_dst_as_num,policy_qos_classification_hierarchy, policy_qos_queue_id FROM Flow_Raw_TS group by timeseries(end_ms,'hour'),in_if_id,app_name,src_tos,src_ip_group,dst_ip_group,protocol_id,src_ip, dst_ip,bgp_src_as_num,bgp_dst_as_num,policy_qos_classification_hierarchy, policy_qos_queue_id").show() val endTime = System.nanoTime val elapsedSeconds = (endTime - startTime) / 1e9d -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4120) select queries against carbondata tables getting stuck when fired through Apache Hive
suyash yadav created CARBONDATA-4120: Summary: select queries against carbondata tables getting stuck when fired through Apache Hive Key: CARBONDATA-4120 URL: https://issues.apache.org/jira/browse/CARBONDATA-4120 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Environment: Apache Hive 3.1.2, Apache carbondata 2.0.1 Reporter: suyash yadav Hi Team need one more help..We have created a table which has around 172 million records and we have connected this table through Apache Hive..but whenever we are running select count(*) on this table through hive, the query gets stuck.We can run the query successfully when we run it through spark shell but through Hive it is always getting stuck.One more observation is, Whenever we run any query which contains join the query gets stuck. Also for where clause the query gets executed with smaller table but when we run it against the bigger table, it also gets stuck. So could you giys guide us how can we run all these queries successfully without any issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4116) Concurrent Data Loading Issue
suyash yadav created CARBONDATA-4116: Summary: Concurrent Data Loading Issue Key: CARBONDATA-4116 URL: https://issues.apache.org/jira/browse/CARBONDATA-4116 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 1.6.1, 2.0.1 Environment: Apache carbondata 2.0.1 Reporter: suyash yadav Even carbon claim that it can support the concurrent data loading together with table compaction, in fact it cannot. We have faced data inconsistent issue in carbon 1.6.1 when loading data concurrently into the table together with compaction. That is why we implement table locking for load data, compact and clean files command. All this is due to the manipulation of the table’s metadata file, i.e .tablestatus. We are facing issue in concurrent data loading together with compaction - gets us to data inconsistency - what is the way out to fix this as we want concurrent loading with compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4108) How to connect carbondata with Hive
suyash yadav created CARBONDATA-4108: Summary: How to connect carbondata with Hive Key: CARBONDATA-4108 URL: https://issues.apache.org/jira/browse/CARBONDATA-4108 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Environment: apache carbondata 2.0.1, spark 2.4.5, Hive 2.0 Reporter: suyash yadav Fix For: 2.0.1 Hi Team, We would like to know how to connect hive with carbondata.We are doing a POC where in we need to access carbondata table through hive but we need this configuration with username and password. So our hive connection should have some username and password configuration to connect to carbondata tables. Could you guys please review above requirement and suggest steps to achieve the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4106) Compaction is not working properly
suyash yadav created CARBONDATA-4106: Summary: Compaction is not working properly Key: CARBONDATA-4106 URL: https://issues.apache.org/jira/browse/CARBONDATA-4106 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Environment: Apache spark 2.4.5, carbonData 2.0.1 Reporter: suyash yadav Fix For: 2.0.1 Attachments: describe_fact_probe_1 Hi Team, We are using apache carbondata 2.0.1 for one of our POC and we observed that we are not getting proper benifit from using compaction (Both majour and minor). Please find below details for the issue we are facing: *Name of the table used*: fact_365_1_probe_1 +*Number of rows:* + select count(*) from fact_365_1_probe_1 ++ |count(1)| ++ |76963753| *Sample data from the table:* == +---+--++--+-+---+ | ts| metric| tags_id| value| epoch| ts2| +---+--++--+-+---+ |2021-01-07 21:05:00|Probe.Duplicate.Poll.Count|c8dead9b-87ae-46ae-8703-bc2b7bfba5d4|39.611356797970274|1610033757768|2021-01-07 00:00:00| |2021-01-07 23:50:00|Probe.Duplicate.Poll.Count|62351ef2-f2ce-49d1-a2fd-a0d1e5f6a1b9| 72.70658115131307|1610043742516|2021-01-07 00:00:00| [^describe_fact_probe_1] I have attached the describe output which will show you the other details of the table. The size of the table is 3.24 GB and even after running minor or majour compaction the size remain almost the same. So we re not getting any benifit by running the compaction.Could you please review the shared details and help us in identifying if we are missing something here or is there any bug? Also we need answer to the following questions about carbondata storate: 1. In case of decimal values, how the storage behaves like if i have one row with 20 digits after decimal and second row has only 5 digits after decimal so how and what would be the difference in the storage taken. 2. My second question is , if i have two tables and one of the table has same values for 100 rows and other table has different values for 100 rows so how carbon will behave as far as the storage is concerned in this scenario. WHich table will take less storage or both will take same storage. 3.Also for string datatype could you please describe what is the storage defined for string datatype. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4085) How to improve query execution time further
suyash yadav created CARBONDATA-4085: Summary: How to improve query execution time further Key: CARBONDATA-4085 URL: https://issues.apache.org/jira/browse/CARBONDATA-4085 Project: CarbonData Issue Type: Improvement Components: sql Affects Versions: 2.0.1 Reporter: suyash yadav Fix For: 2.0.1 Hi Team, We are doing a POC where we would like oour query execution to be fatser, mostly in the range of 3 to 4 seconds. We have read carbon docuements where it has been claimed that carbondata can help to scan PETABYTES of data and present results in 3 to 4 seconds , which does not seem to be the case as per our observation. Our table size is 1.6 billionand query is fetching only 4K records but still it takes around 22 to 25 seconds for query execution. Below is our query that we are firing: == spark.sql("select ts,resource,metric,value from fact_timestamp_global left join tags_10_days_test on fact_timestamp_global.tags_id= tags_10_days_test.id where metric in ('Outbound Utilization (percent)','Inbound Utilization (percent)') and resource='10.212.7.98_if:<0001>' and ts>='2020-09-28 00:00:00' and ts<='2020-09-28 23:55:55'").show(false) = Definition of fact_timestamp_global is like below: spark.sql("create table Fact_timestamp_GLOBAL(ts timestamp,metric string,tags_id string,value double) partitioned by (ts2 timestamp) stored as carbondata TBLPROPERTIES ('SORT_COLUMNS'='ts,metric','SORT_SCOPE'='GLOBAL_SORT')").show() == Definition of tags_10_days_test is like below: spark.sql("create table tags_10_days_test(id string,resource string) stored as carbondata TBLPROPERTIES('SORT_COLUMNS'='id,resource')").show() == Kindly go through above points and help us the query performence further. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-4079) Queries with Date range are taking time
[ https://issues.apache.org/jira/browse/CARBONDATA-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247041#comment-17247041 ] suyash yadav commented on CARBONDATA-4079: -- Below are the complete background of the issue. We created two tables named as fact_10days_data_ts and tags_final and below are the description of these two tables. *Spark.sql("desc extended fact_timestamp_global").show(100,false)* col_name |data_type |comment| +-+-+---+ |ts |timestamp |null | |metric |string |null | |tags_id |string |null | |value |double |null | |ts2 |timestamp |null | | | | | |## Detailed Table Information| | | |Database |default | | |Table |fact_timestamp_global | | |Owner |root | | |Created |Wed Dec 09 19:31:48 MYT 2020 | | |Location |hdfs://10.212.5.2:7200/Data/default/fact_timestamp_global| | |External |false | | |Transactional |true | | |Streaming |false | | |Table Block Size |1024 MB | | |Table Blocklet Size |64 MB | | |Comment | | | |Bad Record Path | | | |Min Input Per Node Per Load |0.0B | | | | | | |## Index Information | | | |Sort Scope |global_sort | | |Sort Columns |ts, metric | | |Inverted Index Columns | | | |Cached Min/Max Index Columns |All columns | | |Min/Max Index Cache Level |BLOCK | | |Table page size in mb | | | | | | | |## Encoding Information | | | |Local Dictionary Enabled |true | | |Local Dictionary Threshold |1 | | |Local Dictionary Include |metric,tags_id | | | | | | |## Compaction Information | | | |MAJOR_COMPACTION_SIZE |1024 | | |AUTO_LOAD_MERGE |false | | |COMPACTION_LEVEL_THRESHOLD |4,3 | | |COMPACTION_PRESERVE_SEGMENTS |0 | | |ALLOWED_COMPACTION_DAYS |0 | | | | | | |## Partition Information | | | |Partition Type |NATIVE_HIVE | | |Partition Columns |ts2:TIMESTAMP
[jira] [Created] (CARBONDATA-4079) Queries with Date range are taking time
suyash yadav created CARBONDATA-4079: Summary: Queries with Date range are taking time Key: CARBONDATA-4079 URL: https://issues.apache.org/jira/browse/CARBONDATA-4079 Project: CarbonData Issue Type: Improvement Components: data-query Affects Versions: 2.1.0 Reporter: suyash yadav Hi Team, We are doing a POC to understand how can we improve performance of the query fired against table created in apache carbondata. Below is the sample query: *spark.sql("select ts,resource,metric,value from fact_timestamp_global left join tags_10_Days_test on fact_timestamp_global.tags_id= tags_10_Days_test.id where metric in ('Outbound Utilization (percent)','Inbound Utilization (percent)') and resource='10.212.7.98_if:<0001>' and ts between '2020-09-21 00:00:00' and '2020-09-21 12:55:55' group by ts,resource,metric,value").show(1,false)* As you can see above query contains the date range filter.We have noticed that due to this date range filter the query time is coming around 15 seconds which is not proving useful as we have to bring down the query execution time to 3 to 4 seconds. Could you please review above query and suggest a better way of framing the above query specially the date range filter which can be helpful to get the desired query execution time? In case you need more details then please do let me know. Regards Suyash Yadav -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-4025) storage space for MV is double to that of a table on which MV has been created.
[ https://issues.apache.org/jira/browse/CARBONDATA-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216618#comment-17216618 ] suyash yadav commented on CARBONDATA-4025: -- Hi Team, Can somebody look into this request? Regards Suyash Yadav > storage space for MV is double to that of a table on which MV has been > created. > --- > > Key: CARBONDATA-4025 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4025 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 2.0.1 > Environment: Apcahe carbondata 2.0.1 > Apache spark 2.4.5 > Hadoop 2.7.2 >Reporter: suyash yadav >Priority: Major > > We are doing a POC based on carbondata but we have observed that when we > create n MV on a table with timeseries function of same granualarity the MV > takes double the space of the table. > > In my scenario, My table has 1.3 million records and MV also has same number > of records but the size of the table is 3.6 MB but the size of the MV is > around 6.5 MB. > This is really important for us as critical business decision are getting > affected due to this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-4025) storage space for MV is double to that of a table on which MV has been created.
[ https://issues.apache.org/jira/browse/CARBONDATA-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213792#comment-17213792 ] suyash yadav commented on CARBONDATA-4025: -- Hi any update on this? > storage space for MV is double to that of a table on which MV has been > created. > --- > > Key: CARBONDATA-4025 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4025 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 2.0.1 > Environment: Apcahe carbondata 2.0.1 > Apache spark 2.4.5 > Hadoop 2.7.2 >Reporter: suyash yadav >Priority: Major > > We are doing a POC based on carbondata but we have observed that when we > create n MV on a table with timeseries function of same granualarity the MV > takes double the space of the table. > > In my scenario, My table has 1.3 million records and MV also has same number > of records but the size of the table is 3.6 MB but the size of the MV is > around 6.5 MB. > This is really important for us as critical business decision are getting > affected due to this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4025) strage size of MV is double to that of a table.
suyash yadav created CARBONDATA-4025: Summary: strage size of MV is double to that of a table. Key: CARBONDATA-4025 URL: https://issues.apache.org/jira/browse/CARBONDATA-4025 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 2.0.1 Environment: Apcahe carbondata 2.0.1 Apache spark 2.4.5 Hadoop 2.7.2 Reporter: suyash yadav We are doing a POC based on carbondata but we have observed that when we create n MV on a table with timeseries function of same granualarity the MV takes double the space of the table. In my scenario, My table has 1.3 million records and MV also has same number of records but the size of the table is 3.6 MB but the size of the MV is around 6.5 MB. This is really important for us as critical business decision are getting affected due to this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4025) storage space for MV is double to that of a table on which MV has been created.
[ https://issues.apache.org/jira/browse/CARBONDATA-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] suyash yadav updated CARBONDATA-4025: - Summary: storage space for MV is double to that of a table on which MV has been created. (was: strage size of MV is double to that of a table.) > storage space for MV is double to that of a table on which MV has been > created. > --- > > Key: CARBONDATA-4025 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4025 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 2.0.1 > Environment: Apcahe carbondata 2.0.1 > Apache spark 2.4.5 > Hadoop 2.7.2 >Reporter: suyash yadav >Priority: Major > > We are doing a POC based on carbondata but we have observed that when we > create n MV on a table with timeseries function of same granualarity the MV > takes double the space of the table. > > In my scenario, My table has 1.3 million records and MV also has same number > of records but the size of the table is 3.6 MB but the size of the MV is > around 6.5 MB. > This is really important for us as critical business decision are getting > affected due to this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3951) Documentation not available for upgrading carbondata from older to newer version
suyash yadav created CARBONDATA-3951: Summary: Documentation not available for upgrading carbondata from older to newer version Key: CARBONDATA-3951 URL: https://issues.apache.org/jira/browse/CARBONDATA-3951 Project: CarbonData Issue Type: Improvement Components: core Affects Versions: 1.6.1 Environment: RHEL 7.x Reporter: suyash yadav Fix For: 2.0.0 Hi Team, We are doing a POC where we want to upgrade carbondata inside our application from old version to latest version. Due to lack of documentation, We are not able to move ahead as we are not sure about he steps needs to be followed to perform the upgrade and then verify if it has been done successfully without any issue. We tried replacing jars with newer ones but we got below exception: Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.util.CarbonReflectionUtils$.getSessionState(CarbonReflectionUtils.scala:223) at org.apache.spark.sql.CarbonSession.sessionState$lzycompute(CarbonSession.scala:57) at org.apache.spark.sql.CarbonSession.sessionState(CarbonSession.scala:56) at org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:258) at org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:165) at persistent.diamond.Server.start(Server.java:176) at persistent.diamond.Server.main(Server.java:595) Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.AbstractSqlParser: method ()V not found at org.apache.spark.sql.parser.CarbonSparkSqlParser.(CarbonSparkSqlParser.scala:39) at org.apache.spark.sql.hive.CarbonSessionStateBuilder.sqlParser$lzycompute(CarbonSessionState.scala:204) at org.apache.spark.sql.hive.CarbonSessionStateBuilder.sqlParser(CarbonSessionState.scala:204) at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:292) ... 11 more Could you please help here to resolve above issue and clearly define the steps for upgrade. Regards Suyash Yadav -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3354) how to use filiters in datamaps
suyash yadav created CARBONDATA-3354: Summary: how to use filiters in datamaps Key: CARBONDATA-3354 URL: https://issues.apache.org/jira/browse/CARBONDATA-3354 Project: CarbonData Issue Type: Task Components: core Affects Versions: 1.5.2 Environment: apache carbon data 1.5.x Reporter: suyash yadav Fix For: NONE Hi Team, We are doing a POC on apache carbon data so that we can verify if this database is capable of handling amount of data we are collecting form network devices. We are stuck on few of our datamap related activities and have below queries: # How to use timiebased filters while creating datamap.We tried a time based condition while creating a datamap but it didn't work. # How to create a timeseries datamap with column which is having value of epoch time.Our query is like below:- *carbon.sql("CREATE DATAMAP test ON TABLE carbon_RT_test USING 'timeseries' DMPROPERTIES ('event_time'='endMs','minute_granularity'='1',) AS SELECT sum(inOctets) FROM carbon_RT_test GROUP BY inIfId")* # *In above query endMs is having epoch time value.* # We got an error like below: "Timeseries event time is only supported on Timestamp column" # Also we need to know if we can have a time granularity other then 1 like in above query, can we have minute_granularity='5*'.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)