Re: Refactored SegmentPropertiesFetcher to resolve pruning problem post the carbon schema restructure.

2018-05-03 Thread xuchuanyin
Will delete/update affect the schema? What's the meaning of 'schema' here? -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

?????? can CarbonThriftServer configure the max number of submit task at the same time?

2018-05-03 Thread ??????
yes, this config works. thanks. My sql is a very big sql?? so 60 concurrent queries does not work. -- -- ??: "xuchuanyin"; : 2018??5??3??(??) 10:41 ??: "dev"; : Re: can CarbonThriftServer configure the max number of submit

Re: Create Carbon data with complex data type (Array )

2018-05-03 Thread xuchuanyin
Yes, it is really a bug. You can raise a jira for this problem. I tried the following queries and they are OK. Hope it will help you bypass the bug. ``` create table IF NOT EXISTS test.Account(CAP_CHARGE Array,CAP_CR_INT Array) partitioned by (current_dt DATE) STORED BY 'carbondata' ``` ``` cr

Re: Grammar about supporting string longer than 32000 characters

2018-05-03 Thread xuchuanyin
Spark dataframe does not have char/varchar datatype, it only has string type. Suppose we are saving spark dataframe to carbonata, if the length of value is longer than 32000 and its datatype is string, how will carbondata know to store it as longstring? In my opinion, a property to indicate whi

Re: Grammar about supporting string longer than 32000 characters

2018-05-03 Thread xm_zzc
+1 for solution 2 I think it's ok to truncate the string value to N if its length is longer than N, it's acceptable. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Grammar about supporting string longer than 32000 characters

2018-05-03 Thread xuchuanyin
In traditional RDBMS, varchar(N) means the value contains at least N characters, at the DBMS will truncate the value if its length is longer than N. Will we implement like this too? Truncate the string value to N if its length is longer than N?

Re: can CarbonThriftServer configure the max number of submit task at the same time?

2018-05-03 Thread xuchuanyin
Maybe you can try `hive.server2.thrift.max.worker.threads` and set a smaller value for it. You can configure it in hive-site.xml or pass the configuration through ??hiveconf when you start the thrift-server. At last, you need to find out the root cause of the failed sqls. 60 concurrent queries

Create Carbon data with complex data type (Array )

2018-05-03 Thread Raviteja
HI I m Getting an error when i run the below query *create table IF NOT EXISTS test.Account(CAP_CHARGE Array,CAP_CR_INT Array) partitioned by (current_dt DATE) STORED BY 'carbondata'* ERROR LOG 18/05/03 18:51:13 ERROR TableNewProcessor$: main Duplicate column found with name: val 18/05/03 18:5

Create Carbon data with complex data type (Array )

2018-05-03 Thread Raviteja
HI I m Getting an error when i run the below query *create table IF NOT EXISTS test.Account(CAP_CHARGE Array,CAP_CR_INT Array) partitioned by (current_dt DATE) STORED BY 'carbondata'* ERROR LOG 18/05/03 18:51:13 ERROR TableNewProcessor$: main Duplicate column found with name: val 18/05/03 18:5

Refactored SegmentPropertiesFetcher to resolve pruning problem post the carbon schema restructure.

2018-05-03 Thread mohdshahidkhan
Hi Dev, Currently DataMap while pruning is considering SegmentProperties from the 0th block of Segment DataMap. It is not correct as post schema restructure if the table is updated (*Update/Delete record*) then within the segment, all the block will not have symmetric schema. To support schema

Re: Grammar about supporting string longer than 32000 characters

2018-05-03 Thread Raghunandan S
+1 for solution 2 On Wed, 2 May 2018 at 9:09 PM, ravipesala wrote: > Hi, > > I agree with option 2 but not new datatype use varchar(size). > There are more optimizations we can do with varchar(size) datatype like > 1. if the size is smaller (less than 8 bytes) then we can write in fixed > length

can CarbonThriftServer configure the max number of submit task at the same time?

2018-05-03 Thread ??????
hi dev, I start CarbonThriftServer, then use beeline to connect it. But when I submit 60 sqls at the same time, all task failed. So I want to know, is there a configure to restrict the max number of running sql in carbonthriftserver, for example, if the number of running sql is larger than this