Hi Ravindran,
Thanks for your quick response. please see my answer as below
What if the order by column is not the first column? It needs to scan all
blocklets to get the data out of it if the order by column is not first column
of mdk
>>>
Hi Jacky,
Please see my comments below;
1. In this phase, is it still using columnar format? Save to a file for
every mini batch? If so, it is only readable after the file has been closed
and some metadata need to be kept to indicate the availability of the new
file.
AA >> yes, for initial phase
How to synchronize carbondata data with hive
Hi Jarck Ma,
It is great to try optimizing Carbondata.
I think this solution comes up with many limitations. What if the order by
column is not the first column? It needs to scan all blocklets to get the
data out of it if the order by column is not first column of mdk.
We used to have multiple pu
Hi Aniket,
This feature looks great, the overall plan also seems fine to me. Thanks for
proposing it.
And I have some doubts inline.
> 在 2017年3月27日，下午6:34，Aniket Adnaik 写道：
>
> Hi All,
>
> I would like to open up a discussion for new feature to support streaming
> ingestion in CarbonData.
>
[ERROR]
/usr/local/soft/incubator-carbondata-CARBONDATA-727/integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java:[87,25]
cannot find symbol
symbol: method getFilterPredicates(org.apache.hadoop.conf.Configuration)
location: class org.apache.carbondata.hive.M
Hi Carbon Dev,
I create table according to the below SQL
cc.sql("""
CREATE TABLE IF NOT EXISTS t3
(ID Int,
date Timestamp,
country String,
name String,
phonetype String,
serialname String,
salary Int,
name1 String,
name2 String,
name3 String,
name4 Stri
Thank you very much!
I have divided 2 billions data into 4 pieces and loaded in the table 。
The three paramaters carbon.graph.rowset.size、 carbon.sort.size
、carbon.number.of.cores.while.loading may be also effect。
Best regards！
At 2017-03-27 13:53:58, "Liang Chen" wrote:
>Hi
>
>1.Use your
Hi Carbon Dev,
currently I have done optimization for ordering by 1 dimension.
my local performance test as below. Please give your suggestion.
my optimization solution for order by 1 dimension as below
mainly filter some unnecessary blocklets and leverage the dimension's order
stored feature
