Hi Liang,
yes, your right, I used the dictionary to do the limit and this solution
can’t make sure the data correctness.
But, I also done order by test on date field which is the first column of MDK
key, date field used the direct dictionary encode, it should can make sure the
data correctness.
after test , it also can improve more than 80％ performance
> 在 2017年9月29日，下午3:34，Liang Chen <chenliang6...@gmail.com> 写道：
>
> Hi Jarck
>
> Did this solution use dictionary to do limit , right ? this solution can't
> make sure the data correctness
> -------------------------------------------------------------------------------
> Use orderby +limit optimized carbondata1.2 master code + spark1.6.3
>
> @Ravindra @Jarck : let us discuss it, see if can find out the common
> solution,
> Proposal 1 : Implement TopN solution.
> Proposal 2 : Decoupled plug-in solution, continue to use Jarck's code,
> just need change order by from dictionary value to actual dimensional value.
>
> Regards
> Liang
>
> simafengyun wrote
>> Recently , I used the latest code done test as below
>> 1. Create Table:
>> CREATE TABLE rx5_tbox_parquet_all(
>> carid STRING,
>> inputstime TIMESTAMP,
>> carsyspwrmod INT,
>> cardofrontpas INT,
>> cardofrontdrv INT,
>> cardorearleft INT,
>> cardorearright INT,
>> carbonnet INT,
>> carboot INT,
>> carwinfrontleft INT,
>> carwinrearleft INT,
>> carwinfrontright INT,
>> carwinrearright INT,
>> carsunroof INT,
>> carcsactive INT,
>> carcsenabled INT,
>> carseatbeltdrv INT
>> )
>> STORED BY 'carbondata'
>> TBLPROPERTIES('SORT_COLUMNS'='carid',
>> 'DICTIONARY_INCLUDE'='carid')
>>
>> 2. Load 0.1 billion data
>>
>> 3. Run the below sql
>>
>> select
>> carid,
>> inputstime,
>> carsyspwrmod,
>> cardofrontpas,
>> cardofrontdrv,
>> cardorearleft,
>> cardorearright,
>> carbonnet,
>> carboot,
>> carwinfrontleft,
>> carwinrearleft,
>> carwinfrontright,
>> carwinrearright
>> from rx5_tbox_parquet_all2
>> order by carid
>> limit 10
>>
>>
>> Use carbondata1.2 master code + spark2.1 to run
>> |carid
>> |inputstime|carsyspwrmod|cardofrontpas|cardofrontdrv|cardorearleft|cardorearright|carbonnet|carboot|carwinfrontleft|carwinrearleft|carwinfrontright|carwinrearright|
>> +-----------------+--------+------------+---------------+---------------+---------------+----------------+---------+-------+------------------+-----------------+-------------------+------------------+
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> +-----------------+--------+------------+---------------+---------------+---------------+----------------+---------+-------+------------------+-----------------+-------------------+------------------+
>>
>> limit 10 query time: 28777 milliseconds
>>
>> Use orderby +limit optimized carbondata1.2 master code + spark1.6.3 to run
>>
>> +-----------------+--------+------------+---------------+---------------+---------------+----------------+---------+-------+------------------+-----------------+-------------------+------------------+
>> |carid
>> |inputstime|carsyspwrmod|cardofrontpas|cardofrontdrv|cardorearleft|cardorearright|carbonnet|carboot|carwinfrontleft|carwinrearleft|carwinfrontright|carwinrearright|
>> +-----------------+--------+------------+---------------+---------------+---------------+----------------+---------+-------+------------------+-----------------+-------------------+------------------+
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> |LSJA24790HS020662|null |2 |0 |0
>> |0
>> |0 |0 |0 |0 |0
>>
>> |0 |0 |
>> +-----------------+--------+------------+---------------+---------------+---------------+----------------+---------+-------+------------------+-----------------+-------------------+------------------+
>>
>> limit 10 query time: 1640 milliseconds
>>
>> Apparently, after optimization, even I use spark1.6.3, it also improved
>> 90%
>> performance
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Thanks
>> 马云
>>
>>
>>
>>
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> <http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/>
