hi ravindra,
Got your point. As i had replied to xuchuyain. We can take these index
datamap enhancement separately.
Thank you
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
+1 for ravin's advice.
We only support lazy/incremental load/rebuild for olap datamap (MV/preagg),
not for index datamap currently.
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Hi Akash,
There is a difference between index datamap (like bloom) and olap datamaps
(like MV). Index datamaps used only for pruning the data while olap datamaps
will be used as pre-computed data which can be fetched directly as per
query.
In OLAP datamap case lazy build or deferred build makes
i got your point. If each segment has status file as i said, we can do
pruning without rebuild also. But need to get others suggestion on this
point. So may be we can take up this in another JIRA and track. In this jira
we can just suppport incremental data load.
--
Sent from:
http://apache-car
I think there is still misunderstanding between us.
Here I only concern about the lazy build for index datamap.
I think each segment should has its own datamap status and based on this we
can support pruning by index datamap for each segment. After this, even the
datamap is lazy, during query we
I agree with you that the index created for old segments will be of no use if
rebuild is not happened and these are not considered in query for pruning.
But we go for datamap pruning (index) based on the datamap status. Status
will be just enabled or disabled. You cannot maintain status for each
s
“For index datamap we can have same behavior as mv datamap only”
===
Emm, That’s just what I am concerned. In your plan, if the index datamap is
lazy, each time after a dataload is completed, this datamap will be ignored
before rebuild index is fired for this segment, even though all the index da
Hi xuchuanyin,
For index datamap we can have same behavior as mv datamap only, but it might
behave differently in case of lucene. This we can decide whether to enable
or lazy load or not.
Currently mv behavior is below
it supports only lazy load. So when the main table data and the datamap
dat
