[
https://issues.apache.org/jira/browse/HIVE-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125502#comment-13125502
]
John Sichi commented on HIVE-1496:
----------------------------------
Discussion from IRC:
{noformat}
ssalbiz: jsichi: I was looking at ashutosh 's patch for 1496, and I was
wondering if the problem with it the lack of atomicity? It seems to chain the
map-red tasks to populate the index with the DDLTask correctly if I'm reading
the patch/testcase right
[4:37pm] jsichi: ssalbiz: you're right; I misread the patch--didn't notice the
addIdxTasks part. ashutosh, sorry about that.
[4:40pm] jsichi: but I don't think we should be calling db.createIndex directly
from DDLSemanticAnalyzer...should still be chaining in the DDLWork for that
[4:41pm] ssalbiz: right, I agree
[4:43pm] ashutosh: jsichi: I rememeber you commenting on the jira that
atomicity will be an issue, but its ok to solve it seperately in a followup work
[4:44pm] jsichi: ashutosh: agreed. But we should still be following the usual
pattern for executing the metastore update from within a task (rather than
analyzer)
[4:45pm] jsichi: Another followup is to support a mode whereby updates to a
table trigger a rebuild of the corresponding index partitions.
[4:47pm] jsichi: I guess the reason you had to do it early (in the analyzer) is
that the build-task generation requires the metastore to already be populated.
[4:47pm] ashutosh: yeah.. correct thats the reason
[4:47pm] jsichi: hmmm
[4:48pm] ashutosh: build-task assumes all the data to be populated
[4:55pm] jsichi: I guess the only way to resolve that would be to factor out
the code that knows how to make up an Index object from a CreateIndexDesc, and
then create a temp during analysis (then discard it), then later create the
real one when the task executes.
[4:56pm] ashutosh: yeah.. i think temp object approach may work
[4:57pm] ashutosh: but probably will churn around lot of code
[4:58pm] jsichi: having EXPLAIN able to show what's gonna be done without
actually doing it seems like a valuable guarantee to preserve
[4:58pm] jsichi: (see e.g. HIVE-2478)
[5:01pm] ashutosh: ⁃actually temp obj approach may not work because createIndex
task connects to metastore to get the metadata … so it must exist in metastore
[5:02pm] jsichi: that's only for verifying that the index name does not
conflict, rigth?
[5:05pm] jsichi: woohoo, finally gonna get a clean trunk build on Jenkins since
I'm about to commit HIVE-2493!
[5:05pm] ashutosh: awesome
[5:05pm] jsichi: already got a clean run on 0.8
[5:07pm] ashutosh: for me HBase tests always fail
[5:07pm] ashutosh: with exception NoRegionServerFound exception
[5:07pm] ashutosh: whats the magic there ?
[5:08pm] ssalbiz: looking at the TableBasedIndexHandler code, it does a bunch
of checks to ensure that the partition specs of the table and index are in sync
in the metastore. I think it would be possible to write a helper method in
TableBasedIndexHandler that can be used to generate Index Map-Red Tasks without
relying on the metastore at all if we can assume that the Index ms partition
spec and the base table partition spec are going to be consistent when the da
[5:09pm] ssalbiz: seems like less code churn than trying to feed the current
method mock metastore/Index objects to make those checks pass
[5:10pm] jsichi: ashutosh: hmm, dunno...I'll bet there's a real exception
buried somewhere deep in the logs...
[5:10pm] jsichi: ssalbiz: I don't think we want to change the index handler
interface though
[5:15pm] ssalbiz: hmmm, ok, in that case I guess we will have to feed temp
objects to the existing generateIndexBuildTaskList method
[5:18pm] ashutosh: one possibility to avoid explain problem is to not execute
ms operation in semantic analyzer if its an explain query
[5:19pm] jsichi: There's already precedent for the temp objects....e.g.
Hive.createIndex already calls indexHandler.analyzeIndexDefinition with
indexDesc and tt params which haven't actually been written to the metastore
yet.
[5:20pm] jsichi: ashutosh: that wouldn't work, would it, since the explain is
supposed to show the index build tasks too
[5:22pm] ashutosh: hmm.. right.. it wont show it.. but will atleast prevent
execution of unwanted ms operation in case of explain ...
[5:22pm] ashutosh: i can take a look at temp object approach
[5:23pm] jsichi: ssalbiz is actually going to work on indexing again as part of
a school project, so if you're OK with it, we can assign back to him.
[5:25pm] ashutosh: ya.. thats fine..
[5:25pm] ashutosh: he can take it up
[5:25pm] jsichi: OK cool. I'll copy-paste this conversation into JIRA in case
we forget anything later.
{noformat}
> enhance CREATE INDEX to support immediate index build
> -----------------------------------------------------
>
> Key: HIVE-1496
> URL: https://issues.apache.org/jira/browse/HIVE-1496
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
> Affects Versions: 0.7.0, 0.8.0
> Reporter: John Sichi
> Assignee: Syed S. Albiz
> Attachments: hive-1496.patch
>
>
> Currently we only support WITH DEFERRED REBUILD.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
