context_ngrams() UDAF for estimating top-k contextual n-grams
-------------------------------------------------------------
Key: HIVE-1518
URL: https://issues.apache.org/jira/browse/HIVE-1518
Project: Hadoop Hive
Issue Type: New Feature
Components: Query Processor
Affects Versions: 0.6.0
Reporter: Mayank Lahiri
Assignee: Mayank Lahiri
Fix For: 0.6.0
Create a new context_ngrams() function that generalizes the ngrams() UDAF to
allow the user to specify context around n-grams. The analogy is
"fill-in-the-blanks", and is best illustrated with an example:
SELECT context_ngrams(sentences(tweets), array("i", "love", null), 300) FROM
twitter;
will estimate the top-300 words that follow the phrase "i love" in a database
of tweets. The position of the null(s) specifies where to generate the n-gram
from, and can be placed anywhere. For example:
SELECT context_ngrams(sentences(tweets), array("i", "love", null, "but",
"hate", null), 300) FROM twitter;
will estimate the top-300 word-pairs that fill in the blanks specified by null.
POSSIBLE USES:
1. Pre-computing search lookaheads
2. Sentiment analysis for products or entities -- e.g., querying with context =
array("twitter", "is", null)
3. Navigation path analysis in URL databases
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
