[ https://issues.apache.org/jira/browse/HIVE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898421#action_12898421 ]
HBase Review Board commented on HIVE-1518: ------------------------------------------ Message from: "John Sichi" <jsi...@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/644/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- review by JVS This addresses bug HIVE-1518. http://issues.apache.org/jira/browse/HIVE-1518 Diffs ----- http://svn.apache.org/repos/asf/hadoop/hive/trunk/data/files/text-en.txt 985013 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 985013 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 985013 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/test/queries/clientpositive/udaf_context_ngrams.q PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/test/queries/clientpositive/udaf_ngrams.q 985013 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out 985013 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/test/results/clientpositive/udaf_context_ngrams.q.out PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/test/results/clientpositive/udaf_ngrams.q.out 985013 Diff: http://review.cloudera.org/r/644/diff Testing ------- Thanks, John > context_ngrams() UDAF for estimating top-k contextual n-grams > ------------------------------------------------------------- > > Key: HIVE-1518 > URL: https://issues.apache.org/jira/browse/HIVE-1518 > Project: Hadoop Hive > Issue Type: New Feature > Components: Query Processor > Affects Versions: 0.6.0 > Reporter: Mayank Lahiri > Assignee: Mayank Lahiri > Fix For: 0.7.0 > > Attachments: HIVE-1518.1.patch, HIVE-1518.2.patch > > > Create a new context_ngrams() function that generalizes the ngrams() UDAF to > allow the user to specify context around n-grams. The analogy is > "fill-in-the-blanks", and is best illustrated with an example: > SELECT context_ngrams(sentences(tweets), array("i", "love", null), 300) FROM > twitter; > will estimate the top-300 words that follow the phrase "i love" in a database > of tweets. The position of the null(s) specifies where to generate the n-gram > from, and can be placed anywhere. For example: > SELECT context_ngrams(sentences(tweets), array("i", "love", null, "but", > "hate", null), 300) FROM twitter; > will estimate the top-300 word-pairs that fill in the blanks specified by > null. > POSSIBLE USES: > 1. Pre-computing search lookaheads > 2. Sentiment analysis for products or entities -- e.g., querying with context > = array("twitter", "is", null) > 3. Navigation path analysis in URL databases -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.