[jira] [Commented] (HIVE-2749) CONV returns incorrect results sometimes
[ https://issues.apache.org/jira/browse/HIVE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196549#comment-13196549 ] Jonathan Chang commented on HIVE-2749: -- Oh sorry, I thought that was only for when the final patch gets approved. Lemme change all my other JIRAs. CONV returns incorrect results sometimes Key: HIVE-2749 URL: https://issues.apache.org/jira/browse/HIVE-2749 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-2749.D1437.1.patch, HIVE-2749.D1437.2.patch ...because it fails to reset state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Allow UDFs to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154727#comment-13154727 ] Jonathan Chang commented on HIVE-2561: -- A dependency declaration allows the resources to be automatically loaded so the end user doesn't have to. I don't think this is a niche case; this has come up a lot and this diff is the result of a brainstorm in response to user complaints. Allow UDFs to specify additional FILE/JAR resources necessary for execution --- Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-2561.D507.1.patch Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153961#comment-13153961 ] Jonathan Chang commented on HIVE-2561: -- I don't think this is purely an administrative problem; right now there's no way for an administrator to express that this file should be added if and only if the query uses a certain UDF. And even if there were it then tightly couples the UDF developer with the administration of the cluster. Thus it seems to me that the responsibility of specifying what the requirements are of the UDF should be the onus of the UDF developer, not the cluster administrator or the end user/query writer. Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-2561.D507.1.patch Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate to UDAF
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153644#comment-13153644 ] Jonathan Chang commented on HIVE-2592: -- @jvs, I have a rather large diff that goes some of the way towards fixing the issue, but it will take a few more large diffs before this one is knocked down completely. Would you be amenable to reviewing/committing one of them? ConstantOI doesn't always propagate to UDAF --- Key: HIVE-2592 URL: https://issues.apache.org/jira/browse/HIVE-2592 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Depending on the query plan, a UDAF may not actually get a constant OI. E.g., set hive.map.aggr=false; SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13148057#comment-13148057 ] Jonathan Chang commented on HIVE-2561: -- Ok, I think I have a reasonable implementation. Does anyone have a good suggestion for a UDF which I could also add and which would be a good testbed for this functionality? Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13148089#comment-13148089 ] Jonathan Chang commented on HIVE-2561: -- External dependencies might lead to license hell. Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13148104#comment-13148104 ] Jonathan Chang commented on HIVE-2561: -- How about something general then which takes a delimited input file and does lookups? Or would that just be overly redundant with a mapjoin? Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13146680#comment-13146680 ] Jonathan Chang commented on HIVE-2561: -- Hm, how would you feel about something more general then, like a method on the class that would be called. That method could potentially access things like environment vars, config settings, etc. to dynamically specify the dependencies? Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2561) Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution
[ https://issues.apache.org/jira/browse/HIVE-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13146703#comment-13146703 ] Jonathan Chang commented on HIVE-2561: -- I believe (and I will check this tomorrow), that UDFs have access to the config during the init phase which happens on the local machine. There is an outstanding JIRA to make the config available to the running UDFs but it's not needed here since all the ADDs happen before execution. I'll probably end up adding some utility functions to make getting these values easier. So for ADD FILE and the like, you should not refer to the absolute path in your UDF. In essence, ADD FILE is copying those paths to some temp directory which is the working directory of the UDF. Add an annotation to UDFs to allow them to specify additional FILE/JAR resources necessary for execution Key: HIVE-2561 URL: https://issues.apache.org/jira/browse/HIVE-2561 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Often times UDFs will have dependencies to external JARs/FILEs. It makes sense for these to be encoded by the UDF (rather than having the caller remember the set of files that need to be ADDed). Let's add an annotation to UDFs which will cause these resources to be auto-added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2527) Consecutive string literals should be combined into a single string literal.
[ https://issues.apache.org/jira/browse/HIVE-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13141675#comment-13141675 ] Jonathan Chang commented on HIVE-2527: -- This just looks like a complaint about the parse trees changing. Do you have the full list of failures? Consecutive string literals should be combined into a single string literal. Key: HIVE-2527 URL: https://issues.apache.org/jira/browse/HIVE-2527 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Priority: Minor Attachments: D147.1.patch, D147.2.patch, D147.3.patch, D147.3.patch C, Python, etc. all support this magical feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2532) Evaluation of non-deterministic/stateful UDFs should not be skipped even if constant oi is returned.
[ https://issues.apache.org/jira/browse/HIVE-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138595#comment-13138595 ] Jonathan Chang commented on HIVE-2532: -- I have a patch but don't have a good testcase yet. This might be a good time to add a side-effecting UDF. Any suggestions? Evaluation of non-deterministic/stateful UDFs should not be skipped even if constant oi is returned. Key: HIVE-2532 URL: https://issues.apache.org/jira/browse/HIVE-2532 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Even if constant oi is returned, these may have stateful/side-effect behavior and hence need to be called each cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2532) Evaluation of non-deterministic/stateful UDFs should not be skipped even if constant oi is returned.
[ https://issues.apache.org/jira/browse/HIVE-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138599#comment-13138599 ] Jonathan Chang commented on HIVE-2532: -- Ooo, how about a counter diff? Evaluation of non-deterministic/stateful UDFs should not be skipped even if constant oi is returned. Key: HIVE-2532 URL: https://issues.apache.org/jira/browse/HIVE-2532 Project: Hive Issue Type: Bug Reporter: Jonathan Chang Assignee: Jonathan Chang Even if constant oi is returned, these may have stateful/side-effect behavior and hence need to be called each cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135093#comment-13135093 ] Jonathan Chang commented on HIVE-2509: -- Done. Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: D15.1.patch, D15.1.patch, D15.2.patch, D15.2.patch, D15.3.patch, D15.3.patch, D15.4.patch, D15.4.patch, D15.4.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135398#comment-13135398 ] Jonathan Chang commented on HIVE-2509: -- How do I get edit rights to the wiki? Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Fix For: 0.9.0 Attachments: D15.1.patch, D15.1.patch, D15.2.patch, D15.2.patch, D15.3.patch, D15.3.patch, D15.4.patch, D15.4.patch, D15.4.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132123#comment-13132123 ] Jonathan Chang commented on HIVE-2509: -- 0L just brings this up to parity with C but C doesn't have convenient shorthands for shorts/chars so I'm open to suggestions on how to best represent these. Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: D15.1.patch, D15.1.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13131169#comment-13131169 ] Jonathan Chang commented on HIVE-2509: -- There are two issues: 1.) There are actually lots of UDFs which are pretty sensitive to type. In particular, widening will not handle cases where arrayint needs to be widened to arraybigint (there's a reason this sort of thing isn't autocasted in C++, Java, etc.). 2.) I generally think widening is a bad idea because it makes people less careful with types. For example, without looking, what will explain select * from (select '0' from akramer_one_row union all select 12345678900 from akramer_one_row) a; return as the type of the outer query? The answer is pretty terrible; if any other language did this to you you'd be pretty furious. Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: D15.1.patch, D15.1.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13131201#comment-13131201 ] Jonathan Chang commented on HIVE-2509: -- 1.) You can't cast arrayint to arraybigint in Hive which is why we had to write an FB UDF for this exact problem. Right now you need to pass in an instance of the desired type which is annoying with BIGINTs. This starts to become a deep rabbit hole. For example, how do you tell if an array is equal to [0L]? Turns out you can't compare arrays in Hive. So we have a UDF, which chokes if you try to compare arrayint to arraybigint, which is IMO reasonable behavior because the complexity/corner cases which we've been talking about here (and on point #2) just gets moved into this UDF which is untenable for the large library of UDFs we have. 2.) Yes, I agree that an error would be good in those cases. Can we make that happen? (Unrelated issue but type widening still fails with NULLs; in general I think type widening is uber complicated and prone to all sorts of issues we wouldn't have if we were more type safe throughout.) Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: D15.1.patch, D15.1.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13131209#comment-13131209 ] Jonathan Chang commented on HIVE-2509: -- To put it another way, once you have complex types (arrays, maps, struct and the like) you either need strong typing (a la C++, Java, etc.) or full-on duck typing (a la Python). I don't think any language has successfully maintained a middle ground (what Hive is trying to do right now). Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: D15.1.patch, D15.1.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2509) Literal bigint
[ https://issues.apache.org/jira/browse/HIVE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13131216#comment-13131216 ] Jonathan Chang commented on HIVE-2509: -- This also matters in a few other places like CTAS. In any case, this is a pretty low-risk change and not too onerous on the code base so I hope this gets in... Literal bigint -- Key: HIVE-2509 URL: https://issues.apache.org/jira/browse/HIVE-2509 Project: Hive Issue Type: New Feature Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: D15.1.patch, D15.1.patch Typing bigints is a pain because you have to write CAST(0 AS BIGINT). We should let people type 0L instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13122420#comment-13122420 ] Jonathan Chang commented on HIVE-2470: -- Won't review board throw an error if it doesn't apply cleanly? Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch, HIVE-2470.patch.3, HIVE-2470.patch.4, HIVE-2470.patch.5, HIVE-2470.patch.6 ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13121540#comment-13121540 ] Jonathan Chang commented on HIVE-2470: -- Ok, fix on its way. Note that I have to manually diff stuff because the junit command gives [junit] diff: file:\|pfile:\|hdfs:\|/tmp/\|invalidscheme:\|lastUpdateTime\|lastAccessTime\|[Oo]wner\|CreateTime\|LastAccessTime\|Location\|LOCATION '\|transient_lastDdlTime\|last_modified_\|java.lang.RuntimeException\|at org\|at sun\|at java\|at junit\|Caused by:\|LOCK_QUERYID:\|LOCK_TIME:\|grantTime\|[.][.][.] [0-9]* more\|job_[0-9]*_[0-9]*\|USING 'java -cp: Regular expression too big Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch, HIVE-2470.patch.3, HIVE-2470.patch.4, HIVE-2470.patch.5 ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13121565#comment-13121565 ] Jonathan Chang commented on HIVE-2470: -- UDF_BITMAP_EMPTY seems broken. Line 50 asserts that the length = 1, but line 70 asserts that length = 2. Can you confirm? Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch, HIVE-2470.patch.3, HIVE-2470.patch.4, HIVE-2470.patch.5 ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120395#comment-13120395 ] Jonathan Chang commented on HIVE-2470: -- Thanks. Fingers crossed. P.S. Is there a way I can kick off these tests on the server next time so you don't have to? Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch, HIVE-2470.patch.3, HIVE-2470.patch.4, HIVE-2470.patch.5 ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120411#comment-13120411 ] Jonathan Chang commented on HIVE-2470: -- Got it. I thought there was some kind of continuous build/test dashboard somewhere to do the adequate yelling automatically. Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch, HIVE-2470.patch.3, HIVE-2470.patch.4, HIVE-2470.patch.5 ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13119746#comment-13119746 ] Jonathan Chang commented on HIVE-2470: -- The .q file in the diff verifies this (if constants don't work, then the input to named struct will not be a constant and an exception will be thrown). Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch, HIVE-2470.patch.3, HIVE-2470.patch.4 ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118281#comment-13118281 ] Jonathan Chang commented on HIVE-2470: -- Oh, I remember why I did things like this. It's because types are checked at parse time (when the ExprGenericFuncNodes are created initialize() is called). Therefore we can't do this in the optimize step because UDFs which depend on a parameter being constant will fail if the constants in their children expressions haven't been folded by the time they get created. This leaves two options. The first is to modify the tree in place during the beast that is semantic analysis (I'm ok with this if you are). The other, which I went with, is to have the UDFs themselves do the folding during their initializations. Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118282#comment-13118282 ] Jonathan Chang commented on HIVE-2470: -- If we go with the second, I can add the functionality to GenericUDF directly but this will involve either changing all the call sites of GenericUDF or all the UDFs both of which seem pretty risky. Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118328#comment-13118328 ] Jonathan Chang commented on HIVE-2470: -- True, I can add a hook in exprnodegenericfuncevaluator so that if the outputOI is a constantOI to skip evaluation. @jvs, does that sound ok? (along with adding deterministic/stateful checks in genericfuncdesc) Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118396#comment-13118396 ] Jonathan Chang commented on HIVE-2470: -- @Igor, how is that different from what is currently being done? Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118407#comment-13118407 ] Jonathan Chang commented on HIVE-2470: -- Ah, interesting. I like the simplicity of that. You onboard with that jvs? Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118425#comment-13118425 ] Jonathan Chang commented on HIVE-2470: -- This should be ok as long as the wrapper subclasses GenericUDF right? I'm not sure what the invariant is in these cases but adding an equals method should be safe. Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[
https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117600#comment-13117600
]
Jonathan Chang commented on HIVE-2470:
--
Reflection is a good example (and is actually the motivation for this diff in
the first place). For example, if you have a function which evaluates an
another function, say EVAL('NUMBER_ROWS'), it won't be known until initialize
time whether EVAL is deterministic.
Improve support for Constant Object Inspectors
--
Key: HIVE-2470
URL: https://issues.apache.org/jira/browse/HIVE-2470
Project: Hive
Issue Type: Improvement
Reporter: Jonathan Chang
Assignee: Jonathan Chang
Attachments: HIVE-1360.part2.patch, HIVE-2470.patch
ConstantObjectInspectors are nice but they need to be more widely supported
and used. In particular,
* More functions need to be using them.
* There need to be facilities for propagating constness.
* Support for complex types must be added.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2470) Improve support for Constant Object Inspectors
[ https://issues.apache.org/jira/browse/HIVE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116717#comment-13116717 ] Jonathan Chang commented on HIVE-2470: -- The goal here was to maximize compatibility; that is, things can be constant folded only if they explicitly ask to be. The other reason was that some UDFs may have special handling for constants (e.g. all the reflection-based stuff). Improve support for Constant Object Inspectors -- Key: HIVE-2470 URL: https://issues.apache.org/jira/browse/HIVE-2470 Project: Hive Issue Type: Improvement Reporter: Jonathan Chang Assignee: Jonathan Chang Attachments: HIVE-1360.part2.patch, HIVE-2470.patch ConstantObjectInspectors are nice but they need to be more widely supported and used. In particular, * More functions need to be using them. * There need to be facilities for propagating constness. * Support for complex types must be added. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1360) Allow UDFs to access constant parameter values at compile time
[ https://issues.apache.org/jira/browse/HIVE-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115993#comment-13115993 ] Jonathan Chang commented on HIVE-1360: -- Please visit HIVE-2470 for followup. Allow UDFs to access constant parameter values at compile time -- Key: HIVE-1360 URL: https://issues.apache.org/jira/browse/HIVE-1360 Project: Hive Issue Type: Improvement Components: Query Processor, UDF Affects Versions: 0.5.0 Reporter: Carl Steinbach Assignee: Jonathan Chang Fix For: 0.8.0 Attachments: HIVE-1360.2.patch.txt, HIVE-1360.2.test-failures.txt, HIVE-1360.3.patch, HIVE-1360.part2.patch, HIVE-1360.patch, HIVE-1360.patch UDFs should be able to access constant parameter values at compile time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
