[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727340#comment-13727340 ] Yin Huai commented on HIVE-4979: Yes, it will stop on compile error. I am running unit tests. From what I have seen, it will continue on test failure (I inserted a failure in TestHadoop20SAuthBridge which is a part of shims). After the tests of shims, it is running test queries right now. I will check the test results tomorrow morning. > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch, > HIVE-4979.D11931.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4915) unit tests fail on windows because of difference in input file size
[ https://issues.apache.org/jira/browse/HIVE-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727331#comment-13727331 ] Ashutosh Chauhan commented on HIVE-4915: +1 > unit tests fail on windows because of difference in input file size > --- > > Key: HIVE-4915 > URL: https://issues.apache.org/jira/browse/HIVE-4915 > Project: Hive > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-4915.1.patch > > > Several qfile based tests fail on windows because in the output of explain > extended, the total file size of input files shown is different on windows. > This is because by default text files on windows are checked out with two > char line endings, and *.dat files used as input files for the tables are > considered as text files. So for every line in the .dat file, the size of the > file is larger by 1 byte on windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727332#comment-13727332 ] Gunther Hagleitner commented on HIVE-4979: -- I like the fix. Did you verify whether it stops on compile error but continues on test failure? If that's the case I'm +1 on this one. > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch, > HIVE-4979.D11931.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4951) combine2_win.q.out needs update for HIVE-3253 (increasing nesting levels)
[ https://issues.apache.org/jira/browse/HIVE-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727330#comment-13727330 ] Ashutosh Chauhan commented on HIVE-4951: +1 > combine2_win.q.out needs update for HIVE-3253 (increasing nesting levels) > - > > Key: HIVE-4951 > URL: https://issues.apache.org/jira/browse/HIVE-4951 > Project: Hive > Issue Type: Bug >Affects Versions: 0.11.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-4951.1.patch > > > combine2.q was updated in HIVE-3253, the corresponding change is missing in > combine2_win.q, causing it to fail on windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4638) Thread local PerfLog can get shared by multiple hiveserver2 sessions
[ https://issues.apache.org/jira/browse/HIVE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4638: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Prasad! > Thread local PerfLog can get shared by multiple hiveserver2 sessions > > > Key: HIVE-4638 > URL: https://issues.apache.org/jira/browse/HIVE-4638 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Affects Versions: 0.11.0 >Reporter: Prasad Mujumdar >Assignee: Prasad Mujumdar > Fix For: 0.12.0 > > Attachments: HIVE-4638-1.patch, HIVE-4638-2.patch > > > The PerfLog is accessed as thread local which can be shared by multiple > hiveserver2 session, overwriting query runtime information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4968) When deduplicating multiple SelectOperators, we should update RowResolver accordinly
[
https://issues.apache.org/jira/browse/HIVE-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4968:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Yin!
> When deduplicating multiple SelectOperators, we should update RowResolver
> accordinly
>
>
> Key: HIVE-4968
> URL: https://issues.apache.org/jira/browse/HIVE-4968
> Project: Hive
> Issue Type: Bug
>Reporter: Yin Huai
>Assignee: Yin Huai
> Fix For: 0.12.0
>
> Attachments: HIVE-4968.D11901.1.patch, HIVE-4968.D11901.2.patch
>
>
> {code:Sql}
> SELECT tmp3.key, tmp3.value, tmp3.count
> FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count
> FROM (SELECT key, value
> FROM src) tmp1
> JOIN (SELECT count(*) as count
> FROM src) tmp2
> ) tmp3;
> {\code}
> The plan is executable.
> {code:sql}
> SELECT tmp3.key, tmp3.value, tmp3.count
> FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count
> FROM (SELECT *
> FROM src) tmp1
> JOIN (SELECT count(*) as count
> FROM src) tmp2
> ) tmp3;
> {\code}
> The plan is executable.
> {code}
> SELECT tmp4.key, tmp4.value, tmp4.count
> FROM (SELECT tmp2.key as key, tmp2.value as value, tmp3.count as count
> FROM (SELECT *
> FROM (SELECT key, value
> FROM src) tmp1 ) tmp2
> JOIN (SELECT count(*) as count
> FROM src) tmp3
> ) tmp4;
> {\code}
> The plan is not executable.
> The plan related to the MapJoin is
> {code}
> Stage: Stage-5
> Map Reduce Local Work
> Alias -> Map Local Tables:
> tmp4:tmp2:tmp1:src
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> tmp4:tmp2:tmp1:src
> TableScan
> alias: src
> Select Operator
> expressions:
> expr: key
> type: string
> expr: value
> type: string
> outputColumnNames: _col0, _col1
> HashTable Sink Operator
> condition expressions:
> 0
> 1 {_col0}
> handleSkewJoin: false
> keys:
> 0 []
> 1 []
> Position of Big Table: 1
> Stage: Stage-4
> Map Reduce
> Alias -> Map Operator Tree:
> $INTNAME
> Map Join Operator
> condition map:
>Inner Join 0 to 1
> condition expressions:
> 0
> 1 {_col0}
> handleSkewJoin: false
> keys:
> 0 []
> 1 []
> outputColumnNames: _col2
> Position of Big Table: 1
> Select Operator
> expressions:
> expr: _col0
> type: string
> expr: _col1
> type: string
> expr: _col2
> type: bigint
> outputColumnNames: _col0, _col1, _col2
> File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Local Work:
> Map Reduce Local Work
> {\code}
> The outputColumnNames of MapJoin is '_col2'. But it should be '_col0, _col1,
> _col2'
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4952) When hive.join.emit.interval is small, queries optimized by Correlation Optimizer may generate wrong results
[
https://issues.apache.org/jira/browse/HIVE-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4952:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Yin!
> When hive.join.emit.interval is small, queries optimized by Correlation
> Optimizer may generate wrong results
>
>
> Key: HIVE-4952
> URL: https://issues.apache.org/jira/browse/HIVE-4952
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Yin Huai
>Assignee: Yin Huai
> Fix For: 0.12.0
>
> Attachments: HIVE-4952.D11889.1.patch, HIVE-4952.D11889.2.patch,
> replay.txt
>
>
> If we have a query like this ...
> {code:sql}
> SELECT xx.key, xx.cnt, yy.key
> FROM
> (SELECT x.key as key, count(1) as cnt FROM src1 x JOIN src1 y ON (x.key =
> y.key) group by x.key) xx
> JOIN src yy
> ON xx.key=yy.key;
> {\code}
> After Correlation Optimizer, the operator tree in the reducer will be
> {code}
> JOIN2
>|
>|
> MUX
> / \
> / \
>GBY |
> | |
> JOIN1|
> \ /
> \ /
> DEMUX
> {\code}
> For JOIN2, the right table will arrive at this operator first. If
> hive.join.emit.interval is small, e.g. 1, JOIN2 will output the results even
> it has not got any row from the left table. The logic related
> hive.join.emit.interval in JoinOperator assumes that inputs will be ordered
> by the tag. But, if a query has been optimized by Correlation Optimizer, this
> assumption may not hold for those JoinOperators inside the reducer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727326#comment-13727326 ] Yin Huai commented on HIVE-4979: Another try. I am running the entire unit tests. > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch, > HIVE-4979.D11931.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4979: -- Attachment: HIVE-4979.D11931.2.patch yhuai updated the revision "HIVE-4979 [jira] If any compiling error exists, test-shims should stop". Try another fix to avoid stop the test when any test failure happens Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D11931 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11931?vs=36777&id=36825#toc BRANCH HIVE-4979 ARCANIST PROJECT hive AFFECTED FILES shims/build.xml To: JIRA, ashutoshc, yhuai > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch, > HIVE-4979.D11931.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4972) update code generated by thrift for DemuxOperator and MuxOperator
[ https://issues.apache.org/jira/browse/HIVE-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4972: - Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) > update code generated by thrift for DemuxOperator and MuxOperator > - > > Key: HIVE-4972 > URL: https://issues.apache.org/jira/browse/HIVE-4972 > Project: Hive > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Yin Huai >Assignee: Yin Huai > Fix For: 0.12.0 > > Attachments: HIVE-4972.D11907.1.patch > > > HIVE-2206 introduces two new operators, which are DemuxOperator and > MuxOperator. queryplan.thrift has been updated. But code generated by thrift > should be also updated -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4960) lastAlias in CommonJoinOperator is not used
[
https://issues.apache.org/jira/browse/HIVE-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727322#comment-13727322
]
Hudson commented on HIVE-4960:
--
SUCCESS: Integrated in Hive-trunk-h0.21 #2238 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/2238/])
HIVE-4960 : lastAlias in CommonJoinOperator is not used (Yin Huai via Ashutosh
Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509314)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
> lastAlias in CommonJoinOperator is not used
> ---
>
> Key: HIVE-4960
> URL: https://issues.apache.org/jira/browse/HIVE-4960
> Project: Hive
> Issue Type: Improvement
>Reporter: Yin Huai
>Assignee: Yin Huai
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4960.D11895.1.patch
>
>
> In CommonJoinOperator, there is object called lastAlias. The initial value of
> this object is 'null'. After tracing the usage of this object, I found that
> there is no place to change the value of this object. Also, it is only used
> in processOp in JoinOperator and MapJoinOperator as
> {code}
> if ((lastAlias == null) || (!lastAlias.equals(alias))) {
> nextSz = joinEmitInterval;
> }
> {\code}
> Since lastAlias will always be null, we will assign joinEmitInterval to
> nextSz every time we get a row. Later in processOp, we have
> {code}
> nextSz = getNextSize(nextSz);
> {\code}
> Because we reset the value of nextSz to joinEmitInterval every time we get a
> row, seems that getNextSize will not be used as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4972) update code generated by thrift for DemuxOperator and MuxOperator
[ https://issues.apache.org/jira/browse/HIVE-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727321#comment-13727321 ] Gunther Hagleitner commented on HIVE-4972: -- Committed to trunk. Thanks Yin! > update code generated by thrift for DemuxOperator and MuxOperator > - > > Key: HIVE-4972 > URL: https://issues.apache.org/jira/browse/HIVE-4972 > Project: Hive > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4972.D11907.1.patch > > > HIVE-2206 introduces two new operators, which are DemuxOperator and > MuxOperator. queryplan.thrift has been updated. But code generated by thrift > should be also updated -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727277#comment-13727277 ] Mark Wagner commented on HIVE-2482: --- Also, this patch relies on changes in HIVE-4970 to apply cleanly (and to fix the test case) > Convenience UDFs for binary data type > - > > Key: HIVE-2482 > URL: https://issues.apache.org/jira/browse/HIVE-2482 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Mark Wagner > Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch > > > HIVE-2380 introduced binary data type in Hive. It will be good to have > following udfs to make it more useful: > * UDF's to convert to/from hex string > * UDF's to convert to/from string using a specific encoding > * UDF's to convert to/from base64 string > * UDF's to convert to/from non-string types using a particular serde -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-2482: -- Attachment: HIVE-2482.2.patch Here's the updated patch with q testing. > Convenience UDFs for binary data type > - > > Key: HIVE-2482 > URL: https://issues.apache.org/jira/browse/HIVE-2482 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Mark Wagner > Attachments: HIVE-2482.1.patch, HIVE-2482.2.patch > > > HIVE-2380 introduced binary data type in Hive. It will be good to have > following udfs to make it more useful: > * UDF's to convert to/from hex string > * UDF's to convert to/from string using a specific encoding > * UDF's to convert to/from base64 string > * UDF's to convert to/from non-string types using a particular serde -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4960) lastAlias in CommonJoinOperator is not used
[
https://issues.apache.org/jira/browse/HIVE-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727262#comment-13727262
]
Hudson commented on HIVE-4960:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #321 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/321/])
HIVE-4960 : lastAlias in CommonJoinOperator is not used (Yin Huai via Ashutosh
Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509314)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
> lastAlias in CommonJoinOperator is not used
> ---
>
> Key: HIVE-4960
> URL: https://issues.apache.org/jira/browse/HIVE-4960
> Project: Hive
> Issue Type: Improvement
>Reporter: Yin Huai
>Assignee: Yin Huai
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4960.D11895.1.patch
>
>
> In CommonJoinOperator, there is object called lastAlias. The initial value of
> this object is 'null'. After tracing the usage of this object, I found that
> there is no place to change the value of this object. Also, it is only used
> in processOp in JoinOperator and MapJoinOperator as
> {code}
> if ((lastAlias == null) || (!lastAlias.equals(alias))) {
> nextSz = joinEmitInterval;
> }
> {\code}
> Since lastAlias will always be null, we will assign joinEmitInterval to
> nextSz every time we get a row. Later in processOp, we have
> {code}
> nextSz = getNextSize(nextSz);
> {\code}
> Because we reset the value of nextSz to joinEmitInterval every time we get a
> row, seems that getNextSize will not be used as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4970) BinaryConverter does not respect nulls
[ https://issues.apache.org/jira/browse/HIVE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-4970: -- Attachment: HIVE-4970.2.patch Added some checks for null in the existing unit tests (for other converters also) and made the existing .q file for binary udfs include some nulls. > BinaryConverter does not respect nulls > -- > > Key: HIVE-4970 > URL: https://issues.apache.org/jira/browse/HIVE-4970 > Project: Hive > Issue Type: Bug >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: HIVE-4970.1.patch, HIVE-4970.2.patch > > > Right now, the BinaryConverter in PrimitiveObjectInspectorConverter does not > handle null values the same as the other converters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
Sergey Shelukhin created HIVE-4985: -- Summary: refactor/clean up partition name pruning to be usable inside metastore server Key: HIVE-4985 URL: https://issues.apache.org/jira/browse/HIVE-4985 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Preliminary for HIVE-4914. The patch is going to be large already, so some refactoring and dead code removal that is non-controversial can be done in advance in a separate patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-4914: --- Summary: filtering via partition name should be done inside metastore server (implementation) (was: filtering via partition name should be done inside metastore server) > filtering via partition name should be done inside metastore server > (implementation) > > > Key: HIVE-4914 > URL: https://issues.apache.org/jira/browse/HIVE-4914 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Currently, if the filter pushdown is impossible (which is most cases), the > client gets all partition names from metastore, filters them, and asks for > partitions by names for the filtered set. > Metastore server code should do that instead; it should check if pushdown is > possible and do it if so; otherwise it should do name-based filtering. > Saves the roundtrip with all partition names from the server to client, and > also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4984) hive metastore should not re-use hadoop proxy configuration
Thejas M Nair created HIVE-4984: --- Summary: hive metastore should not re-use hadoop proxy configuration Key: HIVE-4984 URL: https://issues.apache.org/jira/browse/HIVE-4984 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.12.0 Reporter: Thejas M Nair Hive metastore supports proxyuser/doas functionality like hadoop [1]. Metastore allows anybody who has proxyuser privileges in core-site.xml, to be a metastore proxy user. This is a bad from a security perspective, because when a user is made proxy user for hadoop, it gets automatic privilege as proxy user for metastore as well. The more secure approach is to use metastore specific config parameters, like what oozie does. [2] [1] http://hadoop.apache.org/docs/stable/Secure_Impersonation.html [2] http://oozie.apache.org/docs/3.2.0-incubating/AG_Install.html#User_ProxyUser_Configuration -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4984) hive metastore should not re-use hadoop proxy configuration
[ https://issues.apache.org/jira/browse/HIVE-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-4984: Component/s: Security > hive metastore should not re-use hadoop proxy configuration > --- > > Key: HIVE-4984 > URL: https://issues.apache.org/jira/browse/HIVE-4984 > Project: Hive > Issue Type: Bug > Components: Metastore, Security >Affects Versions: 0.12.0 >Reporter: Thejas M Nair > > Hive metastore supports proxyuser/doas functionality like hadoop [1]. > Metastore allows anybody who has proxyuser privileges in core-site.xml, to be > a metastore proxy user. > This is a bad from a security perspective, because when a user is made proxy > user for hadoop, it gets automatic privilege as proxy user for metastore as > well. > The more secure approach is to use metastore specific config parameters, like > what oozie does. [2] > [1] http://hadoop.apache.org/docs/stable/Secure_Impersonation.html > [2] > http://oozie.apache.org/docs/3.2.0-incubating/AG_Install.html#User_ProxyUser_Configuration -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4983) Hive metastore client doesn't use batching for filter pushdown
Sergey Shelukhin created HIVE-4983: -- Summary: Hive metastore client doesn't use batching for filter pushdown Key: HIVE-4983 URL: https://issues.apache.org/jira/browse/HIVE-4983 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Priority: Minor When getting partitions the usual way (get names, filter, get partitions by filtered names) MS client batches the partition get requests; the default batch size is 300. This can already cause problems (metastore can OOM when getting many partitions if it's not given enough memory). When filter pushdown is improved to be used in more cases the problem will become worse, and the name filtering is moved to server to avoid roundtrips with large number of names going to the client/to decide on pushdown on server, batching will disappear entirely. We might want to introduce it for filter pushdown, direct SQL and all these other cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4983) Hive metastore client doesn't use batching for filter pushdown
[ https://issues.apache.org/jira/browse/HIVE-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-4983: --- Description: When getting partitions the usual way (get names, filter, get partitions by filtered names) MS client batches the partition get requests; the default batch size is 300. However for filter pushdown there's no such logic. This can already cause problems (metastore can OOM when getting many partitions if it's not given enough memory). When filter pushdown is improved to be used in more cases the problem will become worse, and the name filtering is moved to server to avoid roundtrips with large number of names going to the client/to decide on pushdown on server, batching will disappear entirely. We might want to introduce it for filter pushdown, direct SQL and all these other cases. was: When getting partitions the usual way (get names, filter, get partitions by filtered names) MS client batches the partition get requests; the default batch size is 300. This can already cause problems (metastore can OOM when getting many partitions if it's not given enough memory). When filter pushdown is improved to be used in more cases the problem will become worse, and the name filtering is moved to server to avoid roundtrips with large number of names going to the client/to decide on pushdown on server, batching will disappear entirely. We might want to introduce it for filter pushdown, direct SQL and all these other cases. > Hive metastore client doesn't use batching for filter pushdown > -- > > Key: HIVE-4983 > URL: https://issues.apache.org/jira/browse/HIVE-4983 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Minor > > When getting partitions the usual way (get names, filter, get partitions by > filtered names) MS client batches the partition get requests; the default > batch size is 300. However for filter pushdown there's no such logic. > This can already cause problems (metastore can OOM when getting many > partitions if it's not given enough memory). > When filter pushdown is improved to be used in more cases the problem will > become worse, and the name filtering is moved to server to avoid roundtrips > with large number of names going to the client/to decide on pushdown on > server, batching will disappear entirely. > We might want to introduce it for filter pushdown, direct SQL and all these > other cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727217#comment-13727217 ] Thejas M Nair commented on HIVE-2482: - bq. Hey guys, just to make sure I understand, unit tests are still strongly encouraged for new features as well as end-to-end tests? I would agree with that conclusion > Convenience UDFs for binary data type > - > > Key: HIVE-2482 > URL: https://issues.apache.org/jira/browse/HIVE-2482 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Mark Wagner > Attachments: HIVE-2482.1.patch > > > HIVE-2380 introduced binary data type in Hive. It will be good to have > following udfs to make it more useful: > * UDF's to convert to/from hex string > * UDF's to convert to/from string using a specific encoding > * UDF's to convert to/from base64 string > * UDF's to convert to/from non-string types using a particular serde -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727165#comment-13727165 ] Yin Huai commented on HIVE-4979: Yeah... [~hagleitn] you are right. With this patch, tests will stop after shims if there is any test failure Let me see how to solve this problem. > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4979: --- Status: Open (was: Patch Available) > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4982) Hive tarball doesn't contain .git* files
Brock Noland created HIVE-4982: -- Summary: Hive tarball doesn't contain .git* files Key: HIVE-4982 URL: https://issues.apache.org/jira/browse/HIVE-4982 Project: Hive Issue Type: Bug Reporter: Brock Noland Priority: Minor The tarball should contain the .git* files as they are part of the tag. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4794) Unit e2e tests for vectorization
[ https://issues.apache.org/jira/browse/HIVE-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727115#comment-13727115 ] Tony Murphy commented on HIVE-4794: --- For some reason testCliDriver_rename_column didn't show up in the jenkins results. I've filed HIVE-4981 for it. I'm trying to get a repro for testCliDriver_smb_mapjoin_8. It didn't repro when run in isolation, so i've kicked off the entire suite to see if there's a dependency with another test that introduced this failure. > Unit e2e tests for vectorization > > > Key: HIVE-4794 > URL: https://issues.apache.org/jira/browse/HIVE-4794 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4794.1.patch, HIVE-4794.2.patch, HIVE-4794.3.patch, > HIVE-4794.3-vectorization.patch, HIVE-4794.4-vectorization.patch, > hive-4794.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4981) PreCommit Patch Testing Jenkins Result missing failures present in JIRA Comment
Tony Murphy created HIVE-4981: - Summary: PreCommit Patch Testing Jenkins Result missing failures present in JIRA Comment Key: HIVE-4981 URL: https://issues.apache.org/jira/browse/HIVE-4981 Project: Hive Issue Type: Bug Reporter: Tony Murphy the jenkins test results don’t match the message posted to the JIRA: https://builds.apache.org/job/PreCommit-HIVE-Build/267/testReport/ https://issues.apache.org/jira/browse/HIVE-4794?focusedCommentId=13726048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13726048 For some reason the Jenkins test result page does not list testCliDriver_rename_column as failing, but the message in the JIRA does. This caused me to miss that failure, since I was doing my investigation from the Jenkins results page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4838) Refactor MapJoin HashMap code to improve testability and readability
[ https://issues.apache.org/jira/browse/HIVE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727102#comment-13727102 ] Brock Noland commented on HIVE-4838: I guess we could go that route. My thought was that the memory consumption was monitored to be conservative? I've always wondered about this. I mean if an admin sets mapred.child.java.opts and io.sort.mb final on the cluster the settings we are using from a client perspective could be completely different therefore it's possible it "works" locally but fails on the cluster. Another question I had about this is that ORC has a memory manager that assumes it can use a certain percentage of ram but that could conflict with our work here? That is the ORC memory manager could use memory while creating the hash table that we won't use when reading the hash table? Additionally I thought it might make sense to only store offsets into a side file in the hash map to reduce memory consumption and then throw say a 25MB LRU cache on lookups into the file. Since the file is small it should be in OS buffer cache when not in the LRU cache. Maybe we should take up memory management during map joins in another jira? > Refactor MapJoin HashMap code to improve testability and readability > > > Key: HIVE-4838 > URL: https://issues.apache.org/jira/browse/HIVE-4838 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Brock Noland > Attachments: HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, > HIVE-4838.patch, HIVE-4838.patch > > > MapJoin is an essential component for high performance joins in Hive and the > current code has done great service for many years. However, the code is > showing it's age and currently suffers from the following issues: > * Uses static state via the MapJoinMetaData class to pass serialization > metadata to the Key, Row classes. > * The api of a logical "Table Container" is not defined and therefore it's > unclear what apis HashMapWrapper > needs to publicize. Additionally HashMapWrapper has many used public methods. > * HashMapWrapper contains logic to serialize, test memory bounds, and > implement the table container. Ideally these logical units could be seperated > * HashTableSinkObjectCtx has unused fields and unused methods > * CommonJoinOperator and children use ArrayList on left hand side when only > List is required > * There are unused classes MRU, DCLLItemm and classes which duplicate > functionality MapJoinSingleKey and MapJoinDoubleKeys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727100#comment-13727100 ] Yin Huai commented on HIVE-4979: Let me try an entire run of unit tests with HIVE-4979.4980.failedTest.txt > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4971) Unit test failure in TestVectorTimestampExpressions
[
https://issues.apache.org/jira/browse/HIVE-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727083#comment-13727083
]
Hive QA commented on HIVE-4971:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12595508/HIVE-4971.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/282/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-282/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1509469.
At revision 1509469.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
> Unit test failure in TestVectorTimestampExpressions
> ---
>
> Key: HIVE-4971
> URL: https://issues.apache.org/jira/browse/HIVE-4971
> Project: Hive
> Issue Type: Sub-task
> Components: Tests, UDF
>Affects Versions: vectorization-branch
>Reporter: Jitendra Nath Pandey
>Assignee: Gopal V
> Attachments: HIVE-4971.patch
>
>
> Unit test "testVectorUDFUnixTimeStampLong" is failing
> TestVectorTimestampExpressions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[
https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727077#comment-13727077
]
Hive QA commented on HIVE-4980:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12595495/HIVE-4980.D11937.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2757 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/281/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/281/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
> Fix the compiling error in TestHadoop20SAuthBridge
> --
>
> Key: HIVE-4980
> URL: https://issues.apache.org/jira/browse/HIVE-4980
> Project: Hive
> Issue Type: Sub-task
>Reporter: Yin Huai
>Assignee: Yin Huai
> Attachments: HIVE-4980.D11937.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[
https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727062#comment-13727062
]
Yin Huai commented on HIVE-4979:
I did a quick test. If we have test failures, the build will not stop. To try
my test, you can apply
https://issues.apache.org/jira/secure/attachment/12595513/HIVE-4979.4980.failedTest.txt
and then {code}ant test -Dmodule=shims{\code}
The log will look like ...
{code}
est:
[echo] Project: shims
[junit] WARNING: multiple versions of ant detected in path for junit
[junit]
jar:file:/usr/share/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit] and
jar:file:/Users/yhuai/Projects/Hive-YSmart/hive-trunk/build/ivy/lib/hadoop0.20S.shim/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
[junit] Running org.apache.hadoop.hive.thrift.TestDBTokenStore
[junit] 2013-08-01 16:17:21.382 java[4398:1203] Unable to load realm info
from SCDynamicStore
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.458 sec
[junit] Running org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge
[junit] 2013-08-01 16:17:23.191 java[4399:1203] Unable to load realm info
from SCDynamicStore
[junit] Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 20.987 sec
[junit] Test org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge FAILED
[junit] Running org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore
[junit] 2013-08-01 16:17:45.621 java[4401:1203] Unable to load realm info
from SCDynamicStore
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.943 sec
[for] shims: The following error occurred while executing this line:
[for] /Users/yhuai/Projects/Hive-YSmart/hive-trunk/build.xml:351: The
following error occurred while executing this line:
[for] /Users/yhuai/Projects/Hive-YSmart/hive-trunk/build-common.xml:490:
Tests failed!
BUILD FAILED
/Users/yhuai/Projects/Hive-YSmart/hive-trunk/build.xml:349: Keepgoing
execution: 1 of 1 iterations failed.
Total time: 37 seconds
{\code}
> If any compiling error exists, test-shims should stop
> -
>
> Key: HIVE-4979
> URL: https://issues.apache.org/jira/browse/HIVE-4979
> Project: Hive
> Issue Type: Sub-task
>Reporter: Yin Huai
>Assignee: Yin Huai
> Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4980: --- Attachment: HIVE-4827-FailedCase > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4980: --- Attachment: (was: HIVE-4979.4980.failedTest.txt) > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4980: --- Attachment: (was: HIVE-4827-FailedCase) > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4979: --- Attachment: HIVE-4979.4980.failedTest.txt > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.4980.failedTest.txt, HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4980: --- Attachment: HIVE-4979.4980.failedTest.txt > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4971) Unit test failure in TestVectorTimestampExpressions
[ https://issues.apache.org/jira/browse/HIVE-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-4971: -- Release Note: Fix vectorized TIMESTAMP() to handle negative timestamps with fractional seconds Status: Patch Available (was: In Progress) > Unit test failure in TestVectorTimestampExpressions > --- > > Key: HIVE-4971 > URL: https://issues.apache.org/jira/browse/HIVE-4971 > Project: Hive > Issue Type: Sub-task > Components: Tests, UDF >Affects Versions: vectorization-branch >Reporter: Jitendra Nath Pandey >Assignee: Gopal V > Attachments: HIVE-4971.patch > > > Unit test "testVectorUDFUnixTimeStampLong" is failing > TestVectorTimestampExpressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4971) Unit test failure in TestVectorTimestampExpressions
[ https://issues.apache.org/jira/browse/HIVE-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-4971: -- Attachment: HIVE-4971.patch Fix the timestamp math, to match the rounding mechanism for java (all integer divisions round towards zero). > Unit test failure in TestVectorTimestampExpressions > --- > > Key: HIVE-4971 > URL: https://issues.apache.org/jira/browse/HIVE-4971 > Project: Hive > Issue Type: Sub-task > Components: Tests, UDF >Affects Versions: vectorization-branch >Reporter: Jitendra Nath Pandey >Assignee: Gopal V > Attachments: HIVE-4971.patch > > > Unit test "testVectorUDFUnixTimeStampLong" is failing > TestVectorTimestampExpressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4971) Unit test failure in TestVectorTimestampExpressions
[ https://issues.apache.org/jira/browse/HIVE-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-4971: -- Affects Version/s: vectorization-branch > Unit test failure in TestVectorTimestampExpressions > --- > > Key: HIVE-4971 > URL: https://issues.apache.org/jira/browse/HIVE-4971 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Jitendra Nath Pandey >Assignee: Gopal V > > Unit test "testVectorUDFUnixTimeStampLong" is failing > TestVectorTimestampExpressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4971) Unit test failure in TestVectorTimestampExpressions
[ https://issues.apache.org/jira/browse/HIVE-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-4971: -- Component/s: UDF Tests > Unit test failure in TestVectorTimestampExpressions > --- > > Key: HIVE-4971 > URL: https://issues.apache.org/jira/browse/HIVE-4971 > Project: Hive > Issue Type: Sub-task > Components: Tests, UDF >Affects Versions: vectorization-branch >Reporter: Jitendra Nath Pandey >Assignee: Gopal V > > Unit test "testVectorUDFUnixTimeStampLong" is failing > TestVectorTimestampExpressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-4971) Unit test failure in TestVectorTimestampExpressions
[ https://issues.apache.org/jira/browse/HIVE-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-4971 started by Gopal V. > Unit test failure in TestVectorTimestampExpressions > --- > > Key: HIVE-4971 > URL: https://issues.apache.org/jira/browse/HIVE-4971 > Project: Hive > Issue Type: Sub-task > Components: Tests, UDF >Affects Versions: vectorization-branch >Reporter: Jitendra Nath Pandey >Assignee: Gopal V > > Unit test "testVectorUDFUnixTimeStampLong" is failing > TestVectorTimestampExpressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727024#comment-13727024 ] Gunther Hagleitner commented on HIVE-4979: -- I agree with the stop-on-failed-compile part, but this change will also stop the build if the test fails, right? > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727014#comment-13727014 ] Phabricator commented on HIVE-4979: --- ashutoshc has accepted the revision "HIVE-4979 [jira] If any compiling error exists, test-shims should stop". +1 It doesn't make sense to have this configurable. Build should stop if we can't compile. REVISION DETAIL https://reviews.facebook.net/D11931 BRANCH HIVE-4979 ARCANIST PROJECT hive To: JIRA, ashutoshc, yhuai > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727013#comment-13727013 ] Gunther Hagleitner commented on HIVE-4979: -- We've spent some time in the past to make sure tests run to completion in the case of error (depending on flag), but report all errors along the way. I think the change you propose will halt the build when it encounters a test failure in the test shim. Is there a way to do keep the flag behavior but record the error? > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727000#comment-13727000 ] Phabricator commented on HIVE-4980: --- hagleitn has accepted the revision "HIVE-4980 [jira] Fix the compiling error in TestHadoop20SAuthBridge". LGTM +1 REVISION DETAIL https://reviews.facebook.net/D11937 BRANCH HIVE-4980 ARCANIST PROJECT hive To: JIRA, hagleitn, yhuai > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions & is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726974#comment-13726974
]
Hive QA commented on HIVE-4051:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12595477/HIVE-4051.D11805.6.patch
{color:green}SUCCESS:{color} +1 2749 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/278/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/278/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
> Hive's metastore suffers from 1+N queries when querying partitions & is slow
>
>
> Key: HIVE-4051
> URL: https://issues.apache.org/jira/browse/HIVE-4051
> Project: Hive
> Issue Type: Bug
> Components: Clients, Metastore
> Environment: RHEL 6.3 / EC2 C1.XL
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4051.D11805.1.patch, HIVE-4051.D11805.2.patch,
> HIVE-4051.D11805.3.patch, HIVE-4051.D11805.4.patch, HIVE-4051.D11805.5.patch,
> HIVE-4051.D11805.6.patch
>
>
> Hive's query client takes a long time to initialize & start planning queries
> because of delays in creating all the MTable/MPartition objects.
> For a hive db with 1800 partitions, the metastore took 6-7 seconds to
> initialize - firing approximately 5900 queries to the mysql database.
> Several of those queries fetch exactly one row to create a single object on
> the client.
> The following 12 queries were repeated for each partition, generating a storm
> of SQL queries
> {code}
> 4 Query SELECT
> `A0`.`SD_ID`,`B0`.`INPUT_FORMAT`,`B0`.`IS_COMPRESSED`,`B0`.`IS_STOREDASSUBDIRECTORIES`,`B0`.`LOCATION`,`B0`.`NUM_BUCKETS`,`B0`.`OUTPUT_FORMAT`,`B0`.`SD_ID`
> FROM `PARTITIONS` `A0` LEFT OUTER JOIN `SDS` `B0` ON `A0`.`SD_ID` =
> `B0`.`SD_ID` WHERE `A0`.`PART_ID` = 3945
> 4 Query SELECT `A0`.`CD_ID`,`B0`.`CD_ID` FROM `SDS` `A0` LEFT OUTER JOIN
> `CDS` `B0` ON `A0`.`CD_ID` = `B0`.`CD_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `COLUMNS_V2` THIS WHERE THIS.`CD_ID`=1546
> AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT
> `A0`.`COMMENT`,`A0`.`COLUMN_NAME`,`A0`.`TYPE_NAME`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `COLUMNS_V2` `A0` WHERE `A0`.`CD_ID` = 1546 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT `A0`.`SERDE_ID`,`B0`.`NAME`,`B0`.`SLIB`,`B0`.`SERDE_ID`
> FROM `SDS` `A0` LEFT OUTER JOIN `SERDES` `B0` ON `A0`.`SERDE_ID` =
> `B0`.`SERDE_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE THIS.`SD_ID`=4871 AND
> THIS.`INTEGER_IDX`>=0
> 4 Query SELECT `A0`.`COLUMN_NAME`,`A0`.`ORDER`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `SORT_COLS` `A0` WHERE `A0`.`SD_ID` =4871 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_VALUES` THIS WHERE
> THIS.`SD_ID_OID`=4871 AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A1`.`STRING_LIST_ID`,`A0`.`INTEGER_IDX` AS NUCORDER0 FROM
> `SKEWED_VALUES` `A0` INNER JOIN `SKEWED_STRING_LIST` `A1` ON
> `A0`.`STRING_LIST_ID_EID` = `A1`.`STRING_LIST_ID` WHERE `A0`.`SD_ID_OID`
> =4871 AND `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_COL_VALUE_LOC_MAP` WHERE `SD_ID`
> =4871 AND `STRING_LIST_ID_KID` IS NOT NULL
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A0`.`STRING_LIST_ID` FROM `SKEWED_STRING_LIST` `A0` INNER JOIN
> `SKEWED_COL_VALUE_LOC_MAP` `B0` ON `A0`.`STRING_LIST_ID` =
> `B0`.`STRING_LIST_ID_KID` WHERE `B0`.`SD_ID` =4871
> 4 Query SELECT `A0`.`STRING_LIST_ID_KID`,`A0`.`LOCATION` FROM
> `SKEWED_COL_VALUE_LOC_MAP` `A0` WHERE `A0`.`SD_ID` =4871 AND NOT
> (`A0`.`STRING_LIST_ID_KID` IS NULL)
> {code}
> This data is not detached or cached, so this operation is performed during
> every query plan for the partitions, even in the same hive client.
> The queries are automatically generated by JDO/DataNucleus which makes it
> nearly impossible to rewrite it into a single denormalized join operation &
> process it locally.
> Attempts to optimize this with JDO fetch-groups did not bear fruit in
> improving the query count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4976) TestHadoop20SAuthBridge needs to be updated after HIVE-3255
[
https://issues.apache.org/jira/browse/HIVE-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726966#comment-13726966
]
Yin Huai commented on HIVE-4976:
I am addressing this issue in two separate tasks. HIVE-4979 is for the
build.xml. HIVE-4980 is for the compiling error.
> TestHadoop20SAuthBridge needs to be updated after HIVE-3255
> ---
>
> Key: HIVE-4976
> URL: https://issues.apache.org/jira/browse/HIVE-4976
> Project: Hive
> Issue Type: Bug
>Reporter: Yin Huai
>Assignee: Yin Huai
>
> I found the following error appearing. Here is the log from jenkins
> {code}
> compile-test-worker:
> [echo] Project: shims
> [echo] Compiling shim tests against hadoop 20S
> [echo] Test srcdir :
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test
>
> [javac] Compiling 3 source files to
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/build/shims/test/classes
> [javac]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:103:
> cannot find symbol
> [javac] symbol : method
> startDelegationTokenSecretManager(org.apache.hadoop.conf.Configuration)
> [javac] location: class
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.Server
> [javac] super.startDelegationTokenSecretManager(conf);
> [javac] ^
> [javac]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:100:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac] 2 errors
>[subant] Failure for target 'test' of:
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml
>[subant] The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:174:
> The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:154:
> The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:106:
> Compile failed; see the compiler error output for details.
> {\code}
> I found HIVE-3255 changed the interface of
> HadoopThriftAuthBridge20S.Server.startDelegationTokenSecretManager to
> 'startDelegationTokenSecretManager(Configuration conf, Object hms)'. But in
> TestHadoop20SAuthBridge, we are still using the old interface which is
> 'startDelegationTokenSecretManager(conf)'.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4980: --- Status: Patch Available (was: Open) > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4980: -- Attachment: HIVE-4980.D11937.1.patch yhuai requested code review of "HIVE-4980 [jira] Fix the compiling error in TestHadoop20SAuthBridge". Reviewers: JIRA fix TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D11937 AFFECTED FILES shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/28521/ To: JIRA, yhuai > Fix the compiling error in TestHadoop20SAuthBridge > -- > > Key: HIVE-4980 > URL: https://issues.apache.org/jira/browse/HIVE-4980 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4980.D11937.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[
https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726957#comment-13726957
]
Yin Huai commented on HIVE-4979:
The above command cannot verify the fix... Please try
{code}
ant test -Dtestcase=TestHadoop20SAuthBridge
{\code}
For the current trunk, in my mac, the above command will generate "BUILD
SUCCESSFUL". But, when I check the testreport, no unit test has been tested
because the test cannot be compiled.
In my ubuntu vm, the above command will generate "BUILD FAILED". But the build
will not stop right after the point of the compiling error. The log will look
like this ...
{code}
checkstyle-init:
[mkdir] Created dir: /home/yhuai/hive/hive-trunk/hcatalog/build/checkstyle
checkstyle:
[echo] hcatalog
[checkstyle] Running Checkstyle 5.5 on 422 files
[for] hcatalog: The following error occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/build.xml:355: The following error
occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/hcatalog/build.xml:102:
[for] The following error occurred while executing this line:
[for]
/home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/build-common.xml:97: The
following error occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/test.xml:30:
exec returned: 1
[for] The following error occurred while executing this line:
[for]
/home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/build-common.xml:97: The
following error occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/test.xml:30:
exec returned: 1
[for] The following error occurred while executing this line:
[for]
/home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/build-common.xml:97: The
following error occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/test.xml:30:
exec returned: 1
[for] The following error occurred while executing this line:
[for]
/home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/build-common.xml:97: The
following error occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/test.xml:30:
exec returned: 1
[for] The following error occurred while executing this line:
[for]
/home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/build-common.xml:97: The
following error occurred while executing this line:
[for] /home/yhuai/hive/hive-trunk/hcatalog/build-support/ant/test.xml:30:
exec returned: 1
BUILD FAILED
/home/yhuai/hive/hive-trunk/build.xml:353: Keepgoing execution: 1 of 11
iterations failed.
Total time: 2 minutes 11 seconds
{\code}
With this fix, the build will stop right after the point of the compiling
error. The log will look like this ...
{code}
compile-test-worker:
[echo] Project: shims
[echo] Compiling shim tests against hadoop 20S
[echo] Test srcdir :
/home/yhuai/hive/hive-4979/shims/src/common-secure/test
[javac] Compiling 3 source files to
/home/yhuai/hive/hive-4979/build/shims/test/classes
[javac]
/home/yhuai/hive/hive-4979/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:103:
cannot find symbol
[javac] symbol : method
startDelegationTokenSecretManager(org.apache.hadoop.conf.Configuration)
[javac] location: class
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.Server
[javac] super.startDelegationTokenSecretManager(conf);
[javac] ^
[javac]
/home/yhuai/hive/hive-4979/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:100:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac] 2 errors
BUILD FAILED
/home/yhuai/hive/hive-4979/build.xml:341: The following error occurred while
executing this line:
/home/yhuai/hive/hive-4979/build.xml:361: The following error occurred while
executing this line:
/home/yhuai/hive/hive-4979/shims/build.xml:174: The following error occurred
while executing this line:
/home/yhuai/hive/hive-4979/shims/build.xml:154: The following error occurred
while executing this line:
/home/yhuai/hive/hive-4979/shims/build.xml:106: Compile failed; see the
compiler error output for details.
Total time: 1 minute 38 seconds
{\code}
> If any compiling error exists, test-shims should stop
> -
>
> Key: HIVE-4979
> URL: https://issues.apache.org/jira/browse/HIVE-4979
> Project: Hive
> Issue Type: Sub-task
>Reporter: Yin Huai
>Assignee: Yin Huai
> Attachments: HIVE-4979.D11931.1.patch
>
>
--
This message is automatically
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726959#comment-13726959 ] Yin Huai commented on HIVE-4979: We should expect that the precommit test will fail. > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4843) Refactoring MapRedTask and ExecDriver for better re-usability (for tez) and readability
[ https://issues.apache.org/jira/browse/HIVE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726948#comment-13726948 ] Hudson commented on HIVE-4843: -- ABORTED: Integrated in Hive-trunk-hadoop2 #320 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/320/]) HIVE-4843: Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java > Refactoring MapRedTask and ExecDriver for better re-usability (for tez) and > readability > --- > > Key: HIVE-4843 > URL: https://issues.apache.org/jira/browse/HIVE-4843 > Project: Hive > Issue Type: Bug >Affects Versions: 0.12.0, tez-branch >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 0.12.0 > > Attachments: HIVE-4843.1.patch, HIVE-4843.2.patch, HIVE-4843.3.patch, > HIVE-4843.4.patch, HIVE-4843.5.patch > > > Currently, there are static apis in multiple locations in ExecDriver and > MapRedTask that can be leveraged if put in the already existing utility class > in the exec package. This would help making the code more maintainable, > readable and also re-usable by other run-time infra such as tez. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4954) PTFTranslator hardcodes ranking functions
[
https://issues.apache.org/jira/browse/HIVE-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726946#comment-13726946
]
Hudson commented on HIVE-4954:
--
ABORTED: Integrated in Hive-trunk-hadoop2 #320 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/320/])
HIVE-4954: PTFTranslator hardcodes ranking functions (Edward Capriolo via Brock
Noland) (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1508984)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionDescription.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCumeDist.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFDenseRank.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentRank.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRank.java
*
/hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java
> PTFTranslator hardcodes ranking functions
> -
>
> Key: HIVE-4954
> URL: https://issues.apache.org/jira/browse/HIVE-4954
> Project: Hive
> Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Fix For: 0.12.0
>
> Attachments: HIVE-4879.2.patch.txt, HIVE-4954.1.patch.txt
>
>
> protected static final ArrayList RANKING_FUNCS = new
> ArrayList();
> static {
> RANKING_FUNCS.add("rank");
> RANKING_FUNCS.add("dense_rank");
> RANKING_FUNCS.add("percent_rank");
> RANKING_FUNCS.add("cume_dist");
> };
> Move this logic to annotations
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only task to its child task
[ https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726947#comment-13726947 ] Hudson commented on HIVE-4827: -- ABORTED: Integrated in Hive-trunk-hadoop2 #320 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/320/]) HIVE-4827: Merge a Map-only task to its child task (Yin Huai via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509082) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/QueryPlanTreeTransformation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java * /hive/trunk/ql/src/test/queries/clientpositive/auto_join33.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer1.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer3.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer4.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer5.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer6.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer7.q * /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin1.q * /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin2.q * /hive/trunk/ql/src/test/queries/clientpositive/union34.q * /hive/trunk/ql/src/test/results/clientpositive/auto_join0.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join10.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join15.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join2.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join22.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join23.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join24.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join26.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join32.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join33.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer1.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer3.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer4.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer6.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer7.q.out * /hive/trunk/ql/src/test/results/clientpositive/join28.q.out * /hive/trunk/ql/src/test/results/clientpositive/join32.q.out * /hive/trunk/ql/src/test/results/clientpositive/join33.q.out * /hive/trunk/ql/src/test/results/clientpositive/join_star.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery2.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out * /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin1.q.out * /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin2.q.out * /hive/trunk/ql/src/test/results/clientpositive/multi_join_union.q.out * /hive/trunk/ql/src/test/results/clientpositive/union34.q.out > Merge a Map-only task to its child task > --- > > Key: HIVE-4827 > URL: https://issues.apache.org/jira/browse/HIVE-4827 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.12.0 >Reporter: Yin Huai >Assignee: Yin Huai > Fix For: 0.12.0 > > Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch, HIVE-4827.3
[jira] [Commented] (HIVE-2590) HBase bulk load wiki page improvements
[ https://issues.apache.org/jira/browse/HIVE-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726937#comment-13726937 ] Ashutosh Chauhan commented on HIVE-2590: permissions granted. > HBase bulk load wiki page improvements > -- > > Key: HIVE-2590 > URL: https://issues.apache.org/jira/browse/HIVE-2590 > Project: Hive > Issue Type: Bug > Components: Documentation, HBase Handler >Reporter: Ben West >Assignee: Ben West >Priority: Minor > Labels: wiki > Fix For: 0.8.0 > > > Some suggestions on the page > https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad which seems > kind of out of date: > 1. It seems like it's required that the number of reduce tasks in the "Sort > Data" phase be one more than the number of keys selected in the "Range > Partitioning" step, or else you get an error like this: > Caused by: java.lang.IllegalArgumentException: Can't read partitions file > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:91) > ... 15 more > Caused by: java.io.IOException: Wrong number of partitions in keyset > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:72) > ... 15 more > If so, it would be helpful if this was explicitly pointed out. > 2. It recommends that you should use the "loadtable" ruby script to put data > into hbase, but if you run this on newer versions of HBase (e.g. 0.90.3) it > errors: > DISABLED Use completebulkload instead. See tail of > http://hbase.apache.org/bulk-loads.html > The instructions should probably be changed to use completebulkload instead > of this script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4979: --- Status: Patch Available (was: Open) > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4979) If any compiling error exists, test-shims should stop
[
https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726932#comment-13726932
]
Yin Huai commented on HIVE-4979:
To try it, please use
{code}
ant test -Dmodule=shims
{\code}
> If any compiling error exists, test-shims should stop
> -
>
> Key: HIVE-4979
> URL: https://issues.apache.org/jira/browse/HIVE-4979
> Project: Hive
> Issue Type: Sub-task
>Reporter: Yin Huai
>Assignee: Yin Huai
> Attachments: HIVE-4979.D11931.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4979) If any compiling error exists, test-shims should stop
[ https://issues.apache.org/jira/browse/HIVE-4979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4979: -- Attachment: HIVE-4979.D11931.1.patch yhuai requested code review of "HIVE-4979 [jira] If any compiling error exists, test-shims should stop". Reviewers: JIRA update build.xml TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D11931 AFFECTED FILES build.xml MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/28515/ To: JIRA, yhuai > If any compiling error exists, test-shims should stop > - > > Key: HIVE-4979 > URL: https://issues.apache.org/jira/browse/HIVE-4979 > Project: Hive > Issue Type: Sub-task >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4979.D11931.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3607) Set mapreduce.task.classpath.user.precedence to true by default
[ https://issues.apache.org/jira/browse/HIVE-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726925#comment-13726925 ] Konstantin Boudnik commented on HIVE-3607: -- Are there any plans to commit this and HIVE-3606 in the bugfix release for Hive 0.11 (if any?) > Set mapreduce.task.classpath.user.precedence to true by default > --- > > Key: HIVE-3607 > URL: https://issues.apache.org/jira/browse/HIVE-3607 > Project: Hive > Issue Type: Improvement > Components: Configuration >Affects Versions: 0.10.0 >Reporter: Kevin Wilfong >Assignee: Kevin Wilfong > > When queries are actually run in a Hadoop cluster, Hive's jars are appended > to Hadoop's classpath. However, when we test/run jobs locally Hive's > classpath comes first. This leads to issues like the one brought up here > after the patch was committed HIVE-3581 where a change depended on a jar Hive > includes which conflicted with one provided by Hadoop which is an older > version in 0.20 > It's possible that more of the jars we include are getting preceded by older > jars in Hadoop, and we haven't noticed yet. > If we add Hive jars to the beginning of Hadoop's classpath we will be in > control in such situations where the jars are backwards compatible. We will > be able to update the jars in Hive and these will be used at run time, > instead of just compile time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4979) If any compiling error exists, test-shims should stop
Yin Huai created HIVE-4979: -- Summary: If any compiling error exists, test-shims should stop Key: HIVE-4979 URL: https://issues.apache.org/jira/browse/HIVE-4979 Project: Hive Issue Type: Sub-task Reporter: Yin Huai Assignee: Yin Huai -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4980) Fix the compiling error in TestHadoop20SAuthBridge
Yin Huai created HIVE-4980: -- Summary: Fix the compiling error in TestHadoop20SAuthBridge Key: HIVE-4980 URL: https://issues.apache.org/jira/browse/HIVE-4980 Project: Hive Issue Type: Sub-task Reporter: Yin Huai Assignee: Yin Huai -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4960) lastAlias in CommonJoinOperator is not used
[
https://issues.apache.org/jira/browse/HIVE-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726906#comment-13726906
]
Hudson commented on HIVE-4960:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #40 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/40/])
HIVE-4960 : lastAlias in CommonJoinOperator is not used (Yin Huai via Ashutosh
Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509314)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
> lastAlias in CommonJoinOperator is not used
> ---
>
> Key: HIVE-4960
> URL: https://issues.apache.org/jira/browse/HIVE-4960
> Project: Hive
> Issue Type: Improvement
>Reporter: Yin Huai
>Assignee: Yin Huai
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4960.D11895.1.patch
>
>
> In CommonJoinOperator, there is object called lastAlias. The initial value of
> this object is 'null'. After tracing the usage of this object, I found that
> there is no place to change the value of this object. Also, it is only used
> in processOp in JoinOperator and MapJoinOperator as
> {code}
> if ((lastAlias == null) || (!lastAlias.equals(alias))) {
> nextSz = joinEmitInterval;
> }
> {\code}
> Since lastAlias will always be null, we will assign joinEmitInterval to
> nextSz every time we get a row. Later in processOp, we have
> {code}
> nextSz = getNextSize(nextSz);
> {\code}
> Because we reset the value of nextSz to joinEmitInterval every time we get a
> row, seems that getNextSize will not be used as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions & is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-4051:
---
Status: Patch Available (was: Open)
> Hive's metastore suffers from 1+N queries when querying partitions & is slow
>
>
> Key: HIVE-4051
> URL: https://issues.apache.org/jira/browse/HIVE-4051
> Project: Hive
> Issue Type: Bug
> Components: Clients, Metastore
> Environment: RHEL 6.3 / EC2 C1.XL
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4051.D11805.1.patch, HIVE-4051.D11805.2.patch,
> HIVE-4051.D11805.3.patch, HIVE-4051.D11805.4.patch, HIVE-4051.D11805.5.patch,
> HIVE-4051.D11805.6.patch
>
>
> Hive's query client takes a long time to initialize & start planning queries
> because of delays in creating all the MTable/MPartition objects.
> For a hive db with 1800 partitions, the metastore took 6-7 seconds to
> initialize - firing approximately 5900 queries to the mysql database.
> Several of those queries fetch exactly one row to create a single object on
> the client.
> The following 12 queries were repeated for each partition, generating a storm
> of SQL queries
> {code}
> 4 Query SELECT
> `A0`.`SD_ID`,`B0`.`INPUT_FORMAT`,`B0`.`IS_COMPRESSED`,`B0`.`IS_STOREDASSUBDIRECTORIES`,`B0`.`LOCATION`,`B0`.`NUM_BUCKETS`,`B0`.`OUTPUT_FORMAT`,`B0`.`SD_ID`
> FROM `PARTITIONS` `A0` LEFT OUTER JOIN `SDS` `B0` ON `A0`.`SD_ID` =
> `B0`.`SD_ID` WHERE `A0`.`PART_ID` = 3945
> 4 Query SELECT `A0`.`CD_ID`,`B0`.`CD_ID` FROM `SDS` `A0` LEFT OUTER JOIN
> `CDS` `B0` ON `A0`.`CD_ID` = `B0`.`CD_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `COLUMNS_V2` THIS WHERE THIS.`CD_ID`=1546
> AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT
> `A0`.`COMMENT`,`A0`.`COLUMN_NAME`,`A0`.`TYPE_NAME`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `COLUMNS_V2` `A0` WHERE `A0`.`CD_ID` = 1546 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT `A0`.`SERDE_ID`,`B0`.`NAME`,`B0`.`SLIB`,`B0`.`SERDE_ID`
> FROM `SDS` `A0` LEFT OUTER JOIN `SERDES` `B0` ON `A0`.`SERDE_ID` =
> `B0`.`SERDE_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE THIS.`SD_ID`=4871 AND
> THIS.`INTEGER_IDX`>=0
> 4 Query SELECT `A0`.`COLUMN_NAME`,`A0`.`ORDER`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `SORT_COLS` `A0` WHERE `A0`.`SD_ID` =4871 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_VALUES` THIS WHERE
> THIS.`SD_ID_OID`=4871 AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A1`.`STRING_LIST_ID`,`A0`.`INTEGER_IDX` AS NUCORDER0 FROM
> `SKEWED_VALUES` `A0` INNER JOIN `SKEWED_STRING_LIST` `A1` ON
> `A0`.`STRING_LIST_ID_EID` = `A1`.`STRING_LIST_ID` WHERE `A0`.`SD_ID_OID`
> =4871 AND `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_COL_VALUE_LOC_MAP` WHERE `SD_ID`
> =4871 AND `STRING_LIST_ID_KID` IS NOT NULL
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A0`.`STRING_LIST_ID` FROM `SKEWED_STRING_LIST` `A0` INNER JOIN
> `SKEWED_COL_VALUE_LOC_MAP` `B0` ON `A0`.`STRING_LIST_ID` =
> `B0`.`STRING_LIST_ID_KID` WHERE `B0`.`SD_ID` =4871
> 4 Query SELECT `A0`.`STRING_LIST_ID_KID`,`A0`.`LOCATION` FROM
> `SKEWED_COL_VALUE_LOC_MAP` `A0` WHERE `A0`.`SD_ID` =4871 AND NOT
> (`A0`.`STRING_LIST_ID_KID` IS NULL)
> {code}
> This data is not detached or cached, so this operation is performed during
> every query plan for the partitions, even in the same hive client.
> The queries are automatically generated by JDO/DataNucleus which makes it
> nearly impossible to rewrite it into a single denormalized join operation &
> process it locally.
> Attempts to optimize this with JDO fetch-groups did not bear fruit in
> improving the query count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4978) [WebHCat] Close the PrintWriter after writing data
[ https://issues.apache.org/jira/browse/HIVE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venki Korukanti updated HIVE-4978: -- Attachment: HIVE-4978-1.patch > [WebHCat] Close the PrintWriter after writing data > -- > > Key: HIVE-4978 > URL: https://issues.apache.org/jira/browse/HIVE-4978 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.11.0 >Reporter: Venki Korukanti >Priority: Minor > Fix For: 0.12.0 > > Attachments: HIVE-4978-1.patch > > > We are not closing the PrintWriter after writing data into it. I haven't seen > any problems so far, but it is good to close the PrintWriter so that > resources are released properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4978) [WebHCat] Close the PrintWriter after writing data
[ https://issues.apache.org/jira/browse/HIVE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726893#comment-13726893 ] Venki Korukanti commented on HIVE-4978: --- Attached a patch. > [WebHCat] Close the PrintWriter after writing data > -- > > Key: HIVE-4978 > URL: https://issues.apache.org/jira/browse/HIVE-4978 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.11.0 >Reporter: Venki Korukanti >Priority: Minor > Fix For: 0.12.0 > > Attachments: HIVE-4978-1.patch > > > We are not closing the PrintWriter after writing data into it. I haven't seen > any problems so far, but it is good to close the PrintWriter so that > resources are released properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4978) [WebHCat] Close the PrintWriter after writing data
Venki Korukanti created HIVE-4978: - Summary: [WebHCat] Close the PrintWriter after writing data Key: HIVE-4978 URL: https://issues.apache.org/jira/browse/HIVE-4978 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Venki Korukanti Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4978-1.patch We are not closing the PrintWriter after writing data into it. I haven't seen any problems so far, but it is good to close the PrintWriter so that resources are released properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4978) [WebHCat] Close the PrintWriter after writing data
[ https://issues.apache.org/jira/browse/HIVE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venki Korukanti updated HIVE-4978: -- Status: Patch Available (was: Open) > [WebHCat] Close the PrintWriter after writing data > -- > > Key: HIVE-4978 > URL: https://issues.apache.org/jira/browse/HIVE-4978 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.11.0 >Reporter: Venki Korukanti >Priority: Minor > Fix For: 0.12.0 > > Attachments: HIVE-4978-1.patch > > > We are not closing the PrintWriter after writing data into it. I haven't seen > any problems so far, but it is good to close the PrintWriter so that > resources are released properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2590) HBase bulk load wiki page improvements
[ https://issues.apache.org/jira/browse/HIVE-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726874#comment-13726874 ] vikram s commented on HIVE-2590: vikram s > HBase bulk load wiki page improvements > -- > > Key: HIVE-2590 > URL: https://issues.apache.org/jira/browse/HIVE-2590 > Project: Hive > Issue Type: Bug > Components: Documentation, HBase Handler >Reporter: Ben West >Assignee: Ben West >Priority: Minor > Labels: wiki > Fix For: 0.8.0 > > > Some suggestions on the page > https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad which seems > kind of out of date: > 1. It seems like it's required that the number of reduce tasks in the "Sort > Data" phase be one more than the number of keys selected in the "Range > Partitioning" step, or else you get an error like this: > Caused by: java.lang.IllegalArgumentException: Can't read partitions file > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:91) > ... 15 more > Caused by: java.io.IOException: Wrong number of partitions in keyset > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:72) > ... 15 more > If so, it would be helpful if this was explicitly pointed out. > 2. It recommends that you should use the "loadtable" ruby script to put data > into hbase, but if you run this on newer versions of HBase (e.g. 0.90.3) it > errors: > DISABLED Use completebulkload instead. See tail of > http://hbase.apache.org/bulk-loads.html > The instructions should probably be changed to use completebulkload instead > of this script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4976) TestHadoop20SAuthBridge needs to be updated after HIVE-3255
[
https://issues.apache.org/jira/browse/HIVE-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4976:
---
Assignee: Yin Huai
> TestHadoop20SAuthBridge needs to be updated after HIVE-3255
> ---
>
> Key: HIVE-4976
> URL: https://issues.apache.org/jira/browse/HIVE-4976
> Project: Hive
> Issue Type: Bug
>Reporter: Yin Huai
>Assignee: Yin Huai
>
> I found the following error appearing. Here is the log from jenkins
> {code}
> compile-test-worker:
> [echo] Project: shims
> [echo] Compiling shim tests against hadoop 20S
> [echo] Test srcdir :
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test
>
> [javac] Compiling 3 source files to
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/build/shims/test/classes
> [javac]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:103:
> cannot find symbol
> [javac] symbol : method
> startDelegationTokenSecretManager(org.apache.hadoop.conf.Configuration)
> [javac] location: class
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.Server
> [javac] super.startDelegationTokenSecretManager(conf);
> [javac] ^
> [javac]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:100:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac] 2 errors
>[subant] Failure for target 'test' of:
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml
>[subant] The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:174:
> The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:154:
> The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:106:
> Compile failed; see the compiler error output for details.
> {\code}
> I found HIVE-3255 changed the interface of
> HadoopThriftAuthBridge20S.Server.startDelegationTokenSecretManager to
> 'startDelegationTokenSecretManager(Configuration conf, Object hms)'. But in
> TestHadoop20SAuthBridge, we are still using the old interface which is
> 'startDelegationTokenSecretManager(conf)'.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2590) HBase bulk load wiki page improvements
[ https://issues.apache.org/jira/browse/HIVE-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726860#comment-13726860 ] Ashutosh Chauhan commented on HIVE-2590: [~vikramsi] Whats your cwiki id? > HBase bulk load wiki page improvements > -- > > Key: HIVE-2590 > URL: https://issues.apache.org/jira/browse/HIVE-2590 > Project: Hive > Issue Type: Bug > Components: Documentation, HBase Handler >Reporter: Ben West >Assignee: Ben West >Priority: Minor > Labels: wiki > Fix For: 0.8.0 > > > Some suggestions on the page > https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad which seems > kind of out of date: > 1. It seems like it's required that the number of reduce tasks in the "Sort > Data" phase be one more than the number of keys selected in the "Range > Partitioning" step, or else you get an error like this: > Caused by: java.lang.IllegalArgumentException: Can't read partitions file > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:91) > ... 15 more > Caused by: java.io.IOException: Wrong number of partitions in keyset > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:72) > ... 15 more > If so, it would be helpful if this was explicitly pointed out. > 2. It recommends that you should use the "loadtable" ruby script to put data > into hbase, but if you run this on newer versions of HBase (e.g. 0.90.3) it > errors: > DISABLED Use completebulkload instead. See tail of > http://hbase.apache.org/bulk-loads.html > The instructions should probably be changed to use completebulkload instead > of this script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2590) HBase bulk load wiki page improvements
[ https://issues.apache.org/jira/browse/HIVE-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726853#comment-13726853 ] vikram s commented on HIVE-2590: I wanted to update wiki.as most of of the stuff is unclear. > HBase bulk load wiki page improvements > -- > > Key: HIVE-2590 > URL: https://issues.apache.org/jira/browse/HIVE-2590 > Project: Hive > Issue Type: Bug > Components: Documentation, HBase Handler >Reporter: Ben West >Assignee: Ben West >Priority: Minor > Labels: wiki > Fix For: 0.8.0 > > > Some suggestions on the page > https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad which seems > kind of out of date: > 1. It seems like it's required that the number of reduce tasks in the "Sort > Data" phase be one more than the number of keys selected in the "Range > Partitioning" step, or else you get an error like this: > Caused by: java.lang.IllegalArgumentException: Can't read partitions file > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:91) > ... 15 more > Caused by: java.io.IOException: Wrong number of partitions in keyset > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:72) > ... 15 more > If so, it would be helpful if this was explicitly pointed out. > 2. It recommends that you should use the "loadtable" ruby script to put data > into hbase, but if you run this on newer versions of HBase (e.g. 0.90.3) it > errors: > DISABLED Use completebulkload instead. See tail of > http://hbase.apache.org/bulk-loads.html > The instructions should probably be changed to use completebulkload instead > of this script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions & is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-4051:
--
Attachment: HIVE-4051.D11805.6.patch
sershe updated the revision "HIVE-4051 [jira] Hive's metastore suffers from 1+N
queries when querying partitions & is slow".
The above failed tests are due to sorting. Due to so many tests failing I
changed the code to preserve the same sort as JDO, it's actually a very small
change. I am getting some test failures locally that look like they are caused
by my laptop problems; let me try to push to HiveQA again.
Reviewers: JIRA
REVISION DETAIL
https://reviews.facebook.net/D11805
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D11805?vs=36627&id=36771#toc
AFFECTED FILES
build.xml
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
To: JIRA, sershe
Cc: brock
> Hive's metastore suffers from 1+N queries when querying partitions & is slow
>
>
> Key: HIVE-4051
> URL: https://issues.apache.org/jira/browse/HIVE-4051
> Project: Hive
> Issue Type: Bug
> Components: Clients, Metastore
> Environment: RHEL 6.3 / EC2 C1.XL
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4051.D11805.1.patch, HIVE-4051.D11805.2.patch,
> HIVE-4051.D11805.3.patch, HIVE-4051.D11805.4.patch, HIVE-4051.D11805.5.patch,
> HIVE-4051.D11805.6.patch
>
>
> Hive's query client takes a long time to initialize & start planning queries
> because of delays in creating all the MTable/MPartition objects.
> For a hive db with 1800 partitions, the metastore took 6-7 seconds to
> initialize - firing approximately 5900 queries to the mysql database.
> Several of those queries fetch exactly one row to create a single object on
> the client.
> The following 12 queries were repeated for each partition, generating a storm
> of SQL queries
> {code}
> 4 Query SELECT
> `A0`.`SD_ID`,`B0`.`INPUT_FORMAT`,`B0`.`IS_COMPRESSED`,`B0`.`IS_STOREDASSUBDIRECTORIES`,`B0`.`LOCATION`,`B0`.`NUM_BUCKETS`,`B0`.`OUTPUT_FORMAT`,`B0`.`SD_ID`
> FROM `PARTITIONS` `A0` LEFT OUTER JOIN `SDS` `B0` ON `A0`.`SD_ID` =
> `B0`.`SD_ID` WHERE `A0`.`PART_ID` = 3945
> 4 Query SELECT `A0`.`CD_ID`,`B0`.`CD_ID` FROM `SDS` `A0` LEFT OUTER JOIN
> `CDS` `B0` ON `A0`.`CD_ID` = `B0`.`CD_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `COLUMNS_V2` THIS WHERE THIS.`CD_ID`=1546
> AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT
> `A0`.`COMMENT`,`A0`.`COLUMN_NAME`,`A0`.`TYPE_NAME`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `COLUMNS_V2` `A0` WHERE `A0`.`CD_ID` = 1546 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT `A0`.`SERDE_ID`,`B0`.`NAME`,`B0`.`SLIB`,`B0`.`SERDE_ID`
> FROM `SDS` `A0` LEFT OUTER JOIN `SERDES` `B0` ON `A0`.`SERDE_ID` =
> `B0`.`SERDE_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE THIS.`SD_ID`=4871 AND
> THIS.`INTEGER_IDX`>=0
> 4 Query SELECT `A0`.`COLUMN_NAME`,`A0`.`ORDER`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `SORT_COLS` `A0` WHERE `A0`.`SD_ID` =4871 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_VALUES` THIS WHERE
> THIS.`SD_ID_OID`=4871 AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A1`.`STRING_LIST_ID`,`A0`.`INTEGER_IDX` AS NUCORDER0 FROM
> `SKEWED_VALUES` `A0` INNER JOIN `SKEWED_STRING_LIST` `A1` ON
> `A0`.`STRING_LIST_ID_EID` = `A1`.`STRING_LIST_ID` WHERE `A0`.`SD_ID_OID`
> =4871 AND `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_COL_VALUE_LOC_MAP` WHERE `SD_ID`
> =4871 AND `STRING_LIST_ID_KID` IS NOT NULL
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A0`.`STRING_LIST_ID` FROM `SKEWED_STRING_LIST` `A0` INNER JOIN
> `SKEWED_COL_VALUE_LOC_MAP` `B0` ON `A0`.`STRING_LIST_ID` =
> `B0`.`STRING_LIST_ID_KID` WHERE `B0`.`SD_ID` =4871
> 4 Query SELECT `A0`.`STRING_LIST_ID_KID`,`A0`.`LOCATION` FROM
> `SKEWED_COL_VALUE_LOC_MAP` `A0` WHERE `A0`.`SD_ID` =4871 AND NOT
> (`A0`.`STRING_LIST_ID_KID` IS NULL)
> {code}
> This data is not detached or cached, so this operation is performed during
> every query plan for the partitions, even in the same hive client.
> The queries are automatically generated by JDO/DataNucleus which makes it
> nearly impossible to rewrite it into a single denormalized join operation &
> process it locally.
> Attempts to optimize this with JDO fetch-groups did not bear fruit in
> improvi
[jira] [Updated] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions & is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-4051:
---
Status: Open (was: Patch Available)
> Hive's metastore suffers from 1+N queries when querying partitions & is slow
>
>
> Key: HIVE-4051
> URL: https://issues.apache.org/jira/browse/HIVE-4051
> Project: Hive
> Issue Type: Bug
> Components: Clients, Metastore
> Environment: RHEL 6.3 / EC2 C1.XL
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4051.D11805.1.patch, HIVE-4051.D11805.2.patch,
> HIVE-4051.D11805.3.patch, HIVE-4051.D11805.4.patch, HIVE-4051.D11805.5.patch
>
>
> Hive's query client takes a long time to initialize & start planning queries
> because of delays in creating all the MTable/MPartition objects.
> For a hive db with 1800 partitions, the metastore took 6-7 seconds to
> initialize - firing approximately 5900 queries to the mysql database.
> Several of those queries fetch exactly one row to create a single object on
> the client.
> The following 12 queries were repeated for each partition, generating a storm
> of SQL queries
> {code}
> 4 Query SELECT
> `A0`.`SD_ID`,`B0`.`INPUT_FORMAT`,`B0`.`IS_COMPRESSED`,`B0`.`IS_STOREDASSUBDIRECTORIES`,`B0`.`LOCATION`,`B0`.`NUM_BUCKETS`,`B0`.`OUTPUT_FORMAT`,`B0`.`SD_ID`
> FROM `PARTITIONS` `A0` LEFT OUTER JOIN `SDS` `B0` ON `A0`.`SD_ID` =
> `B0`.`SD_ID` WHERE `A0`.`PART_ID` = 3945
> 4 Query SELECT `A0`.`CD_ID`,`B0`.`CD_ID` FROM `SDS` `A0` LEFT OUTER JOIN
> `CDS` `B0` ON `A0`.`CD_ID` = `B0`.`CD_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `COLUMNS_V2` THIS WHERE THIS.`CD_ID`=1546
> AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT
> `A0`.`COMMENT`,`A0`.`COLUMN_NAME`,`A0`.`TYPE_NAME`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `COLUMNS_V2` `A0` WHERE `A0`.`CD_ID` = 1546 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT `A0`.`SERDE_ID`,`B0`.`NAME`,`B0`.`SLIB`,`B0`.`SERDE_ID`
> FROM `SDS` `A0` LEFT OUTER JOIN `SERDES` `B0` ON `A0`.`SERDE_ID` =
> `B0`.`SERDE_ID` WHERE `A0`.`SD_ID` =4871
> 4 Query SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE THIS.`SD_ID`=4871 AND
> THIS.`INTEGER_IDX`>=0
> 4 Query SELECT `A0`.`COLUMN_NAME`,`A0`.`ORDER`,`A0`.`INTEGER_IDX` AS
> NUCORDER0 FROM `SORT_COLS` `A0` WHERE `A0`.`SD_ID` =4871 AND
> `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_VALUES` THIS WHERE
> THIS.`SD_ID_OID`=4871 AND THIS.`INTEGER_IDX`>=0
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A1`.`STRING_LIST_ID`,`A0`.`INTEGER_IDX` AS NUCORDER0 FROM
> `SKEWED_VALUES` `A0` INNER JOIN `SKEWED_STRING_LIST` `A1` ON
> `A0`.`STRING_LIST_ID_EID` = `A1`.`STRING_LIST_ID` WHERE `A0`.`SD_ID_OID`
> =4871 AND `A0`.`INTEGER_IDX` >= 0 ORDER BY NUCORDER0
> 4 Query SELECT COUNT(*) FROM `SKEWED_COL_VALUE_LOC_MAP` WHERE `SD_ID`
> =4871 AND `STRING_LIST_ID_KID` IS NOT NULL
> 4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
> NUCLEUS_TYPE,`A0`.`STRING_LIST_ID` FROM `SKEWED_STRING_LIST` `A0` INNER JOIN
> `SKEWED_COL_VALUE_LOC_MAP` `B0` ON `A0`.`STRING_LIST_ID` =
> `B0`.`STRING_LIST_ID_KID` WHERE `B0`.`SD_ID` =4871
> 4 Query SELECT `A0`.`STRING_LIST_ID_KID`,`A0`.`LOCATION` FROM
> `SKEWED_COL_VALUE_LOC_MAP` `A0` WHERE `A0`.`SD_ID` =4871 AND NOT
> (`A0`.`STRING_LIST_ID_KID` IS NULL)
> {code}
> This data is not detached or cached, so this operation is performed during
> every query plan for the partitions, even in the same hive client.
> The queries are automatically generated by JDO/DataNucleus which makes it
> nearly impossible to rewrite it into a single denormalized join operation &
> process it locally.
> Attempts to optimize this with JDO fetch-groups did not bear fruit in
> improving the query count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4541) Run check-style on the branch and fix style issues.
[ https://issues.apache.org/jira/browse/HIVE-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-4541. Resolution: Fixed Committed to branch. Thanks, Jitendra! > Run check-style on the branch and fix style issues. > --- > > Key: HIVE-4541 > URL: https://issues.apache.org/jira/browse/HIVE-4541 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: vectorization-branch > > Attachments: HIVE-4541.1.patch, HIVE-4541.2.patch > > > We should run check style on the entire branch and fix issues before the > branch is merged back to the trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4967) Don't serialize unnecessary fields in query plan
[ https://issues.apache.org/jira/browse/HIVE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726843#comment-13726843 ] Brock Noland commented on HIVE-4967: This looks good to me, I'll review it tomorrow unless someone else gets to it first. > Don't serialize unnecessary fields in query plan > > > Key: HIVE-4967 > URL: https://issues.apache.org/jira/browse/HIVE-4967 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-4967.patch > > > There are quite a few fields which need not to be serialized since they are > initialized anyways in backend. We need not to serialize them in our plan. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4967) Don't serialize unnecessary fields in query plan
[ https://issues.apache.org/jira/browse/HIVE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726829#comment-13726829 ] Ashutosh Chauhan commented on HIVE-4967: Reported failure is unrelated to change. Phabricator entry is available at https://reviews.facebook.net/D11925 > Don't serialize unnecessary fields in query plan > > > Key: HIVE-4967 > URL: https://issues.apache.org/jira/browse/HIVE-4967 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-4967.patch > > > There are quite a few fields which need not to be serialized since they are > initialized anyways in backend. We need not to serialize them in our plan. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4977) HS2: support an alternate serialization protocol between client and server
Chris Drome created HIVE-4977: - Summary: HS2: support an alternate serialization protocol between client and server Key: HIVE-4977 URL: https://issues.apache.org/jira/browse/HIVE-4977 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0, 0.10.0, 0.12.0 Reporter: Chris Drome Assignee: Chris Drome Current serialization protocol between client and server as defined in cli_service.thrift results in 2x (or more) throughput degradation compared to HS1. Initial proposal is to introduce HS1 serialization protocol as a negotiable alternative. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4794) Unit e2e tests for vectorization
[
https://issues.apache.org/jira/browse/HIVE-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726808#comment-13726808
]
Hive QA commented on HIVE-4794:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12595463/HIVE-4794.4-vectorization.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 3512 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rename_column
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTimestampExpressions.testVectorUDFUnixTimeStampLong
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/277/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/277/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
> Unit e2e tests for vectorization
>
>
> Key: HIVE-4794
> URL: https://issues.apache.org/jira/browse/HIVE-4794
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: vectorization-branch
>Reporter: Tony Murphy
>Assignee: Tony Murphy
> Fix For: vectorization-branch
>
> Attachments: HIVE-4794.1.patch, HIVE-4794.2.patch, HIVE-4794.3.patch,
> HIVE-4794.3-vectorization.patch, HIVE-4794.4-vectorization.patch,
> hive-4794.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4976) TestHadoop20SAuthBridge needs to be updated after HIVE-3255
[
https://issues.apache.org/jira/browse/HIVE-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726793#comment-13726793
]
Gunther Hagleitner commented on HIVE-4976:
--
Good find [~yhuai]. We should really first change the build to fail if tests
don't actually compile and then fix the code to compile properly.
> TestHadoop20SAuthBridge needs to be updated after HIVE-3255
> ---
>
> Key: HIVE-4976
> URL: https://issues.apache.org/jira/browse/HIVE-4976
> Project: Hive
> Issue Type: Bug
>Reporter: Yin Huai
>
> I found the following error appearing. Here is the log from jenkins
> {code}
> compile-test-worker:
> [echo] Project: shims
> [echo] Compiling shim tests against hadoop 20S
> [echo] Test srcdir :
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test
>
> [javac] Compiling 3 source files to
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/build/shims/test/classes
> [javac]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:103:
> cannot find symbol
> [javac] symbol : method
> startDelegationTokenSecretManager(org.apache.hadoop.conf.Configuration)
> [javac] location: class
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.Server
> [javac] super.startDelegationTokenSecretManager(conf);
> [javac] ^
> [javac]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:100:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac] 2 errors
>[subant] Failure for target 'test' of:
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml
>[subant] The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:174:
> The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:154:
> The following error occurred while executing this line:
>[subant]
> /home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:106:
> Compile failed; see the compiler error output for details.
> {\code}
> I found HIVE-3255 changed the interface of
> HadoopThriftAuthBridge20S.Server.startDelegationTokenSecretManager to
> 'startDelegationTokenSecretManager(Configuration conf, Object hms)'. But in
> TestHadoop20SAuthBridge, we are still using the old interface which is
> 'startDelegationTokenSecretManager(conf)'.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4838) Refactor MapJoin HashMap code to improve testability and readability
[ https://issues.apache.org/jira/browse/HIVE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726791#comment-13726791 ] Ashutosh Chauhan commented on HIVE-4838: Yeah I misunderstood that piece. Another question : I see that you have improved memory handling. But I am confused why do we need to monitor memory usage here anyway? This predate your patch so question here really is do we need memory handler here? It seems it was put in place so that we can proactively kill local task before it throws OOM. But since MapRedLocalTask anyways catches OOM exception, it seems like even if local task didnt kill itself before OOM'ing, we should be fine since MapRedLocalTask will take care of OOM exception. > Refactor MapJoin HashMap code to improve testability and readability > > > Key: HIVE-4838 > URL: https://issues.apache.org/jira/browse/HIVE-4838 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Brock Noland > Attachments: HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, > HIVE-4838.patch, HIVE-4838.patch > > > MapJoin is an essential component for high performance joins in Hive and the > current code has done great service for many years. However, the code is > showing it's age and currently suffers from the following issues: > * Uses static state via the MapJoinMetaData class to pass serialization > metadata to the Key, Row classes. > * The api of a logical "Table Container" is not defined and therefore it's > unclear what apis HashMapWrapper > needs to publicize. Additionally HashMapWrapper has many used public methods. > * HashMapWrapper contains logic to serialize, test memory bounds, and > implement the table container. Ideally these logical units could be seperated > * HashTableSinkObjectCtx has unused fields and unused methods > * CommonJoinOperator and children use ArrayList on left hand side when only > List is required > * There are unused classes MRU, DCLLItemm and classes which duplicate > functionality MapJoinSingleKey and MapJoinDoubleKeys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4789) FetchOperator fails on partitioned Avro data
[
https://issues.apache.org/jira/browse/HIVE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726766#comment-13726766
]
Ashutosh Chauhan edited comment on HIVE-4789 at 8/1/13 7:16 PM:
On the second thought I think it doesnt make sense to delay this patch for
HIVE-4967 we can update golden files again if it happens so.
I reviewed the patch and I have question for [~busbey]
As far as I can see, problem should have been fixed by your changes in
FetchOperator.java? I didn't get why you needed to make changes in
MetaStoreUtils.java. Can you briefly explain why is it needed?
was (Author: ashutoshc):
On the second thought I think it doesnt make sense to delay this patch for
HIVE-4967 we can update golden files again if it happens so.
I reviewed the patch and I have question for [~busbey]
As far as I can see, problem should have been fixed by your changes in
FetchTask.java? I didn't get why you needed to make changes in
MetaStoreUtils.java. Can you briefly explain why is it needed?
> FetchOperator fails on partitioned Avro data
>
>
> Key: HIVE-4789
> URL: https://issues.apache.org/jira/browse/HIVE-4789
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Attachments: HIVE-4789.1.patch.txt, HIVE-4789.2.patch.txt
>
>
> HIVE-3953 fixed using partitioned avro tables for anything that used the
> MapOperator, but those that rely on FetchOperator still fail with the same
> error.
> e.g.
> {code}
> SELECT * FROM partitioned_avro LIMIT 5;
> SELECT * FROM partitioned_avro WHERE partition_col=value;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4789) FetchOperator fails on partitioned Avro data
[
https://issues.apache.org/jira/browse/HIVE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726766#comment-13726766
]
Ashutosh Chauhan commented on HIVE-4789:
On the second thought I think it doesnt make sense to delay this patch for
HIVE-4967 we can update golden files again if it happens so.
I reviewed the patch and I have question for [~busbey]
As far as I can see, problem should have been fixed by your changes in
FetchTask.java? I didn't get why you needed to make changes in
MetaStoreUtils.java. Can you briefly explain why is it needed?
> FetchOperator fails on partitioned Avro data
>
>
> Key: HIVE-4789
> URL: https://issues.apache.org/jira/browse/HIVE-4789
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Attachments: HIVE-4789.1.patch.txt, HIVE-4789.2.patch.txt
>
>
> HIVE-3953 fixed using partitioned avro tables for anything that used the
> MapOperator, but those that rely on FetchOperator still fail with the same
> error.
> e.g.
> {code}
> SELECT * FROM partitioned_avro LIMIT 5;
> SELECT * FROM partitioned_avro WHERE partition_col=value;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4976) TestHadoop20SAuthBridge needs to be updated after HIVE-3255
Yin Huai created HIVE-4976:
--
Summary: TestHadoop20SAuthBridge needs to be updated after
HIVE-3255
Key: HIVE-4976
URL: https://issues.apache.org/jira/browse/HIVE-4976
Project: Hive
Issue Type: Bug
Reporter: Yin Huai
I found the following error appearing. Here is the log from jenkins
{code}
compile-test-worker:
[echo] Project: shims
[echo] Compiling shim tests against hadoop 20S
[echo] Test srcdir :
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test
[javac] Compiling 3 source files to
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/build/shims/test/classes
[javac]
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:103:
cannot find symbol
[javac] symbol : method
startDelegationTokenSecretManager(org.apache.hadoop.conf.Configuration)
[javac] location: class
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.Server
[javac] super.startDelegationTokenSecretManager(conf);
[javac] ^
[javac]
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java:100:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac] 2 errors
[subant] Failure for target 'test' of:
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml
[subant] The following error occurred while executing this line:
[subant]
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:174:
The following error occurred while executing this line:
[subant]
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:154:
The following error occurred while executing this line:
[subant]
/home/jenkins/jenkins-slave/workspace/Hive-trunk-h0.21/hive/shims/build.xml:106:
Compile failed; see the compiler error output for details.
{\code}
I found HIVE-3255 changed the interface of
HadoopThriftAuthBridge20S.Server.startDelegationTokenSecretManager to
'startDelegationTokenSecretManager(Configuration conf, Object hms)'. But in
TestHadoop20SAuthBridge, we are still using the old interface which is
'startDelegationTokenSecretManager(conf)'.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4843) Refactoring MapRedTask and ExecDriver for better re-usability (for tez) and readability
[ https://issues.apache.org/jira/browse/HIVE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726738#comment-13726738 ] Hudson commented on HIVE-4843: -- FAILURE: Integrated in Hive-trunk-h0.21 #2237 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2237/]) HIVE-4843: Refactoring MapRedTask and ExecDriver for better re-usability and readability (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509081) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java > Refactoring MapRedTask and ExecDriver for better re-usability (for tez) and > readability > --- > > Key: HIVE-4843 > URL: https://issues.apache.org/jira/browse/HIVE-4843 > Project: Hive > Issue Type: Bug >Affects Versions: 0.12.0, tez-branch >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 0.12.0 > > Attachments: HIVE-4843.1.patch, HIVE-4843.2.patch, HIVE-4843.3.patch, > HIVE-4843.4.patch, HIVE-4843.5.patch > > > Currently, there are static apis in multiple locations in ExecDriver and > MapRedTask that can be leveraged if put in the already existing utility class > in the exec package. This would help making the code more maintainable, > readable and also re-usable by other run-time infra such as tez. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only task to its child task
[ https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726737#comment-13726737 ] Hudson commented on HIVE-4827: -- FAILURE: Integrated in Hive-trunk-h0.21 #2237 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2237/]) HIVE-4827: Merge a Map-only task to its child task (Yin Huai via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509082) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/QueryPlanTreeTransformation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java * /hive/trunk/ql/src/test/queries/clientpositive/auto_join33.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer1.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer3.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer4.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer5.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer6.q * /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer7.q * /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin1.q * /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin2.q * /hive/trunk/ql/src/test/queries/clientpositive/union34.q * /hive/trunk/ql/src/test/results/clientpositive/auto_join0.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join10.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join15.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join2.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join22.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join23.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join24.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join26.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join32.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_join33.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer1.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer3.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer4.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer6.q.out * /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer7.q.out * /hive/trunk/ql/src/test/results/clientpositive/join28.q.out * /hive/trunk/ql/src/test/results/clientpositive/join32.q.out * /hive/trunk/ql/src/test/results/clientpositive/join33.q.out * /hive/trunk/ql/src/test/results/clientpositive/join_star.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery2.q.out * /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out * /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin1.q.out * /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin2.q.out * /hive/trunk/ql/src/test/results/clientpositive/multi_join_union.q.out * /hive/trunk/ql/src/test/results/clientpositive/union34.q.out > Merge a Map-only task to its child task > --- > > Key: HIVE-4827 > URL: https://issues.apache.org/jira/browse/HIVE-4827 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.12.0 >Reporter: Yin Huai >Assignee: Yin Huai > Fix For: 0.12.0 > > Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch, HIVE-4827.3.p
[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
[
https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726735#comment-13726735
]
Hive QA commented on HIVE-4388:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12595442/HIVE-4388.patch
{color:green}SUCCESS:{color} +1 2749 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/276/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/276/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
> HBase tests fail against Hadoop 2
> -
>
> Key: HIVE-4388
> URL: https://issues.apache.org/jira/browse/HIVE-4388
> Project: Hive
> Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Brock Noland
> Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch,
> HIVE-4388.patch, HIVE-4388-wip.txt
>
>
> Currently we're building by default against 0.92. When you run against hadoop
> 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963.
> HIVE-3861 upgrades the version of hbase used. This will get you past the
> problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4972) update code generated by thrift for DemuxOperator and MuxOperator
[ https://issues.apache.org/jira/browse/HIVE-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726725#comment-13726725 ] Gunther Hagleitner commented on HIVE-4972: -- Thanks [~brocknoland]. Looked at it. Seems utterly unrelated. > update code generated by thrift for DemuxOperator and MuxOperator > - > > Key: HIVE-4972 > URL: https://issues.apache.org/jira/browse/HIVE-4972 > Project: Hive > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4972.D11907.1.patch > > > HIVE-2206 introduces two new operators, which are DemuxOperator and > MuxOperator. queryplan.thrift has been updated. But code generated by thrift > should be also updated -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4794) Unit e2e tests for vectorization
[ https://issues.apache.org/jira/browse/HIVE-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Murphy updated HIVE-4794: -- Attachment: HIVE-4794.4-vectorization.patch Handled side effects in other tests (show table) of adding hive table. fix bug to always create the alltypesorc file in the local filesystem. > Unit e2e tests for vectorization > > > Key: HIVE-4794 > URL: https://issues.apache.org/jira/browse/HIVE-4794 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4794.1.patch, HIVE-4794.2.patch, HIVE-4794.3.patch, > HIVE-4794.3-vectorization.patch, HIVE-4794.4-vectorization.patch, > hive-4794.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 13021: Vectorization Tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13021/ --- (Updated Aug. 1, 2013, 6:26 p.m.) Review request for hive, Eric Hanson, Jitendra Pandey, Remus Rusanu, and Sarvesh Sakalanaga. Changes --- Handled side effects in other tests (show table) of adding hive table. fix bug to always create the alltypesorc file in the local filesystem. Bugs: HIVE-4794 https://issues.apache.org/jira/browse/HIVE-4794 Repository: hive-git Description --- These test cover all types, aggregates, and operators currently supported for vectorization. The queries are executed over a specially crafted data set which covers all the interesting classes of batch for each type: all nulls, repeating value, no nulls, and random values, to fully exercise the vectorization stack. The queries were stabilized against a text test oracle in order to validate results. This patch depends on: HIVE-4525 HIVE-4922 HIVE-4931 Diffs (updated) - .gitignore c0e9b3c ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 97436c5 ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 79390a9 ql/src/test/org/apache/hadoop/hive/ql/exec/vector/util/AllVectorTypesRecord.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/exec/vector/util/OrcFileGenerator.java PRE-CREATION ql/src/test/queries/clientpositive/vectorization_0.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_1.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_10.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_11.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_12.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_13.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_14.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_15.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_16.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_2.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_3.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_4.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_5.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_6.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_7.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_8.q PRE-CREATION ql/src/test/queries/clientpositive/vectorization_9.q PRE-CREATION ql/src/test/results/clientpositive/add_part_exist.q.out e64cc3c ql/src/test/results/clientpositive/alter1.q.out a944b29 ql/src/test/results/clientpositive/alter2.q.out 82c78ad ql/src/test/results/clientpositive/alter3.q.out 43064f7 ql/src/test/results/clientpositive/alter4.q.out ed8b4f6 ql/src/test/results/clientpositive/alter5.q.out c1df0c8 ql/src/test/results/clientpositive/alter_index.q.out a1738ad ql/src/test/results/clientpositive/alter_rename_partition.q.out c6fa3f9 ql/src/test/results/clientpositive/describe_table_json.q.out d4f497f ql/src/test/results/clientpositive/index_creation.q.out bddaa35 ql/src/test/results/clientpositive/input2.q.out e99d405 ql/src/test/results/clientpositive/input3.q.out be091ea ql/src/test/results/clientpositive/show_tables.q.out d434c0e ql/src/test/results/clientpositive/vectorization_0.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_1.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_10.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_11.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_12.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_13.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_14.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_15.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_16.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_2.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_3.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_4.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_5.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_6.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_7.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_8.q.out PRE-CREATION ql/src/test/results/clientpositive/vectorization_9.q.out PRE-CREATION Diff: https://reviews.apache.org/r/13021/diff/ Testing --- Thanks, tony murphy
[jira] [Commented] (HIVE-4343) HS2 with kerberos- local task for map join fails
[ https://issues.apache.org/jira/browse/HIVE-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726683#comment-13726683 ] Prasad Mujumdar commented on HIVE-4343: --- I agree with Gunther. We should go ahead with fixing this issue separately from HIVE-4470. > HS2 with kerberos- local task for map join fails > > > Key: HIVE-4343 > URL: https://issues.apache.org/jira/browse/HIVE-4343 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-4343.1.patch > > > With hive server2 configured with kerberos security, when a (map) join query > is run, it results in failure with "GSSException: No valid credentials > provided " -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4789) FetchOperator fails on partitioned Avro data
[
https://issues.apache.org/jira/browse/HIVE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726652#comment-13726652
]
Ashutosh Chauhan commented on HIVE-4789:
Some of the changes in golden file (particularly in .xml plans) we should be
able to avoid by improving HIVE-4967 I will do investigation to check this.
> FetchOperator fails on partitioned Avro data
>
>
> Key: HIVE-4789
> URL: https://issues.apache.org/jira/browse/HIVE-4789
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Attachments: HIVE-4789.1.patch.txt, HIVE-4789.2.patch.txt
>
>
> HIVE-3953 fixed using partitioned avro tables for anything that used the
> MapOperator, but those that rely on FetchOperator still fail with the same
> error.
> e.g.
> {code}
> SELECT * FROM partitioned_avro LIMIT 5;
> SELECT * FROM partitioned_avro WHERE partition_col=value;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4470) HS2 should disable local query execution
[ https://issues.apache.org/jira/browse/HIVE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726650#comment-13726650 ] Prasad Mujumdar commented on HIVE-4470: --- The restrict list parameter itself is part of the list implicitly. > HS2 should disable local query execution > > > Key: HIVE-4470 > URL: https://issues.apache.org/jira/browse/HIVE-4470 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair > > Hive can run queries in local mode (instead of using a cluster), if the size > is small. This happens when "hive.exec.mode.local.auto" is set to true. > This would affect the stability of the hive server2 node, if you have heavy > query processing happening on it. Bugs in udfs triggered by a bad record can > potentially add very heavy load making the server inaccessible. > By default, HS2 should set these parameters to disallow local execution or > send and error message if user tries to set these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4960) lastAlias in CommonJoinOperator is not used
[
https://issues.apache.org/jira/browse/HIVE-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726647#comment-13726647
]
Hudson commented on HIVE-4960:
--
FAILURE: Integrated in Hive-trunk-hadoop1-ptest #112 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/112/])
HIVE-4960 : lastAlias in CommonJoinOperator is not used (Yin Huai via Ashutosh
Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509314)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
> lastAlias in CommonJoinOperator is not used
> ---
>
> Key: HIVE-4960
> URL: https://issues.apache.org/jira/browse/HIVE-4960
> Project: Hive
> Issue Type: Improvement
>Reporter: Yin Huai
>Assignee: Yin Huai
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4960.D11895.1.patch
>
>
> In CommonJoinOperator, there is object called lastAlias. The initial value of
> this object is 'null'. After tracing the usage of this object, I found that
> there is no place to change the value of this object. Also, it is only used
> in processOp in JoinOperator and MapJoinOperator as
> {code}
> if ((lastAlias == null) || (!lastAlias.equals(alias))) {
> nextSz = joinEmitInterval;
> }
> {\code}
> Since lastAlias will always be null, we will assign joinEmitInterval to
> nextSz every time we get a row. Later in processOp, we have
> {code}
> nextSz = getNextSize(nextSz);
> {\code}
> Because we reset the value of nextSz to joinEmitInterval every time we get a
> row, seems that getNextSize will not be used as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726637#comment-13726637 ] Brock Noland commented on HIVE-2482: bq. I usually write end to end tests for new features (mostly for regression testing but also to demonstrate what functionality is getting introduced). For bug fixes I write junit tests to test the part which I am fixing. Hey guys, just to make sure I understand, unit tests are still strongly encouraged for new features as well as end-to-end tests? > Convenience UDFs for binary data type > - > > Key: HIVE-2482 > URL: https://issues.apache.org/jira/browse/HIVE-2482 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Mark Wagner > Attachments: HIVE-2482.1.patch > > > HIVE-2380 introduced binary data type in Hive. It will be good to have > following udfs to make it more useful: > * UDF's to convert to/from hex string > * UDF's to convert to/from string using a specific encoding > * UDF's to convert to/from base64 string > * UDF's to convert to/from non-string types using a particular serde -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[
https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726632#comment-13726632
]
Edward Capriolo commented on HIVE-2482:
---
{quote}
I usually write end to end tests for new features (mostly for regression
testing but also to demonstrate what functionality is getting introduced). For
bug fixes I write junit tests to test the part which I am fixing.
Obviously there are lot more bug fixes than features, so in most circumstances
junit tests are encouraged, though in this particular instance there are new
udfs (thus new features) so here end to end tests does make sense.
{quote}
Well said. Hive is releasable trunk, so every commit for a new feature has to
be usable. q tests show the usable.
Moreover the q tests serve as the ONLY true documentation we have, since most
things never hit the wiki.
> Convenience UDFs for binary data type
> -
>
> Key: HIVE-2482
> URL: https://issues.apache.org/jira/browse/HIVE-2482
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.9.0
>Reporter: Ashutosh Chauhan
>Assignee: Mark Wagner
> Attachments: HIVE-2482.1.patch
>
>
> HIVE-2380 introduced binary data type in Hive. It will be good to have
> following udfs to make it more useful:
> * UDF's to convert to/from hex string
> * UDF's to convert to/from string using a specific encoding
> * UDF's to convert to/from base64 string
> * UDF's to convert to/from non-string types using a particular serde
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[
https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726630#comment-13726630
]
Edward Capriolo commented on HIVE-2482:
---
{quote}
Is it reasonable to say that we strongly encourage unit testing but we'd like
to see at least one end to end test?
{quote}
^ that has always been the case.
I am ok with moving to a simpler/faster/east to test style. But asserting on
evaluate() really does not cover everything (because of the state). Neither
does select udf(x) from src limit 1 (bugs from multiple rows). I think we need
to be simulating from the standpoint of an operator interacting with the UDF
normally. It would not be full end-to-end but more then just cherry picking
execution points and validating results.
> Convenience UDFs for binary data type
> -
>
> Key: HIVE-2482
> URL: https://issues.apache.org/jira/browse/HIVE-2482
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.9.0
>Reporter: Ashutosh Chauhan
>Assignee: Mark Wagner
> Attachments: HIVE-2482.1.patch
>
>
> HIVE-2380 introduced binary data type in Hive. It will be good to have
> following udfs to make it more useful:
> * UDF's to convert to/from hex string
> * UDF's to convert to/from string using a specific encoding
> * UDF's to convert to/from base64 string
> * UDF's to convert to/from non-string types using a particular serde
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726621#comment-13726621 ] Ashutosh Chauhan commented on HIVE-2482: I usually write end to end tests for new features (mostly for regression testing but also to demonstrate what functionality is getting introduced). For bug fixes I write junit tests to test the part which I am fixing. Obviously there are lot more bug fixes than features, so in most circumstances junit tests are encouraged, though in this particular instance there are new udfs (thus new features) so here end to end tests does make sense. > Convenience UDFs for binary data type > - > > Key: HIVE-2482 > URL: https://issues.apache.org/jira/browse/HIVE-2482 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Mark Wagner > Attachments: HIVE-2482.1.patch > > > HIVE-2380 introduced binary data type in Hive. It will be good to have > following udfs to make it more useful: > * UDF's to convert to/from hex string > * UDF's to convert to/from string using a specific encoding > * UDF's to convert to/from base64 string > * UDF's to convert to/from non-string types using a particular serde -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2482) Convenience UDFs for binary data type
[ https://issues.apache.org/jira/browse/HIVE-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726587#comment-13726587 ] Brock Noland commented on HIVE-2482: Just going through this chain. I really appreciate the unit tests as well! There is a good point about object reuse though. Is it reasonable to say that we strongly encourage unit testing but we'd like to see at least one end to end test? > Convenience UDFs for binary data type > - > > Key: HIVE-2482 > URL: https://issues.apache.org/jira/browse/HIVE-2482 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Mark Wagner > Attachments: HIVE-2482.1.patch > > > HIVE-2380 introduced binary data type in Hive. It will be good to have > following udfs to make it more useful: > * UDF's to convert to/from hex string > * UDF's to convert to/from string using a specific encoding > * UDF's to convert to/from base64 string > * UDF's to convert to/from non-string types using a particular serde -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
