[jira] [Commented] (HIVE-13727) Getting error Failed rule: 'orderByClause clusterByClause distributeByClause sortByClause limitClause can only be applied to the whole union.' in subquery
[
https://issues.apache.org/jira/browse/HIVE-13727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571107#comment-15571107
]
Lingang Deng commented on HIVE-13727:
-
I got the exception when parsing sql with hive1.2.1
> Getting error Failed rule: 'orderByClause clusterByClause distributeByClause
> sortByClause limitClause can only be applied to the whole union.' in subquery
> ---
>
> Key: HIVE-13727
> URL: https://issues.apache.org/jira/browse/HIVE-13727
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>
> The error comes in the following query:
> {noformat}
> SELECT *
> FROM
> (SELECT *
>FROM srcpart a
>WHERE a.ds = '2008-04-08'
> AND a.hr = '11'
>ORDER BY a.key LIMIT 5
>UNION ALL
>SELECT *
>FROM srcpart b
>WHERE b.ds = '2008-04-08'
> AND b.hr = '14'
>ORDER BY b.key LIMIT 5) subq
> ORDER BY KEY LIMIT 5
> {noformat}
> But the following query works:
> {noformat}
> SELECT *
> FROM
> (SELECT *
>FROM
> (SELECT *
> FROM srcpart a
> WHERE a.ds = '2008-04-08'
> AND a.hr = '11'
> ORDER BY a.key LIMIT 5) pa
>UNION ALL SELECT *
>FROM
> (SELECT *
> FROM srcpart b
> WHERE b.ds = '2008-04-08'
> AND b.hr = '14'
> ORDER BY b.key LIMIT 5) pb) subq
> ORDER BY KEY LIMIT 5
> {noformat}
> The queries are logically identical, the query that's rejected has dummy
> select * clauses around the sub-queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14933) include argparse with LLAP scripts to support antique Python versions
[
https://issues.apache.org/jira/browse/HIVE-14933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571146#comment-15571146
]
Hive QA commented on HIVE-14933:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12832979/HIVE-14933.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10560 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1521/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1521/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1521/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12832979 - PreCommit-HIVE-Build
> include argparse with LLAP scripts to support antique Python versions
> -
>
> Key: HIVE-14933
> URL: https://issues.apache.org/jira/browse/HIVE-14933
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14933.01.patch, HIVE-14933.patch
>
>
> The module is a standalone file, and it's under Python license that is
> compatible with Apache. In the long term we should probably just move
> LlapServiceDriver code entirely to Java, as right now it's a combination of
> part-py, part-java.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14921) Move slow CliDriver tests to MiniLlap - part 2
[
https://issues.apache.org/jira/browse/HIVE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571154#comment-15571154
]
Hive QA commented on HIVE-14921:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12832981/HIVE-14921.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1522/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1522/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1522/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-13 07:36:44.443
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1522/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 07:36:44.445
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f9843ac HIVE-14928 Analyze table no scan mess up schema (Jimmy
Xiang, reviewed by Prasanth Jayachandran)
+ git clean -f -d
Removing llap-server/src/main/resources/argparse.py
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f9843ac HIVE-14928 Analyze table no scan mess up schema (Jimmy
Xiang, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 07:36:45.423
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
fatal: git diff header lacks filename information when removing 0 leading
pathname components (line 4)
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12832981 - PreCommit-HIVE-Build
> Move slow CliDriver tests to MiniLlap - part 2
> --
>
> Key: HIVE-14921
> URL: https://issues.apache.org/jira/browse/HIVE-14921
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14921.1.patch, HIVE-14921.1.patch,
> HIVE-14921.2.patch
>
>
> Continuation to HIVE-14877
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14921) Move slow CliDriver tests to MiniLlap - part 2
[ https://issues.apache.org/jira/browse/HIVE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14921: - Attachment: HIVE-14921.2.patch > Move slow CliDriver tests to MiniLlap - part 2 > -- > > Key: HIVE-14921 > URL: https://issues.apache.org/jira/browse/HIVE-14921 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14921.1.patch, HIVE-14921.1.patch, > HIVE-14921.2.patch, HIVE-14921.2.patch > > > Continuation to HIVE-14877 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13316) Upgrade to Calcite 1.10
[ https://issues.apache.org/jira/browse/HIVE-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13316: --- Attachment: HIVE-13316.06.patch > Upgrade to Calcite 1.10 > --- > > Key: HIVE-13316 > URL: https://issues.apache.org/jira/browse/HIVE-13316 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13316.01.patch, HIVE-13316.02.patch, > HIVE-13316.05.patch, HIVE-13316.06.patch, HIVE-13316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13316) Upgrade to Calcite 1.10
[ https://issues.apache.org/jira/browse/HIVE-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13316: --- Attachment: HIVE-13316.07.patch > Upgrade to Calcite 1.10 > --- > > Key: HIVE-13316 > URL: https://issues.apache.org/jira/browse/HIVE-13316 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13316.01.patch, HIVE-13316.02.patch, > HIVE-13316.05.patch, HIVE-13316.06.patch, HIVE-13316.07.patch, > HIVE-13316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13316) Upgrade to Calcite 1.10
[ https://issues.apache.org/jira/browse/HIVE-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13316: --- Attachment: (was: HIVE-13316.06.patch) > Upgrade to Calcite 1.10 > --- > > Key: HIVE-13316 > URL: https://issues.apache.org/jira/browse/HIVE-13316 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13316.01.patch, HIVE-13316.02.patch, > HIVE-13316.05.patch, HIVE-13316.07.patch, HIVE-13316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14916) Reduce the memory requirements for Spark tests
[ https://issues.apache.org/jira/browse/HIVE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HIVE-14916: -- Attachment: HIVE-14916.004.patch > Reduce the memory requirements for Spark tests > -- > > Key: HIVE-14916 > URL: https://issues.apache.org/jira/browse/HIVE-14916 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Dapeng Sun > Attachments: HIVE-14916.001.patch, HIVE-14916.002.patch, > HIVE-14916.003.patch, HIVE-14916.004.patch > > > As HIVE-14887, we need to reduce the memory requirements for Spark tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[
https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571266#comment-15571266
]
Hive QA commented on HIVE-14940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833004/HIVE-14940.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_column_names_with_leading_and_trailing_spaces]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1523/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1523/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1523/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833004 - PreCommit-HIVE-Build
> MiniTezCliDriver - switch back to SQL metastore as default
> --
>
> Key: HIVE-14940
> URL: https://issues.apache.org/jira/browse/HIVE-14940
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14940.1.patch
>
>
> HBase setup for metastore in MiniTez is taking around 3 mins for setup. The
> actual runtime of the queries is typically much lower. To avoid the high
> overhead we should be switch back to SQL metastore as default and if required
> we can have dedicated set of tests that run against hbase metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Attachment: HIVE-14940.1.patch > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-11394:
Status: In Progress (was: Patch Available)
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> keys: cint (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false
> usesVectorUDFAdaptor: false
> vectorized: true
> Reducer 2
> Execution mode: vectorized, llap
> Reduce Vectoriza
[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-11394:
Status: Patch Available (was: In Progress)
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> keys: cint (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false
> usesVectorUDFAdaptor: false
> vectorized: true
> Reducer 2
> Execution mode: vectorized, llap
> Reduce Vectoriza
[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-11394:
Attachment: HIVE-11394.093.patch
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> keys: cint (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false
> usesVectorUDFAdaptor: false
> vectorized: true
> Reducer 2
> Execution mode: vectorized, llap
> Reduce Vectorization:
>
[jira] [Commented] (HIVE-12765) Support Intersect (distinct/all) Except (distinct/all) Minus (distinct/all)
[
https://issues.apache.org/jira/browse/HIVE-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571416#comment-15571416
]
Hive QA commented on HIVE-12765:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833005/HIVE-12765.03.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 10566 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join8]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input25]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input26]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join8]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_oneskew_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part14]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_25]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_null_projection]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_1]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_2]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_top_level]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_null_projection]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join8]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join8]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part14]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_25]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_25]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_top_level]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1524/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1524/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1524/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833005 - PreCommit-HIVE-Build
> Support Intersect (distinct/all) Except (distinct/all) Minus (distinct/all)
> ---
>
> Key: HIVE-12765
> URL: https://issues.apache.org/jira/browse/HIVE-12765
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12765.01.patch, HIVE-12765.02.patch,
> HIVE-12765.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571568#comment-15571568
]
Hive QA commented on HIVE-11394:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833079/HIVE-11394.093.patch
{color:green}SUCCESS:{color} +1 due to 160 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10530 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver-orc_llap.q-delete_where_non_partitioned.q-vector_groupby_mapjoin.q-and-27-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1525/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1525/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1525/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833079 - PreCommit-HIVE-Build
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num
[jira] [Commented] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571593#comment-15571593
]
Matt McCline commented on HIVE-11394:
-
Test failures
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
and org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar] are
unrelated.
Committed to master.
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> keys: cint (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
[jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-11394:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> keys: cint (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true
> groupByVectorOutput: true
> inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> allNative: false
> usesVectorUDFAdaptor: false
> vectorized: true
> Red
[jira] [Commented] (HIVE-13873) Column pruning for nested fields
[
https://issues.apache.org/jira/browse/HIVE-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571685#comment-15571685
]
Hive QA commented on HIVE-13873:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833051/HIVE-13873.3.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10569 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[struct_in_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_named_struct]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_struct]
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1526/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1526/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1526/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833051 - PreCommit-HIVE-Build
> Column pruning for nested fields
>
>
> Key: HIVE-13873
> URL: https://issues.apache.org/jira/browse/HIVE-13873
> Project: Hive
> Issue Type: New Feature
> Components: Logical Optimizer
>Reporter: Xuefu Zhang
>Assignee: Ferdinand Xu
> Attachments: HIVE-13873.1.patch, HIVE-13873.2.patch,
> HIVE-13873.3.patch, HIVE-13873.patch, HIVE-13873.wip.patch
>
>
> Some columnar file formats such as Parquet store fields in struct type also
> column by column using encoding described in Google Dramel pager. It's very
> common in big data where data are stored in structs while queries only needs
> a subset of the the fields in the structs. However, presently Hive still
> needs to read the whole struct regardless whether all fields are selected.
> Therefore, pruning unwanted sub-fields in struct or nested fields at file
> reading time would be a big performance boost for such scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14921) Move slow CliDriver tests to MiniLlap - part 2
[
https://issues.apache.org/jira/browse/HIVE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571811#comment-15571811
]
Hive QA commented on HIVE-14921:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833062/HIVE-14921.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10533 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge10]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[pcs]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1527/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1527/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1527/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833062 - PreCommit-HIVE-Build
> Move slow CliDriver tests to MiniLlap - part 2
> --
>
> Key: HIVE-14921
> URL: https://issues.apache.org/jira/browse/HIVE-14921
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14921.1.patch, HIVE-14921.1.patch,
> HIVE-14921.2.patch, HIVE-14921.2.patch
>
>
> Continuation to HIVE-14877
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14679) csv2/tsv2 output format disables quoting by default and it's difficult to enable
[
https://issues.apache.org/jira/browse/HIVE-14679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571892#comment-15571892
]
Kenneth MacArthur commented on HIVE-14679:
--
Section 2.6 of RFC 4180 says:
"Fields containing line breaks (CRLF), double quotes, and commas should be
enclosed in double-quotes."
It seems strange, then, to disable quoting for the csv2 output format by
default.
What's also strange is that when quoting is disabled, values are in fact still
'quoted' with a null character (00), rather than no character at all (as
described in [~ngangam]'s comment on HIVE-9788). This doesn't appear to be
mentioned anywhere in RFC 4180.
May I suggest that:
- Quoting should be enabled by default for csv2, tsv2 and dsv.
- Disabling quoting should be possible using a beeline argument.
- Disabling quoting should not result in the output of a null character in
place of a visible quote - there should simply be no quote character at all in
this case.
> csv2/tsv2 output format disables quoting by default and it's difficult to
> enable
>
>
> Key: HIVE-14679
> URL: https://issues.apache.org/jira/browse/HIVE-14679
> Project: Hive
> Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Jianguo Tian
>
> Over in HIVE-9788 we made quoting optional for csv2/tsv2.
> However I see the following issues:
> * JIRA doc doesn't mention it's disabled by default, this should be there an
> in the output of beeline help.
> * The JIRA says the property is {{--disableQuotingForSV}} but it's actually a
> system property. We should not use a system property as it's non-standard so
> extremely hard for users to set. For example I must do: {{env
> HADOOP_CLIENT_OPTS="-Ddisable.quoting.for.sv=false" beeline ...}}
> * The arg {{--disableQuotingForSV}} should be documented in beeline help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14916) Reduce the memory requirements for Spark tests
[
https://issues.apache.org/jira/browse/HIVE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571950#comment-15571950
]
Hive QA commented on HIVE-14916:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833069/HIVE-14916.004.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10560 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1529/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1529/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1529/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833069 - PreCommit-HIVE-Build
> Reduce the memory requirements for Spark tests
> --
>
> Key: HIVE-14916
> URL: https://issues.apache.org/jira/browse/HIVE-14916
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ferdinand Xu
>Assignee: Dapeng Sun
> Attachments: HIVE-14916.001.patch, HIVE-14916.002.patch,
> HIVE-14916.003.patch, HIVE-14916.004.patch
>
>
> As HIVE-14887, we need to reduce the memory requirements for Spark tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572001#comment-15572001 ] Illya Yalovyy commented on HIVE-14927: -- [~aihuaxu], [~ashutoshc], [~ctang.ma], [~szehon] Could you please take a look at this CR? https://reviews.apache.org/r/52743/ > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572022#comment-15572022 ] Chaoyu Tang commented on HIVE-14927: LGTM, +1 pending precommit tests > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14679) csv2/tsv2 output format disables quoting by default and it's difficult to enable
[
https://issues.apache.org/jira/browse/HIVE-14679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572042#comment-15572042
]
Naveen Gangam commented on HIVE-14679:
--
Its my understanding that quoting is NOT optional for csv2/tsv2 formats. These
formats were introduced specifically to rid the quotes around the column
values. Since we could not just change the original csv/tsv formats to not wrap
values in quotes for backward compatibility reasons, we had to introduce new
output formats.
Its been a while but I believe HIVE-9788 makes it optional for the original sv
formats and not affect the csv2/tsv2 formats. [~Ferd] please correct me if I am
wrong.
> csv2/tsv2 output format disables quoting by default and it's difficult to
> enable
>
>
> Key: HIVE-14679
> URL: https://issues.apache.org/jira/browse/HIVE-14679
> Project: Hive
> Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Jianguo Tian
>
> Over in HIVE-9788 we made quoting optional for csv2/tsv2.
> However I see the following issues:
> * JIRA doc doesn't mention it's disabled by default, this should be there an
> in the output of beeline help.
> * The JIRA says the property is {{--disableQuotingForSV}} but it's actually a
> system property. We should not use a system property as it's non-standard so
> extremely hard for users to set. For example I must do: {{env
> HADOOP_CLIENT_OPTS="-Ddisable.quoting.for.sv=false" beeline ...}}
> * The arg {{--disableQuotingForSV}} should be documented in beeline help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572107#comment-15572107 ] Illya Yalovyy commented on HIVE-14927: -- Thank you! Is Hive QA job is down? > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14679) csv2/tsv2 output format disables quoting by default and it's difficult to enable
[
https://issues.apache.org/jira/browse/HIVE-14679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572121#comment-15572121
]
Ferdinand Xu commented on HIVE-14679:
-
bq. It optional for the original sv formats and not affect the csv2/tsv2 formats
You can see the code for HIVE-9788 that by default the quoting is disabled. And
it is about csv2/tsv2 as the release note said I think. But it has been a while
and need double check. [~JonnyR], can you please confirm this?
{noformat}
+ private boolean isQuotingDisabled() {
+String quotingDisabledStr =
System.getProperty(SeparatedValuesOutputFormat.DISABLE_QUOTING_FOR_SV);
+if (quotingDisabledStr == null || quotingDisabledStr.isEmpty()) {
+ // default is disabling the double quoting for separated value
+ return true;
+}
+String parsedOptionStr = quotingDisabledStr.toLowerCase();
+if (parsedOptionStr.equals("false") || parsedOptionStr.equals("true")) {
+ return Boolean.valueOf(parsedOptionStr);
+} else {
+ beeLine.error("System Property disable.quoting.for.sv is now " +
parsedOptionStr
+ + " which only accepts boolean value");
+ return true;
+}
+ }
{noformat}
> csv2/tsv2 output format disables quoting by default and it's difficult to
> enable
>
>
> Key: HIVE-14679
> URL: https://issues.apache.org/jira/browse/HIVE-14679
> Project: Hive
> Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Jianguo Tian
>
> Over in HIVE-9788 we made quoting optional for csv2/tsv2.
> However I see the following issues:
> * JIRA doc doesn't mention it's disabled by default, this should be there an
> in the output of beeline help.
> * The JIRA says the property is {{--disableQuotingForSV}} but it's actually a
> system property. We should not use a system property as it's non-standard so
> extremely hard for users to set. For example I must do: {{env
> HADOOP_CLIENT_OPTS="-Ddisable.quoting.for.sv=false" beeline ...}}
> * The arg {{--disableQuotingForSV}} should be documented in beeline help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Attachment: HIVE-14926.1.patch > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Status: Patch Available (was: Open) Patch-1: switch the query execution from beeline to use JDBC directly. This doesn't seem to be necessary since internally it goes through JDBC as well but limits the supported functionalities. Switch to call JDBC connection directly with which we can use batch and commit or rollback the change. Also, now dryRun option is doing the real dryRun which will execute the scripts and give you the execution info without the final commit. > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572221#comment-15572221 ] Aihua Xu commented on HIVE-14926: - FYI: [~ychena] and [~ashutoshc] Can you help review the change since you are more familiar with it? > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14373) Add integration tests for hive on S3
[ https://issues.apache.org/jira/browse/HIVE-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14373: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~poeppt] [~ayousufi] for the contribution. I committed to master. > Add integration tests for hive on S3 > > > Key: HIVE-14373 > URL: https://issues.apache.org/jira/browse/HIVE-14373 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Thomas Poepping > Fix For: 2.2.0 > > Attachments: HIVE-14373.02.patch, HIVE-14373.03.patch, > HIVE-14373.04.patch, HIVE-14373.05.patch, HIVE-14373.06.patch, > HIVE-14373.patch > > > With Hive doing improvements to run on S3, it would be ideal to have better > integration testing on S3. > These S3 tests won't be able to be executed by HiveQA because it will need > Amazon credentials. We need to write suite based on ideas from the Hadoop > project where: > - an xml file is provided with S3 credentials > - a committer must run these tests manually to verify it works > - the xml file should not be part of the commit, and hiveqa should not run > these tests. > https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14884) Test result cleanup before 2.1.1 release
[ https://issues.apache.org/jira/browse/HIVE-14884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572253#comment-15572253 ] Sergio Peña commented on HIVE-14884: [~mmccline] This job has the tests failures. It was run manually on branch-2.1 https://builds.apache.org/view/H-L/view/Hive/job/HIVE-Branch-Test/10/ > Test result cleanup before 2.1.1 release > > > Key: HIVE-14884 > URL: https://issues.apache.org/jira/browse/HIVE-14884 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14884-branch-2.1.patch, > HIVE-14884.2-branch-2.1.patch > > > There are multiple tests are failing on 2.1 branch. > Before releasing 2.1.1 it would be good to clean up this list -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[
https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572259#comment-15572259
]
Hive QA commented on HIVE-14926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833143/HIVE-14926.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1532/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1532/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1532/
Messages:
{noformat}
This message was trimmed, see log for full details
[exec] + local md5File=spark-2.0.0-bin-hadoop2-without-hive.tgz.md5sum
[exec] + curl -Sso
./../thirdparty/spark-2.0.0-bin-hadoop2-without-hive.tgz.md5sum
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-2.0.0-bin-hadoop2-without-hive.tgz.md5sum
[exec] + cd ./../thirdparty
[exec] + md5sum -c spark-2.0.0-bin-hadoop2-without-hive.tgz.md5sum
[exec] + cd -
[exec] + tar -zxf ./../thirdparty/spark-2.0.0-bin-hadoop2-without-hive.tgz
-C ./target
[exec] spark-2.0.0-bin-hadoop2-without-hive.tgz: OK
[exec]
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit
[exec] + mv ./target/spark-2.0.0-bin-hadoop2-without-hive ./target/spark
[exec] + cp -f ./target/../../..//data/conf/spark/log4j2.properties
./target/spark/conf/
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-it-unit
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-it-unit ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it-unit ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-it-unit
---
[INFO] Compiling 2 source files to
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-it-unit ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-unit ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/tmp/conf
[copy] Copying 15 files to
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-metastore-scripts) @ hive-it-unit
---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/tmp/scripts/metastore
[copy] Copying 337 files to
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/tmp/scripts/metastore
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-unit ---
[INFO] Compiling 114 source files to
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target/test-classes
[INFO] -
[WARNING] COMPILATION WARNING :
[INFO] -
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
Recompile with -Xlint:unchecked for details.
[INFO] 4 warnings
[INFO] -
[INFO] -
[jira] [Commented] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572287#comment-15572287 ] Chaoyu Tang commented on HIVE-14799: The failed tests should not be related to this patch and I am not able to reproduce it in my local env. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-14927: - Status: Patch Available (was: In Progress) > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-14927: - Status: In Progress (was: Patch Available) > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572357#comment-15572357 ] Aihua Xu commented on HIVE-14926: - Seems I forgot to update TestSchemaTool.java. Will do that. > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1620) Patch to write directly to S3 from Hive
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572539#comment-15572539 ] Sahil Takiar commented on HIVE-1620: Hey [~richcole], [~vaggarw], [~yalovyyi], [~poeppt], As part of HIVE-14271 and HIVE-14269, we are considering implementing something very similar to what this patch did. However, we are still debating between a few different options. Any chance someone could comment on if this approach worked well in production? Were there issues with this approach that caused problems for any users? Some of the concerns we have with the Direct Write to S3 from Hive are that the failure semantics need to be improved when writing to S3. Hive needs to make sure that there aren’t any dangling files left in the final table location on S3. This isn’t really an issue for writing to HDFS because everything is written to a temp directory and only the successfully written files get renamed to their output location. The temp directory is then deleted at the end of the MR job (similar concerns were raised in HIVE-14271). According to the AWS docs, EMR 4.x took the Direct Write approach, but EMR 5.x doesn't (ref: http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-hive-differences.html#emr-hive-diff). The docs say that the Direct Write to S3 was eliminated and that EMR 5.x writes to a staging file on S3, and then copies the data to the final table location on S3. Any chance someone could comment on why the approach was changed? Were there fundamental issues with the approach that caused it to not work well in production. Any help / feedback on this would be greatly appreciated, since we probably shouldn't implement the Direct Write Approach if it doesn't work well. > Patch to write directly to S3 from Hive > --- > > Key: HIVE-1620 > URL: https://issues.apache.org/jira/browse/HIVE-1620 > Project: Hive > Issue Type: New Feature >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > Attachments: HIVE-1620.patch > > > We want to submit a patch to Hive which allows user to write files directly > to S3. > This patch allow user to specify an S3 location as the table output location > and hence eliminates the need of copying data from HDFS to S3. > Users can run Hive queries directly over the data stored in S3. > This patch helps integrate hive with S3 better and quicker. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572287#comment-15572287 ] Chaoyu Tang edited comment on HIVE-14799 at 10/13/16 5:12 PM: -- The failed tests should not be related to this patch and I am not able to reproduce it in my local env. [~sershe] Could you review the revised patch based on your last review comments? Thanks was (Author: ctang.ma): The failed tests should not be related to this patch and I am not able to reproduce it in my local env. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-14942: -- Attachment: HIVE-14942.1.patch > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-14942: -- Status: Patch Available (was: Open) > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14943) Base Implementation
[ https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14943: -- Description: Create the 1st pass functional implementation of MERGE This should run e2e and produce correct results. was:Create the 1st pass functional implementation of MERGE > Base Implementation > --- > > Key: HIVE-14943 > URL: https://issues.apache.org/jira/browse/HIVE-14943 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > Create the 1st pass functional implementation of MERGE > This should run e2e and produce correct results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572679#comment-15572679 ] Tao Li commented on HIVE-14942: --- cc: [~thejas] I was trying to add some unit tests in org.apache.hive.service.cli.session.TestQueryDisplay.testQueryDisplay(). But it looks like that test method is involving dummy queries only. To test the query cancelation I will need some relatively long running queries to be canceled. Given that this change is small and it only affects HS2 UI, I did not spend time setting up a more full-fledged test that involves real queries (like a itest). > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14925) MSCK repair table hang while running with multi threading enabled
[ https://issues.apache.org/jira/browse/HIVE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572698#comment-15572698 ] Pengcheng Xiong commented on HIVE-14925: [~rkamoor], there are several test cases failing, could u take a look at them? And, as [~rajesh.balamohan] suggested, we need to add a test case for the patch. If q tests are hard to add, maybe you can add a JUnit test with some artificial delay, e.g., thread.sleep, etc to expose the problem and prove the benefit of your patch. Thanks again for your efforts. > MSCK repair table hang while running with multi threading enabled > - > > Key: HIVE-14925 > URL: https://issues.apache.org/jira/browse/HIVE-14925 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 2.2.0 >Reporter: Ratheesh Kamoor >Assignee: Ratheesh Kamoor >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-14925.patch > > > MSCK REPAIR TABLE hanging while running with multi-threading enabled > (default). I think it is because of a major design flaw in how thread pool > implemented in HiveMetaSoreChecker class / checkPartitionDirs method. This > method has a thread pool which register Callable but callable makes a > recursive call to checkPartitionDirs method again. This code will hang when > number of directories is more than thread pool size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13316) Upgrade to Calcite 1.10
[ https://issues.apache.org/jira/browse/HIVE-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13316: --- Attachment: HIVE-13316.08.patch > Upgrade to Calcite 1.10 > --- > > Key: HIVE-13316 > URL: https://issues.apache.org/jira/browse/HIVE-13316 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13316.01.patch, HIVE-13316.02.patch, > HIVE-13316.05.patch, HIVE-13316.07.patch, HIVE-13316.08.patch, > HIVE-13316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[
https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572883#comment-15572883
]
Hive QA commented on HIVE-14942:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833162/HIVE-14942.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10530 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver-orc_llap.q-delete_where_non_partitioned.q-vector_groupby_mapjoin.q-and-27-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1533/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1533/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1533/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833162 - PreCommit-HIVE-Build
> HS2 UI: Canceled queries show up in "Open Queries"
> --
>
> Key: HIVE-14942
> URL: https://issues.apache.org/jira/browse/HIVE-14942
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-14942.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14580) Introduce || operator
[ https://issues.apache.org/jira/browse/HIVE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14580: Attachment: HIVE-14580.2.patch > Introduce || operator > - > > Key: HIVE-14580 > URL: https://issues.apache.org/jira/browse/HIVE-14580 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Zoltan Haindrich > Attachments: HIVE-14580.1.patch, HIVE-14580.2.patch > > > Functionally equivalent to concat() udf. But standard allows usage of || for > string concatenations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14580) Introduce || operator
[ https://issues.apache.org/jira/browse/HIVE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14580: Attachment: HIVE-14580.3.patch patch#3: more tests...group by / order by > Introduce || operator > - > > Key: HIVE-14580 > URL: https://issues.apache.org/jira/browse/HIVE-14580 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Zoltan Haindrich > Attachments: HIVE-14580.1.patch, HIVE-14580.2.patch, > HIVE-14580.3.patch > > > Functionally equivalent to concat() udf. But standard allows usage of || for > string concatenations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14950) Support integer data type
[ https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14950: Attachment: HIVE-14950.1.patch > Support integer data type > - > > Key: HIVE-14950 > URL: https://issues.apache.org/jira/browse/HIVE-14950 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14950.1.patch > > > maybe its just me bumping into this difference again and again... > but it's in the sql2011 standard... > adding an alias for int would be easy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14950) Support integer data type
[ https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14950: Status: Patch Available (was: Open) > Support integer data type > - > > Key: HIVE-14950 > URL: https://issues.apache.org/jira/browse/HIVE-14950 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14950.1.patch > > > maybe its just me bumping into this difference again and again... > but it's in the sql2011 standard... > adding an alias for int would be easy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573018#comment-15573018 ] Thejas M Nair commented on HIVE-14942: -- [~vgumashta] Can you please review this ? > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-14927: - Status: Open (was: Patch Available) > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-14927: - Status: Patch Available (was: Open) > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13316) Upgrade to Calcite 1.10
[
https://issues.apache.org/jira/browse/HIVE-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573059#comment-15573059
]
Hive QA commented on HIVE-13316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833169/HIVE-13316.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 10530 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver-orc_llap.q-delete_where_non_partitioned.q-vector_groupby_mapjoin.q-and-27-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_skew_1_23]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[offset_limit_ppd_optimizer]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_pushdown]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[table_access_keys_stats]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[partition_column_names_with_leading_and_trailing_spaces]
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query75]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_1_23]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_skew_1_23]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[table_access_keys_stats]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1534/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1534/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1534/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833169 - PreCommit-HIVE-Build
> Upgrade to Calcite 1.10
> ---
>
> Key: HIVE-13316
> URL: https://issues.apache.org/jira/browse/HIVE-13316
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13316.01.patch, HIVE-13316.02.patch,
> HIVE-13316.05.patch, HIVE-13316.07.patch, HIVE-13316.08.patch,
> HIVE-13316.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573060#comment-15573060 ] Mohit Sabharwal commented on HIVE-14942: LGTM, +1. Looks like I missed this case in HIVE-13099 > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Attachment: HIVE-14926.2.patch PATCH-2: update the test cases. > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Attachment: HIVE-14926.2.patch > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Attachment: (was: HIVE-14926.2.patch) > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator
[
https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573083#comment-15573083
]
Zhiyuan Yang commented on HIVE-14951:
-
Here's the error message:
{code:java}
Error: Error while running task ( failure ) :
attempt_1475017598908_0282_2_02_00_3:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=0) {"key":{"_col0":2},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0) {"key":{"_col0":2},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:284)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=0) {"key":{"_col0":2},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=1) {"key":{"_col0":1},"value":null}
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:416)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:379)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:485)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1016)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:821)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
... 17 more
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=1) {"key":{"_col0":1},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:284)
at
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:404)
... 26 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=1) {"key":{"_col0":1},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
... 27 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:708)
at
o
[jira] [Updated] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez)
[ https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14951: Summary: ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez) (was: ArrayIndexOutOfBoundsException in GroupByOperator) > ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez) > --- > > Key: HIVE-14951 > URL: https://issues.apache.org/jira/browse/HIVE-14951 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang > > Query: > select * from (select distinct a from f16) as f16, (select distinct a from > f1) as fprime where f16.a = fprime.a; > Table: > create table f1 (a int, b string); > create table f16 (a int, b string); > Config: > set hive.auto.convert.sortmerge.join=true; > set hive.auto.convert.join=false; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14951: Summary: ArrayIndexOutOfBoundsException in GroupByOperator (was: ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez)) > ArrayIndexOutOfBoundsException in GroupByOperator > - > > Key: HIVE-14951 > URL: https://issues.apache.org/jira/browse/HIVE-14951 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang > > Engine: > Tez > Query: > select * from (select distinct a from f16) as f16, (select distinct a from > f1) as fprime where f16.a = fprime.a; > Table: > create table f1 (a int, b string); > create table f16 (a int, b string); > Config: > set hive.auto.convert.sortmerge.join=true; > set hive.auto.convert.join=false; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14951) ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez)
[ https://issues.apache.org/jira/browse/HIVE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14951: Description: Engine: Tez Query: select * from (select distinct a from f16) as f16, (select distinct a from f1) as fprime where f16.a = fprime.a; Table: create table f1 (a int, b string); create table f16 (a int, b string); Config: set hive.auto.convert.sortmerge.join=true; set hive.auto.convert.join=false; was: Query: select * from (select distinct a from f16) as f16, (select distinct a from f1) as fprime where f16.a = fprime.a; Table: create table f1 (a int, b string); create table f16 (a int, b string); Config: set hive.auto.convert.sortmerge.join=true; set hive.auto.convert.join=false; > ArrayIndexOutOfBoundsException in GroupByOperator (Hive on Tez) > --- > > Key: HIVE-14951 > URL: https://issues.apache.org/jira/browse/HIVE-14951 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang > > Engine: > Tez > Query: > select * from (select distinct a from f16) as f16, (select distinct a from > f1) as fprime where f16.a = fprime.a; > Table: > create table f1 (a int, b string); > create table f16 (a int, b string); > Config: > set hive.auto.convert.sortmerge.join=true; > set hive.auto.convert.join=false; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Attachment: HIVE-14926.2.patch > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Attachment: (was: HIVE-14926.2.patch) > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573164#comment-15573164 ] Tao Li commented on HIVE-14942: --- [~mohitsabharwal] Thanks for the review. Can you please commit it? > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13554) [Umbrella] SQL:2011 compliance
[ https://issues.apache.org/jira/browse/HIVE-13554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573178#comment-15573178 ] Damien Carol commented on HIVE-13554: - [~ashutoshc] do you have any specification document? What is your reference? > [Umbrella] SQL:2011 compliance > -- > > Key: HIVE-13554 > URL: https://issues.apache.org/jira/browse/HIVE-13554 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > There are various gaps in language which needs to be addressed to bring Hive > under SQL:2011 compliance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14921) Move slow CliDriver tests to MiniLlap - part 2
[ https://issues.apache.org/jira/browse/HIVE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14921: - Attachment: HIVE-14921.3.patch > Move slow CliDriver tests to MiniLlap - part 2 > -- > > Key: HIVE-14921 > URL: https://issues.apache.org/jira/browse/HIVE-14921 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14921.1.patch, HIVE-14921.1.patch, > HIVE-14921.2.patch, HIVE-14921.2.patch, HIVE-14921.3.patch > > > Continuation to HIVE-14877 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Attachment: HIVE-14940.2.patch > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14950) Support integer data type
[
https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573209#comment-15573209
]
Alan Gates commented on HIVE-14950:
---
If you're going to create KW_INTEGER as a new keyword there are several places
that KW_INT is referenced in IdentifiersParser.g that should also be updated.
I'm wondering if instead it makes more sense to change the definition of KW_INT
in HiveLexer.g to be:
{code}
KW_INT: 'INT' | 'INTEGER'
{code}
then you're guaranteed that they're interchangeable now and in the future.
> Support integer data type
> -
>
> Key: HIVE-14950
> URL: https://issues.apache.org/jira/browse/HIVE-14950
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14950.1.patch
>
>
> maybe its just me bumping into this difference again and again...
> but it's in the sql2011 standard...
> adding an alias for int would be easy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14929) Adding JDBC test for query cancellation scenario
[ https://issues.apache.org/jira/browse/HIVE-14929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14929: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. Thanks Deepak. > Adding JDBC test for query cancellation scenario > > > Key: HIVE-14929 > URL: https://issues.apache.org/jira/browse/HIVE-14929 > Project: Hive > Issue Type: Test >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 2.2.0 > > Attachments: HIVE-14929.1.patch, HIVE-14929.2.patch > > > There is some functional testing for query cancellation using JDBC which is > missing in unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14580) Introduce || operator
[
https://issues.apache.org/jira/browse/HIVE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573258#comment-15573258
]
Hive QA commented on HIVE-14580:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833184/HIVE-14580.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10531 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver-orc_llap.q-delete_where_non_partitioned.q-vector_groupby_mapjoin.q-and-27-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1535/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1535/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833184 - PreCommit-HIVE-Build
> Introduce || operator
> -
>
> Key: HIVE-14580
> URL: https://issues.apache.org/jira/browse/HIVE-14580
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Zoltan Haindrich
> Attachments: HIVE-14580.1.patch, HIVE-14580.2.patch,
> HIVE-14580.3.patch
>
>
> Functionally equivalent to concat() udf. But standard allows usage of || for
> string concatenations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14924) MSCK REPAIR table with single threaded is throwing null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-14924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573268#comment-15573268 ] Ashutosh Chauhan commented on HIVE-14924: - +1 > MSCK REPAIR table with single threaded is throwing null pointer exception > - > > Key: HIVE-14924 > URL: https://issues.apache.org/jira/browse/HIVE-14924 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 2.2.0 >Reporter: Ratheesh Kamoor >Assignee: Pengcheng Xiong > Attachments: HIVE-14924.01.patch > > > MSCK REPAIR TABLE is throwing Null Pointer Exception while running on single > threaded mode (hive.mv.files.thread=0) > Error: > 2016-10-10T22:27:13,564 ERROR [e9ce04a8-2a84-426d-8e79-a2d15b8cee09 > main([])]: exec.DDLTask (DDLTask.java:failed(581)) - > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkPartitionDirs(HiveMetaStoreChecker.java:423) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:315) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:291) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:236) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:113) > at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1834) > In order to reproduce: > set hive.mv.files.thread=0 and run MSCK REPAIR TABLE command -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14921) Move slow CliDriver tests to MiniLlap - part 2
[ https://issues.apache.org/jira/browse/HIVE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573315#comment-15573315 ] Siddharth Seth commented on HIVE-14921: --- +1, looks good to me. Lets wait another run to see if the test failures are related. I think testCliDriver[reloadJar] failure was introduced recently. > Move slow CliDriver tests to MiniLlap - part 2 > -- > > Key: HIVE-14921 > URL: https://issues.apache.org/jira/browse/HIVE-14921 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14921.1.patch, HIVE-14921.1.patch, > HIVE-14921.2.patch, HIVE-14921.2.patch, HIVE-14921.3.patch > > > Continuation to HIVE-14877 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1
[ https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573327#comment-15573327 ] Sergio Peña commented on HIVE-14476: [~taoli-hwx] I tried compiling the branch-1 with JDK7, and it does not compile. It might be broken by another commit, and that's why ptest fails too. Does it compile for you? > Fix logging issue for branch-1 > -- > > Key: HIVE-14476 > URL: https://issues.apache.org/jira/browse/HIVE-14476 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14476.1-branch-1.patch > > > This issue is from branch-1 code when we decide if a log entry is an > operational log or not (the operational logs are visible to the client). The > problem is that the code is checking the logging mode at the beginning of the > decide() method, while the logging mode is updated after that check. Due to > this issue, we ran into an issue that an operational log could be filtered > out if it's the very first log being checked from the this method. As a > result, that particular log is not showing up for the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1
[ https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573377#comment-15573377 ] Tao Li commented on HIVE-14476: --- I ran into the build errors below. Are you seeing the same errors? [INFO] BUILD FAILURE [INFO] [INFO] Total time: 9.873 s [INFO] Finished at: 2016-10-13T15:15:01-07:00 [INFO] Final Memory: 58M/626M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure: [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/common/ServerUtils.java:[23,28] cannot find symbol [ERROR] symbol: class FileSystem [ERROR] location: package org.apache.hadoop.fs [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/common/ServerUtils.java:[24,28] cannot find symbol [ERROR] symbol: class Path [ERROR] location: package org.apache.hadoop.fs [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[45,30] package org.apache.hadoop.conf does not exist [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[54,32] package org.apache.hadoop.mapred does not exist [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[55,34] package org.apache.hadoop.security does not exist [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[56,30] package org.apache.hadoop.util does not exist [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[64,31] cannot find symbol [ERROR] symbol: class Configuration [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[2453,31] cannot find symbol [ERROR] symbol: class Configuration [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[2458,32] cannot find symbol [ERROR] symbol: class Configuration [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[2471,33] cannot find symbol [ERROR] symbol: class Configuration [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[2475,33] cannot find symbol [ERROR] symbol: class Configuration [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf [ERROR] /Users/tli/dev2/hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[2547,33] cannot find symbol [ERROR] symbol: class Configuration [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf > Fix logging issue for branch-1 > -- > > Key: HIVE-14476 > URL: https://issues.apache.org/jira/browse/HIVE-14476 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14476.1-branch-1.patch > > > This issue is from branch-1 code when we decide if a log entry is an > operational log or not (the operational logs are visible to the client). The > problem is that the code is checking the logging mode at the beginning of the > decide() method, while the logging mode is updated after that check. Due to > this issue, we ran into an issue that an operational log could be filtered > out if it's the very first log being checked from the this method. As a > result, that particular log is not showing up for the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1
[ https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573383#comment-15573383 ] Tao Li commented on HIVE-14476: --- The build was against branch-1.2. I saw the same errors even without my change. > Fix logging issue for branch-1 > -- > > Key: HIVE-14476 > URL: https://issues.apache.org/jira/browse/HIVE-14476 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14476.1-branch-1.patch > > > This issue is from branch-1 code when we decide if a log entry is an > operational log or not (the operational logs are visible to the client). The > problem is that the code is checking the logging mode at the beginning of the > decide() method, while the logging mode is updated after that check. Due to > this issue, we ran into an issue that an operational log could be filtered > out if it's the very first log being checked from the this method. As a > result, that particular log is not showing up for the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1
[
https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573387#comment-15573387
]
Sergio Peña commented on HIVE-14476:
Nop, I got the same ones from ptest.
Use this {{mvn clean install -DskipTests -Phadoop-2}}
> Fix logging issue for branch-1
> --
>
> Key: HIVE-14476
> URL: https://issues.apache.org/jira/browse/HIVE-14476
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-14476.1-branch-1.patch
>
>
> This issue is from branch-1 code when we decide if a log entry is an
> operational log or not (the operational logs are visible to the client). The
> problem is that the code is checking the logging mode at the beginning of the
> decide() method, while the logging mode is updated after that check. Due to
> this issue, we ran into an issue that an operational log could be filtered
> out if it's the very first log being checked from the this method. As a
> result, that particular log is not showing up for the end user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[
https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573397#comment-15573397
]
Hive QA commented on HIVE-14940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833205/HIVE-14940.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1540/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1540/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1540/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-13 22:26:34.739
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1540/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 22:26:34.743
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/primitive_types.q
Removing ql/src/test/results/clientpositive/primitive_types.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 22:26:35.778
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error:
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
No such file or directory
error:
a/ql/src/test/queries/clientpositive/partition_column_names_with_leading_and_trailing_spaces.q:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/partition_column_names_with_leading_and_trailing_spaces.q.out:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/unionDistinct_2.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/vector_join_part_col_char.q.out: No
such file or directory
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833205 - PreCommit-HIVE-Build
> MiniTezCliDriver - switch back to SQL metastore as default
> --
>
> Key: HIVE-14940
> URL: https://issues.apache.org/jira/browse/HIVE-14940
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch,
> HIVE-14940.2.patch
>
>
> HBase setup for metastore in MiniTez is taking around 3 mins for setup. The
> actual runtime of the queries is typically much lower. To avoid the high
> overhead we should be switch back to SQL metastore as default and if required
> we can have dedicated set of tests that run against hbase metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Attachment: HIVE-14940.2.patch > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch, HIVE-14940.2.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Status: Patch Available (was: Open) > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch, HIVE-14940.2.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Status: Open (was: Patch Available) > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch, HIVE-14940.2.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1
[ https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573427#comment-15573427 ] Tao Li commented on HIVE-14476: --- Sorry, I forgot "-Phadoop-2". Now it builds fine for me. > Fix logging issue for branch-1 > -- > > Key: HIVE-14476 > URL: https://issues.apache.org/jira/browse/HIVE-14476 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14476.1-branch-1.patch > > > This issue is from branch-1 code when we decide if a log entry is an > operational log or not (the operational logs are visible to the client). The > problem is that the code is checking the logging mode at the beginning of the > decide() method, while the logging mode is updated after that check. Due to > this issue, we ran into an issue that an operational log could be filtered > out if it's the very first log being checked from the this method. As a > result, that particular log is not showing up for the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Attachment: HIVE-14940.3.patch > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch, HIVE-14940.2.patch, HIVE-14940.3.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[
https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573460#comment-15573460
]
Hive QA commented on HIVE-14940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833227/HIVE-14940.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1541/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1541/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1541/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-13 22:51:37.564
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1541/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 22:51:37.569
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 22:51:38.500
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error:
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
No such file or directory
error:
a/ql/src/test/queries/clientpositive/partition_column_names_with_leading_and_trailing_spaces.q:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/partition_column_names_with_leading_and_trailing_spaces.q.out:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/unionDistinct_2.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/vector_join_part_col_char.q.out: No
such file or directory
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833227 - PreCommit-HIVE-Build
> MiniTezCliDriver - switch back to SQL metastore as default
> --
>
> Key: HIVE-14940
> URL: https://issues.apache.org/jira/browse/HIVE-14940
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch,
> HIVE-14940.2.patch, HIVE-14940.2.patch, HIVE-14940.3.patch
>
>
> HBase setup for metastore in MiniTez is taking around 3 mins for setup. The
> actual runtime of the queries is typically much lower. To avoid the high
> overhead we should be switch back to SQL metastore as default and if required
> we can have dedicated set of tests that run against hbase metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[
https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573477#comment-15573477
]
Hive QA commented on HIVE-14940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833227/HIVE-14940.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1542/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1542/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1542/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-13 23:01:59.740
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1542/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 23:01:59.743
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 23:02:00.627
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error:
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
No such file or directory
error:
a/ql/src/test/queries/clientpositive/partition_column_names_with_leading_and_trailing_spaces.q:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/partition_column_names_with_leading_and_trailing_spaces.q.out:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/unionDistinct_2.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/vector_join_part_col_char.q.out: No
such file or directory
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833227 - PreCommit-HIVE-Build
> MiniTezCliDriver - switch back to SQL metastore as default
> --
>
> Key: HIVE-14940
> URL: https://issues.apache.org/jira/browse/HIVE-14940
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch,
> HIVE-14940.2.patch, HIVE-14940.2.patch, HIVE-14940.3.patch
>
>
> HBase setup for metastore in MiniTez is taking around 3 mins for setup. The
> actual runtime of the queries is typically much lower. To avoid the high
> overhead we should be switch back to SQL metastore as default and if required
> we can have dedicated set of tests that run against hbase metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14864) Distcp is not called from MoveTask when src is a directory
[
https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-14864:
---
Assignee: Sahil Takiar (was: Vihang Karajgaonkar)
> Distcp is not called from MoveTask when src is a directory
> --
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Sahil Takiar
>
> In FileUtils.java the following code does not get executed even when src
> directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We
> should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() >
> conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
> LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes.
> (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) +
> ")");
> LOG.info("Launch distributed copy (distcp) job.");
> HiveConfUtil.updateJobCredentialProviders(conf);
> copied = shims.runDistCp(src, dst, conf);
> if (copied && deleteSource) {
> srcFS.delete(src, true);
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Attachment: HIVE-14940.3.patch > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch, HIVE-14940.2.patch, HIVE-14940.3.patch, HIVE-14940.3.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[
https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573501#comment-15573501
]
Hive QA commented on HIVE-14940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833233/HIVE-14940.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1543/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1543/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1543/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-13 23:13:53.170
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1543/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 23:13:53.176
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5353161 HIVE-14835: Improve ptest2 build time (Prasanth
Jayachandran reviewed by Sergio Pena)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-13 23:13:54.051
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error:
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
No such file or directory
error:
a/ql/src/test/queries/clientpositive/partition_column_names_with_leading_and_trailing_spaces.q:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/partition_column_names_with_leading_and_trailing_spaces.q.out:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/unionDistinct_2.q.out: No such
file or directory
error:
a/ql/src/test/results/clientpositive/tez/vector_join_part_col_char.q.out: No
such file or directory
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833233 - PreCommit-HIVE-Build
> MiniTezCliDriver - switch back to SQL metastore as default
> --
>
> Key: HIVE-14940
> URL: https://issues.apache.org/jira/browse/HIVE-14940
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch,
> HIVE-14940.2.patch, HIVE-14940.2.patch, HIVE-14940.3.patch, HIVE-14940.3.patch
>
>
> HBase setup for metastore in MiniTez is taking around 3 mins for setup. The
> actual runtime of the queries is typically much lower. To avoid the high
> overhead we should be switch back to SQL metastore as default and if required
> we can have dedicated set of tests that run against hbase metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14864) Distcp is not called from MoveTask when src is a directory
[
https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14864:
Status: Patch Available (was: Open)
> Distcp is not called from MoveTask when src is a directory
> --
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Sahil Takiar
> Attachments: HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src
> directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We
> should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() >
> conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
> LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes.
> (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) +
> ")");
> LOG.info("Launch distributed copy (distcp) job.");
> HiveConfUtil.updateJobCredentialProviders(conf);
> copied = shims.runDistCp(src, dst, conf);
> if (copied && deleteSource) {
> srcFS.delete(src, true);
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14864) Distcp is not called from MoveTask when src is a directory
[
https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14864:
Attachment: HIVE-14864.patch
Attaching a pre-lim patch just for reference. The patch is still a WIP.
Changes are pretty simple, {{FileUtils.copy}} uses {{getContentSummary}} to get
the number of files under the folder. It triggers a Distcp job based on the
size of files under the folder + the number of files under the folder.
If only a single file needs to be copied, the {{ContentSummary}} length will be
the size of that file, and the number of files under it will be 1.
For now the logic is pretty simple, if the number of files exceeds a threshold
set by {{hive.exec.copyfile.maxnumfiles}} (which defaults to 1) and the size of
the files exceeds a threshold set by hive.exec.copyfile.maxsize (which defaults
to 32 MB), the Distcp job will be triggered.
So basically any folder that contains more than 1 file and whose total contents
is greater than 32 MB.
> Distcp is not called from MoveTask when src is a directory
> --
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Sahil Takiar
> Attachments: HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src
> directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We
> should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() >
> conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
> LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes.
> (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) +
> ")");
> LOG.info("Launch distributed copy (distcp) job.");
> HiveConfUtil.updateJobCredentialProviders(conf);
> copied = shims.runDistCp(src, dst, conf);
> if (copied && deleteSource) {
> srcFS.delete(src, true);
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14940) MiniTezCliDriver - switch back to SQL metastore as default
[ https://issues.apache.org/jira/browse/HIVE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14940: - Attachment: HIVE-14940.4.patch > MiniTezCliDriver - switch back to SQL metastore as default > -- > > Key: HIVE-14940 > URL: https://issues.apache.org/jira/browse/HIVE-14940 > Project: Hive > Issue Type: Sub-task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14940.1.patch, HIVE-14940.1.patch, > HIVE-14940.2.patch, HIVE-14940.2.patch, HIVE-14940.3.patch, > HIVE-14940.3.patch, HIVE-14940.4.patch > > > HBase setup for metastore in MiniTez is taking around 3 mins for setup. The > actual runtime of the queries is typically much lower. To avoid the high > overhead we should be switch back to SQL metastore as default and if required > we can have dedicated set of tests that run against hbase metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14643) handle ctas
[ https://issues.apache.org/jira/browse/HIVE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573516#comment-15573516 ] Sergey Shelukhin commented on HIVE-14643: - The patch. This also reworks the union logic which I noticed used the weird path structure (table/1/mm_..., creating multiple directories per write) instead of proper (table/mm_.../1) > handle ctas > --- > > Key: HIVE-14643 > URL: https://issues.apache.org/jira/browse/HIVE-14643 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14643.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14643) handle ctas
[ https://issues.apache.org/jira/browse/HIVE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14643: Attachment: HIVE-14643.patch > handle ctas > --- > > Key: HIVE-14643 > URL: https://issues.apache.org/jira/browse/HIVE-14643 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14643.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1
[ https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573529#comment-15573529 ] Tao Li commented on HIVE-14476: --- [~spena] What issues are you seeing locally? > Fix logging issue for branch-1 > -- > > Key: HIVE-14476 > URL: https://issues.apache.org/jira/browse/HIVE-14476 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14476.1-branch-1.patch > > > This issue is from branch-1 code when we decide if a log entry is an > operational log or not (the operational logs are visible to the client). The > problem is that the code is checking the logging mode at the beginning of the > decide() method, while the logging mode is updated after that check. Due to > this issue, we ran into an issue that an operational log could be filtered > out if it's the very first log being checked from the this method. As a > result, that particular log is not showing up for the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14799: --- Attachment: HIVE-14799.7.patch revised the patch to address an issue raised by [~ychena]. [~sershe] [~ychena], could you review it. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, > HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Status: Patch Available (was: In Progress) > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Status: In Progress (was: Patch Available) > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14640) handle hive.merge.*files in select queries
[ https://issues.apache.org/jira/browse/HIVE-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14640: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to branch > handle hive.merge.*files in select queries > -- > > Key: HIVE-14640 > URL: https://issues.apache.org/jira/browse/HIVE-14640 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hive-14535 > > Attachments: HIVE-14640.WIP.patch, HIVE-14640.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14643) handle ctas
[ https://issues.apache.org/jira/browse/HIVE-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14643: Fix Version/s: hive-14535 Status: Patch Available (was: Open) > handle ctas > --- > > Key: HIVE-14643 > URL: https://issues.apache.org/jira/browse/HIVE-14643 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hive-14535 > > Attachments: HIVE-14643.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14932) handle bucketing for MM tables
[ https://issues.apache.org/jira/browse/HIVE-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-14932: --- Assignee: Sergey Shelukhin > handle bucketing for MM tables > -- > > Key: HIVE-14932 > URL: https://issues.apache.org/jira/browse/HIVE-14932 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14950) Support integer data type
[
https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573651#comment-15573651
]
Hive QA commented on HIVE-14950:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12833195/HIVE-14950.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10534 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reloadJar]
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1544/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1544/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1544/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12833195 - PreCommit-HIVE-Build
> Support integer data type
> -
>
> Key: HIVE-14950
> URL: https://issues.apache.org/jira/browse/HIVE-14950
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14950.1.patch
>
>
> maybe its just me bumping into this difference again and again...
> but it's in the sql2011 standard...
> adding an alias for int would be easy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
