[jira] [Commented] (HIVE-16388) LLAP: Log rotation for daemon, history and gc files
[
https://issues.apache.org/jira/browse/HIVE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961697#comment-15961697
]
Hive QA commented on HIVE-16388:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862561/HIVE-16388.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10561 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=234)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4615/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4615/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4615/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862561 - PreCommit-HIVE-Build
> LLAP: Log rotation for daemon, history and gc files
> ---
>
> Key: HIVE-16388
> URL: https://issues.apache.org/jira/browse/HIVE-16388
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Siddharth Seth
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16388.1.patch, HIVE-16388.2.patch
>
>
> GC logs need to be rotated by date.
> LLAP daemon history logs as well
> Ideally, the daemon.out file needs the same
> Need to be able to download relevant logfiles for a time window.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16368) Unexpected java.lang.ArrayIndexOutOfBoundsException from query with LaterView Operation for hive on MR.
[
https://issues.apache.org/jira/browse/HIVE-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961695#comment-15961695
]
Ashutosh Chauhan commented on HIVE-16368:
-
I see that patch indeed is an improvement. Column Pruner should not change the
order of columns. If you can find a testcase that will be great, else this
still is worth checking in.
lookupColumn() should not return null in this case. Can you change null check
to assert col != null instead.?
> Unexpected java.lang.ArrayIndexOutOfBoundsException from query with LaterView
> Operation for hive on MR.
> ---
>
> Key: HIVE-16368
> URL: https://issues.apache.org/jira/browse/HIVE-16368
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-16368.000.patch, HIVE-16368.001.patch
>
>
> Unexpected java.lang.ArrayIndexOutOfBoundsException from query. It happened
> in LaterView Operation. It happened for hive-on-mr. The reason is because the
> column prune change the column order in LaterView operation, for back-back
> reducesink operators using MR engine, FileSinkOperator and TableScanOperator
> are added before the second ReduceSink operator, The serialization column
> order used by FileSinkOperator in LazyBinarySerDe of previous reducer is
> different from deserialization column order from table desc used by
> MapOperator/TableScanOperator in LazyBinarySerDe of current failed mapper.
> The serialization is decided by the outputObjInspector from
> LateralViewJoinOperator,
> {code}
> ArrayList fieldNames = conf.getOutputInternalColNames();
> outputObjInspector = ObjectInspectorFactory
> .getStandardStructObjectInspector(fieldNames, ois);
> {code}
> So the column order for serialization is decided by getOutputInternalColNames
> in LateralViewJoinOperator.
> The deserialization is decided by TableScanOperator which is created at
> GenMapRedUtils.splitTasks.
> {code}
> TableDesc tt_desc = PlanUtils.getIntermediateFileTableDesc(PlanUtils
> .getFieldSchemasFromRowSchema(parent.getSchema(), "temporarycol"));
> // Create the temporary file, its corresponding FileSinkOperaotr, and
> // its corresponding TableScanOperator.
> TableScanOperator tableScanOp =
> createTemporaryFile(parent, op, taskTmpDir, tt_desc, parseCtx);
> {code}
> The column order for deserialization is decided by rowSchema of
> LateralViewJoinOperator.
> But ColumnPrunerLateralViewJoinProc changed the order of
> outputInternalColNames but still keep the original order of rowSchema,
> Which cause the mismatch between serialization and deserialization for two
> back-to-back MR jobs.
> Similar issue for ColumnPrunerLateralViewForwardProc which change the column
> order of its child selector colList but not rowSchema.
> The exception is
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 875968094
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToLong(LazyBinaryUtils.java:78)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryDouble.init(LazyBinaryDouble.java:43)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:554)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:381)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961689#comment-15961689 ] Prasanth Jayachandran commented on HIVE-16285: -- Yes. PerfLogger it will work regardless of root logger level since we have explicitly specified level for PerfLogger it won't be inherited from root. Will address Gopal's comments in the next patch. > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch, > HIVE-16285.3.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16388) LLAP: Log rotation for daemon, history and gc files
[ https://issues.apache.org/jira/browse/HIVE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16388: - Attachment: HIVE-16388.2.patch Adding LlapDaemon to INFO level in log4j properties. So even if we configure the root logger with WARN, configuration logs will end up being written to log file. > LLAP: Log rotation for daemon, history and gc files > --- > > Key: HIVE-16388 > URL: https://issues.apache.org/jira/browse/HIVE-16388 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Attachments: HIVE-16388.1.patch, HIVE-16388.2.patch > > > GC logs need to be rotated by date. > LLAP daemon history logs as well > Ideally, the daemon.out file needs the same > Need to be able to download relevant logfiles for a time window. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16383) Switch to HikariCP as default connection pooling
[ https://issues.apache.org/jira/browse/HIVE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16383: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Ran TestLlapSignerImpl test locally and it passed for me. It also passed in previous test run. All other tests are failing already in master. Committed to master. > Switch to HikariCP as default connection pooling > > > Key: HIVE-16383 > URL: https://issues.apache.org/jira/browse/HIVE-16383 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-16383.1.patch, HIVE-16383.2.patch > > > Since 3.0 is planned to move to JDK8, we can now switch to HikariCP as > default connection pooling for DN because of its improved performance over > others. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961671#comment-15961671 ] Ashutosh Chauhan commented on HIVE-16407: - yes.. i ran it after adding the ; in the query. > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > Attachments: issue_reproduce.sql > > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14564:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Zhihai !
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:550)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:377)
> ... 13 more
> {code}
> The exception is because the serialization and deserialization doesn't match.
> The serialization by LazyBinarySerDe from previous MapReduce job used
> different order of columns. When the current MapReduce job deserialized the
> intermediate sequence file generated by previous MapReduce job, it will get
> corrupted data from the deserialization using wrong order of columns by
> LazyBinaryStruct. The unmatched columns between serialization and
> deserialization is caused by SelectOperator's Column Pruning
> {{ColumnPrunerSelectProc}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961655#comment-15961655 ] Sudarshan edited comment on HIVE-16407 at 4/8/17 2:58 AM: -- Thanks for taking a look [~ashutoshc] . Attached script was missing a ";" in the last line. ( just deleting the attachment and adding it again). Can you confirm if you had added ";" ? was (Author: sudarshans): [~ashutoshc] Attached script was missing a ";" in the last line. ( just deleting the attachment and adding it again). Can you confirm if you had added ";" ? > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > Attachments: issue_reproduce.sql > > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan updated HIVE-16407: - Attachment: issue_reproduce.sql > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > Attachments: issue_reproduce.sql > > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961655#comment-15961655 ] Sudarshan commented on HIVE-16407: -- [~ashutoshc] Attached script was missing a ";" in the last line. ( just deleting the attachment and adding it again). Can you confirm if you had added ";" ? > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan updated HIVE-16407: - Attachment: (was: issue_reproduce.sql) > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16392) Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance logic
[
https://issues.apache.org/jira/browse/HIVE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-16392:
Resolution: Fixed
Hadoop Flags: Incompatible change
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Sahil!
> Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance
> logic
>
>
> Key: HIVE-16392
> URL: https://issues.apache.org/jira/browse/HIVE-16392
> Project: Hive
> Issue Type: Task
> Components: Security
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Labels: backwards-incompatible
> Fix For: 3.0.0
>
> Attachments: HIVE-16392.1.patch, HIVE-16392.2.patch
>
>
> As discussed in HIVE-16346 we should remove the config
> {{hive.warehouse.subdir.inherit.perms}} and all the permissions inheritance
> logic.
> This feature is no longer needed in Hive as the traditional permission model
> has largely been replaced by external security systems such as Ranger and
> Sentry.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961646#comment-15961646 ] Ashutosh Chauhan commented on HIVE-16407: - This issue is not reproducible on current master. Attached query finished in few seconds. > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > Attachments: issue_reproduce.sql > > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961615#comment-15961615
]
zhihai xu commented on HIVE-14564:
--
All the test failures are not related to my change.
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:550)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:377)
> ... 13 more
> {code}
> The exception is because the serialization and deserialization doesn't match.
> The serialization by LazyBinarySerDe from previous MapReduce job used
> different order of columns. When the current MapReduce job deserialized the
> intermediate sequence file generated by previous MapReduce job, it will get
> corrupted data from the deserialization using wrong order of columns by
> LazyBinaryStruct. The unmatched columns between serialization and
> deserialization is caused by SelectOperator's Column Pruning
> {{ColumnPrunerSelectProc}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16368) Unexpected java.lang.ArrayIndexOutOfBoundsException from query with LaterView Operation for hive on MR.
[
https://issues.apache.org/jira/browse/HIVE-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961604#comment-15961604
]
zhihai xu commented on HIVE-16368:
--
Thanks for the review, My mistake, yes, this test case can't reproduce the
issue. I thought the plan is similar, but after debug it further, the reason is
there are two Select Operators are added after the Lateral View Join Operator
before ReduceSinkOperator So ColumnPrunerSelectProc is called before
ColumnPrunerLateralViewJoinProc is called, then ColumnPrunerSelectProc change
the column order to the correct order at the following code:
{code}
cppCtx.getPrunedColLists().put(op,
cppCtx.getSelectColsFromChildren(op, cols));
public List getSelectColsFromChildren(SelectOperator op,
List colList) {
List cols = new ArrayList<>();
SelectDesc conf = op.getConf();
if (colList != null && conf.isSelStarNoCompute()) {
cols.addAll(colList);
return cols;
}
List selectExprs = conf.getColList();
// The colList is the output columns used by child operators, they are
// different
// from input columns of the current operator. we need to find out which
// input columns are used.
List outputColumnNames = conf.getOutputColumnNames();
for (int i = 0; i < outputColumnNames.size(); i++) {
if (colList == null) {
cols = mergeFieldNodesWithDesc(cols, selectExprs.get(i));
} else {
FieldNode childFn = lookupColumn(colList, outputColumnNames.get(i));
if (childFn != null) {
// In SemanticAnalyzer we inject SEL op before aggregation. The
columns
// in this SEL are derived from the table schema, and do not reflect
the
// actual columns being selected in the current query.
// In this case, we skip the merge and just use the path from the
child ops.
ExprNodeDesc desc = selectExprs.get(i);
if (desc instanceof ExprNodeColumnDesc && ((ExprNodeColumnDesc)
desc).getIsGenerated()) {
FieldNode fn = new FieldNode(((ExprNodeColumnDesc)
desc).getColumn());
fn.setNodes(childFn.getNodes());
cols = mergeFieldNodes(cols, fn);
} else {
cols = mergeFieldNodesWithDesc(cols, selectExprs.get(i));
}
}
}
}
return cols;
}
{code}
Then after column pruning finished, the two redundant Select Operators between
the Lateral View Join Operator and ReduceSinkOperator are removed.
We have a very complicated query which can trigger this issue in our internal
hive branch: there are no Select Operators between the Lateral View Join
Operator and ReduceSinkOperator before ColumnPrunerLateralViewJoinProc is
called. So this issue may potentially exist but currently I can't find a test
case to trigger this error. I will keep looking for a good test case to
reproduce this issue.
> Unexpected java.lang.ArrayIndexOutOfBoundsException from query with LaterView
> Operation for hive on MR.
> ---
>
> Key: HIVE-16368
> URL: https://issues.apache.org/jira/browse/HIVE-16368
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-16368.000.patch, HIVE-16368.001.patch
>
>
> Unexpected java.lang.ArrayIndexOutOfBoundsException from query. It happened
> in LaterView Operation. It happened for hive-on-mr. The reason is because the
> column prune change the column order in LaterView operation, for back-back
> reducesink operators using MR engine, FileSinkOperator and TableScanOperator
> are added before the second ReduceSink operator, The serialization column
> order used by FileSinkOperator in LazyBinarySerDe of previous reducer is
> different from deserialization column order from table desc used by
> MapOperator/TableScanOperator in LazyBinarySerDe of current failed mapper.
> The serialization is decided by the outputObjInspector from
> LateralViewJoinOperator,
> {code}
> ArrayList fieldNames = conf.getOutputInternalColNames();
> outputObjInspector = ObjectInspectorFactory
> .getStandardStructObjectInspector(fieldNames, ois);
> {code}
> So the column order for serialization is decided by getOutputInternalColNames
> in LateralViewJoinOperator.
> The deserialization is decided by TableScanOperator which is created at
> GenMapRedUtils.splitTasks.
> {code}
> TableDesc tt_desc = PlanUtils.getIntermediateFileTableDesc(PlanUtils
> .getFieldSchemasFromRowSchema(parent.getSchema(), "temporarycol"));
> // Create the temporary file, its corresponding FileSinkOperaotr, and
> // its corresponding TableScanOperator.
> TableScanOperator tableScanOp =
>
[jira] [Commented] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961558#comment-15961558
]
Hive QA commented on HIVE-14564:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862536/HIVE-14564.004.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10589 tests
executed
*Failed tests:*
{noformat}
TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=220)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_skewtable]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4614/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4614/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4614/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862536 - PreCommit-HIVE-Build
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
>
[jira] [Updated] (HIVE-16407) Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time.
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan updated HIVE-16407: - Summary: Compilation of a query with lots of "OR" conditions (like 2200) that check for "like" condition take a lot of time. (was: Compilation of a query with lots of conditions (like 2200) in a where clause can take a lot of time) > Compilation of a query with lots of "OR" conditions (like 2200) that check > for "like" condition take a lot of time. > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > Attachments: issue_reproduce.sql > > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16385) StatsNoJobTask could exit early before all partitions have been processed
[
https://issues.apache.org/jira/browse/HIVE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-16385:
Resolution: Fixed
Fix Version/s: 2.3.0
2.2.0
2.1.2
2.0.2
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~xuefuz] for the review.
> StatsNoJobTask could exit early before all partitions have been processed
> -
>
> Key: HIVE-16385
> URL: https://issues.apache.org/jira/browse/HIVE-16385
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.0.2, 2.1.2, 2.2.0, 2.3.0
>
> Attachments: HIVE-16385.1.patch
>
>
> For a partitioned table, the class {{StatsNoJobTask}} is supposed to launch
> threads for all partitions and compute their stats. However, it could exit
> early after at most 100 seconds:
> {code}
> private void shutdownAndAwaitTermination(ExecutorService threadPool) {
> // Disable new tasks from being submitted
> threadPool.shutdown();
> try {
> // Wait a while for existing tasks to terminate
> if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
> // Cancel currently executing tasks
> threadPool.shutdownNow();
> // Wait a while for tasks to respond to being cancelled
> if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
> LOG.debug("Stats collection thread pool did not terminate");
> }
> }
> } catch (InterruptedException ie) {
> // Cancel again if current thread also interrupted
> threadPool.shutdownNow();
> // Preserve interrupt status
> Thread.currentThread().interrupt();
> }
> }
> {code}
> The {{shutdown}} call does not wait for all submitted tasks to complete, and
> the {{awaitTermination}} call waits at most 100 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961525#comment-15961525
]
Hive QA commented on HIVE-16402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862533/HIVE-16402.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 10590 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testDoKillMultiple
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testDontKillMultiple
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testFinishablePreeptsNonFinishable
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionStateOnTaskMoveToFinishableState
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionStateOnTaskMoveToNonFinishableState
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueueAcceptAfterAMTaskReport
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueuePreemption
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparator
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorParallelism
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorWithinDagPriority
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorWithinSameDagPriority
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparator
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorAging
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinSameDagPriority
(batchId=285)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
(batchId=194)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
(batchId=193)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTimeout.org.apache.hadoop.hive.metastore.TestHiveMetaStoreTimeout
(batchId=194)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=194)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4613/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4613/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4613/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862533 - PreCommit-HIVE-Build
> Upgrade to Hadoop 2.8.0
> ---
>
> Key: HIVE-16402
> URL: https://issues.apache.org/jira/browse/HIVE-16402
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch,
> HIVE-16402.3.patch, HIVE-16402.4.patch, HIVE-16402.5.patch
>
>
> Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes
> for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html
> It has a number of useful features, improvements for S3 support, ADLS
> support, etc. along with a bunch of other fixes. This should also help us on
> our way to upgrading to Hadoop
[jira] [Commented] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961490#comment-15961490
]
Sergio Peña commented on HIVE-16389:
done.
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961475#comment-15961475
]
zhihai xu commented on HIVE-14564:
--
[~ashutoshc], Thanks for the review! Yes, use Set will be better, I uploaded a
new patch HIVE-14564.004.patch which updated golden files for test and use Set
for colNames. Please review it.
Looks like the following 5 test failures are related to my change
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar
(batchId=178)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:550)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:377)
> ... 13 more
> {code}
> The exception is because the serialization and deserialization doesn't match.
> The serialization by LazyBinarySerDe from previous MapReduce job used
> different order of columns. When the current MapReduce job deserialized the
> intermediate sequence file generated by previous MapReduce job, it will get
> corrupted data from the deserialization using wrong order of columns by
> LazyBinaryStruct. The unmatched columns between serialization and
> deserialization is caused by SelectOperator's Column Pruning
[jira] [Comment Edited] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961475#comment-15961475
]
zhihai xu edited comment on HIVE-14564 at 4/7/17 9:39 PM:
--
[~ashutoshc], Thanks for the review! Yes, use Set will be better, I uploaded a
new patch HIVE-14564.004.patch which updated golden files for test and use Set
for colNames. Please review it.
Looks like the following 5 test failures are not related to my change
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar
(batchId=178)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
was (Author: zxu):
[~ashutoshc], Thanks for the review! Yes, use Set will be better, I uploaded a
new patch HIVE-14564.004.patch which updated golden files for test and use Set
for colNames. Please review it.
Looks like the following 5 test failures are related to my change
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar
(batchId=178)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
>
[jira] [Comment Edited] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961475#comment-15961475
]
zhihai xu edited comment on HIVE-14564 at 4/7/17 9:39 PM:
--
[~ashutoshc], Thanks for the review! Yes, use Set will be better, I uploaded a
new patch HIVE-14564.004.patch which updated golden files for test and use Set
for colNames. Please review it.
Looks like the following 5 test failures are not related to my change:
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar
(batchId=178)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
was (Author: zxu):
[~ashutoshc], Thanks for the review! Yes, use Set will be better, I uploaded a
new patch HIVE-14564.004.patch which updated golden files for test and use Set
for colNames. Please review it.
Looks like the following 5 test failures are not related to my change
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar
(batchId=178)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
>
[jira] [Updated] (HIVE-14564) Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
[
https://issues.apache.org/jira/browse/HIVE-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-14564:
-
Attachment: HIVE-14564.004.patch
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> ---
>
> Key: HIVE-14564
> URL: https://issues.apache.org/jira/browse/HIVE-14564
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Attachments: HIVE-14564.000.patch, HIVE-14564.001.patch,
> HIVE-14564.002.patch, HIVE-14564.003.patch, HIVE-14564.004.patch
>
>
> Column Pruning generates out of order columns in SelectOperator which cause
> ArrayIndexOutOfBoundsException.
> {code}
> 2016-07-26 21:49:24,390 FATAL [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.io.Text.set(Text.java:225)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:264)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:201)
> at
> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:550)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:377)
> ... 13 more
> {code}
> The exception is because the serialization and deserialization doesn't match.
> The serialization by LazyBinarySerDe from previous MapReduce job used
> different order of columns. When the current MapReduce job deserialized the
> intermediate sequence file generated by previous MapReduce job, it will get
> corrupted data from the deserialization using wrong order of columns by
> LazyBinaryStruct. The unmatched columns between serialization and
> deserialization is caused by SelectOperator's Column Pruning
> {{ColumnPrunerSelectProc}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16402) Upgrade to Hadoop 2.8.0
[ https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-16402: Attachment: HIVE-16402.5.patch > Upgrade to Hadoop 2.8.0 > --- > > Key: HIVE-16402 > URL: https://issues.apache.org/jira/browse/HIVE-16402 > Project: Hive > Issue Type: Bug >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch, > HIVE-16402.3.patch, HIVE-16402.4.patch, HIVE-16402.5.patch > > > Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes > for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html > It has a number of useful features, improvements for S3 support, ADLS > support, etc. along with a bunch of other fixes. This should also help us on > our way to upgrading to Hadoop 3.x (HIVE-15016). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16257) Intermittent issue with incorrect resultset with Spark
[
https://issues.apache.org/jira/browse/HIVE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961436#comment-15961436
]
Xuefu Zhang commented on HIVE-16257:
[~ngangam], Yes, our build has HIVE-12768, which explains why we cannot
reproduce the issue. Nevertheless, you should check why you have 4 tasks for a
5-row file. I also have spark.executor.core=4, and I only see one task. Also, I
was using text file. That can also relate to minSplitSize, etc.
Besides HIVE-12768, we have also fixed many concurrency issues that can impact
HoS.
Thanks for looking to this problem.
> Intermittent issue with incorrect resultset with Spark
> --
>
> Key: HIVE-16257
> URL: https://issues.apache.org/jira/browse/HIVE-16257
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: failed_yarnlogs.log, success_yarnlogs.log
>
>
> This issue is highly intermittent that only seems to occurs with spark engine
> when the query has a GROUPBY clause. The following is the testcase.
> {code}
> drop table if exists test_hos_sample;
> create table test_hos_sample (name string, val1 decimal(18,2), val2
> decimal(20,3));
> insert into test_hos_sample values
> ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
> set hive.execution.engine=spark;
> select name, val1,val2 from test_hos_sample group by name, val1, val2;
> {code}
> Expected Results:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 101.12 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> Incorrect results once in a while:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 104.52 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> 1) Not reproducible with HoMR.
> 2) Not an issue when running from spark-shell.
> 3) Not reproducible when the column data type is String or double. Only
> reproducible with decimal data types. Also works fine for decimal datatype if
> you cast decimal as string on read and cast it back to decimal on select.
> 4) Occurs with parquet and text file format as well. (havent tried with other
> formats).
> 5) Occurs in both scenarios when table data is within encryption zone and
> outside.
> 6) Even in clusters where this is reproducible, this occurs once in like 20
> times or more.
> 7) Occurs with both Beeline and Hive CLI.
> 8) Reproducible only when there is a a groupby clause.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961419#comment-15961419
]
Sahil Takiar commented on HIVE-16389:
-
Thanks [~spena]! Any chance you could re-publish the your comments? For some
reason they aren't showing up for me, are they showing up for you?
(https://reviews.apache.org/r/58236/).
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HIVE-16257) Intermittent issue with incorrect resultset with Spark
[
https://issues.apache.org/jira/browse/HIVE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam resolved HIVE-16257.
--
Resolution: Duplicate
Assignee: Naveen Gangam
> Intermittent issue with incorrect resultset with Spark
> --
>
> Key: HIVE-16257
> URL: https://issues.apache.org/jira/browse/HIVE-16257
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: failed_yarnlogs.log, success_yarnlogs.log
>
>
> This issue is highly intermittent that only seems to occurs with spark engine
> when the query has a GROUPBY clause. The following is the testcase.
> {code}
> drop table if exists test_hos_sample;
> create table test_hos_sample (name string, val1 decimal(18,2), val2
> decimal(20,3));
> insert into test_hos_sample values
> ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
> set hive.execution.engine=spark;
> select name, val1,val2 from test_hos_sample group by name, val1, val2;
> {code}
> Expected Results:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 101.12 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> Incorrect results once in a while:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 104.52 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> 1) Not reproducible with HoMR.
> 2) Not an issue when running from spark-shell.
> 3) Not reproducible when the column data type is String or double. Only
> reproducible with decimal data types. Also works fine for decimal datatype if
> you cast decimal as string on read and cast it back to decimal on select.
> 4) Occurs with parquet and text file format as well. (havent tried with other
> formats).
> 5) Occurs in both scenarios when table data is within encryption zone and
> outside.
> 6) Even in clusters where this is reproducible, this occurs once in like 20
> times or more.
> 7) Occurs with both Beeline and Hive CLI.
> 8) Reproducible only when there is a a groupby clause.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16257) Intermittent issue with incorrect resultset with Spark
[
https://issues.apache.org/jira/browse/HIVE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961411#comment-15961411
]
Naveen Gangam commented on HIVE-16257:
--
[~xuefuz] In my case, it is always coming out to 4 tasks (in the first stage)
for some reason. There is a single input file on hdfs. However, the default
{{spark.executor.cores}} for me is 4. I dont know if they are related. After
adding some instrumentation, the data being passed to reducers was found to be
correct indicating the problem was prior to this phase. Seems like we use serde
to parse decimal types which pointed to HIVE-12768. It does not occur with HoMR
because each mapper is run in a seperate process. It also does not occur when
using a single executor.cores because it uses single thread to process it.
With the fix for HIVE-12768, we are currently unable to reproduce the issue
after several dozens of retries. As far as it being an intermittent issue, I
think the size of the data matters. The larger the dataset, the higher the
chance of reproducing it. In this case, the data was very small and hence was
hard to reproduce. Closing this jira. Thanks for all your help [~xuefuz]
> Intermittent issue with incorrect resultset with Spark
> --
>
> Key: HIVE-16257
> URL: https://issues.apache.org/jira/browse/HIVE-16257
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
> Attachments: failed_yarnlogs.log, success_yarnlogs.log
>
>
> This issue is highly intermittent that only seems to occurs with spark engine
> when the query has a GROUPBY clause. The following is the testcase.
> {code}
> drop table if exists test_hos_sample;
> create table test_hos_sample (name string, val1 decimal(18,2), val2
> decimal(20,3));
> insert into test_hos_sample values
> ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
> set hive.execution.engine=spark;
> select name, val1,val2 from test_hos_sample group by name, val1, val2;
> {code}
> Expected Results:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 101.12 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> Incorrect results once in a while:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 104.52 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> 1) Not reproducible with HoMR.
> 2) Not an issue when running from spark-shell.
> 3) Not reproducible when the column data type is String or double. Only
> reproducible with decimal data types. Also works fine for decimal datatype if
> you cast decimal as string on read and cast it back to decimal on select.
> 4) Occurs with parquet and text file format as well. (havent tried with other
> formats).
> 5) Occurs in both scenarios when table data is within encryption zone and
> outside.
> 6) Even in clusters where this is reproducible, this occurs once in like 20
> times or more.
> 7) Occurs with both Beeline and Hive CLI.
> 8) Reproducible only when there is a a groupby clause.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961200#comment-15961200
]
Johndee Burks edited comment on HIVE-14737 at 4/7/17 8:46 PM:
--
I have looked into this and the problem is the following code in the situation
of hive.server2.webui.use.spenego being set to false in a secure cluster.
[Code
Link|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java#L39]
{code}
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// Do the authorization
if (HttpServer.hasAdministratorAccess(getServletContext(), request,
response)) {
// Authorization is done. Just call super.
super.doGet(request, response);
{code}
In a secure cluster HttpServer.hasAdministratorAccess will always evaluate
false because of HADOOP_SECURITY_AUTHORIZATION. The code can be seen below.
[Code
Link|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/http/HttpServer.java#L259]
{code}
static boolean hasAdministratorAccess(
ServletContext servletContext, HttpServletRequest request,
HttpServletResponse response) throws IOException {
Configuration conf =
(Configuration) servletContext.getAttribute(CONF_CONTEXT_ATTRIBUTE);
// If there is no authorization, anybody has administrator access.
if (!conf.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
return true;
}
{code}
I am fairly certain if HttpServer.hasAdministratorAccess is changed to
HttpServer. isInstrumentationAccessAllowed this would work without issue. I am
looking into the implications of making this change.
was (Author: johndee):
I have looked into this and the problem is the following code.
[Code
Link|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java#L39]
{code}
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// Do the authorization
if (HttpServer.hasAdministratorAccess(getServletContext(), request,
response)) {
// Authorization is done. Just call super.
super.doGet(request, response);
{code}
In a secure cluster HttpServer.hasAdministratorAccess will always evaluate
false because of HADOOP_SECURITY_AUTHORIZATION. The code can be seen below.
[Code
Link|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/http/HttpServer.java#L259]
{code}
static boolean hasAdministratorAccess(
ServletContext servletContext, HttpServletRequest request,
HttpServletResponse response) throws IOException {
Configuration conf =
(Configuration) servletContext.getAttribute(CONF_CONTEXT_ATTRIBUTE);
// If there is no authorization, anybody has administrator access.
if (!conf.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
return true;
}
{code}
I am fairly certain if HttpServer.hasAdministratorAccess is changed to
HttpServer. isInstrumentationAccessAllowed this would work without issue. I am
looking into the implications of making this change.
> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
>
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Johndee Burks
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15272) "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark
[
https://issues.apache.org/jira/browse/HIVE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961373#comment-15961373
]
Miklos Szurap commented on HIVE-15272:
--
As the amount field is a DECIMAL, it is very likely that HIVE-12768 is the root
cause.
> "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark
> --
>
> Key: HIVE-15272
> URL: https://issues.apache.org/jira/browse/HIVE-15272
> Project: Hive
> Issue Type: Bug
> Components: Hive, Spark
>Affects Versions: 1.1.0
> Environment: Hive 1.1.0, CentOS, Cloudera 5.7.4
>Reporter: Vikash Pareek
>Assignee: Rui Li
>
> I ran following Hive query multiple times with execution engine as Hive on
> Spark and Hive on MapReduce.
> {code}
> SELECT COUNT(DISTINCT t1.region, t1.amount)
> FROM my_db.my_table1 t1
> LEFT OUTER
> JOIN my_db.my_table2 t2 ON (t1.id = t2.id
> AND t1.name = t2.name)
> {code}
> With Hive on Spark: Result (count) were different of every execution.
> With Hive on MapReduce: Result (count) were same of every execution.
> Seems like Hive on Spark behaving differently in each execution and does not
> populating correct result.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961365#comment-15961365
]
Hive QA commented on HIVE-16402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862516/HIVE-16402.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 10590 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=161)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testDoKillMultiple
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testDontKillMultiple
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testFinishablePreeptsNonFinishable
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionStateOnTaskMoveToFinishableState
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionStateOnTaskMoveToNonFinishableState
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueueAcceptAfterAMTaskReport
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueuePreemption
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparator
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorParallelism
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorWithinDagPriority
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorWithinSameDagPriority
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparator
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorAging
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
(batchId=285)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinSameDagPriority
(batchId=285)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
(batchId=194)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
(batchId=193)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTimeout.org.apache.hadoop.hive.metastore.TestHiveMetaStoreTimeout
(batchId=194)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=194)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4612/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4612/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4612/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862516 - PreCommit-HIVE-Build
> Upgrade to Hadoop 2.8.0
> ---
>
> Key: HIVE-16402
> URL: https://issues.apache.org/jira/browse/HIVE-16402
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch,
> HIVE-16402.3.patch, HIVE-16402.4.patch
>
>
> Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes
> for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html
> It has a number of useful features, improvements for S3 support, ADLS
> support, etc. along with a bunch of other fixes. This should also help us on
> our way to upgrading to Hadoop 3.x (HIVE-15016).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Issue Comment Deleted] (HIVE-16257) Intermittent issue with incorrect resultset with Spark
[
https://issues.apache.org/jira/browse/HIVE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szurap updated HIVE-16257:
-
Comment: was deleted
(was: It seems like HIVE-12768 is the root cause.)
> Intermittent issue with incorrect resultset with Spark
> --
>
> Key: HIVE-16257
> URL: https://issues.apache.org/jira/browse/HIVE-16257
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
> Attachments: failed_yarnlogs.log, success_yarnlogs.log
>
>
> This issue is highly intermittent that only seems to occurs with spark engine
> when the query has a GROUPBY clause. The following is the testcase.
> {code}
> drop table if exists test_hos_sample;
> create table test_hos_sample (name string, val1 decimal(18,2), val2
> decimal(20,3));
> insert into test_hos_sample values
> ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
> set hive.execution.engine=spark;
> select name, val1,val2 from test_hos_sample group by name, val1, val2;
> {code}
> Expected Results:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 101.12 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> Incorrect results once in a while:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 104.52 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> 1) Not reproducible with HoMR.
> 2) Not an issue when running from spark-shell.
> 3) Not reproducible when the column data type is String or double. Only
> reproducible with decimal data types. Also works fine for decimal datatype if
> you cast decimal as string on read and cast it back to decimal on select.
> 4) Occurs with parquet and text file format as well. (havent tried with other
> formats).
> 5) Occurs in both scenarios when table data is within encryption zone and
> outside.
> 6) Even in clusters where this is reproducible, this occurs once in like 20
> times or more.
> 7) Occurs with both Beeline and Hive CLI.
> 8) Reproducible only when there is a a groupby clause.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16257) Intermittent issue with incorrect resultset with Spark
[
https://issues.apache.org/jira/browse/HIVE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961369#comment-15961369
]
Miklos Szurap commented on HIVE-16257:
--
It seems like HIVE-12768 is the root cause.
> Intermittent issue with incorrect resultset with Spark
> --
>
> Key: HIVE-16257
> URL: https://issues.apache.org/jira/browse/HIVE-16257
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
> Attachments: failed_yarnlogs.log, success_yarnlogs.log
>
>
> This issue is highly intermittent that only seems to occurs with spark engine
> when the query has a GROUPBY clause. The following is the testcase.
> {code}
> drop table if exists test_hos_sample;
> create table test_hos_sample (name string, val1 decimal(18,2), val2
> decimal(20,3));
> insert into test_hos_sample values
> ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
> set hive.execution.engine=spark;
> select name, val1,val2 from test_hos_sample group by name, val1, val2;
> {code}
> Expected Results:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 101.12 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> Incorrect results once in a while:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 104.52 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> 1) Not reproducible with HoMR.
> 2) Not an issue when running from spark-shell.
> 3) Not reproducible when the column data type is String or double. Only
> reproducible with decimal data types. Also works fine for decimal datatype if
> you cast decimal as string on read and cast it back to decimal on select.
> 4) Occurs with parquet and text file format as well. (havent tried with other
> formats).
> 5) Occurs in both scenarios when table data is within encryption zone and
> outside.
> 6) Even in clusters where this is reproducible, this occurs once in like 20
> times or more.
> 7) Occurs with both Beeline and Hive CLI.
> 8) Reproducible only when there is a a groupby clause.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16257) Intermittent issue with incorrect resultset with Spark
[
https://issues.apache.org/jira/browse/HIVE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961367#comment-15961367
]
Miklos Szurap commented on HIVE-16257:
--
It seems like HIVE-12768 is the root cause.
> Intermittent issue with incorrect resultset with Spark
> --
>
> Key: HIVE-16257
> URL: https://issues.apache.org/jira/browse/HIVE-16257
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
> Attachments: failed_yarnlogs.log, success_yarnlogs.log
>
>
> This issue is highly intermittent that only seems to occurs with spark engine
> when the query has a GROUPBY clause. The following is the testcase.
> {code}
> drop table if exists test_hos_sample;
> create table test_hos_sample (name string, val1 decimal(18,2), val2
> decimal(20,3));
> insert into test_hos_sample values
> ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
> set hive.execution.engine=spark;
> select name, val1,val2 from test_hos_sample group by name, val1, val2;
> {code}
> Expected Results:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 101.12 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> Incorrect results once in a while:
> {code}
> nameval1val2
> test5 105.52 105.567
> test3 103.52 102.345
> test1 104.52 102.123
> test4 104.52 104.456
> test2 102.12 103.234
> {code}
> 1) Not reproducible with HoMR.
> 2) Not an issue when running from spark-shell.
> 3) Not reproducible when the column data type is String or double. Only
> reproducible with decimal data types. Also works fine for decimal datatype if
> you cast decimal as string on read and cast it back to decimal on select.
> 4) Occurs with parquet and text file format as well. (havent tried with other
> formats).
> 5) Occurs in both scenarios when table data is within encryption zone and
> outside.
> 6) Even in clusters where this is reproducible, this occurs once in like 20
> times or more.
> 7) Occurs with both Beeline and Hive CLI.
> 8) Reproducible only when there is a a groupby clause.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961360#comment-15961360
]
Sergio Peña commented on HIVE-16389:
The patch looks good. I left a few minor comments on the RB.
+1
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16402) Upgrade to Hadoop 2.8.0
[ https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-16402: Attachment: HIVE-16402.4.patch > Upgrade to Hadoop 2.8.0 > --- > > Key: HIVE-16402 > URL: https://issues.apache.org/jira/browse/HIVE-16402 > Project: Hive > Issue Type: Bug >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch, > HIVE-16402.3.patch, HIVE-16402.4.patch > > > Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes > for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html > It has a number of useful features, improvements for S3 support, ADLS > support, etc. along with a bunch of other fixes. This should also help us on > our way to upgrading to Hadoop 3.x (HIVE-15016). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16407) Compilation of a query with lots of conditions (like 2200) in a where clause can take a lot of time
[ https://issues.apache.org/jira/browse/HIVE-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan updated HIVE-16407: - Attachment: issue_reproduce.sql > Compilation of a query with lots of conditions (like 2200) in a where clause > can take a lot of time > --- > > Key: HIVE-16407 > URL: https://issues.apache.org/jira/browse/HIVE-16407 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sudarshan > Attachments: issue_reproduce.sql > > > Following is the table structure. > > create table test_hello ( > a string); > > A query that involves lots of "OR" conditions can in a "where" clause, can > take a really long time to compile. It doesn't finishes compilation in even > 30 to 40 minutes. > > select * from test_hello > where > ( > a like '%1%' or > > a like '%1%' > ) > + > (Attaching the entire script to the Jira) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HIVE-15535) Flaky test : TestHS2HttpServer.testContextRootUrlRewrite
[
https://issues.apache.org/jira/browse/HIVE-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam reopened HIVE-15535:
--
> Flaky test : TestHS2HttpServer.testContextRootUrlRewrite
>
>
> Key: HIVE-15535
> URL: https://issues.apache.org/jira/browse/HIVE-15535
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sushanth Sowmyan
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15535.01.patch
>
>
> Per recent test failure :
> https://builds.apache.org/job/PreCommit-HIVE-Build/2766/testReport/org.apache.hive.service.server/TestHS2HttpServer/testContextRootUrlRewrite/
> {noformat}
> Stacktrace
> org.junit.ComparisonFailure:
> expected:<...d>Tue Jan 03 11:54:4[6] PST 2017
> ...> but was:<...d>Tue Jan 03 11:54:4[7] PST 2017
> ...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite(TestHS2HttpServer.java:99)
> {noformat}
> Looks like it is overly picky on an exact string match on a field that
> contains a second difference.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961200#comment-15961200
]
Johndee Burks commented on HIVE-14737:
--
I have looked into this and the problem is the following code.
[Code
Link|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java#L39]
{code}
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// Do the authorization
if (HttpServer.hasAdministratorAccess(getServletContext(), request,
response)) {
// Authorization is done. Just call super.
super.doGet(request, response);
{code}
In a secure cluster HttpServer.hasAdministratorAccess will always evaluate
false because of HADOOP_SECURITY_AUTHORIZATION. The code can be seen below.
[Code
Link|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/http/HttpServer.java#L259]
{code}
static boolean hasAdministratorAccess(
ServletContext servletContext, HttpServletRequest request,
HttpServletResponse response) throws IOException {
Configuration conf =
(Configuration) servletContext.getAttribute(CONF_CONTEXT_ATTRIBUTE);
// If there is no authorization, anybody has administrator access.
if (!conf.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
return true;
}
{code}
I am fairly certain if HttpServer.hasAdministratorAccess is changed to
HttpServer. isInstrumentationAccessAllowed this would work without issue. I am
looking into the implications of making this change.
> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
>
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Johndee Burks
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15630) add operation handle before operation.run instead of after operation.run
[
https://issues.apache.org/jira/browse/HIVE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-15630:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks Zhihai for the patch. Pushed to the master branch.
> add operation handle before operation.run instead of after operation.run
>
>
> Key: HIVE-15630
> URL: https://issues.apache.org/jira/browse/HIVE-15630
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15630.000.patch, HIVE-15630.001.patch
>
>
> Add operation handle before operation.run instead of after operation.run. So
> when session is closed, all the running operations from {{operation.run}} can
> also be closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johndee Burks reassigned HIVE-14737: Assignee: Johndee Burks > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Matyas Orhidi >Assignee: Johndee Burks > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[ https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961189#comment-15961189 ] Yongzhi Chen commented on HIVE-16345: - PATCH 3 looks fine +1 > BeeLineDriver should be able to run qtest files which are using default > database tables > --- > > Key: HIVE-16345 > URL: https://issues.apache.org/jira/browse/HIVE-16345 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16345.2.patch, HIVE-16345.3.patch, HIVE-16345.patch > > > It would be good to be able to run the default clientpositive tests. > Currently we can not do that, since we start with a specific database. We > should filter the query input and replace the table references -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15535) Flaky test : TestHS2HttpServer.testContextRootUrlRewrite
[
https://issues.apache.org/jira/browse/HIVE-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961120#comment-15961120
]
Barna Zsombor Klara commented on HIVE-15535:
[~stakiar] yes... because I'm an idiot... I wrote an regexp that matches the
string on the error message to replace it... including the square bracket
characters added for the diff by junit which of course are not in the original
string. I will fix it.
> Flaky test : TestHS2HttpServer.testContextRootUrlRewrite
>
>
> Key: HIVE-15535
> URL: https://issues.apache.org/jira/browse/HIVE-15535
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sushanth Sowmyan
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15535.01.patch
>
>
> Per recent test failure :
> https://builds.apache.org/job/PreCommit-HIVE-Build/2766/testReport/org.apache.hive.service.server/TestHS2HttpServer/testContextRootUrlRewrite/
> {noformat}
> Stacktrace
> org.junit.ComparisonFailure:
> expected:<...d>Tue Jan 03 11:54:4[6] PST 2017
> ...> but was:<...d>Tue Jan 03 11:54:4[7] PST 2017
> ...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite(TestHS2HttpServer.java:99)
> {noformat}
> Looks like it is overly picky on an exact string match on a field that
> contains a second difference.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[ https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961071#comment-15961071 ] Sergio Peña commented on HIVE-16402: Agree. Some folks from Hadoop were telling us in the past to use Hadoop 2.8 because good improvements on S3. > Upgrade to Hadoop 2.8.0 > --- > > Key: HIVE-16402 > URL: https://issues.apache.org/jira/browse/HIVE-16402 > Project: Hive > Issue Type: Bug >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch, > HIVE-16402.3.patch > > > Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes > for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html > It has a number of useful features, improvements for S3 support, ADLS > support, etc. along with a bunch of other fixes. This should also help us on > our way to upgrading to Hadoop 3.x (HIVE-15016). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961028#comment-15961028
]
Hive QA commented on HIVE-16402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862498/HIVE-16402.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4611/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4611/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4611/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-07 16:08:39.422
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4611/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 16:08:39.424
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a5f94c0 HIVE-16291: Hive fails when unions a parquet table with
itself (Yibing Shi, reviewed by Ashutosh Chauhan & Aihua Xu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a5f94c0 HIVE-16291: Hive fails when unions a parquet table with
itself (Yibing Shi, reviewed by Ashutosh Chauhan & Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 16:08:39.900
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file pom.xml
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-shims-0.23: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-0.23:jar:3.0.0-SNAPSHOT: The following
artifacts could not be resolved:
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:2.8.0,
de.ruedigermoeller:fst:jar:2.24,
org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.8.0,
org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:2.8.0,
org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:2.8.0: Could not find
artifact
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:2.8.0 in
datanucleus (http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-0.23
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862498 - PreCommit-HIVE-Build
> Upgrade to Hadoop 2.8.0
> ---
>
> Key: HIVE-16402
> URL: https://issues.apache.org/jira/browse/HIVE-16402
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch,
> HIVE-16402.3.patch
>
>
> Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes
> for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html
> It has a number of useful features, improvements for S3 support, ADLS
> support, etc. along with a bunch of other
[jira] [Updated] (HIVE-16402) Upgrade to Hadoop 2.8.0
[ https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-16402: Attachment: HIVE-16402.3.patch > Upgrade to Hadoop 2.8.0 > --- > > Key: HIVE-16402 > URL: https://issues.apache.org/jira/browse/HIVE-16402 > Project: Hive > Issue Type: Bug >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch, > HIVE-16402.3.patch > > > Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes > for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html > It has a number of useful features, improvements for S3 support, ADLS > support, etc. along with a bunch of other fixes. This should also help us on > our way to upgrading to Hadoop 3.x (HIVE-15016). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11418) Dropping a database in an encryption zone with CASCADE and trash enabled fails
[
https://issues.apache.org/jira/browse/HIVE-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961020#comment-15961020
]
Sahil Takiar commented on HIVE-11418:
-
No, it won't work with older Hadoop versions. I've opened a JIRA to upgrade to
Hadoop 2.8.0 - HIVE-16402
So it may still be worth fixing this, at least in branch-2; thoughts?
> Dropping a database in an encryption zone with CASCADE and trash enabled fails
> --
>
> Key: HIVE-11418
> URL: https://issues.apache.org/jira/browse/HIVE-11418
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 1.2.0
>Reporter: Sergio Peña
>Assignee: Sahil Takiar
>
> Here's the query that fails:
> {noformat}
> hive> CREATE DATABASE db;
> hive> USE db;
> hive> CREATE TABLE a(id int);
> hive> SET fs.trash.interval=1;
> hive> DROP DATABASE db CASCADE;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
> db.a because it is in an encryption zone and trash
> is enabled. Use PURGE option to skip trash.)
> {noformat}
> DROP DATABASE does not support PURGE, so we have to remove the tables one by
> one, and then drop the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961018#comment-15961018
]
Hive QA commented on HIVE-16402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862496/HIVE-16402.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4610/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4610/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4610/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-07 16:01:24.336
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4610/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 16:01:24.338
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a5f94c0 HIVE-16291: Hive fails when unions a parquet table with
itself (Yibing Shi, reviewed by Ashutosh Chauhan & Aihua Xu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a5f94c0 HIVE-16291: Hive fails when unions a parquet table with
itself (Yibing Shi, reviewed by Ashutosh Chauhan & Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 16:01:24.823
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file pom.xml
patching file testutils/ptest2/src/main/resources/source-prep.vm
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-common:jar:3.0.0-SNAPSHOT: The following
artifacts could not be resolved: org.apache.hadoop:hadoop-auth:jar:2.8.0,
com.nimbusds:nimbus-jose-jwt:jar:3.9,
org.apache.hadoop:hadoop-annotations:jar:2.8.0: Could not find artifact
org.apache.hadoop:hadoop-auth:jar:2.8.0 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-common
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862496 - PreCommit-HIVE-Build
> Upgrade to Hadoop 2.8.0
> ---
>
> Key: HIVE-16402
> URL: https://issues.apache.org/jira/browse/HIVE-16402
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch
>
>
> Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes
> for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html
> It has a number of useful features, improvements for S3 support, ADLS
> support, etc. along with a bunch of other fixes. This should also help us on
> our way to upgrading to Hadoop 3.x (HIVE-15016).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16392) Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance logic
[
https://issues.apache.org/jira/browse/HIVE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961007#comment-15961007
]
Sahil Takiar commented on HIVE-16392:
-
Test failures are flaky:
HIVE-16345 - TestBeeLineDriver.testCliDriver[drop_with_concurrency]
HIVE-16320 - TestBeeLineDriver.testCliDriver[escape_comments]
HIVE-16387 - TestJdbcDriver2.testResultSetMetaData
HIVE-15535 - TestHS2HttpServer.testContextRootUrlRewrite
Should be ready to push.
> Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance
> logic
>
>
> Key: HIVE-16392
> URL: https://issues.apache.org/jira/browse/HIVE-16392
> Project: Hive
> Issue Type: Task
> Components: Security
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Labels: backwards-incompatible
> Attachments: HIVE-16392.1.patch, HIVE-16392.2.patch
>
>
> As discussed in HIVE-16346 we should remove the config
> {{hive.warehouse.subdir.inherit.perms}} and all the permissions inheritance
> logic.
> This feature is no longer needed in Hive as the traditional permission model
> has largely been replaced by external security systems such as Ranger and
> Sentry.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-15058) [Umbrella] Current Flaky Tests
[ https://issues.apache.org/jira/browse/HIVE-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-15058: --- Assignee: Sahil Takiar > [Umbrella] Current Flaky Tests > -- > > Key: HIVE-15058 > URL: https://issues.apache.org/jira/browse/HIVE-15058 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Sahil Takiar > > Jira to track known flaky tests, to make it easier to cite jiras before > committing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11418) Dropping a database in an encryption zone with CASCADE and trash enabled fails
[
https://issues.apache.org/jira/browse/HIVE-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961003#comment-15961003
]
Sergio Peña commented on HIVE-11418:
Oh, sorry. I misunderstood the HDFS jira. I see that previously to the patch
the support was to delete only root encryption zones. This new patch on HDFS
now allows to delete any file under an encryption zone and send it to the trash.
We could try to bump our Hadoop version to 2.8.0 and take advantage of this.
But, is this going to work if a user is running Hive on an older Hadoop version?
> Dropping a database in an encryption zone with CASCADE and trash enabled fails
> --
>
> Key: HIVE-11418
> URL: https://issues.apache.org/jira/browse/HIVE-11418
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 1.2.0
>Reporter: Sergio Peña
>Assignee: Sahil Takiar
>
> Here's the query that fails:
> {noformat}
> hive> CREATE DATABASE db;
> hive> USE db;
> hive> CREATE TABLE a(id int);
> hive> SET fs.trash.interval=1;
> hive> DROP DATABASE db CASCADE;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
> db.a because it is in an encryption zone and trash
> is enabled. Use PURGE option to skip trash.)
> {noformat}
> DROP DATABASE does not support PURGE, so we have to remove the tables one by
> one, and then drop the database.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15535) Flaky test : TestHS2HttpServer.testContextRootUrlRewrite
[
https://issues.apache.org/jira/browse/HIVE-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961000#comment-15961000
]
Sahil Takiar commented on HIVE-15535:
-
[~zsombor.klara] could this still be happening. I got a failure recently:
https://builds.apache.org/job/PreCommit-HIVE-Build/4598/testReport/org.apache.hive.service.server/TestHS2HttpServer/testContextRootUrlRewrite/
Stack-trace:
{code}
org.junit.ComparisonFailure: expected:<...d>Thu Apr 06 23:47:1[1] PDT 2017
...> but was:<...d>Thu Apr 06 23:47:1[2] PDT 2017
...>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite(TestHS2HttpServer.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
> Flaky test : TestHS2HttpServer.testContextRootUrlRewrite
>
>
> Key: HIVE-15535
> URL: https://issues.apache.org/jira/browse/HIVE-15535
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sushanth Sowmyan
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15535.01.patch
>
>
> Per recent test failure :
> https://builds.apache.org/job/PreCommit-HIVE-Build/2766/testReport/org.apache.hive.service.server/TestHS2HttpServer/testContextRootUrlRewrite/
> {noformat}
> Stacktrace
> org.junit.ComparisonFailure:
> expected:<...d>Tue Jan 03 11:54:4[6] PST 2017
> ...> but was:<...d>Tue Jan 03 11:54:4[7] PST 2017
> ...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite(TestHS2HttpServer.java:99)
> {noformat}
> Looks like it is overly picky on an exact string match on a field that
> contains a second difference.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[ https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960981#comment-15960981 ] Sahil Takiar commented on HIVE-16402: - This should also help with HIVE-11418 > Upgrade to Hadoop 2.8.0 > --- > > Key: HIVE-16402 > URL: https://issues.apache.org/jira/browse/HIVE-16402 > Project: Hive > Issue Type: Bug >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch > > > Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes > for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html > It has a number of useful features, improvements for S3 support, ADLS > support, etc. along with a bunch of other fixes. This should also help us on > our way to upgrading to Hadoop 3.x (HIVE-15016). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16402) Upgrade to Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16402:
Attachment: HIVE-16402.2.patch
Failed due to:
{code}
[ERROR] Failed to execute goal on project hive-hcatalog: Could not resolve
dependencies for project
org.apache.hive.hcatalog:hive-hcatalog:pom:3.0.0-SNAPSHOT: Failed to collect
dependencies for [org.mockito:mockito-all:jar:1.9.5 (test),
org.apache.hadoop:hadoop-common:jar:2.8.0 (test),
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.0 (test),
org.apache.pig:pig:jar:h2:0.16.0 (test), org.slf4j:slf4j-api:jar:1.7.10
(compile), org.skyscreamer:jsonassert:jar:1.4.0 (test)]: Failed to read
artifact descriptor for org.apache.hadoop:hadoop-common:jar:2.8.0: Could not
find artifact org.apache.hadoop:hadoop-project:pom:2.8.0 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
{code}
Not sure why, build works locally. Think it may be a race condition. I could
re-produce similar errors on a fresh Ubuntu VM; when I removed {{-T 4}} from
the maven build parameters everything worked. Removing the {{-T}} option for
now, will add it back.
> Upgrade to Hadoop 2.8.0
> ---
>
> Key: HIVE-16402
> URL: https://issues.apache.org/jira/browse/HIVE-16402
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16402.1.patch, HIVE-16402.2.patch
>
>
> Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes
> for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html
> It has a number of useful features, improvements for S3 support, ADLS
> support, etc. along with a bunch of other fixes. This should also help us on
> our way to upgrading to Hadoop 3.x (HIVE-15016).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960928#comment-15960928 ] Peter Vary commented on HIVE-16146: --- [~kgyrtkirk], [~ychena], [~aihuaxu], [~vihangk1] you can review if you have time. Thanks, Peter > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960923#comment-15960923 ] Peter Vary commented on HIVE-16146: --- The test failures are not related: - HIVE-16405 - Flaky Test: org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData - HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960881#comment-15960881
]
Hive QA commented on HIVE-16406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862479/HIVE-16406.profiler.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4609/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4609/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4609/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-07 14:32:51.768
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4609/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 14:32:51.771
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
0d0e497..a5f94c0 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 0d0e497 HIVE-16334: Query lock contains the query string, which
can cause OOM on ZooKeeper (Peter Vary via Chaoyu Tang)
+ git clean -f -d
Removing itests/util/src/main/java/org/apache/hive/beeline/QFile.java
Removing
itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java
Removing itests/util/src/main/java/org/apache/hive/beeline/package-info.java
Removing ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppenderForTest.java
Removing ql/src/test/results/clientpositive/beeline/select_dummy_source.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 4 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at a5f94c0 HIVE-16291: Hive fails when unions a parquet table with
itself (Yibing Shi, reviewed by Ashutosh Chauhan & Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 14:32:53.163
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862479 - PreCommit-HIVE-Build
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.2.patch,
> HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[
https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960878#comment-15960878
]
Hive QA commented on HIVE-16146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862465/HIVE-16146.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10596 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4608/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4608/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862465 - PreCommit-HIVE-Build
> If possible find a better way to filter the TestBeeLineDriver output
>
>
> Key: HIVE-16146
> URL: https://issues.apache.org/jira/browse/HIVE-16146
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch,
> HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch
>
>
> Currently we apply a blacklist to filter the output of the BeeLine Qtest runs.
> It might be a good idea to go thorough of the possibilities and find a better
> way, if possible.
> I think our main goal could be for the TestBeeLineDriver test output to match
> the TestCliDriver output of the came query file. Or if it is not possible,
> then at least a similar one
> CC: [~vihangk1]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16316) Prepare master branch for 3.0.0 development.
[
https://issues.apache.org/jira/browse/HIVE-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960856#comment-15960856
]
Naveen Gangam commented on HIVE-16316:
--
[~pxiong] Thanks for the review. That was my first comment in this jira.
I am not quite sure what the tentative release schedule is for 2.3 and 3.0. But
this is similar to what was done when branch-1 was used for 1.3 and 1.2 in the
process of being released and master then was 2.0 release. {{branch-1}} had
scripts for migrating 1.2-to-1.3 and {{master}} had scripts for migrating
1.2-to-2.0. Hope this makes sense. We could adjust it in the future as we get
closer to release dates.
> Prepare master branch for 3.0.0 development.
>
>
> Key: HIVE-16316
> URL: https://issues.apache.org/jira/browse/HIVE-16316
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 3.0.0
>
> Attachments: HIVE-16316.patch
>
>
> master branch is now being used for 3.0.0 development. The build files will
> need to reflect this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16340) Allow Kerberos + SSL connections to HMS
[ https://issues.apache.org/jira/browse/HIVE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960849#comment-15960849 ] Aihua Xu commented on HIVE-16340: - Pushed to master. Thanks Sahil for the work. > Allow Kerberos + SSL connections to HMS > --- > > Key: HIVE-16340 > URL: https://issues.apache.org/jira/browse/HIVE-16340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Fix For: 3.0.0 > > Attachments: HIVE-16340.1.patch, HIVE-16340.2.patch, > HIVE-16340.3.patch > > > It should be possible to connect to HMS with Kerberos authentication and SSL > enabled, at the same time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16400) Fix the MDC reference to use slf4j rather than log4j
[ https://issues.apache.org/jira/browse/HIVE-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16400: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks Yongzhi and Naveen for reviewing. > Fix the MDC reference to use slf4j rather than log4j > > > Key: HIVE-16400 > URL: https://issues.apache.org/jira/browse/HIVE-16400 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 3.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 3.0.0 > > Attachments: HIVE-16400.1.patch > > > HIVE-16061 added MDC reference incorrectly. The MDC reference in LogUtils is > using Log4J version, but we should use slf4j version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16340) Allow Kerberos + SSL connections to HMS
[ https://issues.apache.org/jira/browse/HIVE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16340: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Allow Kerberos + SSL connections to HMS > --- > > Key: HIVE-16340 > URL: https://issues.apache.org/jira/browse/HIVE-16340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Fix For: 3.0.0 > > Attachments: HIVE-16340.1.patch, HIVE-16340.2.patch, > HIVE-16340.3.patch > > > It should be possible to connect to HMS with Kerberos authentication and SSL > enabled, at the same time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16291) Hive fails when unions a parquet table with itself
[
https://issues.apache.org/jira/browse/HIVE-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16291:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Yibing for the work.
> Hive fails when unions a parquet table with itself
> --
>
> Key: HIVE-16291
> URL: https://issues.apache.org/jira/browse/HIVE-16291
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Fix For: 3.0.0
>
> Attachments: HIVE-16291.1.patch, HIVE-16291.2.patch
>
>
> Reproduce commands:
> {code:sql}
> create table tst_unin (col1 int) partitioned by (p_tdate int) stored as
> parquet;
> insert into tst_unin partition (p_tdate=201603) values (20160312), (20160310);
> insert into tst_unin partition (p_tdate=201604) values (20160412), (20160410);
> select count(*) from (select tst_unin.p_tdate from tst_unin where
> tst_unin.col1=20160302 union all select tst_unin.p_tdate from tst_unin) t1;
> {code}
> The table is stored in Parquet format, which is a columnar file format. Hive
> tries to push the query predicates to the table scan operators so that only
> the needed columns are read. This is done by adding the needed column IDs
> into job configuration with property "hive.io.file.readcolumn.ids".
> In above case, the query unions the result of 2 subqueries, which select data
> from one same table. The first subquery doesn't need any column from Parquet
> file, while the second subquery needs a column "col1". Hive has a bug here,
> it finally set "hive.io.file.readcolumn.ids" to a value like "0,,0", which
> method ColumnProjectionUtils.getReadColumnIDs cannot parse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16333) remove the redundant symbol "\" to appear red in sublime text 3
[ https://issues.apache.org/jira/browse/HIVE-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16333: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Saijin for the work. > remove the redundant symbol "\" to appear red in sublime text 3 > --- > > Key: HIVE-16333 > URL: https://issues.apache.org/jira/browse/HIVE-16333 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16333.1.patch, HIVE-16333.2.patch > > > In TxnHandler.java,i found a redundant symbol "\" in function getOpenTxns() > whch leads to appear red in sublime text 3. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16334) Query lock contains the query string, which can cause OOM on ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960825#comment-15960825 ] Peter Vary commented on HIVE-16334: --- Thanks for the commit [~ctang.ma]! Updated the document, added the new property. > Query lock contains the query string, which can cause OOM on ZooKeeper > -- > > Key: HIVE-16334 > URL: https://issues.apache.org/jira/browse/HIVE-16334 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: HIVE-16334.2.patch, HIVE-16334.3.patch, > HIVE-16334.4.patch, HIVE-16334.patch > > > When there are big number of partitions in a query this will result in a huge > number of locks on ZooKeeper. Since the query object contains the whole query > string this might cause serious memory pressure on the ZooKeeper services. > It would be good to have the possibility to truncate the query strings that > are written into the locks -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960804#comment-15960804
]
Hive QA commented on HIVE-16406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862479/HIVE-16406.profiler.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4607/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4607/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4607/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-07 13:36:49.368
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4607/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 13:36:49.370
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
a01a6a3..0d0e497 master -> origin/master
+ git reset --hard HEAD
HEAD is now at a01a6a3 HIVE-15616 : Improve contents of qfile test output
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 0d0e497 HIVE-16334: Query lock contains the query string, which
can cause OOM on ZooKeeper (Peter Vary via Chaoyu Tang)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 13:36:50.628
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862479 - PreCommit-HIVE-Build
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.2.patch,
> HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960801#comment-15960801 ] Aihua Xu commented on HIVE-15104: - [~lirui] I didn't have time to work on that . Feel free to take it over. > Hive on Spark generate more shuffle data than hive on mr > > > Key: HIVE-15104 > URL: https://issues.apache.org/jira/browse/HIVE-15104 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1 >Reporter: wangwenli >Assignee: Aihua Xu > > the same sql, running on spark and mr engine, will generate different size > of shuffle data. > i think it is because of hive on mr just serialize part of HiveKey, but hive > on spark which using kryo will serialize full of Hivekey object. > what is your opionion? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16151) BytesBytesHashTable allocates large arrays
[
https://issues.apache.org/jira/browse/HIVE-16151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960802#comment-15960802
]
Hive QA commented on HIVE-16151:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862414/HIVE-16151.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4606/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4606/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4606/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862414 - PreCommit-HIVE-Build
> BytesBytesHashTable allocates large arrays
> --
>
> Key: HIVE-16151
> URL: https://issues.apache.org/jira/browse/HIVE-16151
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16151.01.patch, HIVE-16151.patch
>
>
> These arrays cause GC pressure and also impose key count limitations on the
> table. Wrt the latter, we won't be able to get rid of it without a 64-bit
> hash function, but for now we can get rid of the former. If we need the
> latter we'd add murmur64 and probably account for it differently for resize
> (we don't want to blow up the hashtable by 4 bytes/key in the common case
> where #of keys is less than ~1.5B :))
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: HIVE-16406.profiler.png
HIVE-16406.2.patch
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.2.patch,
> HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15538) Test HIVE-13884 with more complex query predicates
[ https://issues.apache.org/jira/browse/HIVE-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960754#comment-15960754 ] Marta Kuczora commented on HIVE-15538: -- Thanks a lot [~ctang.ma] for committing the patch. > Test HIVE-13884 with more complex query predicates > -- > > Key: HIVE-15538 > URL: https://issues.apache.org/jira/browse/HIVE-15538 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 2.2.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora > Fix For: 3.0.0 > > Attachments: HIVE-15538.2.patch, HIVE-15538.3.patch, HIVE-15538.patch > > > HIVE-13884 introduced a new property hive.metastore.limit.partition.request. > It would be good to have more tests to cover the cases where the query > predicates (such as like, in) could not be pushed down to see if the fail > back from directsql to ORM works properly if hive.metastore.try.direct.sql is > enabled. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16334) Query lock contains the query string, which can cause OOM on ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-16334: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to 3.0.0. Thanks [~pvary] for the patch. I think you may need to update the document for the new property. > Query lock contains the query string, which can cause OOM on ZooKeeper > -- > > Key: HIVE-16334 > URL: https://issues.apache.org/jira/browse/HIVE-16334 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: HIVE-16334.2.patch, HIVE-16334.3.patch, > HIVE-16334.4.patch, HIVE-16334.patch > > > When there are big number of partitions in a query this will result in a huge > number of locks on ZooKeeper. Since the query object contains the whole query > string this might cause serious memory pressure on the ZooKeeper services. > It would be good to have the possibility to truncate the query strings that > are written into the locks -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16394) HoS does not support queue name change in middle of session
[ https://issues.apache.org/jira/browse/HIVE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-16394: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to 3.0.0. Thanks [~xuefuz], [~lirui] for review. > HoS does not support queue name change in middle of session > --- > > Key: HIVE-16394 > URL: https://issues.apache.org/jira/browse/HIVE-16394 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 3.0.0 > > Attachments: HIVE-16394.patch > > > The mapreduce.job.queuename only effects when HoS executes its query first > time. After that, changing mapreduce.job.queuename won't change the query > yarn scheduler queue name. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15538) Test HIVE-13884 with more complex query predicates
[ https://issues.apache.org/jira/browse/HIVE-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15538: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to 3.0.0. Thanks [~kuczoram] for the patch. > Test HIVE-13884 with more complex query predicates > -- > > Key: HIVE-15538 > URL: https://issues.apache.org/jira/browse/HIVE-15538 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 2.2.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora > Fix For: 3.0.0 > > Attachments: HIVE-15538.2.patch, HIVE-15538.3.patch, HIVE-15538.patch > > > HIVE-13884 introduced a new property hive.metastore.limit.partition.request. > It would be good to have more tests to cover the cases where the query > predicates (such as like, in) could not be pushed down to see if the fail > back from directsql to ORM works properly if hive.metastore.try.direct.sql is > enabled. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16387) Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
[
https://issues.apache.org/jira/browse/HIVE-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960713#comment-15960713
]
Hive QA commented on HIVE-16387:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862408/HIVE-16387.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 600 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=226)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=238)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=238)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=238)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=238)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_char1] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_varchar1]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join17] (batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join1] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join3] (batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join4] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join5] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join6] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join8] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join9] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_13]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_14]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_15]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket1] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket2] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket3] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=54)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: (was: HIVE-16406.profiler.png)
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16403) LLAP UI shows the wrong number of executors
[
https://issues.apache.org/jira/browse/HIVE-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960642#comment-15960642
]
Hive QA commented on HIVE-16403:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862387/HIVE-16403.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862387 - PreCommit-HIVE-Build
> LLAP UI shows the wrong number of executors
> ---
>
> Key: HIVE-16403
> URL: https://issues.apache.org/jira/browse/HIVE-16403
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16403.patch
>
>
> Queued tasks are added twice.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-10299) Enable new cost model for Tez execution engine
[
https://issues.apache.org/jira/browse/HIVE-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960601#comment-15960601
]
Hive QA commented on HIVE-10299:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862372/HIVE-10299.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=97)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query13]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query15]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query17]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query18]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query19]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query1] (batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query22]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query25]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query29]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query30]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query31]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query33]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query38]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query46]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query48]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query54]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query56]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query58]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query60]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query64]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query65]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query66]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query67]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query68]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query6] (batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query72]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query75]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query80]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query81]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query84]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query85]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query87]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query90]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query91]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query93]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94]
(batchId=230)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query95]
(batchId=230)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4603/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4603/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4603/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing
[jira] [Updated] (HIVE-16267) Enable bootstrap function metadata to be loaded in repl load
[ https://issues.apache.org/jira/browse/HIVE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-16267: --- Remaining Estimate: 72h Original Estimate: 72h > Enable bootstrap function metadata to be loaded in repl load > > > Key: HIVE-16267 > URL: https://issues.apache.org/jira/browse/HIVE-16267 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16267.1.patch > > Original Estimate: 72h > Remaining Estimate: 72h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HIVE-16267) Enable bootstrap function metadata to be loaded in repl load
[ https://issues.apache.org/jira/browse/HIVE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-16267 started by anishek. -- > Enable bootstrap function metadata to be loaded in repl load > > > Key: HIVE-16267 > URL: https://issues.apache.org/jira/browse/HIVE-16267 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16267.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16267) Enable bootstrap function metadata to be loaded in repl load
[ https://issues.apache.org/jira/browse/HIVE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-16267: --- Status: Open (was: Patch Available) reworking on a few things before providing the patch again > Enable bootstrap function metadata to be loaded in repl load > > > Key: HIVE-16267 > URL: https://issues.apache.org/jira/browse/HIVE-16267 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16267.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[
https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960562#comment-15960562
]
Hive QA commented on HIVE-11609:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862366/HIVE-11609.7.patch.txt
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3]
(batchId=90)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4602/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4602/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4602/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862366 - PreCommit-HIVE-Build
> Capability to add a filter to hbase scan via composite key doesn't work
> ---
>
> Key: HIVE-11609
> URL: https://issues.apache.org/jira/browse/HIVE-11609
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-11609.1.patch.txt, HIVE-11609.2.patch.txt,
> HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt, HIVE-11609.5.patch,
> HIVE-11609.6.patch.txt, HIVE-11609.7.patch.txt
>
>
> It seems like the capability to add filter to an hbase scan which was added
> as part of HIVE-6411 doesn't work. This is primarily because in the
> HiveHBaseInputFormat, the filter is added in the getsplits instead of
> getrecordreader. This works fine for start and stop keys but not for filter
> because a filter is respected only when an actual scan is performed. This is
> also related to the initial refactoring that was done as part of HIVE-3420.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16146: -- Attachment: HIVE-16146.05.patch Added extra configuration variables for using the truncated log files, and for not removing them after the query, and the session is closed. This will help the other unit tests > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16402) Upgrade to Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HIVE-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960510#comment-15960510
]
Hive QA commented on HIVE-16402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862347/HIVE-16402.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4601/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4601/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4601/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-07 08:57:59.941
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4601/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 08:57:59.944
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a01a6a3 HIVE-15616 : Improve contents of qfile test output
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a01a6a3 HIVE-15616 : Improve contents of qfile test output
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-07 08:58:01.193
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file pom.xml
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-hcatalog: Could not resolve
dependencies for project
org.apache.hive.hcatalog:hive-hcatalog:pom:3.0.0-SNAPSHOT: Failed to collect
dependencies for [org.mockito:mockito-all:jar:1.9.5 (test),
org.apache.hadoop:hadoop-common:jar:2.8.0 (test),
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.0 (test),
org.apache.pig:pig:jar:h2:0.16.0 (test), org.slf4j:slf4j-api:jar:1.7.10
(compile), org.skyscreamer:jsonassert:jar:1.4.0 (test)]: Failed to read
artifact descriptor for org.apache.hadoop:hadoop-common:jar:2.8.0: Could not
find artifact org.apache.hadoop:hadoop-project:pom:2.8.0 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-hcatalog
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862347 - PreCommit-HIVE-Build
> Upgrade to Hadoop 2.8.0
> ---
>
> Key: HIVE-16402
> URL: https://issues.apache.org/jira/browse/HIVE-16402
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16402.1.patch
>
>
> Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes
> for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html
> It has a number of useful features, improvements for S3 support, ADLS
> support, etc. along with a bunch of other fixes. This should also help us on
> our way to upgrading to Hadoop 3.x (HIVE-15016).
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-12636) Ensure that all queries (with DbTxnManager) run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960509#comment-15960509
]
Hive QA commented on HIVE-12636:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862345/HIVE-12636.07.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10570 tests
executed
*Failed tests:*
{noformat}
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=214)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.ql.TestTxnCommands2.testValidTxnsBookkeeping
(batchId=266)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testValidTxnsBookkeeping
(batchId=276)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testValidTxnsBookkeeping
(batchId=273)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4600/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4600/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4600/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862345 - PreCommit-HIVE-Build
> Ensure that all queries (with DbTxnManager) run in a transaction
>
>
> Key: HIVE-12636
> URL: https://issues.apache.org/jira/browse/HIVE-12636
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-12636.01.patch, HIVE-12636.02.patch,
> HIVE-12636.03.patch, HIVE-12636.04.patch, HIVE-12636.05.patch,
> HIVE-12636.06.patch, HIVE-12636.07.patch
>
>
> Assuming Hive is using DbTxnManager
> Currently (as of this writing only auto commit mode is supported), only
> queries that write to an Acid table start a transaction.
> Read-only queries don't open a txn but still acquire locks.
> This makes internal structures confusing/odd.
> The are constantly 2 code paths to deal with which is inconvenient and error
> prone.
> Also, a txn id is convenient "handle" for all locks/resources within a txn.
> Doing thing would mean the client no longer needs to track locks that it
> acquired. This enables further improvements to metastore side of Acid.
> # add metastore call to openTxn() and acquireLocks() in a single call. this
> it to make sure perf doesn't degrade for read-only query. (Would also be
> useful for auto commit write queries)
> # Should RO queries generate txn ids from the same sequence? (they could for
> example use negative values of a different sequence). Txnid is part of the
> delta/base file name. Currently it's 7 digits. If we use the same sequence,
> we'll exceed 7 digits faster. (possible upgrade issue). On the other hand
> there is value in being able to pick txn id and commit timestamp out of the
> same logical sequence.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Status: Patch Available (was: Open)
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: HIVE-16406.1.patch
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.1.patch, HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16406) Remove unwanted interning when creating PartitionDesc
[
https://issues.apache.org/jira/browse/HIVE-16406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16406:
Attachment: HIVE-16406.profiler.png
> Remove unwanted interning when creating PartitionDesc
> -
>
> Key: HIVE-16406
> URL: https://issues.apache.org/jira/browse/HIVE-16406
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16406.profiler.png
>
>
> {{PartitionDesc::getTableDesc}} interns all table description properties by
> default. But the table description properties are already interned and need
> not be interned again.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16386) Add debug logging to describe why runtime filtering semijoins are removed
[
https://issues.apache.org/jira/browse/HIVE-16386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960442#comment-15960442
]
Hive QA commented on HIVE-16386:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862343/HIVE-16386.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4599/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4599/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4599/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862343 - PreCommit-HIVE-Build
> Add debug logging to describe why runtime filtering semijoins are removed
> -
>
> Key: HIVE-16386
> URL: https://issues.apache.org/jira/browse/HIVE-16386
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-16386.1.patch, HIVE-16386.2.patch
>
>
> Add a few logging statements to detail the reason why semijoin optimizations
> are being removed, which can help during debugging.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16380) removing global test dependency of jsonassert
[ https://issues.apache.org/jira/browse/HIVE-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960417#comment-15960417 ] Thejas M Nair commented on HIVE-16380: -- +1 cc [~vgumashta] > removing global test dependency of jsonassert > - > > Key: HIVE-16380 > URL: https://issues.apache.org/jira/browse/HIVE-16380 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16380.1.patch, HIVE-16380.2.patch > > > as part of commit done for HIVE-16219, there seems to additional changes in > the root level pom.xml, they should not be required. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16380) removing global test dependency of jsonassert
[ https://issues.apache.org/jira/browse/HIVE-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960411#comment-15960411 ] anishek commented on HIVE-16380: [~vgumashta]/ [~thejas] /[~sushanth] please review. > removing global test dependency of jsonassert > - > > Key: HIVE-16380 > URL: https://issues.apache.org/jira/browse/HIVE-16380 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16380.1.patch, HIVE-16380.2.patch > > > as part of commit done for HIVE-16219, there seems to additional changes in > the root level pom.xml, they should not be required. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16383) Switch to HikariCP as default connection pooling
[ https://issues.apache.org/jira/browse/HIVE-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960395#comment-15960395 ] Thejas M Nair commented on HIVE-16383: -- +1 Please verify failed tests if they are known flaky issues. > Switch to HikariCP as default connection pooling > > > Key: HIVE-16383 > URL: https://issues.apache.org/jira/browse/HIVE-16383 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16383.1.patch, HIVE-16383.2.patch > > > Since 3.0 is planned to move to JDK8, we can now switch to HikariCP as > default connection pooling for DN because of its improved performance over > others. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[ https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960385#comment-15960385 ] Peter Vary commented on HIVE-16345: --- Test failure not related: - HIVE-16387 - Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData Any more review comments? Thanks, Peter > BeeLineDriver should be able to run qtest files which are using default > database tables > --- > > Key: HIVE-16345 > URL: https://issues.apache.org/jira/browse/HIVE-16345 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16345.2.patch, HIVE-16345.3.patch, HIVE-16345.patch > > > It would be good to be able to run the default clientpositive tests. > Currently we can not do that, since we start with a specific database. We > should filter the query input and replace the table references -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16392) Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance logic
[
https://issues.apache.org/jira/browse/HIVE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960382#comment-15960382
]
Hive QA commented on HIVE-16392:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862341/HIVE-16392.2.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=234)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite
(batchId=191)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4598/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4598/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4598/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862341 - PreCommit-HIVE-Build
> Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance
> logic
>
>
> Key: HIVE-16392
> URL: https://issues.apache.org/jira/browse/HIVE-16392
> Project: Hive
> Issue Type: Task
> Components: Security
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Labels: backwards-incompatible
> Attachments: HIVE-16392.1.patch, HIVE-16392.2.patch
>
>
> As discussed in HIVE-16346 we should remove the config
> {{hive.warehouse.subdir.inherit.perms}} and all the permissions inheritance
> logic.
> This feature is no longer needed in Hive as the traditional permission model
> has largely been replaced by external security systems such as Ranger and
> Sentry.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16334) Query lock contains the query string, which can cause OOM on ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960376#comment-15960376 ] Peter Vary commented on HIVE-16334: --- The failed tests are not related: - TestBeeLineDriver error will be solved with HIVE-16345 - HIVE-16405 - Flaky Test: org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData - HIVE-15165 - Flaky test: TestSparkNegativeCliDriver I think the patch is ready to commit. Thanks, Peter > Query lock contains the query string, which can cause OOM on ZooKeeper > -- > > Key: HIVE-16334 > URL: https://issues.apache.org/jira/browse/HIVE-16334 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16334.2.patch, HIVE-16334.3.patch, > HIVE-16334.4.patch, HIVE-16334.patch > > > When there are big number of partitions in a query this will result in a huge > number of locks on ZooKeeper. Since the query object contains the whole query > string this might cause serious memory pressure on the ZooKeeper services. > It would be good to have the possibility to truncate the query strings that > are written into the locks -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[
https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960372#comment-15960372
]
Gopal V commented on HIVE-16285:
Will test this tomorrow - but as a quick note, it is an XSRF anti-pattern to
use a GET URL to change a system config.
{code}
+ * Set root logger to INFO:
+ * http://:/conflog?logger=INFO
{code}
Adding a POST for write & GET for read would be good.
> Servlet for dynamically configuring log levels
> --
>
> Key: HIVE-16285
> URL: https://issues.apache.org/jira/browse/HIVE-16285
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch,
> HIVE-16285.3.patch
>
>
> Many long running services like HS2, LLAP etc. will benefit from having an
> endpoint to dynamically change log levels for various loggers. This will help
> greatly with debuggability without requiring a restart of the service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960346#comment-15960346 ] Siddharth Seth commented on HIVE-16285: --- Looks good. Will this work with the PerfLogger as well? Would be useful to run some queries to see if and how this impacts performance. [~gopalv], [~rajesh.balamohan] - do you see performance concerns with this? > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch, > HIVE-16285.3.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16334) Query lock contains the query string, which can cause OOM on ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960339#comment-15960339
]
Hive QA commented on HIVE-16334:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862323/HIVE-16334.4.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10576 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=237)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4597/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4597/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4597/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862323 - PreCommit-HIVE-Build
> Query lock contains the query string, which can cause OOM on ZooKeeper
> --
>
> Key: HIVE-16334
> URL: https://issues.apache.org/jira/browse/HIVE-16334
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16334.2.patch, HIVE-16334.3.patch,
> HIVE-16334.4.patch, HIVE-16334.patch
>
>
> When there are big number of partitions in a query this will result in a huge
> number of locks on ZooKeeper. Since the query object contains the whole query
> string this might cause serious memory pressure on the ZooKeeper services.
> It would be good to have the possibility to truncate the query strings that
> are written into the locks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16388) LLAP: Log rotation for daemon, history and gc files
[ https://issues.apache.org/jira/browse/HIVE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960328#comment-15960328 ] Siddharth Seth commented on HIVE-16388: --- Is it possible to use a custom log level instead of WARN for the daemon startup message? WARN can be confused for an error. (Nice to have more than anything else). > LLAP: Log rotation for daemon, history and gc files > --- > > Key: HIVE-16388 > URL: https://issues.apache.org/jira/browse/HIVE-16388 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Attachments: HIVE-16388.1.patch > > > GC logs need to be rotated by date. > LLAP daemon history logs as well > Ideally, the daemon.out file needs the same > Need to be able to download relevant logfiles for a time window. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16390) LLAP IO should take job config into account; also LLAP config should load defaults
[ https://issues.apache.org/jira/browse/HIVE-16390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960327#comment-15960327 ] Siddharth Seth commented on HIVE-16390: --- Patch looks good. One question is what impact does this have when there may be two different queries trying to access the same data. Earlier the read would be using the daemon configuration. Now it's two configuration instance with potentially differing values. Likely does not make a difference. The patch accounts for ZCR. Anything else that should be looked at? > LLAP IO should take job config into account; also LLAP config should load > defaults > -- > > Key: HIVE-16390 > URL: https://issues.apache.org/jira/browse/HIVE-16390 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16390.patch > > > Ensure the config is used consistently with task-based execution by default; > the exceptions should be specific (settings we don't want overridden, like > zero-copy). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
