[jira] [Commented] (HIVE-13913) LLAP: introduce backpressure to recordreader
[
https://issues.apache.org/jira/browse/HIVE-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337618#comment-15337618
]
Hive QA commented on HIVE-13913:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811254/HIVE-13913.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/160/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/160/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-160/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811254 - PreCommit-HIVE-MASTER-Build
> LLAP: introduce backpressure to recordreader
>
>
> Key: HIVE-13913
> URL: https://issues.apache.org/jira/browse/HIVE-13913
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13913.01.patch, HIVE-13913.02.patch,
> HIVE-13913.03.patch, HIVE-13913.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14056) Golden file updates for few tests
[ https://issues.apache.org/jira/browse/HIVE-14056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14056: Status: Patch Available (was: Open) > Golden file updates for few tests > - > > Key: HIVE-14056 > URL: https://issues.apache.org/jira/browse/HIVE-14056 > Project: Hive > Issue Type: Task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14056.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14056) Golden file updates for few tests
[ https://issues.apache.org/jira/browse/HIVE-14056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14056: Attachment: HIVE-14056.patch > Golden file updates for few tests > - > > Key: HIVE-14056 > URL: https://issues.apache.org/jira/browse/HIVE-14056 > Project: Hive > Issue Type: Task > Components: Tests >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14056.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13350) Support Alter commands for Rely/NoRely novalidate for PK/FK constraints
[ https://issues.apache.org/jira/browse/HIVE-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337581#comment-15337581 ] Lefty Leverenz commented on HIVE-13350: --- Doc note: See the PDF doc attached to HIVE-13076. * https://issues.apache.org/jira/secure/attachment/12803522/AddingPKFKconstraints.pdf > Support Alter commands for Rely/NoRely novalidate for PK/FK constraints > > > Key: HIVE-13350 > URL: https://issues.apache.org/jira/browse/HIVE-13350 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13350.1.patch, HIVE-13350.2.patch, > HIVE-13350.final.patch > > > Support commands like : > ALTER TABLE table2 ADD CONSTRAINT pkt2 primary key (a) disable novalidate; > ALTER TABLE table3 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES table2(a) > DISABLE NOVALIDATE RELY; > ALTER TABLE table6 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES table1(a) > DISABLE NOVALIDATE; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337577#comment-15337577 ] Lefty Leverenz commented on HIVE-13351: --- See the PDF doc attached to HIVE-13076: * https://issues.apache.org/jira/secure/attachment/12803522/AddingPKFKconstraints.pdf > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337576#comment-15337576
]
Lefty Leverenz commented on HIVE-13290:
---
See the PDF doc attached to HIVE-13076:
*
https://issues.apache.org/jira/secure/attachment/12803522/AddingPKFKconstraints.pdf
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch,
> HIVE-13290.6.patch, HIVE-13290.7.patch, HIVE-13290.8.patch,
> HIVE-13290.final.patch, test-results.txt
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id) DISABLE NOVALIDATE,
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id) DISABLE NOVALIDATE
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id) DISABLE NOVALIDATE RELY
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table. The
> RELY/NORELY keyword is also optional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13441) LLAPIF: security and signed fragments
[ https://issues.apache.org/jira/browse/HIVE-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337565#comment-15337565 ] Lefty Leverenz commented on HIVE-13441: --- Does this need general documentation, in addition to the subtasks that have TODOC labels for new configuration parameters? * TODOC2.1: HIVE-13444 (LLAP: add HMAC signatures to LLAP; verify them on LLAP side) * TODOC2.2: HIVE-13675 (LLAP: add HMAC signatures to LLAPIF splits) * TODOC2.2: HIVE-13827 (LLAPIF: authentication on the output channel) > LLAPIF: security and signed fragments > - > > Key: HIVE-13441 > URL: https://issues.apache.org/jira/browse/HIVE-13441 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: llap > Fix For: 2.2.0 > > > Allows external clients to get securely signed splits from HS2, and submit > them to LLAP without running as a privileged user; LLAP will verify the > splits before running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13392) disable speculative execution for ACID Compactor
[
https://issues.apache.org/jira/browse/HIVE-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337551#comment-15337551
]
Lefty Leverenz commented on HIVE-13392:
---
Okay, thanks Eugene.
> disable speculative execution for ACID Compactor
>
>
> Key: HIVE-13392
> URL: https://issues.apache.org/jira/browse/HIVE-13392
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.2.0
>
> Attachments: HIVE-13392.2.patch, HIVE-13392.3.patch,
> HIVE-13392.4.patch, HIVE-13392.patch
>
>
> https://developer.yahoo.com/hadoop/tutorial/module4.html
> Speculative execution is enabled by default. You can disable speculative
> execution for the mappers and reducers by setting the
> mapred.map.tasks.speculative.execution and
> mapred.reduce.tasks.speculative.execution JobConf options to false,
> respectively.
> CompactorMR is currently not set up to handle speculative execution and may
> lead to something like
> {code}
> 2016-02-08 22:56:38,256 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
> Failed to CREATE_FILE
> /apps/hive/warehouse/service_logs_v2/ds=2016-01-20/_tmp_6cf08b9f-c2e2-4182-bc81-e032801b147f/base_13858600/bucket_4
> for DFSClient_attempt_1454628390210_27756_m_01_1_131224698_1 on
> 172.18.129.12 because this file lease is currently owned by
> DFSClient_attempt_1454628390210_27756_m_01_0_-2027182532_1 on
> 172.18.129.18
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2451)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> {code}
> Short term: disable speculative execution for this job
> Longer term perhaps make each task write to dir with UUID...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13827) LLAPIF: authentication on the output channel
[ https://issues.apache.org/jira/browse/HIVE-13827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337548#comment-15337548 ] Lefty Leverenz commented on HIVE-13827: --- Doc note: This adds *hive.llap.daemon.output.stream.timeout* to HiveConf.java, so it will need to be documented in the LLAP section of Configuration Properties for the 2.2.0 release. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC2.2 label. > LLAPIF: authentication on the output channel > > > Key: HIVE-13827 > URL: https://issues.apache.org/jira/browse/HIVE-13827 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-13827.01.patch, HIVE-13827.patch > > > The current thinking is that we'd send the token. There's no protocol on the > channel right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13827) LLAPIF: authentication on the output channel
[ https://issues.apache.org/jira/browse/HIVE-13827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13827: -- Labels: TODOC2.2 (was: ) > LLAPIF: authentication on the output channel > > > Key: HIVE-13827 > URL: https://issues.apache.org/jira/browse/HIVE-13827 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-13827.01.patch, HIVE-13827.patch > > > The current thinking is that we'd send the token. There's no protocol on the > channel right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[
https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337545#comment-15337545
]
Hive QA commented on HIVE-13258:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811263/HIVE-13258.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 10238 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_uncompressed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap_counters1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/159/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/159/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-159/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811263 - PreCommit-HIVE-MASTER-Build
> LLAP: Add hdfs bytes read and spilled bytes to tez print summary
>
>
> Key: HIVE-13258
> URL: https://issues.apache.org/jira/browse/HIVE-13258
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13258.1.patch, HIVE-13258.1.patch,
> HIVE-13258.2.patch, HIVE-13258.3.patch, HIVE-13258.4.patch,
> HIVE-13258.5.patch, llap-fs-counters-full-cache-hit.png, llap-fs-counters.png
>
>
> When printing counters to console it will be useful to print hdfs bytes read
> and spilled bytes which will help with debugging issues faster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14018) Make IN clause row selectivity estimation customizable
[ https://issues.apache.org/jira/browse/HIVE-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337522#comment-15337522 ] Lefty Leverenz commented on HIVE-14018: --- Doc note: This adds *hive.stats.filter.in.factor* to HiveConf.java, so it will need to be documented for releases 2.1.1 and 2.2.0. * [Configuration Properties -- Statistics | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Statistics] Added TODOC2.1.1 and TODOC2.2 labels. > Make IN clause row selectivity estimation customizable > -- > > Key: HIVE-14018 > URL: https://issues.apache.org/jira/browse/HIVE-14018 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Labels: TODOC2.1.1, TODOC2.2 > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14018.1.patch, HIVE-14018.patch > > > After HIVE-13287 went in, we calculate IN clause estimates natively (instead > of just dividing incoming number of rows by 2). However, as the distribution > of values of the columns is considered uniform, we might end up heavily > underestimating/overestimating the resulting number of rows. > This issue is to add a factor that multiplies the IN clause estimation so we > can alleviate this problem. The solution is not very elegant, but it is the > best we can do until we have histograms to improve our estimate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14018) Make IN clause row selectivity estimation customizable
[ https://issues.apache.org/jira/browse/HIVE-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14018: -- Labels: TODOC2.1.1 TODOC2.2 (was: ) > Make IN clause row selectivity estimation customizable > -- > > Key: HIVE-14018 > URL: https://issues.apache.org/jira/browse/HIVE-14018 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Labels: TODOC2.1.1, TODOC2.2 > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14018.1.patch, HIVE-14018.patch > > > After HIVE-13287 went in, we calculate IN clause estimates natively (instead > of just dividing incoming number of rows by 2). However, as the distribution > of values of the columns is considered uniform, we might end up heavily > underestimating/overestimating the resulting number of rows. > This issue is to add a factor that multiplies the IN clause estimation so we > can alleviate this problem. The solution is not very elegant, but it is the > best we can do until we have histograms to improve our estimate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Attachment: HIVE-14000.04.patch > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch, > HIVE-14000.04.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Status: Patch Available (was: In Progress) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch, > HIVE-14000.04.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14003) queries running against llap hang at times - preemption issues
[
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337469#comment-15337469
]
Hive QA commented on HIVE-14003:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811266/HIVE-14003.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10220 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/158/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/158/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-158/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811266 - PreCommit-HIVE-MASTER-Build
> queries running against llap hang at times - preemption issues
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
> Attachments: HIVE-14003.01.patch, HIVE-14003.02.patch
>
>
> The preemption logic in the Hive processor needs some more work. There are
> definitely windows where the abort flag is completely dropped within the Hive
> processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13985: - Attachment: HIVE-13985.5.patch > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch, HIVE-13985.5.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14031) cleanup metadataReader in OrcEncodedDataReader
[
https://issues.apache.org/jira/browse/HIVE-14031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337381#comment-15337381
]
Hive QA commented on HIVE-14031:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811001/HIVE-14031.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/157/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/157/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-157/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811001 - PreCommit-HIVE-MASTER-Build
> cleanup metadataReader in OrcEncodedDataReader
> --
>
> Key: HIVE-14031
> URL: https://issues.apache.org/jira/browse/HIVE-14031
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14031.1.patch
>
>
> MetadataReader should be closed in OrcEncodedDataReader as a part of
> cleanupReaders.
> \cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14055) directSql - getting the number of partitions is broken
[ https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14055: Status: Patch Available (was: Open) > directSql - getting the number of partitions is broken > -- > > Key: HIVE-14055 > URL: https://issues.apache.org/jira/browse/HIVE-14055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14055.patch > > > Noticed while looking at something else. If the filter cannot be pushed down > it just returns 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14055) directSql - getting the number of partitions is broken
[ https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14055: Attachment: HIVE-14055.patch [~ashutoshc] [~sushanth] can you take a look? Thanks > directSql - getting the number of partitions is broken > -- > > Key: HIVE-14055 > URL: https://issues.apache.org/jira/browse/HIVE-14055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14055.patch > > > Noticed while looking at something else. If the filter cannot be pushed down > it just returns 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14055) directSql - getting the number of partitions is broken
[ https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14055: Attachment: (was: HIVE-14055.patch) > directSql - getting the number of partitions is broken > -- > > Key: HIVE-14055 > URL: https://issues.apache.org/jira/browse/HIVE-14055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Noticed while looking at something else. If the filter cannot be pushed down > it just returns 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337363#comment-15337363 ] Ashutosh Chauhan commented on HIVE-14054: - and TestHiveMetaStoreChecker failures too. > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14055) directSql - getting the number of partitions is broken
[ https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14055: Attachment: HIVE-14055.patch > directSql - getting the number of partitions is broken > -- > > Key: HIVE-14055 > URL: https://issues.apache.org/jira/browse/HIVE-14055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14055.patch > > > Noticed while looking at something else. If the filter cannot be pushed down > it just returns 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337362#comment-15337362 ] Ashutosh Chauhan commented on HIVE-14054: - Also testCliDriver_repair failure is also likely related. > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14055) directSql - getting the number of partitions is broken
[ https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14055: Description: Noticed while looking at something else. If the filter cannot be pushed down it just returns 0 (was: Noticed while looking at something else) > directSql - getting the number of partitions is broken > -- > > Key: HIVE-14055 > URL: https://issues.apache.org/jira/browse/HIVE-14055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Noticed while looking at something else. If the filter cannot be pushed down > it just returns 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14054) TestHiveMetaStoreChecker fails on master
[ https://issues.apache.org/jira/browse/HIVE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337358#comment-15337358 ] Ashutosh Chauhan commented on HIVE-14054: - [~pxiong] Looks like it started failing because of HIVE-13984 > TestHiveMetaStoreChecker fails on master > - > > Key: HIVE-14054 > URL: https://issues.apache.org/jira/browse/HIVE-14054 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[ https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337352#comment-15337352 ] Ashutosh Chauhan commented on HIVE-14021: - Can you create a RB? Is also diff from Calcite's toCnf() is in terms of overflow exception() ? > When converting to CNF, fail if the expression exceeds a threshold > -- > > Key: HIVE-14021 > URL: https://issues.apache.org/jira/browse/HIVE-14021 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14021.patch > > > When converting to conjunctive normal form (CNF), fail if the expression > exceeds a threshold. CNF can explode exponentially in the size of the input > expression, but rarely does so in practice. Add a maxNodeCount parameter to > RexUtil.toCnf and throw or return null if it is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12977) Pass credentials in the current UGI while creating Tez session
[
https://issues.apache.org/jira/browse/HIVE-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337351#comment-15337351
]
Thejas M Nair commented on HIVE-12977:
--
+1 pending tests
> Pass credentials in the current UGI while creating Tez session
> --
>
> Key: HIVE-12977
> URL: https://issues.apache.org/jira/browse/HIVE-12977
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Vinoth Sathappan
>Assignee: Vinoth Sathappan
> Attachments: HIVE-12977.1.patch, HIVE-12977.1.patch,
> HIVE-12977.2.patch, HIVE-12977.3.patch, HIVE-12977.4.patch
>
>
> The credentials present in the current UGI i.e.
> UserGroupInformation.getCurrentUser().getCredentials() isn't passed to the
> Tez session. It is instantiated with null credentials.
> session = TezClient.create("HIVE-" + sessionId, tezConfig, true,
> commonLocalResources, null);
> In this case, tokens added using hive execution hooks, aren't available to
> Tez even if they are available in memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14031) cleanup metadataReader in OrcEncodedDataReader
[ https://issues.apache.org/jira/browse/HIVE-14031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14031: Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Pushed to master & branch-2.1 Thanks [~rajesh.balamohan] > cleanup metadataReader in OrcEncodedDataReader > -- > > Key: HIVE-14031 > URL: https://issues.apache.org/jira/browse/HIVE-14031 > Project: Hive > Issue Type: Bug > Components: ORC >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14031.1.patch > > > MetadataReader should be closed in OrcEncodedDataReader as a part of > cleanupReaders. > \cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14031) cleanup metadataReader in OrcEncodedDataReader
[ https://issues.apache.org/jira/browse/HIVE-14031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14031: Component/s: ORC > cleanup metadataReader in OrcEncodedDataReader > -- > > Key: HIVE-14031 > URL: https://issues.apache.org/jira/browse/HIVE-14031 > Project: Hive > Issue Type: Bug > Components: ORC >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14031.1.patch > > > MetadataReader should be closed in OrcEncodedDataReader as a part of > cleanupReaders. > \cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13350) Support Alter commands for Rely/NoRely novalidate for PK/FK constraints
[ https://issues.apache.org/jira/browse/HIVE-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337317#comment-15337317 ] Sergey Shelukhin commented on HIVE-13350: - This and other patches left a bunch of blank methods in HBaseStore. When will those be implemented? How did this get committed with a bunch of auto-generated Eclipse comments? > Support Alter commands for Rely/NoRely novalidate for PK/FK constraints > > > Key: HIVE-13350 > URL: https://issues.apache.org/jira/browse/HIVE-13350 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13350.1.patch, HIVE-13350.2.patch, > HIVE-13350.final.patch > > > Support commands like : > ALTER TABLE table2 ADD CONSTRAINT pkt2 primary key (a) disable novalidate; > ALTER TABLE table3 ADD CONSTRAINT fk1 FOREIGN KEY ( x ) REFERENCES table2(a) > DISABLE NOVALIDATE RELY; > ALTER TABLE table6 ADD CONSTRAINT fk4 FOREIGN KEY ( y ) REFERENCES table1(a) > DISABLE NOVALIDATE; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337300#comment-15337300 ] Thejas M Nair commented on HIVE-11527: -- Thanks for working on this [~tasanuma0829]! I have added some comments in review board. This would break backward compatibility with old clients (old jdbc drivers). HS2 supports having older drivers talking to new servers. The bypass should be disabled if the client is using a thrift api version older than this api version, to support this backward compatibility. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-13872:
---
Assignee: Matt McCline
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.WIP.patch
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>customer_demographics.cd_demo_sk = ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'U'
> ))
> ;
> {code}
> {code}
> Map 3
> Map Operator Tree:
> TableScan
> alias: customer_demographics
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> value expressions: cd_demo_sk (type: int),
> cd_marital_status (type: string)
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14003) queries running against llap hang at times - preemption issues
[ https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337250#comment-15337250 ] Sergey Shelukhin commented on HIVE-14003: - +1 > queries running against llap hang at times - preemption issues > -- > > Key: HIVE-14003 > URL: https://issues.apache.org/jira/browse/HIVE-14003 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Siddharth Seth > Attachments: HIVE-14003.01.patch, HIVE-14003.02.patch > > > The preemption logic in the Hive processor needs some more work. There are > definitely windows where the abort flag is completely dropped within the Hive > processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12052) automatically populate file metadata to HBase metastore based on config or table properties
[ https://issues.apache.org/jira/browse/HIVE-12052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337197#comment-15337197 ] Sergey Shelukhin commented on HIVE-12052: - . > automatically populate file metadata to HBase metastore based on config or > table properties > --- > > Key: HIVE-12052 > URL: https://issues.apache.org/jira/browse/HIVE-12052 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > As discussed in HIVE-11500 > Should use a table property similar to auto.purge. > Then, when this setting is set, partitions are added (convertToMPart is a > good source to find all the paths for that), after compactions, after > load/non-ACID insert, and periodically (configurable), the storage locations > should be scanned for new files and cache updated accordingly. All the > updates should probably be in the background thread and taken from queue > (high pri from most ops, low pri from enabling the property and from periodic > updates) to avoid high load on HDFS from metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[
https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337180#comment-15337180
]
Hive QA commented on HIVE-13985:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811333/HIVE-13985.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 181 failed/errored test(s), 10246 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_windowing
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
[jira] [Commented] (HIVE-13932) Hive SMB Map Join with small set of LIMIT failed with NPE
[ https://issues.apache.org/jira/browse/HIVE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337168#comment-15337168 ] Ashutosh Chauhan commented on HIVE-13932: - Shall this be committed to branch-2.1 as well ? > Hive SMB Map Join with small set of LIMIT failed with NPE > - > > Key: HIVE-13932 > URL: https://issues.apache.org/jira/browse/HIVE-13932 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.2.0 > > Attachments: HIVE-13932.1.patch > > > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c limit 1; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14053) Hive should report that primary keys can't be null.
[
https://issues.apache.org/jira/browse/HIVE-14053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carter Shanklin updated HIVE-14053:
---
Priority: Minor (was: Major)
> Hive should report that primary keys can't be null.
> ---
>
> Key: HIVE-14053
> URL: https://issues.apache.org/jira/browse/HIVE-14053
> Project: Hive
> Issue Type: Bug
>Reporter: Carter Shanklin
>Priority: Minor
>
> HIVE-13076 introduces "rely novalidate" primary and foreign keys to Hive.
> With the right driver in place, tools like Tableau can do join elimination
> and queries can run much faster.
> Some gaps remain, currently getAttributes() in HiveDatabaseMetaData doesn't
> work quite right for keys. In particular, primary keys by definition are not
> null and the metadata should reflect this for improved join elimination.
> In this example that uses the TPC-H schema and its constraints, we sum
> l_extendedprice and group by l_shipmode. This query should not use more than
> just the lineitem table.
> With all the constraints in place, Tableau generates this query:
> {code}
> SELECT `lineitem`.`l_shipmode` AS `l_shipmode`,
> SUM(`lineitem`.`l_extendedprice`) AS `sum_l_extendedprice_ok`
> FROM `tpch_bin_flat_orc_2`.`lineitem` `lineitem`
> JOIN `tpch_bin_flat_orc_2`.`orders` `orders` ON (`lineitem`.`l_orderkey` =
> `orders`.`o_orderkey`)
> JOIN `tpch_bin_flat_orc_2`.`customer` `customer` ON (`orders`.`o_custkey` =
> `customer`.`c_custkey`)
> JOIN `tpch_bin_flat_orc_2`.`nation` `nation` ON (`customer`.`c_nationkey` =
> `nation`.`n_nationkey`)
> WHERE NOT (`lineitem`.`l_partkey` IS NULL)) AND (NOT
> (`lineitem`.`l_suppkey` IS NULL))) AND ((NOT (`lineitem`.`l_partkey` IS
> NULL)) AND (NOT (`lineitem`.`l_suppkey` IS NULL AND (NOT
> (`nation`.`n_regionkey` IS NULL)))
> {code}
> Since these are the primary keys the denormalization and the where condition
> is unnecessary and this sort of query can be a lot faster by just accessing
> the lineitem table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13965) Empty resultset run into Exception when using Thrift Binary Serde
[
https://issues.apache.org/jira/browse/HIVE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyang Zhao updated HIVE-13965:
---
Status: Patch Available (was: Open)
> Empty resultset run into Exception when using Thrift Binary Serde
> -
>
> Key: HIVE-13965
> URL: https://issues.apache.org/jira/browse/HIVE-13965
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
> Attachments: HIVE-13965.1.patch.txt
>
>
> This error can be reproduced by enabling thrift binary serde, using beeline
> connect to hiveserver2 and executing the following commands:
> >create table test3(num1 int);
> >create table test4(num1 int);
> >insert into test3 values(1);
> >insert into test4 values(2);
> >select * from test3 join test4 on test3.num1=test4.num1;
> The result should be empty, but it gives an exception:
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1029)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:641)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:195)
> ... 8 more
> This error is caused in FileSinkOperator.java.
> If the resultset is empty, function process() will not be called, so variable
> "fpaths" will not be set. When run into CloseOp(),
> if (conf.isHiveServerQuery() && HiveConf.getBoolVar(hconf,
> HiveConf.ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_SERIALIZE_IN_TASKS)
> &&
>
> serializer.getClass().getName().equalsIgnoreCase(ThriftJDBCBinarySerDe.class.getName()))
> {
> try {
>recordValue = serializer.serialize(null, inputObjInspectors[0]);
>rowOutWriters = fpaths.outWriters;
>rowOutWriters[0].write(recordValue);
> } catch (SerDeException | IOException e) {
>throw new HiveException(e);
> }
> }
> Here fpaths is null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337150#comment-15337150
]
Ashutosh Chauhan commented on HIVE-13901:
-
Following tests failed:
{code}
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition
{code}
Atleast few of these seems to be relevant.
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Status: In Progress (was: Patch Available) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337146#comment-15337146 ] Matt McCline commented on HIVE-14000: - Thank you for your review. > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14052) Cleanup of structures required when LLAP access from external clients completes
[ https://issues.apache.org/jira/browse/HIVE-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337143#comment-15337143 ] Jason Dere commented on HIVE-14052: --- Whoops, forgot to add RB link: https://reviews.apache.org/r/48886/ > Cleanup of structures required when LLAP access from external clients > completes > --- > > Key: HIVE-14052 > URL: https://issues.apache.org/jira/browse/HIVE-14052 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14052.1.patch > > > Per [~sseth]: There's no cleanup at the moment, and structures used in LLAP > to track a query will keep building up slowly over time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337141#comment-15337141 ] Sergey Shelukhin commented on HIVE-14000: - Sure... +1 > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14050) Hive attempts to 'chgrp' files on s3a://
[
https://issues.apache.org/jira/browse/HIVE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337139#comment-15337139
]
Ashutosh Chauhan commented on HIVE-14050:
-
{{set hive.warehouse.subdir.inherit.perms=false;}} for S3 as permission inherit
is of no use in S3.
> Hive attempts to 'chgrp' files on s3a://
>
>
> Key: HIVE-14050
> URL: https://issues.apache.org/jira/browse/HIVE-14050
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Sean Roberts
>Assignee: Chris Nauroth
> Labels: s3
>
> When inserting to a table on s3a://, Hive attempts to `chgrp` the files but
> files in s3a:// do not have group ownership.
> {code}
> hive> insert into INVENTORY select * from INVENTORY_Q1_2006;
> -chgrp: '' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> {code}
> Full output of the query here:
> {code}
> hive> insert into INVENTORY select * from INVENTORY_Q1_2006;
> -chgrp: '' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> Query ID = admin_20160617201151_5f953fbe-acde-4774-9ad7-06cffc76dd72
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1466165341299_0011)
>
> VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED
> KILLED
>
> Map 1 .. SUCCEEDED 1 100 0
> 0
>
> VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 8.71 s
>
> Loading data to table mydb.inventory
> -chgrp: '' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> Table mydb.inventory stats: [numFiles=12, numRows=6020352,
> totalSize=25250706, rawDataSize=96325632]
> OK
> Time taken: 19.123 seconds
> {code}
> The table:
> {code}
> CREATE TABLE IF NOT EXISTS inventory
>(
> MONTH_ID int,
> ITEM_ID int,
> BOH_QTY float,
> EOH_QTY float
>) row format delimited fields terminated by '|' escaped by '\\' stored as
> ORC
> LOCATION 's3a://mybucket/hive/warehouse/mydb.db/inventory'
> tblproperties ("orc.compress"="SNAPPY");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14052) Cleanup of structures required when LLAP access from external clients completes
[
https://issues.apache.org/jira/browse/HIVE-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337137#comment-15337137
]
Sergey Shelukhin commented on HIVE-14052:
-
RB might help
{noformat}
+CancellableCleanerCallable queryCleanerTask =
pendingQueryCleanerTasks.get(queryIdentifier);
+if (queryCleanerTask != null) {
+ boolean cancelled = queryCleanerTask.cancel();
+ if (!cancelled) {
+throw new IOException("Unable to cancel pending query cleanup for
" + queryIdentifier);
{noformat}
what it completes just before we try to cancel?
> Cleanup of structures required when LLAP access from external clients
> completes
> ---
>
> Key: HIVE-14052
> URL: https://issues.apache.org/jira/browse/HIVE-14052
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-14052.1.patch
>
>
> Per [~sseth]: There's no cleanup at the moment, and structures used in LLAP
> to track a query will keep building up slowly over time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13392) disable speculative execution for ACID Compactor
[
https://issues.apache.org/jira/browse/HIVE-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337134#comment-15337134
]
Eugene Koifman commented on HIVE-13392:
---
[~leftylev], no
> disable speculative execution for ACID Compactor
>
>
> Key: HIVE-13392
> URL: https://issues.apache.org/jira/browse/HIVE-13392
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.2.0
>
> Attachments: HIVE-13392.2.patch, HIVE-13392.3.patch,
> HIVE-13392.4.patch, HIVE-13392.patch
>
>
> https://developer.yahoo.com/hadoop/tutorial/module4.html
> Speculative execution is enabled by default. You can disable speculative
> execution for the mappers and reducers by setting the
> mapred.map.tasks.speculative.execution and
> mapred.reduce.tasks.speculative.execution JobConf options to false,
> respectively.
> CompactorMR is currently not set up to handle speculative execution and may
> lead to something like
> {code}
> 2016-02-08 22:56:38,256 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
> Failed to CREATE_FILE
> /apps/hive/warehouse/service_logs_v2/ds=2016-01-20/_tmp_6cf08b9f-c2e2-4182-bc81-e032801b147f/base_13858600/bucket_4
> for DFSClient_attempt_1454628390210_27756_m_01_1_131224698_1 on
> 172.18.129.12 because this file lease is currently owned by
> DFSClient_attempt_1454628390210_27756_m_01_0_-2027182532_1 on
> 172.18.129.18
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2451)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> {code}
> Short term: disable speculative execution for this job
> Longer term perhaps make each task write to dir with UUID...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337131#comment-15337131 ] Matt McCline commented on HIVE-14000: - [~sershe] Yes, I think the float case will be ok. > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13970) refactor LLAPIF splits - get rid of SubmitWorkInfo
[ https://issues.apache.org/jira/browse/HIVE-13970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13970: Assignee: (was: Sergey Shelukhin) > refactor LLAPIF splits - get rid of SubmitWorkInfo > -- > > Key: HIVE-13970 > URL: https://issues.apache.org/jira/browse/HIVE-13970 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > Attachments: HIVE-13970.01.patch, HIVE-13970.only.patch, > HIVE-13970.patch, HIVE-13970.patch > > > First we build the signable vertex spec, convert it into bytes (as we > should), and put it inside SubmitWorkInfo. Then we serialize that into byte[] > and put it into LlapInputSplit. Then we serialize that to return... We should > get rid of one of the steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13441) LLAPIF: security and signed fragments
[ https://issues.apache.org/jira/browse/HIVE-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-13441. - Resolution: Fixed Fix Version/s: 2.2.0 > LLAPIF: security and signed fragments > - > > Key: HIVE-13441 > URL: https://issues.apache.org/jira/browse/HIVE-13441 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: llap > Fix For: 2.2.0 > > > Allows external clients to get securely signed splits from HS2, and submit > them to LLAP without running as a privileged user; LLAP will verify the > splits before running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14052) Cleanup of structures required when LLAP access from external clients completes
[ https://issues.apache.org/jira/browse/HIVE-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14052: -- Status: Patch Available (was: Open) > Cleanup of structures required when LLAP access from external clients > completes > --- > > Key: HIVE-14052 > URL: https://issues.apache.org/jira/browse/HIVE-14052 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14052.1.patch > > > Per [~sseth]: There's no cleanup at the moment, and structures used in LLAP > to track a query will keep building up slowly over time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13970) refactor LLAPIF splits - get rid of SubmitWorkInfo
[ https://issues.apache.org/jira/browse/HIVE-13970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13970: Resolution: Won't Fix Status: Resolved (was: Patch Available) I looked at this a bit and it's probably not worth doing. I was going to change the FN to return 2 cols, first one being bytes for the shared part (one row) and for unique split info (N rows) and the second one being splitNum for splits, with a string or magic value for the shared part); with the structs being protobuf instead of writables; however, it falls under the other JIRA and is anyway probably wasted work in light of proper API work that is surely forthcoming ;) > refactor LLAPIF splits - get rid of SubmitWorkInfo > -- > > Key: HIVE-13970 > URL: https://issues.apache.org/jira/browse/HIVE-13970 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13970.01.patch, HIVE-13970.only.patch, > HIVE-13970.patch, HIVE-13970.patch > > > First we build the signable vertex spec, convert it into bytes (as we > should), and put it inside SubmitWorkInfo. Then we serialize that into byte[] > and put it into LlapInputSplit. Then we serialize that to return... We should > get rid of one of the steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14052) Cleanup of structures required when LLAP access from external clients completes
[ https://issues.apache.org/jira/browse/HIVE-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14052: -- Attachment: HIVE-14052.1.patch > Cleanup of structures required when LLAP access from external clients > completes > --- > > Key: HIVE-14052 > URL: https://issues.apache.org/jira/browse/HIVE-14052 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14052.1.patch > > > Per [~sseth]: There's no cleanup at the moment, and structures used in LLAP > to track a query will keep building up slowly over time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337115#comment-15337115
]
Szehon Ho commented on HIVE-13884:
--
+1 from my side, pending one last comment on RB, and also the other reviews
from Mohit.
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch, HIVE-13884.6.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: proposal.pdf > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: (was: proposal.pdf) > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7443) Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs
[ https://issues.apache.org/jira/browse/HIVE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337051#comment-15337051 ] Chaoyu Tang commented on HIVE-7443: --- LGTM, +1 pending on testing > Fix HiveConnection to communicate with Kerberized Hive JDBC server and > alternative JDKs > --- > > Key: HIVE-7443 > URL: https://issues.apache.org/jira/browse/HIVE-7443 > Project: Hive > Issue Type: Bug > Components: JDBC, Security >Affects Versions: 0.12.0, 0.13.1 > Environment: Kerberos > Run Hive server2 and client with IBM JDK7.1 >Reporter: Yu Gao >Assignee: Aihua Xu > Attachments: HIVE-7443.2.patch, HIVE-7443.patch > > > Hive Kerberos authentication has been enabled in my cluster. I ran kinit to > initialize the current login user's ticket cache successfully, and then tried > to use beeline to connect to Hive Server2, but failed. After I manually added > some logging to catch the failure exception, this is what I got that caused > the failure: > beeline> !connect > jdbc:hive2://:1/default;principal=hive/@REALM.COM > org.apache.hive.jdbc.HiveDriver > scan complete in 2ms > Connecting to > jdbc:hive2://:1/default;principal=hive/@REALM.COM > Enter password for > jdbc:hive2://:1/default;principal=hive/@REALM.COM: > 14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport > javax.security.sasl.SaslException: Failed to open client transport [Caused by > java.io.IOException: Could not instantiate SASL transport] > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78) > at > org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342) > at > org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200) > at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:178) > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:198) > at > org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) > at > org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186) > at org.apache.hive.beeline.Commands.connect(Commands.java:959) > at org.apache.hive.beeline.Commands.connect(Commands.java:880) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at > org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659) > at > org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.io.IOException: Could not instantiate SASL transport > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:177) > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:74) > ... 24 more > Caused by: javax.security.sasl.SaslException: Failure to initialize security > context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0 > major string: Invalid credentials > minor string: SubjectCredFinder: no JAAS Subject] > at > com.ibm.security.sasl.gsskerb.GssKrb5Client.(GssKrb5Client.java:131) > at > com.ibm.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:53) > at javax.security.sasl.Sasl.createSaslClient(Sasl.java:362) > at > org.apache.thrift.transport.TSaslClientTransport.(TSaslClientTransport.java:72) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:169) > ... 25 more > Caused by: org.ietf.jgss.GSSException, major code: 13, minor code
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337044#comment-15337044 ] Sergey Shelukhin commented on HIVE-13930: - [~xuefuz] [~spena] can you please comment? I am inclined to disable the main SparkCliDriver (SparkOnYarn seems to be ok) if these are the only tests that fail due to the packaging scheme; we cannot be blocked on Hadoop version upgrade by this test dependency. Where is the tgz file unpackaged? Perhaps I could try to add code to nuke the hadoop jars and/or replace them with the version used by Hive, to see if that works. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: HIVE-13934.2.patch > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[
https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336964#comment-15336964
]
Hive QA commented on HIVE-14021:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811225/HIVE-14021.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 10221 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_cond_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query13
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query48
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query85
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query88
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query89
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query91
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/155/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/155/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-155/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811225 - PreCommit-HIVE-MASTER-Build
> When converting to CNF, fail if the expression exceeds a threshold
> --
>
> Key: HIVE-14021
> URL: https://issues.apache.org/jira/browse/HIVE-14021
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-14021.patch
>
>
> When converting to conjunctive normal form (CNF), fail if the expression
> exceeds a threshold. CNF can explode exponentially in the size of the input
> expression, but rarely does so in practice. Add a maxNodeCount parameter to
> RexUtil.toCnf and throw or return null if it is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14015:
Attachment: (was: HIVE-14015.1.patch)
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.call
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14015:
Attachment: HIVE-14015.1.patch
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch, HIVE-14015.1.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtoc
[jira] [Updated] (HIVE-14051) Custom authenticaton in Hive JDBC
[ https://issues.apache.org/jira/browse/HIVE-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Sathappan updated HIVE-14051: Status: Patch Available (was: Open) > Custom authenticaton in Hive JDBC > - > > Key: HIVE-14051 > URL: https://issues.apache.org/jira/browse/HIVE-14051 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Vinoth Sathappan >Assignee: Vinoth Sathappan > Attachments: HIVE-14051.1.patch > > > Enable the JDBC driver for Hive to use a pluggable module to connect to HS2 > behind gateways using OAuth, OpenID Connect, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14051) Custom authenticaton in Hive JDBC
[ https://issues.apache.org/jira/browse/HIVE-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Sathappan updated HIVE-14051: Attachment: HIVE-14051.1.patch > Custom authenticaton in Hive JDBC > - > > Key: HIVE-14051 > URL: https://issues.apache.org/jira/browse/HIVE-14051 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Vinoth Sathappan >Assignee: Vinoth Sathappan > Attachments: HIVE-14051.1.patch > > > Enable the JDBC driver for Hive to use a pluggable module to connect to HS2 > behind gateways using OAuth, OpenID Connect, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14012) some ColumnVector-s are missing ensureSize
[ https://issues.apache.org/jira/browse/HIVE-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336944#comment-15336944 ] Prasanth Jayachandran commented on HIVE-14012: -- lgtm, +1. Pending tests > some ColumnVector-s are missing ensureSize > -- > > Key: HIVE-14012 > URL: https://issues.apache.org/jira/browse/HIVE-14012 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14012.01.patch, HIVE-14012.01.patch, > HIVE-14012.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13930: Attachment: HIVE-13930.04.patch And again... > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14012) some ColumnVector-s are missing ensureSize
[ https://issues.apache.org/jira/browse/HIVE-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14012: Attachment: HIVE-14012.01.patch Updated. > some ColumnVector-s are missing ensureSize > -- > > Key: HIVE-14012 > URL: https://issues.apache.org/jira/browse/HIVE-14012 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14012.01.patch, HIVE-14012.01.patch, > HIVE-14012.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13901: Status: Open (was: Patch Available) > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13901: Status: Patch Available (was: Open) > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13901: Attachment: HIVE-13901.6.patch > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14015:
Status: Open (was: Patch Available)
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlock
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[
https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14015:
Status: Patch Available (was: Open)
> SMB MapJoin failed for Hive on Spark when kerberized
>
>
> Key: HIVE-14015
> URL: https://issues.apache.org/jira/browse/HIVE-14015
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14015.1.patch
>
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> It could be reproduced:
> 1) prepare sample data:
> a=1
> while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data
> 2) prepare source hive table:
> CREATE TABLE `s`(`c` string);
> load data local inpath 'data' into table s;
> 3) prepare the bucketed table:
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS;
> insert into t select * from s;
> 4) reproduce this issue:
> SET hive.execution.engine=spark;
> SET hive.auto.convert.sortmerge.join = true;
> SET hive.auto.convert.sortmerge.join.bigtable.selection.policy =
> org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ;
> SET hive.auto.convert.sortmerge.join.noconditionaltask = true;
> SET hive.optimize.bucketmapjoin = true;
> SET hive.optimize.bucketmapjoin.sortedmerge = true;
> select * from t join t t1 on t.c=t1.c;
> The stack is as following:
> {noformat}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 6,
> ychencdh571-2.vpc.cloudera.com): java.lang.RuntimeException: Error processing
> row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
> while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:154)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2003)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"c":"13"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7454)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:662)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:966)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlock
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336905#comment-15336905 ] Prasanth Jayachandran commented on HIVE-13985: -- Committed to branch-1. Thanks [~sershe] for the review! Will upload master patch to RB shortly > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336848#comment-15336848 ] Sergey Shelukhin commented on HIVE-13985: - K +1 for branch-1... there should probably be RB for master > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336844#comment-15336844 ] Sergey Shelukhin commented on HIVE-13159: - +1, should this be committed? > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.2.patch, HIVE-13159.3.patch, HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14012) some ColumnVector-s are missing ensureSize
[ https://issues.apache.org/jira/browse/HIVE-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336842#comment-15336842 ] Prasanth Jayachandran commented on HIVE-14012: -- isRepeating missing inside preserveData condition? > some ColumnVector-s are missing ensureSize > -- > > Key: HIVE-14012 > URL: https://issues.apache.org/jira/browse/HIVE-14012 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14012.01.patch, HIVE-14012.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336825#comment-15336825 ] Prasanth Jayachandran commented on HIVE-13985: -- OrcTail preserves the entire serialized footer from which it derives metadata lazily. RB patch is for branch-1 only which does not have to have deal with metastore cache. Patch for master is where I fixed the metastore cache test failure. As I said before I am not going to commit to master until HIVE-14007. I have just uploaded the patch to kick off pre-commit test run. > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13963) vectorization - string arguments may be converted to decimal null
[ https://issues.apache.org/jira/browse/HIVE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13963: Description: See HIVE-13957. The default precision and scale for the implicit decimal cast are -max,max, ie 38,38-38,18. Those don't do what the code may assume they do. All the values >=1 become invalid and precision-scale enforcement automatically converts them to null. We need to 1) Validate when this happens in/after the conversion code and bail; 2) Or, derive precision and scale from the constants themselves so they all fit, instead; 3) Or, derive it from the type of whatever caused the conversion in the first place (e.g. IN column decimal); however, this could be function-specific (e.g. IN just needs equality, BETWEEN would need at least one extra digit, arithmetic, if this ever happens, would need everything, etc.); 4) Something else? :) was: See HIVE-13957. The default precision and scale for the implicit decimal cast are max,max, ie 38,38. Those don't do what the code may assume they do. All the values >=1 become invalid and precision-scale enforcement automatically converts them to null. We need to 1) Validate when this happens in/after the conversion code and bail; 2) Or, derive precision and scale from the constants themselves so they all fit, instead; 3) Or, derive it from the type of whatever caused the conversion in the first place (e.g. IN column decimal); however, this could be function-specific (e.g. IN just needs equality, BETWEEN would need at least one extra digit, arithmetic, if this ever happens, would need everything, etc.); 4) Something else? :) > vectorization - string arguments may be converted to decimal null > - > > Key: HIVE-13963 > URL: https://issues.apache.org/jira/browse/HIVE-13963 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Matt McCline > > See HIVE-13957. > The default precision and scale for the implicit decimal cast are -max,max, > ie 38,38-38,18. Those don't do what the code may assume they do. All the > values >=1 become invalid and precision-scale enforcement automatically > converts them to null. > We need to > 1) Validate when this happens in/after the conversion code and bail; > 2) Or, derive precision and scale from the constants themselves so they all > fit, instead; > 3) Or, derive it from the type of whatever caused the conversion in the first > place (e.g. IN column decimal); however, this could be function-specific > (e.g. IN just needs equality, BETWEEN would need at least one extra digit, > arithmetic, if this ever happens, would need everything, etc.); > 4) Something else? :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13963) vectorization - string arguments may be converted to decimal null
[ https://issues.apache.org/jira/browse/HIVE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13963: Description: See HIVE-13957. The default precision and scale for the implicit decimal cast are -max,max, ie 38,38- 38,18. -Those don't do what the code may assume they do. All the values >=1- Some values may become invalid and precision-scale enforcement automatically converts them to null. We need to 1) Validate when this happens in/after the conversion code and bail; 2) Or, derive precision and scale from the constants themselves so they all fit, instead; 3) Or, derive it from the type of whatever caused the conversion in the first place (e.g. IN column decimal); however, this could be function-specific (e.g. IN just needs equality, BETWEEN would need at least one extra digit, arithmetic, if this ever happens, would need everything, etc.); 4) Something else? :) was: See HIVE-13957. The default precision and scale for the implicit decimal cast are -max,max, ie 38,38- 38,18. Those don't do what the code may assume they do. All the values >=1 become invalid and precision-scale enforcement automatically converts them to null. We need to 1) Validate when this happens in/after the conversion code and bail; 2) Or, derive precision and scale from the constants themselves so they all fit, instead; 3) Or, derive it from the type of whatever caused the conversion in the first place (e.g. IN column decimal); however, this could be function-specific (e.g. IN just needs equality, BETWEEN would need at least one extra digit, arithmetic, if this ever happens, would need everything, etc.); 4) Something else? :) > vectorization - string arguments may be converted to decimal null > - > > Key: HIVE-13963 > URL: https://issues.apache.org/jira/browse/HIVE-13963 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Matt McCline > > See HIVE-13957. > The default precision and scale for the implicit decimal cast are -max,max, > ie 38,38- 38,18. -Those don't do what the code may assume they do. All the > values >=1- Some values may become invalid and precision-scale enforcement > automatically converts them to null. > We need to > 1) Validate when this happens in/after the conversion code and bail; > 2) Or, derive precision and scale from the constants themselves so they all > fit, instead; > 3) Or, derive it from the type of whatever caused the conversion in the first > place (e.g. IN column decimal); however, this could be function-specific > (e.g. IN just needs equality, BETWEEN would need at least one extra digit, > arithmetic, if this ever happens, would need everything, etc.); > 4) Something else? :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13963) vectorization - string arguments may be converted to decimal null
[ https://issues.apache.org/jira/browse/HIVE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13963: Description: See HIVE-13957. The default precision and scale for the implicit decimal cast are -max,max, ie 38,38- 38,18. Those don't do what the code may assume they do. All the values >=1 become invalid and precision-scale enforcement automatically converts them to null. We need to 1) Validate when this happens in/after the conversion code and bail; 2) Or, derive precision and scale from the constants themselves so they all fit, instead; 3) Or, derive it from the type of whatever caused the conversion in the first place (e.g. IN column decimal); however, this could be function-specific (e.g. IN just needs equality, BETWEEN would need at least one extra digit, arithmetic, if this ever happens, would need everything, etc.); 4) Something else? :) was: See HIVE-13957. The default precision and scale for the implicit decimal cast are -max,max, ie 38,38-38,18. Those don't do what the code may assume they do. All the values >=1 become invalid and precision-scale enforcement automatically converts them to null. We need to 1) Validate when this happens in/after the conversion code and bail; 2) Or, derive precision and scale from the constants themselves so they all fit, instead; 3) Or, derive it from the type of whatever caused the conversion in the first place (e.g. IN column decimal); however, this could be function-specific (e.g. IN just needs equality, BETWEEN would need at least one extra digit, arithmetic, if this ever happens, would need everything, etc.); 4) Something else? :) > vectorization - string arguments may be converted to decimal null > - > > Key: HIVE-13963 > URL: https://issues.apache.org/jira/browse/HIVE-13963 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Matt McCline > > See HIVE-13957. > The default precision and scale for the implicit decimal cast are -max,max, > ie 38,38- 38,18. Those don't do what the code may assume they do. All the > values >=1 become invalid and precision-scale enforcement automatically > converts them to null. > We need to > 1) Validate when this happens in/after the conversion code and bail; > 2) Or, derive precision and scale from the constants themselves so they all > fit, instead; > 3) Or, derive it from the type of whatever caused the conversion in the first > place (e.g. IN column decimal); however, this could be function-specific > (e.g. IN just needs equality, BETWEEN would need at least one extra digit, > arithmetic, if this ever happens, would need everything, etc.); > 4) Something else? :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13963) vectorization - string arguments may be converted to decimal null
[ https://issues.apache.org/jira/browse/HIVE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13963: Priority: Major (was: Critical) > vectorization - string arguments may be converted to decimal null > - > > Key: HIVE-13963 > URL: https://issues.apache.org/jira/browse/HIVE-13963 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Matt McCline > > See HIVE-13957. > The default precision and scale for the implicit decimal cast are max,max, ie > 38,38. Those don't do what the code may assume they do. All the values >=1 > become invalid and precision-scale enforcement automatically converts them to > null. > We need to > 1) Validate when this happens in/after the conversion code and bail; > 2) Or, derive precision and scale from the constants themselves so they all > fit, instead; > 3) Or, derive it from the type of whatever caused the conversion in the first > place (e.g. IN column decimal); however, this could be function-specific > (e.g. IN just needs equality, BETWEEN would need at least one extra digit, > arithmetic, if this ever happens, would need everything, etc.); > 4) Something else? :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14024) setAllColumns is called incorrectly after some changes
[ https://issues.apache.org/jira/browse/HIVE-14024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14024: Attachment: HIVE-14024.01.patch Again for HiveQA... gr > setAllColumns is called incorrectly after some changes > -- > > Key: HIVE-14024 > URL: https://issues.apache.org/jira/browse/HIVE-14024 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14024.01.patch, HIVE-14024.patch > > > h/t [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14012) some ColumnVector-s are missing ensureSize
[ https://issues.apache.org/jira/browse/HIVE-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336780#comment-15336780 ] Sergey Shelukhin commented on HIVE-14012: - [~mmccline] [~owen.omalley] [~prasanth_j] ping? > some ColumnVector-s are missing ensureSize > -- > > Key: HIVE-14012 > URL: https://issues.apache.org/jira/browse/HIVE-14012 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14012.01.patch, HIVE-14012.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7443) Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs
[ https://issues.apache.org/jira/browse/HIVE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336766#comment-15336766 ] Aihua Xu commented on HIVE-7443: Thanks Chaoyu for reviewing the code. I changed to use UserGroupInformation.getCurrentUser() instead of getLoginUser() which also works for the case of fromSubject. So if the JDBC client does the login, then JDBC will get the credential from the subject. We don't separate logic for different JDKs, the same logic works for both of them. > Fix HiveConnection to communicate with Kerberized Hive JDBC server and > alternative JDKs > --- > > Key: HIVE-7443 > URL: https://issues.apache.org/jira/browse/HIVE-7443 > Project: Hive > Issue Type: Bug > Components: JDBC, Security >Affects Versions: 0.12.0, 0.13.1 > Environment: Kerberos > Run Hive server2 and client with IBM JDK7.1 >Reporter: Yu Gao >Assignee: Aihua Xu > Attachments: HIVE-7443.2.patch, HIVE-7443.patch > > > Hive Kerberos authentication has been enabled in my cluster. I ran kinit to > initialize the current login user's ticket cache successfully, and then tried > to use beeline to connect to Hive Server2, but failed. After I manually added > some logging to catch the failure exception, this is what I got that caused > the failure: > beeline> !connect > jdbc:hive2://:1/default;principal=hive/@REALM.COM > org.apache.hive.jdbc.HiveDriver > scan complete in 2ms > Connecting to > jdbc:hive2://:1/default;principal=hive/@REALM.COM > Enter password for > jdbc:hive2://:1/default;principal=hive/@REALM.COM: > 14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport > javax.security.sasl.SaslException: Failed to open client transport [Caused by > java.io.IOException: Could not instantiate SASL transport] > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78) > at > org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342) > at > org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200) > at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:178) > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:198) > at > org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) > at > org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186) > at org.apache.hive.beeline.Commands.connect(Commands.java:959) > at org.apache.hive.beeline.Commands.connect(Commands.java:880) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at > org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659) > at > org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.io.IOException: Could not instantiate SASL transport > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:177) > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:74) > ... 24 more > Caused by: javax.security.sasl.SaslException: Failure to initialize security > context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0 > major string: Invalid credentials > minor string: SubjectCredFinder: no JAAS Subject] > at > com.ibm.security.sasl.gsskerb.GssKrb5Client.(GssKrb5Client.java:131) > at > com.ibm.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:53) > at javax.security.sasl.Sasl.createSaslClient(Sasl.java:362) > at
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336757#comment-15336757 ] Sergey Shelukhin commented on HIVE-14000: - Mostly looks good, one questionable spot (comment) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7443) Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs
[ https://issues.apache.org/jira/browse/HIVE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336756#comment-15336756 ] Chaoyu Tang commented on HIVE-7443: --- Will this IBM JDK issue affect the case where kerberosAuthType is fromSubject? Also should we separate the logic for IBM JDK vs. others? > Fix HiveConnection to communicate with Kerberized Hive JDBC server and > alternative JDKs > --- > > Key: HIVE-7443 > URL: https://issues.apache.org/jira/browse/HIVE-7443 > Project: Hive > Issue Type: Bug > Components: JDBC, Security >Affects Versions: 0.12.0, 0.13.1 > Environment: Kerberos > Run Hive server2 and client with IBM JDK7.1 >Reporter: Yu Gao >Assignee: Aihua Xu > Attachments: HIVE-7443.2.patch, HIVE-7443.patch > > > Hive Kerberos authentication has been enabled in my cluster. I ran kinit to > initialize the current login user's ticket cache successfully, and then tried > to use beeline to connect to Hive Server2, but failed. After I manually added > some logging to catch the failure exception, this is what I got that caused > the failure: > beeline> !connect > jdbc:hive2://:1/default;principal=hive/@REALM.COM > org.apache.hive.jdbc.HiveDriver > scan complete in 2ms > Connecting to > jdbc:hive2://:1/default;principal=hive/@REALM.COM > Enter password for > jdbc:hive2://:1/default;principal=hive/@REALM.COM: > 14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport > javax.security.sasl.SaslException: Failed to open client transport [Caused by > java.io.IOException: Could not instantiate SASL transport] > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78) > at > org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342) > at > org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200) > at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:178) > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:198) > at > org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) > at > org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186) > at org.apache.hive.beeline.Commands.connect(Commands.java:959) > at org.apache.hive.beeline.Commands.connect(Commands.java:880) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at > org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659) > at > org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.io.IOException: Could not instantiate SASL transport > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:177) > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:74) > ... 24 more > Caused by: javax.security.sasl.SaslException: Failure to initialize security > context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0 > major string: Invalid credentials > minor string: SubjectCredFinder: no JAAS Subject] > at > com.ibm.security.sasl.gsskerb.GssKrb5Client.(GssKrb5Client.java:131) > at > com.ibm.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:53) > at javax.security.sasl.Sasl.createSaslClient(Sasl.java:362) > at > org.apache.thrift.transport.TSaslClientTransport.(TSaslClientTransport.java:72) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAut
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336747#comment-15336747 ] Sergey Shelukhin commented on HIVE-13901: - Back to +1 :) > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14034) Vectorization may fail with compex OR conditions
[
https://issues.apache.org/jira/browse/HIVE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14034:
Description:
Example callstack for reference, should somebody hit this:
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 288
at
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.StringGroupColEqualStringGroupScalarBase.evaluate(StringGroupColEqualStringGroupScalarBase.java:74)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.ColOrCol.evaluate(ColOrCol.java:268)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:121)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringScalarStringGroupColumn.evaluate(IfExprStringScalarStringGroupColumn.java:56)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:132)
... 23 more
{noformat}
> Vectorization may fail with compex OR conditions
>
>
> Key: HIVE-14034
> URL: https://issues.apache.org/jira/browse/HIVE-14034
> Project: Hive
> Issue Type: Bug
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14034.patch
>
>
> Example callstack for reference, should somebody hit this:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 288
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.StringGroupColEqualStringGroupScalarBase.evaluate(StringGroupColEqualStringGroupScalarBase.java:74)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.ColOrCol.evaluate(ColOrCol.java:268)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:121)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringScalarStringGroupColumn.evaluate(IfExprStringScalarStringGroupColumn.java:56)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:132)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14034) Vectorization may fail with compex OR conditions
[ https://issues.apache.org/jira/browse/HIVE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14034: Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Committed to the affected branches. > Vectorization may fail with compex OR conditions > > > Key: HIVE-14034 > URL: https://issues.apache.org/jira/browse/HIVE-14034 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14034.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336710#comment-15336710 ] Sergey Shelukhin commented on HIVE-13985: - Btw, metastore cache needs to have metadata... I hope this is preserved. Otherwise it would be impossible to do the split generation without reading the file. > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336708#comment-15336708 ] Sergey Shelukhin commented on HIVE-13985: - What was the fix for those? Doesn't appear to be on RB > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14023) LLAP: Make the Hive query id available in ContainerRunner
[
https://issues.apache.org/jira/browse/HIVE-14023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336689#comment-15336689
]
Hive QA commented on HIVE-14023:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811293/HIVE-14023.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/153/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/153/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-153/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811293 - PreCommit-HIVE-MASTER-Build
> LLAP: Make the Hive query id available in ContainerRunner
> -
>
> Key: HIVE-14023
> URL: https://issues.apache.org/jira/browse/HIVE-14023
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14023.01.patch, HIVE-14023.02.patch
>
>
> Needed to generate logs per query.
> We can use the dag identifier for now, but that isn't very useful. (The
> queryId may not be too useful either if users cannot find it - but that's
> better than a dagIdentifier)
> The queryId is available right now after the Processor starts, which is too
> late for log changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12977) Pass credentials in the current UGI while creating Tez session
[
https://issues.apache.org/jira/browse/HIVE-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Sathappan updated HIVE-12977:
Attachment: HIVE-12977.4.patch
> Pass credentials in the current UGI while creating Tez session
> --
>
> Key: HIVE-12977
> URL: https://issues.apache.org/jira/browse/HIVE-12977
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Vinoth Sathappan
>Assignee: Vinoth Sathappan
> Attachments: HIVE-12977.1.patch, HIVE-12977.1.patch,
> HIVE-12977.2.patch, HIVE-12977.3.patch, HIVE-12977.4.patch
>
>
> The credentials present in the current UGI i.e.
> UserGroupInformation.getCurrentUser().getCredentials() isn't passed to the
> Tez session. It is instantiated with null credentials.
> session = TezClient.create("HIVE-" + sessionId, tezConfig, true,
> commonLocalResources, null);
> In this case, tokens added using hive execution hooks, aren't available to
> Tez even if they are available in memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14048) patch for HIVE-4570 removes protected fields which can break dependencies
[ https://issues.apache.org/jira/browse/HIVE-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336677#comment-15336677 ] Ashutosh Chauhan commented on HIVE-14048: - Those fields are not of public api, there is no contract that we need to deprecate them before removing them. Dependencies which are using non-public apis need to update their source if they rely on non-public apis. > patch for HIVE-4570 removes protected fields which can break dependencies > - > > Key: HIVE-14048 > URL: https://issues.apache.org/jira/browse/HIVE-14048 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Critical > > The patch for HIVE-4570 removes protected fields like initialized, isDone, > started etc and created a TaskState enum to represent these variables. Since > these fields which were removed were marked protected, class which can extend > Task.java (or DDLTask.java) will fail to compile once they get this patch. > The protected fields should probably be marked deprecated instead of removing > them directly since it could break outside dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14049) Password prompt in Beeline is continuously printed
[
https://issues.apache.org/jira/browse/HIVE-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Yousufi updated HIVE-14049:
Description:
I'm experiencing this issue with a Mac, which was not occurring until recently.
{code}
Beeline version 2.2.0-SNAPSHOT by Apache Hive
beeline> !connect jdbc:hive2://localhost:1
Connecting to jdbc:hive2://localhost:1
Enter username for jdbc:hive2://localhost:1: hive
Enter password for jdbc:hive2://localhost:1:
Enter password for jdbc:hive2://localhost:1:
Enter password for jdbc:hive2://localhost:1:
...
{code}
The 'Enter password for jdbc:hive2://localhost:1:' line continues to print
until enter is hit. From looking at the code in Commands.java (lines
1413-1420), it's not quite clear why this happens on the second call to
readLine()) :
{code}
if (username == null) {
username = beeLine.getConsoleReader().readLine("Enter username for " + url +
": ");
}
props.setProperty("user", username);
if (password == null) {
password = beeLine.getConsoleReader().readLine("Enter password for " + url +
": ",
new Character('*'));
}
{code}
was:
I'm experiencing this issue with a Mac, which was not occurring until recently.
{code}
Beeline version 2.2.0-SNAPSHOT by Apache Hive
beeline> !connect jdbc:hive2://localhost:1
Connecting to jdbc:hive2://localhost:1
Enter username for jdbc:hive2://localhost:1: hive
Enter password for jdbc:hive2://localhost:1:
Enter password for jdbc:hive2://localhost:1:
Enter password for jdbc:hive2://localhost:1:
...
{code}
The 'Enter password for jdbc:hive2://localhost:1:' line continues to print
until enter is hit. From looking at the code in Commands.java (lines
1413-1420), it's not quite clear why this happens on the second call to
readLine()) :
{code}
if (username == null) {
username = beeLine.getConsoleReader().readLine("Enter username for " +
url + ": ");
}
props.setProperty("user", username);
if (password == null) {
password = beeLine.getConsoleReader().readLine("Enter password for " +
url + ": ",
new Character('*'));
}
{code}
> Password prompt in Beeline is continuously printed
> --
>
> Key: HIVE-14049
> URL: https://issues.apache.org/jira/browse/HIVE-14049
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>
> I'm experiencing this issue with a Mac, which was not occurring until
> recently.
> {code}
> Beeline version 2.2.0-SNAPSHOT by Apache Hive
> beeline> !connect jdbc:hive2://localhost:1
> Connecting to jdbc:hive2://localhost:1
> Enter username for jdbc:hive2://localhost:1: hive
> Enter password for jdbc:hive2://localhost:1:
> Enter password for jdbc:hive2://localhost:1:
> Enter password for jdbc:hive2://localhost:1:
> ...
> {code}
> The 'Enter password for jdbc:hive2://localhost:1:' line continues to
> print until enter is hit. From looking at the code in Commands.java (lines
> 1413-1420), it's not quite clear why this happens on the second call to
> readLine()) :
> {code}
> if (username == null) {
> username = beeLine.getConsoleReader().readLine("Enter username for " + url
> + ": ");
> }
> props.setProperty("user", username);
> if (password == null) {
> password = beeLine.getConsoleReader().readLine("Enter password for " + url
> + ": ",
> new Character('*'));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files
[
https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336669#comment-15336669
]
Ashutosh Chauhan commented on HIVE-13997:
-
Won't
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3165
delete the directory already. This later calls moveFile() If it doesn't we
should enhance logic in replaceFiles() to do the deletes, instead of in
moveFile()
> Insert overwrite directory doesn't overwrite existing files
> ---
>
> Key: HIVE-13997
> URL: https://issues.apache.org/jira/browse/HIVE-13997
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13997.1.patch
>
>
> Can be easily reproduced by running {{INSERT OVERWRITE DIRECTORY}} to the
> same dir twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[ https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336650#comment-15336650 ] Sergio Peña commented on HIVE-13964: Thanks [~ayousufi]. Changes look good now. Those tests are not related to this patch. +1 > Add a parameter to beeline to allow a properties file to be passed in > - > > Key: HIVE-13964 > URL: https://issues.apache.org/jira/browse/HIVE-13964 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, > HIVE-13964.03.patch, HIVE-13964.04.patch, HIVE-13964.05.patch > > > HIVE-6652 removed the ability to pass in a properties file as a beeline > parameter. It may be a useful feature to be able to pass the file in is a > parameter, such as --property-file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336633#comment-15336633
]
Abdullah Yousufi commented on HIVE-13964:
-
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810921/HIVE-13964.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10234 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/130/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/130/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-130/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810921 - PreCommit-HIVE-MASTER-Build
> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch,
> HIVE-13964.03.patch, HIVE-13964.04.patch, HIVE-13964.05.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline
> parameter. It may be a useful feature to be able to pass the file in is a
> parameter, such as --property-file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Yousufi updated HIVE-13964:
Comment: was deleted
(was: Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810921/HIVE-13964.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10234 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/130/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/130/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-130/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810921 - PreCommit-HIVE-MASTER-Build)
> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch,
> HIVE-13964.03.patch, HIVE-13964.04.patch, HIVE-13964.05.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline
> parameter. It may be a useful feature to be able to pass the file in is a
> parameter, such as --property-file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336623#comment-15336623
]
Abdullah Yousufi commented on HIVE-13964:
-
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810921/HIVE-13964.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10234 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/130/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/130/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-130/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810921 - PreCommit-HIVE-MASTER-Build
> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch,
> HIVE-13964.03.patch, HIVE-13964.04.patch, HIVE-13964.05.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline
> parameter. It may be a useful feature to be able to pass the file in is a
> parameter, such as --property-file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
