[jira] [Reopened] (HIVE-20193) cboInfo is not present in the explain plan json
[ https://issues.apache.org/jira/browse/HIVE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek reopened HIVE-20193: This is not a duplicate of HIVE-19360, both are separate issues, hence reopening this > cboInfo is not present in the explain plan json > --- > > Key: HIVE-20193 > URL: https://issues.apache.org/jira/browse/HIVE-20193 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20193.1.patch, HIVE-20193.2.patch > > > *cboInfo* attribute is not present in the explain plan json that is provided > to the pre exec hook in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581985#comment-16581985
]
Hive QA commented on HIVE-19253:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13256/dev-support/hive-personality.sh
|
| git revision | master / 649d7c1 |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13256/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13256/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13256/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if
[jira] [Commented] (HIVE-20384) Fix flakiness of erasure_commands.q
[
https://issues.apache.org/jira/browse/HIVE-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581964#comment-16581964
]
Hive QA commented on HIVE-20384:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935667/HIVE-20384.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13255/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13255/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12935667/HIVE-20384.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935667 - PreCommit-HIVE-Build
> Fix flakiness of erasure_commands.q
> ---
>
> Key: HIVE-20384
> URL: https://issues.apache.org/jira/browse/HIVE-20384
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20384.0.patch, HIVE-20384.1.patch
>
>
> Qtest erasure_commands.q might fail if erasure_simple.q precedes it in the
> same batch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20388) Move common classes out of metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581963#comment-16581963
]
Hive QA commented on HIVE-20388:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935738/HIVE-20388.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14878 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13254/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13254/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13254/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935738 - PreCommit-HIVE-Build
> Move common classes out of metastore-server
> ---
>
> Key: HIVE-20388
> URL: https://issues.apache.org/jira/browse/HIVE-20388
> Project: Hive
> Issue Type: Sub-task
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20388.01.patch, HIVE-20388.02.patch,
> HIVE-20388.03.patch
>
>
> There are many classes in metastore-server module that should be moved to
> metastore-common.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20391) HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing aggregate function
[ https://issues.apache.org/jira/browse/HIVE-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20391: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing > aggregate function > > > Key: HIVE-20391 > URL: https://issues.apache.org/jira/browse/HIVE-20391 > Project: Hive > Issue Type: Bug > Components: CBO, Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20391.01.patch, HIVE-20391.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20036) Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
[
https://issues.apache.org/jira/browse/HIVE-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matrix0xCC resolved HIVE-20036.
---
Resolution: Not A Bug
Not a bug of Hive, but a compatibility problem of HDFS.
> Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
> -
>
> Key: HIVE-20036
> URL: https://issues.apache.org/jira/browse/HIVE-20036
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Matrix0xCC
>Assignee: Matrix0xCC
>Priority: Major
>
> I'm using Hive 2.3.3 with Hadoop 3.0.0 and Spark 2.2.1.
> I've created a partitioned orc table and enabled compaction.
> But the compaction task keeps failing and complains that a URI cannot be
> resolved.
> here is the yarn application diagnostics log:
> {code:java}
> Application application_1529550480937_0033 failed 2 times due to AM Container
> for appattempt_1529550480937_0033_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: [2018-06-29 17:25:25.656]Port 8020
> specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> java.io.IOException: Port 8020 specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:266)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:217)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:127)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:355)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:289)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:234)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:222)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> For more detailed output, check the application tracking page:
> http://cluster-master:8088/cluster/app/application_1529550480937_0033 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
> This is my core-site.xml and hdfs-site.xml
> {code:xml}
>
>
> hadoop.tmp.dir
> file:/opt/hdfs/tmp/
> A base for other temporary directories.
>
>
> io.file.buffer.size
>
> 131072
>
>
> fs.defaultFS
> hdfs://hadoopcluster
>
>
> hadoop.proxyuser.smsuser.hosts
> *
>
>
> hadoop.proxyuser.smsuser.groups
> *
>
>
> {code}
> {code:xml}
>
>
> dfs.nameservices
> hadoopcluster
>
>
> dfs.ha.namenodes.hadoopcluster
> cluster-master,cluster-backup
>
>
> dfs.namenode.rpc-address.hadoopcluster.cluster-master
> cluster-master:9820
>
>
> dfs.namenode.rpc-address.hadoopcluster.cluster-backup
> cluster-backup:9820
>
>
>
[jira] [Updated] (HIVE-17979) Tez: Improve ReduceRecordSource passDownKey copying
[
https://issues.apache.org/jira/browse/HIVE-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-17979:
Status: Open (was: Patch Available)
> Tez: Improve ReduceRecordSource passDownKey copying
> ---
>
> Key: HIVE-17979
> URL: https://issues.apache.org/jira/browse/HIVE-17979
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-17979.1.patch, HIVE-17979.2.patch
>
>
> Tez does not use a single Key stream for both sides of the join, so each
> input gets its own ReduceRecordSource
> {code}
> sources[tag] = new ReduceRecordSource();
> {code}
> And this means for each input stream, there's a deserialized key (because the
> tag is not part of the Key byte stream), this means for a 2-table join there
> are 2 ReduceRecordSource objects.
> This means that the passDownKey is only an optimization when the Key,
> List has more than 1 value in it. Otherwise the copy is entirely
> wasted CPU cycles, because it deserializes the entire row to extract the key
> and discards the row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20036) Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
[
https://issues.apache.org/jira/browse/HIVE-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matrix0xCC reassigned HIVE-20036:
-
Assignee: Matrix0xCC
> Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
> -
>
> Key: HIVE-20036
> URL: https://issues.apache.org/jira/browse/HIVE-20036
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Matrix0xCC
>Assignee: Matrix0xCC
>Priority: Major
>
> I'm using Hive 2.3.3 with Hadoop 3.0.0 and Spark 2.2.1.
> I've created a partitioned orc table and enabled compaction.
> But the compaction task keeps failing and complains that a URI cannot be
> resolved.
> here is the yarn application diagnostics log:
> {code:java}
> Application application_1529550480937_0033 failed 2 times due to AM Container
> for appattempt_1529550480937_0033_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: [2018-06-29 17:25:25.656]Port 8020
> specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> java.io.IOException: Port 8020 specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:266)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:217)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:127)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:355)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:289)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:234)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:222)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> For more detailed output, check the application tracking page:
> http://cluster-master:8088/cluster/app/application_1529550480937_0033 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
> This is my core-site.xml and hdfs-site.xml
> {code:xml}
>
>
> hadoop.tmp.dir
> file:/opt/hdfs/tmp/
> A base for other temporary directories.
>
>
> io.file.buffer.size
>
> 131072
>
>
> fs.defaultFS
> hdfs://hadoopcluster
>
>
> hadoop.proxyuser.smsuser.hosts
> *
>
>
> hadoop.proxyuser.smsuser.groups
> *
>
>
> {code}
> {code:xml}
>
>
> dfs.nameservices
> hadoopcluster
>
>
> dfs.ha.namenodes.hadoopcluster
> cluster-master,cluster-backup
>
>
> dfs.namenode.rpc-address.hadoopcluster.cluster-master
> cluster-master:9820
>
>
> dfs.namenode.rpc-address.hadoopcluster.cluster-backup
> cluster-backup:9820
>
>
> dfs.namenode.http-address.hadoopcluster.cluster-master
>
[jira] [Updated] (HIVE-17979) Tez: Improve ReduceRecordSource passDownKey copying
[
https://issues.apache.org/jira/browse/HIVE-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-17979:
Status: Patch Available (was: Open)
> Tez: Improve ReduceRecordSource passDownKey copying
> ---
>
> Key: HIVE-17979
> URL: https://issues.apache.org/jira/browse/HIVE-17979
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-17979.1.patch, HIVE-17979.2.patch
>
>
> Tez does not use a single Key stream for both sides of the join, so each
> input gets its own ReduceRecordSource
> {code}
> sources[tag] = new ReduceRecordSource();
> {code}
> And this means for each input stream, there's a deserialized key (because the
> tag is not part of the Key byte stream), this means for a 2-table join there
> are 2 ReduceRecordSource objects.
> This means that the passDownKey is only an optimization when the Key,
> List has more than 1 value in it. Otherwise the copy is entirely
> wasted CPU cycles, because it deserializes the entire row to extract the key
> and discards the row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20036) Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
[
https://issues.apache.org/jira/browse/HIVE-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581956#comment-16581956
]
Matrix0xCC edited comment on HIVE-20036 at 8/16/18 4:56 AM:
Fixed by myself.
Hadoop 3.0.0 use 9820 as the default RPC port while prior and later versions
all used 8020, which leads to a compatibility problem.
I upgrade Hadoop to 3.0.3 and the compaction works perfectly now.
See this issue:
[Change default NameNode RPC port back to
8020|https://issues.apache.org/jira/browse/HDFS-12990]
was (Author: matrix0xcc):
Fixed by myself.
Hadoop 3.0.0 use 9820 as the default RPC port while prior and later versions
all used 8020, which leads to a compatibility problem.
I upgrade Hadoop to 3.0.3 and the compaction works perfectly now.
See this issue:
Change default NameNode RPC port back to 8020
> Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
> -
>
> Key: HIVE-20036
> URL: https://issues.apache.org/jira/browse/HIVE-20036
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Matrix0xCC
>Priority: Major
>
> I'm using Hive 2.3.3 with Hadoop 3.0.0 and Spark 2.2.1.
> I've created a partitioned orc table and enabled compaction.
> But the compaction task keeps failing and complains that a URI cannot be
> resolved.
> here is the yarn application diagnostics log:
> {code:java}
> Application application_1529550480937_0033 failed 2 times due to AM Container
> for appattempt_1529550480937_0033_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: [2018-06-29 17:25:25.656]Port 8020
> specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> java.io.IOException: Port 8020 specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:266)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:217)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:127)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:355)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:289)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:234)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:222)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> For more detailed output, check the application tracking page:
> http://cluster-master:8088/cluster/app/application_1529550480937_0033 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
> This is my
[jira] [Commented] (HIVE-20036) Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
[
https://issues.apache.org/jira/browse/HIVE-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581956#comment-16581956
]
Matrix0xCC commented on HIVE-20036:
---
Fixed by myself.
Hadoop 3.0.0 use 9820 as the default RPC port while prior and later versions
all used 8020, which leads to a compatibility problem.
I upgrade Hadoop to 3.0.3 and the compaction works perfectly now.
See this issue:
Change default NameNode RPC port back to 8020
> Hive Compactor MapReduce task keeps failing due to wrong hadoop URI.
> -
>
> Key: HIVE-20036
> URL: https://issues.apache.org/jira/browse/HIVE-20036
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Matrix0xCC
>Priority: Major
>
> I'm using Hive 2.3.3 with Hadoop 3.0.0 and Spark 2.2.1.
> I've created a partitioned orc table and enabled compaction.
> But the compaction task keeps failing and complains that a URI cannot be
> resolved.
> here is the yarn application diagnostics log:
> {code:java}
> Application application_1529550480937_0033 failed 2 times due to AM Container
> for appattempt_1529550480937_0033_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: [2018-06-29 17:25:25.656]Port 8020
> specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> java.io.IOException: Port 8020 specified in URI
> hdfs://hadoopcluster:8020/tmp/hadoop-yarn/staging/smsuser/.staging/job_1529550480937_0033/job.splitmetainfo
> but host 'hadoopcluster' is a logical (HA) namenode and does not use port
> information.
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:266)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:217)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:127)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:355)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:289)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:234)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:222)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> For more detailed output, check the application tracking page:
> http://cluster-master:8088/cluster/app/application_1529550480937_0033 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
> This is my core-site.xml and hdfs-site.xml
> {code:xml}
>
>
> hadoop.tmp.dir
> file:/opt/hdfs/tmp/
> A base for other temporary directories.
>
>
> io.file.buffer.size
>
> 131072
>
>
> fs.defaultFS
> hdfs://hadoopcluster
>
>
> hadoop.proxyuser.smsuser.hosts
> *
>
>
> hadoop.proxyuser.smsuser.groups
> *
>
>
> {code}
> {code:xml}
>
>
> dfs.nameservices
> hadoopcluster
>
>
> dfs.ha.namenodes.hadoopcluster
>
[jira] [Commented] (HIVE-20388) Move common classes out of metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581952#comment-16581952
]
Hive QA commented on HIVE-20388:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore/metastore-common in master has 9
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 71 unchanged - 3
fixed = 71 total (was 74) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} service: The patch generated 0 new + 7 unchanged - 6
fixed = 7 total (was 13) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
43s{color} | {color:red} standalone-metastore/metastore-common generated 24 new
+ 5 unchanged - 4 fixed = 29 total (was 9) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
54s{color} | {color:red} standalone-metastore_metastore-common generated 15 new
+ 4 unchanged - 0 fixed = 19 total (was 4) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} standalone-metastore_metastore-server generated 0
new + 49 unchanged - 13 fixed = 49 total (was 62) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF
[jira] [Commented] (HIVE-20351) GenericUDFNamedStruct should constant fold at compile time
[
https://issues.apache.org/jira/browse/HIVE-20351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581918#comment-16581918
]
Hive QA commented on HIVE-20351:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935733/HIVE-20351.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14878 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_annotate_stats_select]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13253/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13253/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13253/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935733 - PreCommit-HIVE-Build
> GenericUDFNamedStruct should constant fold at compile time
> --
>
> Key: HIVE-20351
> URL: https://issues.apache.org/jira/browse/HIVE-20351
> Project: Hive
> Issue Type: Bug
>Reporter: Mykhailo Kysliuk
>Assignee: Mykhailo Kysliuk
>Priority: Minor
> Attachments: HIVE-20351.1.patch, HIVE-20351.2.patch
>
>
> Reproduced at hive-3.0.
> When we run hive query:
> {code:java}
> select named_struct('Total','Total') from test;
> {code}
> We could see the ERROR at hiveserver logs:
> {code:java}
> 2018-05-25T15:18:13,182 ERROR [main] optimizer.ConstantPropagateProcFactory:
> Unable to evaluate
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct@a0bf272. Return
> value unrecoginizable.
> {code}
> This error is harmless because all results are correct. But named_struct
> constant values should be processed correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20351) GenericUDFNamedStruct should constant fold at compile time
[
https://issues.apache.org/jira/browse/HIVE-20351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581882#comment-16581882
]
Hive QA commented on HIVE-20351:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 98 unchanged - 2
fixed = 98 total (was 100) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13253/dev-support/hive-personality.sh
|
| git revision | master / 142367d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13253/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> GenericUDFNamedStruct should constant fold at compile time
> --
>
> Key: HIVE-20351
> URL: https://issues.apache.org/jira/browse/HIVE-20351
> Project: Hive
> Issue Type: Bug
>Reporter: Mykhailo Kysliuk
>Assignee: Mykhailo Kysliuk
>Priority: Minor
> Attachments: HIVE-20351.1.patch, HIVE-20351.2.patch
>
>
> Reproduced at hive-3.0.
> When we run hive query:
> {code:java}
> select named_struct('Total','Total') from test;
> {code}
> We could see the ERROR at hiveserver logs:
> {code:java}
> 2018-05-25T15:18:13,182 ERROR [main] optimizer.ConstantPropagateProcFactory:
> Unable to evaluate
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct@a0bf272. Return
> value unrecoginizable.
> {code}
> This error is harmless because all results are correct. But named_struct
> constant values should be processed correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581881#comment-16581881
]
Matt McCline commented on HIVE-20321:
-
With patch #3, there are 4 different kinds of vector hash key wrapper classes
with these members:
1) VectorHashKeyWrapperEmpty (int hashcode)
2) VectorHashKeyWrapperSingleLong (int hashcode, boolean isNull flag, long
value)
3) VectorHashKeyWrapperTwoLong (int hashcode, 2 X boolean isNull flags 2 X
long values)
4) VectorHashKeyWrapperGeneral (original class with many members including
arrays for all data types, etc)
With the new code "framework", we can add special case classes for other keys
as desired.
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch,
> HIVE-20321.03.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20321:
Attachment: HIVE-20321.03.patch
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch,
> HIVE-20321.03.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20321:
Status: Patch Available (was: In Progress)
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch,
> HIVE-20321.03.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20321:
Status: In Progress (was: Patch Available)
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20378: Attachment: HIVE-20378.03.patch > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.01.patch, HIVE-20378.02.patch, > HIVE-20378.03.patch, HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[
https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581861#comment-16581861
]
Hive QA commented on HIVE-20378:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935735/HIVE-20378.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14878 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_9]
(batchId=123)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13252/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13252/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13252/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935735 - PreCommit-HIVE-Build
> don't update stats during alter for txn table conversion
>
>
> Key: HIVE-20378
> URL: https://issues.apache.org/jira/browse/HIVE-20378
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20378.01.patch, HIVE-20378.02.patch,
> HIVE-20378.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20400) create table should always use a fully qualified path to avoid potential FS ambiguity
[ https://issues.apache.org/jira/browse/HIVE-20400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20400: Status: Patch Available (was: Open) > create table should always use a fully qualified path to avoid potential FS > ambiguity > - > > Key: HIVE-20400 > URL: https://issues.apache.org/jira/browse/HIVE-20400 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20400.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20400) create table should always use a fully qualified path to avoid potential FS ambiguity
[ https://issues.apache.org/jira/browse/HIVE-20400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581854#comment-16581854 ] Sergey Shelukhin commented on HIVE-20400: - [~ashutoshc] can you take a look? This might cause some out files to change, we'll see how it goes > create table should always use a fully qualified path to avoid potential FS > ambiguity > - > > Key: HIVE-20400 > URL: https://issues.apache.org/jira/browse/HIVE-20400 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20400.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20400) create table should always use a fully qualified path to avoid potential FS ambiguity
[ https://issues.apache.org/jira/browse/HIVE-20400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20400: Attachment: HIVE-20400.patch > create table should always use a fully qualified path to avoid potential FS > ambiguity > - > > Key: HIVE-20400 > URL: https://issues.apache.org/jira/browse/HIVE-20400 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20400.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20399) CTAS w/a custom table location that is not fully qualified fails for MM tables
[ https://issues.apache.org/jira/browse/HIVE-20399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20399: Status: Patch Available (was: Open) [~ashutoshc] can you take a look? > CTAS w/a custom table location that is not fully qualified fails for MM tables > -- > > Key: HIVE-20399 > URL: https://issues.apache.org/jira/browse/HIVE-20399 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20399.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20399) CTAS w/a custom table location that is not fully qualified fails for MM tables
[ https://issues.apache.org/jira/browse/HIVE-20399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20399: Attachment: HIVE-20399.patch > CTAS w/a custom table location that is not fully qualified fails for MM tables > -- > > Key: HIVE-20399 > URL: https://issues.apache.org/jira/browse/HIVE-20399 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20399.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20400) create table should always use a fully qualified path to avoid potential FS ambiguity
[ https://issues.apache.org/jira/browse/HIVE-20400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20400: --- > create table should always use a fully qualified path to avoid potential FS > ambiguity > - > > Key: HIVE-20400 > URL: https://issues.apache.org/jira/browse/HIVE-20400 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20399) CTAS w/a custom table location that is not fully qualified fails for MM tables
[ https://issues.apache.org/jira/browse/HIVE-20399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20399: --- > CTAS w/a custom table location that is not fully qualified fails for MM tables > -- > > Key: HIVE-20399 > URL: https://issues.apache.org/jira/browse/HIVE-20399 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-14848:
---
Attachment: HIVE-14848.3.patch
Status: Patch Available (was: Open)
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch,
> HIVE-14848.2.patch, HIVE-14848.3.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-14848:
---
Status: Open (was: Patch Available)
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch,
> HIVE-14848.2.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[
https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581838#comment-16581838
]
Hive QA commented on HIVE-20378:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13252/dev-support/hive-personality.sh
|
| git revision | master / 142367d |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13252/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13252/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13252/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> don't update stats during alter for txn table conversion
>
>
> Key: HIVE-20378
> URL: https://issues.apache.org/jira/browse/HIVE-20378
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20378.01.patch, HIVE-20378.02.patch,
> HIVE-20378.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19814) RPC Server port is always random for spark
[
https://issues.apache.org/jira/browse/HIVE-19814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581832#comment-16581832
]
Hive QA commented on HIVE-19814:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935731/HIVE-19814.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14876 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=193)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13251/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13251/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13251/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935731 - PreCommit-HIVE-Build
> RPC Server port is always random for spark
> --
>
> Key: HIVE-19814
> URL: https://issues.apache.org/jira/browse/HIVE-19814
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.3.0, 3.0.0, 2.4.0, 4.0.0
>Reporter: bounkong khamphousone
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19814.1.patch, HIVE-19814.2.patch
>
>
> RPC server port is always a random one. In fact, the problem is in
> RpcConfiguration.HIVE_SPARK_RSC_CONFIGS which doesn't include
> SPARK_RPC_SERVER_PORT.
>
> I've found this issue while trying to make hive-on-spark running inside
> docker.
>
> HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf
> > SparkSessionManagerImpl.setup and the latter call
> SparkClientFactory.initialize(conf) which initialize the rpc server. This
> RPCServer is then used to create the sparkClient which use the rpc server
> port as --remote-port arg. Since initiateSparkConf ignore
> SPARK_RPC_SERVER_PORT, then it will always be a random port.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20398) [Hive Metastore] Add a Configuration Item to Skip the HDFS Data Modification
[
https://issues.apache.org/jira/browse/HIVE-20398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wuchang reassigned HIVE-20398:
--
> [Hive Metastore] Add a Configuration Item to Skip the HDFS Data Modification
>
>
> Key: HIVE-20398
> URL: https://issues.apache.org/jira/browse/HIVE-20398
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Affects Versions: 2.3.2
>Reporter: wuchang
>Assignee: wuchang
>Priority: Major
>
> When we are conducting the hive upgrading, we have following use case:
> We want to sync the operations between two metastore server (A and B) by
> thrift api, but both them are based on the same HDFS. So, for operations like
> *drop_partitions*, *drop_table*, *insert_overwrite, create_table* which will
> cause the data modification in HDFS, we want it to be executed by only
> Metastore Server A. For metastore Server B, he will only change his metadata,
> but didn't do corresponding HDFS files operation.
> So, we need a switch to control this.
> like
>
> {code:java}
> hive.metastore.skip.hdfs{code}
> whose default value is *false* just like what is happening now.
> When its value is true, the metastore server will only conduct the metadata
> modification, but skip the HDFS data modification.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19814) RPC Server port is always random for spark
[
https://issues.apache.org/jira/browse/HIVE-19814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581804#comment-16581804
]
Hive QA commented on HIVE-19814:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} spark-client in master has 10 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
5s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} spark-client: The patch generated 0 new + 23
unchanged - 8 fixed = 23 total (was 31) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13251/dev-support/hive-personality.sh
|
| git revision | master / 489b37a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13251/yetus/patch-mvninstall-ql.txt
|
| modules | C: ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13251/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> RPC Server port is always random for spark
> --
>
> Key: HIVE-19814
> URL: https://issues.apache.org/jira/browse/HIVE-19814
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.3.0, 3.0.0, 2.4.0, 4.0.0
>Reporter: bounkong khamphousone
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19814.1.patch, HIVE-19814.2.patch
>
>
> RPC server port is always a random one. In fact, the problem is in
> RpcConfiguration.HIVE_SPARK_RSC_CONFIGS which doesn't include
> SPARK_RPC_SERVER_PORT.
>
> I've found this issue while trying to make hive-on-spark running inside
> docker.
>
> HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf
> > SparkSessionManagerImpl.setup and the latter call
>
[jira] [Commented] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581793#comment-16581793 ] Matt McCline commented on HIVE-20315: - Reverted the change. > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, > HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch, > HIVE-20315.09.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions
[ https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reopened HIVE-20315: - > Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary > casts/conversions > --- > > Key: HIVE-20315 > URL: https://issues.apache.org/jira/browse/HIVE-20315 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, > HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, > HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch, > HIVE-20315.09.patch > > > Generate multi-byte Unicode characters in addition to regular single byte > characters for random data. > Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in > vectorization without padding. > Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL > constants. > Fix NULL / Wrong Results issues in VectorElt. > Change performance Q files to generate non-user EXPLAIN with VECTORIZATION > display so unnecesary CAST / DECIMAL_64 conversions are visible. > Add test for vectorized STRUCT field reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: (was: HIVE-20246.7.patch)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch, HIVE-20246.6.patch,
> HIVE-20246.7.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.7.patch
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch, HIVE-20246.6.patch,
> HIVE-20246.7.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20227) Exclude glassfish javax.el dependency
[ https://issues.apache.org/jira/browse/HIVE-20227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg resolved HIVE-20227. Resolution: Fixed Fix Version/s: 3.2.0 > Exclude glassfish javax.el dependency > - > > Key: HIVE-20227 > URL: https://issues.apache.org/jira/browse/HIVE-20227 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20227.branch-3.1-1.patch > > > INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ > hive-llap-server --- > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireReleaseDeps > failed with message: > Release builds are not allowed to have SNAPSHOT depenendencies > Found Banned Dependency: org.glassfish:javax.el:jar:3.0.1-b11-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Status: Patch Available (was: Open)
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20227) Exclude glassfish javax.el dependency
[ https://issues.apache.org/jira/browse/HIVE-20227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581791#comment-16581791 ] Vineet Garg commented on HIVE-20227: Pushed this to branch-3. Master already has this change via HIVE-19579 > Exclude glassfish javax.el dependency > - > > Key: HIVE-20227 > URL: https://issues.apache.org/jira/browse/HIVE-20227 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20227.branch-3.1-1.patch > > > INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ > hive-llap-server --- > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireReleaseDeps > failed with message: > Release builds are not allowed to have SNAPSHOT depenendencies > Found Banned Dependency: org.glassfish:javax.el:jar:3.0.1-b11-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Issue Type: Improvement (was: Bug)
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
> The Group key is pushed down to Druid as a Constant Column, this leads to an
> Exception while parsing back the results since Druid Input format does not
> allow Decimals.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Description:
Queries like following gets generated by Tableau.
{code}
SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
GROUP BY 1.1001;
{code}
was:
Queries like following gets generated by Tableau.
{code}
SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
GROUP BY 1.1001;
{code}
The Group key is pushed down to Druid as a Constant Column, this leads to an
Exception while parsing back the results since Druid Input format does not
allow Decimals.
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Attachment: HIVE-19674.1.patch
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
> The Group key is pushed down to Druid as a Constant Column, this leads to an
> Exception while parsing back the results since Druid Input format does not
> allow Decimals.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-19674:
--
Assignee: Vineet Garg (was: slim bouguerra)
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
> The Group key is pushed down to Druid as a Constant Column, this leads to an
> Exception while parsing back the results since Druid Input format does not
> allow Decimals.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581769#comment-16581769
]
Hive QA commented on HIVE-20395:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935723/HIVE-20395.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13250/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13250/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13250/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12935723/HIVE-20395.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935723 - PreCommit-HIVE-Build
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Attachments: HIVE-20395.1.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20357) Introduce initOrUpgradeSchema option to schema tool
[
https://issues.apache.org/jira/browse/HIVE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581768#comment-16581768
]
Hive QA commented on HIVE-20357:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935772/HIVE-20357.1.branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13249/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13249/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13249/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-08-16 00:19:57.246
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-13249/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-08-16 00:19:57.249
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 489b37a HIVE-14898: HS2 shouldn't log callstack for an empty
auth header error
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 489b37a HIVE-14898: HS2 shouldn't log callstack for an empty
auth header error
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-08-16 00:19:58.395
+ rm -rf ../yetus_PreCommit-HIVE-Build-13249
+ mkdir ../yetus_PreCommit-HIVE-Build-13249
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-13249
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-13249/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/metastore/scripts/upgrade/hive/upgrade-3.0.0-to-3.1.0.hive.sql: does
not exist in index
error:
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/tools/MetastoreSchemaTool.java:
does not exist in index
error:
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/tools/SchemaToolCommandLine.java:
does not exist in index
error:
a/standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java:
does not exist in index
error:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/tools/MetastoreSchemaTool.java:
does not exist in index
error:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/tools/SchemaToolCommandLine.java:
does not exist in index
error:
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java:
does not exist in index
error: scripts/upgrade/hive/upgrade-3.0.0-to-3.1.0.hive.sql: does not exist in
index
error:
src/main/java/org/apache/hadoop/hive/metastore/tools/MetastoreSchemaTool.java:
does not exist in index
error:
src/main/java/org/apache/hadoop/hive/metastore/tools/SchemaToolCommandLine.java:
does not exist in index
error:
src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java:
does not exist in index
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-13249
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935772 - PreCommit-HIVE-Build
> Introduce initOrUpgradeSchema option to schema tool
> ---
>
> Key: HIVE-20357
> URL: https://issues.apache.org/jira/browse/HIVE-20357
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
[jira] [Commented] (HIVE-20391) HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing aggregate function
[
https://issues.apache.org/jira/browse/HIVE-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581766#comment-16581766
]
Hive QA commented on HIVE-20391:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935715/HIVE-20391.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14879 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13248/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13248/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13248/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935715 - PreCommit-HIVE-Build
> HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing
> aggregate function
>
>
> Key: HIVE-20391
> URL: https://issues.apache.org/jira/browse/HIVE-20391
> Project: Hive
> Issue Type: Bug
> Components: CBO, Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20391.01.patch, HIVE-20391.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.7.patch
Status: Patch Available (was: Open)
Hi [~aihuaxu],
Thanks for providing the feedback. Have added comments and changed test case
accordingly.
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch, HIVE-20246.6.patch,
> HIVE-20246.7.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Status: Open (was: Patch Available)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch, HIVE-20246.6.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20393) Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
[ https://issues.apache.org/jira/browse/HIVE-20393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581759#comment-16581759 ] Ashutosh Chauhan commented on HIVE-20393: - +1 pending tests > Semijoin Reduction : markSemiJoinForDPP behaves inconsistently > -- > > Key: HIVE-20393 > URL: https://issues.apache.org/jira/browse/HIVE-20393 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20393.1.patch, HIVE-20393.2.patch > > > markSemiJoinForDPP has multiple issues, > > * Uses map tsOps which is wrong as it disallows going thru same TS which may > have filters from more than 1 semijoin edges. This results in inconsistent > plans for same query as semijoin edges may be processed in different order > each time. > * Uses getColumnExpr() which is not as robust as extractColumn() thus > resulting in NPEs. > * The logic to mark an edge useful when NPE is hit may end up having bad > edge. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20237) Do Not Print StackTraces to STDERR in HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-20237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581757#comment-16581757
]
Alice Fan edited comment on HIVE-20237 at 8/16/18 12:11 AM:
Hi [~aihuaxu] and [~belugabehr], Could you please help to review it? Thanks.
was (Author: afan):
Hi [~aihuaxu], Could you please help to review it? Thanks!
> Do Not Print StackTraces to STDERR in HiveMetaStore
> ---
>
> Key: HIVE-20237
> URL: https://issues.apache.org/jira/browse/HIVE-20237
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Trivial
> Labels: newbie, noob
> Attachments: HIVE-20237.1.patch
>
>
> {code:java|title=HiveMetaStore.java}
> } catch (Throwable x) {
> x.printStackTrace();
> HMSHandler.LOG.error(StringUtils.stringifyException(x));
> throw x;
> }
> {code}
> Bad design here of "log and throw". Don't do it. Just throw the exception
> and let it be handled, and logged, in one place. At the very least, we don't
> need the error message to go into the STDERR logs with {{printStackTrace}},
> please remove. And remove the {{stringifyException}} code. Just use the
> normal logging faciltiies at the 'debug' level logging to hide the stack
> trace during normal operations.
> {code}
> HMSHandler.LOG.debug("Error", e);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20237) Do Not Print StackTraces to STDERR in HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-20237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581757#comment-16581757
]
Alice Fan commented on HIVE-20237:
--
Hi [~aihuaxu], Could you please help to review it? Thanks!
> Do Not Print StackTraces to STDERR in HiveMetaStore
> ---
>
> Key: HIVE-20237
> URL: https://issues.apache.org/jira/browse/HIVE-20237
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Trivial
> Labels: newbie, noob
> Attachments: HIVE-20237.1.patch
>
>
> {code:java|title=HiveMetaStore.java}
> } catch (Throwable x) {
> x.printStackTrace();
> HMSHandler.LOG.error(StringUtils.stringifyException(x));
> throw x;
> }
> {code}
> Bad design here of "log and throw". Don't do it. Just throw the exception
> and let it be handled, and logged, in one place. At the very least, we don't
> need the error message to go into the STDERR logs with {{printStackTrace}},
> please remove. And remove the {{stringifyException}} code. Just use the
> normal logging faciltiies at the 'debug' level logging to hide the stack
> trace during normal operations.
> {code}
> HMSHandler.LOG.debug("Error", e);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20227) Exclude glassfish javax.el dependency
[ https://issues.apache.org/jira/browse/HIVE-20227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581754#comment-16581754 ] Vineet Garg commented on HIVE-20227: [~thejas] Yes I'll push this to master and branch-3. > Exclude glassfish javax.el dependency > - > > Key: HIVE-20227 > URL: https://issues.apache.org/jira/browse/HIVE-20227 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20227.branch-3.1-1.patch > > > INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ > hive-llap-server --- > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireReleaseDeps > failed with message: > Release builds are not allowed to have SNAPSHOT depenendencies > Found Banned Dependency: org.glassfish:javax.el:jar:3.0.1-b11-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20227) Exclude glassfish javax.el dependency
[ https://issues.apache.org/jira/browse/HIVE-20227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581750#comment-16581750 ] Thejas M Nair commented on HIVE-20227: -- [~vgarg] This is committed to branch 3.1, but not to branch-3 and master. Do you plan to commit to branch-3 and master as well ? > Exclude glassfish javax.el dependency > - > > Key: HIVE-20227 > URL: https://issues.apache.org/jira/browse/HIVE-20227 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20227.branch-3.1-1.patch > > > INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ > hive-llap-server --- > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireReleaseDeps > failed with message: > Release builds are not allowed to have SNAPSHOT depenendencies > Found Banned Dependency: org.glassfish:javax.el:jar:3.0.1-b11-SNAPSHOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581745#comment-16581745
]
Jaume M commented on HIVE-14848:
This should be done together with
https://issues.apache.org/jira/browse/TEZ-3979 since now sensitive information
will be sent o tez.
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch,
> HIVE-14848.2.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-14848:
---
Attachment: HIVE-14848.2.patch
Status: Patch Available (was: Open)
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch,
> HIVE-14848.2.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-14848:
---
Status: Open (was: Patch Available)
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M reassigned HIVE-14848:
--
Assignee: Jaume M
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20391) HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing aggregate function
[
https://issues.apache.org/jira/browse/HIVE-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581743#comment-16581743
]
Hive QA commented on HIVE-20391:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 14 new + 4 unchanged - 0 fixed
= 18 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13248/dev-support/hive-personality.sh
|
| git revision | master / 489b37a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13248/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13248/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveAggregateReduceFunctionsRule may infer wrong return type when decomposing
> aggregate function
>
>
> Key: HIVE-20391
> URL: https://issues.apache.org/jira/browse/HIVE-20391
> Project: Hive
> Issue Type: Bug
> Components: CBO, Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20391.01.patch, HIVE-20391.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20357) Introduce initOrUpgradeSchema option to schema tool
[ https://issues.apache.org/jira/browse/HIVE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581737#comment-16581737 ] Daniel Dai commented on HIVE-20357: --- Same patch for branch-3. > Introduce initOrUpgradeSchema option to schema tool > --- > > Key: HIVE-20357 > URL: https://issues.apache.org/jira/browse/HIVE-20357 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20357.1.branch-3.patch, HIVE-20357.1.patch > > > Currently, schematool has two option: initSchema/upgradeSchema. User needs to > use different command line for different action. However, from the schema > version stored in db, we shall able to figure out if there's a need to > init/upgrade, and choose the right action automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20357) Introduce initOrUpgradeSchema option to schema tool
[ https://issues.apache.org/jira/browse/HIVE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20357: -- Attachment: HIVE-20357.1.branch-3.patch > Introduce initOrUpgradeSchema option to schema tool > --- > > Key: HIVE-20357 > URL: https://issues.apache.org/jira/browse/HIVE-20357 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20357.1.branch-3.patch, HIVE-20357.1.patch > > > Currently, schematool has two option: initSchema/upgradeSchema. User needs to > use different command line for different action. However, from the schema > version stored in db, we shall able to figure out if there's a need to > init/upgrade, and choose the right action automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581733#comment-16581733
]
Aihua Xu commented on HIVE-20246:
-
The change looks good.
Minor change. Probably add comment to the function. The test can just throw
Exception rather than throwing many individual ones. The location in the test
can be a little real: part.getSd().setLocation(tbl.getSd().getLocation() +
"/parCol=3");
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch, HIVE-20246.6.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581731#comment-16581731
]
Hive QA commented on HIVE-20377:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} itests/qtest-druid in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 84 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 1 new + 26 unchanged
- 4 fixed = 27 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} patch/itests/qtest-druid cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red} patch/itests/util cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} patch/kafka-handler cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} patch/llap-server cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m
6s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 5m
41s{color} | {color:red} root in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13246/dev-support/hive-personality.sh
|
| git revision | master / b7b5cb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13246/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.8.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.4.patch, HIVE-20377.5.patch,
> HIVE-20377.6.patch, HIVE-20377.8.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Attachment: HIVE-19253.04.patch
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Attachment: HIVE-19253.03.patch
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581721#comment-16581721
]
Hive QA commented on HIVE-20395:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935723/HIVE-20395.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13247/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13247/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13247/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12935723/HIVE-20395.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935723 - PreCommit-HIVE-Build
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Attachments: HIVE-20395.1.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581719#comment-16581719
]
Hive QA commented on HIVE-20377:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935716/HIVE-20377.8.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14880 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=194)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13246/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13246/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13246/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935716 - PreCommit-HIVE-Build
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.4.patch, HIVE-20377.5.patch,
> HIVE-20377.6.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps
[jira] [Updated] (HIVE-14898) HS2 shouldn't log callstack for an empty auth header error
[
https://issues.apache.org/jira/browse/HIVE-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-14898:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Patch pushed to master. Thanks Thejas for review!
> HS2 shouldn't log callstack for an empty auth header error
> --
>
> Key: HIVE-14898
> URL: https://issues.apache.org/jira/browse/HIVE-14898
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-14898.1.patch
>
>
> Currently when the auth header is not sent by the client (Knox seems to do
> this every time - it only adds auth header after receiving 401), HS2 logs the
> following twice, for two principals.
> The callstack is useless because this is an expected condition and 401 is
> returned to the client.
> {noformat}
> 2016-10-05 15:32:02,408 ERROR [HiveServer2-HttpHandler-Pool: Thread-199]:
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(169)) -
> Failed to authenticate with hive/_HOST kerberos principal
> 2016-10-05 15:32:02,408 ERROR [HiveServer2-HttpHandler-Pool: Thread-199]:
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(104)) - Error:
> org.apache.hive.service.auth.HttpAuthenticationException:
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:170)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:83)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:167)
> ... 23 more
> Caused by: org.apache.hive.service.auth.HttpAuthenticationException:
> Authorization header received from the client is empty.
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.getAuthHeader(ThriftHttpServlet.java:311)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.access$100(ThriftHttpServlet.java:59)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:212)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ... 24 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20397: -- Status: Patch Available (was: Open) [~ekoifman] can you take a look? > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20397.1.patch > > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20397: -- Attachment: HIVE-20397.1.patch > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20397.1.patch > > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20393) Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
[ https://issues.apache.org/jira/browse/HIVE-20393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20393: -- Attachment: HIVE-20393.2.patch > Semijoin Reduction : markSemiJoinForDPP behaves inconsistently > -- > > Key: HIVE-20393 > URL: https://issues.apache.org/jira/browse/HIVE-20393 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20393.1.patch, HIVE-20393.2.patch > > > markSemiJoinForDPP has multiple issues, > > * Uses map tsOps which is wrong as it disallows going thru same TS which may > have filters from more than 1 semijoin edges. This results in inconsistent > plans for same query as semijoin edges may be processed in different order > each time. > * Uses getColumnExpr() which is not as robust as extractColumn() thus > resulting in NPEs. > * The logic to mark an edge useful when NPE is hit may end up having bad > edge. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20397: - > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581664#comment-16581664
]
Hive QA commented on HIVE-20395:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935723/HIVE-20395.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14879 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_fs] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_fs_overwrite]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_orc_part]
(batchId=15)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_load_data_to_encrypted_tables]
(batchId=183)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_orc_negative_part]
(batchId=98)
org.apache.hive.service.TestDFSErrorHandling.testAccessDenied (batchId=249)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13245/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13245/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13245/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935723 - PreCommit-HIVE-Build
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Attachments: HIVE-20395.1.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19254) NumberFormatException in MetaStoreUtils.isFastStatsSame
[
https://issues.apache.org/jira/browse/HIVE-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581640#comment-16581640
]
Vihang Karajgaonkar commented on HIVE-19254:
Looks good to me +1
> NumberFormatException in MetaStoreUtils.isFastStatsSame
> ---
>
> Key: HIVE-19254
> URL: https://issues.apache.org/jira/browse/HIVE-19254
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19254.1.patch
>
>
> I see the following exception under some cases in the logs. This possibly
> happens when you try to add empty partitions.
> {noformat}
> 2018-04-19T19:32:19,260 ERROR [pool-7-thread-7] metastore.RetryingHMSHandler:
> MetaException(message:java.lang.NumberFormatException: null)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6824)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:4864)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:4801)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy24.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:16046)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:16030)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: null
> at java.lang.Long.parseLong(Long.java:552)
> at java.lang.Long.parseLong(Long.java:631)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:632)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:743)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:4827)
> ... 21 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581635#comment-16581635
]
Alexander Kolbasov commented on HIVE-19253:
---
In addition to that, the check for EXTERNAL table in
{{MetastoreUtils#isExternalTable}} only checks for params and ignores table
type.
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Attachment: HIVE-19253.03.patch
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581629#comment-16581629
]
Hive QA commented on HIVE-20395:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13245/dev-support/hive-personality.sh
|
| git revision | master / b7b5cb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13245/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Attachments: HIVE-20395.1.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20384) Fix flakiness of erasure_commands.q
[
https://issues.apache.org/jira/browse/HIVE-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581589#comment-16581589
]
Hive QA commented on HIVE-20384:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935667/HIVE-20384.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13244/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13244/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13244/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12935667/HIVE-20384.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935667 - PreCommit-HIVE-Build
> Fix flakiness of erasure_commands.q
> ---
>
> Key: HIVE-20384
> URL: https://issues.apache.org/jira/browse/HIVE-20384
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20384.0.patch, HIVE-20384.1.patch
>
>
> Qtest erasure_commands.q might fail if erasure_simple.q precedes it in the
> same batch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20384) Fix flakiness of erasure_commands.q
[
https://issues.apache.org/jira/browse/HIVE-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581586#comment-16581586
]
Hive QA commented on HIVE-20384:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935667/HIVE-20384.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14879 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13243/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13243/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13243/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935667 - PreCommit-HIVE-Build
> Fix flakiness of erasure_commands.q
> ---
>
> Key: HIVE-20384
> URL: https://issues.apache.org/jira/browse/HIVE-20384
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20384.0.patch, HIVE-20384.1.patch
>
>
> Qtest erasure_commands.q might fail if erasure_simple.q precedes it in the
> same batch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18021) Insert overwrite on acid table with Union All optimizations
[
https://issues.apache.org/jira/browse/HIVE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18021:
--
Attachment: HIVE-18021.not_atomic.patch
> Insert overwrite on acid table with Union All optimizations
> ---
>
> Key: HIVE-18021
> URL: https://issues.apache.org/jira/browse/HIVE-18021
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18021.not_atomic.patch
>
>
> This is a followup from HIVE-14988.
> T is unbucketed acid table
> {noformat}
> insert into T select a,b from S union all select a,b from S1
> {noformat}
> will create a separate subdirectory for each leg of the union in the target
> table
> (automatically on Tez, with some props enabled on MR)
> Regular Insert will make each subdirectory be a delta_x_x_0, delta_x_x_1.
> See HIVE-15899.
> There is no such suffix mechanism for base_x/.
> Need to figure how this should work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18774) ACID: Use the _copy_N files copyNumber as the implicit statement-id
[ https://issues.apache.org/jira/browse/HIVE-18774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18774: -- Attachment: HIVE-18774.03.wip.patch > ACID: Use the _copy_N files copyNumber as the implicit statement-id > --- > > Key: HIVE-18774 > URL: https://issues.apache.org/jira/browse/HIVE-18774 > Project: Hive > Issue Type: Bug > Components: Transactions > Environment: if this is not done in 3.0 it cannot be done at all >Reporter: Gopal V >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18774.03.wip.patch > > > When upgrading flat ORC files to ACID, use the _copy_N numbering as a > statement-id to avoid having to align the row numbering between _copy_1 and > _copy_2 files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20321:
Status: Patch Available (was: In Progress)
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20321:
Attachment: HIVE-20321.02.patch
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20321:
Status: In Progress (was: Patch Available)
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20321.01.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Attachment: HIVE-19253.02.patch
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19985) ACID: Skip decoding the ROW__ID sections for read-only queries
[
https://issues.apache.org/jira/browse/HIVE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581555#comment-16581555
]
Eugene Koifman edited comment on HIVE-19985 at 8/15/18 8:13 PM:
patch 5 includes a fix to a stupid bug in
{{VectorizedOrcAcidRowBatchReader.copyFromBase()}} wrt {{payloadCol}}
calculation that broke non LLAP path and some additional tests
was (Author: ekoifman):
patch 5 includes a stupid bug in
{{VectorizedOrcAcidRowBatchReader.copyFromBase()}} wrt {{payloadCol}}
calculation that broke non LLAP path and some additional tests
> ACID: Skip decoding the ROW__ID sections for read-only queries
> ---
>
> Key: HIVE-19985
> URL: https://issues.apache.org/jira/browse/HIVE-19985
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Eugene Koifman
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-19985.01.patch, HIVE-19985.04.patch,
> HIVE-19985.05.patch
>
>
> For a base_n file there are no aborted transactions within the file and if
> there are no pending delete deltas, the entire ACID ROW__ID can be skipped
> for all read-only queries (i.e SELECT), though it still needs to be projected
> out for MERGE, UPDATE and DELETE queries.
> This patch tries to entirely ignore the ACID ROW__ID fields for all tables
> where there are no possible deletes or aborted transactions for an ACID split.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581556#comment-16581556 ] Sergey Shelukhin commented on HIVE-20378: - [~ekoifman] ping? I actually wonder if we should even allow stats on txn tables without write ID/etc. Perhaps it would be better in a follow up to make sure we create write ID for conversion, nothing in the DB structure prevents it as far as I can tell. > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.01.patch, HIVE-20378.02.patch, > HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19985) ACID: Skip decoding the ROW__ID sections for read-only queries
[
https://issues.apache.org/jira/browse/HIVE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581555#comment-16581555
]
Eugene Koifman commented on HIVE-19985:
---
patch 5 includes a stupid bug in
{{VectorizedOrcAcidRowBatchReader.copyFromBase()}} wrt {{payloadCol}}
calculation that broke non LLAP path and some additional tests
> ACID: Skip decoding the ROW__ID sections for read-only queries
> ---
>
> Key: HIVE-19985
> URL: https://issues.apache.org/jira/browse/HIVE-19985
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Eugene Koifman
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-19985.01.patch, HIVE-19985.04.patch,
> HIVE-19985.05.patch
>
>
> For a base_n file there are no aborted transactions within the file and if
> there are no pending delete deltas, the entire ACID ROW__ID can be skipped
> for all read-only queries (i.e SELECT), though it still needs to be projected
> out for MERGE, UPDATE and DELETE queries.
> This patch tries to entirely ignore the ACID ROW__ID fields for all tables
> where there are no possible deletes or aborted transactions for an ACID split.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19985) ACID: Skip decoding the ROW__ID sections for read-only queries
[ https://issues.apache.org/jira/browse/HIVE-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19985: -- Attachment: HIVE-19985.05.patch > ACID: Skip decoding the ROW__ID sections for read-only queries > --- > > Key: HIVE-19985 > URL: https://issues.apache.org/jira/browse/HIVE-19985 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Eugene Koifman >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-19985.01.patch, HIVE-19985.04.patch, > HIVE-19985.05.patch > > > For a base_n file there are no aborted transactions within the file and if > there are no pending delete deltas, the entire ACID ROW__ID can be skipped > for all read-only queries (i.e SELECT), though it still needs to be projected > out for MERGE, UPDATE and DELETE queries. > This patch tries to entirely ignore the ACID ROW__ID fields for all tables > where there are no possible deletes or aborted transactions for an ACID split. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20396) Test HS2 open_connection metrics
[ https://issues.apache.org/jira/browse/HIVE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-20396: > Test HS2 open_connection metrics > > > Key: HIVE-20396 > URL: https://issues.apache.org/jira/browse/HIVE-20396 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > > HiveServer2 is emitting metrics _default.General.open_connections_ in both > binary and http mode. These metrics should be tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20384) Fix flakiness of erasure_commands.q
[
https://issues.apache.org/jira/browse/HIVE-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581535#comment-16581535
]
Hive QA commented on HIVE-20384:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13243/dev-support/hive-personality.sh
|
| git revision | master / b7b5cb4 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13243/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flakiness of erasure_commands.q
> ---
>
> Key: HIVE-20384
> URL: https://issues.apache.org/jira/browse/HIVE-20384
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20384.0.patch, HIVE-20384.1.patch
>
>
> Qtest erasure_commands.q might fail if erasure_simple.q precedes it in the
> same batch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20393) Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
[
https://issues.apache.org/jira/browse/HIVE-20393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581528#comment-16581528
]
Hive QA commented on HIVE-20393:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935661/HIVE-20393.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13242/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13242/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13242/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12935661/HIVE-20393.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935661 - PreCommit-HIVE-Build
> Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
> --
>
> Key: HIVE-20393
> URL: https://issues.apache.org/jira/browse/HIVE-20393
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20393.1.patch
>
>
> markSemiJoinForDPP has multiple issues,
>
> * Uses map tsOps which is wrong as it disallows going thru same TS which may
> have filters from more than 1 semijoin edges. This results in inconsistent
> plans for same query as semijoin edges may be processed in different order
> each time.
> * Uses getColumnExpr() which is not as robust as extractColumn() thus
> resulting in NPEs.
> * The logic to mark an edge useful when NPE is hit may end up having bad
> edge.
> cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20393) Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
[
https://issues.apache.org/jira/browse/HIVE-20393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581527#comment-16581527
]
Hive QA commented on HIVE-20393:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935661/HIVE-20393.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14879 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13241/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935661 - PreCommit-HIVE-Build
> Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
> --
>
> Key: HIVE-20393
> URL: https://issues.apache.org/jira/browse/HIVE-20393
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20393.1.patch
>
>
> markSemiJoinForDPP has multiple issues,
>
> * Uses map tsOps which is wrong as it disallows going thru same TS which may
> have filters from more than 1 semijoin edges. This results in inconsistent
> plans for same query as semijoin edges may be processed in different order
> each time.
> * Uses getColumnExpr() which is not as robust as extractColumn() thus
> resulting in NPEs.
> * The logic to mark an edge useful when NPE is hit may end up having bad
> edge.
> cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20348) Hive HCat does not create a proper 'client' on kerberos cluster without hive metastore
[
https://issues.apache.org/jira/browse/HIVE-20348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581526#comment-16581526
]
Lefty Leverenz commented on HIVE-20348:
---
[~osayankin], you named me as a reviewer on RB but I'm not qualified to review
code. Perhaps someone else can handle it.
https://reviews.apache.org/r/68275/
> Hive HCat does not create a proper 'client' on kerberos cluster without hive
> metastore
> --
>
> Key: HIVE-20348
> URL: https://issues.apache.org/jira/browse/HIVE-20348
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-20348.1.patch
>
>
> *STEPS TO REPRODUCE:*
> 1. Configure Hive to use embedded Metastore (do not specify
> {{hive.metastore.uris}} in {{hive-site.xml}});
> 2. Create a database and a table in MySQL:
> {code:java}
> mysql -uroot -p123456 -e "CREATE DATABASE test;CREATE TABLE test.test (id
> INT);INSERT INTO test.test VALUES (1),(2),(3)"
> {code}
> 3. Create a table in Hive:
> {code:java}
> hive -e "CREATE TABLE default.test (id INT)"
> {code}
> 4. Run Sqoop import command:
> {code:java}
> sqoop import --connect 'jdbc:mysql://localhost:3306/test' --username root
> --password 123456 --table test --hcatalog-database "default"
> --hcatalog-table "test" --verbose -m 1
> {code}
> *ACTUAL RESULT:*
> Sqoop import command fails with an exception:
> {code:java}
> 18/08/08 01:07:09 ERROR tool.ImportTool: Encountered IOException running
> import job: org.apache.hive.hcatalog.common.HCatException : 2001 : Error
> setting output information. Cause : java.lang.NullPointerException
> at
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:220)
> at
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:70)
> at
> org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:361)
> at
> org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:783)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:98)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:689)
> at
> org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:498)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:606)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.decodeWritable(Token.java:256)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:275)
> at
> org.apache.hive.hcatalog.common.HCatUtil.extractThriftToken(HCatUtil.java:351)
> at
> org.apache.hive.hcatalog.mapreduce.Security.handleSecurity(Security.java:139)
> at
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:214)
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Affects Version/s: (was: 2.0.2)
4.0.0
Status: Patch Available (was: Open)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.0, 3.0.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Attachment: HIVE-19253.01.patch
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20393) Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
[
https://issues.apache.org/jira/browse/HIVE-20393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581493#comment-16581493
]
Hive QA commented on HIVE-20393:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13241/dev-support/hive-personality.sh
|
| git revision | master / b7b5cb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13241/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Semijoin Reduction : markSemiJoinForDPP behaves inconsistently
> --
>
> Key: HIVE-20393
> URL: https://issues.apache.org/jira/browse/HIVE-20393
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20393.1.patch
>
>
> markSemiJoinForDPP has multiple issues,
>
> * Uses map tsOps which is wrong as it disallows going thru same TS which may
> have filters from more than 1 semijoin edges. This results in inconsistent
> plans for same query as semijoin edges may be processed in different order
> each time.
> * Uses getColumnExpr() which is not as robust as extractColumn() thus
> resulting in NPEs.
> * The logic to mark an edge useful when NPE is hit may end up having bad
> edge.
> cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20378: Attachment: (was: HIVE-20378.02.patch) > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.01.patch, HIVE-20378.02.patch, > HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20388) Move common classes out of metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581477#comment-16581477 ] Alexander Kolbasov commented on HIVE-20388: --- Resubmitting the same patch to check whether the test failure is intermittent. The patch doesn't introduce any new code so findbug rants can be ignored. > Move common classes out of metastore-server > --- > > Key: HIVE-20388 > URL: https://issues.apache.org/jira/browse/HIVE-20388 > Project: Hive > Issue Type: Sub-task >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20388.01.patch, HIVE-20388.02.patch, > HIVE-20388.03.patch > > > There are many classes in metastore-server module that should be moved to > metastore-common. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
