[jira] [Commented] (HIVE-20451) Metastore client and server tarball issues
[
https://issues.apache.org/jira/browse/HIVE-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598273#comment-16598273
]
Hive QA commented on HIVE-20451:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13522/dev-support/hive-personality.sh
|
| git revision | master / a77e046 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore standalone-metastore/metastore-common
standalone-metastore/metastore-server standalone-metastore/metastore-tools U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13522/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore client and server tarball issues
> --
>
> Key: HIVE-20451
> URL: https://issues.apache.org/jira/browse/HIVE-20451
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Alan Gates
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20451.01.patch, HIVE-20451.02.patch,
> HIVE-20451.03.patch, HIVE-20451.04.patch, HIVE-20451.05.patch
>
>
> With the split of the metastore into common and server there are now two sets
> of tarballs. There are a couple of issues here.
> # It doesn't make sense to have separate source tarballs for each. The
> source release should still be done from the standalone-metastore directory
> and include all code for the metastore.
> # The binary tarballs should have separate names. At the moment both are
> named apache-hive-metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: In Progress) > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[
https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598252#comment-16598252
]
Hive QA commented on HIVE-20291:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937547/HIVE-20291.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14917 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13521/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13521/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13521/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12937547 - PreCommit-HIVE-Build
> Allow HiveStreamingConnection to receive a WriteId
> --
>
> Key: HIVE-20291
> URL: https://issues.apache.org/jira/browse/HIVE-20291
> Project: Hive
> Issue Type: Improvement
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch,
> HIVE-20291.3.patch
>
>
> If the writeId is received externally it won't need to open connections to
> the metastore. It won't be able to the commit in this case as well so it must
> be done by the entity passing the writeId.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[
https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598244#comment-16598244
]
Hive QA commented on HIVE-20291:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} streaming: The patch generated 4 new + 647 unchanged -
86 fixed = 651 total (was 733) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13521/dev-support/hive-personality.sh
|
| git revision | master / a77e046 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13521/yetus/diff-checkstyle-streaming.txt
|
| modules | C: ql streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13521/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow HiveStreamingConnection to receive a WriteId
> --
>
> Key: HIVE-20291
> URL: https://issues.apache.org/jira/browse/HIVE-20291
> Project: Hive
> Issue Type: Improvement
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch,
> HIVE-20291.3.patch
>
>
> If the writeId is received externally it won't need to open connections to
> the metastore. It won't be able to the commit in this case as well so it must
> be done by the entity passing the writeId.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HIVE-20395: - Attachment: HIVE-20395.4.patch > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.1.patch, HIVE-20395.2.patch, > HIVE-20395.3.patch, HIVE-20395.4.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20451) Metastore client and server tarball issues
[ https://issues.apache.org/jira/browse/HIVE-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598237#comment-16598237 ] Alexander Kolbasov commented on HIVE-20451: --- Looks like the test failure is unrelated, attaching the same patch again. [~alangates] The latest patch creates only a single source tarball and a single binary tarball. What do you mean by a "client binary"? What should it include? Would you be able to review and commit the patch? > Metastore client and server tarball issues > -- > > Key: HIVE-20451 > URL: https://issues.apache.org/jira/browse/HIVE-20451 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Alan Gates >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20451.01.patch, HIVE-20451.02.patch, > HIVE-20451.03.patch, HIVE-20451.04.patch, HIVE-20451.05.patch > > > With the split of the metastore into common and server there are now two sets > of tarballs. There are a couple of issues here. > # It doesn't make sense to have separate source tarballs for each. The > source release should still be done from the standalone-metastore directory > and include all code for the metastore. > # The binary tarballs should have separate names. At the moment both are > named apache-hive-metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20451) Metastore client and server tarball issues
[ https://issues.apache.org/jira/browse/HIVE-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20451: -- Attachment: HIVE-20451.05.patch > Metastore client and server tarball issues > -- > > Key: HIVE-20451 > URL: https://issues.apache.org/jira/browse/HIVE-20451 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Alan Gates >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20451.01.patch, HIVE-20451.02.patch, > HIVE-20451.03.patch, HIVE-20451.04.patch, HIVE-20451.05.patch > > > With the split of the metastore into common and server there are now two sets > of tarballs. There are a couple of issues here. > # It doesn't make sense to have separate source tarballs for each. The > source release should still be done from the standalone-metastore directory > and include all code for the metastore. > # The binary tarballs should have separate names. At the moment both are > named apache-hive-metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: Patch Available (was: In Progress)
resubmit the patch for the branch-2 after pulling the latest change
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.2.patch, HIVE-20225.3.patch,
> HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch,
> HIVE-20225.7.patch, HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Attachment: HIVE-20225.16-branch-2.patch
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.2.patch, HIVE-20225.3.patch,
> HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch,
> HIVE-20225.7.patch, HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: In Progress (was: Patch Available)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch, HIVE-20225.2.patch,
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch,
> HIVE-20225.6.patch, HIVE-20225.7.patch, HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20451) Metastore client and server tarball issues
[
https://issues.apache.org/jira/browse/HIVE-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598224#comment-16598224
]
Hive QA commented on HIVE-20451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937674/HIVE-20451.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14916 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13519/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13519/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13519/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12937674 - PreCommit-HIVE-Build
> Metastore client and server tarball issues
> --
>
> Key: HIVE-20451
> URL: https://issues.apache.org/jira/browse/HIVE-20451
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Alan Gates
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20451.01.patch, HIVE-20451.02.patch,
> HIVE-20451.03.patch, HIVE-20451.04.patch
>
>
> With the split of the metastore into common and server there are now two sets
> of tarballs. There are a couple of issues here.
> # It doesn't make sense to have separate source tarballs for each. The
> source release should still be done from the standalone-metastore directory
> and include all code for the metastore.
> # The binary tarballs should have separate names. At the moment both are
> named apache-hive-metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20451) Metastore client and server tarball issues
[
https://issues.apache.org/jira/browse/HIVE-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598160#comment-16598160
]
Hive QA commented on HIVE-20451:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13519/dev-support/hive-personality.sh
|
| git revision | master / a77e046 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore standalone-metastore/metastore-common
standalone-metastore/metastore-server standalone-metastore/metastore-tools U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13519/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore client and server tarball issues
> --
>
> Key: HIVE-20451
> URL: https://issues.apache.org/jira/browse/HIVE-20451
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Alan Gates
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20451.01.patch, HIVE-20451.02.patch,
> HIVE-20451.03.patch, HIVE-20451.04.patch
>
>
> With the split of the metastore into common and server there are now two sets
> of tarballs. There are a couple of issues here.
> # It doesn't make sense to have separate source tarballs for each. The
> source release should still be done from the standalone-metastore directory
> and include all code for the metastore.
> # The binary tarballs should have separate names. At the moment both are
> named apache-hive-metastore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598108#comment-16598108
]
Vineet Garg commented on HIVE-18873:
[~elserj] That query was just to demonstrate/reproduce the bug. This query is
not part of test. Test has join queries for which Hive automatically generate
is not null filter on join keys and passes it to storage handler (not sure why
since Accumulo handler isn't able to handle it).
It was probably not caught earlier because hive.optimize.index.filter was off
by default (therefore hive wasn't pushing the filter to storage handler).
hive.optimize.index.filter was turned on by HIVE-12342 but then HIVE-15680
caused hive to not push the filter to storage handler.
Basically it looks like this has been in the system for a while and is being
exposed now due to these two jiras.
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598104#comment-16598104
]
Mass Dosage commented on HIVE-18767:
[~pvary] I put the ticket back into "in progress", added the single file that I
mention in the above comment, then put the ticket back into "patch available"
but nothing appears to have happened. Have I missed something to get the build
going?
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2.patch, HIVE-18767.3.patch,
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20306: -- Attachment: HIVE-20306.09.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20306: -- Attachment: (was: HIVE-20306.09.patch) > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20306: -- Attachment: HIVE-20306.09.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598062#comment-16598062
]
Josh Elser commented on HIVE-18873:
---
{code:java}
SELECT u.key, u.country FROM users u where u.key is not null;{code}
This query doesn't really make any sense for an HBase or Accumulo because you
cannot have a null Key (in their bigtable data model). Where did this query
come from in the first place? Would rather not see all of TestAccumuloCliDriver
ignored (and just this one query).
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20489) Explain plan of query hangs
[ https://issues.apache.org/jira/browse/HIVE-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598052#comment-16598052 ] Vineet Garg commented on HIVE-20489: [~janulatha] Out of curiosity can you share the query? > Explain plan of query hangs > --- > > Key: HIVE-20489 > URL: https://issues.apache.org/jira/browse/HIVE-20489 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20489.1.patch > > > Explain on a query that joins 47 views, in effect around 94 joins after view > expansion seems to take forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598037#comment-16598037
]
Vineet Garg commented on HIVE-18873:
Because this is an existing issue with Accumulo I believe it could be
investigated in separate JIRA. We should disable the test, get the current
patch in and open new JIRA for fixing it. What do you think [~prasanth_j]?
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch,
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598030#comment-16598030
]
Vineet Garg commented on HIVE-18873:
I looked into the test failure (TestAccumuloCliDriver). Query {{SELECT u.key,
u.country, c.name, c.key FROM users u JOIN countries c ON (u.country = c.key)}}
throws following exception:
{noformat}
java.lang.RuntimeException: Unexpected residual predicate: key is not null
at
org.apache.hadoop.hive.accumulo.predicate.AccumuloPredicateHandler.getSearchConditions(AccumuloPredicateHandler.java:396)
at
org.apache.hadoop.hive.accumulo.predicate.AccumuloPredicateHandler.getIterators(AccumuloPredicateHandler.java:315)
at
org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableInputFormat.getSplits(HiveAccumuloTableInputFormat.java:146)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:506)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:752)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:341)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:332)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:203)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:423)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:149)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2727)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2379)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1749)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1743)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1347)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1321)
at
org.apache.hadoop.hive.cli.control.CoreAccumuloCliDriver.runTest(CoreAccumuloCliDriver.java:113)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver(TestAccumuloCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
[jira] [Commented] (HIVE-20020) Hive contrib jar should not be in lib
[ https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597943#comment-16597943 ] Alice Fan commented on HIVE-20020: -- [~johndee] and [~belugabehr], Do you have any suggestion for the directory name of contrib jar? I am thinking to create a directory called "extralib" for all the future experimental features. This will be generated under $HIVE_HOME directory, which is similar to 'jdbc'. > Hive contrib jar should not be in lib > - > > Key: HIVE-20020 > URL: https://issues.apache.org/jira/browse/HIVE-20020 > Project: Hive > Issue Type: Improvement > Components: Contrib >Reporter: Johndee Burks >Assignee: Alice Fan >Priority: Trivial > Attachments: HIVE-20020.2.patch > > > Currently the way hive is packaged it includes hive-contrib-.jar in > lib, we should not include it here because it is picked up by services like > HS2. This creates a situation in which experimental features such as the > [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] > are accessible without understanding how to really install and use it. For > example you can create a table using HS2 via beeline with the aforementioned > SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not > work because the SerDe is not in aux to get shipped into distcache. I propose > we do not package it this way and if someone would like to leverage an > experimental feature they can add it manually to their environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Manamcheri updated HIVE-20372: -- Attachment: HIVE-20372.2.patch > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > Attachments: HIVE-20372.2.patch, HIVE-20372.patch > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20489) Explain plan of query hangs
[ https://issues.apache.org/jira/browse/HIVE-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20489: --- Status: Patch Available (was: Open) > Explain plan of query hangs > --- > > Key: HIVE-20489 > URL: https://issues.apache.org/jira/browse/HIVE-20489 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20489.1.patch > > > Explain on a query that joins 47 views, in effect around 94 joins after view > expansion seems to take forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20489) Explain plan of query hangs
[ https://issues.apache.org/jira/browse/HIVE-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20489: --- Attachment: HIVE-20489.1.patch > Explain plan of query hangs > --- > > Key: HIVE-20489 > URL: https://issues.apache.org/jira/browse/HIVE-20489 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20489.1.patch > > > Explain on a query that joins 47 views, in effect around 94 joins after view > expansion seems to take forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20489) Explain plan of query hangs
[ https://issues.apache.org/jira/browse/HIVE-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani reassigned HIVE-20489: -- Assignee: Janaki Lahorani > Explain plan of query hangs > --- > > Key: HIVE-20489 > URL: https://issues.apache.org/jira/browse/HIVE-20489 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > > Explain on a query that joins 47 views, in effect around 94 joins after view > expansion seems to take forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: In Progress (was: Patch Available) > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597732#comment-16597732 ] Karthik Manamcheri commented on HIVE-20372: --- The word "write" has been misspelled as "wrtie" in two places # In a log in TxnHandler.java # In a local variable in UpgradeTool.java > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > Attachments: HIVE-20372.patch > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597725#comment-16597725 ] Alexander Kolbasov commented on HIVE-20372: --- It would be nice to tell what is the typo in JIRA. > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > Attachments: HIVE-20372.patch > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20451) Metastore client and server tarball issues
[ https://issues.apache.org/jira/browse/HIVE-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597665#comment-16597665 ] Alan Gates commented on HIVE-20451: --- Won't we want a client binary? Is your plan to add that later? I agree there's no need for a common binary. We don't want submodule tarballs. Only the top one in standalone-metastore is needed, as that is the only artifact we'll release. Creating source tarballs in common and server is a waste of build time. > Metastore client and server tarball issues > -- > > Key: HIVE-20451 > URL: https://issues.apache.org/jira/browse/HIVE-20451 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Alan Gates >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20451.01.patch, HIVE-20451.02.patch, > HIVE-20451.03.patch, HIVE-20451.04.patch > > > With the split of the metastore into common and server there are now two sets > of tarballs. There are a couple of issues here. > # It doesn't make sense to have separate source tarballs for each. The > source release should still be done from the standalone-metastore directory > and include all code for the metastore. > # The binary tarballs should have separate names. At the moment both are > named apache-hive-metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Manamcheri updated HIVE-20372: -- Attachment: HIVE-20372.patch Status: Patch Available (was: In Progress) > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > Attachments: HIVE-20372.patch > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20372 started by Karthik Manamcheri. - > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597637#comment-16597637 ] Sahil Takiar commented on HIVE-14162: - [~szita] thanks for taking a look. Addressed your comments and added some more javadocs to make the code easier to understand. Updated the RB. > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch, > HIVE-14162.6.patch, HIVE-14162.7.patch, HIVE-14162.8.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-14162: Attachment: HIVE-14162.8.patch > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch, > HIVE-14162.6.patch, HIVE-14162.7.patch, HIVE-14162.8.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.19.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.18.patch,
> HIVE-20377.18.patch, HIVE-20377.19.patch, HIVE-20377.19.patch,
> HIVE-20377.4.patch, HIVE-20377.5.patch, HIVE-20377.6.patch,
> HIVE-20377.8.patch, HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
> h2. More examples with Avro
> {code}
> CREATE EXTERNAL TABLE wiki_kafka_avro_table
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wiki_kafka_avro_table",
> "kafka.bootstrap.servers"="localhost:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe",
> 'avro.schema.literal'='{
> "type" : "record",
> "name" : "Wikipedia",
> "namespace" : "org.apache.hive.kafka",
> "version": "1",
> "fields" : [ {
> "name" : "isrobot",
> "type" : "boolean"
> }, {
> "name" : "channel",
> "type" : "string"
> }, {
> "name" : "timestamp",
> "type" : "string"
> }, {
> "name" : "flags",
> "type" : "string"
> }, {
> "name" :
[jira] [Commented] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597609#comment-16597609 ] Karthik Manamcheri commented on HIVE-20372: --- One more minor typo at https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java#L370 > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597603#comment-16597603 ] mahesh kumar behera commented on HIVE-20293: [~sankarh] Please review the pull request > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20293.01.patch > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Manamcheri reassigned HIVE-20372: - Assignee: Karthik Manamcheri > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Assignee: Karthik Manamcheri >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20293: --- Status: Patch Available (was: Open) Added the write id to the alter table / alter partition messages to get the valid write id lists on the target cluster. Added a new test for alter table. The truncate table operation (removing files) is still non transactional. > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20293.01.patch > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20293: --- Attachment: HIVE-20293.01.patch > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20293.01.patch > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Attachment: HIVE-17300.6.patch > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597413#comment-16597413 ] Adam Szita commented on HIVE-14162: --- Hi [~stakiar], I left some comments on RB > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch, > HIVE-14162.6.patch, HIVE-14162.7.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20287) Document the differences between managed and external tables
[ https://issues.apache.org/jira/browse/HIVE-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597324#comment-16597324 ] Lars Francke commented on HIVE-20287: - The docs currently say: ??An external table describes the metadata / schema on external files. External table files can be accessed and managed by processes outside of Hive. External tables can access data stored in sources such as Azure Storage Volumes (ASV) or remote HDFS locations. If the structure or partitioning of an external table is changed, an [MSCK REPAIR TABLE table_name|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RecoverPartitions(MSCKREPAIRTABLE)] statement can be used to refresh metadata information.?? This makes it sound as if MSCK only works for external tables. Is that true? I'd need to check. Is anyone certain? > Document the differences between managed and external tables > > > Key: HIVE-20287 > URL: https://issues.apache.org/jira/browse/HIVE-20287 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Major > > We should document all the differences between managed and external tables. I > need everyone's help for that though. > This is what I know: > * ARCHIVE/UNARCHIVE - DDLTask - Only works for managed tables > * TRUNCATE - DDLSemanticAnalyzer - Only works for managed tables > * MERGE/CONCATENATE - HiveRelOpMaterializationValidator - Only works for > managed tables > * Constraints - DDLSemanticAnalyzer - (NOT NULL, DEFAULT, CHECK, only RELY > ist allowed) > * IMPORT - ImportSemanticAnalyzer - This has some wild restrictions I didn't > follow for external tables > * Query Results Caching - https://issues.apache.org/jira/browse/HIVE-18513 > SemanticAnalyzer > > Hortonworks has extra documentation listing these things: > * Query cache > * Materialized views, except in a limited way > * Default statistics gathering > * Compute queries using statistics > * Automatic runtime filtering > * File merging after insert > > It'd be great if someone (from Hortonworks or otherwise) could elaborate on > those. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20287) Document the differences between managed and external tables
[ https://issues.apache.org/jira/browse/HIVE-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597318#comment-16597318 ] Lars Francke commented on HIVE-20287: - Thanks Eugene! > Document the differences between managed and external tables > > > Key: HIVE-20287 > URL: https://issues.apache.org/jira/browse/HIVE-20287 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Major > > We should document all the differences between managed and external tables. I > need everyone's help for that though. > This is what I know: > * ARCHIVE/UNARCHIVE - DDLTask - Only works for managed tables > * TRUNCATE - DDLSemanticAnalyzer - Only works for managed tables > * MERGE/CONCATENATE - HiveRelOpMaterializationValidator - Only works for > managed tables > * Constraints - DDLSemanticAnalyzer - (NOT NULL, DEFAULT, CHECK, only RELY > ist allowed) > * IMPORT - ImportSemanticAnalyzer - This has some wild restrictions I didn't > follow for external tables > * Query Results Caching - https://issues.apache.org/jira/browse/HIVE-18513 > SemanticAnalyzer > > Hortonworks has extra documentation listing these things: > * Query cache > * Materialized views, except in a limited way > * Default statistics gathering > * Compute queries using statistics > * Automatic runtime filtering > * File merging after insert > > It'd be great if someone (from Hortonworks or otherwise) could elaborate on > those. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20466) Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience
[ https://issues.apache.org/jira/browse/HIVE-20466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597231#comment-16597231 ] Adam Szita edited comment on HIVE-20466 at 8/30/18 9:23 AM: +1, committed to master Thanks for the patch Denys, and thanks for reviewing Peter was (Author: szita): +1, committed to master > Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience > -- > > Key: HIVE-20466 > URL: https://issues.apache.org/jira/browse/HIVE-20466 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20466.1.patch, HIVE-20466.2.patch, > HIVE-20466.3.patch, HIVE-20466.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20466) Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience
[ https://issues.apache.org/jira/browse/HIVE-20466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597231#comment-16597231 ] Adam Szita commented on HIVE-20466: --- +1, committed to master > Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience > -- > > Key: HIVE-20466 > URL: https://issues.apache.org/jira/browse/HIVE-20466 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20466.1.patch, HIVE-20466.2.patch, > HIVE-20466.3.patch, HIVE-20466.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20466) Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience
[ https://issues.apache.org/jira/browse/HIVE-20466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-20466: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience > -- > > Key: HIVE-20466 > URL: https://issues.apache.org/jira/browse/HIVE-20466 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20466.1.patch, HIVE-20466.2.patch, > HIVE-20466.3.patch, HIVE-20466.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20487) Update committer's list at hive.apache.org/people.html
[ https://issues.apache.org/jira/browse/HIVE-20487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597169#comment-16597169 ] Adam Szita commented on HIVE-20487: --- Thanks for reviewing Marta, this is now committed and published > Update committer's list at hive.apache.org/people.html > -- > > Key: HIVE-20487 > URL: https://issues.apache.org/jira/browse/HIVE-20487 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-20487.0.patch > > > Adding pvary, kuczoram and szita > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20487) Update committer's list at hive.apache.org/people.html
[ https://issues.apache.org/jira/browse/HIVE-20487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita resolved HIVE-20487. --- Resolution: Fixed > Update committer's list at hive.apache.org/people.html > -- > > Key: HIVE-20487 > URL: https://issues.apache.org/jira/browse/HIVE-20487 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-20487.0.patch > > > Adding pvary, kuczoram and szita > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20487) Update committer's list at hive.apache.org/people.html
[ https://issues.apache.org/jira/browse/HIVE-20487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597168#comment-16597168 ] Marta Kuczora commented on HIVE-20487: -- +1 Thanks [~szita] for updating the page. > Update committer's list at hive.apache.org/people.html > -- > > Key: HIVE-20487 > URL: https://issues.apache.org/jira/browse/HIVE-20487 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-20487.0.patch > > > Adding pvary, kuczoram and szita > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20487) Update committer's list at hive.apache.org/people.html
[ https://issues.apache.org/jira/browse/HIVE-20487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-20487: -- Attachment: HIVE-20487.0.patch > Update committer's list at hive.apache.org/people.html > -- > > Key: HIVE-20487 > URL: https://issues.apache.org/jira/browse/HIVE-20487 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-20487.0.patch > > > Adding pvary, kuczoram and szita > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20487) Update committer's list at hive.apache.org/people.html
[ https://issues.apache.org/jira/browse/HIVE-20487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-20487: -- Description: Adding pvary, kuczoram and szita NO PRECOMMIT TESTS was:Adding pvary, kuczoram and szita > Update committer's list at hive.apache.org/people.html > -- > > Key: HIVE-20487 > URL: https://issues.apache.org/jira/browse/HIVE-20487 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-20487.0.patch > > > Adding pvary, kuczoram and szita > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20487) Update committer's list at hive.apache.org/people.html
[ https://issues.apache.org/jira/browse/HIVE-20487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita reassigned HIVE-20487: - > Update committer's list at hive.apache.org/people.html > -- > > Key: HIVE-20487 > URL: https://issues.apache.org/jira/browse/HIVE-20487 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > > Adding pvary, kuczoram and szita -- This message was sent by Atlassian JIRA (v7.6.3#76005)
