[jira] [Reopened] (HIVE-19569) alter table db1.t1 rename db2.t2 generates MetaStoreEventListener.onDropTable()
[
https://issues.apache.org/jira/browse/HIVE-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan reopened HIVE-19569:
-
Need to remove the check which restricts rename of transactional table across
databases.
> alter table db1.t1 rename db2.t2 generates
> MetaStoreEventListener.onDropTable()
> ---
>
> Key: HIVE-19569
> URL: https://issues.apache.org/jira/browse/HIVE-19569
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: mahesh kumar behera
>Priority: Major
>
> When renaming a table within the same DB, this operation causes
> {{MetaStoreEventListener.onAlterTable()}} to fire but when changing DB name
> for a table it causes {{MetaStoreEventListener.onDropTable()}} +
> {{MetaStoreEventListener.onCreateTable()}}.
> The files from original table are moved to new table location.
> This creates confusing semantics since any logic in {{onDropTable()}} doesn't
> know about the larger context, i.e. that there will be a matching
> {{onCreateTable()}}.
> In particular, this causes a problem for Acid tables since files moved from
> old table use WriteIDs that are not meaningful with the context of new table.
> Current implementation is due to replication. This should ideally be changed
> to raise a "not supported" error for tables that are marked for replication.
> cc [~sankarh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19488) Enable CM root based on db parameter, identifying a db as source of replication.
[ https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19488: Resolution: Fixed Status: Resolved (was: Patch Available) > Enable CM root based on db parameter, identifying a db as source of > replication. > > > Key: HIVE-19488 > URL: https://issues.apache.org/jira/browse/HIVE-19488 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19488.01.patch, HIVE-19488.02.patch, > HIVE-19488.03.patch, HIVE-19488.04.patch, HIVE-19488.05.patch, > HIVE-19488.06.patch, HIVE-19488.07.patch, HIVE-19488.08-branch-3.patch, > HIVE-19488.08.patch, HIVE-19488.09-branch-3.patch > > > * add a parameter at db level to identify if its a source of replication. > user should set this. > * Enable CM root only for databases that are a source of a replication > policy, for other db's skip the CM root functionality. > * prevent database drop if the parameter indicating its source of a > replication, is set. > * as an upgrade to this version, user should set the property on all > existing database policies, in affect. > * the parameter should be of the form . – repl.source.for : List < policy > ids > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19488) Enable CM root based on db parameter, identifying a db as source of replication.
[ https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19488: Fix Version/s: 3.1.0 > Enable CM root based on db parameter, identifying a db as source of > replication. > > > Key: HIVE-19488 > URL: https://issues.apache.org/jira/browse/HIVE-19488 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19488.01.patch, HIVE-19488.02.patch, > HIVE-19488.03.patch, HIVE-19488.04.patch, HIVE-19488.05.patch, > HIVE-19488.06.patch, HIVE-19488.07.patch, HIVE-19488.08-branch-3.patch, > HIVE-19488.08.patch, HIVE-19488.09-branch-3.patch > > > * add a parameter at db level to identify if its a source of replication. > user should set this. > * Enable CM root only for databases that are a source of a replication > policy, for other db's skip the CM root functionality. > * prevent database drop if the parameter indicating its source of a > replication, is set. > * as an upgrade to this version, user should set the property on all > existing database policies, in affect. > * the parameter should be of the form . – repl.source.for : List < policy > ids > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19488) Enable CM root based on db parameter, identifying a db as source of replication.
[ https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501315#comment-16501315 ] Sankar Hariappan commented on HIVE-19488: - [^HIVE-19488.09-branch-3.patch] is committed to branch-3. > Enable CM root based on db parameter, identifying a db as source of > replication. > > > Key: HIVE-19488 > URL: https://issues.apache.org/jira/browse/HIVE-19488 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19488.01.patch, HIVE-19488.02.patch, > HIVE-19488.03.patch, HIVE-19488.04.patch, HIVE-19488.05.patch, > HIVE-19488.06.patch, HIVE-19488.07.patch, HIVE-19488.08-branch-3.patch, > HIVE-19488.08.patch, HIVE-19488.09-branch-3.patch > > > * add a parameter at db level to identify if its a source of replication. > user should set this. > * Enable CM root only for databases that are a source of a replication > policy, for other db's skip the CM root functionality. > * prevent database drop if the parameter indicating its source of a > replication, is set. > * as an upgrade to this version, user should set the property on all > existing database policies, in affect. > * the parameter should be of the form . – repl.source.for : List < policy > ids > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501297#comment-16501297
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926112/HIVE-17683.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14463 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_columnPruning]
(batchId=84)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11516/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11516/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11516/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926112 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19488) Enable CM root based on db parameter, identifying a db as source of replication.
[ https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19488: Target Version/s: 3.1.0 (was: 3.0.0, 3.1.0) > Enable CM root based on db parameter, identifying a db as source of > replication. > > > Key: HIVE-19488 > URL: https://issues.apache.org/jira/browse/HIVE-19488 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19488.01.patch, HIVE-19488.02.patch, > HIVE-19488.03.patch, HIVE-19488.04.patch, HIVE-19488.05.patch, > HIVE-19488.06.patch, HIVE-19488.07.patch, HIVE-19488.08-branch-3.patch, > HIVE-19488.08.patch, HIVE-19488.09-branch-3.patch > > > * add a parameter at db level to identify if its a source of replication. > user should set this. > * Enable CM root only for databases that are a source of a replication > policy, for other db's skip the CM root functionality. > * prevent database drop if the parameter indicating its source of a > replication, is set. > * as an upgrade to this version, user should set the property on all > existing database policies, in affect. > * the parameter should be of the form . – repl.source.for : List < policy > ids > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19739) Bootstrap REPL LOAD to use checkpoints to validate and skip the loaded data/metadata.
[ https://issues.apache.org/jira/browse/HIVE-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19739: Target Version/s: 3.1.0, 4.0.0 (was: 3.0.0, 3.1.0, 4.0.0) > Bootstrap REPL LOAD to use checkpoints to validate and skip the loaded > data/metadata. > - > > Key: HIVE-19739 > URL: https://issues.apache.org/jira/browse/HIVE-19739 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0 > > > Currently. bootstrap REPL LOAD have added checkpoint identifiers in > DB/table/partition object properties once the data/metadata related to the > object is successfully loaded. > If the Db exist and is not empty, then currently we are throwing exception. > But need to support it for the retry scenario after a failure. > If there is a retry of bootstrap load using the same dump, then instead of > throwing error, we should check if any of the tables/partitions are > completely loaded using the checkpoint identifiers. If yes, then skip it or > else drop/create them again. > If the bootstrap load is performed using different dump, then it should throw > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[ https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19708: Fix Version/s: (was: 3.1.0) > Repl copy retrying with cm path even if the failure is due to network issue > --- > > Key: HIVE-19708 > URL: https://issues.apache.org/jira/browse/HIVE-19708 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch, > HIVE-19708.04.patch, HIVE-19708.05.patch, HIVE-19708.06.patch > > > * During repl load > ** for filesystem based copying of file if the copy fails due to a > connection error to source Name Node, we should recreate the filesystem > object. > ** the retry logic for local file copy should be triggered using the > original source file path ( and not the CM root path ) since failure can be > due to network issues between DFSClient and NN. > * When listing files in tables / partition to include them in _files, we > should add retry logic when failure occurs. FileSystem object here also > should be recreated since the existing one might be in inconsistent state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[ https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19708: Fix Version/s: 4.0.0 > Repl copy retrying with cm path even if the failure is due to network issue > --- > > Key: HIVE-19708 > URL: https://issues.apache.org/jira/browse/HIVE-19708 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch, > HIVE-19708.04.patch, HIVE-19708.05.patch, HIVE-19708.06.patch > > > * During repl load > ** for filesystem based copying of file if the copy fails due to a > connection error to source Name Node, we should recreate the filesystem > object. > ** the retry logic for local file copy should be triggered using the > original source file path ( and not the CM root path ) since failure can be > due to network issues between DFSClient and NN. > * When listing files in tables / partition to include them in _files, we > should add retry logic when failure occurs. FileSystem object here also > should be recreated since the existing one might be in inconsistent state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[ https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501287#comment-16501287 ] Sankar Hariappan commented on HIVE-19708: - +1 06.patch is committed to master. Thanks [~maheshk114] for the contribution! > Repl copy retrying with cm path even if the failure is due to network issue > --- > > Key: HIVE-19708 > URL: https://issues.apache.org/jira/browse/HIVE-19708 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch, > HIVE-19708.04.patch, HIVE-19708.05.patch, HIVE-19708.06.patch > > > * During repl load > ** for filesystem based copying of file if the copy fails due to a > connection error to source Name Node, we should recreate the filesystem > object. > ** the retry logic for local file copy should be triggered using the > original source file path ( and not the CM root path ) since failure can be > due to network issues between DFSClient and NN. > * When listing files in tables / partition to include them in _files, we > should add retry logic when failure occurs. FileSystem object here also > should be recreated since the existing one might be in inconsistent state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)
[
https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501285#comment-16501285
]
Saisai Shao commented on HIVE-16391:
Hi [~joshrosen] I'm trying to make the hive changes as you mentioned above
using the new classifier {{core-spark}}. I found one problem about release two
shaded jars (one is hive-exec, another is hive-exec-core-spark). The published
pom file is still reduced pom file, which is related to hive-exec, so when
Spark using hive-exec-core-spark jar, it should explicitly declare all the
transitive dependencies of hive-exec.
I'm not sure if there's a way to publish two pom files mapping to two different
shaded jars, or it is acceptable for Spark to explicitly declare all the
transitive dependencies, like {{core}} classifier you used before?
> Publish proper Hive 1.2 jars (without including all dependencies in uber jar)
> -
>
> Key: HIVE-16391
> URL: https://issues.apache.org/jira/browse/HIVE-16391
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Reporter: Reynold Xin
>Priority: Major
>
> Apache Spark currently depends on a forked version of Apache Hive. AFAIK, the
> only change in the fork is to work around the issue that Hive publishes only
> two sets of jars: one set with no dependency declared, and another with all
> the dependencies included in the published uber jar. That is to say, Hive
> doesn't publish a set of jars with the proper dependencies declared.
> There is general consensus on both sides that we should remove the forked
> Hive.
> The change in the forked version is recorded here
> https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2
> Note that the fork in the past included other fixes but those have all become
> unnecessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster
[ https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19340: --- Attachment: HIVE-19340.06.patch > Disable timeout of transactions opened by replication task at target cluster > > > Key: HIVE-19340 > URL: https://issues.apache.org/jira/browse/HIVE-19340 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-19340.01.patch, HIVE-19340.02.patch, > HIVE-19340.03-branch-3.patch, HIVE-19340.03.patch, > HIVE-19340.04-branch-3.patch, HIVE-19340.06.patch > > > The transactions opened by applying EVENT_OPEN_TXN should never be aborted > automatically due to time-out. Aborting of transaction started by replication > task may leads to inconsistent state at target which needs additional > overhead to clean-up. So, it is proposed to mark the transactions opened by > replication task as special ones and shouldn't be aborted if heart beat is > lost. This helps to ensure all ABORT and COMMIT events will always find the > corresponding txn at target to operate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501283#comment-16501283
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
24s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
32s{color} | {color:blue} standalone-metastore in master has 214 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
43s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 10 new + 323 unchanged - 0
fixed = 333 total (was 323) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
35s{color} | {color:red} ql generated 1 new + 2277 unchanged - 0 fixed = 2278
total (was 2277) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Null pointer dereference of t in
org.apache.hadoop.hive.ql.exec.ExplainTask.getJsonLocks(ExplainWork)
Dereferenced at ExplainTask.java:in
org.apache.hadoop.hive.ql.exec.ExplainTask.getJsonLocks(ExplainWork)
Dereferenced at ExplainTask.java:[line 345] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11516/dev-support/hive-personality.sh
|
| git revision | master / f567a82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus/new-findbugs-ql.html
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11516/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter:
[jira] [Commented] (HIVE-19764) Add --SORT_QUERY_RESULTS to hive-blobstore/map_join.q.out
[
https://issues.apache.org/jira/browse/HIVE-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501268#comment-16501268
]
Hive QA commented on HIVE-19764:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926108/HIVE-19764.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14461 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11515/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11515/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11515/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926108 - PreCommit-HIVE-Build
> Add --SORT_QUERY_RESULTS to hive-blobstore/map_join.q.out
> -
>
> Key: HIVE-19764
> URL: https://issues.apache.org/jira/browse/HIVE-19764
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19764.1.patch
>
>
> Fixes flakiness with this test
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[ https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19358: Resolution: Fixed Fix Version/s: 3.1.0 Target Version/s: (was: 3.0.0) Status: Resolved (was: Patch Available) Pushed to master & branch-3. Thanks, Jesus! > CBO decorrelation logic should generate Hive operators > -- > > Key: HIVE-19358 > URL: https://issues.apache.org/jira/browse/HIVE-19358 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch, > HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch, > HIVE-19358.patch, fix.patch > > > Decorrelation logic may generate logical instances of the operators in the > plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while > costing the tree in the Volcano planner (used in MV rewriting), since logical > operators do not have a cost associated to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19762) Druid Queries containing Joins gives wrong results.
[
https://issues.apache.org/jira/browse/HIVE-19762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19762:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master and branch-3. Thanks, Nishant!
> Druid Queries containing Joins gives wrong results.
>
>
> Key: HIVE-19762
> URL: https://issues.apache.org/jira/browse/HIVE-19762
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19762.patch
>
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19488) Enable CM root based on db parameter, identifying a db as source of replication.
[ https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501255#comment-16501255 ] mahesh kumar behera commented on HIVE-19488: ptest build in master has run successfully. In branch-3 ptest, there are 2 failures which are not related to this patch. > Enable CM root based on db parameter, identifying a db as source of > replication. > > > Key: HIVE-19488 > URL: https://issues.apache.org/jira/browse/HIVE-19488 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19488.01.patch, HIVE-19488.02.patch, > HIVE-19488.03.patch, HIVE-19488.04.patch, HIVE-19488.05.patch, > HIVE-19488.06.patch, HIVE-19488.07.patch, HIVE-19488.08-branch-3.patch, > HIVE-19488.08.patch, HIVE-19488.09-branch-3.patch > > > * add a parameter at db level to identify if its a source of replication. > user should set this. > * Enable CM root only for databases that are a source of a replication > policy, for other db's skip the CM root functionality. > * prevent database drop if the parameter indicating its source of a > replication, is set. > * as an upgrade to this version, user should set the property on all > existing database policies, in affect. > * the parameter should be of the form . – repl.source.for : List < policy > ids > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[ https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501247#comment-16501247 ] Gunther Hagleitner commented on HIVE-18875: --- Left comments on review board. The main things are: a) I'm still hoping to remove the new code that checks cols in convertJoinMapJoin. That seems redundant. b) I'm hoping that new code in gby isn't needed. I left more thoughts on RB. I'm hoping that can be taken out right away. Other than that - the patch looks good. > Enable SMB Join by default in Tez > - > > Key: HIVE-18875 > URL: https://issues.apache.org/jira/browse/HIVE-18875 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18875.1.patch, HIVE-18875.10.patch, > HIVE-18875.11.patch, HIVE-18875.12.patch, HIVE-18875.13.patch, > HIVE-18875.2.patch, HIVE-18875.3.patch, HIVE-18875.4.patch, > HIVE-18875.5.patch, HIVE-18875.6.patch, HIVE-18875.7.patch, > HIVE-18875.8.patch, HIVE-18875.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19764) Add --SORT_QUERY_RESULTS to hive-blobstore/map_join.q.out
[
https://issues.apache.org/jira/browse/HIVE-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501234#comment-16501234
]
Hive QA commented on HIVE-19764:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11515/dev-support/hive-personality.sh
|
| git revision | master / 91cdd4f |
| modules | C: itests/hive-blobstore U: itests/hive-blobstore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11515/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add --SORT_QUERY_RESULTS to hive-blobstore/map_join.q.out
> -
>
> Key: HIVE-19764
> URL: https://issues.apache.org/jira/browse/HIVE-19764
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19764.1.patch
>
>
> Fixes flakiness with this test
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501228#comment-16501228
]
Hive QA commented on HIVE-19358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926107/HIVE-19358.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14461 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11514/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11514/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11514/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926107 - PreCommit-HIVE-Build
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch,
> HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates
[ https://issues.apache.org/jira/browse/HIVE-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19793: Attachment: HIVE-19793.patch > disable LLAP IO batch-to-row wrapper for ACID deletes/updates > - > > Key: HIVE-19793 > URL: https://issues.apache.org/jira/browse/HIVE-19793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19793.patch > > > 1) Batch to row converter doesn't propagate columns correctly because they > are not in the schema. > 2) Then, even if it did, the current VrbCtx model of ACID column propagation > only works with VectorMapOperator. Regular MapOperator has no such context; > the reader ends up storing the vector in some fake temporary ctx. I left a > TODO that combined with a fix to (1) could fix this instead of disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates
[ https://issues.apache.org/jira/browse/HIVE-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19793: Attachment: (was: HIVE-19793.patch) > disable LLAP IO batch-to-row wrapper for ACID deletes/updates > - > > Key: HIVE-19793 > URL: https://issues.apache.org/jira/browse/HIVE-19793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > 1) Batch to row converter doesn't propagate columns correctly because they > are not in the schema. > 2) Then, even if it did, the current VrbCtx model of ACID column propagation > only works with VectorMapOperator. Regular MapOperator has no such context; > the reader ends up storing the vector in some fake temporary ctx. I left a > TODO that combined with a fix to (1) could fix this instead of disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates
[ https://issues.apache.org/jira/browse/HIVE-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19793: Description: 1) Batch to row converter doesn't propagate columns correctly because they are not in the schema. 2) Then, even if it did, the current VrbCtx model of ACID column propagation only works with VectorMapOperator. Regular MapOperator has no such context; the reader ends up storing the vector in some fake temporary ctx. I left a TODO that combined with a fix to (1) could fix this instead of disabling it. > disable LLAP IO batch-to-row wrapper for ACID deletes/updates > - > > Key: HIVE-19793 > URL: https://issues.apache.org/jira/browse/HIVE-19793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19793.patch > > > 1) Batch to row converter doesn't propagate columns correctly because they > are not in the schema. > 2) Then, even if it did, the current VrbCtx model of ACID column propagation > only works with VectorMapOperator. Regular MapOperator has no such context; > the reader ends up storing the vector in some fake temporary ctx. I left a > TODO that combined with a fix to (1) could fix this instead of disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates
[ https://issues.apache.org/jira/browse/HIVE-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19793: Attachment: HIVE-19793.patch > disable LLAP IO batch-to-row wrapper for ACID deletes/updates > - > > Key: HIVE-19793 > URL: https://issues.apache.org/jira/browse/HIVE-19793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19793.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates
[ https://issues.apache.org/jira/browse/HIVE-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19793: Status: Patch Available (was: Open) [~ekoifman] can you take a look? > disable LLAP IO batch-to-row wrapper for ACID deletes/updates > - > > Key: HIVE-19793 > URL: https://issues.apache.org/jira/browse/HIVE-19793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19793.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19793) disable LLAP IO batch-to-row wrapper for ACID deletes/updates
[ https://issues.apache.org/jira/browse/HIVE-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19793: --- > disable LLAP IO batch-to-row wrapper for ACID deletes/updates > - > > Key: HIVE-19793 > URL: https://issues.apache.org/jira/browse/HIVE-19793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19358) CBO decorrelation logic should generate Hive operators
[
https://issues.apache.org/jira/browse/HIVE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501203#comment-16501203
]
Hive QA commented on HIVE-19358:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
29s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 1 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11514/dev-support/hive-personality.sh
|
| git revision | master / 91cdd4f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11514/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11514/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO decorrelation logic should generate Hive operators
> --
>
> Key: HIVE-19358
> URL: https://issues.apache.org/jira/browse/HIVE-19358
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19358.01.patch, HIVE-19358.02.patch,
> HIVE-19358.03.patch, HIVE-19358.04.patch, HIVE-19358.05.patch,
> HIVE-19358.patch, fix.patch
>
>
> Decorrelation logic may generate logical instances of the operators in the
> plan (e.g., LogicalFilter instead of HiveFilter). This leads to errors while
> costing the tree in the Volcano planner (used in MV rewriting), since logical
> operators do not have a cost associated to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19736) Backport part of HIVE-18031: Support alter database events to branch-2
[
https://issues.apache.org/jira/browse/HIVE-19736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501156#comment-16501156
]
Hive QA commented on HIVE-19736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926104/HIVE-19736.1-branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10683 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.getOnlyMaxEvents
(batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11512/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11512/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11512/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926104 - PreCommit-HIVE-Build
> Backport part of HIVE-18031: Support alter database events to branch-2
> --
>
> Key: HIVE-19736
> URL: https://issues.apache.org/jira/browse/HIVE-19736
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 2.4.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Major
> Attachments: HIVE-19736.1-branch-2.patch
>
>
> This is a partial backport of HIVE-18031 which adds support for replication
> alter database events. The partial backport just adds support for alter
> database events but it does not add the replication part to avoid api
> incompatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19772:
-
Attachment: HIVE-19772.2.patch
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
> org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546) at
> org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370) at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:60) at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:90) at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1124)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2373)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:1000)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:977)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19762) Druid Queries containing Joins gives wrong results.
[
https://issues.apache.org/jira/browse/HIVE-19762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501113#comment-16501113
]
Hive QA commented on HIVE-19762:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926099/HIVE-19762.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14461 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druidmini_joins]
(batchId=87)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11511/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11511/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11511/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926099 - PreCommit-HIVE-Build
> Druid Queries containing Joins gives wrong results.
>
>
> Key: HIVE-19762
> URL: https://issues.apache.org/jira/browse/HIVE-19762
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-19762.patch
>
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19738) Update committer-list
[
https://issues.apache.org/jira/browse/HIVE-19738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501101#comment-16501101
]
Teddy Choi commented on HIVE-19738:
---
Thanks, Zoltan!
> Update committer-list
> -
>
> Key: HIVE-19738
> URL: https://issues.apache.org/jira/browse/HIVE-19738
> Project: Hive
> Issue Type: Task
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Trivial
> Attachments: HIVE-19738.patch
>
>
> Adding new entry to committer-list:
> {noformat}
> +
> +tchoi
> +Teddy Choi
> + href="http://hortonworks.com/;>Hortonworks
> +
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19738) Update committer-list
[
https://issues.apache.org/jira/browse/HIVE-19738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-19738:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Update committer-list
> -
>
> Key: HIVE-19738
> URL: https://issues.apache.org/jira/browse/HIVE-19738
> Project: Hive
> Issue Type: Task
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Trivial
> Attachments: HIVE-19738.patch
>
>
> Adding new entry to committer-list:
> {noformat}
> +
> +tchoi
> +Teddy Choi
> + href="http://hortonworks.com/;>Hortonworks
> +
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19792) Enable schema evolution tests for decimal 64
[ https://issues.apache.org/jira/browse/HIVE-19792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19792: - Description: Following tests are disabled in HIVE-19629 as orc ConvertTreeReaderFactory does not handle Decimal64ColumnVectors. This jira is to re-enable those tests after orc supports it. 1) type_change_test_int_vectorized.q 2) type_change_test_int.q 3) orc_schema_evolution_float.q was:Following tests are disabled in HIVE-19629 as orc ConvertTreeReaderFactory does not handle Decimal64ColumnVectors. This jira is to re-enable those tests after orc supports it. > Enable schema evolution tests for decimal 64 > > > Key: HIVE-19792 > URL: https://issues.apache.org/jira/browse/HIVE-19792 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Priority: Major > > Following tests are disabled in HIVE-19629 as orc ConvertTreeReaderFactory > does not handle Decimal64ColumnVectors. This jira is to re-enable those tests > after orc supports it. > 1) type_change_test_int_vectorized.q > 2) type_change_test_int.q > 3) orc_schema_evolution_float.q -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[ https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19629: - Attachment: HIVE-19629.9.patch > Enable Decimal64 reader after orc version upgrade > - > > Key: HIVE-19629 > URL: https://issues.apache.org/jira/browse/HIVE-19629 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch, > HIVE-19629.3.patch, HIVE-19629.4.patch, HIVE-19629.5.patch, > HIVE-19629.6.patch, HIVE-19629.7.patch, HIVE-19629.8.patch, HIVE-19629.9.patch > > > ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for > making use of decimal 64 column vectors. Also LLAP IO will need a new reader > to reader from long stream to decimal 64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[ https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501093#comment-16501093 ] Prasanth Jayachandran commented on HIVE-19629: -- Fixes spark test diffs. > Enable Decimal64 reader after orc version upgrade > - > > Key: HIVE-19629 > URL: https://issues.apache.org/jira/browse/HIVE-19629 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch, > HIVE-19629.3.patch, HIVE-19629.4.patch, HIVE-19629.5.patch, > HIVE-19629.6.patch, HIVE-19629.7.patch, HIVE-19629.8.patch, HIVE-19629.9.patch > > > ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for > making use of decimal 64 column vectors. Also LLAP IO will need a new reader > to reader from long stream to decimal 64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19762) Druid Queries containing Joins gives wrong results.
[
https://issues.apache.org/jira/browse/HIVE-19762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501088#comment-16501088
]
Hive QA commented on HIVE-19762:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
26s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore in master has 214 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11511/dev-support/hive-personality.sh
|
| git revision | master / 85ac544 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11511/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Druid Queries containing Joins gives wrong results.
>
>
> Key: HIVE-19762
> URL: https://issues.apache.org/jira/browse/HIVE-19762
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-19762.patch
>
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties
[jira] [Updated] (HIVE-19334) Use actual file size rather than stats for fetch task optimization with external tables
[ https://issues.apache.org/jira/browse/HIVE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19334: -- Resolution: Fixed Fix Version/s: 4.0.0 3.10 Status: Resolved (was: Patch Available) Committed to master/branch-3 > Use actual file size rather than stats for fetch task optimization with > external tables > --- > > Key: HIVE-19334 > URL: https://issues.apache.org/jira/browse/HIVE-19334 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.10, 4.0.0 > > Attachments: HIVE-19334.1.patch, HIVE-19334.2.patch, > HIVE-19334.3.patch, HIVE-19334.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19332: -- Resolution: Fixed Fix Version/s: 4.0.0 3.1.0 Status: Resolved (was: Patch Available) Committed to master/branch-3 > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19332.1.patch, HIVE-19332.10.patch, > HIVE-19332.2.patch, HIVE-19332.3.patch, HIVE-19332.4.patch, > HIVE-19332.5.patch, HIVE-19332.6.patch, HIVE-19332.7.patch, > HIVE-19332.8.patch, HIVE-19332.9.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18690) Integrate with Spark OutputMetrics
[
https://issues.apache.org/jira/browse/HIVE-18690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501065#comment-16501065
]
Hive QA commented on HIVE-18690:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926097/HIVE-18690.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11509/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11509/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11509/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-05 00:09:27.506
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11509/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-05 00:09:27.508
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
5667af3..bf70bd2 master -> origin/master
5ec8e35..ab1be56 branch-3 -> origin/branch-3
+ git reset --hard HEAD
HEAD is now at 5667af3 HIVE-19690 : multi-insert query with multiple GBY, and
distinct in only some branches can produce incorrect results (Sergey Shelukhin,
reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at bf70bd2 HIVE-19096: query result cache interferes with explain
analyze (Jason Dere, reviewed by Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-05 00:09:29.623
+ rm -rf ../yetus_PreCommit-HIVE-Build-11509
+ mkdir ../yetus_PreCommit-HIVE-Build-11509
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11509
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11509/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/exec/spark/TestSparkStatistics.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/Statistic/SparkStatisticsNames.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/SparkMetricsUtils.java:
does not exist in index
error:
a/spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java:
does not exist in index
error:
a/spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java:
does not exist in index
error:
a/spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java:
does not exist in index
error: patch failed:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/exec/spark/TestSparkStatistics.java:81
Falling back to three-way merge...
Applied patch to
'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/exec/spark/TestSparkStatistics.java'
with conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/Statistic/SparkStatisticsNames.java:41
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/spark/Statistic/SparkStatisticsNames.java'
cleanly.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/SparkMetricsUtils.java:58
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/SparkMetricsUtils.java'
cleanly.
error: patch failed:
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501063#comment-16501063
]
Hive QA commented on HIVE-18533:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926092/HIVE-18533.98.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14462 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11508/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11508/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11508/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926092 - PreCommit-HIVE-Build
> Add option to use InProcessLauncher to submit spark jobs
>
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch,
> HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch,
> HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch,
> HIVE-18533.9.patch, HIVE-18533.91.patch, HIVE-18533.94.patch,
> HIVE-18533.95.patch, HIVE-18533.96.patch, HIVE-18533.97.patch,
> HIVE-18533.98.patch, HIVE-18831.93.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a
> HoS session + debuggability (no need launch a separate process to run a Spark
> app).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19785) Race condition when timeout task is invoked during SASL negotation
[
https://issues.apache.org/jira/browse/HIVE-19785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-19785:
---
Assignee: Aihua Xu
> Race condition when timeout task is invoked during SASL negotation
> --
>
> Key: HIVE-19785
> URL: https://issues.apache.org/jira/browse/HIVE-19785
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
>
> There is a race condition that leads to some extraneous exception messages
> when the timeout task is invoked in {{RpcServer}}.
> If a timeout is triggered by {{RpcServer#registerClient}} the method will
> remove the {{clientId}} from {{pendingClients}}. However, if the SASL
> negotiation is in progress when the timeout task is invoked, then
> {{SaslServerHandler#update}} will throw an {{IllegalArgumentException}}
> complaining that it can't find the {{clientId}} in the map of
> {{pendingClients}}.
> The timeout still succeeds, but the logging is confusing and multiple
> exceptions make this difficult to debug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19422) Create Docker env for running HoS locally
[
https://issues.apache.org/jira/browse/HIVE-19422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-19422:
---
Assignee: Aihua Xu
> Create Docker env for running HoS locally
> -
>
> Key: HIVE-19422
> URL: https://issues.apache.org/jira/browse/HIVE-19422
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
>
> It's really hard to run HoS on a locally installed distribution of Hive built
> using {{mvn package}}. The only way developers can really run HoS is via the
> Spark CLI Drivers. However, there are occasions where devs need to run HoS on
> a proper Hive distribution in order to validate some behavior.
> The docker image will also be useful to users who want to play around with
> HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[ https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501033#comment-16501033 ] Teddy Choi commented on HIVE-19723: --- I'm sorry. I'll fix it as soon as possible. > Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)" > - > > Key: HIVE-19723 > URL: https://issues.apache.org/jira/browse/HIVE-19723 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19723.1.patch, HIVE-19732.2.patch > > > Spark's Arrow support only provides Timestamp at MICROSECOND granularity. > Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND. > The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need > to change the assertion to test microsecond. And we'll need to add this to > documentation on supported datatypes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501009#comment-16501009 ] Laszlo Bodor commented on HIVE-17852: - build failure, merged again in 06.patch > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch, > HIVE-17852.06.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-17852: Attachment: HIVE-17852.06.patch > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch, > HIVE-17852.06.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19096) query result cache interferes with explain analyze
[
https://issues.apache.org/jira/browse/HIVE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-19096:
--
Resolution: Fixed
Fix Version/s: 4.0.0
3.1.0
Status: Resolved (was: Patch Available)
Committed to master/branch-3
> query result cache interferes with explain analyze
> ---
>
> Key: HIVE-19096
> URL: https://issues.apache.org/jira/browse/HIVE-19096
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Jason Dere
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19096.1.patch, HIVE-19096.2.patch,
> HIVE-19096.3.patch
>
>
> if result cache is active; the explain analyze doesn't really return usefull
> informations; even for unseen queries the result is like this:
> {code}
> ++
> |Explain |
> ++
> | Stage-0|
> | Fetch Operator |
> | Cached Query Result:true,limit:-1 |
> ||
> ++
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-17852: Attachment: (was: HIVE-17852.06.patch) > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-17852: Attachment: HIVE-17852.06.patch > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch, > HIVE-17852.06.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19791) Modify TableDesc to contain the catalog
[ https://issues.apache.org/jira/browse/HIVE-19791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-19791: - > Modify TableDesc to contain the catalog > --- > > Key: HIVE-19791 > URL: https://issues.apache.org/jira/browse/HIVE-19791 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > TableDesc currently only contains a table's database and tablename. It needs > to also have the catalog name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16505) Support "unknown" boolean truth value
[
https://issues.apache.org/jira/browse/HIVE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500987#comment-16500987
]
Laszlo Bodor commented on HIVE-16505:
-
input31 failure is unrelated, but let's try again with 03.patch
> Support "unknown" boolean truth value
> -
>
> Key: HIVE-16505
> URL: https://issues.apache.org/jira/browse/HIVE-16505
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-16505.01.patch, HIVE-16505.02.patch,
> HIVE-16505.03.patch
>
>
> according to the standard, boolean truth value might be:
> {{TRUE|FALSE|UNKNOWN}}.
> similar queries to the following should be supported:
> {code:java}
> select 1 where null is unknown;
> select 1 where (select cast(null as boolean) ) is unknown;
> {code}
> "unknown" behaves similarily to null. {{(null=null) is null}}
>
> "All boolean values and SQL truth values are comparable and all are
> assignable to a site of type boolean. The value True is greater than the
> value False, and any comparison involving the null value or an Unknown truth
> value will return an Unknown result. The values True and False may be
> assigned to any site having a boolean data type; assignment of Unknown, or
> the null value, is subject to the nullability characteristic of the target."
>
> *Truth table for the AND boolean operator*
> AND True False Unknown
> True True False Unknown
> False False False False
> Unknown Unknown False Unknown
> *Truth table for the OR boolean operator*
> OR True False Unknown
> True True True True
> False True False Unknown
> Unknown True Unknown Unknown
> *Truth table for the IS boolean operator*
> IS TRUE FALSE UNKNOWN
> True True False False
> False False True False
> Unknown False False True
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16505) Support "unknown" boolean truth value
[
https://issues.apache.org/jira/browse/HIVE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-16505:
Attachment: HIVE-16505.03.patch
> Support "unknown" boolean truth value
> -
>
> Key: HIVE-16505
> URL: https://issues.apache.org/jira/browse/HIVE-16505
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-16505.01.patch, HIVE-16505.02.patch,
> HIVE-16505.03.patch
>
>
> according to the standard, boolean truth value might be:
> {{TRUE|FALSE|UNKNOWN}}.
> similar queries to the following should be supported:
> {code:java}
> select 1 where null is unknown;
> select 1 where (select cast(null as boolean) ) is unknown;
> {code}
> "unknown" behaves similarily to null. {{(null=null) is null}}
>
> "All boolean values and SQL truth values are comparable and all are
> assignable to a site of type boolean. The value True is greater than the
> value False, and any comparison involving the null value or an Unknown truth
> value will return an Unknown result. The values True and False may be
> assigned to any site having a boolean data type; assignment of Unknown, or
> the null value, is subject to the nullability characteristic of the target."
>
> *Truth table for the AND boolean operator*
> AND True False Unknown
> True True False Unknown
> False False False False
> Unknown Unknown False Unknown
> *Truth table for the OR boolean operator*
> OR True False Unknown
> True True True True
> False True False Unknown
> Unknown True Unknown Unknown
> *Truth table for the IS boolean operator*
> IS TRUE FALSE UNKNOWN
> True True False False
> False False True False
> Unknown False False True
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19334) Use actual file size rather than stats for fetch task optimization with external tables
[ https://issues.apache.org/jira/browse/HIVE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19334: -- Target Version/s: 3.1.0 > Use actual file size rather than stats for fetch task optimization with > external tables > --- > > Key: HIVE-19334 > URL: https://issues.apache.org/jira/browse/HIVE-19334 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19334.1.patch, HIVE-19334.2.patch, > HIVE-19334.3.patch, HIVE-19334.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19332: -- Target Version/s: 3.1.0 > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.10.patch, > HIVE-19332.2.patch, HIVE-19332.3.patch, HIVE-19332.4.patch, > HIVE-19332.5.patch, HIVE-19332.6.patch, HIVE-19332.7.patch, > HIVE-19332.8.patch, HIVE-19332.9.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500969#comment-16500969
]
Hive QA commented on HIVE-18533:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
27s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} spark-client in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} spark-client: The patch generated 11 new + 25
unchanged - 9 fixed = 36 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} spark-client generated 2 new + 10 unchanged - 5 fixed
= 12 total (was 15) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:spark-client |
| | Class org.apache.hive.spark.client.SparkLauncherSparkClient defines
non-transient non-serializable instance field sparkLauncher In
SparkLauncherSparkClient.java:instance field sparkLauncher In
SparkLauncherSparkClient.java |
| | org.apache.spark.launcher.InProcessLauncher stored into non-transient
field SparkLauncherSparkClient.sparkLauncher At
SparkLauncherSparkClient.java:SparkLauncherSparkClient.sparkLauncher At
SparkLauncherSparkClient.java:[line 182] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11508/dev-support/hive-personality.sh
|
| git revision | master / 5667af3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11508/yetus/diff-checkstyle-spark-client.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11508/yetus/new-findbugs-spark-client.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11508/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests itests/qtest-spark ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11508/yetus.txt |
| Powered by | Apache Yetus
[jira] [Updated] (HIVE-19467) Make storage format configurable for temp tables created using LLAP external client
[ https://issues.apache.org/jira/browse/HIVE-19467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19467: -- Fix Version/s: 3.1.0 > Make storage format configurable for temp tables created using LLAP external > client > --- > > Key: HIVE-19467 > URL: https://issues.apache.org/jira/browse/HIVE-19467 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19467.1.patch > > > Temp tables created for complex queries when using the LLAP external client > are created using the default storage format. Default to orc, and make > configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19467) Make storage format configurable for temp tables created using LLAP external client
[ https://issues.apache.org/jira/browse/HIVE-19467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500966#comment-16500966 ] Jason Dere commented on HIVE-19467: --- Committed to branch-3 > Make storage format configurable for temp tables created using LLAP external > client > --- > > Key: HIVE-19467 > URL: https://issues.apache.org/jira/browse/HIVE-19467 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19467.1.patch > > > Temp tables created for complex queries when using the LLAP external client > are created using the default storage format. Default to orc, and make > configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19053:
Status: Patch Available (was: Open)
patch-1: simple improvement to handle InterruptedException and
ExecutionException and throw different errors.
[~stakiar] can you take a look if it makes sense?
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19467) Make storage format configurable for temp tables created using LLAP external client
[ https://issues.apache.org/jira/browse/HIVE-19467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500953#comment-16500953 ] Jason Dere commented on HIVE-19467: --- Will add to branch-3 > Make storage format configurable for temp tables created using LLAP external > client > --- > > Key: HIVE-19467 > URL: https://issues.apache.org/jira/browse/HIVE-19467 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19467.1.patch > > > Temp tables created for complex queries when using the LLAP external client > are created using the default storage format. Default to orc, and make > configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19467) Make storage format configurable for temp tables created using LLAP external client
[ https://issues.apache.org/jira/browse/HIVE-19467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19467: -- Target Version/s: 3.1.0 > Make storage format configurable for temp tables created using LLAP external > client > --- > > Key: HIVE-19467 > URL: https://issues.apache.org/jira/browse/HIVE-19467 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19467.1.patch > > > Temp tables created for complex queries when using the LLAP external client > are created using the default storage format. Default to orc, and make > configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-19053:
---
Assignee: Aihua Xu
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19053:
Attachment: HIVE-19053.1.patch
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19720) backport multiple MM commits to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19720: Description: To avoid chained test runs of branch-3 backporting one by one, I will run HiveQA on an epic combined patch, then commit patches w/proper commit separation via cherry-pick: 0930aec69b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey Shelukhin, reviewed by Gunther Hagleitner) 99a2b8bd6b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey Shelukhin, reviewed by Gunther Hagleitner) ADDENDUM 7ebcdeb951 HIVE-17657 : export/import for MM tables is broken (Sergey Shelukhin, reviewed by Eugene Koifman) 8db979f1ff (part not previously backported) HIVE-19476: Fix failures in TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and TestCopyUtils (Sankar Hariappan, reviewed by Sergey Shelukhin) f4352e5339 HIVE-19258 : add originals support to MM tables (and make the conversion a metadata only operation) (Sergey Shelukhin, reviewed by Jason Dere) 36d66f0cf27 HIVE-19643 : MM table conversion doesn't need full ACID structure checks (Sergey Shelukhin, reviewed by Eugene Koifman) was: To avoid chained test runs of branch-3 backporting one by one, I will run HiveQA on an epic combined patch, then commit patches w/proper commit separation via cherry-pick: 0930aec69b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey Shelukhin, reviewed by Gunther Hagleitner) 99a2b8bd6b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey Shelukhin, reviewed by Gunther Hagleitner) ADDENDUM 7ebcdeb951 HIVE-17657 : export/import for MM tables is broken (Sergey Shelukhin, reviewed by Eugene Koifman) 8db979f1ff (part not previously backported) HIVE-19476: Fix failures in TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and TestCopyUtils (Sankar Hariappan, reviewed by Sergey Shelukhin) f4352e5339 HIVE-19258 : add originals support to MM tables (and make the conversion a metadata only operation) (Sergey Shelukhin, reviewed by Jason Dere) Need to add: 36d66f0cf27 HIVE-19643 : MM table conversion doesn't need full ACID structure checks (Sergey Shelukhin, reviewed by Eugene Koifman) > backport multiple MM commits to branch-3 > > > Key: HIVE-19720 > URL: https://issues.apache.org/jira/browse/HIVE-19720 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19720.01-branch-3.patch, > HIVE-19720.02-branch-3.patch, HIVE-19720.03-branch-3.patch, > HIVE-19720.04-branch-3.patch, HIVE-19720.05-branch-3.patch > > > To avoid chained test runs of branch-3 backporting one by one, I will run > HiveQA on an epic combined patch, then commit patches w/proper commit > separation via cherry-pick: > 0930aec69b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) > 99a2b8bd6b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) ADDENDUM > 7ebcdeb951 HIVE-17657 : export/import for MM tables is broken (Sergey > Shelukhin, reviewed by Eugene Koifman) > 8db979f1ff (part not previously backported) HIVE-19476: Fix failures in > TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and > TestCopyUtils (Sankar Hariappan, reviewed by Sergey Shelukhin) > f4352e5339 HIVE-19258 : add originals support to MM tables (and make the > conversion a metadata only operation) (Sergey Shelukhin, reviewed by Jason > Dere) > 36d66f0cf27 HIVE-19643 : MM table conversion doesn't need full ACID structure > checks (Sergey Shelukhin, reviewed by Eugene Koifman) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19720) backport multiple MM commits to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500937#comment-16500937 ] Sergey Shelukhin commented on HIVE-19720: - If some more failures like these happen on the next run, I will just push these patches since branch-3 test don't appear to be in a good state.. none of these are relevant. > backport multiple MM commits to branch-3 > > > Key: HIVE-19720 > URL: https://issues.apache.org/jira/browse/HIVE-19720 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19720.01-branch-3.patch, > HIVE-19720.02-branch-3.patch, HIVE-19720.03-branch-3.patch, > HIVE-19720.04-branch-3.patch, HIVE-19720.05-branch-3.patch > > > To avoid chained test runs of branch-3 backporting one by one, I will run > HiveQA on an epic combined patch, then commit patches w/proper commit > separation via cherry-pick: > 0930aec69b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) > 99a2b8bd6b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) ADDENDUM > 7ebcdeb951 HIVE-17657 : export/import for MM tables is broken (Sergey > Shelukhin, reviewed by Eugene Koifman) > 8db979f1ff (part not previously backported) HIVE-19476: Fix failures in > TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and > TestCopyUtils (Sankar Hariappan, reviewed by Sergey Shelukhin) > f4352e5339 HIVE-19258 : add originals support to MM tables (and make the > conversion a metadata only operation) (Sergey Shelukhin, reviewed by Jason > Dere) > Need to add: > 36d66f0cf27 HIVE-19643 : MM table conversion doesn't need full ACID structure > checks (Sergey Shelukhin, reviewed by Eugene Koifman) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19788) Flaky test: HCatalog-PigServer tests
[
https://issues.apache.org/jira/browse/HIVE-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19788:
Attachment: HIVE-19788.1.patch
> Flaky test: HCatalog-PigServer tests
>
>
> Key: HIVE-19788
> URL: https://issues.apache.org/jira/browse/HIVE-19788
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19788.1.patch
>
>
> {{TestHCatLoaderComplexSchema}} is still flaky because its writing to
> {{/tmp/}} - HIVE-19731 was meant to fix this, and that fixes the tmp dir for
> any Hive queries, but these tests run a bunch of Pig queries too, and those
> queries write to {{/tmp/}} - we need to pass in custom configs to the
> embedded {{PigServer}} that is being created as part of these tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19788) Flaky test: HCatalog-PigServer tests
[
https://issues.apache.org/jira/browse/HIVE-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19788:
Status: Patch Available (was: Open)
> Flaky test: HCatalog-PigServer tests
>
>
> Key: HIVE-19788
> URL: https://issues.apache.org/jira/browse/HIVE-19788
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19788.1.patch
>
>
> {{TestHCatLoaderComplexSchema}} is still flaky because its writing to
> {{/tmp/}} - HIVE-19731 was meant to fix this, and that fixes the tmp dir for
> any Hive queries, but these tests run a bunch of Pig queries too, and those
> queries write to {{/tmp/}} - we need to pass in custom configs to the
> embedded {{PigServer}} that is being created as part of these tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[
https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500927#comment-16500927
]
Hive QA commented on HIVE-19708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926282/HIVE-19708.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14459 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11507/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11507/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11507/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926282 - PreCommit-HIVE-Build
> Repl copy retrying with cm path even if the failure is due to network issue
> ---
>
> Key: HIVE-19708
> URL: https://issues.apache.org/jira/browse/HIVE-19708
> Project: Hive
> Issue Type: Task
> Components: Hive, HiveServer2, repl
>Affects Versions: 3.1.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch,
> HIVE-19708.04.patch, HIVE-19708.05.patch, HIVE-19708.06.patch
>
>
> * During repl load
> ** for filesystem based copying of file if the copy fails due to a
> connection error to source Name Node, we should recreate the filesystem
> object.
> ** the retry logic for local file copy should be triggered using the
> original source file path ( and not the CM root path ) since failure can be
> due to network issues between DFSClient and NN.
> * When listing files in tables / partition to include them in _files, we
> should add retry logic when failure occurs. FileSystem object here also
> should be recreated since the existing one might be in inconsistent state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19788) Flaky test: HCatalog-PigServer tests
[
https://issues.apache.org/jira/browse/HIVE-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19788:
Summary: Flaky test: HCatalog-PigServer tests (was: Flaky test:
TestHCatLoaderComplexSchema)
> Flaky test: HCatalog-PigServer tests
>
>
> Key: HIVE-19788
> URL: https://issues.apache.org/jira/browse/HIVE-19788
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{TestHCatLoaderComplexSchema}} is still flaky because its writing to
> {{/tmp/}} - HIVE-19731 was meant to fix this, and that fixes the tmp dir for
> any Hive queries, but these tests run a bunch of Pig queries too, and those
> queries write to {{/tmp/}} - we need to pass in custom configs to the
> embedded {{PigServer}} that is being created as part of these tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19778) Flaky test: TestCliDriver#input31
[
https://issues.apache.org/jira/browse/HIVE-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500924#comment-16500924
]
Jason Dere commented on HIVE-19778:
---
That's fine .. maybe we can create another item to troubleshoot for later.
+1
> Flaky test: TestCliDriver#input31
> -
>
> Key: HIVE-19778
> URL: https://issues.apache.org/jira/browse/HIVE-19778
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19778.patch
>
>
> Noticed this one has been failing occasionally on precommit test runs.
> {noformat}
> Running: diff -a
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/itests/qtest/target/qfile-results/clientpositive/input31.q.out
>
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/ql/src/test/results/clientpositive/input31.q.out
> 128c128
> < 496
> ---
> > 242
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19720) backport multiple MM commits to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19720: Attachment: HIVE-19720.05-branch-3.patch > backport multiple MM commits to branch-3 > > > Key: HIVE-19720 > URL: https://issues.apache.org/jira/browse/HIVE-19720 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19720.01-branch-3.patch, > HIVE-19720.02-branch-3.patch, HIVE-19720.03-branch-3.patch, > HIVE-19720.04-branch-3.patch, HIVE-19720.05-branch-3.patch > > > To avoid chained test runs of branch-3 backporting one by one, I will run > HiveQA on an epic combined patch, then commit patches w/proper commit > separation via cherry-pick: > 0930aec69b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) > 99a2b8bd6b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) ADDENDUM > 7ebcdeb951 HIVE-17657 : export/import for MM tables is broken (Sergey > Shelukhin, reviewed by Eugene Koifman) > 8db979f1ff (part not previously backported) HIVE-19476: Fix failures in > TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and > TestCopyUtils (Sankar Hariappan, reviewed by Sergey Shelukhin) > f4352e5339 HIVE-19258 : add originals support to MM tables (and make the > conversion a metadata only operation) (Sergey Shelukhin, reviewed by Jason > Dere) > Need to add: > 36d66f0cf27 HIVE-19643 : MM table conversion doesn't need full ACID structure > checks (Sergey Shelukhin, reviewed by Eugene Koifman) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19690: Resolution: Fixed Fix Version/s: 4.0.0 3.1.0 Status: Resolved (was: Patch Available) Committed to master and branch-3 > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch, > HIVE-19690.03.patch, HIVE-19690.04.patch, HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[
https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500903#comment-16500903
]
Bryan Cutler commented on HIVE-19723:
-
> My understanding is that since the primary use-case for ArrowUtils is Python
> integration, some of the conversions are currently somewhat particular for
> Python. Perhaps Python/Pandas only supports MICROSECOND timestamps.
Python, with pandas and pyarrow, supports timestamps down to nanoseconds. The
reason for for using microseconds in Spark {{ArrowUtils}} is to match Sparks
internal representation, which is in microseconds. This way avoids any further
conversions once read into the Spark JVM.
> Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
> -
>
> Key: HIVE-19723
> URL: https://issues.apache.org/jira/browse/HIVE-19723
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19723.1.patch, HIVE-19732.2.patch
>
>
> Spark's Arrow support only provides Timestamp at MICROSECOND granularity.
> Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND.
> The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need
> to change the assertion to test microsecond. And we'll need to add this to
> documentation on supported datatypes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500880#comment-16500880 ] Ashutosh Chauhan commented on HIVE-19332: - +1 Agreed no need for new run. Push it in. > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.10.patch, > HIVE-19332.2.patch, HIVE-19332.3.patch, HIVE-19332.4.patch, > HIVE-19332.5.patch, HIVE-19332.6.patch, HIVE-19332.7.patch, > HIVE-19332.8.patch, HIVE-19332.9.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19789) reenable orc_llap test
[ https://issues.apache.org/jira/browse/HIVE-19789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19789: --- > reenable orc_llap test > -- > > Key: HIVE-19789 > URL: https://issues.apache.org/jira/browse/HIVE-19789 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Matt McCline >Priority: Major > > Test has been disabled, looks like by mistake (or due to some issue with the > patch there that was never addressed), in HIVE-11394. > It needs to be reenabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-11394) Enhance EXPLAIN display for vectorization
[
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500871#comment-16500871
]
Sergey Shelukhin commented on HIVE-11394:
-
This patch has permanently disabled orc_llap. This test needs to be reenabled.
> Enhance EXPLAIN display for vectorization
> -
>
> Key: HIVE-11394
> URL: https://issues.apache.org/jira/browse/HIVE-11394
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Labels: TODOC2.2
> Fix For: 2.3.0
>
> Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch,
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch,
> HIVE-11394.06.patch, HIVE-11394.07.patch, HIVE-11394.08.patch,
> HIVE-11394.09.patch, HIVE-11394.091.patch, HIVE-11394.092.patch,
> HIVE-11394.093.patch, HIVE-11394.094.patch, HIVE-11394.095.patch,
> HIVE-11394.096.patch, HIVE-11394.097.patch, HIVE-11394.098.patch,
> HIVE-11394.099.patch, HIVE-11394.0991.patch, HIVE-11394.0992.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\]
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators. E.g. Filter
> Vectorization. It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions. E.g.
> predicateExpression. It includes all information of SUMMARY and OPERATOR,
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> ---
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION
> SUMMARY.
> Under Reducer 3’s "Reduce Vectorization:" you’ll see
> notVectorizedReason: Aggregation Function UDF avg parameter expression for
> GROUPBY operator: Data type struct of
> Column\[VALUE._col2\] not supported
> For Reducer 2’s "Reduce Vectorization:" you’ll see "groupByVectorOutput:":
> "false" which says a node has a GROUP BY with an AVG or some other aggregator
> that outputs a non-PRIMITIVE type (e.g. STRUCT) and all downstream operators
> are row-mode. I.e. not vector output.
> If "usesVectorUDFAdaptor:": "false" were true, it would say there was at
> least one vectorized expression is using VectorUDFAdaptor.
> And, "allNative:": "false" will be true when all operators are native.
> Today, GROUP BY and FILE SINK are not native. MAP JOIN and REDUCE SINK are
> conditionally native. FILTER and SELECT are native.
> {code}
> PLAN VECTORIZATION:
> enabled: true
> enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> ...
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: alltypesorc
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: cint (type: int)
> outputColumnNames: cint
> Statistics: Num rows: 12288 Data size: 36696 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> keys: cint (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: int)
> sort order: +
> Map-reduce partition columns: _col0 (type: int)
> Statistics: Num rows: 5775 Data size: 17248 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map Vectorization:
> enabled: true
> enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true
>
[jira] [Commented] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[
https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500873#comment-16500873
]
Hive QA commented on HIVE-19708:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 1 new + 10 unchanged - 0 fixed
= 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11507/dev-support/hive-personality.sh
|
| git revision | master / 43e331e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11507/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11507/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl copy retrying with cm path even if the failure is due to network issue
> ---
>
> Key: HIVE-19708
> URL: https://issues.apache.org/jira/browse/HIVE-19708
> Project: Hive
> Issue Type: Task
> Components: Hive, HiveServer2, repl
>Affects Versions: 3.1.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch,
> HIVE-19708.04.patch, HIVE-19708.05.patch, HIVE-19708.06.patch
>
>
> * During repl load
> ** for filesystem based copying of file if the copy fails due to a
> connection error to source Name Node, we should recreate the filesystem
> object.
> ** the retry logic for local file copy should be triggered using the
> original source file path ( and not the CM root path ) since failure can be
> due to network issues between DFSClient and NN.
> * When
[jira] [Commented] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500870#comment-16500870 ] Jason Dere commented on HIVE-19332: --- Updated patch to change areBasicStatsUptoDateForPlanning/areColumnStatsUptoDateForPlanning() to areBasicStatsUptoDateForQueryAnswering/areColumnStatsUptoDateForQueryAnswering(). [~ashutoshc] is this ok now? Probably doesn't need another PTest run since the only change is to those method names. > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.10.patch, > HIVE-19332.2.patch, HIVE-19332.3.patch, HIVE-19332.4.patch, > HIVE-19332.5.patch, HIVE-19332.6.patch, HIVE-19332.7.patch, > HIVE-19332.8.patch, HIVE-19332.9.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500868#comment-16500868 ] Sergey Shelukhin commented on HIVE-19690: - I'm disabling input31 in HIVE-19778, flaky test > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch, > HIVE-19690.03.patch, HIVE-19690.04.patch, HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500864#comment-16500864 ] Vihang Karajgaonkar commented on HIVE-19685: Looks like Todd is busy. Rebased the patch and reattached. > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: HIVE-19685.02.patch, hive-19685.patch, hive-19685.patch, > trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19778) Flaky test: TestCliDriver#input31
[
https://issues.apache.org/jira/browse/HIVE-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19778:
Status: Patch Available (was: Open)
Disabling for now. [~jdere] does this make sense?
> Flaky test: TestCliDriver#input31
> -
>
> Key: HIVE-19778
> URL: https://issues.apache.org/jira/browse/HIVE-19778
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19778.patch
>
>
> Noticed this one has been failing occasionally on precommit test runs.
> {noformat}
> Running: diff -a
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/itests/qtest/target/qfile-results/clientpositive/input31.q.out
>
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/ql/src/test/results/clientpositive/input31.q.out
> 128c128
> < 496
> ---
> > 242
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19778) Flaky test: TestCliDriver#input31
[
https://issues.apache.org/jira/browse/HIVE-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19778:
Attachment: HIVE-19778.patch
> Flaky test: TestCliDriver#input31
> -
>
> Key: HIVE-19778
> URL: https://issues.apache.org/jira/browse/HIVE-19778
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19778.patch
>
>
> Noticed this one has been failing occasionally on precommit test runs.
> {noformat}
> Running: diff -a
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/itests/qtest/target/qfile-results/clientpositive/input31.q.out
>
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/ql/src/test/results/clientpositive/input31.q.out
> 128c128
> < 496
> ---
> > 242
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19332: -- Attachment: HIVE-19332.10.patch > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.10.patch, > HIVE-19332.2.patch, HIVE-19332.3.patch, HIVE-19332.4.patch, > HIVE-19332.5.patch, HIVE-19332.6.patch, HIVE-19332.7.patch, > HIVE-19332.8.patch, HIVE-19332.9.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19778) Flaky test: TestCliDriver#input31
[
https://issues.apache.org/jira/browse/HIVE-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-19778:
---
Assignee: Sergey Shelukhin
> Flaky test: TestCliDriver#input31
> -
>
> Key: HIVE-19778
> URL: https://issues.apache.org/jira/browse/HIVE-19778
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Noticed this one has been failing occasionally on precommit test runs.
> {noformat}
> Running: diff -a
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/itests/qtest/target/qfile-results/clientpositive/input31.q.out
>
> /home/hiveptest/35.193.227.186-hiveptest-1/apache-github-source-source/ql/src/test/results/clientpositive/input31.q.out
> 128c128
> < 496
> ---
> > 242
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-19685: -- Assignee: Todd Lipcon (was: Vihang Karajgaonkar) > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: HIVE-19685.02.patch, hive-19685.patch, hive-19685.patch, > trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-19685: --- Attachment: HIVE-19685.02.patch > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19685.02.patch, hive-19685.patch, hive-19685.patch, > trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-19685: -- Assignee: Vihang Karajgaonkar (was: Todd Lipcon) > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19685.02.patch, hive-19685.patch, hive-19685.patch, > trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent
[
https://issues.apache.org/jira/browse/HIVE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18766:
Status: Patch Available (was: Open)
patch-1: when the RemoteDriver sends the error to the HS2, this patch adds the
logic to wait for a configured time (we reuse the same configuration
hive.spark.client.future.timeout).
[~stakiar] Can you help take a look at this? It's a simple patch so I didn't
create CR.
> Race condition during shutdown of RemoteDriver, error messages aren't always
> sent
> -
>
> Key: HIVE-18766
> URL: https://issues.apache.org/jira/browse/HIVE-18766
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18766.1.patch
>
>
> When we send an error during shutdown of the {{RemoteDriver}}, we don't wait
> for the error message to be sent. We just send it and then close the RPC
> channel. For a graceful shutdown, that doesn't seem ideal. We should at least
> wait a bit for the RPC message to be sent before shutting things down. I ran
> some tests locally and its pretty easy to hit a situation where the error
> message doesn't even get sent to the {{SparkClientImpl}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent
[
https://issues.apache.org/jira/browse/HIVE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18766:
Attachment: HIVE-18766.1.patch
> Race condition during shutdown of RemoteDriver, error messages aren't always
> sent
> -
>
> Key: HIVE-18766
> URL: https://issues.apache.org/jira/browse/HIVE-18766
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18766.1.patch
>
>
> When we send an error during shutdown of the {{RemoteDriver}}, we don't wait
> for the error message to be sent. We just send it and then close the RPC
> channel. For a graceful shutdown, that doesn't seem ideal. We should at least
> wait a bit for the RPC message to be sent before shutting things down. I ran
> some tests locally and its pretty easy to hit a situation where the error
> message doesn't even get sent to the {{SparkClientImpl}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500844#comment-16500844 ] Sergey Shelukhin commented on HIVE-19418: - Cannot repro this after a few runs and the logs are gone. Trying again... > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.05.patch, > HIVE-19418.06.patch, HIVE-19418.06.patch, HIVE-19418.07.patch, > HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19418: Attachment: HIVE-19418.07.patch > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.05.patch, > HIVE-19418.06.patch, HIVE-19418.06.patch, HIVE-19418.07.patch, > HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500840#comment-16500840
]
Hive QA commented on HIVE-19079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926089/HIVE-19079.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14460 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926089 - PreCommit-HIVE-Build
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, Spark Collapse Truncated Query.png,
> Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19788) Flaky test: TestHCatLoaderComplexSchema
[
https://issues.apache.org/jira/browse/HIVE-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500834#comment-16500834
]
Sahil Takiar commented on HIVE-19788:
-
Need to do something like HIVE-19594 / HIVE-19620
> Flaky test: TestHCatLoaderComplexSchema
> ---
>
> Key: HIVE-19788
> URL: https://issues.apache.org/jira/browse/HIVE-19788
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{TestHCatLoaderComplexSchema}} is still flaky because its writing to
> {{/tmp/}} - HIVE-19731 was meant to fix this, and that fixes the tmp dir for
> any Hive queries, but these tests run a bunch of Pig queries too, and those
> queries write to {{/tmp/}} - we need to pass in custom configs to the
> embedded {{PigServer}} that is being created as part of these tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19788) Flaky test: TestHCatLoaderComplexSchema
[
https://issues.apache.org/jira/browse/HIVE-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-19788:
---
> Flaky test: TestHCatLoaderComplexSchema
> ---
>
> Key: HIVE-19788
> URL: https://issues.apache.org/jira/browse/HIVE-19788
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{TestHCatLoaderComplexSchema}} is still flaky because its writing to
> {{/tmp/}} - HIVE-19731 was meant to fix this, and that fixes the tmp dir for
> any Hive queries, but these tests run a bunch of Pig queries too, and those
> queries write to {{/tmp/}} - we need to pass in custom configs to the
> embedded {{PigServer}} that is being created as part of these tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19759) Flaky test: TestRpc#testServerPort
[
https://issues.apache.org/jira/browse/HIVE-19759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19759:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Peter for the review!
> Flaky test: TestRpc#testServerPort
> --
>
> Key: HIVE-19759
> URL: https://issues.apache.org/jira/browse/HIVE-19759
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19759.1.patch, HIVE-19759.2.patch,
> HIVE-19759.3.patch
>
>
> Conflict when opening ports:
> {code}
> java.io.IOException: Remote Spark Driver RPC Server cannot bind to any of the
> configured ports: [65535, 21, 22, 23]
> at
> org.apache.hive.spark.client.rpc.RpcServer.bindServerPort(RpcServer.java:150)
> at org.apache.hive.spark.client.rpc.RpcServer.(RpcServer.java:117)
> at
> org.apache.hive.spark.client.rpc.TestRpc.testServerPort(TestRpc.java:209)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500800#comment-16500800
]
Hive QA commented on HIVE-19079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 2 new + 142 unchanged - 0
fixed = 144 total (was 142) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11506/dev-support/hive-personality.sh
|
| git revision | master / d682ca9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11506/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11506/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11506/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, Spark Collapse Truncated Query.png,
> Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19752) PerfLogger integration for critical Hive-on-S3 paths
[
https://issues.apache.org/jira/browse/HIVE-19752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500799#comment-16500799
]
Sahil Takiar commented on HIVE-19752:
-
Thanks for taking a look [~vihangk1]. I double checked, and the {{HashMap}} in
each {{PerfLogger}} gets reset by the {{Driver}} so we shouldn't be leaking
them. Each {{PerfLogger}} is {{ThreadLocal}} too, so it should be garbage
collected after each session is closed.
> PerfLogger integration for critical Hive-on-S3 paths
>
>
> Key: HIVE-19752
> URL: https://issues.apache.org/jira/browse/HIVE-19752
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19752.1.patch
>
>
> There are several areas where Hive performs a lot of S3 operations, it would
> be good to add PerfLogger statements around this so we can measure how long
> they take.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19752) PerfLogger integration for critical Hive-on-S3 paths
[
https://issues.apache.org/jira/browse/HIVE-19752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500792#comment-16500792
]
Vihang Karajgaonkar commented on HIVE-19752:
Ah, I see. Thanks [~stakiar] LGTM +1 On a side note, I am curious to understand
the behavior of Metrics scope when there are exceptions and we don't call the
{{PerfLogEnd}}. I hope we don't leak them.
> PerfLogger integration for critical Hive-on-S3 paths
>
>
> Key: HIVE-19752
> URL: https://issues.apache.org/jira/browse/HIVE-19752
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19752.1.patch
>
>
> There are several areas where Hive performs a lot of S3 operations, it would
> be good to add PerfLogger statements around this so we can measure how long
> they take.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19605) TAB_COL_STATS table has no index on db/table name
[ https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-19605: --- Attachment: HIVE-19605.02.patch > TAB_COL_STATS table has no index on db/table name > - > > Key: HIVE-19605 > URL: https://issues.apache.org/jira/browse/HIVE-19605 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Todd Lipcon >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19605.01.patch, HIVE-19605.02.patch > > > The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, > TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. > This makes those queries take a significant amount of time in large > metastores since they do a full table scan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19605) TAB_COL_STATS table has no index on db/table name
[ https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500789#comment-16500789 ] Vihang Karajgaonkar commented on HIVE-19605: Tests work for me locally. Resubmitting the patch. > TAB_COL_STATS table has no index on db/table name > - > > Key: HIVE-19605 > URL: https://issues.apache.org/jira/browse/HIVE-19605 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Todd Lipcon >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19605.01.patch, HIVE-19605.02.patch > > > The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, > TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. > This makes those queries take a significant amount of time in large > metastores since they do a full table scan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500776#comment-16500776 ] Ashutosh Chauhan commented on HIVE-19332: - areColumnStatsUptoDateForPlanning() is misnomer. It should be areColumnStatsUptoDateForQueryAnswering() since for planning they are considered upto date. Its for answering queries they are not. Looks good otherwise. > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.2.patch, > HIVE-19332.3.patch, HIVE-19332.4.patch, HIVE-19332.5.patch, > HIVE-19332.6.patch, HIVE-19332.7.patch, HIVE-19332.8.patch, HIVE-19332.9.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19558) HiveAuthorizationProviderBase gets catalog name from config rather than db object
[ https://issues.apache.org/jira/browse/HIVE-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19558: -- Attachment: HIVE-19558.1take9.patch > HiveAuthorizationProviderBase gets catalog name from config rather than db > object > - > > Key: HIVE-19558 > URL: https://issues.apache.org/jira/browse/HIVE-19558 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-19558.1take2.patch, HIVE-19558.1take3.patch, > HIVE-19558.1take4.patch, HIVE-19558.1take5.patch, HIVE-19558.1take6.patch, > HIVE-19558.1take7.patch, HIVE-19558.1take8.patch, HIVE-19558.1take9.patch, > HIVE-19558.patch > > > HiveAuthorizationProviderBase.getDatabase uses just the database name to > fetch the database, relying on getDefaultCatalog() to fetch the catalog name > from the conf file. This does not work when the client has passed in an > object for a different catalog. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
