[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-21283: -- Status: In Progress (was: Patch Available) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.2.PATCH, HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, > image-2019-03-16-21-33-18-898.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-21283: -- Status: Patch Available (was: In Progress) Resubmitted the patch to avoid the falky test cases > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.2.PATCH, HIVE.21283.PATCH, > image-2019-03-16-21-31-15-541.png, image-2019-03-16-21-33-18-898.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-21283: -- Attachment: HIVE.21283.08.PATCH > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.2.PATCH, HIVE.21283.PATCH, > image-2019-03-16-21-31-15-541.png, image-2019-03-16-21-33-18-898.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21283) Create Synonym mid for substr, position for locate
[
https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794448#comment-16794448
]
Hive QA commented on HIVE-21283:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16548/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16548/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create Synonym mid for substr, position for locate
>
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
> Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
> Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH,
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH,
> HIVE.21283.08.PATCH, HIVE.21283.2.PATCH, HIVE.21283.PATCH,
> image-2019-03-16-21-31-15-541.png, image-2019-03-16-21-33-18-898.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Create new synonym for the existing function
>
> Mid for substr
> postiion for locate
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21283) Create Synonym mid for substr, position for locate
[
https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794456#comment-16794456
]
Hive QA commented on HIVE-21283:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962723/HIVE.21283.08.PATCH
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15835 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16548/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16548/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16548/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962723 - PreCommit-HIVE-Build
> Create Synonym mid for substr, position for locate
>
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
> Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
> Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH,
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH,
> HIVE.21283.08.PATCH, HIVE.21283.2.PATCH, HIVE.21283.PATCH,
> image-2019-03-16-21-31-15-541.png, image-2019-03-16-21-33-18-898.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Create new synonym for the existing function
>
> Mid for substr
> postiion for locate
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[ https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Zhi updated HIVE-21392: Attachment: HIVE-21392.06.patch > Misconfigurations of DataNucleus log in log4j.properties > > > Key: HIVE-21392 > URL: https://issues.apache.org/jira/browse/HIVE-21392 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.0.0 >Reporter: Chen Zhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21392.02.patch, HIVE-21392.03.patch, > HIVE-21392.04.patch, HIVE-21392.05.patch, HIVE-21392.06.patch, > HIVE-21392.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In the patch of > [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the > DataNucleus related logging configuration from nine fine-grained loggers with > three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth > Jayachandran > [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612], > these three loggers are the top-level logger in DataNucleus, so that we > don't need to specify other loggers for DataNucleus. However, according to > the > [documents|http://www.datanucleus.org/products/accessplatform/logging.html] > and [source > codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108] > of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`. > Therefore, we just need to keep the right one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[
https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794464#comment-16794464
]
Hive QA commented on HIVE-21392:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16549/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16549/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Misconfigurations of DataNucleus log in log4j.properties
>
>
> Key: HIVE-21392
> URL: https://issues.apache.org/jira/browse/HIVE-21392
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Chen Zhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21392.02.patch, HIVE-21392.03.patch,
> HIVE-21392.04.patch, HIVE-21392.05.patch, HIVE-21392.06.patch,
> HIVE-21392.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In the patch of
> [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the
> DataNucleus related logging configuration from nine fine-grained loggers with
> three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth
> Jayachandran
> [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612],
> these three loggers are the top-level logger in DataNucleus, so that we
> don't need to specify other loggers for DataNucleus. However, according to
> the
> [documents|http://www.datanucleus.org/products/accessplatform/logging.html]
> and [source
> codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108]
> of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`.
> Therefore, we just need to keep the right one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[
https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794476#comment-16794476
]
Hive QA commented on HIVE-21392:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962727/HIVE-21392.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1
(batchId=319)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testCancelRenewTokenFlow
(batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testConnection (batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValid (batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValidNeg (batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeProxyAuth (batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth (batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testProxyAuth (batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testRenewDelegationToken
(batchId=276)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16549/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16549/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16549/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962727 - PreCommit-HIVE-Build
> Misconfigurations of DataNucleus log in log4j.properties
>
>
> Key: HIVE-21392
> URL: https://issues.apache.org/jira/browse/HIVE-21392
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Chen Zhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21392.02.patch, HIVE-21392.03.patch,
> HIVE-21392.04.patch, HIVE-21392.05.patch, HIVE-21392.06.patch,
> HIVE-21392.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In the patch of
> [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the
> DataNucleus related logging configuration from nine fine-grained loggers with
> three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth
> Jayachandran
> [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612],
> these three loggers are the top-level logger in DataNucleus, so that we
> don't need to specify other loggers for DataNucleus. However, according to
> the
> [documents|http://www.datanucleus.org/products/accessplatform/logging.html]
> and [source
> codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108]
> of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`.
> Therefore, we just need to keep the right one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
[
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794511#comment-16794511
]
Eugene Koifman commented on HIVE-20580:
---
note that Query based compactor doesn't produce hive.acid.index so this jira is
important once that is enabled. cc [~vgumashta]
> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch,
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking
> for presence of {{hive.acid.key.index}} in the footer. This is only created
> when the file is written by {{OrcRecordUpdater}}. It should instead check
> for presence of Acid metadata columns so that a file can be produced by
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1
> type of event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-2816) Extend IMPORT/EXPORT with metadata only option
[ https://issues.apache.org/jira/browse/HIVE-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794555#comment-16794555 ] Guy Melul commented on HIVE-2816: - Is this functionality still relevant? > Extend IMPORT/EXPORT with metadata only option > -- > > Key: HIVE-2816 > URL: https://issues.apache.org/jira/browse/HIVE-2816 > Project: Hive > Issue Type: New Feature > Components: Import/Export >Reporter: Carl Steinbach >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-2816) Extend IMPORT/EXPORT with metadata only option
[ https://issues.apache.org/jira/browse/HIVE-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794557#comment-16794557 ] Shawn Weeks commented on HIVE-2816: --- I think it would be useful. I’ve had to build some tools to do this for testing deployments. It would be great if you could dump the metadata to a common format and import on another system. > Extend IMPORT/EXPORT with metadata only option > -- > > Key: HIVE-2816 > URL: https://issues.apache.org/jira/browse/HIVE-2816 > Project: Hive > Issue Type: New Feature > Components: Import/Export >Reporter: Carl Steinbach >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21437) Vectorization: Decimal64 division with integer columns
[
https://issues.apache.org/jira/browse/HIVE-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-21437:
--
Attachment: HIVE-21437.3.patch
> Vectorization: Decimal64 division with integer columns
> --
>
> Key: HIVE-21437
> URL: https://issues.apache.org/jira/browse/HIVE-21437
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21437.1.patch, HIVE-21437.2.patch,
> HIVE-21437.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorizer fails for
> {code}
> CREATE temporary TABLE `catalog_Sales`(
> `cs_quantity` int,
> `cs_wholesale_cost` decimal(7,2),
> `cs_list_price` decimal(7,2),
> `cs_sales_price` decimal(7,2),
> `cs_ext_discount_amt` decimal(7,2),
> `cs_ext_sales_price` decimal(7,2),
> `cs_ext_wholesale_cost` decimal(7,2),
> `cs_ext_list_price` decimal(7,2),
> `cs_ext_tax` decimal(7,2),
> `cs_coupon_amt` decimal(7,2),
> `cs_ext_ship_cost` decimal(7,2),
> `cs_net_paid` decimal(7,2),
> `cs_net_paid_inc_tax` decimal(7,2),
> `cs_net_paid_inc_ship` decimal(7,2),
> `cs_net_paid_inc_ship_tax` decimal(7,2),
> `cs_net_profit` decimal(7,2))
> ;
> explain vectorization detail select maxcs_ext_list_price -
> cs_ext_wholesale_cost) - cs_ext_discount_amt) + cs_ext_sales_price) / 2) from
> catalog_sales;
> {code}
> {code}
> 'Map Vectorization:'
> 'enabled: true'
> 'enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true'
> 'inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> 'notVectorizedReason: SELECT operator: Could not instantiate
> DecimalColDivideDecimalScalar with arguments arguments: [21, 20, 22],
> argument classes: [Integer, Integer, Integer], exception:
> java.lang.IllegalArgumentException: java.lang.ClassCastException@63b56be0
> stack trace:
> sun.reflect.GeneratedConstructorAccessor.newInstance(Unknown
> Source),
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45),
> java.lang.reflect.Constructor.newInstance(Constructor.java:423),
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.instantiateExpression(VectorizationContext.java:2088),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4662),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4602),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4584),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5171),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:923),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:809),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:776),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:240),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2038),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:1990),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1963),
> ...'
> 'vectorized: false'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21437) Vectorization: Decimal64 division with integer columns
[
https://issues.apache.org/jira/browse/HIVE-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794626#comment-16794626
]
Hive QA commented on HIVE-21437:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 17 new + 470 unchanged - 0
fixed = 487 total (was 470) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16550/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16550/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16550/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16550/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Decimal64 division with integer columns
> --
>
> Key: HIVE-21437
> URL: https://issues.apache.org/jira/browse/HIVE-21437
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21437.1.patch, HIVE-21437.2.patch,
> HIVE-21437.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorizer fails for
> {code}
> CREATE temporary TABLE `catalog_Sales`(
> `cs_quantity` int,
> `cs_wholesale_cost` decimal(7,2),
> `cs_list_price` decimal(7,2),
> `cs_sales_price` decimal(7,2),
> `cs_ext_discount_amt` decimal(7,2),
> `cs_ext_sales_price` decimal(7,2),
> `cs_ext_wholesale_cost` decimal(7,2),
> `cs_ext_list_price` decimal(7,2),
> `cs_ext_tax` decimal(7,2),
> `cs_coupon_amt` decimal(7,2),
> `cs_ext_ship_cost` decimal(7,2),
> `cs_net_paid` decimal(7,2),
> `cs_net_paid_inc_tax` decimal(7,2),
> `cs_net_paid_inc_ship` decimal(7,2),
> `cs_net_paid_inc_ship_tax` decimal(7,2),
> `cs_net_profit` decimal(7,2))
> ;
> explain vectorization detail select maxcs_ext_list_price -
[jira] [Updated] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[ https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Zhi updated HIVE-21392: Attachment: HIVE-21392.07.patch > Misconfigurations of DataNucleus log in log4j.properties > > > Key: HIVE-21392 > URL: https://issues.apache.org/jira/browse/HIVE-21392 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.0.0 >Reporter: Chen Zhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21392.02.patch, HIVE-21392.03.patch, > HIVE-21392.04.patch, HIVE-21392.05.patch, HIVE-21392.06.patch, > HIVE-21392.07.patch, HIVE-21392.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In the patch of > [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the > DataNucleus related logging configuration from nine fine-grained loggers with > three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth > Jayachandran > [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612], > these three loggers are the top-level logger in DataNucleus, so that we > don't need to specify other loggers for DataNucleus. However, according to > the > [documents|http://www.datanucleus.org/products/accessplatform/logging.html] > and [source > codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108] > of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`. > Therefore, we just need to keep the right one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21437) Vectorization: Decimal64 division with integer columns
[
https://issues.apache.org/jira/browse/HIVE-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794632#comment-16794632
]
Hive QA commented on HIVE-21437:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962751/HIVE-21437.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16550/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16550/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16550/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962751 - PreCommit-HIVE-Build
> Vectorization: Decimal64 division with integer columns
> --
>
> Key: HIVE-21437
> URL: https://issues.apache.org/jira/browse/HIVE-21437
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21437.1.patch, HIVE-21437.2.patch,
> HIVE-21437.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorizer fails for
> {code}
> CREATE temporary TABLE `catalog_Sales`(
> `cs_quantity` int,
> `cs_wholesale_cost` decimal(7,2),
> `cs_list_price` decimal(7,2),
> `cs_sales_price` decimal(7,2),
> `cs_ext_discount_amt` decimal(7,2),
> `cs_ext_sales_price` decimal(7,2),
> `cs_ext_wholesale_cost` decimal(7,2),
> `cs_ext_list_price` decimal(7,2),
> `cs_ext_tax` decimal(7,2),
> `cs_coupon_amt` decimal(7,2),
> `cs_ext_ship_cost` decimal(7,2),
> `cs_net_paid` decimal(7,2),
> `cs_net_paid_inc_tax` decimal(7,2),
> `cs_net_paid_inc_ship` decimal(7,2),
> `cs_net_paid_inc_ship_tax` decimal(7,2),
> `cs_net_profit` decimal(7,2))
> ;
> explain vectorization detail select maxcs_ext_list_price -
> cs_ext_wholesale_cost) - cs_ext_discount_amt) + cs_ext_sales_price) / 2) from
> catalog_sales;
> {code}
> {code}
> 'Map Vectorization:'
> 'enabled: true'
> 'enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true'
> 'inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> 'notVectorizedReason: SELECT operator: Could not instantiate
> DecimalColDivideDecimalScalar with arguments arguments: [21, 20, 22],
> argument classes: [Integer, Integer, Integer], exception:
> java.lang.IllegalArgumentException: java.lang.ClassCastException@63b56be0
> stack trace:
> sun.reflect.GeneratedConstructorAccessor.newInstance(Unknown
> Source),
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45),
> java.lang.reflect.Constructor.newInstance(Constructor.java:423),
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.instantiateExpression(VectorizationContext.java:2088),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4662),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4602),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4584),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5171),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:923),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:809),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:776),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:240),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2038),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:1990),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$Ve
[jira] [Commented] (HIVE-21436) "Malformed ORC file" when only one data-file in external table directory
[
https://issues.apache.org/jira/browse/HIVE-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794635#comment-16794635
]
archon gum commented on HIVE-21436:
---
[~damien.carol] presto 305 sql
{code:java}
insert into dim_date2 values (current_date);{code}
java -jar orc-tools-1.5.1-uber.jar meta data1.orc
{code:java}
Processing data file data1.orc [length: 255]
Structure for data1.orc
File Version: 0.12 with PRESTO_ORIGINAL
Rows: 1
Compression: ZLIB
Compression size: 262144
Type: struct
Stripe Statistics:
Stripe 1:
Column 0: count: 1 hasNull: true
Column 1: count: 1 hasNull: true min: 2019-03-13 max: 2019-03-13
File Statistics:
Column 0: count: 1 hasNull: true
Column 1: count: 1 hasNull: true min: 2019-03-13 max: 2019-03-13
Stripes:
Stripe: offset: 3 data: 7 rows: 1 tail: 45 index: 24
Stream: column 1 section ROW_INDEX start: 3 length 24
Stream: column 1 section DATA start: 27 length 7
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
File length: 255 bytes
Padding length: 0 bytes
Padding ratio: 0%
User Metadata:
presto.writer.version=305
presto_query_id=20190313_060328_00174_vcbrs
presto_version=305
{code}
> "Malformed ORC file" when only one data-file in external table directory
>
>
> Key: HIVE-21436
> URL: https://issues.apache.org/jira/browse/HIVE-21436
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: archon gum
>Priority: Blocker
> Attachments: 1.jpg, 2.jpg, data1.orc
>
>
> h1. env
> * Presto 305
> * Hive 3.1.0
>
> h1. step
>
> {code:java}
> -- create external table using hiveserver2
> CREATE EXTERNAL TABLE `dw.dim_date2`(
> `d` date
> )
> STORED AS ORC
> LOCATION
> 'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
> ;
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
>
>
> when using `hiveserver2` to query, it works only at the first query and error
> after then
> !1.jpg!
>
> If I insert another row, it works
> {code:java}
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
> !2.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[
https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794642#comment-16794642
]
Hive QA commented on HIVE-21392:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16551/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16551/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Misconfigurations of DataNucleus log in log4j.properties
>
>
> Key: HIVE-21392
> URL: https://issues.apache.org/jira/browse/HIVE-21392
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Chen Zhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21392.02.patch, HIVE-21392.03.patch,
> HIVE-21392.04.patch, HIVE-21392.05.patch, HIVE-21392.06.patch,
> HIVE-21392.07.patch, HIVE-21392.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In the patch of
> [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the
> DataNucleus related logging configuration from nine fine-grained loggers with
> three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth
> Jayachandran
> [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612],
> these three loggers are the top-level logger in DataNucleus, so that we
> don't need to specify other loggers for DataNucleus. However, according to
> the
> [documents|http://www.datanucleus.org/products/accessplatform/logging.html]
> and [source
> codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108]
> of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`.
> Therefore, we just need to keep the right one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[
https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794655#comment-16794655
]
Hive QA commented on HIVE-21392:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962752/HIVE-21392.07.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16551/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16551/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16551/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962752 - PreCommit-HIVE-Build
> Misconfigurations of DataNucleus log in log4j.properties
>
>
> Key: HIVE-21392
> URL: https://issues.apache.org/jira/browse/HIVE-21392
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Chen Zhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21392.02.patch, HIVE-21392.03.patch,
> HIVE-21392.04.patch, HIVE-21392.05.patch, HIVE-21392.06.patch,
> HIVE-21392.07.patch, HIVE-21392.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In the patch of
> [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the
> DataNucleus related logging configuration from nine fine-grained loggers with
> three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth
> Jayachandran
> [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612],
> these three loggers are the top-level logger in DataNucleus, so that we
> don't need to specify other loggers for DataNucleus. However, according to
> the
> [documents|http://www.datanucleus.org/products/accessplatform/logging.html]
> and [source
> codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108]
> of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`.
> Therefore, we just need to keep the right one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21436) "Malformed ORC file" when only one data-file in external table directory
[
https://issues.apache.org/jira/browse/HIVE-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
archon gum updated HIVE-21436:
--
Attachment: presto-insert-into.orc
org-apache-orc-java-code.orc
hive-insert-into.orc
> "Malformed ORC file" when only one data-file in external table directory
>
>
> Key: HIVE-21436
> URL: https://issues.apache.org/jira/browse/HIVE-21436
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: archon gum
>Priority: Blocker
> Attachments: 1.jpg, 2.jpg, hive-insert-into.orc,
> org-apache-orc-java-code.orc, presto-insert-into.orc
>
>
> h1. env
> * Presto 305
> * Hive 3.1.0
>
> h1. step
>
> {code:java}
> -- create external table using hiveserver2
> CREATE EXTERNAL TABLE `dw.dim_date2`(
> `d` date
> )
> STORED AS ORC
> LOCATION
> 'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
> ;
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
>
>
> when using `hiveserver2` to query, it works only at the first query and error
> after then
> !1.jpg!
>
> If I insert another row, it works
> {code:java}
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
> !2.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21436) "Malformed ORC file" when only one data-file in external table directory
[
https://issues.apache.org/jira/browse/HIVE-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
archon gum updated HIVE-21436:
--
Description:
h1. env
* Presto 305
* Hive 3.1.0
h1. step
{code:java}
-- create external table using hiveserver2
CREATE EXTERNAL TABLE `dw.dim_date2`(
`d` date
)
STORED AS ORC
LOCATION
'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
;
-- upload the 'presto-insert-into.orc' file from attachments
-- OR
-- insert one row using presto
insert into dim_date2 values (current_date);
{code}
when using `hiveserver2` to query, it works only at the first query and error
after then
!1.jpg!
If I insert another row, it works
{code:java}
-- upload the 'presto-insert-into.orc' file from attachments
-- OR
-- insert one row using presto
insert into dim_date2 values (current_date);
{code}
!2.jpg!
was:
h1. env
* Presto 305
* Hive 3.1.0
h1. step
{code:java}
-- create external table using hiveserver2
CREATE EXTERNAL TABLE `dw.dim_date2`(
`d` date
)
STORED AS ORC
LOCATION
'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
;
-- upload the 'data1.orc' file from attachments
-- OR
-- insert one row using presto
insert into dim_date2 values (current_date);
{code}
when using `hiveserver2` to query, it works only at the first query and error
after then
!1.jpg!
If I insert another row, it works
{code:java}
-- upload the 'data1.orc' file from attachments
-- OR
-- insert one row using presto
insert into dim_date2 values (current_date);
{code}
!2.jpg!
> "Malformed ORC file" when only one data-file in external table directory
>
>
> Key: HIVE-21436
> URL: https://issues.apache.org/jira/browse/HIVE-21436
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: archon gum
>Priority: Blocker
> Attachments: 1.jpg, 2.jpg, hive-insert-into.orc,
> org-apache-orc-java-code.orc, presto-insert-into.orc
>
>
> h1. env
> * Presto 305
> * Hive 3.1.0
>
> h1. step
>
> {code:java}
> -- create external table using hiveserver2
> CREATE EXTERNAL TABLE `dw.dim_date2`(
> `d` date
> )
> STORED AS ORC
> LOCATION
> 'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
> ;
> -- upload the 'presto-insert-into.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
>
>
> when using `hiveserver2` to query, it works only at the first query and error
> after then
> !1.jpg!
>
> If I insert another row, it works
> {code:java}
> -- upload the 'presto-insert-into.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
> !2.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21436) "Malformed ORC file" when only one data-file in external table directory
[
https://issues.apache.org/jira/browse/HIVE-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794682#comment-16794682
]
archon gum commented on HIVE-21436:
---
I think it's because of the different `orc` module presto and hive used.
[prestosql/presto
#456|[https://github.com/prestosql/presto/issues/456#issuecomment-473755989]]
> "Malformed ORC file" when only one data-file in external table directory
>
>
> Key: HIVE-21436
> URL: https://issues.apache.org/jira/browse/HIVE-21436
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: archon gum
>Priority: Blocker
> Attachments: 1.jpg, 2.jpg, data1.orc
>
>
> h1. env
> * Presto 305
> * Hive 3.1.0
>
> h1. step
>
> {code:java}
> -- create external table using hiveserver2
> CREATE EXTERNAL TABLE `dw.dim_date2`(
> `d` date
> )
> STORED AS ORC
> LOCATION
> 'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
> ;
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
>
>
> when using `hiveserver2` to query, it works only at the first query and error
> after then
> !1.jpg!
>
> If I insert another row, it works
> {code:java}
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
> !2.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21436) "Malformed ORC file" when only one data-file in external table directory
[
https://issues.apache.org/jira/browse/HIVE-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
archon gum updated HIVE-21436:
--
Attachment: (was: data1.orc)
> "Malformed ORC file" when only one data-file in external table directory
>
>
> Key: HIVE-21436
> URL: https://issues.apache.org/jira/browse/HIVE-21436
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: archon gum
>Priority: Blocker
> Attachments: 1.jpg, 2.jpg, hive-insert-into.orc,
> org-apache-orc-java-code.orc, presto-insert-into.orc
>
>
> h1. env
> * Presto 305
> * Hive 3.1.0
>
> h1. step
>
> {code:java}
> -- create external table using hiveserver2
> CREATE EXTERNAL TABLE `dw.dim_date2`(
> `d` date
> )
> STORED AS ORC
> LOCATION
> 'hdfs://datacenter1:8020/user/hive/warehouse/dw.db/dim_date2'
> ;
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
>
>
> when using `hiveserver2` to query, it works only at the first query and error
> after then
> !1.jpg!
>
> If I insert another row, it works
> {code:java}
> -- upload the 'data1.orc' file from attachments
> -- OR
> -- insert one row using presto
> insert into dim_date2 values (current_date);
> {code}
> !2.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat reassigned HIVE-21462: - > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17404) Orc split generation cache does not handle files without file tail
[
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Shah reassigned HIVE-17404:
--
Assignee: Aditya Shah (was: Prasanth Jayachandran)
> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
>
> Some old files do not have Orc FileTail. If file tail does not exist, split
> generation should fallback to old way of storing footers.
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid
> postscript length 9
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.orc.FileFormatException: Malformed ORC file. Invalid
> postscript length 9
> at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:297)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:470)
> at
> org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:804)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:922)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:891)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1763)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1707)
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17404) Orc split generation cache does not handle files without file tail
[
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Shah updated HIVE-17404:
---
Attachment: HIVE-17404.patch
Target Version/s: (was: 3.0.0, 2.4.0)
Status: Patch Available (was: Open)
Have submitted a patch which Adds a check for ORC bytes in Orctail before
putting it in the local cache. This issue was faced because in HIVE-16133 we
minimize the tail data stored in the cache. This cause a call to extractTails
which rebuilds the OrcTail while using it. This further causes a check for
footer and results in an error being thrown. Because for old orc files when the
tail is not present we check the head for the “ORC” text, but in the case where
we just have a tail as in this call, it causes an exception.
cc [~prasanth_j] [~rajesh.balamohan] [~andrewom]
> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
> Attachments: HIVE-17404.patch
>
>
> Some old files do not have Orc FileTail. If file tail does not exist, split
> generation should fallback to old way of storing footers.
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid
> postscript length 9
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.orc.FileFormatException: Malformed ORC file. Invalid
> postscript length 9
> at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:297)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:470)
> at
> org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:804)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:922)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:891)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1763)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1707)
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214573&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214573
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266296453
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
Review comment:
It can be just info or debug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214573)
Time Spent: 20m (was: 10m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214577
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266297902
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -136,7 +143,11 @@ private void doCopyRetry(FileSystem sourceFs,
List s
}
// looks like some network outrage, reset the file system object and
retry.
- FileSystem.closeAllForUGI(Utils.getUGI());
+ if (proxyUser == null) {
+FileSystem.closeAllForUGI(Utils.getUGI());
+ } else {
+FileSystem.closeAllForUGI(proxyUser);
+ }
sourceFs = pathList.get(0).getFileSystem(hiveConf);
destinationFs = destination.getFileSystem(hiveConf);
Review comment:
We need to propagate the destinationFs to caller as well as caller still
points to old file system object which is invalid. As caller uses this object
in a loop, it is necessary to propagate it back to caller.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214577)
Time Spent: 50m (was: 40m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214576&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214576
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266297006
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -238,23 +249,54 @@ private boolean isSourceFileMismatch(FileSystem
sourceFs, ReplChangeManager.File
return false;
}
+ private UserGroupInformation getProxyUser() throws LoginException,
IOException {
+if (copyAsUser == null) {
+ return null;
+}
+UserGroupInformation proxyUser = null;
+int currentRetry = 0;
+while (currentRetry <= MAX_IO_RETRY) {
+ try {
+UserGroupInformation ugi = Utils.getUGI();
+String currentUser = ugi.getShortUserName();
+if (!currentUser.equals(copyAsUser)) {
+ proxyUser = UserGroupInformation.createProxyUser(
+ copyAsUser, UserGroupInformation.getLoginUser());
+}
+return proxyUser;
+ } catch (IOException e) {
+currentRetry++;
+if (currentRetry <= MAX_IO_RETRY) {
+ LOG.warn("Unable to get UGI info", e);
Review comment:
Can log the retry number in the log msg.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214576)
Time Spent: 40m (was: 0.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214574&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214574
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266296535
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
+if (!checkIfPathExist(sourcePath, proxyUser)) {
+ LOG.warn("Source path is missing. Ignoring exception.");
Review comment:
Can be just info log as it is possible valid scenario in case of external
tables.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214574)
Time Spent: 0.5h (was: 20m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214575
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266298085
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
-for (Map.Entry>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map> destMap = entry.getValue();
- for (Map.Entry> destMapEntry :
destMap.entrySet()) {
-Path destination = destMapEntry.getKey();
-List fileInfoList =
destMapEntry.getValue();
-boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
-if (!destinationFs.exists(destination)
-&& !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
-}
+UserGroupInformation proxyUser = getProxyUser();
+try {
+ for (Map.Entry>>
entry : map.entrySet()) {
+FileSystem sourceFs = entry.getKey();
Review comment:
The sourceFs should be re-get from HDFS if previous copy closed all file
systems object for proxy user.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214575)
Time Spent: 40m (was: 0.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=214588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214588 ] ASF GitHub Bot logged work on HIVE-21283: - Author: ASF GitHub Bot Created on: 18/Mar/19 06:12 Start Date: 18/Mar/19 06:12 Worklog Time Spent: 10m Work Description: rmsmani commented on issue #540: HIVE-21283 Synonyms for the existing functions URL: https://github.com/apache/hive/pull/540#issuecomment-473521823 HI @sankarh Included Unit Test Cases. Check and merge the code This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 214588) Time Spent: 2h (was: 1h 50m) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.2.PATCH, HIVE.21283.PATCH, > image-2019-03-16-21-31-15-541.png, image-2019-03-16-21-33-18-898.png > > Time Spent: 2h > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17404) Orc split generation cache does not handle files without file tail
[
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794756#comment-16794756
]
Hive QA commented on HIVE-17404:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16552/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16552/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
> Attachments: HIVE-17404.patch
>
>
> Some old files do not have Orc FileTail. If file tail does not exist, split
> generation should fallback to old way of storing footers.
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid
> postscript length 9
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.19
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.44.patch > Upgrade to calcite-1.19 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch, > HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch, > HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch, > HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch, > HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch, > HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch, > HIVE-21001.43.patch, HIVE-21001.44.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794758#comment-16794758 ] Zoltan Haindrich commented on HIVE-20546: - [~nishantbangarwa]: this patch seems to have added a "druid-apache-rc-testing" repository to the pom.xml - is that needed? note: based on my recent experiences how the ptest servers seem to cache artifacts...it seems like even if the rc was rolled back; the old code (jar) is sometimes used - not sure if this is relevant to Druid...but it could cause raised eyebrows (at least it caused some for me) :D > Upgrade to Apache Druid 0.13.0-incubating > - > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, > HIVE-20546.3.patch, HIVE-20546.4.patch, HIVE-20546.5.patch, > HIVE-20546.6.patch, HIVE-20546.7.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21454) Tez default configs get overwritten by MR default configs
[ https://issues.apache.org/jira/browse/HIVE-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-21454: - Description: Due to changes done in HIVE-17781 Tez default configs such as tez.counters.max which has a default value of 1200 gets overwritten by mapreduce.job.counters.max which has a default value of 120. So one of the obvious solution is to initialize TezConfig with its default value before the method in HIVE-17781 is called and hence overwritten won't happen. cc [~mithun] was: Due to changes done in HIVE-17781 Tez default configs such as tez.counters.max which has a default value of 1200 gets overwritten by mapreduce.job.counters.max which has a default value of 120 cc [~mithun] > Tez default configs get overwritten by MR default configs > - > > Key: HIVE-21454 > URL: https://issues.apache.org/jira/browse/HIVE-21454 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Priority: Major > > Due to changes done in HIVE-17781 Tez default configs such as > tez.counters.max which has a default value of 1200 gets overwritten by > mapreduce.job.counters.max which has a default value of 120. > So one of the obvious solution is to initialize TezConfig with its default > value before the method in HIVE-17781 is called and hence overwritten won't > happen. > cc [~mithun] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17404) Orc split generation cache does not handle files without file tail
[
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794770#comment-16794770
]
Hive QA commented on HIVE-17404:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962771/HIVE-17404.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16552/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16552/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16552/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962771 - PreCommit-HIVE-Build
> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
> Attachments: HIVE-17404.patch
>
>
> Some old files do not have Orc FileTail. If file tail does not exist, split
> generation should fallback to old way of storing footers.
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid
> postscript length 9
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.orc.FileFormatException: Malformed ORC file. Invalid
> postscript length 9
> at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:297)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:470)
> at
> org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:804)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:922)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:891)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1763)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1707)
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
