[jira] [Commented] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[
https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109948#comment-17109948
]
Hive QA commented on HIVE-23281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003247/HIVE-23281.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22428/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22428/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22428/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-18 06:54:44.443
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22428/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-18 06:54:44.446
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 28bdeeb HIVE-23476 : LLAP: Preallocate arenas for mmap case as
well (Prasanth J via Gopal V)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 28bdeeb HIVE-23476 : LLAP: Preallocate arenas for mmap case as
well (Prasanth J via Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-18 06:54:45.477
+ rm -rf ../yetus_PreCommit-HIVE-Build-22428
+ mkdir ../yetus_PreCommit-HIVE-Build-22428
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22428
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22428/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
does not exist in index
error: a/ql/src/test/results/clientpositive/llap/acid_nullscan.q.out: does not
exist in index
error:
a/ql/src/test/results/clientpositive/llap/acid_table_directories_test.q.out:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
Trying to apply the patch with -p1
error: patch failed:
ql/src/test/results/clientpositive/llap/acid_nullscan.q.out:91
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/acid_nullscan.q.out'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/test/results/clientpositive/llap/acid_nullscan.q.out:91
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/acid_nullscan.q.out'
with conflicts.
U ql/src/test/results/clientpositive/llap/acid_nullscan.q.out
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22428
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003247 - PreCommit-HIVE-Build
> ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to
> DB for ACID tables
> --
>
> Key: HIVE-23281
> URL: https://issues.apache.org/jira/browse/HIVE-23281
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch,
> image-2020-04-23-13-56-17-210
[jira] [Commented] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109945#comment-17109945
]
Hive QA commented on HIVE-23477:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003240/HIVE-23477.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22427/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22427/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22427/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003240 - PreCommit-HIVE-Build
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch,
> HIVE-23477.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRIn
[jira] [Commented] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[ https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109917#comment-17109917 ] Ashutosh Chauhan commented on HIVE-23488: - +1 > Optimise PartitionManagementTask::Msck::repair > -- > > Key: HIVE-23488 > URL: https://issues.apache.org/jira/browse/HIVE-23488 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23488.1.patch, Screenshot 2020-05-18 at 5.06.15 > AM.png > > > Ends up fetching table information twice. > !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754! > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Attachment: HIVE-23468.2.patch > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Status: Open (was: Patch Available) > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Status: Patch Available (was: Open) > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109914#comment-17109914
]
Hive QA commented on HIVE-23477:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22427/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22427/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch,
> HIVE-23477.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(
[jira] [Commented] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109913#comment-17109913 ] Ashutosh Chauhan commented on HIVE-23281: - This patch fixes following from original patch: a) Compiler was always looking for skewed col info, even when its not needed and not returned from metastore which was causing NPE b) Bucket cols are needed for Acid tables, since Acid tables could be bucketed. Was causing failures. c) We need to also implement JDO path necessarily since tests are run to make sure results are identical from two. Not having it implemented caused test fails. > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Attachment: HIVE-23281.2.patch > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Patch Available (was: Open) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Open (was: Patch Available) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[
https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109901#comment-17109901
]
Hive QA commented on HIVE-23484:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003239/HIVE-23484.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dml]
(batchId=66)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22426/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22426/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22426/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003239 - PreCommit-HIVE-Build
> Ensure result order for special_character_in_tabnames_1.q
> -
>
> Key: HIVE-23484
> URL: https://issues.apache.org/jira/browse/HIVE-23484
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23484.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109897#comment-17109897 ] Ashutosh Chauhan commented on HIVE-23468: - +1 > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[
https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109841#comment-17109841
]
Hive QA commented on HIVE-23484:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22426/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22426/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ensure result order for special_character_in_tabnames_1.q
> -
>
> Key: HIVE-23484
> URL: https://issues.apache.org/jira/browse/HIVE-23484
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23484.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[
https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109830#comment-17109830
]
Hive QA commented on HIVE-23488:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003237/HIVE-23488.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testComplexQuery
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testDataTypes
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testEscapedStrings
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testInvalidReferenceCountScenario
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testKillQuery
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testLlapInputFormatEndToEnd
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testLlapInputFormatEndToEndWithMultipleBatches
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testMultipleBatchesOfComplexTypes
(batchId=218)
org.apache.hive.jdbc.TestNewGetSplitsFormatReturnPath.testNonAsciiStrings
(batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22425/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22425/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22425/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003237 - PreCommit-HIVE-Build
> Optimise PartitionManagementTask::Msck::repair
> --
>
> Key: HIVE-23488
> URL: https://issues.apache.org/jira/browse/HIVE-23488
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23488.1.patch, Screenshot 2020-05-18 at 5.06.15
> AM.png
>
>
> Ends up fetching table information twice.
> !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754!
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Attachment: HIVE-23477.3.patch
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch,
> HIVE-23477.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109818#comment-17109818 ] Adesh Kumar Rao commented on HIVE-21637: Thanks [~kishendas]. We can collaborate once the subtasks are ready. > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[
https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109817#comment-17109817
]
Hive QA commented on HIVE-23488:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 50 unchanged - 1 fixed = 51 total (was 51) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22425/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22425/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22425/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22425/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22425/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimise PartitionManagementTask::Msck::repair
> --
>
> Key: HIVE-23488
> URL: https://issues.apache.org/jira/browse/HIVE-23488
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23488.1.patch, Screenshot 2020-05-18 at 5.06.15
> AM.png
>
>
> Ends up fetching table information twice.
> !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754!
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113]
[jira] [Commented] (HIVE-23487) Optimise PartitionManagementTask
[
https://issues.apache.org/jira/browse/HIVE-23487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109798#comment-17109798
]
Hive QA commented on HIVE-23487:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003235/HIVE-23487.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=128)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22424/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22424/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22424/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003235 - PreCommit-HIVE-Build
> Optimise PartitionManagementTask
>
>
> Key: HIVE-23487
> URL: https://issues.apache.org/jira/browse/HIVE-23487
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23487.1.patch, Screenshot 2020-05-18 at 4.19.48
> AM.png
>
>
> Msck.init for every table takes more time than the actual table repair. This
> was observed on a system which had lots of DB and tables.
>
> !Screenshot 2020-05-18 at 4.19.48 AM.png|width=1014,height=732!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23487) Optimise PartitionManagementTask
[
https://issues.apache.org/jira/browse/HIVE-23487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109787#comment-17109787
]
Hive QA commented on HIVE-23487:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 54 unchanged - 1 fixed = 54 total (was 55) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22424/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22424/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22424/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22424/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimise PartitionManagementTask
>
>
> Key: HIVE-23487
> URL: https://issues.apache.org/jira/browse/HIVE-23487
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23487.1.patch, Screenshot 2020-05-18 at 4.19.48
> AM.png
>
>
> Msck.init for every table takes more time than the actual table repair. This
> was observed on a system which had lots of DB and tables.
>
> !Screenshot 2020-05-18 at 4.19.48 AM.png|width=1014,height=732!
--
This message was se
[jira] [Commented] (HIVE-22368) Hive JDBC Storage Handler: some mysql data type can not be cast to hive data type
[
https://issues.apache.org/jira/browse/HIVE-22368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109756#comment-17109756
]
Hive QA commented on HIVE-22368:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12983353/HIVE-22368.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=128)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22423/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22423/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22423/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12983353 - PreCommit-HIVE-Build
> Hive JDBC Storage Handler: some mysql data type can not be cast to hive data
> type
> -
>
> Key: HIVE-22368
> URL: https://issues.apache.org/jira/browse/HIVE-22368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0, 3.1.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22368.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Mysql data type（date、timestamp、decimal）can not be cast to hive data
> type（date、timestamp、decimal）。
> step to repo（take date type for example）:
> {code:java}
> //MySQL table:
> create table testdate(id date);
> //Hive table:
> CREATE EXTERNAL TABLE `hive_date`(
> id date )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testdate');
> //Hive query:
> select * from hive_date;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date (state=,code=0)
> //Error stack trace
> Caused by: java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:41)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:27)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:422)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.toThriftPayload(SerDeUtils.java:173)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.thrift.ThriftFormatter.convert(ThriftFormatter.java:49)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Oper
[jira] [Updated] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[ https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23484: -- Attachment: HIVE-23484.01.patch > Ensure result order for special_character_in_tabnames_1.q > - > > Key: HIVE-23484 > URL: https://issues.apache.org/jira/browse/HIVE-23484 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23484.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[ https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23484: -- Attachment: (was: HIVE-23484.01.patch) > Ensure result order for special_character_in_tabnames_1.q > - > > Key: HIVE-23484 > URL: https://issues.apache.org/jira/browse/HIVE-23484 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23484.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22368) Hive JDBC Storage Handler: some mysql data type can not be cast to hive data type
[
https://issues.apache.org/jira/browse/HIVE-22368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109725#comment-17109725
]
Hive QA commented on HIVE-22368:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22423/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22423/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive JDBC Storage Handler: some mysql data type can not be cast to hive data
> type
> -
>
> Key: HIVE-22368
> URL: https://issues.apache.org/jira/browse/HIVE-22368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0, 3.1.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22368.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Mysql data type（date、timestamp、decimal）can not be cast to hive data
> type（date、timestamp、decimal）。
> step to repo（take date type for example）:
> {code:java}
> //MySQL table:
> create table testdate(id date);
> //Hive table:
> CREATE EXTERNAL TABLE `hive_date`(
> id date )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testdate');
> //Hive query:
> select * from hive_date;
> Error: java.io.IOExcepti
[jira] [Updated] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[ https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23488: Attachment: HIVE-23488.1.patch > Optimise PartitionManagementTask::Msck::repair > -- > > Key: HIVE-23488 > URL: https://issues.apache.org/jira/browse/HIVE-23488 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23488.1.patch, Screenshot 2020-05-18 at 5.06.15 > AM.png > > > Ends up fetching table information twice. > !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754! > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[ https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23488: Assignee: Rajesh Balamohan Status: Patch Available (was: Open) > Optimise PartitionManagementTask::Msck::repair > -- > > Key: HIVE-23488 > URL: https://issues.apache.org/jira/browse/HIVE-23488 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23488.1.patch, Screenshot 2020-05-18 at 5.06.15 > AM.png > > > Ends up fetching table information twice. > !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754! > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[
https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109716#comment-17109716
]
Hive QA commented on HIVE-23433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003222/HIVE-23433.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17266 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
org.apache.hadoop.hive.ql.exec.repl.TestRangerLoadTask.testSuccessAddDenyRangerPolicies
(batchId=292)
org.apache.hadoop.hive.ql.exec.repl.TestRangerLoadTask.testSuccessRangerDumpMetrics
(batchId=292)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22422/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22422/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22422/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003222 - PreCommit-HIVE-Build
> Add Deny Policy on Target Database After Ranger Replication to avoid writes
> ---
>
> Key: HIVE-23433
> URL: https://issues.apache.org/jira/browse/HIVE-23433
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch,
> HIVE-23433.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23487) Optimise PartitionManagementTask
[ https://issues.apache.org/jira/browse/HIVE-23487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23487: Assignee: Rajesh Balamohan Status: Patch Available (was: Open) > Optimise PartitionManagementTask > > > Key: HIVE-23487 > URL: https://issues.apache.org/jira/browse/HIVE-23487 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23487.1.patch, Screenshot 2020-05-18 at 4.19.48 > AM.png > > > Msck.init for every table takes more time than the actual table repair. This > was observed on a system which had lots of DB and tables. > > !Screenshot 2020-05-18 at 4.19.48 AM.png|width=1014,height=732! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23487) Optimise PartitionManagementTask
[ https://issues.apache.org/jira/browse/HIVE-23487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23487: Attachment: HIVE-23487.1.patch > Optimise PartitionManagementTask > > > Key: HIVE-23487 > URL: https://issues.apache.org/jira/browse/HIVE-23487 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23487.1.patch, Screenshot 2020-05-18 at 4.19.48 > AM.png > > > Msck.init for every table takes more time than the actual table repair. This > was observed on a system which had lots of DB and tables. > > !Screenshot 2020-05-18 at 4.19.48 AM.png|width=1014,height=732! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[
https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109712#comment-17109712
]
Hive QA commented on HIVE-23433:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
58s{color} | {color:red} ql: The patch generated 3 new + 0 unchanged - 0 fixed
= 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22422/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22422/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22422/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Deny Policy on Target Database After Ranger Replication to avoid writes
> ---
>
> Key: HIVE-23433
> URL: https://issues.apache.org/jira/browse/HIVE-23433
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch,
> HIVE-23433.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[
https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109706#comment-17109706
]
Hive QA commented on HIVE-23484:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003221/HIVE-23484.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17265 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService.cleansAllCommittedTxns
(batchId=152)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22421/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22421/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22421/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003221 - PreCommit-HIVE-Build
> Ensure result order for special_character_in_tabnames_1.q
> -
>
> Key: HIVE-23484
> URL: https://issues.apache.org/jira/browse/HIVE-23484
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23484.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[
https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109696#comment-17109696
]
Hive QA commented on HIVE-23484:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22421/dev-support/hive-personality.sh
|
| git revision | master / 28bdeeb |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22421/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ensure result order for special_character_in_tabnames_1.q
> -
>
> Key: HIVE-23484
> URL: https://issues.apache.org/jira/browse/HIVE-23484
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23484.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23476: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Prasanth! > LLAP: Preallocate arenas for mmap case as well > -- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23476.1.patch, HIVE-23476.2.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[
https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109693#comment-17109693
]
Hive QA commented on HIVE-23476:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003216/HIVE-23476.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22420/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22420/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22420/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003216 - PreCommit-HIVE-Build
> LLAP: Preallocate arenas for mmap case as well
> --
>
> Key: HIVE-23476
> URL: https://issues.apache.org/jira/browse/HIVE-23476
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23476.1.patch, HIVE-23476.2.patch
>
>
> BuddyAllocator pre-allocation of arenas does not happen for mmap cache case.
> Since we are not filling up the mmap'ed buffers the upfront allocations in
> constructor is cheap. This can avoid lock free allocation of arenas later in
> the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[
https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109679#comment-17109679
]
Hive QA commented on HIVE-23476:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22420/dev-support/hive-personality.sh
|
| git revision | master / 7fe1076 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22420/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Preallocate arenas for mmap case as well
> --
>
> Key: HIVE-23476
> URL: https://issues.apache.org/jira/browse/HIVE-23476
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23476.1.patch, HIVE-23476.2.patch
>
>
> BuddyAllocator pre-allocation of arenas does not happen for mmap cache case.
> Since we are not filling up the mmap'ed buffers the upfront allocations in
> constructor is cheap. This can avoid lock free allocation of arenas later in
> the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23486) Compiler support for row-level filtering on transactional deletes
[ https://issues.apache.org/jira/browse/HIVE-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23486: - > Compiler support for row-level filtering on transactional deletes > - > > Key: HIVE-23486 > URL: https://issues.apache.org/jira/browse/HIVE-23486 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23167) Compiler support for row-level filtering on static filters
[ https://issues.apache.org/jira/browse/HIVE-23167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23167: -- Summary: Compiler support for row-level filtering on static filters (was: Extend compiler support for Probe static filters) > Compiler support for row-level filtering on static filters > --- > > Key: HIVE-23167 > URL: https://issues.apache.org/jira/browse/HIVE-23167 > Project: Hive > Issue Type: New Feature >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > Extend HIVE-23006 rules to support static filter expressions like: > select * from sales where sold_state = 'PR'; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23167) Extend compiler support for Probe static filters
[ https://issues.apache.org/jira/browse/HIVE-23167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23167: -- Parent: (was: HIVE-22731) Issue Type: New Feature (was: Sub-task) > Extend compiler support for Probe static filters > > > Key: HIVE-23167 > URL: https://issues.apache.org/jira/browse/HIVE-23167 > Project: Hive > Issue Type: New Feature >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > > Extend HIVE-23006 rules to support static filter expressions like: > select * from sales where sold_state = 'PR'; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109653#comment-17109653
]
Hive QA commented on HIVE-23477:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003215/HIVE-23477.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_part]
(batchId=80)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22419/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22419/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22419/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003215 - PreCommit-HIVE-Build
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderM
[jira] [Commented] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109639#comment-17109639
]
Hive QA commented on HIVE-23477:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22419/dev-support/hive-personality.sh
|
| git revision | master / c5c0767 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22419/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:145
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23478: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, John! > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[ https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23292: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Ramesh & Rajesh! > Reduce PartitionDesc payload in MapWork > --- > > Key: HIVE-23292 > URL: https://issues.apache.org/jira/browse/HIVE-23292 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23292.1.patch, HIVE-23292.1.patch, > HIVE-23292.2.patch, HIVE-23292.3.patch, HIVE-23292.4.patch > > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java#L105 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[
https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109629#comment-17109629
]
Hive QA commented on HIVE-23292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003213/HIVE-23292.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.testAuthorization1
(batchId=235)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22418/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22418/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22418/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003213 - PreCommit-HIVE-Build
> Reduce PartitionDesc payload in MapWork
> ---
>
> Key: HIVE-23292
> URL: https://issues.apache.org/jira/browse/HIVE-23292
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23292.1.patch, HIVE-23292.1.patch,
> HIVE-23292.2.patch, HIVE-23292.3.patch, HIVE-23292.4.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java#L105
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23314) Upgrade to Kudu 1.12
[ https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23314: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you Miklos for reviewing the changes! > Upgrade to Kudu 1.12 > > > Key: HIVE-23314 > URL: https://issues.apache.org/jira/browse/HIVE-23314 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch, > HIVE-23314.01.patch, HIVE-23314.01.patch, HIVE-23314.01.patch > > > we need KUDU-3044 because it could cause random failures... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[
https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109621#comment-17109621
]
Hive QA commented on HIVE-23292:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} metastore-common in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 19 unchanged - 1 fixed = 20 total (was 20) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} serde: The patch generated 3 new + 7 unchanged - 0
fixed = 10 total (was 7) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
2s{color} | {color:red} ql: The patch generated 2 new + 8 unchanged - 1 fixed =
10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22418/dev-support/hive-personality.sh
|
| git revision | master / e85731c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22418/yetus/branch-findbugs-standalone-metastore_metastore-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22418/yetus/diff-checkstyle-standalone-metastore_metastore-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22418/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22418/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22418/yetus/patch-findbugs-standalone-metastore_metastore-common.txt
|
| modules | C: standalone-metastore/metastore-common serde ql
itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22418/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
[jira] [Work logged] (HIVE-22368) Hive JDBC Storage Handler: some mysql data type can not be cast to hive data type
[
https://issues.apache.org/jira/browse/HIVE-22368?focusedWorklogId=434235&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434235
]
ASF GitHub Bot logged work on HIVE-22368:
-
Author: ASF GitHub Bot
Created on: 17/May/20 19:18
Start Date: 17/May/20 19:18
Worklog Time Spent: 10m
Work Description: ches commented on a change in pull request #822:
URL: https://github.com/apache/hive/pull/822#discussion_r426294836
##
File path:
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaDateObjectInspector.java
##
@@ -38,7 +38,7 @@ public DateWritableV2 getPrimitiveWritableObject(Object o) {
@Override
public Date getPrimitiveJavaObject(Object o) {
-return o == null ? null : (Date) o;
+return o == null ? null : Date.valueOf(o.toString());
Review comment:
If this is the correct fix, I believe it's also needed on the `new
DateWritableV2((Date) o)` in the method above. I'm having code break there on a
write path.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434235)
Time Spent: 20m (was: 10m)
> Hive JDBC Storage Handler: some mysql data type can not be cast to hive data
> type
> -
>
> Key: HIVE-22368
> URL: https://issues.apache.org/jira/browse/HIVE-22368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0, 3.1.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22368.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Mysql data type（date、timestamp、decimal）can not be cast to hive data
> type（date、timestamp、decimal）。
> step to repo（take date type for example）:
> {code:java}
> //MySQL table:
> create table testdate(id date);
> //Hive table:
> CREATE EXTERNAL TABLE `hive_date`(
> id date )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testdate');
> //Hive query:
> select * from hive_date;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date (state=,code=0)
> //Error stack trace
> Caused by: java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:41)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:27)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:422)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.toThriftPayload(SerDeUtils.java:173)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.thrift.ThriftFormatter.convert(ThriftFormatter.java:49)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[
https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109608#comment-17109608
]
Hive QA commented on HIVE-23478:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003223/HIVE-23478.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22417/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22417/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22417/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003223 - PreCommit-HIVE-Build
> Fix flaky special_character_in_tabnames_quotes_1 test
> -
>
> Key: HIVE-23478
> URL: https://issues.apache.org/jira/browse/HIVE-23478
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch,
> HIVE-23478.3.patch
>
>
> While testing https://issues.apache.org/jira/browse/HIVE-23354
> special_character_in_tabnames_quotes_1 failed. Searching for the test, it
> seems other patches have also had failures. I noticed that
> special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1
> use the same database/table names. I suspect this is responsible for some of
> the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?focusedWorklogId=434227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434227
]
ASF GitHub Bot logged work on HIVE-23443:
-
Author: ASF GitHub Bot
Created on: 17/May/20 18:09
Start Date: 17/May/20 18:09
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #1012:
URL: https://github.com/apache/hive/pull/1012#discussion_r426288840
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -884,10 +885,20 @@ private void finishableStateUpdated(TaskWrapper
taskWrapper, boolean newFinishab
taskWrapper.updateCanFinishForPriority(newFinishableState);
forceReinsertIntoQueue(taskWrapper, isRemoved);
} else {
-taskWrapper.updateCanFinishForPriority(newFinishableState);
-if (!newFinishableState && !taskWrapper.isInPreemptionQueue()) {
- // No need to check guaranteed here; if it was false we would
already be in the queue.
+// if speculative task, any finishable state change should re-order
the queue as speculative tasks are always
+// not-guaranteed (re-order helps put non-finishable's ahead of
finishable)
+if (!taskWrapper.isGuaranteed()) {
+ removeFromPreemptionQueue(taskWrapper);
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
addToPreemptionQueue(taskWrapper);
+} else {
+ // if guaranteed task, if the finishable state changed to
non-finishable and if the task doesn't exist
+ // pre-emption queue, then add it so that it becomes candidate to
kill
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
Review comment:
@prasanthj thanks for fixing this! Patch looks good and is now committed!
At some point I would also change the first comment of the method to clarify
that a task that is both Guaranteed and Finishable should never be in the
preemption queue.
https://github.com/apache/hive/pull/1012/files#diff-16658bf15468ecd089c4fd32e75fa8b2R876
I am believe there is value to put some effort documenting and describing
how the scheduler works (started keeping some noted but happy to add more
work). I personally find the information in this area of the project very
limited.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434227)
Time Spent: 50m (was: 40m)
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch,
> HIVE-23443.3.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109597#comment-17109597 ] Ashutosh Chauhan commented on HIVE-23478: - +1 > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[
https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109595#comment-17109595
]
Hive QA commented on HIVE-23478:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22417/dev-support/hive-personality.sh
|
| git revision | master / e85731c |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22417/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flaky special_character_in_tabnames_quotes_1 test
> -
>
> Key: HIVE-23478
> URL: https://issues.apache.org/jira/browse/HIVE-23478
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch,
> HIVE-23478.3.patch
>
>
> While testing https://issues.apache.org/jira/browse/HIVE-23354
> special_character_in_tabnames_quotes_1 failed. Searching for the test, it
> seems other patches have also had failures. I noticed that
> special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1
> use the same database/table names. I suspect this is responsible for some of
> the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23354: -- Labels: pull-request-available (was: ) > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, > HIVE-23354.6.patch, HIVE-23354.7.patch > > Time Spent: 10m > Remaining Estimate: 0h > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?focusedWorklogId=434225&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434225 ] ASF GitHub Bot logged work on HIVE-23354: - Author: ASF GitHub Bot Created on: 17/May/20 18:00 Start Date: 17/May/20 18:00 Worklog Time Spent: 10m Work Description: jfsii opened a new pull request #1022: URL: https://github.com/apache/hive/pull/1022 …ateFiles Change-Id: I12a76ff1c98e10941e7ef76932a5249ed4a5472a This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434225) Remaining Estimate: 0h Time Spent: 10m > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, > HIVE-23354.6.patch, HIVE-23354.7.patch > > Time Spent: 10m > Remaining Estimate: 0h > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[
https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109594#comment-17109594
]
Hive QA commented on HIVE-23354:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003211/HIVE-23354.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22416/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22416/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22416/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003211 - PreCommit-HIVE-Build
> Remove file size sanity checking from compareTempOrDuplicateFiles
> -
>
> Key: HIVE-23354
> URL: https://issues.apache.org/jira/browse/HIVE-23354
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch,
> HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch,
> HIVE-23354.6.patch, HIVE-23354.7.patch
>
>
> [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010]
> compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to
> determine which file(s) to keep.
> I've seen instances where this function throws an exception due to the fact
> that the newer attemptId file size is less than the older attemptId (thus
> failing the query).
> I think this assumption is faulty, due to various factors such as file
> compression and the order in which values are written. It may be prudent to
> trust that the newest attemptId is in fact the best choice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Status: Patch Available (was: Reopened) > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[
https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109585#comment-17109585
]
Hive QA commented on HIVE-23354:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
0s{color} | {color:red} ql: The patch generated 2 new + 228 unchanged - 2 fixed
= 230 total (was 230) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22416/dev-support/hive-personality.sh
|
| git revision | master / e85731c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22416/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22416/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove file size sanity checking from compareTempOrDuplicateFiles
> -
>
> Key: HIVE-23354
> URL: https://issues.apache.org/jira/browse/HIVE-23354
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch,
> HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch,
> HIVE-23354.6.patch, HIVE-23354.7.patch
>
>
> [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010]
> compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to
> determine which file(s) to keep.
> I've seen instances where this function throws an exception due to the fact
> that the newer attemptId file size is less than the older attemptId (thus
> failing the query).
> I think this assumption is faulty, due
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Attachment: HIVE-23478.3.patch > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109584#comment-17109584 ] John Sherman commented on HIVE-23478: - Incorporated changes from HIVE-23484 > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Attachment: HIVE-23433.03.patch Status: Patch Available (was: In Progress) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > --- > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch, > HIVE-23433.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Status: In Progress (was: Patch Available) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > --- > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23470) Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver
[ https://issues.apache.org/jira/browse/HIVE-23470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109578#comment-17109578 ] Jesus Camacho Rodriguez commented on HIVE-23470: +1 > Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with > TestMiniLlapLocalCliDriver > - > > Key: HIVE-23470 > URL: https://issues.apache.org/jira/browse/HIVE-23470 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23470.01.patch > > > Some tests are failing with TestMiniLlapLocalCliDriver, but running fine with > TestMiniLlapCliDriver, let's move them there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109577#comment-17109577 ] Ashutosh Chauhan commented on HIVE-23478: - Doing both won't hurt and makes test more resilient. So, can add unique name as well as order by. > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[ https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23484: -- Attachment: (was: HIVE-23484.01.patch) > Ensure result order for special_character_in_tabnames_1.q > - > > Key: HIVE-23484 > URL: https://issues.apache.org/jira/browse/HIVE-23484 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23484.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[ https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23484: -- Attachment: HIVE-23484.01.patch > Ensure result order for special_character_in_tabnames_1.q > - > > Key: HIVE-23484 > URL: https://issues.apache.org/jira/browse/HIVE-23484 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23484.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[
https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109566#comment-17109566
]
Hive QA commented on HIVE-23314:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003194/HIVE-23314.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17268 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22415/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22415/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22415/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003194 - PreCommit-HIVE-Build
> Upgrade to Kudu 1.12
>
>
> Key: HIVE-23314
> URL: https://issues.apache.org/jira/browse/HIVE-23314
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch,
> HIVE-23314.01.patch, HIVE-23314.01.patch, HIVE-23314.01.patch
>
>
> we need KUDU-3044 because it could cause random failures...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[
https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109559#comment-17109559
]
Hive QA commented on HIVE-23314:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22415/dev-support/hive-personality.sh
|
| git revision | master / 5c9fa2a |
| Default Java | 1.8.0_111 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22415/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Kudu 1.12
>
>
> Key: HIVE-23314
> URL: https://issues.apache.org/jira/browse/HIVE-23314
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch,
> HIVE-23314.01.patch, HIVE-23314.01.patch, HIVE-23314.01.patch
>
>
> we need KUDU-3044 because it could cause random failures...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109555#comment-17109555 ] John Sherman commented on HIVE-23478: - Actually, I think I confused myself with similar named tests. I'll update this patch with the ordering change. > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reopened HIVE-23478: - > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman resolved HIVE-23478. - Resolution: Duplicate Closing as duplicate of HIVE-23484 > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Status: Open (was: Patch Available) > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109550#comment-17109550 ] John Sherman commented on HIVE-23478: - [~ashutoshc] I was just looking at that one, I do believe that is the correct approach. I'm going to suggest making the DB name on that patch unique also. I had seen problems running both test together locally which I surmised to be the DB name, but I didn't repro the ordering formatting change due to order. I'll abandon this one. > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[
https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109544#comment-17109544
]
Ashutosh Chauhan commented on HIVE-23478:
-
I also hit this flaky test in one of my patches. Failure I saw was following :
{code}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing special_character_in_tabnames_quotes_1.q
15534c15534
< 14 2
---
> 14 2
{code}
I dont think using unique name will suffice. Issue seems to be order of rows in
test output. It probably needs fix similar to HIVE-23484
> Fix flaky special_character_in_tabnames_quotes_1 test
> -
>
> Key: HIVE-23478
> URL: https://issues.apache.org/jira/browse/HIVE-23478
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch
>
>
> While testing https://issues.apache.org/jira/browse/HIVE-23354
> special_character_in_tabnames_quotes_1 failed. Searching for the test, it
> seems other patches have also had failures. I noticed that
> special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1
> use the same database/table names. I suspect this is responsible for some of
> the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23443:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Prasanth!
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch,
> HIVE-23443.3.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23476: Status: Patch Available (was: Open) > LLAP: Preallocate arenas for mmap case as well > -- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch, HIVE-23476.2.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23476: Attachment: HIVE-23476.2.patch > LLAP: Preallocate arenas for mmap case as well > -- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch, HIVE-23476.2.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23476: Status: Open (was: Patch Available) > LLAP: Preallocate arenas for mmap case as well > -- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch, HIVE-23476.2.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109537#comment-17109537
]
Hive QA commented on HIVE-23353:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
0s{color} | {color:red} ql: The patch generated 38 new + 18 unchanged - 0 fixed
= 56 total (was 18) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m
50s{color} | {color:red} root: The patch generated 39 new + 509 unchanged - 0
fixed = 548 total (was 509) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 114
unchanged - 0 fixed = 115 total (was 114) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m
52s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
11s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22414/dev-support/hive-personality.sh
|
| git revision | master / 5c9fa2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22414/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22414/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22414/yetus/diff-checkstyle-itests_hive-unit.txt
|
| f
[jira] [Updated] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23477:
Status: Patch Available (was: Open)
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(T
[jira] [Updated] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23477:
Status: Open (was: Patch Available)
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(T
[jira] [Updated] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23477:
Attachment: HIVE-23477.2.patch
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch, HIVE-23477.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRun
[jira] [Updated] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[ https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23292: Status: Patch Available (was: Open) > Reduce PartitionDesc payload in MapWork > --- > > Key: HIVE-23292 > URL: https://issues.apache.org/jira/browse/HIVE-23292 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23292.1.patch, HIVE-23292.1.patch, > HIVE-23292.2.patch, HIVE-23292.3.patch, HIVE-23292.4.patch > > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java#L105 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[ https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23292: Status: Open (was: Patch Available) > Reduce PartitionDesc payload in MapWork > --- > > Key: HIVE-23292 > URL: https://issues.apache.org/jira/browse/HIVE-23292 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23292.1.patch, HIVE-23292.1.patch, > HIVE-23292.2.patch, HIVE-23292.3.patch, HIVE-23292.4.patch > > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java#L105 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[ https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23292: Attachment: HIVE-23292.4.patch > Reduce PartitionDesc payload in MapWork > --- > > Key: HIVE-23292 > URL: https://issues.apache.org/jira/browse/HIVE-23292 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23292.1.patch, HIVE-23292.1.patch, > HIVE-23292.2.patch, HIVE-23292.3.patch, HIVE-23292.4.patch > > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java#L105 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Attachment: HIVE-23478.2.patch > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.7.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, > HIVE-23354.6.patch, HIVE-23354.7.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109528#comment-17109528 ] John Sherman commented on HIVE-23354: - Re-uploaded patch in hopes test infra runs this time. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, > HIVE-23354.6.patch, HIVE-23354.7.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109527#comment-17109527
]
Hive QA commented on HIVE-23353:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003193/HIVE-23353.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17272 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=78)
org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService.cleansAllCommittedTxns
(batchId=152)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22414/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22414/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22414/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003193 - PreCommit-HIVE-Build
> Atlas metadata replication scheduling
> -
>
> Key: HIVE-23353
> URL: https://issues.apache.org/jira/browse/HIVE-23353
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23353.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23361) Optimising privilege synchroniser
[
https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109493#comment-17109493
]
Hive QA commented on HIVE-23361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003185/hive-23361.7.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17270 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003185 - PreCommit-HIVE-Build
> Optimising privilege synchroniser
> -
>
> Key: HIVE-23361
> URL: https://issues.apache.org/jira/browse/HIVE-23361
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Simhadri G
>Assignee: Simhadri G
>Priority: Minor
> Attachments: hive-23361.1.patch, hive-23361.2.patch,
> hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch,
> hive-23361.6.patch, hive-23361.7.patch, hive-23361.patch, hive-23361.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> This patch store the privilege information as bit string. This is done to
> reduce the size of the tbl_col_privs tables in metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23361) Optimising privilege synchroniser
[
https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109486#comment-17109486
]
Hive QA commented on HIVE-23361:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 6m
51s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22413/dev-support/hive-personality.sh
|
| git revision | master / 5c9fa2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: metastore ql itests itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22413/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimising privilege synchroniser
> -
>
> Key: HIVE-23361
> URL: https://issues.apache.org/jira/browse/HIVE-23361
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Simhadri G
>Assignee: Simhadri G
>Priority: Minor
> Attachments: hive-23361.1.patch, hive-23361.2.patch,
> hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch,
> hive-23361.6.patch, hive-23361.7.patch, hive-23361.patch, hive-23361.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> This patch store the privilege information as bit string. This is done to
> reduce the size of the tbl_col_privs tables in metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23482) Use junit5 to execute tests
[
https://issues.apache.org/jira/browse/HIVE-23482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109484#comment-17109484
]
Hive QA commented on HIVE-23482:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 15m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} metastore in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} llap-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} llap-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} jdbc in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} contrib in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hbase-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} svr in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} hplsql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} kudu-handler in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
55s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 22m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22411/dev-support/hive-personality.sh
|
| git revision | master / 5c9fa2a |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22411/yetus/patch-mvninstall-metastore.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22411/yetus/patch-mvninstall-llap-common.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22411/yetus/patch-mvninstall-llap-client.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommi
[jira] [Updated] (HIVE-23314) Upgrade to Kudu 1.12
[ https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23314: Attachment: HIVE-23314.01.patch > Upgrade to Kudu 1.12 > > > Key: HIVE-23314 > URL: https://issues.apache.org/jira/browse/HIVE-23314 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch, > HIVE-23314.01.patch, HIVE-23314.01.patch, HIVE-23314.01.patch > > > we need KUDU-3044 because it could cause random failures... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[
https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109466#comment-17109466
]
Hive QA commented on HIVE-23484:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003178/HIVE-23484.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.security.TestZookeeperTokenStoreSSLEnabled.org.apache.hadoop.hive.metastore.security.TestZookeeperTokenStoreSSLEnabled
(batchId=175)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=205)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22412/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22412/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22412/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003178 - PreCommit-HIVE-Build
> Ensure result order for special_character_in_tabnames_1.q
> -
>
> Key: HIVE-23484
> URL: https://issues.apache.org/jira/browse/HIVE-23484
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23484.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Attachment: HIVE-23353.01.patch > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23353: -- Labels: pull-request-available (was: ) > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434184&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434184 ] ASF GitHub Bot logged work on HIVE-23353: - Author: ASF GitHub Bot Created on: 17/May/20 12:51 Start Date: 17/May/20 12:51 Worklog Time Spent: 10m Work Description: pkumarsinha opened a new pull request #1021: URL: https://github.com/apache/hive/pull/1021 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 434184) Remaining Estimate: 0h Time Spent: 10m > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Status: Patch Available (was: In Progress) > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23484) Ensure result order for special_character_in_tabnames_1.q
[
https://issues.apache.org/jira/browse/HIVE-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109445#comment-17109445
]
Hive QA commented on HIVE-23484:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22412/dev-support/hive-personality.sh
|
| git revision | master / 5c9fa2a |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22412/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ensure result order for special_character_in_tabnames_1.q
> -
>
> Key: HIVE-23484
> URL: https://issues.apache.org/jira/browse/HIVE-23484
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23484.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[ https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109440#comment-17109440 ] Syed Shameerur Rahman commented on HIVE-23452: -- [~lide] Looks like this issue is solved in https://issues.apache.org/jira/browse/HIVE-20652 > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez > --- > > Key: HIVE-23452 > URL: https://issues.apache.org/jira/browse/HIVE-23452 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: De Li >Priority: Major > > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by > Tez and it works when query with MR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[
https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109435#comment-17109435
]
Hive QA commented on HIVE-23314:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003173/HIVE-23314.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=207)
org.apache.hive.service.cli.thrift.TestThriftCliServiceMessageSize.testMessageSize
(batchId=209)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22410/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22410/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22410/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003173 - PreCommit-HIVE-Build
> Upgrade to Kudu 1.12
>
>
> Key: HIVE-23314
> URL: https://issues.apache.org/jira/browse/HIVE-23314
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch,
> HIVE-23314.01.patch, HIVE-23314.01.patch
>
>
> we need KUDU-3044 because it could cause random failures...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[
https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109430#comment-17109430
]
Hive QA commented on HIVE-23314:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22410/dev-support/hive-personality.sh
|
| git revision | master / 5c9fa2a |
| Default Java | 1.8.0_111 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22410/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Kudu 1.12
>
>
> Key: HIVE-23314
> URL: https://issues.apache.org/jira/browse/HIVE-23314
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch,
> HIVE-23314.01.patch, HIVE-23314.01.patch
>
>
> we need KUDU-3044 because it could cause random failures...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Status: In Progress (was: Patch Available) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, > hive-23361.6.patch, hive-23361.7.patch, hive-23361.patch, hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
