[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas
[
https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902359#comment-15902359
]
Hive QA commented on HIVE-15988:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856810/HIVE-15988.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4033/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4033/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4033/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:24:44.545
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4033/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:24:44.548
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 348a592 HIVE-16114 : NullPointerException in
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey
Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 348a592 HIVE-16114 : NullPointerException in
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey
Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:24:45.439
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/test/results/clientpositive/perf/query14.q.out:1
error: ql/src/test/results/clientpositive/perf/query14.q.out: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856810 - PreCommit-HIVE-Build
> LLAP: cache ORC ACID delete deltas
> --
>
> Key: HIVE-15988
> URL: https://issues.apache.org/jira/browse/HIVE-15988
> Project: Hive
> Issue Type: Improvement
> Components: llap, Transactions
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-15988.1.patch
>
>
> HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID
> delete delta in LLAP, it will be faster.
> The cache is currently low-level (CB-level in ORC), so we could just use it
> to read bases and deltas (deltas should be cached with higher priority) and
> merge as usual. We could also cache merged representation in future.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas
[
https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902356#comment-15902356
]
Hive QA commented on HIVE-15988:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856810/HIVE-15988.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4032/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4032/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4032/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:23:28.426
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4032/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:23:28.429
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
bfe930c..348a592 master -> origin/master
+ git reset --hard HEAD
HEAD is now at bfe930c HIVE-16072: LLAP: Add some additional jvm metrics for
hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 348a592 HIVE-16114 : NullPointerException in
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey
Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:23:29.671
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/test/results/clientpositive/perf/query14.q.out:1
error: ql/src/test/results/clientpositive/perf/query14.q.out: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856810 - PreCommit-HIVE-Build
> LLAP: cache ORC ACID delete deltas
> --
>
> Key: HIVE-15988
> URL: https://issues.apache.org/jira/browse/HIVE-15988
> Project: Hive
> Issue Type: Improvement
> Components: llap, Transactions
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-15988.1.patch
>
>
> HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID
> delete delta in LLAP, it will be faster.
> The cache is currently low-level (CB-level in ORC), so we could just use it
> to read bases and deltas (deltas should be cached with higher priority) and
> merge as usual. We could also cache merged representation in future.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas
[ https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901442#comment-15901442 ] Teddy Choi commented on HIVE-15988: --- [~gopalv], it makes sense. Maybe we don't need this issue. > LLAP: cache ORC ACID delete deltas > -- > > Key: HIVE-15988 > URL: https://issues.apache.org/jira/browse/HIVE-15988 > Project: Hive > Issue Type: Improvement > Components: llap, Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi > Attachments: HIVE-15988.1.patch > > > HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID > delete delta in LLAP, it will be faster. > The cache is currently low-level (CB-level in ORC), so we could just use it > to read bases and deltas (deltas should be cached with higher priority) and > merge as usual. We could also cache merged representation in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas
[
https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901394#comment-15901394
]
Gopal V commented on HIVE-15988:
The delete deltas are still really small in memory, so this is more of a
hypothetical concern for a large cache (~4 bits per delete row?).
Looks like we already have a configuration parameter which changes the way ACID
deletes are read into memory, which can also apply to the caching operations -
{{hive.transactional.events.mem}}
> LLAP: cache ORC ACID delete deltas
> --
>
> Key: HIVE-15988
> URL: https://issues.apache.org/jira/browse/HIVE-15988
> Project: Hive
> Issue Type: Improvement
> Components: llap, Transactions
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-15988.1.patch
>
>
> HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID
> delete delta in LLAP, it will be faster.
> The cache is currently low-level (CB-level in ORC), so we could just use it
> to read bases and deltas (deltas should be cached with higher priority) and
> merge as usual. We could also cache merged representation in future.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas
[ https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901370#comment-15901370 ] Gopal V commented on HIVE-15988: [~teddy.choi]: the delete deltas pose a deadlock possibility - since they are not opened one by one, but all at the same time, they have the possibility of locking out the entire cache under some circumstances (i.e all deltas are open, but not all have been consumed). This is not the case with the insert deltas, since only one will be opened per split at a time, with the previous one being closed before the next one is opened - the eviction behaviour is much more easy to model. > LLAP: cache ORC ACID delete deltas > -- > > Key: HIVE-15988 > URL: https://issues.apache.org/jira/browse/HIVE-15988 > Project: Hive > Issue Type: Improvement > Components: llap, Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi > Attachments: HIVE-15988.1.patch > > > HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID > delete delta in LLAP, it will be faster. > The cache is currently low-level (CB-level in ORC), so we could just use it > to read bases and deltas (deltas should be cached with higher priority) and > merge as usual. We could also cache merged representation in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
