[jira] [Commented] (HIVE-10128) LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access
[ https://issues.apache.org/jira/browse/HIVE-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387388#comment-14387388 ] Sergey Shelukhin commented on HIVE-10128: - https://reviews.apache.org/r/32637/ LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access --- Key: HIVE-10128 URL: https://issues.apache.org/jira/browse/HIVE-10128 Project: Hive Issue Type: Sub-task Reporter: Gopal V Assignee: Sergey Shelukhin Fix For: llap Attachments: HIVE-10128.01.patch, HIVE-10128.patch, hashmap-after.png, hashmap-sync-source.png, hashmap-sync.png The multi-threaded performance takes a serious hit when LLAP shares hashtables between the probe threads running in parallel. !hashmap-sync.png! This is an explicit synchronized block inside ReusableRowContainer which triggers this particular pattern. !hashmap-sync-source.png! Looking deeper into the code, the synchronization seems to be caused due to the fact that WriteBuffers.setReadPoint modifies the otherwise read-only hashtable. To generate this sort of result, run LLAP at a WARN log-level, to avoid all the log synchronization that otherwise affects the thread sync. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10128) LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access
[ https://issues.apache.org/jira/browse/HIVE-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387368#comment-14387368 ] Sergey Shelukhin commented on HIVE-10128: - Meanwhile, already committed to llap LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access --- Key: HIVE-10128 URL: https://issues.apache.org/jira/browse/HIVE-10128 Project: Hive Issue Type: Sub-task Reporter: Gopal V Assignee: Sergey Shelukhin Fix For: llap Attachments: HIVE-10128.01.patch, HIVE-10128.patch, hashmap-after.png, hashmap-sync-source.png, hashmap-sync.png The multi-threaded performance takes a serious hit when LLAP shares hashtables between the probe threads running in parallel. !hashmap-sync.png! This is an explicit synchronized block inside ReusableRowContainer which triggers this particular pattern. !hashmap-sync-source.png! Looking deeper into the code, the synchronization seems to be caused due to the fact that WriteBuffers.setReadPoint modifies the otherwise read-only hashtable. To generate this sort of result, run LLAP at a WARN log-level, to avoid all the log synchronization that otherwise affects the thread sync. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10128) LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access
[
https://issues.apache.org/jira/browse/HIVE-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387458#comment-14387458
]
Hive QA commented on HIVE-10128:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12708229/HIVE-10128.01.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3207/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3207/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3207/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-client ---
[INFO] Compiling 5 source files to
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client ---
[INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar
[INFO] Copying guava-14.0.1.jar to
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/dependency/guava-14.0.1.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/spark-client-1.2.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
spark-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/target/spark-client-1.2.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/spark-client/1.2.0-SNAPSHOT/spark-client-1.2.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/spark-client/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/spark-client/1.2.0-SNAPSHOT/spark-client-1.2.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Query Language 1.2.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/ql (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveParser.g
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as {KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
[jira] [Commented] (HIVE-10128) LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access
[ https://issues.apache.org/jira/browse/HIVE-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384970#comment-14384970 ] Sergey Shelukhin commented on HIVE-10128: - rather, bugs were before sync blocks were added earlier today. LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access --- Key: HIVE-10128 URL: https://issues.apache.org/jira/browse/HIVE-10128 Project: Hive Issue Type: Sub-task Reporter: Gopal V Assignee: Sergey Shelukhin Attachments: hashmap-sync-source.png, hashmap-sync.png The multi-threaded performance takes a serious hit when LLAP shares hashtables between the probe threads running in parallel. !hashmap-sync.png! This is an explicit synchronized block inside ReusableRowContainer which triggers this particular pattern. !hashmap-sync-source.png! Looking deeper into the code, the synchronization seems to be caused due to the fact that WriteBuffers.setReadPoint modifies the otherwise read-only hashtable. To generate this sort of result, run LLAP at a WARN log-level, to avoid all the log synchronization that otherwise affects the thread sync. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10128) LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access
[ https://issues.apache.org/jira/browse/HIVE-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385062#comment-14385062 ] Sergey Shelukhin commented on HIVE-10128: - As for logging, I have a patch for that that someone needs to +1 ;) LLAP: BytesBytesMultiHashMap does not allow concurrent read-only access --- Key: HIVE-10128 URL: https://issues.apache.org/jira/browse/HIVE-10128 Project: Hive Issue Type: Sub-task Reporter: Gopal V Assignee: Sergey Shelukhin Attachments: HIVE-10128.patch, hashmap-sync-source.png, hashmap-sync.png The multi-threaded performance takes a serious hit when LLAP shares hashtables between the probe threads running in parallel. !hashmap-sync.png! This is an explicit synchronized block inside ReusableRowContainer which triggers this particular pattern. !hashmap-sync-source.png! Looking deeper into the code, the synchronization seems to be caused due to the fact that WriteBuffers.setReadPoint modifies the otherwise read-only hashtable. To generate this sort of result, run LLAP at a WARN log-level, to avoid all the log synchronization that otherwise affects the thread sync. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
