[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[ https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284367#comment-14284367 ] Brock Noland commented on HIVE-8121: +1 > Create micro-benchmarks for ParquetSerde and evaluate performance > - > > Key: HIVE-8121 > URL: https://issues.apache.org/jira/browse/HIVE-8121 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Sergio Peña > Attachments: HIVE-8121.6.patch > > > These benchmarks should not execute queries but test only the ParquetSerde > code to ensure we are as efficient as possible. > The output of this JIRA is: > 1) Benchmark tool exists > 2) We create new tasks under HIVE-8120 to track the improvements required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[
https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284101#comment-14284101
]
Hive QA commented on HIVE-8121:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12693315/HIVE-8121.6.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7332 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_window
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2440/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2440/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2440/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12693315 - PreCommit-HIVE-TRUNK-Build
> Create micro-benchmarks for ParquetSerde and evaluate performance
> -
>
> Key: HIVE-8121
> URL: https://issues.apache.org/jira/browse/HIVE-8121
> Project: Hive
> Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Sergio Peña
> Attachments: HIVE-8121.6.patch
>
>
> These benchmarks should not execute queries but test only the ParquetSerde
> code to ensure we are as efficient as possible.
> The output of this JIRA is:
> 1) Benchmark tool exists
> 2) We create new tasks under HIVE-8120 to track the improvements required
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[
https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281248#comment-14281248
]
Hive QA commented on HIVE-8121:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12692771/HIVE-8121.5.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 7308 tests executed
*Failed tests:*
{noformat}
TestSparkClient - did not produce a TEST-*.xml file
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterPartition
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropPartition
org.apache.hive.hcatalog.listener.TestDbNotificationListener.filterWithMax
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2402/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2402/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2402/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12692771 - PreCommit-HIVE-TRUNK-Build
> Create micro-benchmarks for ParquetSerde and evaluate performance
> -
>
> Key: HIVE-8121
> URL: https://issues.apache.org/jira/browse/HIVE-8121
> Project: Hive
> Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Sergio Peña
> Attachments: HIVE-8121.5.patch
>
>
> These benchmarks should not execute queries but test only the ParquetSerde
> code to ensure we are as efficient as possible.
> The output of this JIRA is:
> 1) Benchmark tool exists
> 2) We create new tasks under HIVE-8120 to track the improvements required
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[ https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271653#comment-14271653 ] Sergio Peña commented on HIVE-8121: --- Hi All, I added a new patch to review that uses the JMH framework. https://reviews.apache.org/r/28964/diff/# > Create micro-benchmarks for ParquetSerde and evaluate performance > - > > Key: HIVE-8121 > URL: https://issues.apache.org/jira/browse/HIVE-8121 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Sergio Peña > > These benchmarks should not execute queries but test only the ParquetSerde > code to ensure we are as efficient as possible. > The output of this JIRA is: > 1) Benchmark tool exists > 2) We create new tasks under HIVE-8120 to track the improvements required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[ https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264799#comment-14264799 ] Sergio Peña commented on HIVE-8121: --- Thanks [~sershe] [~prasanth_j] I'll take a look at JMH. > Create micro-benchmarks for ParquetSerde and evaluate performance > - > > Key: HIVE-8121 > URL: https://issues.apache.org/jira/browse/HIVE-8121 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Sergio Peña > > These benchmarks should not execute queries but test only the ParquetSerde > code to ensure we are as efficient as possible. > The output of this JIRA is: > 1) Benchmark tool exists > 2) We create new tasks under HIVE-8120 to track the improvements required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[ https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243152#comment-14243152 ] Prasanth Jayachandran commented on HIVE-8121: - IMO, there are many pitfalls of doing microbenchmarking like the one in this patch. Lots of them are highlighted in this video by Aleksey Shipilëv (the guy who leads the development of JMH) https://parleys.com/play/52ae09bde4b04354fb7e7cfd/chapter67/about IMHO, we should use JMH for any sorts of micro-benchmarking to get correct and more accurate results. > Create micro-benchmarks for ParquetSerde and evaluate performance > - > > Key: HIVE-8121 > URL: https://issues.apache.org/jira/browse/HIVE-8121 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Sergio Peña > > These benchmarks should not execute queries but test only the ParquetSerde > code to ensure we are as efficient as possible. > The output of this JIRA is: > 1) Benchmark tool exists > 2) We create new tasks under HIVE-8120 to track the improvements required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[ https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243141#comment-14243141 ] Sergey Shelukhin commented on HIVE-8121: Have you considered using some standard micro-benchmarking tools, like JMH? [~prasanth_j] fyi > Create micro-benchmarks for ParquetSerde and evaluate performance > - > > Key: HIVE-8121 > URL: https://issues.apache.org/jira/browse/HIVE-8121 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Sergio Peña > > These benchmarks should not execute queries but test only the ParquetSerde > code to ensure we are as efficient as possible. > The output of this JIRA is: > 1) Benchmark tool exists > 2) We create new tasks under HIVE-8120 to track the improvements required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8121) Create micro-benchmarks for ParquetSerde and evaluate performance
[
https://issues.apache.org/jira/browse/HIVE-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243031#comment-14243031
]
Sergio Peña commented on HIVE-8121:
---
You can run ORC and PARQUET tests in the patch attached using the following
parameters:
{noformat}
# java TestHiveStoragePerformance -f parquet
# java TestHiveStoragePerformance -f orc
{noformat}
> Create micro-benchmarks for ParquetSerde and evaluate performance
> -
>
> Key: HIVE-8121
> URL: https://issues.apache.org/jira/browse/HIVE-8121
> Project: Hive
> Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Sergio Peña
>
> These benchmarks should not execute queries but test only the ParquetSerde
> code to ensure we are as efficient as possible.
> The output of this JIRA is:
> 1) Benchmark tool exists
> 2) We create new tasks under HIVE-8120 to track the improvements required
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
