[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902346#comment-15902346
]
Hive QA commented on HIVE-16119:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856809/HIVE-16119.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10335 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4031/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4031/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4031/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856809 - PreCommit-HIVE-Build
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch, HIVE-16119.2.patch,
> HIVE-16119.3.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1596#comment-1596
]
Hive QA commented on HIVE-16119:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856608/HIVE-16119.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10328 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3999/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3999/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3999/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856608 - PreCommit-HIVE-Build
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch, HIVE-16119.2.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899887#comment-15899887
]
Ashutosh Chauhan commented on HIVE-16119:
-
+1
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch, HIVE-16119.2.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899840#comment-15899840
]
Vihang Karajgaonkar commented on HIVE-16119:
Thanks for the changes. LGTM
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch, HIVE-16119.2.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897817#comment-15897817
]
Ashutosh Chauhan commented on HIVE-16119:
-
I agree there is no need to disregard user config anymore.
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897755#comment-15897755
]
Vihang Karajgaonkar commented on HIVE-16119:
On a second thought, I wonder if we really need to increase the pool size
disregarding user configs. The reason it was added in the first place was due
to an issue with the recursive call. Since that is no more the case
(HIVE-15879), we should probably keep using the poolSize given by the user.
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897749#comment-15897749
]
Vihang Karajgaonkar commented on HIVE-16119:
Thanks for the patch [~kgyrtkirk]. Always, a good idea to remove redundant
code. Just one comment from my side. The conditional check {{poolSize =
poolSize == 0 ? poolSize : Math.max(poolSize, getMinPoolSize());}} is not
needed since it is executing in the else block. We could just replace it with
{{poolSize = Math.max(poolSize, getMinPoolSize());}}
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication
[
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897643#comment-15897643
]
Hive QA commented on HIVE-16119:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856282/HIVE-16119.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3964/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3964/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3964/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856282 - PreCommit-HIVE-Build
> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
