[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410492#comment-15410492 ] Lefty Leverenz commented on HIVE-13901: --- HIVE-14423 changes the default value of *hive.metastore.fshandler.threads* for release 2.2.0. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Labels: TODOC2.1.1, TODOC2.2 > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch, HIVE-13901.8.patch, HIVE-13901.9.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376690#comment-15376690 ] Lefty Leverenz commented on HIVE-13901: --- Doc note: This adds *hive.metastore.fshandler.threads* to HiveConf.java, so it will need to be documented in the wiki for releases 2.1.1 and 2.2.0. (The wiki should spell out "fs" as "file system" in the description.) * [Configuration Properties -- MetaStore | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] Added TODOC2.1.1 and TODOC2.2 labels. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Labels: TODOC2.1.1, TODOC2.2 > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch, HIVE-13901.8.patch, HIVE-13901.9.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369119#comment-15369119 ] Ashutosh Chauhan commented on HIVE-13901: - +1 > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch, HIVE-13901.8.patch, HIVE-13901.9.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368945#comment-15368945
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816878/HIVE-13901.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10294 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_external2
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/433/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/433/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-433/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816878 - PreCommit-HIVE-MASTER-Build
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch, HIVE-13901.7.patch, HIVE-13901.8.patch, HIVE-13901.9.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367175#comment-15367175 ] Rajesh Balamohan commented on HIVE-13901: - Thanks [~ashutoshc]; Was able to reproduce the issue you encountered. I have updated the patch in RB https://reviews.apache.org/r/48159/ > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365360#comment-15365360
]
Ashutosh Chauhan commented on HIVE-13901:
-
I once again did following:
{code}
git pull
patch -p1 < HIVE-13901.7.patch
mvn clean install -DskipTests && cd itests && mvn clean install -DskipTests &&
cd hive-unit && mvn test
-Dtest=TestEmbeddedHiveMetaStore,TestRemoteHiveMetaStore,TestSetUGIOnBothClientServer,TestSetUGIOnOnlyClient,TestSetUGIOnOnlyServer
{code}
and all of the tests corresponding to testPartition() failed (as reported here
as well). I did this on MacOS. Rajesh, can you try above steps one more time?
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch, HIVE-13901.7.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364737#comment-15364737 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-13901: -- [~rajesh.balamohan] Is the patch ready to be committed? Are the above failures related to the change. Thanks Hari > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352977#comment-15352977
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814005/HIVE-13901.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10285 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityDelayedAllocation
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartition
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/287/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/287/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814005 - PreCommit-HIVE-MASTER-Build
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch, HIVE-13901.7.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340743#comment-15340743
]
Ashutosh Chauhan commented on HIVE-13901:
-
Some of these tests are still failing to me when ran locally:
{code}
Running org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
Tests run: 34, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 91.734 sec <<<
FAILURE! - in org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
testPartition(org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore) Time
elapsed: 1.749 sec <<< FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:201)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.partitionTester(TestHiveMetaStore.java:443)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.testPartition(TestHiveMetaStore.java:146)
{code}
{code}
Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
Tests run: 34, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 91.111 sec <<<
FAILURE! - in org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
testTransactionalValidation(org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore)
Time elapsed: 0.143 sec <<< ERROR!
org.apache.hadoop.hive.metastore.api.AlreadyExistsException: Table acidTable
already exists
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41480)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41466)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:41392)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1183)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1169)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2325)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:726)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.createTable(TestHiveMetaStore.java:2967)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.testTransactionalValidation(TestHiveMetaStore.java:2897)
{code}
{code}
testPartition(org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore) Time
elapsed: 1.675 sec <<< FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:201)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.partitionTester(TestHiveMetaStore.java:443)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.testPartition(TestHiveMetaStore.java:146)
{code}
{code}
testPartition(org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer) Time
elapsed: 1.771 sec <<< FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:201)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.partitionTester(TestHiveMetaStore.java:443)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.testPartition(TestHiveMetaStore.java:146)
{code}
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340475#comment-15340475 ] Sergey Shelukhin commented on HIVE-13901: - +1 > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340468#comment-15340468 ] Rajesh Balamohan commented on HIVE-13901: - This is due to HIVE-14054. With HIVE-14054, all these tests pass in my machine. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339928#comment-15339928 ] Sergey Shelukhin commented on HIVE-13901: - testPartition failures may be related > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338358#comment-15338358
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811456/HIVE-13901.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartition
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/172/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/172/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-172/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811456 - PreCommit-HIVE-MASTER-Build
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337150#comment-15337150
]
Ashutosh Chauhan commented on HIVE-13901:
-
Following tests failed:
{code}
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition
{code}
Atleast few of these seems to be relevant.
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336747#comment-15336747 ] Sergey Shelukhin commented on HIVE-13901: - Back to +1 :) > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335474#comment-15335474 ] Rajesh Balamohan commented on HIVE-13901: - Wrong patch got uploaded in RB causing the issue. Earlier patch ended up removing TP changes in add_partitions_pspec_core accidently. Uploaded the correct patch in RB. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330108#comment-15330108 ] Sergey Shelukhin commented on HIVE-13901: - Um, no +1 for now, some feedback on RB > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328900#comment-15328900 ] Rajesh Balamohan commented on HIVE-13901: - These tests had multiple creations of HSMHandler creating threadpools often than needed. "hive.metastore.fshandler.threads" can be a static daemon thread pool which can be shared in the same JVM. Will make the changes along with rename and post it. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325602#comment-15325602 ] Sergey Shelukhin commented on HIVE-13901: - +1 pending tests and [~ashutoshc] feedback > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325520#comment-15325520 ] Ashutosh Chauhan commented on HIVE-13901: - * Are any of the failures related ? * I think we should pick different name for config like: hive.metastore.fshandler.threads or something similar. * [~sershe] Can you take another look at the patch? > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315770#comment-15315770
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12807865/HIVE-13901.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 75 failed/errored test(s), 10141 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-join4.q-alter_rename_table.q-add_part_exist.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-parallel_join1.q-create_view_partitioned.q-tez_union_group_by.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-smb_mapjoin_15.q-masking_disablecbo_3.q-ppd_udf_case.q-and-12-more
- did not produce a TEST-*.xml file
TestCompactor - did not produce a TEST-*.xml file
TestStreaming - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_external2
org.apache.hadoop.hive.metastore.TestFilterHooks.testDefaultFilter
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestMetaStoreEventListener.testListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
org.apache.hadoop.hive.ql.security.TestAuthorizationPreEventListener.testListener
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[0]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[1]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[2]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[3]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[0]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[1]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[2]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[3]
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[6]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[0]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[1]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[2]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[3]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[0]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[1]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[2]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[3]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[6]
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatExternalDynamicCustomLocation[0]
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315208#comment-15315208 ] Ashutosh Chauhan commented on HIVE-13901: - Reason to not add it is to not bloat Hive's code. Its very easy to introduce bugs in multi-threaded case. Not to speak about bugs related to leaking threads, file descriptors etc. in long running metastore process > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315166#comment-15315166 ] Thomas Poepping commented on HIVE-13901: Seems to me that if the multithreaded approach aids performance in any way, and is actually thread-safe, then there's no reason not to add it. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315163#comment-15315163 ] Ashutosh Chauhan commented on HIVE-13901: - If FileSystem implements listFiles(Path p, boolean recursive) or listStatus(Path[]) then we won't need this multithreaded approach. Seems like atleast HDFS has efficient implementation of listFiles(Path, boolean). > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313410#comment-15313410 ] Sergey Shelukhin commented on HIVE-13901: - left some comments on RB > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313105#comment-15313105
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12807314/HIVE-13901.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/502/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/502/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-502/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-service-rpc ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-service-rpc ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-service-rpc ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-service-rpc ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-service-rpc ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT.pom
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
[INFO]
[INFO]
[INFO] Building Hive Serde 2.2.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-serde ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/serde/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/serde
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-serde ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-serde
---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/serde/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/serde/src/gen/thrift/gen-javabean
added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-serde ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/serde/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-serde ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-serde ---
[INFO] Compiling 414 source files to
/data/hive-ptest/working/apache-github-source-source/serde/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/AbstractPrimitiveLazyObjectInspector.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/AbstractPrimitiveLazyObjectInspector.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15312061#comment-15312061
]
Steve Loughran commented on HIVE-13901:
---
Well, I'm not deeply involved enough in Hive to have any binding opinions, but
I'd still be rejecting this on the basis that you aren't initializing the inner
cause of an exception,
StringifyException does dump the exception to a string, but in the process,
removes it from the data structures which exist precisely for the purpose of
debugging wrapped exceptions.
irrespective of how {{MetaException }} was generated, or what constructor it
has, it will have an {{initCause()}} method.
This will work
{code}
throw (MetaException)new MetaException(e.toString()).initCause(e);
{code}
or more verbosely
{code}
MetaException ex = new MetaException(e.toString());
ex.initCause(e);
throw ex;
{code}
Notice how I also switched from {{getMessage()}} to ((toString()}}. Some
exceptions (NPE) have a null getMessage().
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311421#comment-15311421 ] Rajesh Balamohan commented on HIVE-13901: - Thanks [~steve_l]. MetaException is thrift generated one; instead of e.getMessage, I added StringUtils.stringifyException(e) to retain the contents. > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311419#comment-15311419 ] Rajesh Balamohan commented on HIVE-13901: - Here is the RB link https://reviews.apache.org/r/48159/ > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310604#comment-15310604
]
Steve Loughran commented on HIVE-13901:
---
Those catch and wrap exceptions need to retain the inner one as otherwise the
exception cause can never be really identified.
{code}
// from
throw new MetaException(e.getMessage());
// to
throw new MetaException(e.getMessage(), e);
// or, if that constructor isn't there
throw (MetaException)new MetaException(e.getMessage()).initCause(e);
{code}
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
