[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411459#comment-15411459 ] Lefty Leverenz commented on HIVE-11358: --- Also, HIVE-12687 changes the description of *hive.llap.daemon.work.dirs* in 2.0.0 and HIVE-14392 changes it again in 2.2.0. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118995#comment-15118995 ] Lefty Leverenz commented on HIVE-11358: --- For what it's worth, HIVE-12470 changes the default value of *hive.llap.daemon.service.hosts* from "" to null in release 2.0.0. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043530#comment-15043530 ] Lefty Leverenz commented on HIVE-11358: --- Doc note: All the LLAP configs need to be added in a new section of the wiki. * [Configuration Properties | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] For the record, here are the 28 new configs: * hive.llap.daemon.rpc.num.handlers * hive.llap.daemon.work.dirs * hive.llap.daemon.yarn.shuffle.port * hive.llap.daemon.yarn.container.mb * hive.llap.daemon.shuffle.dir.watcher.enabled * hive.llap.daemon.am.liveness.heartbeat.interval.ms * hive.llap.am.liveness.connection.timeout.ms * hive.llap.am.liveness.connection.sleep.between.retries.ms (not used yet, according to the comment) * hive.llap.daemon.num.executors * hive.llap.daemon.rpc.port * hive.llap.daemon.memory.per.instance.mb * hive.llap.daemon.vcpus.per.instance * hive.llap.daemon.num.file.cleaner.threads * hive.llap.file.cleanup.delay.seconds * hive.llap.daemon.service.hosts * hive.llap.daemon.service.refresh.interval.sec * hive.llap.daemon.communicator.num.threads * hive.llap.task.scheduler.node.reenable.min.timeout.ms * hive.llap.task.scheduler.node.reenable.max.timeout.ms * hive.llap.task.scheduler.node.disable.backoff.factor * hive.llap.task.scheduler.num.schedulable.tasks.per.node * hive.llap.daemon.task.scheduler.wait.queue.size * hive.llap.daemon.wait.queue.comparator.class.name * hive.llap.daemon.task.scheduler.enable.preemption * hive.llap.task.communicator.connection.timeout.ms * hive.llap.task.communicator.connection.sleep.between.retries.ms * hive.llap.daemon.web.port * hive.llap.daemon.web.ssl > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15040812#comment-15040812 ] Lefty Leverenz commented on HIVE-11358: --- Yes, +1 for all the configuration parameters. Now the light dawns. Thanks for the explanation, I was just being dunderheaded. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[
https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037542#comment-15037542
]
Hive QA commented on HIVE-11358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775207/HIVE-11358.05.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9885 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6210/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6210/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775207 - PreCommit-HIVE-TRUNK-Build
> LLAP: move LlapConfiguration into HiveConf and document the settings
>
>
> Key: HIVE-11358
> URL: https://issues.apache.org/jira/browse/HIVE-11358
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch,
> HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch,
> HIVE-11358.patch
>
>
> Hive uses HiveConf for configuration. LlapConfiguration should be replaced
> with parameters in HiveConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[
https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037026#comment-15037026
]
Sergey Shelukhin commented on HIVE-11358:
-
Should I assume the last comment on RB is a +1? :) I will see if I can reword
the description better, but I think it's already understandable..
{noformat}
+ "Connection timeout (in milliseconds) before a failure to an LLAP daemon
from Tez AM.",
+ "llap.task.communicator.connection.timeout-millis"),
and
+ "Sleep duration (in milliseconds) to wait before retrying on error when
obtaining a\n" +
+ "connection to LLAP daemon from Tez AM.",
{noformat}
One is the standard connection timeout before we give up and there's a failure,
the other is a wait interval before retrying.
> LLAP: move LlapConfiguration into HiveConf and document the settings
>
>
> Key: HIVE-11358
> URL: https://issues.apache.org/jira/browse/HIVE-11358
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch,
> HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch,
> HIVE-11358.patch
>
>
> Hive uses HiveConf for configuration. LlapConfiguration should be replaced
> with parameters in HiveConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1509#comment-1509 ] Lefty Leverenz commented on HIVE-11358: --- I left a couple of comments on RB -- one small correction, one bit of confusion. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032352#comment-15032352 ] Sergey Shelukhin commented on HIVE-11358: - Test failures are unrelated. [~sseth] [~leftylev] do you have any more feedback? > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[
https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030918#comment-15030918
]
Hive QA commented on HIVE-11358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774456/HIVE-11358.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9865 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6162/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6162/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6162/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774456 - PreCommit-HIVE-TRUNK-Build
> LLAP: move LlapConfiguration into HiveConf and document the settings
>
>
> Key: HIVE-11358
> URL: https://issues.apache.org/jira/browse/HIVE-11358
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch,
> HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.patch
>
>
> Hive uses HiveConf for configuration. LlapConfiguration should be replaced
> with parameters in HiveConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[
https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027218#comment-15027218
]
Hive QA commented on HIVE-11358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12773911/HIVE-11358.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 9825 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_nonascii
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_join_transpose
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_fetchwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_mapwork_table
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_fetchwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dynpart_hashjoin_3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6124/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6124/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6124/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12773911 - PreCommit-HIVE-TRUNK-Build
> LLAP: move LlapConfiguration into HiveConf and document the settings
>
>
> Key: HIVE-11358
> URL: https://issues.apache.org/jira/browse/HIVE-11358
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch,
> HIVE-11358.03.patch, HIVE-11358.patch
>
>
> Hive uses HiveConf for configuration. LlapConfiguration should be replaced
> with parameters in HiveConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023138#comment-15023138 ] Sergey Shelukhin commented on HIVE-11358: - Addressed the comments. TODOs will be addressed in a separate JIRA. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020341#comment-15020341 ] Lefty Leverenz commented on HIVE-11358: --- [~sershe], I put review comments on RB, most of them trivial. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014656#comment-15014656 ] Sergey Shelukhin commented on HIVE-11358: - [~leftylev] can you look at the lastest patch? thanks! > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015359#comment-15015359 ] Lefty Leverenz commented on HIVE-11358: --- I can review it Friday, [~sershe]. Thanks for all the fixes. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[
https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009022#comment-15009022
]
Hive QA commented on HIVE-11358:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772583/HIVE-11358.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9746 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6056/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6056/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772583 - PreCommit-HIVE-TRUNK-Build
> LLAP: move LlapConfiguration into HiveConf and document the settings
>
>
> Key: HIVE-11358
> URL: https://issues.apache.org/jira/browse/HIVE-11358
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11358.01.patch, HIVE-11358.patch
>
>
> Hive uses HiveConf for configuration. LlapConfiguration should be replaced
> with parameters in HiveConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009330#comment-15009330 ] Siddharth Seth commented on HIVE-11358: --- bq. One question is, do we need a separate daemon config, or should we just use hive-site.xml for everything? Separate is better IMO, something other than hive-site.xml is less ambiguous. Also leaving this as hive-site likely means a large (100+ k-v) config file for LLAP - of which only about 10 are required. Given LLAP is a new component - we could get started with separating client and server configs - along with the files. e.g. llap.daemon.rpc.num.handlers (and a lot of other configs) <- useless in client configs (hive-site on the client / hive-server) llap.daemon.num.executors (and a few other configs) <- #threads for a daemon. Used by the client to determine available slots when not using slider. The client config could be renamed. IAC, that's beyond the scope of what this jira is doing. Using annotations to separate config parameters to generate the public list of configurable fields would work. I'll create a separate jira to separate llap server-side and client-side configs. +1 for the patch. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010257#comment-15010257 ] Lefty Leverenz commented on HIVE-11358: --- General questions: 1. Why don't the parameters that take a time value use a TimeValidator? For examples, see hive.spark.client.future.timeout in the patch (seconds) and hive.stats.retries.wait in HiveConf.java (milliseconds). 2. Why is milliseconds "ms" in some parameter names and "millis" in others? 3. Why do several parameter names use hyphens where normally there would be dots? Hyphens in parameter names: * hive.llap.daemon.shuffle.dir-watcher.enabled * hive.llap.daemon.am.liveness.heartbeat.interval-ms * hive.llap.am.liveness.connection.timeout-millis * hive.llap.am.liveness.connection.sleep-between-retries-millis * hive.llap.file.cleanup.delay-seconds * hive.llap.task.communicator.connection.timeout-millis * hive.llap.task.communicator.connection.sleep-between-retries-millis Also hive.llap.task.scheduler.node.re-enable.min.timeout.ms & hive.llap.task.scheduler.node.re-enable.max.timeout.ms but those are genuine hyphens. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007381#comment-15007381 ] Siddharth Seth commented on HIVE-11358: --- Is there a way to group the LLAP variables separately in HiveConf ? The mechanics for validation and parameter access offered by HiveConf are definitely useful. At the same time, it would be useful to separate out LLAP parameters (and within that separate out client / daemon config properties at a later point). llap-daemon-site still exists - but there will be confusion about what goes into this, vs what goes into hive-site.xml. Does the daemon require a hive-site while starting up ? With YARN and HDFS - a perpetual source of confusion has been separation of client vs server configs. This ends up with servers and clients using the same config files (sane path), and the same property names. As a result - knowing whether a parameter change affects the server / client is confusing, also where a change is required is confusing. Separating the configs within HiveConf would be useful for something like this. Functionally, the patch looks good. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007709#comment-15007709 ] Sergey Shelukhin commented on HIVE-11358: - Hive has settings for several servers, in particular metastore and HS2, already, and they use the same config. Existing LLAP settings for IO also live there. HS2, CLI and metastore configs also have settings that are used in 2 or all 3 components. I am not sure there can be any way to separate them that is actually helpful. Users won't read the source to determine which config to use. One question is, do we need a separate daemon config, or should we just use hive-site.xml for everything? We can add a tool to generate configs for each scope by adding some scope setting to HiveConf vars, but it should be done in separate JIRA. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
