[jira] [Commented] (HIVE-14392) llap daemons should try using YARN local dirs, if available
[ https://issues.apache.org/jira/browse/HIVE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411448#comment-15411448 ] Lefty Leverenz commented on HIVE-14392: --- Doc note: This changes the description of *hive.llap.daemon.work.dirs* in 2.2.0, so it will need to be documented in the wiki. Previous JIRA issues for *hive.llap.daemon.work.dirs* are HIVE-11358 (2.0.0) and HIVE-12687 (also 2.0.0). * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC2.2 label. > llap daemons should try using YARN local dirs, if available > --- > > Key: HIVE-14392 > URL: https://issues.apache.org/jira/browse/HIVE-14392 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14392.01.patch, HIVE-14392.02.patch > > > LLAP required hive.llap.daemon.work.dirs to be specified. When running as a > YARN app - this can use the local dirs for the container - removing the > requirement to setup this parameter (for secure and non-secure clusters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14392) llap daemons should try using YARN local dirs, if available
[ https://issues.apache.org/jira/browse/HIVE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404567#comment-15404567 ] Sergey Shelukhin commented on HIVE-14392: - +1, there was some other feedback above > llap daemons should try using YARN local dirs, if available > --- > > Key: HIVE-14392 > URL: https://issues.apache.org/jira/browse/HIVE-14392 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14392.01.patch > > > LLAP required hive.llap.daemon.work.dirs to be specified. When running as a > YARN app - this can use the local dirs for the container - removing the > requirement to setup this parameter (for secure and non-secure clusters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14392) llap daemons should try using YARN local dirs, if available
[ https://issues.apache.org/jira/browse/HIVE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402904#comment-15402904 ] Siddharth Seth commented on HIVE-14392: --- [~leftylev] - thanks for the detailed look at the description. Will incorporate most of that in the next patch. bq. Why did we make it required in the first place? I remember there was an explicit reason, I just don't remember what it was. We were always running on YARN only, too LLAP started with running without Slider - i.e. setup daemons manually on individual nodes (outside of YARN). At that point the work.dir configuration was required. I think we just never really needed to change the way it was used, so this did not get attention. > llap daemons should try using YARN local dirs, if available > --- > > Key: HIVE-14392 > URL: https://issues.apache.org/jira/browse/HIVE-14392 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14392.01.patch > > > LLAP required hive.llap.daemon.work.dirs to be specified. When running as a > YARN app - this can use the local dirs for the container - removing the > requirement to setup this parameter (for secure and non-secure clusters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14392) llap daemons should try using YARN local dirs, if available
[ https://issues.apache.org/jira/browse/HIVE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402567#comment-15402567 ] Sergey Shelukhin commented on HIVE-14392: - Why did we make it required in the first place? I remember there was an explicit reason, I just don't remember what it was. > llap daemons should try using YARN local dirs, if available > --- > > Key: HIVE-14392 > URL: https://issues.apache.org/jira/browse/HIVE-14392 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14392.01.patch > > > LLAP required hive.llap.daemon.work.dirs to be specified. When running as a > YARN app - this can use the local dirs for the container - removing the > requirement to setup this parameter (for secure and non-secure clusters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14392) llap daemons should try using YARN local dirs, if available
[
https://issues.apache.org/jira/browse/HIVE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401860#comment-15401860
]
Hive QA commented on HIVE-14392:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12821183/HIVE-14392.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10418 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testNegativeCliDriver_groupby2_map_skew_multi_distinct
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testNegativeCliDriver_groupby2_multi_distinct
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testNegativeCliDriver_groupby3_map_skew_multi_distinct
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testNegativeCliDriver_groupby3_multi_distinct
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets7
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/719/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/719/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-719/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12821183 - PreCommit-HIVE-MASTER-Build
> llap daemons should try using YARN local dirs, if available
> ---
>
> Key: HIVE-14392
> URL: https://issues.apache.org/jira/browse/HIVE-14392
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14392.01.patch
>
>
> LLAP required hive.llap.daemon.work.dirs to be specified. When running as a
> YARN app - this can use the local dirs for the container - removing the
> requirement to setup this parameter (for secure and non-secure clusters).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14392) llap daemons should try using YARN local dirs, if available
[
https://issues.apache.org/jira/browse/HIVE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401070#comment-15401070
]
Lefty Leverenz commented on HIVE-14392:
---
Review of parameter description in the first patch:
{code}
LLAP_DAEMON_WORK_DIRS("hive.llap.daemon.work.dirs", "",
- "Working directories for the daemon. Needs to be set for a secure
cluster, since LLAP may\n" +
- "not have access to the default YARN working directories.
yarn.nodemanager.local-dirs is\n" +
- "used if this is not set", "llap.daemon.work.dirs"),
+"Working directories for the daemon. This should not be set if running
as a Slider" +
+" app on YARN. It must be setup when not running as a Slider
application. If the value" +
+" is set when running as a Slider app - the specified value will
be used.",
+"llap.daemon.work.dirs"),
{code}
1. The first three lines of the new description need to end with newline (\n)
just like the old description. Otherwise it will appear as a single line in
the generated file hive-default.xml.template.
2. Indentation should be uniform, just like the old description, and no
leading space is needed on the second and third lines if \n is used.
3. "It must be setup" --> "It must be set" (or "It must be set up").
4. Slider app / application / app: although "app" is okay, you should spell
out "application" the first time you use it, not the second time. Spelling it
out all three times would also be good.
5. "as a Slider app - the specified value" --> "as a Slider app, the specified
value" (or use two hyphens if you really want a dash there).
> llap daemons should try using YARN local dirs, if available
> ---
>
> Key: HIVE-14392
> URL: https://issues.apache.org/jira/browse/HIVE-14392
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14392.01.patch
>
>
> LLAP required hive.llap.daemon.work.dirs to be specified. When running as a
> YARN app - this can use the local dirs for the container - removing the
> requirement to setup this parameter (for secure and non-secure clusters).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
