[
https://issues.apache.org/jira/browse/HIVE-28237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-28237:
--------------------------------
Description:
currently, data/conf looks like:
{code}
ls -la data/conf
total 88
drwx------ 15 laszlobodor staff 480 Apr 30 11:52 .
drwx------ 7 laszlobodor staff 224 Sep 20 2023 ..
-rw-r--r-- 1 laszlobodor staff 442 Aug 8 2023 fair-scheduler-test.xml
-rw-r--r-- 1 laszlobodor staff 5862 Mar 20 11:53 hive-log4j2.properties
-rw-r--r-- 1 laszlobodor staff 6706 Aug 8 2023 hive-site-old.xml
-rw-r--r-- 1 laszlobodor staff 11934 Apr 30 11:43 hive-site.xml
-rw-r--r-- 1 laszlobodor staff 1681 Sep 26 2023 hivemetastore-site.xml
-rw-r--r-- 1 laszlobodor staff 1582 Aug 8 2023 hiveserver2-site.xml
drwxr-xr-x 4 laszlobodor staff 128 Mar 20 11:53 iceberg
drwxr-xr-x 3 laszlobodor staff 96 Mar 20 11:53 impala
drwx------ 5 laszlobodor staff 160 Apr 29 20:08 llap
drwxr-xr-x 3 laszlobodor staff 96 Mar 20 11:53 perf
drwx------ 4 laszlobodor staff 128 Mar 20 11:53 rlist
drwx------ 4 laszlobodor staff 128 Apr 29 20:08 tez
{code}
so there is no default tez-site.xml, making it confusing for users e.g. of the
StartMiniHS2Cluster
StartMiniHS2Cluster works from this folder, so it has a target (after a mvn
clean install command) like:
{code}
ls -la itests/hive-unit/target/testconf
total 88
drwxr-xr-x 15 laszlobodor staff 480 Apr 30 11:52 .
drwxr-xr-x 15 laszlobodor staff 480 Apr 30 11:53 ..
-rw-r--r-- 1 laszlobodor staff 442 Apr 30 11:52 fair-scheduler-test.xml
-rw-r--r-- 1 laszlobodor staff 5862 Apr 30 11:52 hive-log4j2.properties
-rw-r--r-- 1 laszlobodor staff 6706 Apr 30 11:52 hive-site-old.xml
-rw-r--r-- 1 laszlobodor staff 11934 Apr 30 11:52 hive-site.xml
-rw-r--r-- 1 laszlobodor staff 1681 Apr 30 11:52 hivemetastore-site.xml
-rw-r--r-- 1 laszlobodor staff 1582 Apr 30 11:52 hiveserver2-site.xml
drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 iceberg
drwxr-xr-x 3 laszlobodor staff 96 Apr 30 11:52 impala
drwxr-xr-x 5 laszlobodor staff 160 Apr 30 11:52 llap
drwxr-xr-x 3 laszlobodor staff 96 Apr 30 11:52 perf
drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 rlist
drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 tez
{code}
for a dev, it's confusing as even the default TezConfiguration adds
tez-site.xml as a default resource, it's not obvious where a tez site config is
supposed to come from, ending up hacking tez options into hive-site.xml, which
doesn't work (TezConfiguration object doesn't have hive-site.xml as a default
resource)
the log message of StartMiniHS2Cluster shows:
{code}
2024-04-30T02:53:15,120 INFO [main] conf.HiveConf: Found configuration file
file:/Users/laszlobodor/CDH/hive/itests/hive-unit/target/testconf/hive-site.xml
{code}
which is because we can inject a hive-site.xml directly for testing purposes:
https://github.com/apache/hive/blob/a6069ee816804477c42c254fbec47228e3491982/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/StartMiniHS2Cluster.java#L61
but we cannot do the same for tez-site.xml, and it's okay, and here is where
this jira is supposed to help by adding a tez-site.xml to data/conf that can be
picked up in scenarios where data/conf is used as a resource folder
was:
currently, data/conf looks like:
{code}
ls -la data/conf
total 88
drwx------ 15 laszlobodor staff 480 Apr 30 11:52 .
drwx------ 7 laszlobodor staff 224 Sep 20 2023 ..
-rw-r--r-- 1 laszlobodor staff 442 Aug 8 2023 fair-scheduler-test.xml
-rw-r--r-- 1 laszlobodor staff 5862 Mar 20 11:53 hive-log4j2.properties
-rw-r--r-- 1 laszlobodor staff 6706 Aug 8 2023 hive-site-old.xml
-rw-r--r-- 1 laszlobodor staff 11934 Apr 30 11:43 hive-site.xml
-rw-r--r-- 1 laszlobodor staff 1681 Sep 26 2023 hivemetastore-site.xml
-rw-r--r-- 1 laszlobodor staff 1582 Aug 8 2023 hiveserver2-site.xml
drwxr-xr-x 4 laszlobodor staff 128 Mar 20 11:53 iceberg
drwxr-xr-x 3 laszlobodor staff 96 Mar 20 11:53 impala
drwx------ 5 laszlobodor staff 160 Apr 29 20:08 llap
drwxr-xr-x 3 laszlobodor staff 96 Mar 20 11:53 perf
drwx------ 4 laszlobodor staff 128 Mar 20 11:53 rlist
drwx------ 4 laszlobodor staff 128 Apr 29 20:08 tez
{code}
so there is no default tez-site.xml, making it confusing for users e.g. of the
StartMiniHS2Cluster
StartMiniHS2Cluster works from this folder, so it has a target (after a mvn
clean install command) like:
{code}
ls -la itests/hive-unit/target/testconf
total 88
drwxr-xr-x 15 laszlobodor staff 480 Apr 30 11:52 .
drwxr-xr-x 15 laszlobodor staff 480 Apr 30 11:53 ..
-rw-r--r-- 1 laszlobodor staff 442 Apr 30 11:52 fair-scheduler-test.xml
-rw-r--r-- 1 laszlobodor staff 5862 Apr 30 11:52 hive-log4j2.properties
-rw-r--r-- 1 laszlobodor staff 6706 Apr 30 11:52 hive-site-old.xml
-rw-r--r-- 1 laszlobodor staff 11934 Apr 30 11:52 hive-site.xml
-rw-r--r-- 1 laszlobodor staff 1681 Apr 30 11:52 hivemetastore-site.xml
-rw-r--r-- 1 laszlobodor staff 1582 Apr 30 11:52 hiveserver2-site.xml
drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 iceberg
drwxr-xr-x 3 laszlobodor staff 96 Apr 30 11:52 impala
drwxr-xr-x 5 laszlobodor staff 160 Apr 30 11:52 llap
drwxr-xr-x 3 laszlobodor staff 96 Apr 30 11:52 perf
drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 rlist
drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 tez
{code}
for a dev, it's confusing as even the default TezConfiguration adds
tez-site.xml as a default resource, it's not obvious where a tez site config is
supposed to come from, ending up hacking tez options into hive-site.xml, which
doesn't work (TezConfiguration object doesn't have hive-site.xml as a default
resource)
the log message of StartMiniHS2Cluster shows:
{code}
2024-04-30T02:53:15,120 INFO [main] conf.HiveConf: Found configuration file
file:/Users/laszlobodor/CDH/hive/itests/hive-unit/target/testconf/hive-site.xml
{code}
which is because we can inject a hive-site.xml directly for testing purposes:
https://github.com/apache/hive/blob/a6069ee816804477c42c254fbec47228e3491982/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/StartMiniHS2Cluster.java#L61
but we cannot do the same for tez-site.xml, and it's okay
> Add a default tez-site.xml to data/conf
> ---------------------------------------
>
> Key: HIVE-28237
> URL: https://issues.apache.org/jira/browse/HIVE-28237
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> currently, data/conf looks like:
> {code}
> ls -la data/conf
> total 88
> drwx------ 15 laszlobodor staff 480 Apr 30 11:52 .
> drwx------ 7 laszlobodor staff 224 Sep 20 2023 ..
> -rw-r--r-- 1 laszlobodor staff 442 Aug 8 2023 fair-scheduler-test.xml
> -rw-r--r-- 1 laszlobodor staff 5862 Mar 20 11:53 hive-log4j2.properties
> -rw-r--r-- 1 laszlobodor staff 6706 Aug 8 2023 hive-site-old.xml
> -rw-r--r-- 1 laszlobodor staff 11934 Apr 30 11:43 hive-site.xml
> -rw-r--r-- 1 laszlobodor staff 1681 Sep 26 2023 hivemetastore-site.xml
> -rw-r--r-- 1 laszlobodor staff 1582 Aug 8 2023 hiveserver2-site.xml
> drwxr-xr-x 4 laszlobodor staff 128 Mar 20 11:53 iceberg
> drwxr-xr-x 3 laszlobodor staff 96 Mar 20 11:53 impala
> drwx------ 5 laszlobodor staff 160 Apr 29 20:08 llap
> drwxr-xr-x 3 laszlobodor staff 96 Mar 20 11:53 perf
> drwx------ 4 laszlobodor staff 128 Mar 20 11:53 rlist
> drwx------ 4 laszlobodor staff 128 Apr 29 20:08 tez
> {code}
> so there is no default tez-site.xml, making it confusing for users e.g. of
> the StartMiniHS2Cluster
> StartMiniHS2Cluster works from this folder, so it has a target (after a mvn
> clean install command) like:
> {code}
> ls -la itests/hive-unit/target/testconf
> total 88
> drwxr-xr-x 15 laszlobodor staff 480 Apr 30 11:52 .
> drwxr-xr-x 15 laszlobodor staff 480 Apr 30 11:53 ..
> -rw-r--r-- 1 laszlobodor staff 442 Apr 30 11:52 fair-scheduler-test.xml
> -rw-r--r-- 1 laszlobodor staff 5862 Apr 30 11:52 hive-log4j2.properties
> -rw-r--r-- 1 laszlobodor staff 6706 Apr 30 11:52 hive-site-old.xml
> -rw-r--r-- 1 laszlobodor staff 11934 Apr 30 11:52 hive-site.xml
> -rw-r--r-- 1 laszlobodor staff 1681 Apr 30 11:52 hivemetastore-site.xml
> -rw-r--r-- 1 laszlobodor staff 1582 Apr 30 11:52 hiveserver2-site.xml
> drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 iceberg
> drwxr-xr-x 3 laszlobodor staff 96 Apr 30 11:52 impala
> drwxr-xr-x 5 laszlobodor staff 160 Apr 30 11:52 llap
> drwxr-xr-x 3 laszlobodor staff 96 Apr 30 11:52 perf
> drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 rlist
> drwxr-xr-x 4 laszlobodor staff 128 Apr 30 11:52 tez
> {code}
> for a dev, it's confusing as even the default TezConfiguration adds
> tez-site.xml as a default resource, it's not obvious where a tez site config
> is supposed to come from, ending up hacking tez options into hive-site.xml,
> which doesn't work (TezConfiguration object doesn't have hive-site.xml as a
> default resource)
> the log message of StartMiniHS2Cluster shows:
> {code}
> 2024-04-30T02:53:15,120 INFO [main] conf.HiveConf: Found configuration file
> file:/Users/laszlobodor/CDH/hive/itests/hive-unit/target/testconf/hive-site.xml
> {code}
> which is because we can inject a hive-site.xml directly for testing purposes:
> https://github.com/apache/hive/blob/a6069ee816804477c42c254fbec47228e3491982/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/StartMiniHS2Cluster.java#L61
> but we cannot do the same for tez-site.xml, and it's okay, and here is where
> this jira is supposed to help by adding a tez-site.xml to data/conf that can
> be picked up in scenarios where data/conf is used as a resource folder
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
