[
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingxuan Fu updated HIVE-26213:
-------------------------------
Description:
In hive-default.xml.template
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.1</value>
<description>
Expects value between 0.0f and 1.0f.
The fraction of available memory to be used for buffering rows in
Reducesink operator for limit pushdown optimization.
</description>
</property>
Based on the description of hive-default.xml.template,
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
memory of all buffered lines for the limit pushdown optimization, and
successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a
client to access hive, using JDBCDemo as an example.
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception {
JDBCUtils.init();
JDBCUtils.createDatabase();
JDBCUtils.showDatabases();
JDBCUtils.createTable();
JDBCUtils.showTables();
JDBCUtils.descTable();
JDBCUtils.loadData();
JDBCUtils.selectData();
JDBCUtils.countData();
JDBCUtils.dropDatabase();
JDBCUtils.dropTable();
JDBCUtils.destory();
}
}
After running the client program, both the client and the hiveserver throw
exceptions.
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno int
ename string
job string
mgr int
hiredate string
sal double
comm double
deptno int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt'
overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error
while compiling statement: FAILED: SemanticException Invalid memory usage value
1.0 for hive.limit.pushdown.memory.usage
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
at demo.test.JDBCDemo.main(JDBCDemo.java:19)
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
*hive-default.xml.template is not clear enough for the description of the
boundary of the value, it is better to use the interval to indicate the value
that is [0.0,1.0).*
was:
In hive-default.xml.template
{code:java}
<property> <name>hive.limit.pushdown.memory.usage</name> <value>0.1</value>
<description> Expects value between 0.0f and 1.0f. The fraction of available
memory to be used for buffering rows in Reducesink operator for limit pushdown
optimization. </description> </property>
{code}
Based on the description of hive-default.xml.template,
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
memory of all buffered lines for the limit pushdown optimization, and
successfully start hiveserver2.
Then, call the java api to write a program to establish a jdbc connection as a
client to access hive, using JDBCDemo as an example.
{code:java}
import demo.utils.JDBCUtils; public class JDBCDemo{ public static void
main(String[] args) throws Exception { JDBCUtils.init();
JDBCUtils.createDatabase(); JDBCUtils.showDatabases();
JDBCUtils.createTable(); JDBCUtils.showTables(); JDBCUtils.descTable();
JDBCUtils.loadData(); JDBCUtils.selectData(); JDBCUtils.countData();
JDBCUtils.dropDatabase(); JDBCUtils.dropTable(); JDBCUtils.destory(); } }
{code}
After running the client program, both the client and the hiveserver throw
exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2 SLF4J: Class path contains multiple
SLF4J bindings. SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation. SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID =
67a6db8d-f957-4d5d-ac18-28403adab7f3 Hive Session ID =
f9f8772c-5765-4c3e-bcff-ca605c667be7 OK OK OK OK OK OK OK Loading data to table
default.emp OK FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh SLF4J: Class path contains
multiple SLF4J bindings. SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation. SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory] Running: drop database if exists
hive_jdbc_test Running: create database hive_jdbc_test Running: show databases
default hive_jdbc_test Running: drop table if exists emp Running: create table
emp( empno int, ename string, job string, mgr int, hiredate string, sal double,
comm double, deptno int ) row format delimited fields terminated by '\t'
Running: show tables emp Running: desc emp empno int ename string job string
mgr int hiredate string sal double comm double deptno int Running: load data
local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp
Running: select * from emp Exception in thread "main"
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: SemanticException Invalid memory usage value 1.0 for
hive.limit.pushdown.memory.usage at
org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380) at
org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366) at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354) at
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293) at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509) at
demo.utils.JDBCUtils.selectData(JDBCUtils.java:98) at
demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
*hive-default.xml.template is not clear enough for the description of the
boundary of the value, it is better to use the interval to indicate the value
that is [0.0,1.0).*
> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it
> will raise an error
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
> Reporter: Jingxuan Fu
> Assignee: Jingxuan Fu
> Priority: Major
>
> In hive-default.xml.template
> <property>
> <name>hive.limit.pushdown.memory.usage</name>
> <value>0.1</value>
> <description>
> Expects value between 0.0f and 1.0f.
> The fraction of available memory to be used for buffering rows in
> Reducesink operator for limit pushdown optimization.
> </description>
> </property>
> Based on the description of hive-default.xml.template,
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available
> memory of all buffered lines for the limit pushdown optimization, and
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as
> a client to access hive, using JDBCDemo as an example.
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception {
> JDBCUtils.init();
> JDBCUtils.createDatabase();
> JDBCUtils.showDatabases();
> JDBCUtils.createTable();
> JDBCUtils.showTables();
> JDBCUtils.descTable();
> JDBCUtils.loadData();
> JDBCUtils.selectData();
> JDBCUtils.countData();
> JDBCUtils.dropDatabase();
> JDBCUtils.dropTable();
> JDBCUtils.destory();
> }
> }
> After running the client program, both the client and the hiveserver throw
> exceptions.
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for
> hive.limit.pushdown.memory.usage
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno int
> ename string
> job string
> mgr int
> hiredate string
> sal double
> comm double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt'
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException:
> Error while compiling statement: FAILED: SemanticException Invalid memory
> usage value 1.0 for hive.limit.pushdown.memory.usage
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
> at
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
> at
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
> at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
> at demo.test.JDBCDemo.main(JDBCDemo.java:19)
> Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.
> So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable,
> *hive-default.xml.template is not clear enough for the description of the
> boundary of the value, it is better to use the interval to indicate the value
> that is [0.0,1.0).*
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
