[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-11-06 Thread ECFuzz (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ECFuzz updated HIVE-26213:
--
Component/s: Documentation

> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it 
> will raise an error
> ---
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
>Reporter: ECFuzz
>Assignee: ECFuzz
>Priority: Minor
>
> In hive-default.xml.template
> {code:java}
> 
>   hive.limit.pushdown.memory.usage
>   0.1
>   
>     Expects value between 0.0f and 1.0f.
>     The fraction of available memory to be used for buffering rows in 
> Reducesink operator for limit pushdown optimization.
>   
> {code}
> Based on the description of hive-default.xml.template, 
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
> memory of all buffered lines for the limit pushdown optimization, and 
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as 
> a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> {   JDBCUtils.init();   JDBCUtils.createDatabase();   
> JDBCUtils.showDatabases();   JDBCUtils.createTable();   
> JDBCUtils.showTables();   JDBCUtils.descTable();   JDBCUtils.loadData();   
> JDBCUtils.selectData();   JDBCUtils.countData();   JDBCUtils.dropDatabase();  
>  JDBCUtils.dropTable();   JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw 
> exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for 
> hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno   int
> ename   string
> job     string
> mgr     int
> hiredate       string
> sal     double
> comm   double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: 
> Error while compiling statement: FAILED: SemanticException Invalid memory 
> usage value 1.0 for hive.limit.pushdown.memory.usage
>       at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
>       at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
>       at 
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
>       at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
>       at 
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
>       at demo.utils.JDBCUtils

[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-05-09 Thread Jingxuan Fu (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingxuan Fu updated HIVE-26213:
---
Description: 
In hive-default.xml.template

    hive.limit.pushdown.memory.usage
    0.1
    
      Expects value between 0.0f and 1.0f.
      The fraction of available memory to be used for buffering rows in 
Reducesink operator for limit pushdown optimization.
    
  
Based on the description of hive-default.xml.template, 
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
memory of all buffered lines for the limit pushdown optimization, and 
successfully start hiveserver2.

Then, call the java api to write a program to establish a jdbc connection as a 
client to access hive, using JDBCDemo as an example.
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception {
    JDBCUtils.init();
    JDBCUtils.createDatabase();
    JDBCUtils.showDatabases();
    JDBCUtils.createTable();
    JDBCUtils.showTables();
    JDBCUtils.descTable();
    JDBCUtils.loadData();
    JDBCUtils.selectData();
    JDBCUtils.countData();
    JDBCUtils.dropDatabase();
    JDBCUtils.dropTable();
    JDBCUtils.destory();
}
}
After running the client program, both the client and the hiveserver throw 
exceptions.
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for 
hive.limit.pushdown.memory.usage
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno   int
ename   string
job     string
mgr     int
hiredate        string
sal     double
comm    double
deptno  int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error 
while compiling statement: FAILED: SemanticException Invalid memory usage value 
1.0 for hive.limit.pushdown.memory.usage
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
        at 
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
        at 
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
        at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
        at demo.test.JDBCDemo.main(JDBCDemo.java:19)
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.

So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, 
*hive-default.xml.template is not clear enough for the description of the 
boundary of the value, it is better to use the interval to indicate the value 
that is [0.0,1.0).*

  was:
In hive-default.xml.template

 

 

 

 
{code:java}
 hive.limit.pushdown.memory.usage 0.1 
 Expects value between 0.0f and 1.0f. The fraction of available 
memory to be used for buffering rows in Reducesink operator for limit pushdown 
optimization.  
{code}
 

 

Based on the description of hive-default.xml.template, 
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
memory of all buffered lin

[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-05-09 Thread Jingxuan Fu (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingxuan Fu updated HIVE-26213:
---
Description: 
In hive-default.xml.template
{code:java}

  hive.limit.pushdown.memory.usage
  0.1
  
    Expects value between 0.0f and 1.0f.
    The fraction of available memory to be used for buffering rows in 
Reducesink operator for limit pushdown optimization.
  
{code}
Based on the description of hive-default.xml.template, 
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
memory of all buffered lines for the limit pushdown optimization, and 
successfully start hiveserver2.

 

Then, call the java api to write a program to establish a jdbc connection as a 
client to access hive, using JDBCDemo as an example.

    hive.limit.pushdown.memory.usage
    0.1
    
      Expects value between 0.0f and 1.0f.
      The fraction of available memory to be used for buffering rows in 
Reducesink operator for limit pushdown optimization.
    
  
Based on the description of hive-default.xml.template, 
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
memory of all buffered lines for the limit pushdown optimization, and 
successfully start hiveserver2.

Then, call the java api to write a program to establish a jdbc connection as a 
client to access hive, using JDBCDemo as an example.
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception {
    JDBCUtils.init();
    JDBCUtils.createDatabase();
    JDBCUtils.showDatabases();
    JDBCUtils.createTable();
    JDBCUtils.showTables();
    JDBCUtils.descTable();
    JDBCUtils.loadData();
    JDBCUtils.selectData();
    JDBCUtils.countData();
    JDBCUtils.dropDatabase();
    JDBCUtils.dropTable();
    JDBCUtils.destory();
}
}
After running the client program, both the client and the hiveserver throw 
exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for 
hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno   int
ename   string
job     string
mgr     int
hiredate       string
sal     double
comm   double
deptno int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error 
while compiling statement: FAILED: SemanticException Invalid memory usage value 
1.0 for hive.limit.pushdown.memory.usage
      at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
      at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
      at 
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
      at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
      at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
      at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
      at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.

So, setting hive.limit.pushdown.memory.usage to 1.0 is 

[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-05-09 Thread Jingxuan Fu (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingxuan Fu updated HIVE-26213:
---
Description: 
In hive-default.xml.template
{code:java}

  hive.limit.pushdown.memory.usage
  0.1
  
    Expects value between 0.0f and 1.0f.
    The fraction of available memory to be used for buffering rows in 
Reducesink operator for limit pushdown optimization.
  
{code}
Based on the description of hive-default.xml.template, 
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
memory of all buffered lines for the limit pushdown optimization, and 
successfully start hiveserver2.

Then, call the java api to write a program to establish a jdbc connection as a 
client to access hive, using JDBCDemo as an example.
{code:java}
import demo.utils.JDBCUtils;
public class JDBCDemo{
public static void main(String[] args) throws Exception
{   JDBCUtils.init();   JDBCUtils.createDatabase();   
JDBCUtils.showDatabases();   JDBCUtils.createTable();   JDBCUtils.showTables(); 
  JDBCUtils.descTable();   JDBCUtils.loadData();   JDBCUtils.selectData();   
JDBCUtils.countData();   JDBCUtils.dropDatabase();   JDBCUtils.dropTable();   
JDBCUtils.destory(); }
}
{code}
After running the client program, both the client and the hiveserver throw 
exceptions.
{code:java}
2022-05-09 19:05:36: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
OK
OK
OK
OK
OK
OK
OK
Loading data to table default.emp
OK
FAILED: SemanticException Invalid memory usage value 1.0 for 
hive.limit.pushdown.memory.usage{code}
{code:java}
liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Running: drop database if exists hive_jdbc_test
Running: create database hive_jdbc_test
Running: show databases
default
hive_jdbc_test
Running: drop table if exists emp
Running: create table emp(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
Running: show tables
emp
Running: desc emp
empno   int
ename   string
job     string
mgr     int
hiredate       string
sal     double
comm   double
deptno int
Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
overwrite into table emp
Running: select * from emp
Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error 
while compiling statement: FAILED: SemanticException Invalid memory usage value 
1.0 for hive.limit.pushdown.memory.usage
      at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
      at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
      at 
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
      at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
      at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
      at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
      at demo.test.JDBCDemo.main(JDBCDemo.java:19){code}
Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.

So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, 
*hive-default.xml.template is not clear enough for the description of the 
boundary of the value, it is better to use the interval to indicate the value 
that is [0.0,1.0).*

  was:
In hive-default.xml.template
{code:java}

  hive.limit.pushdown.memory.usage
  0.1
  
    Expects value between 0.0f and 1.0f.
    The fraction of available memory to be used for buffering rows in 
Reducesink operator for limit pushdown optimization.
  
{code}
Based on the description of hive-default.xml.template, 
hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
memory of 

[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-07-26 Thread Jingxuan Fu (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingxuan Fu updated HIVE-26213:
---
Issue Type: Improvement  (was: Bug)

> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it 
> will raise an error
> ---
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Major
>
> In hive-default.xml.template
> {code:java}
> 
>   hive.limit.pushdown.memory.usage
>   0.1
>   
>     Expects value between 0.0f and 1.0f.
>     The fraction of available memory to be used for buffering rows in 
> Reducesink operator for limit pushdown optimization.
>   
> {code}
> Based on the description of hive-default.xml.template, 
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
> memory of all buffered lines for the limit pushdown optimization, and 
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as 
> a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> {   JDBCUtils.init();   JDBCUtils.createDatabase();   
> JDBCUtils.showDatabases();   JDBCUtils.createTable();   
> JDBCUtils.showTables();   JDBCUtils.descTable();   JDBCUtils.loadData();   
> JDBCUtils.selectData();   JDBCUtils.countData();   JDBCUtils.dropDatabase();  
>  JDBCUtils.dropTable();   JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw 
> exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for 
> hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno   int
> ename   string
> job     string
> mgr     int
> hiredate       string
> sal     double
> comm   double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: 
> Error while compiling statement: FAILED: SemanticException Invalid memory 
> usage value 1.0 for hive.limit.pushdown.memory.usage
>       at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
>       at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
>       at 
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
>       at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
>       at 
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
>       at demo.utils.JDBCUtils

[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-07-26 Thread Jingxuan Fu (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingxuan Fu updated HIVE-26213:
---
Priority: Minor  (was: Major)

> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it 
> will raise an error
> ---
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Minor
>
> In hive-default.xml.template
> {code:java}
> 
>   hive.limit.pushdown.memory.usage
>   0.1
>   
>     Expects value between 0.0f and 1.0f.
>     The fraction of available memory to be used for buffering rows in 
> Reducesink operator for limit pushdown optimization.
>   
> {code}
> Based on the description of hive-default.xml.template, 
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
> memory of all buffered lines for the limit pushdown optimization, and 
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as 
> a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> {   JDBCUtils.init();   JDBCUtils.createDatabase();   
> JDBCUtils.showDatabases();   JDBCUtils.createTable();   
> JDBCUtils.showTables();   JDBCUtils.descTable();   JDBCUtils.loadData();   
> JDBCUtils.selectData();   JDBCUtils.countData();   JDBCUtils.dropDatabase();  
>  JDBCUtils.dropTable();   JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw 
> exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for 
> hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno   int
> ename   string
> job     string
> mgr     int
> hiredate       string
> sal     double
> comm   double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: 
> Error while compiling statement: FAILED: SemanticException Invalid memory 
> usage value 1.0 for hive.limit.pushdown.memory.usage
>       at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
>       at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
>       at 
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
>       at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
>       at 
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
>       at demo.utils.JDBCUtils.selec

[jira] [Updated] (HIVE-26213) "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

2022-08-06 Thread Jingxuan Fu (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingxuan Fu updated HIVE-26213:
---
Issue Type: Bug  (was: Improvement)

> "hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it 
> will raise an error
> ---
>
> Key: HIVE-26213
> URL: https://issues.apache.org/jira/browse/HIVE-26213
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
> Environment: Hive 3.1.2
> os.name=Linux
> os.arch=amd64
> os.version=5.4.0-72-generic
> java.version=1.8.0_162
> java.vendor=Oracle Corporation
>Reporter: Jingxuan Fu
>Assignee: Jingxuan Fu
>Priority: Minor
>
> In hive-default.xml.template
> {code:java}
> 
>   hive.limit.pushdown.memory.usage
>   0.1
>   
>     Expects value between 0.0f and 1.0f.
>     The fraction of available memory to be used for buffering rows in 
> Reducesink operator for limit pushdown optimization.
>   
> {code}
> Based on the description of hive-default.xml.template, 
> hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting 
> hive.limit.pushdown.memory.usage to 1.0 means that it expects the available 
> memory of all buffered lines for the limit pushdown optimization, and 
> successfully start hiveserver2.
> Then, call the java api to write a program to establish a jdbc connection as 
> a client to access hive, using JDBCDemo as an example.
> {code:java}
> import demo.utils.JDBCUtils;
> public class JDBCDemo{
> public static void main(String[] args) throws Exception
> {   JDBCUtils.init();   JDBCUtils.createDatabase();   
> JDBCUtils.showDatabases();   JDBCUtils.createTable();   
> JDBCUtils.showTables();   JDBCUtils.descTable();   JDBCUtils.loadData();   
> JDBCUtils.selectData();   JDBCUtils.countData();   JDBCUtils.dropDatabase();  
>  JDBCUtils.dropTable();   JDBCUtils.destory(); }
> }
> {code}
> After running the client program, both the client and the hiveserver throw 
> exceptions.
> {code:java}
> 2022-05-09 19:05:36: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
> Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
> OK
> OK
> OK
> OK
> OK
> OK
> OK
> Loading data to table default.emp
> OK
> FAILED: SemanticException Invalid memory usage value 1.0 for 
> hive.limit.pushdown.memory.usage{code}
> {code:java}
> liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Running: drop database if exists hive_jdbc_test
> Running: create database hive_jdbc_test
> Running: show databases
> default
> hive_jdbc_test
> Running: drop table if exists emp
> Running: create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )
> row format delimited fields terminated by '\t'
> Running: show tables
> emp
> Running: desc emp
> empno   int
> ename   string
> job     string
> mgr     int
> hiredate       string
> sal     double
> comm   double
> deptno int
> Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' 
> overwrite into table emp
> Running: select * from emp
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: 
> Error while compiling statement: FAILED: SemanticException Invalid memory 
> usage value 1.0 for hive.limit.pushdown.memory.usage
>       at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
>       at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
>       at 
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
>       at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
>       at 
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
>       at demo.utils.JDBCUtils.selectD