[jira] [Commented] (SPARK-32345) SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session
[ https://issues.apache.org/jira/browse/SPARK-32345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177468#comment-17177468 ] ZhouDaHong commented on SPARK-32345: If the cause of the version conflict is excluded. You can look at queue resources. If the queue resource reaches 100% and there is no free task resource released for creating spark session in a short time, the task will fail and this exception will be thrown. Solution: increase the connection time interval of hive client to 15 minutes; set hive.spark.client . server.connect.timeout=90 ; > SemanticException Failed to get a spark session: > org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark > client for Spark session > -- > > Key: SPARK-32345 > URL: https://issues.apache.org/jira/browse/SPARK-32345 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: 任建亭 >Priority: Blocker > > when using hive on spark engine: > FAILED: SemanticException Failed to get a spark session: > org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark > client for Spark session > hadoop version: 2.7.3 / hive version: 3.1.2 / spark version 3.0.0 > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-32587) SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing NULL values
[ https://issues.apache.org/jira/browse/SPARK-32587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177465#comment-17177465 ] ZhouDaHong edited comment on SPARK-32587 at 8/14/20, 3:54 AM: -- Sorry, I don't really understand your problem. Do you mean that data cannot be written to the SQL Server database when there is a null value column? If this is the case, please check the structure of the table to see if the error reporting field is defined as "not null" in the database? was (Author: zdh): Sorry, I don't really understand your problem. Do you mean that data cannot be written to the SQL Server database when there is a null value column? If this is the case, please check the structure of the table to see if the error reporting field is defined as "not null" in the database? > SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing > NULL values > - > > Key: SPARK-32587 > URL: https://issues.apache.org/jira/browse/SPARK-32587 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.5 >Reporter: Mohit Dave >Priority: Major > > While writing to a target in SQL Server using Microsoft's SQL Server driver > using dataframe.write API the target is storing NULL values for BIT columns. > > Table definition > Azure SQL DB > 1)Create 2 tables with column type as bit > 2)Insert some record into 1 table > Create a SPARK job > 1)Create a Dataframe using spark.read with the following query > select from > 2)Write the dataframe to a target table with bit type as column. > > Observation : Bit type is getting converted to NULL at the target > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-32587) SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing NULL values
[ https://issues.apache.org/jira/browse/SPARK-32587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177465#comment-17177465 ] ZhouDaHong edited comment on SPARK-32587 at 8/14/20, 3:54 AM: -- Sorry, I don't really understand your problem. Do you mean that data cannot be written to the SQL Server database when there is a null value column? If this is the case, please check the structure of the table to see if the error reporting field is defined as "not null" in the database? was (Author: zdh): 抱歉，我不是特别明白你的问题。你是不是说数据存在空值列的时候，无法写入到sql server数据库？如果是这样的话，请查看待写入的表的结构，查看报错字段是否在数据库中定义了“not null”？ > SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing > NULL values > - > > Key: SPARK-32587 > URL: https://issues.apache.org/jira/browse/SPARK-32587 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.5 >Reporter: Mohit Dave >Priority: Major > > While writing to a target in SQL Server using Microsoft's SQL Server driver > using dataframe.write API the target is storing NULL values for BIT columns. > > Table definition > Azure SQL DB > 1)Create 2 tables with column type as bit > 2)Insert some record into 1 table > Create a SPARK job > 1)Create a Dataframe using spark.read with the following query > select from > 2)Write the dataframe to a target table with bit type as column. > > Observation : Bit type is getting converted to NULL at the target > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32587) SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing NULL values
[ https://issues.apache.org/jira/browse/SPARK-32587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177465#comment-17177465 ] ZhouDaHong commented on SPARK-32587: 抱歉，我不是特别明白你的问题。你是不是说数据存在空值列的时候，无法写入到sql server数据库？如果是这样的话，请查看待写入的表的结构，查看报错字段是否在数据库中定义了“not null”？ > SPARK SQL writing to JDBC target with bit datatype using Dataframe is writing > NULL values > - > > Key: SPARK-32587 > URL: https://issues.apache.org/jira/browse/SPARK-32587 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.5 >Reporter: Mohit Dave >Priority: Major > > While writing to a target in SQL Server using Microsoft's SQL Server driver > using dataframe.write API the target is storing NULL values for BIT columns. > > Table definition > Azure SQL DB > 1)Create 2 tables with column type as bit > 2)Insert some record into 1 table > Create a SPARK job > 1)Create a Dataframe using spark.read with the following query > select from > 2)Write the dataframe to a target table with bit type as column. > > Observation : Bit type is getting converted to NULL at the target > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type
[
https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177461#comment-17177461
]
ZhouDaHong commented on SPARK-21774:
Hello, you compare the value of a field of string type with the 0 in your sql.
Due to the different data types, (the 0 may be judged as boolean type, or 0 as
int type).
Therefore, the SQL statement [ select a, B from TB where a = 0 ] cannot get the
result you expect.
It is suggested to change to [ select a, B from TB where a ='0' ]
> The rule PromoteStrings cast string to a wrong data type
>
>
> Key: SPARK-21774
> URL: https://issues.apache.org/jira/browse/SPARK-21774
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: StanZhai
>Priority: Critical
> Labels: correctness
>
> Data
> {code}
> create temporary view tb as select * from values
> ("0", 1),
> ("-0.1", 2),
> ("1", 3)
> as grouping(a, b)
> {code}
> SQL:
> {code}
> select a, b from tb where a=0
> {code}
> The result which is wrong:
> {code}
> ++---+
> | a| b|
> ++---+
> | 0| 1|
> |-0.1| 2|
> ++---+
> {code}
> Logical Plan:
> {code}
> == Parsed Logical Plan ==
> 'Project ['a]
> +- 'Filter ('a = 0)
>+- 'UnresolvedRelation `src`
> == Analyzed Logical Plan ==
> a: string
> Project [a#8528]
> +- Filter (cast(a#8528 as int) = 0)
>+- SubqueryAlias src
> +- Project [_1#8525 AS a#8528, _2#8526 AS b#8529]
> +- LocalRelation [_1#8525, _2#8526]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source
[
https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177455#comment-17177455
]
ZhouDaHong commented on SPARK-9182:
---
Hello, it seems that the problem is that the "Sal" field is of numerical type,
but in the actual SQL process, it is impossible to match the numeric value non
equivalently. Try changing the "Sal" field to int or double.
> filter and groupBy on DataFrames are not passed through to jdbc source
> --
>
> Key: SPARK-9182
> URL: https://issues.apache.org/jira/browse/SPARK-9182
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.4.1
>Reporter: Greg Rahn
>Assignee: Yijie Shen
>Priority: Critical
>
> When running all of these API calls, the only one that passes the filter
> through to the backend jdbc source is equality. All filters in these
> commands should be able to be passed through to the jdbc database source.
> {code}
> val url="jdbc:postgresql:grahn"
> val prop = new java.util.Properties
> val emp = sqlContext.read.jdbc(url, "emp", prop)
> emp.filter(emp("sal") === 5000).show()
> emp.filter(emp("sal") < 5000).show()
> emp.filter("sal = 3000").show()
> emp.filter("sal > 2500").show()
> emp.filter("sal >= 2500").show()
> emp.filter("sal < 2500").show()
> emp.filter("sal <= 2500").show()
> emp.filter("sal != 3000").show()
> emp.filter("sal between 3000 and 5000").show()
> emp.filter("ename in ('SCOTT','BLAKE')").show()
> {code}
> We see from the PostgreSQL query log the following is run, and see that only
> equality predicates are passed through.
> {code}
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE
> sal = 5000
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE
> sal = 3000
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute : SET extra_float_digits = 3
> LOG: execute : SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
