Shrikant Prasad created SPARK-42655:
---------------------------------------
Summary: Incorrect ambiguous column reference error
Key: SPARK-42655
URL: https://issues.apache.org/jira/browse/SPARK-42655
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.2.0
Reporter: Shrikant Prasad
val df1 =
sc.parallelize(List((1,2,3,4,5),(1,2,3,4,5))).toDF("id","col2","col3","col4",
"col5")
val op_cols_same_case = List("id","col2","col3","col4", "col5", "id")
val df2 = df1.select(op_cols_same_case .head, op_cols_same_case .tail: _*)
df2.select("id").show()
This query runs fine.
But when we change the casing of the op_cols to have mix of upper & lower case
("id" & "ID") it throws an ambiguous col ref error:
val df1 =
sc.parallelize(List((1,2,3,4,5),(1,2,3,4,5))).toDF("id","col2","col3","col4",
"col5")
val op_cols_same_case = List("id","col2","col3","col4", "col5", "ID")
val df2 = df1.select(op_cols_same_case .head, op_cols_same_case .tail: _*)
df2.select("id").show()
Since, Spark is case insensitive, it should work for second case also when we
have upper and lower case column names in the column list.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
