Tomasz Bartczak created SPARK-23479:
---------------------------------------
Summary: struct() cannot be combined with alias(metadata={})
Key: SPARK-23479
URL: https://issues.apache.org/jira/browse/SPARK-23479
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.1
Reporter: Tomasz Bartczak
when creating a struct with 'struct()' function - metadata added with alias()
is lost:
{code:java}
df =spark.createDataFrame([{'a':1}])
df.select(struct(col('a').alias('a',metadata={'description':'xxx'}))).schema.fields[0].dataType.fields[0].metadata{code}
gives:
{code:java}
{}{code}
workaround is to create the column with metadata before adding it to the
struct, but this is obviously bad behaviour:
{code:java}
df =spark.createDataFrame([{'a':1}])
df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct('a')).schema.fields[0].dataType.fields[0].metadata{code}
keeps metadata:
{code:java}
{'description': 'xxx'}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
