[
https://issues.apache.org/jira/browse/SPARK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519191#comment-16519191
]
Ruben Berenguel commented on SPARK-24347:
-----------------------------------------
[~holdenkarau] [~ueshin] (I ping you since you have reviewed past PRs from me
and I know you have worked around PySpark and "Scala" Spark, sorry if I should
have addressed somebody else) this "error" is somewhat puzzling.
In the Scala side this will also fail with a DataFrame in general, since
`col(name)` or `$"name"` will create the column with an Expression, not with a
NamedExpression. Since `Column` doesn't know it has metadata unless it is
created as a NamedExpression, it gets lost on aliasing methods (since plain
Expression contains no medatada).
This makes scaladoc and pydoc of alias, as and name confusing I think.
If this has to be fixed there is no easy workaround, to get the NamedExpression
properly we need to go to the DataFrame:
{code:java}
val df = spark.sqlContext.createDataFrame(List(("Foo", 1))).toDF("name",
"number")
val metadata = org.apache.spark.sql.types.Metadata.fromJson("""{"foo":
"bar"}""")
val withMeta = df.select($"name".as("name2", metadata))
// Column with metadata, since it comes from the dataframe as a NamedExpression
(this is what is done in Dataset.resolve, which is private). If we use as,
alias or name on it, metadata is preserved
new Column(withMeta.queryExecution.analyzed.resolveQuoted("name2",
spark.sessionState.analyzer.resolver).get
// Column with Expression, has no metadata and using as, alias or name on it
will clean metadata, even if in its dataframe "name2" had metadata.
col(withMeta("name2"))
{code}
> df.alias() in python API should not clear metadata by default
> -------------------------------------------------------------
>
> Key: SPARK-24347
> URL: https://issues.apache.org/jira/browse/SPARK-24347
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.3.0
> Reporter: Tomasz Bartczak
> Priority: Minor
>
> currently when doing an alias on a column in pyspark I lose metadata:
> {code:java}
> print("just select = ", df.select(col("v")).schema.fields[0].metadata.keys())
> print("select alias= ",
> df.select(col("v").alias("vv")).schema.fields[0].metadata.keys()){code}
> gives:
> {code:java}
> just select = dict_keys(['ml_attr'])
> select alias= dict_keys([]){code}
> After looking at alias() documentation I see that metadata is an optional
> param. But it should not clear the metadata when it is not set. A default
> solution should be to keep it as-is.
> Otherwise - it generates problems in a later part of the processing pipeline
> when someone is depending on the metadata.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
