[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121297#comment-16121297
]
Paul Praet edited comment on SPARK-21402 at 8/10/17 9:03 AM:
-------------------------------------------------------------
I can confirm this problem persists in Spark 2.2.0: fields get all swapped when
you use the bean encoder on a dataset with an array of structs. A plain struct
works, an array of structs does not. Pretty big issue if you ask me.
{noformat}
root
|-- writeKey: string (nullable = false)
|-- id: string (nullable = false)
|-- type: string (nullable = false)
|-- ssid: string (nullable = false)
+------------+------+--------+--------+
|writeKey |id |type |ssid |
+------------+------+--------+--------+
|someWriteKey|someId|someType|someSSID|
+------------+------+--------+--------+
{noformat}
When I convert into a struct, everything is still fine:
{noformat}
root
|-- writeKey: string (nullable = false)
|-- nodes: struct (nullable = false)
| |-- id: string (nullable = false)
| |-- type: string (nullable = false)
| |-- ssid: string (nullable = false)
+------------+--------------------------+
|writeKey |nodes |
+------------+--------------------------+
|someWriteKey|[someId,someType,someSSID]|
+------------+--------------------------+
{noformat}
When I do a groupBy on writeKey and a collect_set() on the nodes, we get:
{noformat}
root
|-- writeKey: string (nullable = false)
|-- nodes: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- id: string (nullable = false)
| | |-- type: string (nullable = false)
| | |-- ssid: string (nullable = false)
+------------+----------------------------+
|writeKey |nodes |
+------------+----------------------------+
|someWriteKey|[[someId,someType,someSSID]]|
+------------+----------------------------+
{noformat}
When I convert this to Java...
{code:java}
Dataset<Row> dfArray = dfStruct.groupBy("writeKey")
.agg(functions.collect_set("nodes").alias("nodes"));
Encoder<Topology> topologyEncoder = Encoders.bean(Topology.class);
Dataset<Topology> datasetMultiple = dfArray.as(topologyEncoder);
System.out.println(datasetMultiple.first());
{code}
This prints:
{noformat}
Topology{writeKey='someWriteKey', nodes=[Node{id='someId', type='someSSID',
ssid='someType'}]}
{noformat}
You can clearly see the type and ssid fields were swapped.
POJO classes:
{code:java}
public static class Topology {
private String writeKey;
private List<Node> nodes;
public Topology() {
}
public String getWriteKey() {
return writeKey;
}
public void setWriteKey(String writeKey) {
this.writeKey = writeKey;
}
public List<Node> getNodes() {
return nodes;
}
public void setNodes(List<Node> nodes) {
this.nodes = nodes;
}
@Override
public String toString() {
return "Topology{" +
"writeKey='" + writeKey + '\'' +
", nodes=" + nodes +
'}';
}
}
public static class Node {
private String id;
private String type;
private String ssid;
public Node() {
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getSsid() {
return ssid;
}
public void setSsid(String ssid) {
this.ssid = ssid;
}
@Override
public String toString() {
return "Node{" +
"id='" + id + '\'' +
", type='" + type + '\'' +
", ssid='" + ssid + '\'' +
'}';
}
}
{code}
was (Author: praetp):
I can confirm this problem persists in Spark 2.2.0: fields get all swapped when
you use the bean encoder on a dataset with an array of structs. A plain struct
works, an array of structs does not. Pretty big issue if you ask me.
I have a datamodel like this (all Strings)
{noformat}
+------------+------+--------+--------+
|writeKey |id |type |ssid |
+------------+------+--------+--------+
|someWriteKey|someId|someType|someSSID|
+------------+------+--------+--------+
{noformat}
When I convert into a struct, everything is still fine:
{noformat}
root
|-- writeKey: string (nullable = false)
|-- nodes: struct (nullable = false)
| |-- id: string (nullable = false)
| |-- type: string (nullable = false)
| |-- ssid: string (nullable = false)
+------------+--------------------------+
|writeKey |nodes |
+------------+--------------------------+
|someWriteKey|[someId,someType,someSSID]|
+------------+--------------------------+
{noformat}
When I do a groupBy on writeKey and a collect_set() on the nodes, we get:
{noformat}
root
|-- writeKey: string (nullable = false)
|-- nodes: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- id: string (nullable = false)
| | |-- type: string (nullable = false)
| | |-- ssid: string (nullable = false)
+------------+----------------------------+
|writeKey |nodes |
+------------+----------------------------+
|someWriteKey|[[someId,someType,someSSID]]|
+------------+----------------------------+
{noformat}
When I convert this to Java...
{code:java}
Dataset<Row> dfArray = dfStruct.groupBy("writeKey")
.agg(functions.collect_set("nodes").alias("nodes"));
Encoder<Topology> topologyEncoder = Encoders.bean(Topology.class);
Dataset<Topology> datasetMultiple = dfArray.as(topologyEncoder);
System.out.println(datasetMultiple.first());
{code}
This prints:
{noformat}
Topology{writeKey='someWriteKey', nodes=[Node{id='someId', type='someSSID',
ssid='someType'}]}
{noformat}
You can clearly see the type and ssid fields were swapped.
POJO classes:
{code:java}
public static class Topology {
private String writeKey;
private List<Node> nodes;
public Topology() {
}
public String getWriteKey() {
return writeKey;
}
public void setWriteKey(String writeKey) {
this.writeKey = writeKey;
}
public List<Node> getNodes() {
return nodes;
}
public void setNodes(List<Node> nodes) {
this.nodes = nodes;
}
@Override
public String toString() {
return "Topology{" +
"writeKey='" + writeKey + '\'' +
", nodes=" + nodes +
'}';
}
}
public static class Node {
private String id;
private String type;
private String ssid;
public Node() {
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getSsid() {
return ssid;
}
public void setSsid(String ssid) {
this.ssid = ssid;
}
@Override
public String toString() {
return "Node{" +
"id='" + id + '\'' +
", type='" + type + '\'' +
", ssid='" + ssid + '\'' +
'}';
}
}
{code}
> Java encoders - switch fields on collectAsList
> ----------------------------------------------
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
> Reporter: Tom
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> | |-- key: string
> | |-- value: struct (valueContainsNull = true)
> | | |-- startTime: long (nullable = true)
> | | |-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map<String, MyDTO> data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder<MyClass> myClassEncoder = Encoders.bean(MyClass.class);
> Dataset<MyClass> results = raw_df.as(myClassEncoder);
> List<MyClass> lst = results.collectAsList();
> {code}
>
> I do several calculations to get the result I want and the result is correct
> all through the way before I collect it.
> This is the result for -
> {code:java}
> results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
> {code}
> |data[2017-07-01].startTime|data[2017-07-01].endTime|
> +-----------------------------+--------------+
> |1498854000 |1498870800 |
> This is the result after collecting the reuslts for -
> {code:java}
> MyClass userData = results.collectAsList().get(0);
> MyDTO userDTO = userData.getData().get("2017-07-01");
> System.out.println("userDTO startTime: " + userDTO.getStartTime());
> System.out.println("userDTO endTime: " + userDTO.getEndTime());
> {code}
> --
> data startTime: 1498870800
> data endTime: 1498854000
> I tend to believe it is a spark issue. Would love any suggestions on how to
> bypass it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
