Hi, Have you tried using spark dataframe's Pivot feature ?
-----Original Message----- From: nookala [mailto:srinook...@gmail.com] Sent: Thursday, July 26, 2018 7:33 AM To: user@spark.apache.org Subject: Split a row into multiple rows Java I'm trying to generate multiple rows from a single row I have schema Name Id Date 0100 0200 0300 0400 and would like to make it into a vertical format with schema Name Id Date Time I have the code below and get the error Caused by: java.lang.RuntimeException: org.apache.spark.sql.catalyst.expressions.GenericRow is not a valid external type for schema of string at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) StructType schemata = DataTypes.createStructType( new StructField[]{ DataTypes.createStructField("Name", DataTypes.StringType, false), DataTypes.createStructField("Id", DataTypes.StringType, false), DataTypes.createStructField("Date", DataTypes.StringType, false), DataTypes.createStructField("Time", DataTypes.StringType, false) } ); ExpressionEncoder<Row> encoder = RowEncoder.apply(schemata); Dataset<Row> modifiedRDD = intervalDF.flatMap(new FlatMapFunction<Row,Row>() { @Override public Iterator<Row> call (Row row) throws Exception { List<Row> rowList = new ArrayList<Row>(); String[] timeList = {"0100", "0200", "0300", "0400"} for (String time : timeList) { Row r1 = RowFactory.create(row.<String>getAs("sdp_id"), "WGL", row.<String>getAs("Name"), row.<String>getAs("Id"), row.<String>getAs("Date"), timeList[0], row.<String>getAs(timeList[0])); //updated row by creating new Row rowList.add(RowFactory.create(r1)); } return rowList.iterator(); } }, encoder); modifiedRDD.write().csv("file:///Users/mod/out"); -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org ________________________________ The information contained in this message is intended only for the recipient, and may be a confidential attorney-client communication or may otherwise be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, please be aware that any dissemination or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by replying to the message and deleting it from your computer. S&P Global Inc. reserves the right, subject to applicable local law, to monitor, review and process the content of any electronic message or information sent to or from S&P Global Inc. e-mail addresses without informing the sender or recipient of the message. By sending electronic message or information to S&P Global Inc. e-mail addresses you, as the sender, are consenting to S&P Global Inc. processing any of your personal data therein. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
