Hello! I am running Spark on Java and bumped into a problem I can't solve or find anything helpful among answered questions, so I would really appreciate your help.
I am running some calculations, creating rows for each result: List<Row> results = new LinkedList<Row>(); for(something){ results.add(RowFactory.create( someStringVariable, someIntegerVariable )); } Now I ended up with a list of rows I need to turn into dataframe to perform some spark sql operations on them, like groupings and sorting. Would like to keep the dataTypes. I tried: Dataset<Row> toShow = spark.createDataFrame(results, Row.class); but it throws nullpointer. (spark being SparkSession) Is my logic wrong there somewhere, should this operation be possible, resulting in what I want? Or do I have to create a custom class which extends serializable and create a list of those objects rather than Rows? Will I be able to perform SQL queries on dataset consisting of custom class objects rather than rows? I'm sorry if this is a duplicate question. Thank you for your help! Karin