Re: Help Required on Spark - Convert DataFrame to List with out using collect

[Deepak Sharma]

I am not sure about java but in scala it would be something like
df.rdd.map{ x => MyClass(x.getString(0),.....)}

HTH

--Deepak

On Dec 19, 2017 09:25, "Sunitha Chennareddy" <chennareddysuni...@gmail.com>
wrote:

Hi All,

I am new to Spark, I want to convert DataFrame to List<JavaClass> with out
using collect().

Main requirement is I need to iterate through the rows of dataframe and
call another function by passing column value of each row (person.getId())

Here is the snippet I have tried, Kindly help me to resolve the issue,
personLst is returning 0:

List<Person> personLst= new ArrayList<Person>();
JavaRDD<Person> personRDD = person_dataframe.toJavaRDD().map(new
Function<Row, Person>() {
  public Person call(Row row)  throws Exception{
  Person person = new Person();
  person.setId(row.getDecimal(0).longValue());
  person.setName(row.getString(1));

personLst.add(person);
// here I tried to call another function but control never passed
    return person;
  }
});
logger.info("personLst size =="+personLst.size());
logger.info("personRDD count ==="+personRDD.count());

//output is
personLst size == 0
personRDD count === 3