Or you can just call describe() on the dataframe? In addition to min-max, you'll also get the mean, and count of non-null and non-NA elements as well.
Burak On Fri, Aug 28, 2015 at 10:09 AM, java8964 <java8...@hotmail.com> wrote: > Or RDD.max() and RDD.min() won't work for you? > > Yong > > ------------------------------ > Subject: Re: Calculating Min and Max Values using Spark Transformations? > To: as...@wso2.com > CC: user@spark.apache.org > From: jfc...@us.ibm.com > Date: Fri, 28 Aug 2015 09:28:43 -0700 > > > If you already loaded csv data into a dataframe, why not register it as a > table, and use Spark SQL > to find max/min or any other aggregates? SELECT MAX(column_name) FROM > dftable_name ... seems natural. > > > > > > *JESSE CHEN* > Big Data Performance | IBM Analytics > > Office: 408 463 2296 > Mobile: 408 828 9068 > Email: jfc...@us.ibm.com > > > > [image: Inactive hide details for ashensw ---08/28/2015 05:40:07 AM---Hi > all, I have a dataset which consist of large number of feature]ashensw > ---08/28/2015 05:40:07 AM---Hi all, I have a dataset which consist of large > number of features(columns). It is > > From: ashensw <as...@wso2.com> > To: user@spark.apache.org > Date: 08/28/2015 05:40 AM > Subject: Calculating Min and Max Values using Spark Transformations? > > ------------------------------ > > > > Hi all, > > I have a dataset which consist of large number of features(columns). It is > in csv format. So I loaded it into a spark dataframe. Then I converted it > into a JavaRDD<Row> Then using a spark transformation I converted that into > JavaRDD<String[]>. Then again converted it into a JavaRDD<double[]>. So now > I have a JavaRDD<double[]>. So is there any method to calculate max and min > values of each columns in this JavaRDD<double[]> ? > > Or Is there any way to access the array if I store max and min values to a > array inside the spark transformation class? > > Thanks. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Calculating-Min-and-Max-Values-using-Spark-Transformations-tp24491.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > >
