Re: Check if column exists in Schema
I'm trying to do it before the column is bound to a dataframe. I guess I'm looking for something like col("x").isNull but that would return whether a column exists or not On Mon, Feb 15, 2016 at 10:31 PM Mohammed Guller <moham...@glassbeam.com> wrote: > The DataFrame class has a method named columns, which returns all column > names as an array. You can then use the contains method in the Scala Array > class to check whether a column exists. > > > > Mohammed > > Author: Big Data Analytics with Spark > <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> > > > > *From:* Sebastian Piu [mailto:sebastian@gmail.com] > *Sent:* Monday, February 15, 2016 11:21 AM > *To:* user > *Subject:* Re: Check if column exists in Schema > > > > I just realised this is a bit vague, I'm looking to create a function that > looks into different columns to get a value. So depending on a type I might > look into a given path or another (which might or might not exist). > > > > Example if column *some.path.to.my.date *exists I'd return that, if it > doesn't or it is null, i'd get it from some other place > > > > On Mon, Feb 15, 2016 at 7:17 PM Sebastian Piu <sebastian@gmail.com> > wrote: > > Is there any way of checking if a given column exists in a Dataframe? > >
RE: Check if column exists in Schema
The DataFrame class has a method named columns, which returns all column names as an array. You can then use the contains method in the Scala Array class to check whether a column exists. Mohammed Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Sebastian Piu [mailto:sebastian@gmail.com] Sent: Monday, February 15, 2016 11:21 AM To: user Subject: Re: Check if column exists in Schema I just realised this is a bit vague, I'm looking to create a function that looks into different columns to get a value. So depending on a type I might look into a given path or another (which might or might not exist). Example if column some.path.to.my.date exists I'd return that, if it doesn't or it is null, i'd get it from some other place On Mon, Feb 15, 2016 at 7:17 PM Sebastian Piu <sebastian@gmail.com<mailto:sebastian@gmail.com>> wrote: Is there any way of checking if a given column exists in a Dataframe?
Re: Check if column exists in Schema
I just realised this is a bit vague, I'm looking to create a function that looks into different columns to get a value. So depending on a type I might look into a given path or another (which might or might not exist). Example if column *some.path.to.my.date *exists I'd return that, if it doesn't or it is null, i'd get it from some other place On Mon, Feb 15, 2016 at 7:17 PM Sebastian Piuwrote: > Is there any way of checking if a given column exists in a Dataframe? >
Check if column exists in Schema
Is there any way of checking if a given column exists in a Dataframe?