I usually check the list of Hive UDFs as Spark has implemented almost all of them https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions
Or/and check `org.apache.spark.sql.functions` directly: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/sql/functions.html Also you can check the list of all Datetime functions here https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L368-L399 and what they do here https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala On Tue, Oct 3, 2017 at 1:43 PM, Adaryl Wakefield < adaryl.wakefi...@hotmail.com> wrote: > I gave myself a project to start actually writing Spark programs. I’m > using Scala and Spark 2.2.0. In my project, I had to do some grouping and > filtering by dates. It was awful and took forever. I was trying to use > dataframes and SQL as much as possible. I see that there are date functions > in the dataframe API but trying to use them was frustrating. Even following > code samples was a headache because apparently the code is different > depending on which version of Spark you are using. I was really hoping for > a rich set of date functions like you’d find in T-SQL but I never really > found them. > > > > Is there a best practice for dealing with dates and time in Spark? I feel > like taking a date/time string and converting it to a date/time object and > then manipulating data based on the various components of the timestamp > object (hour, day, year etc.) should be a heck of a lot easier than what > I’m finding and perhaps I’m just not looking in the right place. > > > > You can see my work here: https://github.com/BobLovesData/Apache-Spark-In- > 24-Hours/blob/master/src/net/massstreet/hour10/BayAreaBikeAnalysis.scala > > > > Adaryl "Bob" Wakefield, MBA > Principal > Mass Street Analytics, LLC > 913.938.6685 <(913)%20938-6685> > > www.massstreet.net > > www.linkedin.com/in/bobwakefieldmba > Twitter: @BobLovesData <http://twitter.com/BobLovesData> > > > > >
