Hi Sonia
I believe you are using java? Take a look at Java Date I am sure you will
find lots of examples of how to format dates
Enjoy share
Andy
/**
* saves tweets to disk. This a replacement for
* @param tweets
* @param outputURI
*/
private static void saveTweets(JavaDStream<String> jsonTweets, String
outputURI) {
/*
using saveAsTestFiles will cause lots of empty directories to be
created.
DStream<String> data = jsonTweets.dstream();
data.saveAsTextFiles(outputURI, null);
*/
jsonTweets.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
private static final long serialVersionUID =
-5482893563183573691L;
@Override
public Void call(JavaRDD<String> rdd, Time time) throws
Exception {
if(!rdd.isEmpty()) {
String dirPath = outputURI + "-" + time.milliseconds();
rdd.saveAsTextFile(dirPath);
}
return null;
}
});
From: Soni spark <[email protected]>
Date: Tuesday, December 8, 2015 at 6:26 AM
To: Andrew Davidson <[email protected]>
Subject: epoch date format to normal date format while loading the files to
HDFS
> Hi Andy,
>
> How are you? i need your help again.
>
> I have written a spark streaming program in Java to access twitter tweets and
> it is working fine. I can able to copy the twitter feeds to HDFS location by
> batch wise.For each batch, it is creating a folder with epoch time stamp. for
> example,
>
> If i give HDFS location as hdfs://localhost:54310/twitter/, the files are
> creating like below
>
> /spark/twitter/-1449580800000/
> /spark/twitter/-1449579840000/
>
> I want to create a folder name like yyyy-MM-dd-HH format instead of by default
> epoch format.
>
> I want it like below so that i can do hive partitions easily to access the
> data.
>
> /spark/twitter/2015-12-08-01/
>
>
> Can you help me. Thank you so much in advance.
>
>
> Thanks
> Soniya
