Thanks alot ,
Because i just want to log timestamp and unique message id and not full
RDD .
On Tue, Jun 23, 2015 at 12:41 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Why don't you do a normal .saveAsTextFiles?
>
> Thanks
> Best Regards
>
> On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <anshushuk...@gmail.com>
> wrote:
>
>> Thanx for reply !!
>>
>> YES , Either it should write on any machine of cluster or Can you
>> please help me ... that how to do this . Previously i was using writing
>> using collect () , so some of my tuples are missing while writing.
>>
>> //previous logic that was just creating the file on master -
>>
>> newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
>>
>> @Override
>> public Void call(JavaRDD<String> v1, Time v2) throws Exception {
>> for(String s:v1.collect()) {
>> // System.out.println("v1 here is " + v1 + "-------" + s);
>> spoutlog.batchLogwriter(System.currentTimeMillis(),
>> "spout-MSGID," + msgeditor.getMessageId(s));
>> // System.out.println(msgeditor.getMessageId(s));
>> }
>> return null;
>> }
>> });
>>
>>
>>
>>
>> On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <
>> rmarsc...@localytics.com> wrote:
>>
>>> Is spoutLog just a non-spark file writer? If you run that in the map
>>> call on a cluster its going to be writing in the filesystem of the executor
>>> its being run on. I'm not sure if that's what you intended.
>>>
>>> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <anshushuk...@gmail.com>
>>> wrote:
>>>
>>>> Running perfectly in local system but not writing to file in cluster mode
>>>> .ANY suggestions please ..
>>>>
>>>>
>>>> //msgid is long counter
>>>>
>>>> JavaDStream<String> newinputStream=inputStream.map(new Function<String,
>>>> String>() {
>>>> @Override
>>>> public String call(String v1) throws Exception {
>>>> String s1=msgId+"@"+v1;
>>>> System.out.println(s1);
>>>> msgId++;
>>>> try {
>>>> *//filewriter logic
>>>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," +
>>>> msgeditor.getMessageId(s1));*
>>>> } catch (Exception e) {
>>>>
>>>> System.out.println("exeception is here");
>>>> e.printStackTrace();
>>>> throw e;
>>>> }
>>>> System.out.println("msgid,"+msgId);
>>>> return msgeditor.addMessageId(v1,msgId);
>>>> }
>>>> });
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anshu Shukla
>>>>
>>>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <anshushuk...@gmail.com>
>>>> wrote:
>>>>
>>>>> Can not we write some data to a txt file in parallel with multiple
>>>>> executors running in parallel ??
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anshu Shukla
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anshu Shukla
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
--
Thanks & Regards,
Anshu Shukla
