Is spoutLog just a non-spark file writer? If you run that in the map call on a cluster its going to be writing in the filesystem of the executor its being run on. I'm not sure if that's what you intended.
On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <anshushuk...@gmail.com> wrote: > Running perfectly in local system but not writing to file in cluster mode > .ANY suggestions please .. > > > //msgid is long counter > > JavaDStream<String> newinputStream=inputStream.map(new Function<String, > String>() { > @Override > public String call(String v1) throws Exception { > String s1=msgId+"@"+v1; > System.out.println(s1); > msgId++; > try { > *//filewriter logic > spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + > msgeditor.getMessageId(s1));* > } catch (Exception e) { > > System.out.println("exeception is here"); > e.printStackTrace(); > throw e; > } > System.out.println("msgid,"+msgId); > return msgeditor.addMessageId(v1,msgId); > } > }); > > > -- > Thanks & Regards, > Anshu Shukla > > On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <anshushuk...@gmail.com> > wrote: > >> Can not we write some data to a txt file in parallel with multiple >> executors running in parallel ?? >> >> >> -- >> Thanks & Regards, >> Anshu Shukla >> > > > > -- > Thanks & Regards, > Anshu Shukla >