unsubscribe

2016-04-27 Thread Harjit Singh 








signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: JavaSparkContext.wholeTextFiles read directory

2016-04-26 Thread Harjit Singh 
You will have to write your customReceiver to do that. I don’t think 
wholeTextFile is designed for that.

- Harjit
> On Apr 26, 2016, at 7:19 PM, Mail.com <pradeep.mi...@mail.com> wrote:
> 
> 
> Hi All,
> I am reading entire directory of gz XML files with wholeTextFiles.
> 
> I understand as it is gz and with wholeTextFiles the individual files are not 
> splittable but why the entire directory is read by one executor, single task. 
> I have provided number of executors as number of files in that directory.
> 
> Is the only option here is to repartition after the xmls are read and parsed 
> with JaxB.
> 
> Regards,
> Pradeep
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
> 

v/r,
Harjit Singh
Decipher Technology Studios
email:harjit.sin...@deciphernow.com 
<mailto:email%3ajon.hol...@deciphertechstudios.com>
mobile: 303- 870-0883
website: deciphertechstudios.com <http://deciphertechstudios.com/> 
<http://deciphertechstudios.com/ <http://deciphertechstudios.com/>>

GPG:
keyserver: hkps://hkps.pool.sks-keyservers.net 
<http://hkps.pool.sks-keyservers.net/>
keyid: D814A2EF







signature.asc
Description: Message signed with OpenPGP using GPGMail

Last RDD always being Run

2016-04-26 Thread Harjit Singh 
i'm running the LogAnalyzerStreaming Example. Its processing the files fine but 
it keeps emitting the output of last processed RDD until it gets a new one. Is 
there a way to prevent that. I'm planning to use this example in a real 
scenario where when I have processed the data, I would be pushing it to a 
Database. So if the RDD keeps emitting the same values, I would be persisting 
the same values again and again, till I don't get new one. Any ideas how to do 
it? I have tried using unPersist() on the RDD, but it doesn't help.

Thanks

Harjit






signature.asc
Description: Message signed with OpenPGP using GPGMail

test

2016-04-26 Thread Harjit Singh 








signature.asc
Description: Message signed with OpenPGP using GPGMail