Using Hadoop Custom Input format in Spark

2015-10-27 Thread Balachandar R.A.
Hello, I have developed a hadoop based solution that process a binary file. This uses classic hadoop MR technique. The binary file is about 10GB and divided into 73 HDFS blocks, and the business logic written as map process operates on each of these 73 blocks. We have developed a

Re: Using Hadoop Custom Input format in Spark

2015-10-27 Thread ayan guha
Mind sharing the error you are getting? On 28 Oct 2015 03:53, "Balachandar R.A." wrote: > Hello, > > > I have developed a hadoop based solution that process a binary file. This > uses classic hadoop MR technique. The binary file is about 10GB and divided > into 73 HDFS

Re: Using Hadoop Custom Input format in Spark

2015-10-27 Thread Sabarish Sasidharan
Did you try the sc.binaryFiles() which gives you an RDD of PortableDataStream that wraps around the underlying bytes. On Tue, Oct 27, 2015 at 10:23 PM, Balachandar R.A. wrote: > Hello, > > > I have developed a hadoop based solution that process a binary file. This >