Hi Shuai,
It should certainly be possible to do it that way, but I would recommend
against it. If you look at HadoopRDD, its doing all sorts of little
book-keeping that you would most likely want to mimic. eg., tracking the
number of bytes records that are read, setting up all the hadoop
configuration, splits, readers, scheduling tasks for locality, etc. Thats
why I suggested that really you want to just create a small variant of
HadoopRDD.
hope that helps,
Imran
On Sat, Mar 14, 2015 at 11:10 AM, Shawn Zheng szheng.c...@gmail.com wrote:
Sorry for reply late.
But I just think of one solution: if I load all the file name itself (not
the contain of the file), so I have a RDD[key, iterable[filename]], then I
run mapPartitionsToPair on it with preservesPartitioning=true
Do you think it is a right solution? I am not sure whether it has
potential issue if I try to fake/enforce the partition in my own way.
Regards,
Shuai
On Wed, Mar 11, 2015 at 8:09 PM, Imran Rashid iras...@cloudera.com
wrote:
It should be *possible* to do what you want ... but if I understand you
right, there isn't really any very easy way to do it. I think you would
need to write your own subclass of RDD, which has its own logic on how the
input files get put divided among partitions. You can probably subclass
HadoopRDD and just modify getPartitions(). your logic could look at the
day of each filename to decide which partition it goes into. You'd need to
make corresponding changes for HadoopPartition the compute() method.
(or if you can't subclass HadoopRDD directly you can use it for
inspiration.)
On Mon, Mar 9, 2015 at 11:18 AM, Shuai Zheng szheng.c...@gmail.com
wrote:
Hi All,
If I have a set of time series data files, they are in parquet format
and the data for each day are store in naming convention, but I will not
know how many files for one day.
20150101a.parq
20150101b.parq
20150102a.parq
20150102b.parq
20150102c.parq
…
201501010a.parq
…
Now I try to write a program to process the data. And I want to make
sure each day’s data into one partition, of course I can load all into one
big RDD to do partition but it will be very slow. As I already know the
time series of the file name, is it possible for me to load the data into
the RDD also preserve the partition? I know I can preserve the partition by
each file, but is it anyway for me to load the RDD and preserve partition
based on a set of files: one partition multiple files?
I think it is possible because when I load a RDD from 100 files (assume
cross 100 days), I will have 100 partitions (if I disable file split when
load file). Then I can use a special coalesce to repartition the RDD? But I
don’t know is it possible to do that in current Spark now?
Regards,
Shuai
