Re: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation ?

2015-11-10 Thread Sasaki Kai
Did you indicate CsvRelation in spark-csv package? LibSVMRelation is included in spark core package, but CsvRelation(spark-csv) is not. Is it necessary for us to modify also spark-csv as you proposed in SPARK-11622? Regards Kai > On Nov 5, 2015, at 11:30 AM, Jeff Zhang

Re: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation ?

2015-11-10 Thread Jeff Zhang
Yes Kai, I also to plan to do for CsvRelation, will create PR for spark-csv On Wed, Nov 11, 2015 at 9:10 AM, Sasaki Kai wrote: > Did you indicate CsvRelation in spark-csv package? LibSVMRelation is > included in spark core package, but CsvRelation(spark-csv) is not. > Is it

Re: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation ?

2015-11-10 Thread Sasaki Kai
Great, thank you! > On Nov 11, 2015, at 11:41 AM, Jeff Zhang wrote: > > Yes Kai, I also to plan to do for CsvRelation, will create PR for spark-csv > > On Wed, Nov 11, 2015 at 9:10 AM, Sasaki Kai > wrote: > Did you indicate

RE: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation ?

2015-11-04 Thread Cheng, Hao
Probably 2 reasons: 1. HadoopFsRelation was introduced since 1.4, but seems CsvRelation was created based on 1.3 2. HadoopFsRelation introduces the concept of Partition, which probably not necessary for LibSVMRelation. But I think it will be easy to change as extending from

Re: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation ?

2015-11-04 Thread Jeff Zhang
Thanks Hao. I have ready made it extends HadoopFsRelation and it works. Will create a jira for that. Besides that, I noticed that in DataSourceStrategy, spark build physical plan based on the trait of the BaseRelation in pattern matching (e.g. CatalystScan, TableScan, HadoopFsRelation). That

RE: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation ?

2015-11-04 Thread Cheng, Hao
problem as you described, probably we can add additional checking / reporting rule for the abuse. From: Jeff Zhang [mailto:zjf...@gmail.com] Sent: Thursday, November 5, 2015 1:55 PM To: Cheng, Hao Cc: dev@spark.apache.org Subject: Re: Why LibSVMRelation and CsvRelation don't extends HadoopFsRelation