Re: DataFrame support for hadoop glob patterns

i tried with avro input, something like /data/txn_*/* and it works for me On Wed, Mar 9, 2016 at 12:12 PM, Ted Yu wrote: > Koert: > I meant org.apache.hadoop.mapred.FileInputFormat doesn't support multi > level wildcard. > > Cheers > > On Wed, Mar 9, 2016 at 8:22 AM, Koert

Re: DataFrame support for hadoop glob patterns

Koert: I meant org.apache.hadoop.mapred.FileInputFormat doesn't support multi level wildcard. Cheers On Wed, Mar 9, 2016 at 8:22 AM, Koert Kuipers wrote: > i use multi level wildcard with hadoop fs -ls, which is the exact same > glob function call > > On Wed, Mar 9, 2016 at

Re: DataFrame support for hadoop glob patterns

i use multi level wildcard with hadoop fs -ls, which is the exact same glob function call On Wed, Mar 9, 2016 at 9:24 AM, Ted Yu wrote: > Hadoop glob pattern doesn't support multi level wildcard. > > Thanks > > On Mar 9, 2016, at 6:15 AM, Koert Kuipers

Re: DataFrame support for hadoop glob patterns

Hi, Unless I've misunderstood what you want to achieve, you could use: sqlContext.read.json(sc.textFile("/mnt/views-p/base/2016/01/*/*-xyz.json")) Regards, Christophe. On 09/03/16 15:24, Ted Yu wrote: Hadoop glob pattern doesn't support multi level wildcard. Thanks On Mar 9, 2016, at 6:15 AM,

Re: DataFrame support for hadoop glob patterns

Hadoop glob pattern doesn't support multi level wildcard. Thanks > On Mar 9, 2016, at 6:15 AM, Koert Kuipers wrote: > > if its based on HadoopFsRelation shouldn't it support it? HadoopFsRelation > handles globs > >> On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu

Re: DataFrame support for hadoop glob patterns

if its based on HadoopFsRelation shouldn't it support it? HadoopFsRelation handles globs On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu wrote: > This is currently not supported. > > On Mar 9, 2016, at 4:38 AM, Jakub Liska wrote: > > Hey, > > is something

Re: DataFrame support for hadoop glob patterns

This is currently not supported. > On Mar 9, 2016, at 4:38 AM, Jakub Liska wrote: > > Hey, > > is something like this possible? > > sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json") > > I switched to DataFrames because my source files changed from TSV to

DataFrame support for hadoop glob patterns

Hey, is something like this possible? sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json") I switched to DataFrames because my source files changed from TSV to JSON but now I'm not able to load the files as I did before. I get this error if I try that :