Re: hive issue with sub-directories
One way it was solved by an user earlier was by subclassing the InputFormat class and overriding the listStatus method so that you can ignore subdirectories. This was done in 0.7.1 version. Not sure if there is any better way in later versions. At least you can use this approach until someone comes up with a better way. Please see the message at http://mail-archives.apache.org/mod_mbox/hive-user/201108.mbox/%3CCAC80dVSwRz3GpTSAzpS1eQw_P79ko+-=jmQVkOE2VMvMam=0...@mail.gmail.com%3E -Ramki. On Sun, Mar 10, 2013 at 11:36 PM, wrote: > ** > Hi Suresh > > AFAIK as of now a partition cannot contain sub directories, it can contain > only files. > > You may have to move the sub dirs out of the parent dir 'a' and create > separate partitions for those. > Regards > Bejoy KS > > Sent from remote device, Please excuse typos > -- > *From: * Suresh Krishnappa > *Date: *Mon, 11 Mar 2013 10:58:05 +0530 > *To: * > *ReplyTo: * user@hive.apache.org > *Subject: *Re: hive issue with sub-directories > > Hi Mark, > I am using external table in HIVE. > > This is how I am adding the partition > > > alter table add partition (pt=1) location '/test/a/'; > > I am able to run HIVE queries only if '/test/a/b' folder is deleted. > > How can I retain this folder structure and still issue queries? > > Thanks > Suresh > > On Sun, Mar 10, 2013 at 12:48 AM, Mark Grover > wrote: > >> Suresh, >> By default, the partition column name has to be appear in HDFS >> directory structure. >> >> e.g. >> /user/hive/warehouse//=> value>/data1.txt >> /user/hive/warehouse//=> value>/data2.txt >> >> >> On Thu, Mar 7, 2013 at 7:20 AM, Suresh Krishnappa >> wrote: >> > Hi All, >> > I have the following directory structure in hdfs >> > >> > /test/a/ >> > /test/a/1.avro >> > /test/a/2.avro >> > /test/a/b/ >> > /test/a/b/3.avro >> > >> > I created an external HIVE table using Avro Serde and added /test/a as a >> > partition to this table. >> > >> > I am not able to run a select query. Always getting the error 'not a >> file' >> > on '/test/a/b' >> > >> > Is this by design, a bug or am I missing some configuration? >> > I am using HIVE 0.10 >> > >> > Thanks >> > Suresh >> > >> > >
Re: hive issue with sub-directories
Hi Suresh AFAIK as of now a partition cannot contain sub directories, it can contain only files. You may have to move the sub dirs out of the parent dir 'a' and create separate partitions for those. Regards Bejoy KS Sent from remote device, Please excuse typos -Original Message- From: Suresh Krishnappa Date: Mon, 11 Mar 2013 10:58:05 To: Reply-To: user@hive.apache.org Subject: Re: hive issue with sub-directories Hi Mark, I am using external table in HIVE. This is how I am adding the partition > alter table add partition (pt=1) location '/test/a/'; I am able to run HIVE queries only if '/test/a/b' folder is deleted. How can I retain this folder structure and still issue queries? Thanks Suresh On Sun, Mar 10, 2013 at 12:48 AM, Mark Grover wrote: > Suresh, > By default, the partition column name has to be appear in HDFS > directory structure. > > e.g. > /user/hive/warehouse//= value>/data1.txt > /user/hive/warehouse//= value>/data2.txt > > > On Thu, Mar 7, 2013 at 7:20 AM, Suresh Krishnappa > wrote: > > Hi All, > > I have the following directory structure in hdfs > > > > /test/a/ > > /test/a/1.avro > > /test/a/2.avro > > /test/a/b/ > > /test/a/b/3.avro > > > > I created an external HIVE table using Avro Serde and added /test/a as a > > partition to this table. > > > > I am not able to run a select query. Always getting the error 'not a > file' > > on '/test/a/b' > > > > Is this by design, a bug or am I missing some configuration? > > I am using HIVE 0.10 > > > > Thanks > > Suresh > > >
Re: hive issue with sub-directories
Hi Mark, I am using external table in HIVE. This is how I am adding the partition > alter table add partition (pt=1) location '/test/a/'; I am able to run HIVE queries only if '/test/a/b' folder is deleted. How can I retain this folder structure and still issue queries? Thanks Suresh On Sun, Mar 10, 2013 at 12:48 AM, Mark Grover wrote: > Suresh, > By default, the partition column name has to be appear in HDFS > directory structure. > > e.g. > /user/hive/warehouse//= value>/data1.txt > /user/hive/warehouse//= value>/data2.txt > > > On Thu, Mar 7, 2013 at 7:20 AM, Suresh Krishnappa > wrote: > > Hi All, > > I have the following directory structure in hdfs > > > > /test/a/ > > /test/a/1.avro > > /test/a/2.avro > > /test/a/b/ > > /test/a/b/3.avro > > > > I created an external HIVE table using Avro Serde and added /test/a as a > > partition to this table. > > > > I am not able to run a select query. Always getting the error 'not a > file' > > on '/test/a/b' > > > > Is this by design, a bug or am I missing some configuration? > > I am using HIVE 0.10 > > > > Thanks > > Suresh > > >
Re: hive issue with sub-directories
Suresh, By default, the partition column name has to be appear in HDFS directory structure. e.g. /user/hive/warehouse//=/data1.txt /user/hive/warehouse//=/data2.txt On Thu, Mar 7, 2013 at 7:20 AM, Suresh Krishnappa wrote: > Hi All, > I have the following directory structure in hdfs > > /test/a/ > /test/a/1.avro > /test/a/2.avro > /test/a/b/ > /test/a/b/3.avro > > I created an external HIVE table using Avro Serde and added /test/a as a > partition to this table. > > I am not able to run a select query. Always getting the error 'not a file' > on '/test/a/b' > > Is this by design, a bug or am I missing some configuration? > I am using HIVE 0.10 > > Thanks > Suresh >
