Load Hive query result with array field into pig
I'm executing a hive query in which one of the fields an array and writing it to a file using: INSERT OVERWRITE '/path/to/output' SELECT ... This query works well. I would like to load this data into pig, but I'm quite sure how to get the array properly into pig. My output file from the query doesn't look like it has any array notation when using the default ^A separator. Is there a way I can execute a query with an array field output that can then be directly loaded into pig as a tuple? Thank you.
Re: Load Hive query result with array field into pig
Correction - it looks like the query uses \u002 to separate array elements
and \u001 to separate the other fields. The question is still similar
though in wondering how I can load that array into pig.
Note - If my data is formatted as a tsv with parentheses surrounding the
array:
(element1,element2,element3)anotherfield
This loads properly when I use LOAD '/my/tsvfile' USING PigStorage('\t') AS
(elements:tuple(),afield:chararray);
On Fri, Mar 21, 2014 at 8:38 AM, Jeff Storey storey.j...@gmail.com wrote:
I'm executing a hive query in which one of the fields an array and writing
it to a file using:
INSERT OVERWRITE '/path/to/output' SELECT ...
This query works well. I would like to load this data into pig, but I'm
quite sure how to get the array properly into pig.
My output file from the query doesn't look like it has any array notation
when using the default ^A separator.
Is there a way I can execute a query with an array field output that can
then be directly loaded into pig as a tuple?
Thank you.
Re: Load Hive query result with array field into pig
Sorry for another post on this thread. I had an error in my pigscript that
had the wrong unicode character to split on. Using STRSPLIT worked well.
On Fri, Mar 21, 2014 at 8:46 AM, Jeff Storey storey.j...@gmail.com wrote:
Correction - it looks like the query uses \u002 to separate array elements
and \u001 to separate the other fields. The question is still similar
though in wondering how I can load that array into pig.
Note - If my data is formatted as a tsv with parentheses surrounding the
array:
(element1,element2,element3)anotherfield
This loads properly when I use LOAD '/my/tsvfile' USING PigStorage('\t')
AS (elements:tuple(),afield:chararray);
On Fri, Mar 21, 2014 at 8:38 AM, Jeff Storey storey.j...@gmail.comwrote:
I'm executing a hive query in which one of the fields an array and
writing it to a file using:
INSERT OVERWRITE '/path/to/output' SELECT ...
This query works well. I would like to load this data into pig, but I'm
quite sure how to get the array properly into pig.
My output file from the query doesn't look like it has any array notation
when using the default ^A separator.
Is there a way I can execute a query with an array field output that can
then be directly loaded into pig as a tuple?
Thank you.
Re: Hive 13
We do not have a firm release date yet. The branch has been cut. I think Harish said he’d like to have a first RC early next week. It usually takes 1 to 2 weeks after the first RC, depending on any show stoppers found in it, etc. Alan. On Mar 19, 2014, at 6:50 AM, Bryan Jeffrey bryan.jeff...@gmail.com wrote: Hello. Is there a firm release date for Hive 13? I know there was talk several weeks ago about cutting a branch and looking at stability. Regards, Bryan Jeffrey -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Issue with Querying External Hive Table created on hbase
I am not sure about your question. Do you mean the query runs very fast if you run like 'select * from hbase_table', but very slow for 'select * from hbase where row_key = ?' I think it should be the other way round, right? Yong Date: Wed, 19 Mar 2014 11:42:39 -0700 From: sunil_ra...@yahoo.com Subject: Issue with Querying External Hive Table created on hbase To: user@hive.apache.org Hi All I am trying to query External Hive Table created on hbase ( hbase table is compressed using gzip) . I am getting quick response, if I use select * from hbase_acct_pref_dim_, but the query is taking for ever if I try to retrieve data based on the row_key. hive select * from hbase_acct_pref_dim_ where key = 30001; Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 02014-03-19 11:14:04,432 Stage-1 map = 0%, reduce = 0%2014-03-19 11:15:04,617 Stage-1 map = 0%, reduce = 0%2014-03-19 11:16:04,792 Stage-1 map = 0%, reduce = 0%2014-03-19 11:17:04,969 Stage-1 map = 0%, reduce = 0%2014-03-19 11:18:05,140 Stage-1 map = 0%, reduce = 0%2014-03-19 11:19:05,315 Stage-1 map = 0%, reduce = 0%2014-03-19 11:20:05,484 Stage-1 map = 0%, reduce = 0%2014-03-19 11:21:05,667 Stage-1 map = 0%, reduce = 0%2014-03-19 11:22:05,835 Stage-1 map = 0%, reduce = 0% Any Help is appreciated. Thanks, Sunil S Ranka Blog :: http://sranka.wordpress.com Superior BI is the antidote to Business Failure NOTHING IS IMPOSSIBLE EVEN THE WORD IMPOSSIBLE SAYS ,I M POSSIBLE.
