Re: Avoid creation of _SUCCESS folder in reducer output folder
mapreduce.fileoutputcommitter.marksuccessfuljobs=false; MAPREDUCE-947, i guess.. ~Aniket On Thu, Oct 4, 2012 at 11:06 PM, Balaraman, Anand < anand_balara...@syntelinc.com> wrote: > Hi > > ** ** > > While using Map reduce programs, the output folder where reducer writes > out the result contains 2 auto-generated folders: *_SUCCESS* and *_logs*.* > *** > > To avoid generation of *_log* folder, I can set the configuration > parameter “*hadoop.job.history.user.location*” with value as “*none*”. > > But, I don’t know how to avoid the creation of *_SUCCESS* folder. Can > anyone please help. > > ** ** > > *Note*: For now, as a work around, I am using FileSystem commands at the > end of job to delete the folder manually. > > ** ** > > Anand B > Confidential: This electronic message and all contents contain > information from Syntel, Inc. which may be privileged, confidential or > otherwise protected from disclosure. The information is intended to be for > the addressee only. If you are not the addressee, any disclosure, copy, > distribution or use of the contents of this message is prohibited. If you > have received this electronic message in error, please notify the sender > immediately and destroy the original message and all copies. > -- "...:::Aniket:::... Quetzalco@tl"
Avoid creation of _SUCCESS folder in reducer output folder
Hi While using Map reduce programs, the output folder where reducer writes out the result contains 2 auto-generated folders: _SUCCESS and _logs. To avoid generation of _log folder, I can set the configuration parameter "hadoop.job.history.user.location" with value as "none". But, I don't know how to avoid the creation of _SUCCESS folder. Can anyone please help. Note: For now, as a work around, I am using FileSystem commands at the end of job to delete the folder manually. Anand B Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Lateral Views and Multi Table Insert
Hi, I have a question about using lateral views with multi table insert. I have a table of data that represents raw log data, the structure of which makes it onerous to query directly largely because it requires UNIONTYPE columns. So, I transform that raw table into 3 new tables, a primary table and 2 1-to-many tables. The raw table is similar to this: CREATE TABLE IF NOT EXISTS events_raw ( event_id STRING, event_data_0 INT, event_data_1 BIGINT, packed_event_data_2 UNIONTYPE < INT, STRUCT < event_data_2:INT, event_data_2_sub_0:BOOLEAN, event_data_2_sub_1:BOOLEAN>>, packed_event_data_3 UNIONTYPE < BIGINT, ARRAY < STRUCT < event_data_3_metadata_key:STRING, event_data_3_metadata_value:STRING>>>, packed_event_data_4 UNIONTYPE < BOOLEAN, STRUCT < event_data_4:BOOLEAN, event_data_4_metadata:ARRAY < STRUCT < event_data_4_metadata_key:STRING, event_data_4_metadata_value:STRING); This is to be transformed into these tables: CREATE TABLE IF NOT EXISTS events ( event_id STRING, event_data_0 INT, event_data_1 BIGINT, event_data_2 INT, event_data_2_sub_0 BOOLEAN, event_data_2_sub_1 BOOLEAN, event_data_3 BIGINT, event_data_4 BOOLEAN); CREATE TABLE IF NOT EXISTS event_data_3_metadata ( event_id STRING, metadata_key STRING, metadata_value STRING); CREATE TABLE IF NOT EXISTS event_data_4_metadata ( event_id STRING, metadata_key STRING, metadata_value STRING); The only way I know how to unpack and/or explode the UNIONTYPEs is to create custom UDTFs for each UNIONTYPE column. For example, I created an unpack_packed_event_data_2 function which maps an single UnionObject to a STRUCT. Similarly, I created UDTFs to explode the ARRAY elements contained in the UNIONTYPE columns. Using those UDTFs I devised these queries to build the transformed tables: FROM events_raw LATERAL VIEW unpack_event_data_2 (packed_event_data_2) event_data_2_struct AS event_data_2, event_data_2_sub_0, event_data_2_sub_1 LATERAL VIEW unpack_event_data_3 (packed_event_data_3) event_data_3_struct AS event_data_3 LATERAL VIEW unpack_event_data_4 (packed_event_data_4) event_data_4_struct AS event_data_4 INSERT INTO TABLE events SELECT event_id, event_data_0, event_data_1, event_data_2_struct.event_data_2, event_data_2_struct.event_data_2_sub_0, event_data_2_struct.event_data_2_sub_1, event_data_3_struct.event_data_3, event_data_4_struct.event_data_4); FROM events_raw LATERAL VIEW explode_event_data_3 (packed_event_data_3) event_data_3_array_element AS metadata_key, metadata_value INSERT INTO TABLE event_data_3_metadata SELECT event_id, event_data_3_array_element.metadata_key, event_data_3_array_element.metadata_value); FROM events_raw LATERAL VIEW explode_event_data_4 (packed_event_data_4) event_data_4_array_element AS metadata_key, metadata_value INSERT INTO TABLE event_data_3_metadata SELECT event_id, event_data_4_array_element.metadata_key, event_data_4_array_element.metadata_value); This works correctly, the tables are filled with the appropriate number of rows. However, the raw table is scanned 3 times to accomplish this and that is very costly given the amount of data. When I combine those 3 statements into one Multi Table Insert: FROM events_raw LATERAL VIEW unpack_event_data_2 (packed_event_data_2) event_data_2_struct AS event_data_2, event_data_2_sub_0, event_data_2_sub_1 LATERAL VIEW unpack_event_data_3 (packed_event_data_3) event_data_3_struct AS event_data_3 LATERAL VIEW unpack_event_data_4 (packed_event_data_4) event_data_4_struct AS event_data_4 LATERAL VIEW explode_event_data_3 (packed_event_data_3) event_data_3_array_element AS metadata_key, metadata_value LATERAL VIEW explode_event_data_4 (packed_event_data_4) event_data_4_array_element AS metadata_key, metadata_value INSERT INTO TABLE events SELECT event_id, event_data_0, event_data_1, event_data_2_struct.event_data_2, event_data_2_struct.event_data_2_sub_0, event_data_2_struct.event_data_2_sub_1, event_data_3_struct.event_data_3, event_data_4_struct.event_data_4 INSERT INTO TABLE event_data_3_metadata SELECT event_id, event_data_3_array_element.metadata_key, event_data_3_array_element.metadata_value INSERT INTO TABLE event_data_4_metadata SELECT event_id, event_data_4_array_element.metadata_key, event_data_4_array_element.metadata_value; The query fails with: [Hive Error]: Query returned non-zero code: 10, cause: FAILED: Error in semantic analysis: Column packed_event_data_3 Found in more than One Tables/Subqueries. I don't know how to get around having separate unpack_event_data_3 and explode_event_data_3 functions. Combining them would seem to marry the functions' output signatures and in instances when the BIGINT is type of the UNIONTYPE there shouldn't be a row of NULL values in the event_data
Re: Date Comparisons. in Hive
I suggest you store unix timestamp in hive, and so you can compare it
as BIGINT without worrying about STRING comparison.
And if your data is to be queried on daily bases, you can split one
big file into small files, say, one file per day, then add them as
partitions of soj_session_container. This way can optimize hive a
little since your queries won't have to read all records in
soj_session_container.
CREATE TABLE soj_session_container (
events MAP
)
PARTITIONED BY (date STRING);
ALTER TABLE soj_session_container ADD PARTITION (date = '20120918')
location 'loc1';
SELECT * FROM soj_session_container LATERAL VIEW explode(a.events) t
AS event WHERE date = '20120918' AND event.event_timestamp >=
unix_timestamp('2012-09-18 00:00:00') AND event.event_timestamp <=
unix_timestamp('2012-09-18 02:00:00');
On Thu, Oct 4, 2012 at 8:20 AM, Raihan Jamal wrote:
> I have this below query from which I am trying to find out those records
> that fall between midnight and 2 A.M on 18th September.
> And SojTimestampToDate function will give me date in this format /MM/dd
> HH:mm:ss
>
> I am not sure whether the date comparison I did is right or not. And it will
> give me all those records between midnight and 2 AM.
>
> SELECT event.app_payload ['n'] AS changed_cguid
> FROM soj_session_container a LATERAL VIEW explode(a.events) t AS event
> WHERE a.dt = '20120918'
> AND SojTimestampToDate(event.event_timestamp) >= '2012/09/18 00:00:00'
> AND SojTimestampToDate(event.event_timestamp) <= '2012/09/18 02:00:00'
>
> Can anyone shed some light on this whether I am doing right or not?
>
>
>
> Raihan Jamal
>
Re: no data in external table
Can you try creating a table like this:
CREATE EXTERNAL TABLE hbase_table_2(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "xyz");
Now do a select * from hbase_table_2;
Do you see any data now?
On Thu, Oct 4, 2012 at 5:10 PM, wrote:
> Hi,
>
> In the hbase table I do not see column qualifier, only family.
> For testing connection to hbase I also created a table using
>
> CREATE TABLE hbase_table_1(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "xyz");
>
> I see xyz table in hbase. then I added a row in hbase using put 'xyz',
> 'row1', 'cf1', 'abc'
>
>
> Then in hive I did: select * from hbase_table_1;
> No results are returned, but scan xys in hbase returns 1 row.
>
> Thanks.
>
> Alex.
>
> -Original Message-
> From: kulkarni.swarnim
> To: user
> Sent: Thu, Oct 4, 2012 3:00 pm
> Subject: Re: no data in external table
> > "hbase.columns.mapping" = ":key,mtdt:string,il:string,ol:string"
>
> This doesn't look right. The mapping should be of form
> COLUMN_FAMILY:COLUMN_QUALIFIER. In this case it seems to be
> COLUMN_FAMILY:TYPE which is not right.
>
> On Thu, Oct 4, 2012 at 3:25 PM, wrote:
>
>> Hi,
>>
>> In hive shell I did
>>
>> create external table myextrenaltable (key string, metadata string,
>> inlinks string, outlinks string) stored by
>> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>> with serdeproperties ("hbase.columns.mapping" =
>> ":key,mtdt:string,il:string,ol:string")
>> tblproperties ("hbase.table.name" = "myextrenaltable");
>>
>> In tasktracker log I do not see anything relevant to hbase. In jobdetails
>> page I see a few successful jobs. in hive shell I see
>>
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> Number of reduce tasks is set to 0 since there's no reduce operator
>> Starting Job = job_201210031146_0016, Tracking URL =
>> http://localhost:50030/jobdetails.jsp?jobid=job_201210031146_0016
>> Kill Command = /home/dev/hadoop-0.20.2/bin/../bin/hadoop job
>> -Dmapred.job.tracker=localhost:9001 -kill job_201210031146_0016
>> Hadoop job information for Stage-1: number of mappers: 1; number of
>> reducers: 0
>> 2012-10-04 13:19:06,581 Stage-1 map = 0%, reduce = 0%
>> 2012-10-04 13:19:12,629 Stage-1 map = 100%, reduce = 0%
>> 2012-10-04 13:19:15,657 Stage-1 map = 100%, reduce = 100%
>> Ended Job = job_201210031146_0016
>> MapReduce Jobs Launched:
>> Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 SUCCESS
>> Total MapReduce CPU Time Spent: 0 msec
>> OK
>> Time taken: 17.47 seconds
>>
>>
>>
>> Thanks in advance.
>> Alex.
>>
>> -Original Message-
>> From: Ted Yu
>> To: user
>> Sent: Thu, Oct 4, 2012 11:33 am
>> Subject: Re: no data in external table
>>
>> Can you tell us how you created mapping for the existing table ?
>>
>> In task log, do you see any connection attempt to HBase ?
>>
>> Cheers
>>
>> On Thu, Oct 4, 2012 at 11:30 AM, wrote:
>>
>>> Hello,
>>>
>>> I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created
>>> external table, mapping it to an existing table in hbase. When I do "select
>>> * from myextrenaltable" it returns no results, although scan in hbase shows
>>> data, and I do not see any errors in jobtracker log.
>>>
>>> Any ideas how to debug this issue.
>>>
>>> Thanks.
>>> Alex.
>>>
>>
>>
>
>
> --
> Swarnim
>
--
Swarnim
Re: no data in external table
Hi,
In the hbase table I do not see column qualifier, only family.
For testing connection to hbase I also created a table using
CREATE TABLE hbase_table_1(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "xyz");
I see xyz table in hbase. then I added a row in hbase using put 'xyz', 'row1',
'cf1', 'abc'
Then in hive I did: select * from hbase_table_1;
No results are returned, but scan xys in hbase returns 1 row.
Thanks.
Alex.
-Original Message-
From: kulkarni.swarnim
To: user
Sent: Thu, Oct 4, 2012 3:00 pm
Subject: Re: no data in external table
> "hbase.columns.mapping" = ":key,mtdt:string,il:string,ol:string"
This doesn't look right. The mapping should be of form
COLUMN_FAMILY:COLUMN_QUALIFIER. In this case it seems to be COLUMN_FAMILY:TYPE
which is not right.
On Thu, Oct 4, 2012 at 3:25 PM, wrote:
Hi,
In hive shell I did
create external table myextrenaltable (key string, metadata string, inlinks
string, outlinks string) stored by
'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties ("hbase.columns.mapping" =
":key,mtdt:string,il:string,ol:string")
tblproperties ("hbase.table.name" = "myextrenaltable");
In tasktracker log I do not see anything relevant to hbase. In jobdetails page
I see a few successful jobs. in hive shell I see
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201210031146_0016, Tracking URL =
http://localhost:50030/jobdetails.jsp?jobid=job_201210031146_0016
Kill Command = /home/dev/hadoop-0.20.2/bin/../bin/hadoop job
-Dmapred.job.tracker=localhost:9001 -kill job_201210031146_0016
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-10-04 13:19:06,581 Stage-1 map = 0%, reduce = 0%
2012-10-04 13:19:12,629 Stage-1 map = 100%, reduce = 0%
2012-10-04 13:19:15,657 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201210031146_0016
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 17.47 seconds
Thanks in advance.
Alex.
-Original Message-
From: Ted Yu
To: user
Sent: Thu, Oct 4, 2012 11:33 am
Subject: Re: no data in external table
Can you tell us how you created mapping for the existing table ?
In task log, do you see any connection attempt to HBase ?
Cheers
On Thu, Oct 4, 2012 at 11:30 AM, wrote:
Hello,
I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created external
table, mapping it to an existing table in hbase. When I do "select * from
myextrenaltable" it returns no results, although scan in hbase shows data, and
I do not see any errors in jobtracker log.
Any ideas how to debug this issue.
Thanks.
Alex.
--
Swarnim
Re: no data in external table
> "hbase.columns.mapping" = ":key,mtdt:string,il:string,ol:string"
This doesn't look right. The mapping should be of form
COLUMN_FAMILY:COLUMN_QUALIFIER. In this case it seems to be
COLUMN_FAMILY:TYPE which is not right.
On Thu, Oct 4, 2012 at 3:25 PM, wrote:
> Hi,
>
> In hive shell I did
>
> create external table myextrenaltable (key string, metadata string,
> inlinks string, outlinks string) stored by
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with serdeproperties ("hbase.columns.mapping" =
> ":key,mtdt:string,il:string,ol:string")
> tblproperties ("hbase.table.name" = "myextrenaltable");
>
> In tasktracker log I do not see anything relevant to hbase. In jobdetails
> page I see a few successful jobs. in hive shell I see
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201210031146_0016, Tracking URL =
> http://localhost:50030/jobdetails.jsp?jobid=job_201210031146_0016
> Kill Command = /home/dev/hadoop-0.20.2/bin/../bin/hadoop job
> -Dmapred.job.tracker=localhost:9001 -kill job_201210031146_0016
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2012-10-04 13:19:06,581 Stage-1 map = 0%, reduce = 0%
> 2012-10-04 13:19:12,629 Stage-1 map = 100%, reduce = 0%
> 2012-10-04 13:19:15,657 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201210031146_0016
> MapReduce Jobs Launched:
> Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 17.47 seconds
>
>
>
> Thanks in advance.
> Alex.
>
> -Original Message-
> From: Ted Yu
> To: user
> Sent: Thu, Oct 4, 2012 11:33 am
> Subject: Re: no data in external table
>
> Can you tell us how you created mapping for the existing table ?
>
> In task log, do you see any connection attempt to HBase ?
>
> Cheers
>
> On Thu, Oct 4, 2012 at 11:30 AM, wrote:
>
>> Hello,
>>
>> I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created
>> external table, mapping it to an existing table in hbase. When I do "select
>> * from myextrenaltable" it returns no results, although scan in hbase shows
>> data, and I do not see any errors in jobtracker log.
>>
>> Any ideas how to debug this issue.
>>
>> Thanks.
>> Alex.
>>
>
>
--
Swarnim
Re: Class 'ArcanistDifferentialRevisionRef' not found
The update did not succeed with this error. Did anyone have similar case before or know anything about this? On Thu, Oct 4, 2012 at 10:23 AM, Feng Lu wrote: > Thanks for your reply, Edward. > But for this case, the update did not succeed. > > > > On Thu, Oct 4, 2012 at 9:27 AM, Edward Capriolo wrote: > >> Even with this exception I thing the update still succeeds. I do not >> think arc is working 100% correct for anyone (for any version of it). >> >> >> >> On Wed, Oct 3, 2012 at 11:05 PM, Feng Lu >> wrote: >> > Hi, >> > >> > I was trying to do "arc diff --update ..." under Ubuntu and got this >> error: >> > >> > >> > PHP Fatal error: Class 'ArcanistDifferentialRevisionRef' not found in >> > >> /home/feng/projects/hive/hive-trunk2/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php >> > on line 0 >> > >> > Fatal error: Class 'ArcanistDifferentialRevisionRef' not found in >> > >> /home/feng/projects/hive/hive-trunk2/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php >> > on line 0 >> > >> > >> > There was a post with a similar problem here >> > >> > >> http://mail-archives.apache.org/mod_mbox/hive-dev/201205.mbox/%3CCA%2BFBdFQU2nQXK9c0sLHJz8UAvXuiCxhz2XMuuCLS-Xb9bb2Xgg%40mail.gmail.com%3E >> > >> > But I didn't find an answer from this thread. >> > >> > Does anyone know how to solve this problem? >> > >> > Thanks, >> > Feng >> > >
RE: Limit to columns or nesting of Hive table?
The issue apparently is not just the number of levels of nesting. I just created a Hive table with 20 levels of structs within each other. It created fine. This is more levels than the table that was failing for me. The failing table had many more fields throughout the levels. Chuck -Original Message- From: Connell, Chuck [mailto:chuck.conn...@nuance.com] Sent: Thursday, October 04, 2012 12:09 PM To: user@hive.apache.org Subject: RE: Limit to columns or nesting of Hive table? Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that this limit cannot easily be raised? Chuck -Original Message- From: Edward Capriolo [mailto:edlinuxg...@gmail.com] Sent: Thursday, October 04, 2012 11:57 AM To: user@hive.apache.org Subject: Re: Limit to columns or nesting of Hive table? There is an open jira ticket on this. There is a hard coded limit but it could be raised with some mostly minor code changes. One of the bigger problems is that hive stores the definition of a column in JDBC "column" and for some databases larger nested structs can case issues. Edward On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck wrote: > I am trying to create a large Hive table, with many columns and deeply > nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException: > 10. > > > > Before I spend a lot of time debugging my table declaration, is there > some limit here I should know about? Max number of columns? Max depth > of struct nesting? > > > > Thanks, > > Chuck > >
Re: no data in external table
Hi,
In hive shell I did
create external table myextrenaltable (key string, metadata string, inlinks
string, outlinks string) stored by
'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties ("hbase.columns.mapping" =
":key,mtdt:string,il:string,ol:string")
tblproperties ("hbase.table.name" = "myextrenaltable");
In tasktracker log I do not see anything relevant to hbase. In jobdetails page
I see a few successful jobs. in hive shell I see
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201210031146_0016, Tracking URL =
http://localhost:50030/jobdetails.jsp?jobid=job_201210031146_0016
Kill Command = /home/dev/hadoop-0.20.2/bin/../bin/hadoop job
-Dmapred.job.tracker=localhost:9001 -kill job_201210031146_0016
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-10-04 13:19:06,581 Stage-1 map = 0%, reduce = 0%
2012-10-04 13:19:12,629 Stage-1 map = 100%, reduce = 0%
2012-10-04 13:19:15,657 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201210031146_0016
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 17.47 seconds
Thanks in advance.
Alex.
-Original Message-
From: Ted Yu
To: user
Sent: Thu, Oct 4, 2012 11:33 am
Subject: Re: no data in external table
Can you tell us how you created mapping for the existing table ?
In task log, do you see any connection attempt to HBase ?
Cheers
On Thu, Oct 4, 2012 at 11:30 AM, wrote:
Hello,
I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created external
table, mapping it to an existing table in hbase. When I do "select * from
myextrenaltable" it returns no results, although scan in hbase shows data, and
I do not see any errors in jobtracker log.
Any ideas how to debug this issue.
Thanks.
Alex.
Re: no data in external table
Can you tell us how you created mapping for the existing table ? In task log, do you see any connection attempt to HBase ? Cheers On Thu, Oct 4, 2012 at 11:30 AM, wrote: > Hello, > > I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created > external table, mapping it to an existing table in hbase. When I do "select > * from myextrenaltable" it returns no results, although scan in hbase shows > data, and I do not see any errors in jobtracker log. > > Any ideas how to debug this issue. > > Thanks. > Alex. >
no data in external table
Hello, I use hive-0.9.0 with hadoop-0.20.2 and hbase -0.92.1. I have created external table, mapping it to an existing table in hbase. When I do "select * from myextrenaltable" it returns no results, although scan in hbase shows data, and I do not see any errors in jobtracker log. Any ideas how to debug this issue. Thanks. Alex.
RE: Limit to columns or nesting of Hive table?
Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that this limit cannot easily be raised? Chuck -Original Message- From: Edward Capriolo [mailto:edlinuxg...@gmail.com] Sent: Thursday, October 04, 2012 11:57 AM To: user@hive.apache.org Subject: Re: Limit to columns or nesting of Hive table? There is an open jira ticket on this. There is a hard coded limit but it could be raised with some mostly minor code changes. One of the bigger problems is that hive stores the definition of a column in JDBC "column" and for some databases larger nested structs can case issues. Edward On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck wrote: > I am trying to create a large Hive table, with many columns and deeply > nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException: > 10. > > > > Before I spend a lot of time debugging my table declaration, is there > some limit here I should know about? Max number of columns? Max depth > of struct nesting? > > > > Thanks, > > Chuck > >
Re: Limit to columns or nesting of Hive table?
There is an open jira ticket on this. There is a hard coded limit but it could be raised with some mostly minor code changes. One of the bigger problems is that hive stores the definition of a column in JDBC "column" and for some databases larger nested structs can case issues. Edward On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck wrote: > I am trying to create a large Hive table, with many columns and deeply > nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException: > 10. > > > > Before I spend a lot of time debugging my table declaration, is there some > limit here I should know about? Max number of columns? Max depth of struct > nesting? > > > > Thanks, > > Chuck > >
Limit to columns or nesting of Hive table?
I am trying to create a large Hive table, with many columns and deeply nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException: 10. Before I spend a lot of time debugging my table declaration, is there some limit here I should know about? Max number of columns? Max depth of struct nesting? Thanks, Chuck
Re: Class 'ArcanistDifferentialRevisionRef' not found
Thanks for your reply, Edward. But for this case, the update did not succeed. On Thu, Oct 4, 2012 at 9:27 AM, Edward Capriolo wrote: > Even with this exception I thing the update still succeeds. I do not > think arc is working 100% correct for anyone (for any version of it). > > > > On Wed, Oct 3, 2012 at 11:05 PM, Feng Lu > wrote: > > Hi, > > > > I was trying to do "arc diff --update ..." under Ubuntu and got this > error: > > > > > > PHP Fatal error: Class 'ArcanistDifferentialRevisionRef' not found in > > > /home/feng/projects/hive/hive-trunk2/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php > > on line 0 > > > > Fatal error: Class 'ArcanistDifferentialRevisionRef' not found in > > > /home/feng/projects/hive/hive-trunk2/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php > > on line 0 > > > > > > There was a post with a similar problem here > > > > > http://mail-archives.apache.org/mod_mbox/hive-dev/201205.mbox/%3CCA%2BFBdFQU2nQXK9c0sLHJz8UAvXuiCxhz2XMuuCLS-Xb9bb2Xgg%40mail.gmail.com%3E > > > > But I didn't find an answer from this thread. > > > > Does anyone know how to solve this problem? > > > > Thanks, > > Feng >
Re: Class 'ArcanistDifferentialRevisionRef' not found
Even with this exception I thing the update still succeeds. I do not think arc is working 100% correct for anyone (for any version of it). On Wed, Oct 3, 2012 at 11:05 PM, Feng Lu wrote: > Hi, > > I was trying to do "arc diff --update ..." under Ubuntu and got this error: > > > PHP Fatal error: Class 'ArcanistDifferentialRevisionRef' not found in > /home/feng/projects/hive/hive-trunk2/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php > on line 0 > > Fatal error: Class 'ArcanistDifferentialRevisionRef' not found in > /home/feng/projects/hive/hive-trunk2/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php > on line 0 > > > There was a post with a similar problem here > > http://mail-archives.apache.org/mod_mbox/hive-dev/201205.mbox/%3CCA%2BFBdFQU2nQXK9c0sLHJz8UAvXuiCxhz2XMuuCLS-Xb9bb2Xgg%40mail.gmail.com%3E > > But I didn't find an answer from this thread. > > Does anyone know how to solve this problem? > > Thanks, > Feng
