You need to install 0.6.0 or later to use dynamic partitions
From: Sunil Subrahmanyam [sunil.subrahman...@logile.com]
Sent: Wednesday, September 08, 2010 3:13 PM
To: hive-user@hadoop.apache.org
Subject: RE: Load data from file header
I am very new to hive
urday, September 04, 2010 12:17 AM
To: hive-user@hadoop.apache.org
Subject: RE: Load data from file header
create 2 tables T1 and T2.
T1 has the schema of the file - no partitioning column (say
c1,c2,store#)
T2 is partitioned on (store#) - and the schema is 1 less column (c1, c2
partitioned by
Cant you parse the file to get 2 files ?
From: Sunil Subrahmanyam [sunil.subrahman...@logile.com]
Sent: Saturday, September 04, 2010 6:59 AM
To: hive-user@hadoop.apache.org
Subject: RE: Load data from file header
Thanks Namit for the response.
The format
t;(\\d{6}).{3}(\\d{6}).*") stored as textfile;
Thanks,
Sunil
-Original Message-
From: Namit Jain [mailto:nj...@facebook.com]
Sent: Saturday, September 04, 2010 12:17 AM
To: hive-user@hadoop.apache.org
Subject: RE: Load data from file header
create 2 tables T1 and T2.
T1 has the sche
From: Sunil Subrahmanyam [sunil.subrahman...@logile.com]
Sent: Friday, September 03, 2010 8:11 PM
To: hive-user@hadoop.apache.org
Subject: Load data from file header
Hi,
My data files have a single line (first line) of header information followed by
many
Hi,
My data files have a single line (first line) of header information
followed by many lines of actual data. I am able to load the data into
hive table using RegexSerDe. But I want to save the information in the
header with every data row or use it to partition the table. How do I do
it?
,505 Stage-1 map = 100%, reduce = 100%
>>> 10/06/10 13:51:58 INFO exec.ExecDriver: 2010-06-10 13:51:58,505 Stage-1
>>> map = 100%, reduce = 100%
>>> Ended Job = job_201006101118_0009 with errors
>>> 10/06/10 13:51:58 ERROR exec.ExecDriver: Ended Job =
>>> job_201006101118_0009 w
18_0009&tipid=task_201006101118_0009_m_00
>> -
>>
>> 10/06/10 13:51:58 ERROR exec.ExecDriver:
>> Task with the most failures(4):
>> -
>> Task ID:
>> task_201006101118_0009_m_00
&
this helps.
Carl
On Thu, Jun 10, 2010 at 1:38 PM, Shuja Rehman wrote:
> Hi All
> I want to load data from xml file to hive tables. Any suggestion how can I
> achieve this??
>
> --
> Regards
> Shuja-ur-Rehman Baig
> _
> MS CS - School
Hi All
I want to load data from xml file to hive tables. Any suggestion how can I
achieve this??
--
Regards
Shuja-ur-Rehman Baig
_
MS CS - School of Science and Engineering
Lahore University of Management Sciences (LUMS)
Sector U, DHA, Lahore, 54792, Pakistan
Cell
Hi Carl
Thanks for guidance.
I have loaded the xml file in this way. First I create a table with single
column like this
*create table test (xmlFile String);*
Then I load complete xml in this table using this script
*LOAD DATA LOCAL INPATH '1.xml'
OVERWRITE INTO TABLE test;*
Now
; *Sent:* Wednesday, June 09, 2010 3:07 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Load data from xml using Mapper.py in hive
>
> Hi
> I have created a table in hive (Suppose table1 with two columns, col1 and
> col2 )
>
> now i have an xml file for which i have wri
code 2
> from org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
> -
>
> Any clue???
>
>
> On Thu, Jun 10, 201
ehman
> wrote:
> > Hi
> > I have try to do as you described. Let me explain in steps.
> >
> > 1- create table test (xmlFile String);
> >
> --
> >
> > 2-LOAD DATA LOCAL INPATH '1.xml'
> > OVERWRITE IN
steps.
>
> 1- create table test (xmlFile String);
> ------
>
> 2-LOAD DATA LOCAL INPATH '1.xml'
> OVERWRITE INTO TABLE test;
>
Hi
I have try to do as you described. Let me explain in steps.
1- create table test (xmlFile String);
--
2-LOAD DATA LOCAL INPATH '1.xml'
OVERWRITE INTO
From: Shuja Rehman [mailto:shujamug...@gmail.com]
Sent: Wednesday, June 09, 2010 3:07 PM
To: hive-user@hadoop.apache.org
Subject: Load data from xml using Mapper.py in hive
Hi
I have created a table in hive (Suppose table1 with two columns, col1 and col2 )
now i
Hi
I have created a table in hive (Suppose table1 with two columns, col1 and
col2 )
now i have an xml file for which i have write a python script which read the
xml file and transform it in single row with tab seperated
e.g the output of python script can be
row 1 = val1 val2
row2 = val3
Thanks boss!!
On Thu, Feb 4, 2010 at 12:42 PM, Zheng Shao wrote:
> We can load data/insert overwrite data concurrently as long as they
> are different partitions.
>
> On Thu, Feb 4, 2010 at 6:51 AM, Ryan LeCompte wrote:
> > Hey guys,
> >
> > Is it possible to co
We can load data/insert overwrite data concurrently as long as they
are different partitions.
On Thu, Feb 4, 2010 at 6:51 AM, Ryan LeCompte wrote:
> Hey guys,
>
> Is it possible to concurrently load data into Hive tables (same table,
> different partition)? I'd like to concurr
Hey guys,
Is it possible to concurrently load data into Hive tables (same table,
different partition)? I'd like to concurrently execute the LOAD DATA command
by two separate processes.
Is Hive thread-safe in this regard? Or is it best to run the LOAD DATA
commands serially? How about runnin
Hi Zheng,
Could you please verify my sql;
add jar
CREATE TABLE IF NOT EXISTS PARSE_XML(
column1 String, column2 String
)
STORED AS
INPUTFORMAT 'org.apache.mahout.classifier.bayes.XmlInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
I was thinking that i could
I don't know the details about Amazon S3, but I believe they have
support for their customers.
Would you contact them and let us know the details?
Zheng
On Thu, Jan 21, 2010 at 2:00 PM, ankit bhatnagar wrote:
> Thanks you guys .
> I have a Question if I use Mahout Project's XmlInputFormat
>
>
Thanks you guys .
I have a Question if I use Mahout Project's XmlInputFormat
Then while running on Amazon S3 how would the hive know this format?
I saw that when Amazon executes jobs firstly it installs hive.
I hope this make sense?
Ankit
You might want to take a look through the Hadoop Wiki site and browse their
various tutorials. In addition, you can also follow Cloudera's wonderful
tutorials if you download their virtual machine:
http://www.cloudera.com/hadoop-training-virtual-machine
On Thu, Jan 21, 2010 at 9:45 AM, ankit bhat
Hi Zheng,
Thanks for the info.
One Question though - could be a bad one.
Where does a hadoop sits ?
Ankit
load the data:
4. See http://wiki.apache.org/hadoop/Hive/LanguageManual/DML for load
data command syntax.
Zheng
On Wed, Jan 20, 2010 at 2:00 PM, ankit bhatnagar wrote:
> Mahout project has the XMLInputFormat.
>
> Could you please brief me how would I load data into hive
>
Mahout <http://lucene.apache.org/mahout/> project has the XMLInputFormat.
Could you please brief me how would I load data into hive
Ankit
We need an input format that can read your XML data.
I think Hadoop has a XMLInputFormat or something. Have you tried to
access your data using hadoop?
Zheng
On Wed, Jan 20, 2010 at 1:37 PM, ankit bhatnagar wrote:
> Hi there,
>
> I am new to hive
> is there a way to import data from XML.
>
>
> A
Hi there,
I am new to hive
is there a way to import data from XML.
Ankit
ubject: HDFS move instead of LOAD DATA INPATH?
Hi,
When the "LOAD DATA INPATH" is issued, does Hive modify the metastore data, or
do anything else special besides just moving the files in HDFS?
I've got a daily MR job that runs before I need to load data into a daily Hive
par
Hi,
When the "LOAD DATA INPATH" is issued, does Hive modify the metastore data,
or do anything else special besides just moving the files in HDFS?
I've got a daily MR job that runs before I need to load data into a daily
Hive partition and using the FileSystem class to move the
Mayuran Yogarajah wrote:
I have a table which has a partition that I load data into with:
LOAD DATA INPATH 'filepath' INTO TABLE summary PARTITION('..');
However, this moves the file addressed by /filepath/ into the table or
partition. I
found out from the mailing list t
I have a table which has a partition that I load data into with:
LOAD DATA INPATH 'filepath' INTO TABLE summary PARTITION('..');
However, this moves the file addressed by /filepath/ into the table or
partition. I
found out from the mailing list that I can get around th
lowing
> statement would not move the data files, but only create the directory on
> HDFS.
> hive -e "load data inpath
> '/user/output/nccp/P20090731T29/sessionsFacts_L20090718T03-r-0' into
> table nccp_session_facts partition (dateint = 20090718, hour = 4)
&g
We discovered a problem where loading into a new partition in hive w/o
specifying overwrite¹ doesn¹t work.
If the specified partition does not exist yet, running the following
statement would not move the data files, but only create the directory on
HDFS.
hive -e "load data inpath
&
ve a simple table defined with hive for access logs and
> was trying to import log files with the LOAD DATA LOCAL INPATH command.
> Here's how my command looks like:
>
> LOAD DATA LOCAL INPATH '../test/test.log' INTO TABLE accesslog PARTITION
> (dt='2009-07-14'
Hi,
I'm pretty new to hadoop/hive. I have everything running pretty good on a
single server. I have a simple table defined with hive for access logs and
was trying to import log files with the LOAD DATA LOCAL INPATH command.
Here's how my command looks like:
LOAD DATA LOCAL INPAT
t of your file.
Hive supports TEXTFILE and SEQUENCEFILE natively.
If your file has a customized format, you need to write your own
fileformat classes.
Please take a look at the example added by:
https://issues.apache.org/jira/browse/HIVE-639
Zheng
On Mon, Jul 20, 2009 at 3:53 PM, chen keven wrote:
>
I'm trying to load data into table using the command below. However, I only
got a bunch of NULL in the field. The data fields are seperated by tab.
CREATE TABLE IF NOT EXISTS userweight(source INT, dist INT, weight DOUBLE)
row format delimited fields terminated by " \t";
load da
; page_views_prod SELECT pvs.project_code, pvs.page,
> pvs.referrer, pvs.ip, pvs.created WHERE pvs.created > 1235948360
>
> Can you not, not use OVERWRITE? I want to load data in the staging table
> into the production table but I want it to append the data not e
aid 10k were loaded.
Question #1:
For: FROM page_views_stage pvs INSERT OVERWRITE TABLE
page_views_prod SELECT pvs.project_code, pvs.page,
pvs.referrer, pvs.ip, pvs.created WHERE pvs.created > 1235948360
Can you not, not use OVERWRITE? I want to load data in the staging table
in
>>So evidentially LOAD DATA actually just copies a file to hdfs. What is the
>>solution if you have thousands of files and attempt a hive query because my
>>understanding is that this will be dead slow later.
Loading thousands of files is very slow. I have an application rea
-08.txt /user/data/staging/page_view
FROM page_view_stg pvs
INSERT OVERWRITE TABLE page_view PARTITION(dt='2008-06-08', country='US')
SELECT pvs.viewTime, pvs.userid, pvs.page_url, pvs.referrer_url,
null, null, pvs.ip
WHERE pvs.country = 'US';
On Tu
So evidentially LOAD DATA actually just copies a file to hdfs. What is the
solution if you have thousands of files and attempt a hive query because my
understanding is that this will be dead slow later.
Suhail
On Sun, Apr 5, 2009 at 10:52 AM, Suhail Doshi wrote:
> Ragu,
>
> I managed
t;> the
>>>> > file is located in a directory this user owns.
>>>> >
>>>> > I am not sure why it's not loading it still.
>>>> >
>>>> > Suhail
>>>> >
>>>> > On Sat, Apr 4, 2009 at 10:14 PM, Raghu Murth
nd
>>> the
>>> > file is located in a directory this user owns.
>>> >
>>> > I am not sure why it's not loading it still.
>>> >
>>> > Suhail
>>> >
>>> > On Sat, Apr 4, 2009 at 10:14 PM, Raghu Murthy
this user owns.
>> >
>> > I am not sure why it's not loading it still.
>> >
>> > Suhail
>> >
>> > On Sat, Apr 4, 2009 at 10:14 PM, Raghu Murthy
>> wrote:
>> >> Is the file accessible to the HiveServer? We currently don'
il
> >
> > On Sat, Apr 4, 2009 at 10:14 PM, Raghu Murthy
> wrote:
> >> Is the file accessible to the HiveServer? We currently don't ship the
> file
> >> from the client machine to the server machine.
> >>
> >>
> >> On 4/3/09 10:26
erver machine.
>>
>>
>> On 4/3/09 10:26 PM, "Suhail Doshi" wrote:
>>
>>>> I seem to be having problems with LOAD DATA with a file on my local system
>>>> trying get it into hive:
>>>>
>>>> li57-125 ~/test: python
Raghu Murthy wrote:
> Is the file accessible to the HiveServer? We currently don't ship the file
> from the client machine to the server machine.
>
>
> On 4/3/09 10:26 PM, "Suhail Doshi" wrote:
>
> > I seem to be having problems with LOAD DATA with a file on m
Is the file accessible to the HiveServer? We currently don't ship the file
from the client machine to the server machine.
On 4/3/09 10:26 PM, "Suhail Doshi" wrote:
> I seem to be having problems with LOAD DATA with a file on my local system
> trying get it into hive:
I seem to be having problems with LOAD DATA with a file on my local system
trying get it into hive:
li57-125 ~/test: python hive_test.py
Connecting to HiveServer
Opening transport...
LOAD DATA LOCAL INPATH '/home/hadoop/test/page_view.log.2' INTO TABLE
page_views
Traceback (most r
already has a file whose name
>>> collides with any of the filenames contained in *filepath* - then the
>>> existing file will be replaced with the new file."
>>>
>>> I have rotating data logs that start at log.1 and go to log.512 and wrap
>>> aroun
laced with the new file."
I have rotating data logs that start at log.1 and go to log.512 and
wrap around back to log.1, does this mean that when I try to LOAD
DATA log.1 again it's going to overwrite the other one?
In normal MySQL, this data is just constantly appended regardles
the target table (or partition) already has a file whose name
>> collides with any of the filenames contained in *filepath* - then the
>> existing file will be replaced with the new file."
>>
>> I have rotating data logs that start at log.1 and go to log.512 and
has a file whose name
> collides with any of the filenames contained in *filepath* - then the
> existing file will be replaced with the new file."
>
> I have rotating data logs that start at log.1 and go to log.512 and wrap
> around back to log.1, does this mean that when I try
e rotating data logs that start at log.1 and go to log.512 and wrap
around back to log.1, does this mean that when I try to LOAD DATA log.1
again it's going to overwrite the other one?
In normal MySQL, this data is just constantly appended regardless of the
file name, but given how it'
: Tuesday, December 02, 2008 10:40 AM
To: hive-user@hadoop.apache.org; Zheng Shao
Subject: RE: Re:RE: [hive-users] "LOAD DATA" From hdfs can'r work under hadoop
0.19
Hi Paradisehi
The issue is that the default file system uri obtained from hadoop config
variable fs.default.na
, December 02, 2008 1:46 AM
To: Zheng Shao
Cc: hive-user@hadoop.apache.org
Subject: Re:RE: [hive-users] "LOAD DATA" From hdfs can'r work under hadoop 0.19
hive> LOAD DATA INPATH 'hdfs://xx-mapred000.xx01:54310/test/shixing/log' INTO
TABLE log_stg4;
FAILED: Error in seman
hive> LOAD DATA INPATH 'hdfs://xx-mapred000.xx01:54310/test/shixing/log' INTO
TABLE log_stg4;
FAILED: Error in semantic analysis: line 1:17 Path is not legal
'hdfs://xx-mapred000.xx01:54310/test/shixing/log': Cannot load data across
filesystems, use load data local
The syntax should be
LOAD DATA LOCAL INPATH
instead of
LOAD DATA LOCAL INFILE
Please follow information at
http://wiki.apache.org/hadoop/Hive/GettingStarted
Ashish
From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of tim robertson [EMAIL
62 matches
Mail list logo
