Re:

2013-06-30 Thread m_muthukumar

http://pcvsokolka.com.pl/wel.come.php



Hive Compile Error: WrappedRuntimeException

2013-06-30 Thread Xun Tang
Hi,

** **

I am trying to compile hive 0.10.

Build fails with error ‘WrappedRuntimeException’, related to this jira.

https://issues.apache.org/jira/browse/HIVE-4231

** **

The jira says ‘easy workaround: unset _JAVA_OPTIONS’.

However, I didn’t set any java options, nor did the command ‘unset
_JAVA_OPTIONS’ change anything. Same error.

** **

Here is more info.

OS: Ubuntu 12.10

Java: sun 1.6.0_45

** **

Any suggestion welcome.

** **

Thanks,

Alice


Re: Performance difference between tuning reducer num and partition table

2013-06-30 Thread Felix . 徐
Hi Dean,

Thanks for your reply. If I don't set the number of reducers in the 1st run
, the number of reducers will be much smaller and the performance will be
worse. The total output file size is about 200MB, I see that many reduce
output files are empty, only 10 of them have data.

Another question is that , is there any documentation about the job
specific parameters of MapReduce and Hive?




2013/6/29 Dean Wampler deanwamp...@gmail.com

 What happens if you don't set the number of reducers in the 1st run? How
 many reducers are executed. If it's a much smaller number, the extra
 overhead could matter. Another clue is the size of the files the first run
 produced, i.e., do you have 30 small (much less than a block size) files?

 On Sat, Jun 29, 2013 at 12:27 AM, Felix.徐 ygnhz...@gmail.com wrote:

 Hi Stephen,

 My query is actually more complex , hive will generate 2 mapreduces,
 in the first solution , it runs 17 mappers / 30 reducers and 10 mappers /
 30 reducers (reducer num is set manually)
 in the second solution , it runs 6 mappers / 1 reducer and 4 mappers / 1
 reducers for each partition

 I do not know whether they could achieve the same performance if the
 reducers num is set properly.


 2013/6/29 Stephen Sprague sprag...@gmail.com

 great question.  your parallelization seems to trump hadoop's.I
 guess i'd ask what are the _total_ number of Mappers and Reducers that run
 on your cluster for these two scenarios?   I'd be curious if there are the
 same.




 On Fri, Jun 28, 2013 at 8:40 AM, Felix.徐 ygnhz...@gmail.com wrote:

 Hi all,

 Here is the scenario, suppose I have 2 tables A and B, I would like to
 perform a simple join on them,

 We can do it like this:

 INSERT OVERWRITE TABLE C
 SELECT  FROM A JOIN B on A.id=B.id

 In order to speed up this query since table A and B have lots of data,
 another approach is :

 Say I partition table A and B into 10 partitions respectively, and
 write the query like this

 INSERT OVERWRITE TABLE C PARTITION(pid=1)
 SELECT  FROM A JOIN B on A.id=B.id WHERE A.pid=1 AND B.pid=1

 then I run this query 10 times concurrently (pid ranges from 1 to 10)

 And my question is that , in my observation of some more complex
 queries, the second solution is about 15% faster than the first solution,
 is it simply because the setting of reducer num is not optimal?
 If the resource is not a limit and it is possible to set the proper
 reducer nums in the first solution , can they achieve the same performance?
 Is there any other fact that can cause performance difference between
 them(non-partition VS partition+concurrent) besides the job parameter
 issues?

 Thanks!






 --
 Dean Wampler, Ph.D.
 @deanwampler
 http://polyglotprogramming.com


Correct way of using regexserde

2013-06-30 Thread Mohammad Tariq
Hello list,

 I would really appreciate if someone could show me the correct way
of using regexserde as i'm having some hard time using it. I have verified
my regex through http://www.regexplanet.com/advanced/java/index.html and
it's working fine there. But when i'm using the same pattern with
regexserde i'm getting NULL.

My input looks like this :
SOME_CHARACTER_STRING

and I want to extract the characters enclosed between the angle brackets.

This is the command i'm using :
hive CREATE TABLE s(f1 STRING) ROW FORMAT SERDE
'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
 WITH SERDEPROPERTIES ( input.regex = (\\w*), output.regex =
%1$s) STORED AS TEXTFILE;

LOAD works fine, but SELECT * gives me NULL.

I am on hadoop-1.0.4 and hive-0.10.0

Thank you so much for you time.


Warm Regards,
Tariq
cloudfront.blogspot.com


Re: show table throwing strange error

2013-06-30 Thread Mohammad Tariq
Hello all,

  Apologies for being unresponsive. Was busy with some urgent
deliverable.

Inspite of all your help and trying continuously for several days it didn't
work. I tried almost everything, including whatever you guys had suggested.
As a result I had to reconfigure Hive and now it's working perfectly fine.
Still I would love to hear if someone has something to about this.

Thank you so much for your precious time.

Warm Regards,
Tariq
cloudfront.blogspot.com


On Sat, Jun 22, 2013 at 3:28 PM, shashwat shriparv 
dwivedishash...@gmail.com wrote:

 Create hive-site.xml paste following

 and try

 ?xml version=1.0?
 ?xml-stylesheet type=text/xsl href=configuration.xsl?
 !--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the License); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an AS IS BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
 --


 configuration
 property
  namejavax.jdo.option.ConnectionURL/name

 valuejdbc:mysql://localhost:3306/hivemeta?createDatabaseIfNotExist=true/value
   descriptionthe URL of the MySQL database/description

 /property

 property
   namejavax.jdo.option.ConnectionDriverName/name
   valuecom.mysql.jdbc.Driver/value
 /property

 property
   namejavax.jdo.option.ConnectionUserName/name
valueusername/value

 /property

 property
   namejavax.jdo.option.ConnectionPassword/name
   valuepasswore/value
 /property
 /configuration



 *Thanks  Regards*

 ∞
 Shashwat Shriparv



 On Sat, Jun 22, 2013 at 2:26 AM, kulkarni.swar...@gmail.com 
 kulkarni.swar...@gmail.com wrote:

 More often than not in my experience is caused by a malformed
 hive-site.xml(or hive-default.xml). When this happened to me, it was
 because I somehow had tab characters in my hive-site.xml. Try dropping the
 file(s) and recreate with appropriate formatting.


 On Fri, Jun 21, 2013 at 2:17 PM, Sanjay Subramanian 
 sanjay.subraman...@wizecommerce.com wrote:

  Can u stop following services
 hive-server
 hive-meta-store
 Hive-server2 (if u r running that)

  Move current hive.log some place else

  Start following services
  hive-server
 hive-meta-store
 Hive-server2 (if u r running that)


  And check hive.log ?

  Also can u paste the CREATE TABLe script verbatim here…I feel if u are
 using some custom INPUTFORMAT / OUTPUTFORMAT class  that have to be
 specified in quotes…u may have to be *escape* that

  Plus try and add a semicolon to the end of the create table script ...

  sanjay

   From: Mohammad Tariq donta...@gmail.com
 Reply-To: user@hive.apache.org user@hive.apache.org
 Date: Thursday, June 20, 2013 12:52 PM

 To: user user@hive.apache.org
 Subject: Re: show table throwing strange error

   Thank you for looking into it Sanjay. show tables is working fine
 from both Ubuntu and Hive shell. But i'm getting the same error as
 yesterday when i'm running create table, which is :

  line 1:30 character '' not supported here
 line 1:31 character '' not supported here
 line 1:32 character '' not supported here
 line 1:33 character '' not supported here
 line 1:34 character '' not supported here
 line 1:35 character '' not supported here
 line 1:36 character '' not supported here
 line 1:37 character '' not supported here
 line 1:38 character '' not supported here
 line 1:39 character '' not supported here
 line 1:40 character '' not supported here
 line 1:41 character '' not supported here
 line 1:42 character '' not supported here
 .
 .
 .
 .

  Also, I have noticed 1 strange thing. hive.log is totally messed up.
 Looks like logs are getting written in some binary encoding. I have
 attached a snapshot of the same. Any idea?

  Warm Regards,
 Tariq
 cloudfront.blogspot.com


 On Fri, Jun 21, 2013 at 1:03 AM, Sanjay Subramanian 
 sanjay.subraman...@wizecommerce.com wrote:

  Can u try from your ubuntu command prompt
 $ hive -e show tables

   From: Mohammad Tariq donta...@gmail.com
 Reply-To: user@hive.apache.org user@hive.apache.org
 Date: Thursday, June 20, 2013 4:28 AM
 To: user user@hive.apache.org

 Subject: Re: show table throwing strange error

   Thank you for the response ma'am. It didn't help either.

  Warm Regards,
 Tariq
 cloudfront.blogspot.com


 On Thu, Jun 20, 2013 at 8:43 AM, Sunita Arvind 
 sunitarv...@gmail.comwrote:

  Your issue seems familiar. Try logging out of hive session and
 re-login.

  Sunita


 On Wed, Jun 19, 2013 at 8:53 PM, Mohammad