Re: Setting s3 credentials in cloudera

2014-04-21 Thread Kishore kumar 
With the same credentials I am able to download the s3 file to my local
filesystem.


On Tue, Apr 22, 2014 at 11:17 AM, Kishore kumar wrote:

> No, I am running in cli.
>
>
> On Mon, Apr 21, 2014 at 8:43 PM, j.barrett Strausser <
> j.barrett.straus...@gmail.com> wrote:
>
>> You mention cloudera, are you trying to execute the query from HUE?  That
>> requires altering the setting for HUE and not HIVE.
>>
>>
>> On Mon, Apr 21, 2014 at 11:12 AM, j.barrett Strausser <
>> j.barrett.straus...@gmail.com> wrote:
>>
>>> Hope those aren't you actual credentials.
>>>
>>>
>>> On Mon, Apr 21, 2014 at 11:05 AM, Kishore kumar 
>>> wrote:
>>>
 I Edited "Cluster-wide Configuration Safety Valve for core-site.xml"
 in cm, and specified as below, but still the problem is same.

 
 fs.s3.awsAccessKeyId
 AKIAJNIM5P2SASWJPHSA
 

 
 fs.s3.awsSecretAccessKey
 BN1hkKD7JY4LGGNbjxmnFE0ehs12vXmP44GCKV2N
 


 FAILED: Error in metadata:
 MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
 and Secret Access Key must be specified as the username or password
 (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
 fs.s3.awsSecretAccessKey properties (respectively).)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask

 Thanks,
 Kishore.


 On Mon, Apr 21, 2014 at 8:17 PM, Kishore kumar 
 wrote:

> I set the credentials from hive command line, still I am getting the
> error. please help me.
>
> hive> set fs.s3.awsAccessKeyId = x;
>  hive> set fs.s3.awsSecretAccessKey = xxx;
>
> FAILED: Error in metadata:
> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key 
> ID
> and Secret Access Key must be specified as the username or password
> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
> fs.s3.awsSecretAccessKey properties (respectively).)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> Thanks,
> Kishore.
>
>
>
> On Mon, Apr 21, 2014 at 7:33 PM, Kishore kumar 
> wrote:
>
>> Hi Experts,
>>
>> I am trying to create table against my s3 file, I faced the below
>> issue, where to set these credentials in clouderamanager4.8. I got this
>> link (
>> http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
>> after some research but please explain me clearly after edited
>> "Cluster-wide Configuration Safety Valve for core-site.xml" how to
>> specify the values.
>>
>> -- Thanks,
>>
>>
>> *Kishore *
>>
>
>
>
> --
>
>


 --


>>>
>>>
>>> --
>>>
>>>
>>> 
>>>
>>
>
> --
>
>


-- 

*Kishore Kumar*
ITIM

Re: Setting s3 credentials in cloudera

2014-04-21 Thread Kishore kumar 
No, I am running in cli.


On Mon, Apr 21, 2014 at 8:43 PM, j.barrett Strausser <
j.barrett.straus...@gmail.com> wrote:

> You mention cloudera, are you trying to execute the query from HUE?  That
> requires altering the setting for HUE and not HIVE.
>
>
> On Mon, Apr 21, 2014 at 11:12 AM, j.barrett Strausser <
> j.barrett.straus...@gmail.com> wrote:
>
>> Hope those aren't you actual credentials.
>>
>>
>> On Mon, Apr 21, 2014 at 11:05 AM, Kishore kumar wrote:
>>
>>> I Edited "Cluster-wide Configuration Safety Valve for core-site.xml" in
>>> cm, and specified as below, but still the problem is same.
>>>
>>> 
>>> fs.s3.awsAccessKeyId
>>> AKIAJNIM5P2SASWJPHSA
>>> 
>>>
>>> 
>>> fs.s3.awsSecretAccessKey
>>> BN1hkKD7JY4LGGNbjxmnFE0ehs12vXmP44GCKV2N
>>> 
>>>
>>>
>>> FAILED: Error in metadata:
>>> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
>>> and Secret Access Key must be specified as the username or password
>>> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
>>> fs.s3.awsSecretAccessKey properties (respectively).)
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>>
>>> Thanks,
>>> Kishore.
>>>
>>>
>>> On Mon, Apr 21, 2014 at 8:17 PM, Kishore kumar wrote:
>>>
 I set the credentials from hive command line, still I am getting the
 error. please help me.

 hive> set fs.s3.awsAccessKeyId = x;
  hive> set fs.s3.awsSecretAccessKey = xxx;

 FAILED: Error in metadata:
 MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
 and Secret Access Key must be specified as the username or password
 (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
 fs.s3.awsSecretAccessKey properties (respectively).)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask

 Thanks,
 Kishore.



 On Mon, Apr 21, 2014 at 7:33 PM, Kishore kumar 
 wrote:

> Hi Experts,
>
> I am trying to create table against my s3 file, I faced the below
> issue, where to set these credentials in clouderamanager4.8. I got this
> link (
> http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
> after some research but please explain me clearly after edited
> "Cluster-wide Configuration Safety Valve for core-site.xml" how to
> specify the values.
>
> -- Thanks,
>
>
> *Kishore *
>



 --

 *Kishore Kumar*
 ITIM

 Bidstalk - Ingenius Programmatic Platform

 Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741
 135 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter:
 __kishorealajangi
 [image: Inline image 1]

>>>
>>>
>>>
>>> --
>>>
>>> *Kishore Kumar*
>>> ITIM
>>>
>>> Bidstalk - Ingenius Programmatic Platform
>>>
>>> Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
>>> 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter:
>>> __kishorealajangi
>>> [image: Inline image 1]
>>>
>>
>>
>>
>> --
>>
>>
>> https://github.com/bearrito
>> @deepbearrito
>>
>
>
>
> --
>
>
> https://github.com/bearrito
> @deepbearrito
>



-- 

*Kishore Kumar*
ITIM

Bidstalk - Ingenius Programmatic Platform

Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter: __kishorealajangi
[image: Inline image 1]

Re: question about hive sql

2014-04-21 Thread Shengjun Xin 
You need to check the container log for the details


On Tue, Apr 22, 2014 at 10:27 AM, EdwardKing  wrote:

>  I use hive under hadoop 2.2.0, first I start hive
> [hadoop@master sbin]$ hive
> 14/04/21 19:06:32 INFO Configuration.deprecation:
> mapred.input.dir.recursive is deprecated. Instead, use
> mapreduce.input.fileinputformat.input.dir.recursive
> 14/04/21 19:06:32 INFO Configuration.deprecation: mapred.max.split.size is
> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
> 14/04/21 19:06:32 INFO Configuration.deprecation: mapred.min.split.size is
> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
> 14/04/21 19:06:32 INFO Configuration.deprecation:
> mapred.min.split.size.per.rack is deprecated. Instead, use
> mapreduce.input.fileinputformat.split.minsize.per.rack
> 14/04/21 19:06:32 INFO Configuration.deprecation:
> mapred.min.split.size.per.node is deprecated. Instead, use
> mapreduce.input.fileinputformat.split.minsize.per.node
> 14/04/21 19:06:32 INFO Configuration.deprecation: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 14/04/21 19:06:32 INFO Configuration.deprecation:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> 14/04/21 19:06:32 WARN conf.Configuration:
> org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@2128d0:an attempt
> to override final parameter:
> mapreduce.job.end-notification.max.retry.interval;  Ignoring.
> 14/04/21 19:06:32 WARN conf.Configuration:
> org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@2128d0:an attempt
> to override final parameter: mapreduce.job.end-notification.max.attempts;
> Ignoring.
> Logging initialized using configuration in
> jar:file:/home/software/hive-0.11.0/lib/hive-common-0.11.0.jar!/hive-log4j.properties
> Hive history
> file=/tmp/hadoop/hive_job_log_hadoop_7623@master_201404211906_2069310090.txt
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/software/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/software/hive-0.11.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Then I creat a table
> hive> create table test(id STRING);
> OK
> Time taken: 17.277 seconds
> Then  I insert some date into test
> hive> load data inpath 'a.txt' overwrite into table test;
> Loading data to table default.test
> rmr: DEPRECATED: Please use 'rm -r' instead.
> Deleted /user/hive/warehouse/test
> Table default.test stats: [num_partitions: 0, num_files: 1, num_rows: 0,
> total_size: 19, raw_data_size: 0]
> OK
> Time taken: 1.855 seconds
>
> hive> select * from test;
> OK
> China
> US
> Australia
> Time taken: 0.526 seconds, Fetched: 3 row(s)
> Now I use count command, I expected the result value is 3, but it runs
> failure!  Why? Where is wrong? I am puzzled with it for several days.
> Anyone could tell me how to correct it?
> hive> select count(*) from test;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
>   set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
>   set hive.exec.reducers.max=
> In order to set a constant number of reducers:
>   set mapred.reduce.tasks=
> Starting Job = job_1398132272370_0001, Tracking URL =
> http://master:8088/proxy/application_1398132272370_0001/
> Kill Command = /home/software/hadoop-2.2.0/bin/hadoop job  -kill
> job_1398132272370_0001
> Hadoop job information for Stage-1: number of mappers: 0; number of
> reducers: 0
> 2014-04-21 19:15:56,684 Stage-1 map = 0%,  reduce = 0%
> Ended Job = job_1398132272370_0001 with errors
> Error during job, obtaining debugging information...
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0:  HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
> hive>
>
> Error information under
> http://172.11.12.6:8088/cluster/app/application_1398132272370_0001
> User:  hadoop
> Name:  select count(*) from test(Stage-1)
> Application Type:  MAPREDUCE
> State:  FAILED
> FinalStatus:  FAILED
> Started:  21-Apr-2014 19:14:55
> Elapsed:  57sec
> Tracking URL:  History
> Diagnostics:
> Application application_1398132272370_0001 failed 2 times due to AM
> Container for appattempt_1398132272370_0001_02 exited with exitCode: 1
> due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.jav

Re: Hive 0.13.0 - IndexOutOfBounds Exception

2014-04-21 Thread Prasanth Jayachandran 
Hi Bryan

Can you provide more information about the input and output tables? Schema? 
Partitioning and bucketing information? Explain plan of your insert query? 

These information will help to diagnose the issue.  

Thanks
Prasanth

Sent from my iPhone

> On Apr 21, 2014, at 7:00 PM, Bryan Jeffrey  wrote:
> 
> Hello.
> 
> I am running Hadoop 2.4.0 and Hive 0.13.0.  I am encountering the following 
> error when converting a text table to ORC via the following command:
> 
> Error:
> 
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row { - Removed -}
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row { - Removed -}
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:327)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
> ... 9 more
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at 
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:121)
> at 
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at 
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:283)
> at 
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:268)
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:251)
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:264)
> ... 15 more
> 
> Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> 
> There are a number of older issues associated with IndexOutOfBounds errors 
> within the serde, but nothing that appears to specifically match this error.  
> This occurs with all tables (including those consisting of exclusively 
> integers).  Any thoughts?
> 
> Regards,
> 
> Bryan Jeffrey

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

question about hive sql

2014-04-21 Thread EdwardKing 
I use hive under hadoop 2.2.0, first I start hive
[hadoop@master sbin]$ hive
14/04/21 19:06:32 INFO Configuration.deprecation: mapred.input.dir.recursive is 
deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/04/21 19:06:32 INFO Configuration.deprecation: mapred.max.split.size is 
deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/04/21 19:06:32 INFO Configuration.deprecation: mapred.min.split.size is 
deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/04/21 19:06:32 INFO Configuration.deprecation: 
mapred.min.split.size.per.rack is deprecated. Instead, use 
mapreduce.input.fileinputformat.split.minsize.per.rack
14/04/21 19:06:32 INFO Configuration.deprecation: 
mapred.min.split.size.per.node is deprecated. Instead, use 
mapreduce.input.fileinputformat.split.minsize.per.node
14/04/21 19:06:32 INFO Configuration.deprecation: mapred.reduce.tasks is 
deprecated. Instead, use mapreduce.job.reduces
14/04/21 19:06:32 INFO Configuration.deprecation: 
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use 
mapreduce.reduce.speculative
14/04/21 19:06:32 WARN conf.Configuration: 
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@2128d0:an attempt to 
override final parameter: mapreduce.job.end-notification.max.retry.interval;  
Ignoring.
14/04/21 19:06:32 WARN conf.Configuration: 
org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@2128d0:an attempt to 
override final parameter: mapreduce.job.end-notification.max.attempts;  
Ignoring.
Logging initialized using configuration in 
jar:file:/home/software/hive-0.11.0/lib/hive-common-0.11.0.jar!/hive-log4j.properties
Hive history 
file=/tmp/hadoop/hive_job_log_hadoop_7623@master_201404211906_2069310090.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/software/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/software/hive-0.11.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Then I creat a table
hive> create table test(id STRING);
OK
Time taken: 17.277 seconds

Then  I insert some date into test  
hive> load data inpath 'a.txt' overwrite into table test;
Loading data to table default.test
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted /user/hive/warehouse/test
Table default.test stats: [num_partitions: 0, num_files: 1, num_rows: 0, 
total_size: 19, raw_data_size: 0]
OK
Time taken: 1.855 seconds

hive> select * from test;
OK
China
US
Australia
Time taken: 0.526 seconds, Fetched: 3 row(s)

Now I use count command, I expected the result value is 3, but it runs failure! 
 Why? Where is wrong? I am puzzled with it for several days. Anyone could tell 
me how to correct it?
hive> select count(*) from test;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapred.reduce.tasks=
Starting Job = job_1398132272370_0001, Tracking URL = 
http://master:8088/proxy/application_1398132272370_0001/
Kill Command = /home/software/hadoop-2.2.0/bin/hadoop job  -kill 
job_1398132272370_0001
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2014-04-21 19:15:56,684 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_1398132272370_0001 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched: 
Job 0:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive> 


Error information under 
http://172.11.12.6:8088/cluster/app/application_1398132272370_0001
User:  hadoop
Name:  select count(*) from test(Stage-1)
Application Type:  MAPREDUCE
State:  FAILED
FinalStatus:  FAILED
Started:  21-Apr-2014 19:14:55
Elapsed:  57sec
Tracking URL:  History
Diagnostics:  
Application application_1398132272370_0001 failed 2 times due to AM Container 
for appattempt_1398132272370_0001_02 exited with exitCode: 1 due to: 
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at 
org.apache.hadoop.yarn.server.n

Hive 0.13.0 - IndexOutOfBounds Exception

2014-04-21 Thread Bryan Jeffrey 
Hello.

I am running Hadoop 2.4.0 and Hive 0.13.0.  I am encountering the following
error when converting a text table to ORC via the following command:

Error:

Diagnostic Messages for this Task:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row { - Removed -}
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row { - Removed -}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:327)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
... 9 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:121)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:283)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:268)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:251)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:264)
... 15 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

There are a number of older issues associated with IndexOutOfBounds errors
within the serde, but nothing that appears to specifically match this
error.  This occurs with all tables (including those consisting of
exclusively integers).  Any thoughts?

Regards,

Bryan Jeffrey

Re: [ANNOUNCE] Apache Hive 0.13.0 Released

2014-04-21 Thread Harish Butani 
The link to the Release Notes is wrong.
Thanks Szehon Ho for pointing this out.
The correct link is:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12324986&styleName=Text&projectId=12310843


On Mon, Apr 21, 2014 at 4:23 PM, Thejas Nair  wrote:

> Thanks to Harish for all the hard work managing and getting the release
> out!
>
> This is great news! This is a significant release in hive! This has
> more than twice the number of jiras included (see release note link),
> compared to 0.12, and earlier releases which were also out after a
> similar gap of 5-6 months. It shows tremendous growth in hive
> community activity!
>
> hive 0.13 - 1081
> hive 0.12 - 439
> hive 0.11 - 374
>
> -Thejas
>
> On Mon, Apr 21, 2014 at 3:17 PM, Harish Butani 
> wrote:
> > The Apache Hive team is proud to announce the the release of Apache
> > Hive version 0.13.0.
> >
> > The Apache Hive (TM) data warehouse software facilitates querying and
> > managing large datasets residing in distributed storage. Built on top
> > of Apache Hadoop (TM), it provides:
> >
> > * Tools to enable easy data extract/transform/load (ETL)
> >
> > * A mechanism to impose structure on a variety of data formats
> >
> > * Access to files stored either directly in Apache HDFS (TM) or in other
> >   data storage systems such as Apache HBase (TM)
> >
> > * Query execution via MapReduce
> >
> > For Hive release details and downloads, please visit:
> > http://www.apache.org/dyn/closer.cgi/hive/
> >
> > Hive 0.13.0 Release Notes are available here:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12324312&styleName=Text&projectId=12310843
> >
> > We would like to thank the many contributors who made this release
> > possible.
> >
> > Regards,
> >
> > The Apache Hive Team
> >
> > PS: we are having technical difficulty updating the website. Will resolve
> > this shortly.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [ANNOUNCE] Apache Hive 0.13.0 Released

2014-04-21 Thread Thejas Nair 
Thanks to Harish for all the hard work managing and getting the release out!

This is great news! This is a significant release in hive! This has
more than twice the number of jiras included (see release note link),
compared to 0.12, and earlier releases which were also out after a
similar gap of 5-6 months. It shows tremendous growth in hive
community activity!

hive 0.13 - 1081
hive 0.12 - 439
hive 0.11 - 374

-Thejas

On Mon, Apr 21, 2014 at 3:17 PM, Harish Butani  wrote:
> The Apache Hive team is proud to announce the the release of Apache
> Hive version 0.13.0.
>
> The Apache Hive (TM) data warehouse software facilitates querying and
> managing large datasets residing in distributed storage. Built on top
> of Apache Hadoop (TM), it provides:
>
> * Tools to enable easy data extract/transform/load (ETL)
>
> * A mechanism to impose structure on a variety of data formats
>
> * Access to files stored either directly in Apache HDFS (TM) or in other
>   data storage systems such as Apache HBase (TM)
>
> * Query execution via MapReduce
>
> For Hive release details and downloads, please visit:
> http://www.apache.org/dyn/closer.cgi/hive/
>
> Hive 0.13.0 Release Notes are available here:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12324312&styleName=Text&projectId=12310843
>
> We would like to thank the many contributors who made this release
> possible.
>
> Regards,
>
> The Apache Hive Team
>
> PS: we are having technical difficulty updating the website. Will resolve
> this shortly.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Meta data tables - Hive

2014-04-21 Thread Alan Gates 
Hive does not have a traditional SQL information schema.  Instead it uses MySQL 
style show/describe.  So it has show tables, etc.  See 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show

Alan.

On Apr 21, 2014, at 7:10 AM, Ravi Prasad  wrote:

> Hi all,
>   In Hive do we have any meta data tables where I can see all the table's  / 
> view's / Indexes  information.
>  
> For example,  in Oracle,   we have USER_TABLES  meta data table  to know 
> about all the tables available in the user.
> USER_VIEWS to know about all the view information available in the user.
>  
> Like this , do we have any meta data tables ( data dictionary tables)  
> avilable in Hive to know about the  Hive tables / views information.
> 
> 
> -- 
> Regards,
> RAVI PRASAD. T


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Executing Hive Queries in Parallel

2014-04-21 Thread Subramanian, Sanjay (HQP) 
Hey

Instead of going into HIVE CLI
I would propose 2 ways

NOHUP
nohup hive -f path/to/query/file/hive1.hql >> ./hive1.hql_`date 
+%Y-%m-%d-%H–%M–%S`.log 2>&1
nohup hive -f path/to/query/file/hive2.hql >> ./hive2.hql_`date 
+%Y-%m-%d-%H–%M–%S`.log 2>&1
nohup hive -f path/to/query/file/hive3.hql >> ./hive3.hql_`date 
+%Y-%m-%d-%H–%M–%S`.log 2>&1
nohup hive -f path/to/query/file/hive4.hql >> ./hive4.hql_`date 
+%Y-%m-%d-%H–%M–%S`.log 2>&1
nohup hive -f path/to/query/file/hive5.hql >> ./hive5.hql_`date 
+%Y-%m-%d-%H–%M–%S`.log 2>&1

Each statement above will launch MR jobs on your cluster and depending on the 
cluster configs the jobs will run parallelly
Scheduling jobs on the MR cluster is independent of Hive

SCREEN sessions

  *   Create a Screen session
 *   screen  –S  hive_query1
 *   U r inside the screen session hive_query1
*   hive -f path/to/query/file/hive1.hql
 *   Ctrl A D
*   U detach from a screen session
  *   Repeat for each hive query u want to run
 *   I.e. Say 5 screen sessions, each running a have query
  *   To display screen session active
 *   screen -x
  *   To attach to a screen session
 *   screen  -x hive_query1

Thanks
Warm Regards

Sanjay

From: saurabh mailto:mpp.databa...@gmail.com>>
Reply-To: "user@hive.apache.org" 
mailto:user@hive.apache.org>>
Date: Monday, April 21, 2014 at 1:53 PM
To: "user@hive.apache.org" 
mailto:user@hive.apache.org>>
Subject: Executing Hive Queries in Parallel


Hi,
I need some inputs to execute hive queries in parallel. I tried doing this 
using CLI (by opening multiple ssh connection) and executed 4 HQL's; it was 
observed that the queries are getting executed sequentially. All the FOUR 
queries got submitted however while the first one was in execution mode the 
other were in pending state. I was performing this activity on the EMR running 
on Batch mode hence didn't able to dig into the logs.

The hive CLI uses native hive connection which by default uses the FIFO 
scheduler.  This might be one of the reason for the queries getting executed in 
sequence.

I also observed that when multiple queries are executed using multiple HUE 
sessions, it provides the parallel execution functionality. Can you please 
suggest how the functionality of HUE can be replicated using CLI?

I am aware of beeswax client however i am not sure how this can be used during 
EMR- batch mode processing.

Thanks in advance for going through this. Kindly let me know your thoughts on 
the same.

Executing Hive Queries in Parallel

2014-04-21 Thread saurabh 
Hi,
I need some inputs to execute hive queries in parallel. I tried doing this
using CLI (by opening multiple ssh connection) and executed 4 HQL's; it was
observed that the queries are getting executed sequentially. All the FOUR
queries got submitted however while the first one was in execution mode the
other were in pending state. I was performing this activity on the EMR
running on Batch mode hence didn't able to dig into the logs.

The hive CLI uses native hive connection which by default uses the FIFO
scheduler.  This might be one of the reason for the queries getting
executed in sequence.

I also observed that when multiple queries are executed using multiple HUE
sessions, it provides the parallel execution functionality. Can you please
suggest how the functionality of HUE can be replicated using CLI?

I am aware of beeswax client however i am not sure how this can be used
during EMR- batch mode processing.

Thanks in advance for going through this. Kindly let me know your thoughts
on the same.

"create table as" fails with error

2014-04-21 Thread Subramanian, Sanjay (HQP) 
Hey guys

THIS QUERY FAILS

create table olena.temp8 as select * from olena.temp7 group by person_id, 
level, technical, business, liberalarts, lifesciences, other, school_name, 
degree_complete_flag

FAILED: Error in metadata: InvalidObjectException(message:temp8 is not a valid 
object name)

14/04/21 12:02:32 ERROR exec.Task: FAILED: Error in metadata: 
InvalidObjectException(message:temp8 is not a valid object name)

org.apache.hadoop.hive.ql.metadata.HiveException: 
InvalidObjectException(message:temp8 is not a valid object name)

at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:582)

at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3719)

at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:254)

at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)

at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66)

at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1383)

at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1169)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)

at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)

at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:706)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Caused by: InvalidObjectException(message:temp8 is not a valid object name)

at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result$create_table_resultStandardScheme.read(ThriftHiveMetastore.java:20140)

at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result$create_table_resultStandardScheme.read(ThriftHiveMetastore.java:20117)

at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result.read(ThriftHiveMetastore.java:20043)

at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)

at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table(ThriftHiveMetastore.java:732)

at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table(ThriftHiveMetastore.java:719)

at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:432)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)

at $Proxy9.createTable(Unknown Source)

at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:576)

... 19 more



THIS QUERY SUCCEEDS (Removed “CREATE TABLE”)

select person_id, level, technical, business, liberalarts, lifesciences, other, 
school_name, degree_complete_flag from olena.temp7 group by person_id, level, 
technical, business, liberalarts, lifesciences, other, school_name, 
degree_complete_flag



THIS QUERY SUCCEEDS (Instead of select * , I did select col1,col2,col3…)

create table olena.temp9ish as select person_id, level, technical, business, 
liberalarts, lifesciences, other, school_name, degree_complete_flag from 
olena.temp7 group by person_id, level, technical, business, liberalarts, 
lifesciences, other, school_name, degree_complete_flag



Thanks
Warm Regards

Sanjay

Re: Help - Hadoop jar null org.apache.hadoop.hive.ql.exec.ExecDriver

2014-04-21 Thread Chinna Rao Lalam 
Hi,

Check may be hive-exec.jar is corrupted.


Hope It Helps,
Chinna Rao Lalam


On Sat, Apr 19, 2014 at 2:38 AM, Abhishek Girish  wrote:

> Hello,
>
> I am hitting an error while executing a Hive job inside MapReduce:
>
> *Code snippet:*
> 
> String select1 = "SELECT a FROM abc";
>
> driver.run(select1);
>
> ...
>
> *Error:*
> INFO exec.ExecDriver: Executing: /usr/local/hadoop-0.20.2/bin/*hadoop jar
> null* org.apache.hadoop.hive.ql.exec.ExecDriver  -plan
> file:/tmp/hadoop/hive_2014-04-17_19-46-20_938_6272043157613581856/-local-10003/plan.xml
>   -jobconffile
> file:/tmp/hadoop/hive_2014-04-17_19-46-20_938_6272043157613581856/-local-10002/jobconf.xml
>
> Exception in thread "main" java.io.IOException: Error opening job jar: null
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
>
> Caused by: java.util.zip.ZipException: error in opening zip file
>
> at java.util.zip.ZipFile.open(Native Method)
>
> at java.util.zip.ZipFile.(ZipFile.java:128)
>
> at java.util.jar.JarFile.(JarFile.java:136)
>
> at java.util.jar.JarFile.(JarFile.java:73)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
> ...
>
> ...
>
> 14/04/17 19:46:21 ERROR exec.ExecDriver: Execution failed with exit
> status: 1
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
>
> 14/04/17 19:46:21 ERROR ql.Driver: FAILED: Execution Error, return code 1
> from org.apache.hadoop.hive.ql.exec.MapRedTask
> Can someone please let me know what could be wrong? I was unable to find
> any details online.
>
> -Abhishek
>



-- 
Hope It Helps,
Chinna

Re: All Hive JDBC Queries Fail with Same Error: “Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTa

2014-04-21 Thread Chinna Rao Lalam 
Hi,

Here the MR job is failed, Check why the MR job is failed (From the job
logs) ..


Hope It Helps,
Chinna Rao Lalam


On Fri, Apr 18, 2014 at 9:53 PM, Vince George (vincgeor)  wrote:

>  We have just configured a new Hive JDBC client with an upgraded support
> for per user Kerberos authentication. The JDBC file set applied which works
> fine in an alternate setup is:
>
>
>
> All Hive queries fail with a similar attached stack trace that at the
> bottom is a Hive error “Caused by: java.sql.SQLException: Error while
> processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask”.
>
>
>
> Any ideas as to what is the issue?
>
>
>
> Regards,
>
> 
>
> *Vince George | Solutions Manager - Consultant*
>
> Mobile: (201) 519-3777
>
> vincg...@cisco.com | www.cisco.com
>
>
>
> [image: Description: cid:F9A08884-B7D9-4EB1-B808-54E75B51A1ED]
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/index.html
>
>
>
>
>



-- 
Hope It Helps,
Chinna

Re: Setting s3 credentials in cloudera

2014-04-21 Thread j.barrett Strausser 
You mention cloudera, are you trying to execute the query from HUE?  That
requires altering the setting for HUE and not HIVE.


On Mon, Apr 21, 2014 at 11:12 AM, j.barrett Strausser <
j.barrett.straus...@gmail.com> wrote:

> Hope those aren't you actual credentials.
>
>
> On Mon, Apr 21, 2014 at 11:05 AM, Kishore kumar wrote:
>
>> I Edited "Cluster-wide Configuration Safety Valve for core-site.xml" in
>> cm, and specified as below, but still the problem is same.
>>
>> 
>> fs.s3.awsAccessKeyId
>> AKIAJNIM5P2SASWJPHSA
>> 
>>
>> 
>> fs.s3.awsSecretAccessKey
>> BN1hkKD7JY4LGGNbjxmnFE0ehs12vXmP44GCKV2N
>> 
>>
>>
>> FAILED: Error in metadata:
>> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
>> and Secret Access Key must be specified as the username or password
>> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
>> fs.s3.awsSecretAccessKey properties (respectively).)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>>
>> Thanks,
>> Kishore.
>>
>>
>> On Mon, Apr 21, 2014 at 8:17 PM, Kishore kumar wrote:
>>
>>> I set the credentials from hive command line, still I am getting the
>>> error. please help me.
>>>
>>> hive> set fs.s3.awsAccessKeyId = x;
>>>  hive> set fs.s3.awsSecretAccessKey = xxx;
>>>
>>> FAILED: Error in metadata:
>>> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
>>> and Secret Access Key must be specified as the username or password
>>> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
>>> fs.s3.awsSecretAccessKey properties (respectively).)
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>>
>>> Thanks,
>>> Kishore.
>>>
>>>
>>>
>>> On Mon, Apr 21, 2014 at 7:33 PM, Kishore kumar wrote:
>>>
 Hi Experts,

 I am trying to create table against my s3 file, I faced the below
 issue, where to set these credentials in clouderamanager4.8. I got this
 link (
 http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
 after some research but please explain me clearly after edited
 "Cluster-wide Configuration Safety Valve for core-site.xml" how to
 specify the values.

 -- Thanks,


 *Kishore *

>>>
>>>
>>>
>>> --
>>>
>>> *Kishore Kumar*
>>> ITIM
>>>
>>> Bidstalk - Ingenius Programmatic Platform
>>>
>>> Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
>>> 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter:
>>> __kishorealajangi
>>> [image: Inline image 1]
>>>
>>
>>
>>
>> --
>>
>> *Kishore Kumar*
>> ITIM
>>
>> Bidstalk - Ingenius Programmatic Platform
>>
>> Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
>> 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter:
>> __kishorealajangi
>> [image: Inline image 1]
>>
>
>
>
> --
>
>
> https://github.com/bearrito
> @deepbearrito
>



-- 


https://github.com/bearrito
@deepbearrito

Re: Setting s3 credentials in cloudera

2014-04-21 Thread j.barrett Strausser 
Hope those aren't you actual credentials.


On Mon, Apr 21, 2014 at 11:05 AM, Kishore kumar wrote:

> I Edited "Cluster-wide Configuration Safety Valve for core-site.xml" in
> cm, and specified as below, but still the problem is same.
>
> 
> fs.s3.awsAccessKeyId
> AKIAJNIM5P2SASWJPHSA
> 
>
> 
> fs.s3.awsSecretAccessKey
> BN1hkKD7JY4LGGNbjxmnFE0ehs12vXmP44GCKV2N
> 
>
>
> FAILED: Error in metadata:
> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
> and Secret Access Key must be specified as the username or password
> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
> fs.s3.awsSecretAccessKey properties (respectively).)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> Thanks,
> Kishore.
>
>
> On Mon, Apr 21, 2014 at 8:17 PM, Kishore kumar wrote:
>
>> I set the credentials from hive command line, still I am getting the
>> error. please help me.
>>
>> hive> set fs.s3.awsAccessKeyId = x;
>>  hive> set fs.s3.awsSecretAccessKey = xxx;
>>
>> FAILED: Error in metadata:
>> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
>> and Secret Access Key must be specified as the username or password
>> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
>> fs.s3.awsSecretAccessKey properties (respectively).)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>>
>> Thanks,
>> Kishore.
>>
>>
>>
>> On Mon, Apr 21, 2014 at 7:33 PM, Kishore kumar wrote:
>>
>>> Hi Experts,
>>>
>>> I am trying to create table against my s3 file, I faced the below issue,
>>> where to set these credentials in clouderamanager4.8. I got this link (
>>> http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
>>> after some research but please explain me clearly after edited
>>> "Cluster-wide Configuration Safety Valve for core-site.xml" how to
>>> specify the values.
>>>
>>> -- Thanks,
>>>
>>>
>>> *Kishore *
>>>
>>
>>
>>
>> --
>>
>> *Kishore Kumar*
>> ITIM
>>
>> Bidstalk - Ingenius Programmatic Platform
>>
>> Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
>> 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter:
>> __kishorealajangi
>> [image: Inline image 1]
>>
>
>
>
> --
>
> *Kishore Kumar*
> ITIM
>
> Bidstalk - Ingenius Programmatic Platform
>
> Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
> 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter:
> __kishorealajangi
> [image: Inline image 1]
>



-- 


https://github.com/bearrito
@deepbearrito

Re: Setting s3 credentials in cloudera

2014-04-21 Thread Kishore kumar 
I Edited "Cluster-wide Configuration Safety Valve for core-site.xml" in cm,
and specified as below, but still the problem is same.


fs.s3.awsAccessKeyId
AKIAJNIM5P2SASWJPHSA



fs.s3.awsSecretAccessKey
BN1hkKD7JY4LGGNbjxmnFE0ehs12vXmP44GCKV2N


FAILED: Error in metadata:
MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
and Secret Access Key must be specified as the username or password
(respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
fs.s3.awsSecretAccessKey properties (respectively).)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask

Thanks,
Kishore.


On Mon, Apr 21, 2014 at 8:17 PM, Kishore kumar wrote:

> I set the credentials from hive command line, still I am getting the
> error. please help me.
>
> hive> set fs.s3.awsAccessKeyId = x;
>  hive> set fs.s3.awsSecretAccessKey = xxx;
>
> FAILED: Error in metadata:
> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
> and Secret Access Key must be specified as the username or password
> (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
> fs.s3.awsSecretAccessKey properties (respectively).)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> Thanks,
> Kishore.
>
>
>
> On Mon, Apr 21, 2014 at 7:33 PM, Kishore kumar wrote:
>
>> Hi Experts,
>>
>> I am trying to create table against my s3 file, I faced the below issue,
>> where to set these credentials in clouderamanager4.8. I got this link (
>> http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
>> after some research but please explain me clearly after edited
>> "Cluster-wide Configuration Safety Valve for core-site.xml" how to
>> specify the values.
>>
>> -- Thanks,
>>
>>
>> *Kishore *
>>
>
>
>
> --
>
> *Kishore Kumar*
> ITIM
>
> Bidstalk - Ingenius Programmatic Platform
>
> Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
> 8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter: __kishorealajangi
> [image: Inline image 1]
>



-- 

*Kishore Kumar*
ITIM

Bidstalk - Ingenius Programmatic Platform

Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter: __kishorealajangi
[image: Inline image 1]

Re: Setting s3 credentials in cloudera

2014-04-21 Thread Kishore kumar 
I set the credentials from hive command line, still I am getting the error.
please help me.

hive> set fs.s3.awsAccessKeyId = x;
 hive> set fs.s3.awsSecretAccessKey = xxx;

FAILED: Error in metadata:
MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
and Secret Access Key must be specified as the username or password
(respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or
fs.s3.awsSecretAccessKey properties (respectively).)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask

Thanks,
Kishore.



On Mon, Apr 21, 2014 at 7:33 PM, Kishore kumar wrote:

> Hi Experts,
>
> I am trying to create table against my s3 file, I faced the below issue,
> where to set these credentials in clouderamanager4.8. I got this link (
> http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
> after some research but please explain me clearly after edited
> "Cluster-wide Configuration Safety Valve for core-site.xml" how to
> specify the values.
>
> -- Thanks,
>
>
> *Kishore *
>



-- 

*Kishore Kumar*
ITIM

Bidstalk - Ingenius Programmatic Platform

Email: kish...@techdigita.in| Tel: +1 415 423 8230  | Cell: +91 741 135
8658 | skype: kishore.alajangi | YM: kk_asn2004 | Twitter: __kishorealajangi
[image: Inline image 1]

Meta data tables - Hive

2014-04-21 Thread Ravi Prasad 
Hi all,
  In Hive do we have any meta data tables where I can see all the table's
/ view's / Indexes  information.

For example,  in Oracle,   we have *USER_TABLES*  meta data table  to know
about all the tables available in the user.
*USER_VIEWS* to know about all the view information available in the user.

Like this , do we have any meta data tables ( data dictionary tables)
 avilable in *Hive* to know about the  Hive tables / views information.


-- 
Regards,
RAVI PRASAD. T

Setting s3 credentials in cloudera

2014-04-21 Thread Kishore kumar 
Hi Experts,

I am trying to create table against my s3 file, I faced the below issue,
where to set these credentials in clouderamanager4.8. I got this link (
http://community.cloudera.com/t5/Cloudera-Manager-Installation/AWS-Access-Key-ID-and-Secret-Access-Key-must-be-specified-as-the/td-p/495)
after some research but please explain me clearly after edited
"Cluster-wide Configuration Safety Valve for core-site.xml" how to specify
the values.

-- Thanks,


*Kishore *

Analyzing data resides on s3 from local hadoop cluster

2014-04-21 Thread Kishore kumar 
Hi Experts,

We are running four node cluster which is installed cdh4.5 with cm4.8, We
have large size files in zip format in s3, we want to analyze that files
for every hour in hive, which is the best way to do that, please help me
with examples or with any reference links.

-- Thanks,


*Kishore *

Analyzing data resides on s3 from local hadoop cluster

2014-04-21 Thread Kishore kumar 
Hi Experts,

After I changed the column names in hive table,m the result showing all
null values with new column names, if i query with select * from table
giving the actual values result, what could be the problem please explain
what should i do now, help me.

-- Thanks,


*Kishore *