Re: [ANNOUNCE] New Hive PMC Member - Sergey Shelukhin
Congratulations and keep up the great work! +Vinod On Feb 25, 2015, at 8:43 AM, Carl Steinbach c...@apache.org wrote: I am pleased to announce that Sergey Shelukhin has been elected to the Hive Project Management Committee. Please join me in congratulating Sergey! Thanks. - Carl
Intermittent BindException during long MR jobs
Hi, we occasionally run into a BindException causing long running jobs to occasionally fail. The stacktrace is below. Any ideas what this could be caused by? Cheers, Krishna Stacktrace: 379969 [Thread-980] ERROR org.apache.hadoop.hive.ql.exec.Task - Job Submission failed with exception 'java.net.BindException(Problem binding to [back10/10.4.2.10:0] java.net.BindException: Cann ot assign requested address; For more details see: http://wiki.apache.org/hadoop/BindException)' java.net.BindException: Problem binding to [back10/10.4.2.10:0] java.net.BindException: Cannot assign requested address; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:718) at org.apache.hadoop.ipc.Client.call(Client.java:1242) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) at com.sun.proxy.$Proxy10.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:193) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at com.sun.proxy.$Proxy11.create(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.init(DFSOutputStream.java:1376) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1395) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1255) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1212) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:276) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:265) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:888) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:869) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:768) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:757) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:558) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createFile(JobSplitWriter.java:96) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:85) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:517) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:487) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:369) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1286) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1283) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1283) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:448) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:56)
programmatic filter
Hello I would like to generate predicates in the ExprNodeDesc form, as required by the HiveStoragePredicateHandler.decomposePredicate https://github.com/apache/hive/blob/b8250ac2f30539f6b23ce80a20a9e338d3d31458/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java#L58 API. The goal would be to break encapsulation as little as possible. Here is the context: I have a map-reduce job that needs to read data from a hive table Ideally I'd have my MR job (my choice of mapper, reducer, but also partitioner and output format) integrated in a hive plan. Maybe some day... Alas, the current solution interacts with HiveMetastoreClient and figures out the Partitions, the StorageDescriptor-specified input formats and SerDes. Long story short, I get my data out of object inspectors. Now I would like to take advantage of predicate push down to optimize filtering data. I need to generate ExprNodeDesc objects for the HiveStoragePredicateHandler.decomposePredicate API, and optionally to make a ExprNodeEvaluator. Any advice, tips, or hints are most welcome thank you Gabriel Balan -- The statements and opinions expressed here are my own and do not necessarily represent those of Oracle Corporation.
Re: column selection for insert
Specifying which columns in the target table to insert into (emp in your case) is not supported in 0.14. You have to insert into all the columns in the table and they have to match the order of items in the select or values clause. This will be supported in 1.2. Alan. Srinivas Thunga mailto:srinivas.thu...@gmail.com February 26, 2015 at 4:34 Hi, I am using Hive 0.14 Can any one help me on to execute the below type query insert into table emp (eid,ename) select * from dept where emp.id http://emp.id = dept.id http://dept.id /Thanks Regards,/ / / /Srinivas T/
Re: Reading tables will lock entire database in Hive 0.13
If you are using DummyTxnManager, please see HIVE-9199 which should resolve the issue. Thanks Chaoyu On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates alanfga...@gmail.com wrote: What lock manager are you using? Alan. Jim Green openkbi...@gmail.com February 25, 2015 at 17:24 Hi Team, Here is old thread about this issue: http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive I met the same issue that when reading one table in session A, we can not create a table in the same database in session B. Basically it causes database level lock in Hive 0.13 when hive.support.concurrency=true. Minimum reproduce: --Session A: create table passwords (col0 string, col1 string, col2 string, col3 string, col4 string, col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY :; load data local inpath /etc/passwd into table passwords; select count(*) from passwords a,passwords b, passwords c, passwords d where a.col0=b.col0 and a.col0=c.col0 and a.col0=d.col0; --Session B: hive create table test(id int); conflicting lock present for default mode EXCLUSIVE *Is this fixed or any jira related?* -- Thanks, www.openkb.info (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
column selection for insert
Hi, I am using Hive 0.14 Can any one help me on to execute the below type query insert into table emp (eid,ename) select * from dept where emp.id = dept.id *Thanks Regards,* *Srinivas T*
Re: [ANNOUNCE] New Hive PMC Member - Sergey Shelukhin
Congrats Sergey! On Thu, Feb 26, 2015 at 9:22 AM, Nick Dimiduk ndimi...@gmail.com wrote: Nice work Sergey! On Wednesday, February 25, 2015, Carl Steinbach c...@apache.org wrote: I am pleased to announce that Sergey Shelukhin has been elected to the Hive Project Management Committee. Please join me in congratulating Sergey! Thanks. - Carl
Re: [ANNOUNCE] New Hive PMC Member - Sergey Shelukhin
Nice work Sergey! On Wednesday, February 25, 2015, Carl Steinbach c...@apache.org wrote: I am pleased to announce that Sergey Shelukhin has been elected to the Hive Project Management Committee. Please join me in congratulating Sergey! Thanks. - Carl
Re: [ANNOUNCE] New Hive PMC Member - Sergey Shelukhin
Well done Sergey! From: Jimmy Xiang jxi...@cloudera.com Sent: Thursday, February 26, 2015 9:26 AM To: user@hive.apache.org Cc: d...@hive.apache.org; ser...@apache.org Subject: Re: [ANNOUNCE] New Hive PMC Member - Sergey Shelukhin Congrats Sergey! On Thu, Feb 26, 2015 at 9:22 AM, Nick Dimiduk ndimi...@gmail.com wrote: Nice work Sergey! On Wednesday, February 25, 2015, Carl Steinbach c...@apache.org wrote: I am pleased to announce that Sergey Shelukhin has been elected to the Hive Project Management Committee. Please join me in congratulating Sergey! Thanks. - Carl
Re: Reading tables will lock entire database in Hive 0.13
Hi Chaoyu, Thanks for the information. Actually by default, if we set hive.support.concurrency=true in Hive 0.13: hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager Hive-9199 mentioned it is fixed in 1.1, do you know if it will be fixed under DummyTxnManager? Or does it mean it is fixed using new DbTxnManager? Thanks. On Thu, Feb 26, 2015 at 4:51 AM, Chaoyu Tang ct...@cloudera.com wrote: If you are using DummyTxnManager, please see HIVE-9199 which should resolve the issue. Thanks Chaoyu On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates alanfga...@gmail.com wrote: What lock manager are you using? Alan. Jim Green openkbi...@gmail.com February 25, 2015 at 17:24 Hi Team, Here is old thread about this issue: http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive I met the same issue that when reading one table in session A, we can not create a table in the same database in session B. Basically it causes database level lock in Hive 0.13 when hive.support.concurrency=true. Minimum reproduce: --Session A: create table passwords (col0 string, col1 string, col2 string, col3 string, col4 string, col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY :; load data local inpath /etc/passwd into table passwords; select count(*) from passwords a,passwords b, passwords c, passwords d where a.col0=b.col0 and a.col0=c.col0 and a.col0=d.col0; --Session B: hive create table test(id int); conflicting lock present for default mode EXCLUSIVE *Is this fixed or any jira related?* -- Thanks, www.openkb.info (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool) -- Thanks, www.openkb.info (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
RE: GSS initiate failed exception
I have below code. This does the renewal right?. What I am not getting is, How
drivermanager know this config/user to use when establishing connection?
UserGroupInformation.setConfiguration(conf);
if (loginUser == null) {
UserGroupInformation.loginUserFromKeytab(keytabPrincipal,
keytabLocation)
loginUser = UserGroupInformation.getLoginUser();
} else {
loginUser.checkTGTAndReloginFromKeytab();
}
con = DriverManager
.getConnection(jdbc:hive2://x:1/abc;principal=hive/hostname@xx.com);
stmt = con.createStatement();
Thanks,
Chandra
From: Vaibhav Gumashta [mailto:vgumas...@hortonworks.com]
Sent: Thursday, February 26, 2015 2:03 AM
To: user@hive.apache.org
Subject: Re: GSS initiate failed exception
Looks like your Kerberos authentication failed. Did you renew the ticket?
Typical expiry is 24 hours.
From: Bogala, Chandra Reddy
chandra.bog...@gs.commailto:chandra.bog...@gs.com
Reply-To: user@hive.apache.orgmailto:user@hive.apache.org
user@hive.apache.orgmailto:user@hive.apache.org
Date: Wednesday, February 25, 2015 at 3:42 AM
To: user@hive.apache.orgmailto:user@hive.apache.org
user@hive.apache.orgmailto:user@hive.apache.org
Subject: GSS initiate failed exception
Hi,
My Hive jdbc client queries ( hiveserver2) to secured cluster fails with
below exception after one or two days running fine from tomcat. Any idea what
might be the issue? Is it a known issue?
2015-02-25 04:49:43,174 ERROR [org.apache.thrift.transport.TSaslTransport] SASL
negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
~[na:1.7.0_06]
at
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
[hive-exec-0.13.0.jar:0.13.0]
--
-
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
at
org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:203)
Thanks,
Chandra
Re: [ANNOUNCE] New Hive PMC Member - Sergey Shelukhin
Way to go Sergey! -- Lefty On Wed, Feb 25, 2015 at 4:19 PM, Sergey Shelukhin ser...@hortonworks.com wrote: Thanks guys! On 15/2/25, 16:02, Xiaobing Zhou xz...@hortonworks.com wrote: Congrats Sergey! On Feb 25, 2015, at 1:56 PM, Prasanth Jayachandran pjayachand...@hortonworks.com wrote: Congrats Sergey! On Feb 25, 2015, at 1:50 PM, Alexander Pivovarov apivova...@gmail.commailto:apivova...@gmail.com wrote: Congrats! On Wed, Feb 25, 2015 at 12:33 PM, Vaibhav Gumashta vgumas...@hortonworks.commailto:vgumas...@hortonworks.com wrote: Congrats Sergey! On 2/25/15, 9:06 AM, Vikram Dixit vik...@hortonworks.commailto:vik...@hortonworks.com wrote: Congrats Sergey! On 2/25/15, 8:43 AM, Carl Steinbach c...@apache.orgmailto:c...@apache.org wrote: I am pleased to announce that Sergey Shelukhin has been elected to the Hive Project Management Committee. Please join me in congratulating Sergey! Thanks. - Carl
Re: column selection for insert
Not yet ! Sent from my iPhone On Feb 26, 2015, at 8:23 PM, Srinivas Thunga srinivas.thu...@gmail.com wrote: Hi, Thanks for the prompt response. Cann't i insert specific columns which i want? Like we do in oracle. Thanks Regards, Srinivas T On Thu, Feb 26, 2015 at 8:55 PM, Alan Gates alanfga...@gmail.com wrote: Specifying which columns in the target table to insert into (emp in your case) is not supported in 0.14. You have to insert into all the columns in the table and they have to match the order of items in the select or values clause. This will be supported in 1.2. Alan. compose-unknown-contact.jpg Srinivas Thunga February 26, 2015 at 4:34 Hi, I am using Hive 0.14 Can any one help me on to execute the below type query insert into table emp (eid,ename) select * from dept where emp.id = dept.id Thanks Regards, Srinivas T
Re: column selection for insert
Hi, Thanks for the prompt response. Cann't i insert specific columns which i want? Like we do in oracle. *Thanks Regards,* *Srinivas T* On Thu, Feb 26, 2015 at 8:55 PM, Alan Gates alanfga...@gmail.com wrote: Specifying which columns in the target table to insert into (emp in your case) is not supported in 0.14. You have to insert into all the columns in the table and they have to match the order of items in the select or values clause. This will be supported in 1.2. Alan. Srinivas Thunga srinivas.thu...@gmail.com February 26, 2015 at 4:34 Hi, I am using Hive 0.14 Can any one help me on to execute the below type query insert into table emp (eid,ename) select * from dept where emp.id = dept.id *Thanks Regards,* *Srinivas T*
Re: column selection for insert
Hi, Then is there any way to achieve this? *Thanks Regards,* *Srinivas T* On Fri, Feb 27, 2015 at 9:53 AM, Siddharth Tiwari siddharth.tiw...@live.com wrote: Not yet ! Sent from my iPhone On Feb 26, 2015, at 8:23 PM, Srinivas Thunga srinivas.thu...@gmail.com wrote: Hi, Thanks for the prompt response. Cann't i insert specific columns which i want? Like we do in oracle. *Thanks Regards,* *Srinivas T* On Thu, Feb 26, 2015 at 8:55 PM, Alan Gates alanfga...@gmail.com wrote: Specifying which columns in the target table to insert into (emp in your case) is not supported in 0.14. You have to insert into all the columns in the table and they have to match the order of items in the select or values clause. This will be supported in 1.2. Alan. compose-unknown-contact.jpg Srinivas Thunga srinivas.thu...@gmail.com February 26, 2015 at 4:34 Hi, I am using Hive 0.14 Can any one help me on to execute the below type query insert into table emp (eid,ename) select * from dept where emp.id = dept.id *Thanks Regards,* *Srinivas T*
