Re: Setting | Verifying | Hive Query Parameters from Java
The answer is inlined On Sat, Mar 8, 2014 at 12:38 PM, Garg, Rinku rinku.g...@fisglobal.comwrote: Hi Gordon, Thanks a lot for reply. We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(mapreduce.map.memory.mb,50); props.setProperty(mapreduce.reduce.memory.mb,500); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. QUERY: 1. *Are we doing rightly for setting up the hive properties, if yes then how can we verify that?* You can execute HQL like set fs.defaultFS; to check the props in your jdbc session. 2. *If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC?* I think you may be wrong. I've never set props through driver manager API. From hive code, looks like only user/password will be handled when the props are thrown through driver manager API. The hive conf props may be ignored. (I haven't tested this yet. Anyway, you can use the method in the last question to check whether your props take effect ) I suggest that you execute HQL like set fs.defaultFS=hdfs://hostA:8020; after an jdbc connection is established. Please reply Thanks Rinku Garg *From:* Gordon Wang [mailto:gw...@gopivotal.com] *Sent:* Friday, March 07, 2014 2:45 PM *To:* user *Subject:* Re: Setting | Verifying | Hive Query Parameters from Java Hi Rinku, I think yarn.nodemanager.resource.memory-mb is a static configuration prop. You can not change this dynamically. The other MR props looks OK to me. On Fri, Mar 7, 2014 at 2:58 PM, Garg, Rinku rinku.g...@fisglobal.com wrote: Hi Gordon, Thanks a lot for your reply. The properties mentioned in mail trail are just an example. Actual properties that we want to set are as given below: set yarn.nodemanager.resource.memory-mb=16384; set mapreduce.map.memory.mb=2048; set mapreduce.reduce.memory.mb=2048; set mapreduce.map.java.opts=-Xmx2048M; set yarn.app.mapreduce.am.command-opts=-Xmx2048m; Please suggest. Thanks Rinku Garg *From:* Gordon Wang [mailto:gw...@gopivotal.com] *Sent:* Friday, March 07, 2014 11:49 AM *To:* user@hive.apache.org *Subject:* Re: Setting | Verifying | Hive Query Parameters from Java The 2 following props are for hive server2. I don't think you can change it in your jdbc session. I am wondering why you need to change them in your jdbc connection. props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); You can set props in your jdbc connetion with HQL like propA=valueA; More, your connection url is for hive server, it does not work for hive server2. If you need to use hive server2, you have to use jdbc:hive2://master149:1/default On Tue, Mar 4, 2014 at 7:43 PM, Garg, Rinku rinku.g...@fisglobal.com wrote: Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. *QUERY: * *1.** Are we doing rightly for setting up the hive properties, if yes then how can we verify that?* *2.* *If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC?* Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message
RE: Setting | Verifying | Hive Query Parameters from Java
Hi, yarn.nodemanager.resource.memory-mb is YARN configuration. Others are client configurations. In order to take effect, you need to set at NodeManger configurations and restart NM. This change can be observed in RM web UI. Thanks Regards Rohith Sharma K S From: Gordon Wang [mailto:gw...@gopivotal.com] Sent: 07 March 2014 14:45 To: user Subject: Re: Setting | Verifying | Hive Query Parameters from Java Hi Rinku, I think yarn.nodemanager.resource.memory-mb is a static configuration prop. You can not change this dynamically. The other MR props looks OK to me. On Fri, Mar 7, 2014 at 2:58 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi Gordon, Thanks a lot for your reply. The properties mentioned in mail trail are just an example. Actual properties that we want to set are as given below: set yarn.nodemanager.resource.memory-mb=16384; set mapreduce.map.memory.mb=2048; set mapreduce.reduce.memory.mb=2048; set mapreduce.map.java.opts=-Xmx2048M; set yarn.app.mapreduce.am.command-opts=-Xmx2048m; Please suggest. Thanks Rinku Garg From: Gordon Wang [mailto:gw...@gopivotal.commailto:gw...@gopivotal.com] Sent: Friday, March 07, 2014 11:49 AM To: user@hive.apache.orgmailto:user@hive.apache.org Subject: Re: Setting | Verifying | Hive Query Parameters from Java The 2 following props are for hive server2. I don't think you can change it in your jdbc session. I am wondering why you need to change them in your jdbc connection. props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); You can set props in your jdbc connetion with HQL like propA=valueA; More, your connection url is for hive server, it does not work for hive server2. If you need to use hive server2, you have to use jdbc:hive2://master149:1/default On Tue, Mar 4, 2014 at 7:43 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. QUERY: 1. Are we doing rightly for setting up the hive properties, if yes then how can we verify that? 2. If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC? Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Regards Gordon Wang _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Regards Gordon Wang
Re: Setting | Verifying | Hive Query Parameters from Java
Hi Rinku, I think yarn.nodemanager.resource.memory-mb is a static configuration prop. You can not change this dynamically. The other MR props looks OK to me. On Fri, Mar 7, 2014 at 2:58 PM, Garg, Rinku rinku.g...@fisglobal.comwrote: Hi Gordon, Thanks a lot for your reply. The properties mentioned in mail trail are just an example. Actual properties that we want to set are as given below: set yarn.nodemanager.resource.memory-mb=16384; set mapreduce.map.memory.mb=2048; set mapreduce.reduce.memory.mb=2048; set mapreduce.map.java.opts=-Xmx2048M; set yarn.app.mapreduce.am.command-opts=-Xmx2048m; Please suggest. Thanks Rinku Garg *From:* Gordon Wang [mailto:gw...@gopivotal.com] *Sent:* Friday, March 07, 2014 11:49 AM *To:* user@hive.apache.org *Subject:* Re: Setting | Verifying | Hive Query Parameters from Java The 2 following props are for hive server2. I don't think you can change it in your jdbc session. I am wondering why you need to change them in your jdbc connection. props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); You can set props in your jdbc connetion with HQL like propA=valueA; More, your connection url is for hive server, it does not work for hive server2. If you need to use hive server2, you have to use jdbc:hive2://master149:1/default On Tue, Mar 4, 2014 at 7:43 PM, Garg, Rinku rinku.g...@fisglobal.com wrote: Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. *QUERY: * *1.** Are we doing rightly for setting up the hive properties, if yes then how can we verify that?* *2.* *If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC?* Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Regards Gordon Wang _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Regards Gordon Wang
RE: Setting | Verifying | Hive Query Parameters from Java
Hi All, Can anybody help me on below mail trail. Thanks Rinku Garg From: Garg, Rinku Sent: Tuesday, March 04, 2014 5:14 PM To: user@hive.apache.org Subject: Setting | Verifying | Hive Query Parameters from Java Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. QUERY: 1. Are we doing rightly for setting up the hive properties, if yes then how can we verify that? 2. If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC? Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
RE: Setting | Verifying | Hive Query Parameters from Java
If you want to set some properties of hive, just run it as it is in your JDBC connection. Any command in the hive JDBC will send to the server as the same if you run set hive.server2.async.exec.threads=50; in the hive session. Run the command set hive.server2.async.exec.threads=50; as a SQL statement, it will adjust the value for your JDBC connection. About the properties setting, I am not sure if it will work in Hive JDBC. Hive JDBC is a limited JDBC implementation based on Hive, so it maybe won't work, but I don't know for sure. Yong From: rinku.g...@fisglobal.com To: user@hive.apache.org Subject: RE: Setting | Verifying | Hive Query Parameters from Java Date: Thu, 6 Mar 2014 11:12:52 + Hi All, Can anybody help me on below mail trail. Thanks Rinku Garg From: Garg, Rinku Sent: Tuesday, March 04, 2014 5:14 PM To: user@hive.apache.org Subject: Setting | Verifying | Hive Query Parameters from Java Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. QUERY: 1. Are we doing rightly for setting up the hive properties, if yes then how can we verify that? 2. If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC? Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Setting | Verifying | Hive Query Parameters from Java
The 2 following props are for hive server2. I don't think you can change it in your jdbc session. I am wondering why you need to change them in your jdbc connection. props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); You can set props in your jdbc connetion with HQL like propA=valueA; More, your connection url is for hive server, it does not work for hive server2. If you need to use hive server2, you have to use jdbc:hive2://master149:1/default On Tue, Mar 4, 2014 at 7:43 PM, Garg, Rinku rinku.g...@fisglobal.comwrote: Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. *QUERY: * *1. **Are we doing rightly for setting up the hive properties, if yes then how can we verify that?* *2. **If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC?* Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Regards Gordon Wang
RE: Setting | Verifying | Hive Query Parameters from Java
Hi Gordon, Thanks a lot for your reply. The properties mentioned in mail trail are just an example. Actual properties that we want to set are as given below: set yarn.nodemanager.resource.memory-mb=16384; set mapreduce.map.memory.mb=2048; set mapreduce.reduce.memory.mb=2048; set mapreduce.map.java.opts=-Xmx2048M; set yarn.app.mapreduce.am.command-opts=-Xmx2048m; Please suggest. Thanks Rinku Garg From: Gordon Wang [mailto:gw...@gopivotal.com] Sent: Friday, March 07, 2014 11:49 AM To: user@hive.apache.org Subject: Re: Setting | Verifying | Hive Query Parameters from Java The 2 following props are for hive server2. I don't think you can change it in your jdbc session. I am wondering why you need to change them in your jdbc connection. props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); You can set props in your jdbc connetion with HQL like propA=valueA; More, your connection url is for hive server, it does not work for hive server2. If you need to use hive server2, you have to use jdbc:hive2://master149:1/default On Tue, Mar 4, 2014 at 7:43 PM, Garg, Rinku rinku.g...@fisglobal.commailto:rinku.g...@fisglobal.com wrote: Hi All, We have installed CDH4.2.0 and hive-0.10.0-cdh4.2.0. Both are working as desired. We need to set hive configuration parameter from Java while making JDBC connection. We have written a java program to execute queries on hive server with some configurations properties setting dynamically . We are doing it as below CONNECTION_URL=jdbc:hive://master149:1/default Next, we are doing following method to set properties through java props.setProperty(hive.server2.async.exec.threads,50); props.setProperty(hive.server2.thrift.max.worker.threads,500); props.setProperty(hive.groupby.orderby.position.alias,false); and a hive connection is made as given below hiveConnection = DriverManager.getConnection(connectionURL,props); by above steps when a hive connection is made using hive-jdbc and we are getting hive query results as desired. QUERY: 1. Are we doing rightly for setting up the hive properties, if yes then how can we verify that? 2. If the above is not the right way, then how can we achieve setting hive configuration parameters from Java using JDBC? Thanks Rinku Garg _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. -- Regards Gordon Wang _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
