Re: Question on bucketed map join

2012-03-26 Thread Amit Sharma
Hi Bejoy, I am joining two tables which are both bucketed 64 ways and i want to do a bucketed map join on them. I set the flag set hive.optimize.bucketmapjoin = true;. The auto.convert.join is always false on our cluster. When i run the following query: select /*+ MAPJOIN(b) */ a.visitor_id FROM

Re: Zero Byte file in HDFS

2012-03-26 Thread Abhishek Pratap Singh
This sounds goods as long as the output of select query has at least one row. But in my case it can be zero rows. Thanks, Abhishek On Mon, Mar 26, 2012 at 2:48 PM, Bejoy KS wrote: > ** > Hi Abshiek > Merging happens as a last stage of hive jobs. Say your hive query is > translated to n MR jobs

Re: Zero Byte file in HDFS

2012-03-26 Thread Bejoy KS
Hi Abshiek Merging happens as a last stage of hive jobs. Say your hive query is translated to n MR jobs when you enable merge you can set a size that is needed to merge (usually block size). So after n MR jobs there would be a map only job that is automatically triggered from hive which merges

Re: Zero Byte file in HDFS

2012-03-26 Thread Abhishek Pratap Singh
Thanks Bejoy for this input, Does this merging will combine all the small files to least block size for the very first mappers of the hive job? Well i ll explore on this, my interest on deleting the zero byte files from HDFS comes from reducing the cost of Bookkeeping these files in system. The met

Zero Byte file in HDFS

2012-03-26 Thread Abhishek Pratap Singh
Hi All, I was just going through the implementation scenario of avoiding or deleting Zero byte file in HDFS. I m using Hive partition table where the data in partition come from INSERT OVERWRITE command using the SELECT from few other tables. Sometimes 0 Byte files are being generated in those par

Re:Re: Re: hive cann't support setting queryTimeout

2012-03-26 Thread 王锋
thanks for Pawar. the metadata store is fine and it was in our mysql safe plateform. When I restart the hiveserver ,it 's ok. but when the hiveserver occurred the question,it has been running about one week, and the mem is about 5g . so I agree 1). If the large mem of hive cause the question, I

Re: Re: hive cann't support setting queryTimeout

2012-03-26 Thread Nitin Pawar
Hi, can you check the logs on hiveserver? Is your metadata store working fine? Cause this issue comes only if there is a lag with metadata store or hiveserver is overly loaded. 1) are there any queries running in hiveserver? if yes .. then what is memory status of the server? 2) Also check if th

Re:Re: hive cann't support setting queryTimeout

2012-03-26 Thread 王锋
yes At 2012-03-26 15:26:49,"Nitin Pawar" wrote: Hi Weng, are you using hiveserver ? Thanks, Nitin you can not set a query On Mon, Mar 26, 2012 at 12:45 PM, 王锋 wrote: Hi,all when I using hive through jdbc,and execute the code below. Statement stmt = con.createStatement(); stmt.setQu

Re:Re: hive cann't support setting queryTimeout

2012-03-26 Thread 王锋
ya,this morning,we see a serious question. when our system send sql "CREATE DATABASE IF NOT EXISTS dbname" to hiveserver through jdbc, the result had not been returned after several hours. And this is the second time. 是的。 今天凌晨 系统遇到一个问题, 执行 CREATE DATABASE IF NOT EXISTS dbname时,hiveserver 一直没有返回。

Re: hive cann't support setting queryTimeout

2012-03-26 Thread Nitin Pawar
Hi Weng, are you using hiveserver ? Thanks, Nitin you can not set a query On Mon, Mar 26, 2012 at 12:45 PM, 王锋 wrote: > Hi,all > when I using hive through jdbc,and execute the code below. > > Statement stmt = con.createStatement(); > stmt.setQueryTimeout(10); > > hive thrown the exception "Me

Re: hive cann't support setting queryTimeout

2012-03-26 Thread badqiu
hive jdbc里面还有好多方法没有实现。具体自己查看源代码吧。 在 2012年3月26日 下午3:15,王锋 写道: > Hi,all > when I using hive through jdbc,and execute the code below. > > Statement stmt = con.createStatement(); > stmt.setQueryTimeout(10); > > hive thrown the exception "Method not support." so how can I set the > query timeout like