Re: Hadoop streaming with insert dynamic partition generate many small files

2014-02-02 Thread Chen Wang
it seems that hive.exec.reducers.bytes.per.reducer is still not big enough: I added another 0, and now i only gets one file under each partition. On Sun, Feb 2, 2014 at 10:14 PM, Chen Wang wrote: > Hi, > I am using java reducer reading from a table, and then write to another > one: > > FROM (

Re: What are all the factors that go into the number of mappers - ORC

2014-02-02 Thread Prasanth Jayachandran
Hi John Number of mappers is equal to the number of splits generated. Following are the factors that go into split generation 1) HDFS block size 2) Max split size a split is cut when 1) the cumulative size of all adjacent stripes are greater than HDFS block size 2) the cumulative size of all adj

Hadoop streaming with insert dynamic partition generate many small files

2014-02-02 Thread Chen Wang
Hi, I am using java reducer reading from a table, and then write to another one: FROM ( FROM ( SELECT column1,... FROM table1 WHERE ( partition > 6 and partition < 12 ) ) A MAP A.co

Re: Map-side join memory limit is too low

2014-02-02 Thread Navis류승우
try "set hive.mapred.local.mem=7000" or add it to hive-site.xml instead of modifying hive-env.sh HADOOP_HEAPSIZE is not in use. Should fix documentation of it. Thanks, Navis 2014-01-31 Avrilia Floratou : > Hi, > I'm running hive 0.12 on yarn and I'm trying to convert a common join into > a map

What are all the factors that go into the number of mappers - ORC

2014-02-02 Thread John Omernik
I have two clusters, but small dev clusters, and I loaded the same dataset into both of them. The data size on disk is within 2000 Bytes. Both are ORC, one is Hive 11 and one is Hive 12. One is allocating about 8 more mappers to the exact same query. I am just curious what settings would change

Re: Performance problem with HBase

2014-02-02 Thread Navis류승우
1. current implementation of hbase handler cannot pushdown filter with 'like' expression. You might rewrite the query some thing like "key >= 0010_0 AND key <= 0010_9" 2. Each of all tasks seemed to be scanning whole table (ie. 1000+ time), which is fixed in HIVE-342

Re: Hive 0.11 & Hive 0.12 - Hive MySQL Metastore is not used for INSERT

2014-02-02 Thread Naga Vijay
I have modified the title of the JIRA to properly reflect the issue faced - Hive 0.11 & Hive 0.12 - derby.log and TempStatsStore are created when using MySQL Metastore Regards Naga On Sun, Feb 2, 2014 at 2:15 PM, Naga Vijay wrote: > Hi, > > I have checked my hive-site.xml ; it is fine ; hive

Re: DESCRIBE EXTENDED show numRows=0

2014-02-02 Thread Navis류승우
Could you check the task log? For stat gathering, hive uses derby by default. If the jdbc driver for derby is not in auxlib, task cannot publish stats. 2014-01-30 Stephen Sprague : > the answer to this would seemingly be no. i just tried it in hive v0.12. > > numRows=0 before and numRows=0 aft

Re: Hive 0.11 & Hive 0.12 - Hive MySQL Metastore is not used for INSERT

2014-02-02 Thread Naga Vijay
Hi, I have checked my hive-site.xml ; it is fine ; hive metastore service is running as well. I have logged this - https://issues.apache.org/jira/browse/HIVE-6352 Regards Naga On Sun, Feb 2, 2014 at 9:11 AM, Matouk IFTISSEN wrote: > Hello, > see your hie-site.xml file, and check the database

Re: Hive 0.11 & Hive 0.12 - Hive MySQL Metastore is not used for INSERT

2014-02-02 Thread Matouk IFTISSEN
Hello, see your hie-site.xml file, and check the database connexion( connexion string) because in this file, that happend connexion to the metastore when you start Hive. Hope this helps you. regards. Matouk 2014-02-02 Naga Vijay : > Hi, > > I am facing this situation for the below mentioned hql

Re: Find a date that is in the range of any array dates in Hive

2014-02-02 Thread Matouk IFTISSEN
Hé You can use explode function that is an udf in hive: like this SELECT explode(ActivityDates) AS ActivityDates FROM myMapTable; and think who to map the result whith rates ;) Regadrs. Matouk 2014-02-01 Raj Hadoop : > Hi, > > > I have the following requirement from a Hive table below. > > Cus

Hive equivalent of dump() in Oracle

2014-02-02 Thread pandees waran
Hi, In oracle,DUMP returns a VARCHAR2 value containing the datatype code, length in bytes, and internal representation of expr. SELECT DUMP('abc', 1016) FROM DUAL; DUMP('ABC',1016) -- Typ=96 Len=3 CharacterSet=WE8DEC: 61,62,63 Do we have any equivalen