The rate at which "Map input bytes" and "Map input records" is growing is extremely slow. Is something wrong with the HDFS configuration? But, it was working perfectly fine with the previous Hive version.
Saurabh. On Wed, Dec 30, 2009 at 12:20 PM, Saurabh Nanda <[email protected]>wrote: > Attached are the task logs of one of the tasks. > > Saurabh. > > > On Wed, Dec 30, 2009 at 12:16 PM, Zheng Shao <[email protected]> wrote: > >> This should be compiled into a single map-only job. >> Can you take a look at the progress and the task logs of the job? >> >> We are not aware of any changes that might cause this problem. >> >> Zheng >> >> On Tue, Dec 29, 2009 at 10:35 PM, Saurabh Nanda <[email protected]> >> wrote: >> > Picking up data from the 'raw' table, filtering the unwanted lines and >> > inserting into 'raw_compressed' table which is stored as sequencefile: >> > >> > insert overwrite table raw_compressed partition(dt='2009-04-01') select >> line >> > from raw where dt='2009-04-01' and lower(line) rlike '.*get >> > .*/confirmation.*http.*' and not lower(line) rlike >> > >> '(/images.*?|/styles.*?|/javascripts.*?|/adserver.*?|.*?favicon.*?|/includes/thwarte-logo.html.*)'; >> > >> > Saurabh. >> > >> > On Wed, Dec 30, 2009 at 11:59 AM, Zheng Shao <[email protected]> wrote: >> >> >> >> What is the import query? Do you mean "load data"? >> >> Can you give an example? >> >> >> >> Zheng >> >> >> >> On Tue, Dec 29, 2009 at 10:22 PM, Saurabh Nanda < >> [email protected]> >> >> wrote: >> >> > Also has something changed drastically in Hive over the last 2-3 >> months? >> >> > A >> >> > simply import query seems to be taking forever now! >> >> > >> >> > Saurabh. >> >> > >> >> > On Wed, Dec 30, 2009 at 11:48 AM, Saurabh Nanda < >> [email protected]> >> >> > wrote: >> >> >> >> >> >> I'm taking a look at the HDFS directories through the web interface >> and >> >> >> I >> >> >> can see only 5 files there, not 6. I tried creating the partition >> using >> >> >> the >> >> >> ADD PARTITION command. After that all 6 files get imported >> >> >> successfully. >> >> >> >> >> >> Saurabh. >> >> >> >> >> >> On Wed, Dec 30, 2009 at 11:39 AM, Zheng Shao <[email protected]> >> wrote: >> >> >>> >> >> >>> Can you list the HDFS directories? Are the files in the >> corresponding >> >> >>> directories yet? >> >> >>> >> >> >>> >> >> >>> Zheng >> >> >>> >> >> >>> On Tue, Dec 29, 2009 at 9:57 PM, Saurabh Nanda >> >> >>> <[email protected]> >> >> >>> wrote: >> >> >>> > Hi, >> >> >>> > >> >> >>> > I'm revisiting Hive after a long hiatus, so I may not be aware of >> >> >>> > any >> >> >>> > new >> >> >>> > developments. I had written a script some time back to import >> >> >>> > webserver >> >> >>> > logs >> >> >>> > for a day into a new partition. The same script now running on >> the >> >> >>> > latest >> >> >>> > version of Hive (r894548 compiled off trunk) seems to be >> >> >>> > misbehaving. >> >> >>> > >> >> >>> > I'm importing about 6 files into each partition. However, after >> the >> >> >>> > script >> >> >>> > ends, only 5 files show up in each partition. Do I need to >> >> >>> > explicitly >> >> >>> > issue >> >> >>> > the ADD PARTITION command before loading data? Isn't the >> partition >> >> >>> > implicitly created? >> >> >>> > >> >> >>> > Saurabh. >> >> >>> > -- >> >> >>> > http://nandz.blogspot.com >> >> >>> > http://foodieforlife.blogspot.com >> >> >>> > >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Yours, >> >> >>> Zheng >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> http://nandz.blogspot.com >> >> >> http://foodieforlife.blogspot.com >> >> > >> >> > >> >> > >> >> > -- >> >> > http://nandz.blogspot.com >> >> > http://foodieforlife.blogspot.com >> >> > >> >> >> >> >> >> >> >> -- >> >> Yours, >> >> Zheng >> > >> > >> > >> > -- >> > http://nandz.blogspot.com >> > http://foodieforlife.blogspot.com >> > >> >> >> >> -- >> Yours, >> Zheng >> > > > > -- > http://nandz.blogspot.com > http://foodieforlife.blogspot.com > -- http://nandz.blogspot.com http://foodieforlife.blogspot.com
