Thanks for the pointers Gabriel! Will give it a shot now! On 16 September 2015 at 12:15, Gabriel Reid <gabriel.r...@gmail.com> wrote:
> Yes, there is post-processing that goes on within the driver program (i.e. > the command line tool with which you started the import job). > > The MapReduce job actually just creates HFiles, and then the > post-processing simply involves telling HBase to use these HFiles. If your > terminal closed while running the tool, then the HFiles won't be handed > over to HBase, which will result in what you're seeing. > > I usually start import jobs like this using screen [1] so that losing a > client terminal connection won't get in the way of the full job completing. > > > - Gabriel > > > > 1. https://www.gnu.org/software/screen/manual/screen.html > > On Wed, Sep 16, 2015 at 9:07 PM, Gaurav Kanade <gaurav.kan...@gmail.com> > wrote: > >> Sure, attached below the job counter values. I checked the final status >> of the job and it said succeeded. I could not see the import tool exactly >> because I ran it overnight and my machine rebooted at some point for some >> updates - I wonder if there is some post-processing after the MR job which >> might have failed due to this ? >> >> Thanks for the help ! >> ---------------- >> Logged in as: dr.who >> Counters for job_1442389862209_0002 >> Application Job >> >> - Overview >> >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/job/job_1442389862209_0002> >> - Counters >> >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/jobcounters/job_1442389862209_0002> >> - Configuration >> >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/conf/job_1442389862209_0002> >> - Map tasks >> >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/tasks/job_1442389862209_0002/m> >> - Reduce tasks >> >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/tasks/job_1442389862209_0002/r> >> >> Tools >> Counter Group Counters File System Counters >> Name >> Map >> Reduce >> Total >> FILE: Number of bytes read >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/FILE_BYTES_READ> >> 1520770904675 >> 2604849340144 4125620244819 FILE: Number of bytes written >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/FILE_BYTES_WRITTEN> >> 3031784709196 >> 2616689890216 5648474599412 FILE: Number of large read operations >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/FILE_LARGE_READ_OPS> >> 0 >> 0 0 FILE: Number of read operations >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/FILE_READ_OPS> >> 0 >> 0 0 FILE: Number of write operations >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/FILE_WRITE_OPS> >> 0 >> 0 0 WASB: Number of bytes read >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/WASB_BYTES_READ> >> 186405294283 >> 0 186405294283 WASB: Number of bytes written >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/WASB_BYTES_WRITTEN> >> 0 >> 363027342839 363027342839 WASB: Number of large read operations >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/WASB_LARGE_READ_OPS> >> 0 >> 0 0 WASB: Number of read operations >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/WASB_READ_OPS> >> 0 >> 0 0 WASB: Number of write operations >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.FileSystemCounter/WASB_WRITE_OPS> >> 0 >> 0 0 >> Job Counters >> Name >> Map >> Reduce >> Total >> Launched map tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/TOTAL_LAUNCHED_MAPS> >> 0 >> 0 348 Launched reduce tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/TOTAL_LAUNCHED_REDUCES> >> 0 >> 0 9 Rack-local map tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/RACK_LOCAL_MAPS> >> 0 >> 0 348 Total megabyte-seconds taken by all map tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/MB_MILLIS_MAPS> >> 0 >> 0 460560315648 Total megabyte-seconds taken by all reduce tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/MB_MILLIS_REDUCES> >> 0 >> 0 158604449280 Total time spent by all map tasks (ms) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/MILLIS_MAPS> >> 0 >> 0 599687911 Total time spent by all maps in occupied slots (ms) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/SLOTS_MILLIS_MAPS> >> 0 >> 0 599687911 Total time spent by all reduce tasks (ms) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/MILLIS_REDUCES> >> 0 >> 0 103258105 Total time spent by all reduces in occupied slots (ms) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/SLOTS_MILLIS_REDUCES> >> 0 >> 0 206516210 Total vcore-seconds taken by all map tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/VCORES_MILLIS_MAPS> >> 0 >> 0 599687911 Total vcore-seconds taken by all reduce tasks >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.JobCounter/VCORES_MILLIS_REDUCES> >> 0 >> 0 103258105 >> Map-Reduce Framework >> Name >> Map >> Reduce >> Total >> Combine input records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/COMBINE_INPUT_RECORDS> >> 0 >> 0 0 Combine output records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/COMBINE_OUTPUT_RECORDS> >> 0 >> 0 0 CPU time spent (ms) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/CPU_MILLISECONDS> >> 162773540 >> 90154160 252927700 Failed Shuffles >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/FAILED_SHUFFLE> >> 0 >> 0 0 GC time elapsed (ms) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/GC_TIME_MILLIS> >> 7667781 >> 1607188 9274969 Input split bytes >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/SPLIT_RAW_BYTES> >> 52548 >> 0 52548 Map input records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/MAP_INPUT_RECORDS> >> 861890673 >> 0 861890673 Map output bytes >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/MAP_OUTPUT_BYTES> >> 1488284643774 >> 0 1488284643774 Map output materialized bytes >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/MAP_OUTPUT_MATERIALIZED_BYTES> >> 1515865164102 >> 0 1515865164102 Map output records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/MAP_OUTPUT_RECORDS> >> 13790250768 >> 0 13790250768 Merged Map outputs >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/MERGED_MAP_OUTPUTS> >> 0 >> 3132 3132 Physical memory (bytes) snapshot >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/PHYSICAL_MEMORY_BYTES> >> 192242380800 >> 4546826240 196789207040 Reduce input groups >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/REDUCE_INPUT_GROUPS> >> 0 >> 861890673 861890673 Reduce input records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/REDUCE_INPUT_RECORDS> >> 0 >> 13790250768 13790250768 Reduce output records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/REDUCE_OUTPUT_RECORDS> >> 0 >> 13790250768 13790250768 Reduce shuffle bytes >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/REDUCE_SHUFFLE_BYTES> >> 0 >> 1515865164102 1515865164102 Shuffled Maps >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/SHUFFLED_MAPS> >> 0 >> 3132 3132 Spilled Records >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/SPILLED_RECORDS> >> 27580501536 >> 23694179168 51274680704 Total committed heap usage (bytes) >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/COMMITTED_HEAP_BYTES> >> 186401685504 >> 3023044608 189424730112 Virtual memory (bytes) snapshot >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.TaskCounter/VIRTUAL_MEMORY_BYTES> >> 537370951680 >> 19158048768 556529000448 >> Phoenix MapReduce Import >> Name >> Map >> Reduce >> Total >> Upserts Done >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Phoenix%20MapReduce%20Import/Upserts%20Done> >> 861890673 >> 0 861890673 >> Shuffle Errors >> Name >> Map >> Reduce >> Total >> BAD_ID >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Shuffle%20Errors/BAD_ID> >> 0 >> 0 0 CONNECTION >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Shuffle%20Errors/CONNECTION> >> 0 >> 0 0 IO_ERROR >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Shuffle%20Errors/IO_ERROR> >> 0 >> 0 0 WRONG_LENGTH >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Shuffle%20Errors/WRONG_LENGTH> >> 0 >> 0 0 WRONG_MAP >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Shuffle%20Errors/WRONG_MAP> >> 0 >> 0 0 WRONG_REDUCE >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/Shuffle%20Errors/WRONG_REDUCE> >> 0 >> 0 0 >> File Input Format Counters >> Name >> Map >> Reduce >> Total >> Bytes Read >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter/BYTES_READ> >> 186395934997 >> 0 186395934997 >> File Output Format Counters >> Name >> Map >> Reduce >> Total >> Bytes Written >> <http://headnode0.ctlynvnzlysu3nnyyhqmcwjbee.gx.internal.cloudapp.net:19888/jobhistory/singlejobcounter/job_1442389862209_0002/org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter/BYTES_WRITTEN> >> 0 >> 363027342839 363027342839 >> >> On 16 September 2015 at 11:46, Gabriel Reid <gabriel.r...@gmail.com> >> wrote: >> >>> Can you view (and post) the job counters values from the import job? >>> These should be visible in the job history server. >>> >>> Also, did you see the import tool exit successfully (in the terminal >>> where you started it?) >>> >>> - Gabriel >>> >>> On Wed, Sep 16, 2015 at 6:24 PM, Gaurav Kanade <gaurav.kan...@gmail.com> >>> wrote: >>> > Hi guys >>> > >>> > I was able to get this to work after using bigger VMs for data nodes; >>> > however now the bigger problem I am facing is after my MR job completes >>> > successfully I am not seeing any rows loaded in my table (count shows >>> 0 both >>> > via phoenix and hbase) >>> > >>> > Am I missing something simple ? >>> > >>> > Thanks >>> > Gaurav >>> > >>> > >>> > On 12 September 2015 at 11:16, Gabriel Reid <gabriel.r...@gmail.com> >>> wrote: >>> >> >>> >> Around 1400 mappers sounds about normal to me -- I assume your block >>> >> size on HDFS is 128 MB, which works out to 1500 mappers for 200 GB of >>> >> input. >>> >> >>> >> To add to what Krishna asked, can you be a bit more specific on what >>> >> you're seeing (in log files or elsewhere) which leads you to believe >>> >> the data nodes are running out of capacity? Are map tasks failing? >>> >> >>> >> If this is indeed a capacity issue, one thing you should ensure is >>> >> that map output comression is enabled. This doc from Cloudera explains >>> >> this (and the same information applies whether you're using CDH or >>> >> not) - >>> >> >>> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_23_3.html >>> >> >>> >> In any case, apart from that there isn't any basic thing that you're >>> >> probably missing, so any additional information that you can supply >>> >> about what you're running into would be useful. >>> >> >>> >> - Gabriel >>> >> >>> >> >>> >> On Sat, Sep 12, 2015 at 2:17 AM, Krishna <research...@gmail.com> >>> wrote: >>> >> > 1400 mappers on 9 nodes is about 155 mappers per datanode which >>> sounds >>> >> > high >>> >> > to me. There are very few specifics in your mail. Are you using >>> YARN? >>> >> > Can >>> >> > you provide details like table structure, # of rows & columns, etc. >>> Do >>> >> > you >>> >> > have an error stack? >>> >> > >>> >> > >>> >> > On Friday, September 11, 2015, Gaurav Kanade < >>> gaurav.kan...@gmail.com> >>> >> > wrote: >>> >> >> >>> >> >> Hi All >>> >> >> >>> >> >> I am new to Apache Phoenix (and relatively new to MR in general) >>> but I >>> >> >> am >>> >> >> trying a bulk insert of a 200GB tar separated file in an HBase >>> table. >>> >> >> This >>> >> >> seems to start off fine and kicks off about ~1400 mappers and 9 >>> >> >> reducers (I >>> >> >> have 9 data nodes in my setup). >>> >> >> >>> >> >> At some point I seem to be running into problems with this process >>> as >>> >> >> it >>> >> >> seems the data nodes run out of capacity (from what I can see my >>> data >>> >> >> nodes >>> >> >> have 400GB local space). It does seem that certain reducers eat up >>> most >>> >> >> of >>> >> >> the capacity on these - thus slowing down the process to a crawl >>> and >>> >> >> ultimately leading to Node Managers complaining that Node Health >>> is bad >>> >> >> (log-dirs and local-dirs are bad) >>> >> >> >>> >> >> Is there some inherent setting I am missing that I need to set up >>> for >>> >> >> the >>> >> >> particular job ? >>> >> >> >>> >> >> Any pointers would be appreciated >>> >> >> >>> >> >> Thanks >>> >> >> >>> >> >> -- >>> >> >> Gaurav Kanade, >>> >> >> Software Engineer >>> >> >> Big Data >>> >> >> Cloud and Enterprise Division >>> >> >> Microsoft >>> > >>> > >>> > >>> > >>> > -- >>> > Gaurav Kanade, >>> > Software Engineer >>> > Big Data >>> > Cloud and Enterprise Division >>> > Microsoft >>> >> >> >> >> -- >> Gaurav Kanade, >> Software Engineer >> Big Data >> Cloud and Enterprise Division >> Microsoft >> > > -- Gaurav Kanade, Software Engineer Big Data Cloud and Enterprise Division Microsoft
