Yes, correct. I typically start jobs up via screen (not just bulk load jobs, but any kind of job), as then if there is any kind of cleanup or something similar done in the driver program, it'll still get run even if the ssh session gets dropped.
- Gabriel On Thu, Jun 25, 2015 at 8:47 PM Riesland, Zack <[email protected]> wrote: > OH! > > > > So even though I can still see my job running in the YARN resource manager > after losing my SSH window, THAT job is simply to create the HFiles. But > they won’t actually be written because the session is gone. > > > > Is that correct? > > > > That would clear up a lot of confusion… > > > > *From:* Gabriel Reid [mailto:[email protected]] > *Sent:* Thursday, June 25, 2015 2:44 PM > > > *To:* [email protected] > *Subject:* Re: Bug in CsvBulkLoad tool? > > > > Hi Zack, > > > > The job counters are available in the YARN resource manager and/or YARN > historyserver web interface. Your import job will have an entry in that web > interface, and you can then click through to the full list of counters for > the job. > > > > Something else that you mentioned has made me think of another possible > cause for what's going on here. If I understand correctly, you're > connecting via ssh to a gateway machine to launch your job, and then > sometimes you lose your ssh connection while the job is running. If this is > indeed the case, the output from the job won't be written to HBase. > > > > The CsvBulkLoadTool works by writing data to HFiles first, and then once > all the MapReduce work is done, these HFiles are handed off to HBase. If > your ssh connection closes before the full process has completed, the > HFiles won't be handed off to HBase. > > > > Is it possible that something like this is going on? If so, I recommend > you start your job up from within screen [1] or tmux if they're available > to you so that your terminal session doesn't end even if you drop your ssh > connection. > > > > - Gabriel > > > > On Thu, Jun 25, 2015 at 8:27 PM Riesland, Zack <[email protected]> > wrote: > > Thanks Gabriel, > > > > Then perhaps I discovered something interesting. > > > > After my last email, I created a new table with the exact same script, > except I changed the name of the table and the name of the primary key. > > > > Then, I ran the CsvBulkLoad tool using the same folder with the 35 CSV > files. > > > > This time, my table has 1,479,451,977 rows, as expected. > > > > Just like before, there were 1042 mappers, 1792 reducers, etc. > > > > It might be interesting to write a simple test like this: > > > > Create table X with primary key name ‘X’. > > > > Ingest some data. > > > > Create table Y with primary key name ‘X’. > > > > Ingest the same data. > > > > See whether the expected results are achieved. > > > > One clarification: if I run the CsvBulkLoad tool at the command line and > have the same SSH window open when it finishes, it is easy to see all the > statistics. > > > > But where I can find this data in the logs? Since these ingests can take > several hours, I sometimes lose my VPN connection and my SSH window goes > stale. > > > > Thanks! > > > > *From:* Gabriel Reid [mailto:[email protected]] > *Sent:* Thursday, June 25, 2015 2:18 PM > > > *To:* [email protected] > *Subject:* Re: Bug in CsvBulkLoad tool? > > > > Hi Zack, > > > > No, you don't need to worry about the name of the primary key getting in > the way of the rows being added. > > > > Like Anil pointed out, the best thing to look at first is the job > counters. The relevant ones for debugging this situation are the total map > inputs and total map outputs, total reduce inputs and total reduce outputs, > as well as reduce input groups, and finally the PhoenixJobCounters > (INPUT_RECORDS, FAILED_RECORDS, and OUTPUT_RECORDS). > > > > The INPUT_RECORDS and OUTPUT_RECORDS should both be around the number of > rows that you expected (i.e. 1.7 million), along with map input records. If > I remember correctly, the reduce input groups should be around the same > value as well. > > > > Could you post the values that you've got on those counters? > > > > - Gabriel > > > > > > On Thu, Jun 25, 2015 at 4:41 PM Riesland, Zack <[email protected]> > wrote: > > I started writing a long response, and then noticed something: > > > > When I created my new table, I copied/pasted the script and made some > changes, but didn’t change the name of the primary key. > > > > Is it possible that any row being inserted on the new table with a key > that matches a row in the OTHER table is being thrown away? > > > > *From:* anil gupta [mailto:[email protected]] > *Sent:* Thursday, June 25, 2015 10:20 AM > *To:* [email protected] > *Subject:* Re: Bug in CsvBulkLoad tool? > > > > Hi Zack, > > Can you share counters of csvbulkload job? Also, did you run one > csvbulkload job or 35 bulkload job? Whats the schema of phoenix table? How > are you making sure that you have no duplicate rowkey in your dataset? > > If you have duplicate rowkeys. Then cells in that row in HBase will have > more than 1 version. That is something i would check on HBase side to > investigate this problem. > > > > Thanks, > > Anil Gupta > > > > On Thu, Jun 25, 2015 at 3:11 AM, Riesland, Zack <[email protected]> > wrote: > > Earlier this week I was surprised to find that, after dumping tons of data > from a Hive table to an HBase table, about half of the data didn’t end up > in HBase. > > > > So, yesterday, I created a new Phoenix table. > > > > This time, I’m splitting on the first 6 characters of the key, which gives > me about 1700 regions (across 6 fairly beefy region servers). > > > > My 7 billion Hive rows live in 125 5GB csv files on HDFS. > > > > I copied 35 of them to a separate folder, and ran the CsvBuolkLoad tool > against that folder. > > > > The application manager tells me that the job ran to completion. 1042/1042 > successful maps and 1792/1792 successful reduces. > > > > However, when I run the mapreduce.RowCounter against the new table, it > only shows about 300 million rows. > > > > I should see 35/125 * 7 billion = ~ 1.7 billion rows. > > > > These are not primary key collisions. > > > > Can someone please help me understand what is going on? > > > > > > > > > -- > > Thanks & Regards, > Anil Gupta > >
