Thanks Gabriel,
Below is a redacted stack trace snippet.
I think you're on to something: The "error" line references a "table" but
actually mentions an index:
15/10/23 06:00:56 ERROR mapreduce.CsvBulkLoadTool: Import job on table=<Index
name> failed due to exception:java.lang.IllegalArgumentException: No regions
passed
The index/table in question DOES appear to exist.
Any thoughts?
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Opening socket connection to
server <server name>.<domain>/x.y.z.65:2181. Will not attempt to authenticate
using SASL (unknown error)
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Socket connection established to
<server name>.<domain>/x.y.z.65:2181, initiating session
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Session establishment complete on
server <server name>.<domain>/x.y.z.65:2181, sessionid = 0x350672b547f9b44,
negotiated timeout = 40000
15/10/23 06:00:56 INFO client.ConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
15/10/23 06:00:56 INFO client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x350672b547f9b44
15/10/23 06:00:56 INFO zookeeper.ZooKeeper: Session: 0x350672b547f9b44 closed
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: EventThread shut down
15/10/23 06:00:56 INFO query.ConnectionQueryServicesImpl: Found quorum: <server
name>.<domain>:2181,<SERVER NAME>.<domain>:2181,<SERVER>-001.<domain>:2181
15/10/23 06:00:56 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=<server name>.<domain>:2181,<SERVER
NAME>.<domain>:2181,<SERVER>-001.<domain>:2181 sessionTimeout=120000
watcher=hconnection-0x40d2382e, quorum=<server name>.<domain>:2181,<SERVER
NAME>.<domain>:2181,<SERVER>-001.<domain>:2181, baseZNode=/hbase
15/10/23 06:00:56 INFO zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x40d2382e connecting to ZooKeeper ensemble=<server
name>.<domain>:2181,<SERVER NAME>.<domain>:2181,<SERVER>-001.<domain>:2181
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Opening socket connection to
server <SERVER NAME>.<domain>/x.y.z.64:2181. Will not attempt to authenticate
using SASL (unknown error)
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Socket connection established to
<SERVER NAME>.<domain>/x.y.z.64:2181, initiating session
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Session establishment complete on
server <SERVER NAME>.<domain>/x.y.z.64:2181, sessionid = 0x250672b527a97bc,
negotiated timeout = 40000
15/10/23 06:00:56 INFO client.ConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
15/10/23 06:00:56 INFO client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x250672b527a97bc
15/10/23 06:00:56 INFO zookeeper.ZooKeeper: Session: 0x250672b527a97bc closed
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: EventThread shut down
15/10/23 06:00:56 WARN hbase.HBaseConfiguration: Config option
"hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
15/10/23 06:00:56 INFO mapreduce.CsvBulkLoadTool: Configuring ZK quorum to
<SERVER NAME>.<domain>,<server name>.<domain>,<SERVER>-001.<domain>
15/10/23 06:00:56 WARN hbase.HBaseConfiguration: Config option
"hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
15/10/23 06:00:56 INFO query.ConnectionQueryServicesImpl: Table <Table Name>
has been added into latestMetaData
15/10/23 06:00:56 INFO mapreduce.CsvBulkLoadTool: Configuring HFile output path
to /tmp/caa620d5-f3fe-4523-a04e-51cf163f690d/<Table Name>
15/10/23 06:00:56 INFO mapreduce.CsvBulkLoadTool: Configuring HFile output path
to /tmp/caa620d5-f3fe-4523-a04e-51cf163f690d/<Index name>
15/10/23 06:00:56 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=<server name>.<domain>:2181,<SERVER
NAME>.<domain>:2181,<SERVER>-001.<domain>:2181 sessionTimeout=120000
watcher=hconnection-0x33f3af12, quorum=<server name>.<domain>:2181,<SERVER
NAME>.<domain>:2181,<SERVER>-001.<domain>:2181, baseZNode=/hbase
15/10/23 06:00:56 INFO zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x33f3af12 connecting to ZooKeeper ensemble=<server
name>.<domain>:2181,<SERVER NAME>.<domain>:2181,<SERVER>-001.<domain>:2181
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Opening socket connection to
server <server name>.<domain>/x.y.z.65:2181. Will not attempt to authenticate
using SASL (unknown error)
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Socket connection established to
<server name>.<domain>/x.y.z.65:2181, initiating session
15/10/23 06:00:56 INFO zookeeper.ClientCnxn: Session establishment complete on
server <server name>.<domain>/x.y.z.65:2181, sessionid = 0x350672b547f9b45,
negotiated timeout = 40000
15/10/23 06:00:56 INFO mapreduce.HFileOutputFormat2: Looking up current regions
for table <Table Name>
15/10/23 06:00:56 INFO mapreduce.HFileOutputFormat2: Looking up current regions
for table <Index name>
15/10/23 06:00:56 WARN hbase.HBaseConfiguration: Config option
"hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
15/10/23 06:00:56 WARN hbase.HBaseConfiguration: Config option
"hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
15/10/23 06:00:56 INFO mapreduce.HFileOutputFormat2: Configuring 0 reduce
partitions to match current region count
15/10/23 06:00:56 INFO mapreduce.HFileOutputFormat2: Writing partition
information to /tmp/partitions_2d12d0c0-17a0-46b6-9f0c-147f6d817ecb
15/10/23 06:00:56 ERROR mapreduce.CsvBulkLoadTool: Import job on table=<Index
name> failed due to exception:java.lang.IllegalArgumentException: No regions
passed
15/10/23 06:00:56 INFO mapreduce.HFileOutputFormat2: Configuring 513 reduce
partitions to match current region count
15/10/23 06:00:56 INFO mapreduce.HFileOutputFormat2: Writing partition
information to /tmp/partitions_754b7c35-ab5d-40b2-bb21-dc324689f4ff
15/10/23 06:00:56 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
15/10/23 06:00:56 INFO compress.CodecPool: Got brand-new compressor [.deflate]
15/10/23 06:00:57 INFO mapreduce.HFileOutputFormat2: Incremental table <Table
Name> output configured.
15/10/23 06:00:57 INFO mapreduce.CsvBulkLoadTool: Running MapReduce import job
from /user/hdfs/er_v12_staging_1420070400-1445644799 to
/tmp/caa620d5-f3fe-4523-a04e-51cf163f690d/<Table Name>
15/10/23 06:00:57 INFO impl.TimelineClientImpl: Timeline service address:
http://<server name>.<domain>:8188/ws/v1/timeline/
15/10/23 06:00:57 INFO client.RMProxy: Connecting to ResourceManager at <server
name>.<domain>/x.y.z.65:8050
15/10/23 06:00:58 INFO input.FileInputFormat: Total input paths to process : 97
15/10/23 06:00:58 INFO mapreduce.JobSubmitter: number of splits:97
15/10/23 06:00:58 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1445519089673_0106
15/10/23 06:00:58 INFO impl.YarnClientImpl: Submitted application
application_1445519089673_0106
15/10/23 06:00:58 INFO mapreduce.Job: The url to track the job: http://<server
name>.<domain>:8088/proxy/application_1445519089673_0106/
15/10/23 06:00:58 INFO mapreduce.Job: Running job: job_1445519089673_0106
15/10/23 06:01:06 INFO mapreduce.Job: Job job_1445519089673_0106 running in
uber mode : false
15/10/23 06:01:06 INFO mapreduce.Job: map 0% reduce 0%
15/10/23 06:01:25 INFO mapreduce.Job: map 1% reduce 0%
15/10/23 06:01:26 INFO mapreduce.Job: map 3% reduce 0%
15/10/23 06:01:27 INFO mapreduce.Job: map 5% reduce 0%
15/10/23 06:01:28 INFO mapreduce.Job: map 8% reduce 0%
15/10/23 06:01:29 INFO mapreduce.Job: map 11% reduce 0%
15/10/23 06:01:31 INFO mapreduce.Job: map 12% reduce 0%
15/10/23 06:01:33 INFO mapreduce.Job: map 13% reduce 0%
15/10/23 06:01:34 INFO mapreduce.Job: map 14% reduce 0%
15/10/23 06:01:35 INFO mapreduce.Job: map 16% reduce 0%
15/10/23 06:01:36 INFO mapreduce.Job: map 19% reduce 0%
15/10/23 06:01:37 INFO mapreduce.Job: map 22% reduce 0%
15/10/23 06:01:38 INFO mapreduce.Job: map 30% reduce 0%
15/10/23 06:01:39 INFO mapreduce.Job: map 36% reduce 0%
15/10/23 06:01:40 INFO mapreduce.Job: map 45% reduce 0%
15/10/23 06:01:41 INFO mapreduce.Job: map 52% reduce 0%
15/10/23 06:01:42 INFO mapreduce.Job: map 61% reduce 0%
15/10/23 06:01:43 INFO mapreduce.Job: map 72% reduce 1%
15/10/23 06:01:44 INFO mapreduce.Job: map 80% reduce 1%
15/10/23 06:01:45 INFO mapreduce.Job: map 86% reduce 1%
15/10/23 06:01:46 INFO mapreduce.Job: map 92% reduce 2%
15/10/23 06:01:48 INFO mapreduce.Job: map 96% reduce 2%
15/10/23 06:01:49 INFO mapreduce.Job: map 99% reduce 3%
15/10/23 06:01:50 INFO mapreduce.Job: map 99% reduce 4%
15/10/23 06:01:51 INFO mapreduce.Job: map 100% reduce 5%
15/10/23 06:01:52 INFO mapreduce.Job: map 100% reduce 6%
15/10/23 06:01:53 INFO mapreduce.Job: map 100% reduce 9%
15/10/23 06:01:54 INFO mapreduce.Job: map 100% reduce 12%
15/10/23 06:01:55 INFO mapreduce.Job: map 100% reduce 15%
15/10/23 06:01:56 INFO mapreduce.Job: map 100% reduce 19%
15/10/23 06:01:57 INFO mapreduce.Job: map 100% reduce 24%
15/10/23 06:01:58 INFO mapreduce.Job: map 100% reduce 30%
15/10/23 06:01:59 INFO mapreduce.Job: map 100% reduce 34%
15/10/23 06:02:00 INFO mapreduce.Job: map 100% reduce 37%
15/10/23 06:02:01 INFO mapreduce.Job: map 100% reduce 39%
15/10/23 06:02:02 INFO mapreduce.Job: map 100% reduce 40%
15/10/23 06:02:03 INFO mapreduce.Job: map 100% reduce 42%
15/10/23 06:02:04 INFO mapreduce.Job: map 100% reduce 45%
15/10/23 06:02:05 INFO mapreduce.Job: map 100% reduce 47%
15/10/23 06:02:06 INFO mapreduce.Job: map 100% reduce 50%
15/10/23 06:02:07 INFO mapreduce.Job: map 100% reduce 53%
15/10/23 06:02:08 INFO mapreduce.Job: map 100% reduce 56%
15/10/23 06:02:09 INFO mapreduce.Job: map 100% reduce 62%
15/10/23 06:02:10 INFO mapreduce.Job: map 100% reduce 70%
15/10/23 06:02:11 INFO mapreduce.Job: map 100% reduce 76%
15/10/23 06:02:12 INFO mapreduce.Job: map 100% reduce 80%
15/10/23 06:02:13 INFO mapreduce.Job: map 100% reduce 85%
15/10/23 06:02:14 INFO mapreduce.Job: map 100% reduce 91%
15/10/23 06:02:15 INFO mapreduce.Job: map 100% reduce 97%
15/10/23 06:02:16 INFO mapreduce.Job: map 100% reduce 100%
15/10/23 06:02:19 INFO mapreduce.Job: Job job_1445519089673_0106 completed
successfully
15/10/23 06:02:19 INFO mapreduce.Job: Counters: 51
File System Counters
FILE: Number of bytes read=6255686056
FILE: Number of bytes written=12606758779
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=614122608
HDFS: Number of bytes written=78045672
HDFS: Number of read operations=2886
HDFS: Number of large read operations=0
HDFS: Number of write operations=1416
Job Counters
Launched map tasks=97
Launched reduce tasks=513
Data-local map tasks=84
Rack-local map tasks=13
Total time spent by all maps in occupied slots (ms)=3350846
Total time spent by all reduces in occupied slots (ms)=6020424
Total time spent by all map tasks (ms)=3350846
Total time spent by all reduce tasks (ms)=6020424
Total vcore-seconds taken by all map tasks=3350846
Total vcore-seconds taken by all reduce tasks=6020424
Total megabyte-seconds taken by all map tasks=24018864128
Total megabyte-seconds taken by all reduce tasks=43154399232
Map-Reduce Framework
Map input records=4957008
Map output records=54527088
Map output bytes=6146628802
Map output materialized bytes=6255981544
Input split bytes=12901
Combine input records=0
Combine output records=0
Reduce input groups=4957008
Reduce shuffle bytes=6255981544
Reduce input records=54527088
Reduce output records=54527088
Spilled Records=109054176
Shuffled Maps =49761
Failed Shuffles=0
Merged Map outputs=49761
GC time elapsed (ms)=432301
CPU time spent (ms)=6820780
Physical memory (bytes) snapshot=420588826624
Virtual memory (bytes) snapshot=4303403663360
Total committed heap usage (bytes)=1420226461696
Phoenix MapReduce Import
Upserts Done=4957008
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=612651700
File Output Format Counters
Bytes Written=78045672
15/10/23 06:02:19 INFO mapreduce.CsvBulkLoadTool: Loading HFiles from
/tmp/caa620d5-f3fe-4523-a04e-51cf163f690d/<Table Name>
15/10/23 06:02:19 WARN hbase.HBaseConfiguration: Config option
"hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
15/10/23 06:02:19 WARN mapreduce.LoadIncrementalHFiles: Skipping non-directory
hdfs://surus/tmp/caa620d5-f3fe-4523-a04e-51cf163f690d/<Table Name>/_SUCCESS
15/10/23 06:02:19 WARN hbase.HBaseConfiguration: Config option
"hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
15/10/23 06:02:19 INFO util.ChecksumType: Checksum using
org.apache.hadoop.util.PureJavaCrc32
15/10/23 06:02:19 INFO util.ChecksumType: Checksum can use
org.apache.hadoop.util.PureJavaCrc32C
15/10/23 06:02:19 INFO compress.CodecPool: Got brand-new decompressor [.gz]
15/10/23 06:02:19 INFO compress.CodecPool: Got brand-new decompressor [.gz]
15/10/23 06:02:19 INFO compress.CodecPool: Got brand-new decompressor [.gz]
15/10/23 06:02:19 INFO compress.CodecPool: Got brand-new decompressor [.gz]
15/10/23 06:02:19 INFO compress.CodecPool: Got brand-new decompressor [.gz]
15/10/23 06:02:19 INFO compress.CodecPool: Got brand-new decompressor [.gz]
15/10/23 06:02:19 INFO mapreduce.LoadIncrementalHFiles: Trying to load
hfile=hdfs://surus/tmp/caa620d5-f3fe-4523-a04e-51cf163f690d/<Table
Name>/0/0078777f27b24fbda6bb1db054e5f915
first=268-CSTX_B77653169_80221136\x00\xD5[z
last=268-CSTX_B78594909_82187464\x00\xD6')\xE0
-----Original Message-----
From: Gabriel Reid [mailto:[email protected]]
Sent: Friday, October 23, 2015 9:39 AM
To: [email protected]
Subject: Re: Help With CSVBulkLoadTool
Do you have a stack trace from the log output from when you got this error?
And could you tell me if the table name that is being complained about there is
an index table name?
Tracing through the code, it looks like you could get this exception if an
index table doesn't exist (or somehow isn't available), which would explain how
the data would be getting into your main table.
- Gabriel
On Fri, Oct 23, 2015 at 11:18 AM, Riesland, Zack <[email protected]>
wrote:
> Thanks Gabriel,
>
> From what I can see in the logs, it happens consistently on most if not all
> tables that we import to.
>
> However, it does not appear to actually prevent the data from getting to the
> table.
>
> When I first raised this question, I noticed missing data and saw the error,
> but I have since found another error (bad data of the wrong type) that I
> think was the root cause of my job actually failing.
>
> I certainly never noticed this before, though.
>
> The main things that we have changed since these scripts worked cleanly were
> upgrading our stack and adding new region servers.
>
> Does that help at all?
>
> -----Original Message-----
> From: Gabriel Reid [mailto:[email protected]]
> Sent: Friday, October 23, 2015 1:19 AM
> To: [email protected]
> Subject: Re: Help With CSVBulkLoadTool
>
> Hi Zack,
>
> I can't give you any information about compatibility of a given Phoenix
> version with a given version of HDP (because I don't know).
>
> However, could you give a bit more info on what you're seeing? Are all import
> jobs failing with this error for a given set of tables? Or is this a random
> failure that can happen on any table?
>
> This error looks to me like it would be some kind of configuration issue with
> your cluster(s), but if that's the case then I would expect that you'd be
> getting the same error every time.
>
> - Gabriel
>
> On Wed, Oct 21, 2015 at 2:42 PM, Riesland, Zack <[email protected]>
> wrote:
>> Hello,
>>
>>
>>
>> We recently upgraded our Hadoop stack from HDP 2.2.0 to 2.2.8
>>
>>
>>
>> The phoenix version (phoenix-4.2.0.2.2.8.0) and HBase version
>> (0.98.4.2.2.8.0) did not change (from what I can tell).
>>
>>
>>
>> However, some of our CSVBulkLoadTool jobs have started to fail.
>>
>>
>>
>> I’m not sure whether this is related to the upgrade or not, but the
>> timing seems suspicious.
>>
>>
>>
>> The particular error I’m seeing is like this:
>>
>>
>>
>> ERROR mapreduce.CsvBulkLoadTool: Import job on table=<table name>
>> failed due to exception:java.lang.IllegalArgumentException: No
>> regions passed
>>
>>
>>
>> The Phoenix table in question has 7 regions and millions of rows in
>> it already.
>>
>>
>>
>> The syntax I’m using is
>>
>>
>>
>> HADOOP_CLASSPATH=/<classpath stuff> hadoop jar <path to
>> phoenix-client.jar> org.apache.phoenix.mapreduce.CsvBulkLoadTool
>> –Dfs.permissions.umask-mode=000 --z <zookeeper quarum> --table <my
>> table> --input <my hdfs file>
>>
>>
>>
>> Can anyone help me understand the solution here?
>>
>>
>>
>> Also, does anyone know the most recent version of Phoenix that is
>> compatible with HDP 2.2.8 / HBase 0.98.4.2.2 ?
>>
>>
>>
>> Thanks!
