Thanks Gabriel for quick response,
You are right about my Hadoop Config, its somehow using local
filesystem. (let me find in CDH on how to change this)
So for tool "--output" is optional arg, so should work without that
argument.
But in that case we are not at all able to loadHfiles. Tool get
stuck at that for 15-20 mins and then fails without ANY record in Phoenix.
See following logs..
2015-09-23 19:09:57,618 [phoenix-2-thread-0] INFO
org.apache.phoenix.mapreduce.CsvBulkLoadTool - Loading HFiles from
/tmp/a87eaa26-8cf4-4589-9ac9-6662e86f8547/EXAMPLE
2015-09-23 19:09:59,261 [phoenix-2-thread-0] WARN
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles - Skipping
non-directory
file:/tmp/a87eaa26-8cf4-4589-9ac9-6662e86f8547/EXAMPLE/_SUCCESS
2015-09-23 19:10:01,469 [LoadIncrementalHFiles-0] INFO
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles - Trying to
load
hfile=file:/tmp/a87eaa26-8cf4-4589-9ac9-6662e86f8547/EXAMPLE/M/4854e10c013944e4ab1195c70ff530ee
first=\x80\x00\x00\x00\x00\x0009 last=\x80\x00\x00\x00\x00\x01\x092
That's why we explicitly provided ---output as hdfs:// and things
atleast worked.
--
Dhruv
On Wednesday 23 September 2015 06:54 PM, Gabriel Reid wrote:
Hi Dhruv,
This is a bug in Phoenix, although it appears that your hadoop
configuration is also somewhat unusual.
As far as I can see, your hadoop configuration is set up to use the
local filesystem, and not hdfs. You can test this by running the
following command:
hadoop dfs -ls /
If that command lists the contents of the root directory on your local
machine, then hadoop is set up to use your local filesystem, and not
HDFS.
The bulk load tool currently uses the file system that is configured
for Hadoop for deleting the (temporary) output directory. However,
because your setup uses the local filesystem by default, but you're
using HDFS via the file paths that you supply to the tool, deleting
the temporary output directory fails.
The quick fix (if feasible) is to set up your hadoop command to use
HDFS as the default file system. However, could you also log this as a
bug in the JIRA (https://issues.apache.org/jira/browse/PHOENIX)
- Gabriel
On Wed, Sep 23, 2015 at 2:45 PM, Dhruv Gohil <[email protected]> wrote:
Hi,
I am able to successfully use BulkLoadTool to load millions of rows in
phoenixTable, but at end of each execution following error occurs. Need your
inputs to make runs full green.
Following is minimal reproduction using EXAMPLE given in documentation.
Env:
CDH 5.3
Hbase 0.98
Phoenix 4.2.2
Running BulkloadTool from one of the HDFS-CDH cluster machine.
Exactly following instructions :
https://phoenix.apache.org/bulk_dataload.html
HADOOP_CLASSPATH=/path/to/hbase-protocol.jar:/path/to/hbase/conf hadoop jar
phoenix-4.2.2-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool
--table EXAMPLE --input
hdfs://cdh3.st.comany.org:8020/data/example.csv --output
hdfs://cdh3.st.comany.org:8020/user/ --zookeeper 10.10.10.1
CsvBulkLoadTool - Import job on table=EXAMPLE failed due to
exception:java.lang.IllegalArgumentException: Wrong FS:
hdfs://cdh3.st.comany.org:8020/user/EXAMPLE, expected: file:///
Job output log:
2015-09-23 17:54:09,633 [phoenix-2-thread-0] INFO
org.apache.hadoop.mapreduce.Job - Job job_local1620809330_0001 completed
successfully
2015-09-23 17:54:09,663 [phoenix-2-thread-0] INFO
org.apache.hadoop.mapreduce.Job - Counters: 33
File System Counters
FILE: Number of bytes read=34018940
FILE: Number of bytes written=34812676
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=68
HDFS: Number of bytes written=1241
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=5
Map-Reduce Framework
Map input records=2
Map output records=6
Map output bytes=330
Map output materialized bytes=348
Input split bytes=117
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=0
Reduce input records=6
Reduce output records=6
Spilled Records=12
Shuffled Maps =0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=736100352
Phoenix MapReduce Import
Upserts Done=2
File Input Format Counters
Bytes Read=34
File Output Format Counters
Bytes Written=1241
2015-09-23 17:54:09,664 [phoenix-2-thread-0] INFO
org.apache.phoenix.mapreduce.CsvBulkLoadTool - Loading HFiles from
hdfs://cdh3.st.comany.org:8020/user/EXAMPLE
2015-09-23 17:54:12,428 [phoenix-2-thread-0] WARN
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles - Skipping
non-directory hdfs://cdh3.st.comany.org:8020/user/EXAMPLE/_SUCCESS
2015-09-23 17:54:16,233 [LoadIncrementalHFiles-0] INFO
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles - Trying to load
hfile=hdfs://cdh3.st.comany.org:8020/user/EXAMPLE/M/1fe6f08c1a2f431ba87e08ce27ec813d
first=\x80\x00\x00\x00\x00\x0009 last=\x80\x00\x00\x00\x00\x01\x092
2015-09-23 17:54:17,898 [phoenix-2-thread-0] INFO
org.apache.phoenix.mapreduce.CsvBulkLoadTool - Incremental load complete for
table=EXAMPLE
2015-09-23 17:54:17,899 [phoenix-2-thread-0] INFO
org.apache.phoenix.mapreduce.CsvBulkLoadTool - Removing output directory
hdfs://cdh3.st.comany.org:8020/user/EXAMPLE
2015-09-23 17:54:17,900 [phoenix-2-thread-0] ERROR
org.apache.phoenix.mapreduce.CsvBulkLoadTool - Import job on table=EXAMPLE
failed due to exception:java.lang.IllegalArgumentException: Wrong FS:
hdfs://cdh3.st.comany.org:8020/user/EXAMPLE, expected: file:///
--
Dhruv
