Re: fails to alter table concatenate

2015-06-30 Thread Nitin Pawar 
can you try doing same by changing the query engine from tez to mr1? not
sure if its hive bug or tez bug

On Tue, Jun 30, 2015 at 1:46 PM, patcharee patcharee.thong...@uni.no
wrote:

 Hi,

 I am using hive 0.14. It fails to alter table concatenate occasionally
 (see the exception below). It is strange that it fails from time to time
 not predictable. Is there any suggestion/clue?

 hive alter table 4dim partition(zone=2,z=15,year=2005,month=4)
 CONCATENATE;

 
 VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING FAILED
 KILLED

 
 File MergeFAILED -1  00 -1   0   0

 
 VERTICES: 00/01  [--] 0%ELAPSED TIME:
 1435651968.00 s

 
 Status: Failed
 Vertex failed, vertexName=File Merge,
 vertexId=vertex_1435307579867_0041_1_00, diagnostics=[Vertex
 vertex_1435307579867_0041_1_00 [File Merge] killed/failed due
 to:ROOT_INPUT_INIT_FAILURE, Vertex Input:
 [hdfs://service-10-0.local:8020/apps/hive/warehouse/wrf_tables/4dim/zone=2/z=15/year=2005/month=4]
 initializer failed, vertex=vertex_1435307579867_0041_1_00 [File Merge],
 java.lang.NullPointerException
 at org.apache.hadoop.hive.ql.io
 .HiveInputFormat.init(HiveInputFormat.java:265)
 at org.apache.hadoop.hive.ql.io
 .CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:452)
 at
 org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:441)
 at
 org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
 at
 org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:124)
 at
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
 at
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
 at
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
 ]
 DAG failed due to vertex failure. failedVertices:1 killedVertices:0
 FAILED: Execution Error, return code 2 from
 org.apache.hadoop.hive.ql.exec.DDLTask

 BR,
 Patcharee




-- 
Nitin Pawar

Re: fails to alter table concatenate

2015-06-30 Thread patcharee 

Actually it works on mr. So the problem is from tez. thanks!

BR,
Patcharee

On 30. juni 2015 10:23, Nitin Pawar wrote:
can you try doing same by changing the query engine from tez to mr1? 
not sure if its hive bug or tez bug


On Tue, Jun 30, 2015 at 1:46 PM, patcharee patcharee.thong...@uni.no 
mailto:patcharee.thong...@uni.no wrote:


Hi,

I am using hive 0.14. It fails to alter table concatenate
occasionally (see the exception below). It is strange that it
fails from time to time not predictable. Is there any suggestion/clue?

hive alter table 4dim partition(zone=2,z=15,year=2005,month=4)
CONCATENATE;


VERTICES  STATUS  TOTAL  COMPLETED  RUNNING PENDING
FAILED  KILLED


File MergeFAILED -1  00 -1  0 
 0



VERTICES: 00/01  [--] 0% ELAPSED TIME:
1435651968.00 s


Status: Failed
Vertex failed, vertexName=File Merge,
vertexId=vertex_1435307579867_0041_1_00, diagnostics=[Vertex
vertex_1435307579867_0041_1_00 [File Merge] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input:

[hdfs://service-10-0.local:8020/apps/hive/warehouse/wrf_tables/4dim/zone=2/z=15/year=2005/month=4]
initializer failed, vertex=vertex_1435307579867_0041_1_00 [File
Merge], java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io

http://org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
at org.apache.hadoop.hive.ql.io

http://org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:452)
at

org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:441)
at

org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
at

org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:124)
at

org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at

org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at

org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at

org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
]
DAG failed due to vertex failure. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.DDLTask

BR,
Patcharee




--
Nitin Pawar

fails to alter table concatenate

2015-06-30 Thread patcharee 

Hi,

I am using hive 0.14. It fails to alter table concatenate occasionally 
(see the exception below). It is strange that it fails from time to time 
not predictable. Is there any suggestion/clue?


hive alter table 4dim partition(zone=2,z=15,year=2005,month=4) CONCATENATE;

VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING 
FAILED  KILLED


File MergeFAILED -1  00 -1   0   0

VERTICES: 00/01  [--] 0%ELAPSED TIME: 
1435651968.00 s


Status: Failed
Vertex failed, vertexName=File Merge, 
vertexId=vertex_1435307579867_0041_1_00, diagnostics=[Vertex 
vertex_1435307579867_0041_1_00 [File Merge] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: 
[hdfs://service-10-0.local:8020/apps/hive/warehouse/wrf_tables/4dim/zone=2/z=15/year=2005/month=4] 
initializer failed, vertex=vertex_1435307579867_0041_1_00 [File Merge], 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:452)
at 
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:441)
at 
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
at 
org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:124)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)

at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)
]
DAG failed due to vertex failure. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.DDLTask


BR,
Patcharee

Re: fails to alter table concatenate

2015-06-30 Thread Hitesh Shah 
Move to user@hive. BCC’ed user@tez.

— Hitesh 

On Jun 30, 2015, at 1:44 AM, patcharee patcharee.thong...@uni.no wrote:

 Hi,
 
 I am using hive 0.14 + tez 0.5. It fails to alter table concatenate 
 occasionally (see the exception below). It is strange that it fails from time 
 to time not predictable. However, it works on mr.
 Is there any suggestion/clue?
 
 hive alter table 4dim partition(zone=2,z=15,year=2005,month=4) CONCATENATE;
 
  
VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING FAILED  KILLED
 
  
 File MergeFAILED -1  00 -1 0   0
 
  
 VERTICES: 00/01  [--] 0%ELAPSED TIME: 
 1435651968.00 s
 
  
 Status: Failed
 Vertex failed, vertexName=File Merge, 
 vertexId=vertex_1435307579867_0041_1_00, diagnostics=[Vertex 
 vertex_1435307579867_0041_1_00 [File Merge] killed/failed due 
 to:ROOT_INPUT_INIT_FAILURE, Vertex Input: 
 [hdfs://service-10-0.local:8020/apps/hive/warehouse/wrf_tables/4dim/zone=2/z=15/year=2005/month=4]
  initializer failed, vertex=vertex_1435307579867_0041_1_00 [File Merge], 
 java.lang.NullPointerException
at 
 org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:452)
at 
 org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:441)
at 
 org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
at 
 org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:124)
at 
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at 
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at 
 org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
 ]
 DAG failed due to vertex failure. failedVertices:1 killedVertices:0
 FAILED: Execution Error, return code 2 from 
 org.apache.hadoop.hive.ql.exec.DDLTask
 
 BR,
 Patcharee