Hi,
Pig is a power tool to run map reduces job but i want to run pig in local
mode.
The same pig works fine in mapreduce mode and same pig runs well in local
mode also when data is present.
But it fails when data is not present in local mode but run well in
mapreduce mode.
When data is not present i have configured path of empty dir like shown
below
A= load '
*hdfs://localhost:9000/user/vishaldonderia/data/DataModelsLocalMode/ProcessDataModelsAction/tmp/1388538000/empty'*
USING PigStorage(',')
*Failed Jobs:*
*JobId Alias Feature Message Outputs*
*N/A
data_records,message_records,msc_voice_data_without_fwding_records,out_roamer_trip_model_domestic_usage_data,trip_model_msc_message_data,trip_model_msc_voice_data,trip_model_sgsn_data,voice_records
MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException:
ERROR 2118: Input path does not exist:
file:/tmp/temp616730105/tmp1274352221*
* at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:275)*
* at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)*
* at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)*
* at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)*
* at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)*
* at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)*
* at java.security.AccessController.doPrivileged(Native Method)*
* at javax.security.auth.Subject.doAs(Subject.java:415)*
* at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)*
* at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)*
* at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)*
* at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)*
* at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)*
* at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
* at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)*
* at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
* at java.lang.reflect.Method.invoke(Method.java:606)*
* at
org.apache.pig.backend.hadoop20.PigJobControl.mainLoopAction(PigJobControl.java:157)*
* at
org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:134)*
* at java.lang.Thread.run(Thread.java:745)*
* at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:279)*
*Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input path does not exist: file:/tmp/temp616730105/tmp1274352221*
* at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)*
* at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigFileInputFormat.listStatus(PigFileInputFormat.java:37)*
* at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)*
* at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:261)*
* ... 20 more*
* file:/tmp/temp616730105/tmp1115492154,*
i*nput(s):*
*Successfully read 0 records from:
"hdfs://localhost:9000/user/vishaldonderia/data/DataModelsLocalMode/ProcessDataModelsAction/tmp/1388538000/empty"*
*Successfully read 0 records from:
"hdfs://localhost:9000/user/vishaldonderia/data/DataModelsLocalMode/ProcessDataModelsAction/tmp/1388538000/empty"*
*Output(s):*
*Failed to produce result in "file:/tmp/temp616730105/tmp1115492154"*
*Counters:*
*Total records written : 0*
*Total bytes written : 0*
*Spillable Memory Manager spill count : 0*
*Total bags proactively spilled: 0*
*Total records proactively spilled: 0*
*Job DAG:*
*job_local410195113_0001 -> null,*
*job_local1526910904_0002 -> null,*
*null*
There is Error in getSplits() function.
Then i run pig in local mode in grunt line by line and i was surpised to
find out that there is one line
D = Union A,B,C;
I am able to dump all 3 alias A , B , C , its works fine but when i am try
to dump D it is giving the same error as above.
All three alias have same schema, no casting problem , not permisson
problem in /tmp dir
Note: same thing works fine in mapreduce mode.
Please Help
--
Thanking You
With Regards
Vishal Donderia
[email protected]
+91-9711556310
