You can't do some pig operations in windows especially with hadoop 1.x. Following article talks about couple of options(hacks) that you can use to run Pig scripts on windows.
http://simpletoad.blogspot.com/2013/05/pigunit-issue-on-windows.html?m=1 On Jul 13, 2014 12:42 PM, "Krishnan K" <[email protected]> wrote: > Hi Suraj, > > Thanks for replying. > > > \tmp\hadoop-krkrishnamoorthy\mapred\staging\krkrishnamoorthy502928296\.staging > seems to refer to a path in the unix filesystem. I dont have c:\temp. > > it is trying to set the permissions to 700, which should apparently be > possible in the Unix environment. > > I'll try to setup cygwin. Is that all that is required ? > > Thanks! > > > > > On Sun, Jul 13, 2014 at 7:51 AM, Suraj Nayak M <[email protected]> wrote: > > > Hi Krishnan, > > > > Regarding the error, I can see line > > > > 14/07/12 17:55:31 ERROR security.UserGroupInformation: > > PriviledgedActionException as:krkrishnamoorthy cause:java.io.IOException: > > Failed to set permissions of path: > > > > > \tmp\hadoop-krkrishnamoorthy\mapred\staging\krkrishnamoorthy502928296\.staging > > to 0700 > > > > Do you have C:\tmp in C drive ? > > > > I have added my replies to your questions inline below. > > > > > > On Sunday 13 July 2014 09:44 AM, Krishnan K wrote: > > > > Hi, > > > > I'm running a PigScript on my Windows machine. I don't have a hadoop/pig > > environment installed. > > > > Some questions : > > 1. Can I run PigUnit test cases in *Windows *without having any > *hadoop*/*pig > > environment setup *? > > > > You can run PigUnit test cases locally. I have tried in Linux, it works > > and do not require hadoop to be installed. If you have cygwin installed, > > you should also be able to run PigUnit test cases. > > > > 2. Can I run PigUnit testcases in *local *mode through eclipse if I can > > configure the cluster details ? If yes, where can I provide my cluster > > details ? > > > > No need of cluster configuration in local mode > > > > 3. Can I run PigUnit testcases in *mapreduce *mode through eclipse if I > can > > configure the cluster details ? If yes, where can I provide my cluster > > details ? > > > > Copy *.xml from cluster to local machine (in a folder) and add the > folder > > to classpath. (I have not tested this). > > > > 4. Can I build maven jar without running test cases in my Windows > machine > > and deploy them in a cluster having hadoop/pig ? > > > > Yes. U can use* -DskipTests* option in maven goal. Alternatively, If you > > are using eclipse to build maven jar, in the build dialog of > eclipse(where > > you specify goals), you can check the option to skip the tests option. > > > > > > Appreciate your help. > > > > I executed a pigunit test case and it errored out. Please find the log > > below which has error details : > > > > 14/07/12 17:55:30 INFO pigunit.PigTest: Using default local mode > > 14/07/12 17:55:30 INFO executionengine.HExecutionEngine: Connecting to > > hadoop file system at: file:/// > > 14/07/12 17:55:30 INFO pigunit.PigTest: -- Load users from hdfs > > users = LOAD 'src/test/resources/input/users.txt' USING PigStorage(',') > AS > > (id:long, firstName:chararray, lastName:chararray, country:chararray, > > city:chararray, company:chararray); > > > > -- Load ratings from hdfs > > awesomenessRating = LOAD 'src/test/resources/input/rating.txt' USING > > PigStorage(',') AS (userId:long, rating:long); > > > > -- Join records by userId > > joinedRecords = JOIN users BY id, awesomenessRating BY userId; > > > > -- Filter users with awesomenessRating > 150 > > filteredRecords = FILTER joinedRecords BY awesomenessRating::rating > > 150; > > > > -- Generate fields that we are interested in > > generatedRecords = FOREACH filteredRecords GENERATE > > users::id AS id, > > users::firstName AS firstName, > > users::country AS country, > > awesomenessRating::rating AS rating; > > > > -- Store results > > STORE generatedRecords INTO 'src/test/resources/results/awesomeness' > USING > > PigStorage(); > > > > 14/07/12 17:55:30 INFO util.Utils: Default bootup file > > C:\Users\krkrishnamoorthy/.pigbootup not found > > users = LOAD 'src/test/resources/input/users.txt' USING PigStorage(',') > AS > > (id:long, firstName:chararray, lastName:chararray, country:chararray, > > city:chararray, company:chararray); > > --> users = LOAD 'src/test/resources/input/users.txt' USING > PigStorage(',') > > AS > > > (id:long,firstName:chararray,lastName:chararray,country:chararray,city:chararray,company:chararray); > > awesomenessRating = LOAD 'src/test/resources/input/rating.txt' USING > > PigStorage(',') AS (userId:long, rating:long); > > --> awesomenessRating = LOAD > > 'src/test/resources/input/awesomeness-rating.txt' USING PigStorage(',') > AS > > (userId:long, rating:long); > > STORE generatedRecords INTO 'src/test/resources/results/awesomeness' > USING > > PigStorage(); > > --> none > > 14/07/12 17:55:31 INFO pigstats.ScriptState: Pig features used in the > > script: HASH_JOIN > > 14/07/12 17:55:31 INFO optimizer.LogicalPlanOptimizer: > > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, > > DuplicateForEachColumnRewrite, FilterLogicExpressionSimplifier, > > GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, > > LoadTypeCastInserter, MergeFilter, MergeForEach, > > NewPartitionFilterOptimizer, PartitionFilterOptimizer, > > PushDownForEachFlatten, PushUpFilter, SplitFilter, > StreamTypeCastInserter]} > > 14/07/12 17:55:31 INFO mapReduceLayer.MRCompiler: File concatenation > > threshold: 100 optimistic? false > > 14/07/12 17:55:31 INFO > > mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: > > POPackage->POForEach to POJoinPackage > > 14/07/12 17:55:31 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size > > before optimization: 1 > > 14/07/12 17:55:31 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size > > after optimization: 1 > > 14/07/12 17:55:31 INFO pigstats.ScriptState: Pig script settings are > added > > to the job > > 14/07/12 17:55:31 INFO mapReduceLayer.JobControlCompiler: > > mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > > 14/07/12 17:55:31 INFO mapReduceLayer.JobControlCompiler: Setting up > single > > store job > > 14/07/12 17:55:31 INFO data.SchemaTupleFrontend: Key [pig.schematuple] is > > false, will not generate code. > > 14/07/12 17:55:31 INFO data.SchemaTupleFrontend: Starting process to move > > generated code to distributed cache > > 14/07/12 17:55:31 INFO data.SchemaTupleFrontend: Distributed cache not > > supported or needed in local mode. Setting key > [pig.schematuple.local.dir] > > with code temp directory: > > C:\Users\KRKRIS~1\AppData\Local\Temp\1405212931260-0 > > 14/07/12 17:55:31 INFO mapReduceLayer.JobControlCompiler: Reduce phase > > detected, estimating # of required reducers. > > 14/07/12 17:55:31 INFO mapReduceLayer.JobControlCompiler: Using reducer > > estimator: > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator > > 14/07/12 17:55:31 INFO mapReduceLayer.InputSizeReducerEstimator: > > BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=-1 > > 14/07/12 17:55:31 INFO mapReduceLayer.JobControlCompiler: Could not > > estimate number of reducers and no requested or default parallelism set. > > Defaulting to 1 reducer. > > 14/07/12 17:55:31 INFO mapReduceLayer.JobControlCompiler: Setting > > Parallelism to 1 > > 14/07/12 17:55:31 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce > > job(s) waiting for submission. > > 14/07/12 17:55:31 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > 14/07/12 17:55:31 ERROR security.UserGroupInformation: > > PriviledgedActionException as:krkrishnamoorthy cause:java.io.IOException: > > Failed to set permissions of path: > > > \tmp\hadoop-krkrishnamoorthy\mapred\staging\krkrishnamoorthy502928296\.staging > > to 0700 > > 14/07/12 17:55:31 INFO mapReduceLayer.MapReduceLauncher: 0% complete > > 14/07/12 17:55:31 WARN mapReduceLayer.MapReduceLauncher: Ooops! Some job > > has failed! Specify -stop_on_failure if you want Pig to stop immediately > on > > failure. > > 14/07/12 17:55:31 INFO mapReduceLayer.MapReduceLauncher: job null has > > failed! Stop running all dependent jobs > > 14/07/12 17:55:31 INFO mapReduceLayer.MapReduceLauncher: 100% complete > > 14/07/12 17:55:31 WARN mapReduceLayer.Launcher: There is no log file to > > write to. > > 14/07/12 17:55:31 ERROR mapReduceLayer.Launcher: Backend error message > > during job submission > > java.io.IOException: Failed to set permissions of path: > > > \tmp\hadoop-krkrishnamoorthy\mapred\staging\krkrishnamoorthy502928296\.staging > > to 0700 > > at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691) > > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) > > at > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193) > > at > > > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910) > > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) > > at > > > org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:483) > > at > > > org.apache.pig.backend.hadoop20.PigJobControl.mainLoopAction(PigJobControl.java:157) > > at > > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:134) > > at java.lang.Thread.run(Thread.java:744) > > at > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270) > > > > 14/07/12 17:55:31 ERROR pigstats.SimplePigStats: ERROR: Failed to set > > permissions of path: > > > \tmp\hadoop-krkrishnamoorthy\mapred\staging\krkrishnamoorthy502928296\.staging > > to 0700 > > 14/07/12 17:55:31 ERROR pigstats.PigStatsUtil: 1 map reduce job(s) > failed! > > 14/07/12 17:55:31 INFO pigstats.SimplePigStats: Detected Local mode. > Stats > > reported below may be incomplete > > 14/07/12 17:55:31 INFO pigstats.SimplePigStats: Script Statistics: > > > > HadoopVersion PigVersion UserId StartedAt FinishedAt Features > > 1.2.1 0.12.0 krkrishnamoorthy 2014-07-12 17:55:31 2014-07-12 17:55:31 > > HASH_JOIN > > > > Failed! > > > > Failed Jobs: > > JobId Alias Feature Message Outputs > > N/A awesomenessRating,joinedRecords,users HASH_JOIN Message: > > java.io.IOException: Failed to set permissions of path: > > > \tmp\hadoop-krkrishnamoorthy\mapred\staging\krkrishnamoorthy502928296\.staging > > to 0700 > > at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691) > > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) > > at > > > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) > > at > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193) > > at > > > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942) > > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910) > > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) > > at > > > org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:483) > > at > > > org.apache.pig.backend.hadoop20.PigJobControl.mainLoopAction(PigJobControl.java:157) > > at > > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:134) > > at java.lang.Thread.run(Thread.java:744) > > at > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270) > > file:/tmp/temp49116140/tmp1118481539, > > > > Input(s): > > Failed to read data > from"file:///C:/Users/krkrishnamoorthy/workspace/test/pig-unit-example/src/test/resources/input/awesomeness-rating.txt" > > Failed to read data > from"file:///C:/Users/krkrishnamoorthy/workspace/test/pig-unit-example/src/test/resources/input/users.txt" > > > > Output(s): > > Failed to produce result in "file:/tmp/temp49116140/tmp1118481539" > > > > Job DAG: > > null > > > > 14/07/12 17:55:32 INFO mapReduceLayer.MapReduceLauncher: Failed! > > > > > > Thanks, > > Krishnan > > > > > > > > >
