[appengine-java] map-reduce
Hi All! I catch the following exception when trying to launch map-reduce job in maven project: AccessControlException: access denied (javax.security.auth.AuthPermission getSubject) at java.security.AccessControlContext.checkPermission(AccessControlContext.java: 355) at java.security.AccessController.checkPermission(AccessController.java: 567) at java.lang.SecurityManager.checkPermission(SecurityManager.java: 549) at com.google.apphosting.runtime.security.CustomSecurityManager.checkPermission(CustomSecurityManager.java: 56) at javax.security.auth.Subject.getSubject(Subject.java:288) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java: 394) at org.apache.hadoop.mapreduce.JobContext.(JobContext.java:80) at com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java: 132) at com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java: 124) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleStart(MapReduceServlet.java: 799) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleStartJob(MapReduceServlet.java: 423) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleCommand(MapReduceServlet.java: 301) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost(MapReduceServlet.java: 238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) What can be the cause? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Map Reduce
I tried using the mapper with namespace entities. The mapper fails to run and complains com.google.appengine.tools.mapreduce.MapReduceServlet handleCommand: Got exception while running command java.lang.RuntimeException: Got an IOException while trying to make splits Caused by: java.io.IOException: No entities in the datastore query to split. Any ways to configure the mapper for working with all namespaces? -Aswath On Mon, May 16, 2011 at 10:18 PM, Stephen Johnson wrote: > I have two questions on the Java version of mapreduce since the docs seem > pretty sparse. > 1.) Is it possible to use mapreduce over a namespace and if so, how do you > configure it? > 2.) Is only inputing entity keys and not the entire values supported on the > Java version and if so, how do I configure that as well? > Thanks, > Stephen > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Map Reduce
I have two questions on the Java version of mapreduce since the docs seem pretty sparse. 1.) Is it possible to use mapreduce over a namespace and if so, how do you configure it? 2.) Is only inputing entity keys and not the entire values supported on the Java version and if so, how do I configure that as well? Thanks, Stephen -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Map-reduce: Out of mapper quota error
Hello,
I am running a mapper on a table with about 600 entities (for a
start). In the dev. environment, this runs fine on all entities.
However, on GAE I can see a "processMapper: Out of mapper quota" error
on /mapreduce/mapperCallback in the logs. Only 2 (and at times 4) of
the /mapperCallback calls succeed - however, when I look into the
datastore, all entities are not yet processed.
I've tried to increase the "mapreduce.mapper.inputprocessingrate", as
well as "mapreduce.mapper.shardcount", but without luck.
Please help.
Regards,
Arun
Log trace below:
01-10 04:01AM 59.443 /mapreduce/mapperCallback 200 48ms 160cpu_ms
90api_cpu_ms 0kb AppEngine-Google; (+http://code.google.com/appengine)
See details
0.1.0.2 - - [10/Jan/2011:04:01:59 -0800] "POST /mapreduce/
mapperCallback HTTP/1.1" 200 0 "http://samaanguru-dev.appspot.com/
mapreduce/mapperCallback" "AppEngine-Google; (+http://code.google.com/
appengine)" "samaanguru-dev.appspot.com" ms=48 cpu_ms=160
api_cpu_ms=90 cpm_usd=0.004507 queue_name=default task_name=worker-
attempt-12946609185070f6bdf701e18498d94d10cddf76406af-0001-
m-00-1--3
I 01-10 04:01AM 59.464
com.google.appengine.tools.mapreduce.DatastoreInputSplit
readFields: Initialized DatastoreInputSplit
TransDaySliceTmp("2e1e4f633a53710dc31fe5aae82ee50c")
TransDaySliceTmp("f79ab5df13d2fa8aa59ff2072f570570")
I 01-10 04:01AM 59.467
com.google.appengine.tools.mapreduce.MapReduceServlet
processMapper: Out of mapper quota. Aborting request until quota is
replenished. Consider increasing mapreduce.mapper.inputprocessingrate
(default 1000) if you would like your mapper job to complete faster.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Map Reduce
Hi, I am curious about what will be possible with the planned map reduce functionality. Could this be used as the basis of a free text search or facetted break down of search results? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
