Hi Jerry, I am not familiar with the pull operation on the Storm website... Can you do it when you get a chance?
Also, I wonder how can I accurately specify the resource requirement, especially the cpu demand, for each task? Best, Wuyang ᐧ On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng <jerry.boyang.p...@gmail.com> wrote: > Yup, an error in the documentation > > Feel free to submit a pull request to fix the documentation. > > Best, > > Jerry > > On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu> > wrote: > >> Ok, I find it works by adding >> storm.scheduler: org.apache.storm.scheduler.res >> ource.ResourceAwareScheduler >> >> instead of adding >> >> storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler” >> >> suggested by the website.. >> >> ᐧ >> >> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu> >> wrote: >> >>> When I redo the procedure, here is the more log >>> >>> *2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom >>> scheduler: “org.apache.storm.scheduler.re >>> <http://org.apache.storm.scheduler.re>source.ResourceAwareScheduler”* >>> *2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on >>> initialization of server service-handler* >>> *java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re >>> <http://org.apache.storm.scheduler.re>source.ResourceAwareScheduler”* >>> * at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> ~[?:1.8.0_151]* >>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> ~[?:1.8.0_151]* >>> * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) >>> ~[?:1.8.0_151]* >>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> ~[?:1.8.0_151]* >>> * at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]* >>> * at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]* >>> * at org.apache.storm.util$new_instance.invoke(util.clj:1027) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl >>> <http://nimbus.cl>j:127) ~[storm-core-1.1.1.jar:1.1.1]* >>> * at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at >>> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto____11008.invoke(nimbus.clj:2451) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]* >>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]* >>> * at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]* >>> * at >>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]* >>> * at >>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]* >>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]* >>> * at org.apache.storm.daemon.nimbus.main(Unknown Source) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> *2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: >>> ("Error on initialization")* >>> *java.lang.RuntimeException: ("Error on initialization")* >>> * at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]* >>> * at >>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]* >>> * at >>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]* >>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]* >>> * at org.apache.storm.daemon.nimbus.main(Unknown Source) >>> ~[storm-core-1.1.1.jar:1.1.1]* >>> >>> I download a binary version of Storm 1.1.1 from >>> http://www.apache.org/dyn/closer.lua/storm/apache-storm >>> -1.1.1/apache-storm-1.1.1.tar.gz. >>> ᐧ >>> >>> On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu >>> > wrote: >>> >>>> Hi Jerry, >>>> >>>> I just retry to use the resource aware scheduler in a single node with >>>> 4 slots without any further settings. The scheduler still kills the nimbus >>>> process. Please find nimbus.log and storm.yaml attached. >>>> >>>> Without add the line >>>> >>>> storm.scheduler: >>>> “org.apache.storm.scheduler.resource.ResourceAwareScheduler” >>>> >>>> in storm.yaml, everything works fine. >>>> >>>> I also tried >>>> >>>> conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class); >>>> >>>> The cluster can run, but I cannot see any log related to >>>> DefaultResourceAware. It seems like that it doesn't really run. >>>> >>>> Please take a look at the log and see if I miss anything to enable them >>>> to work? >>>> >>>> Best, >>>> Wuyang >>>> ᐧ >>>> >>>> On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang < >>>> qingshanyou...@gmail.com> wrote: >>>> >>>>> Hi Jerry, >>>>> >>>>> Thank you for the reply. >>>>> >>>>> Please find the storm.yaml below: >>>>> >>>>> ########### These MUST be filled in for a storm configuration >>>>> storm.zookeeper.servers: >>>>> - "10.10.21.1" >>>>> - "10.10.21.2" >>>>> - "10.10.21.3" >>>>> - "10.10.21.4" >>>>> - "10.10.21.5" >>>>> - "10.10.21.6" >>>>> - "10.10.21.7" >>>>> >>>>> nimbus.seeds: ["10.10.21.1"] >>>>> >>>>> storm.zookeeper.port: 2181 >>>>> ###### These may optionally be filled in: >>>>> >>>>> worker.heap.memory.mb: 8192 >>>>> >>>>> supervisor.scheduler.meta: >>>>> tag: 10.10.20.1 >>>>> >>>>> #storm.scheduler: storm.EmptyScheduler >>>>> storm.scheduler: “org.apache.storm.scheduler.re >>>>> source.ResourceAwareScheduler” >>>>> supervisor.memory.capacity.mb: 2048.0 >>>>> supervisor.cpu.capacity: 4000.0 >>>>> supervisor.slots.ports: >>>>> - 6700 >>>>> - 6701 >>>>> - 6702 >>>>> - 6703 >>>>> >>>>> worker.childopts: "-Xmx%HEAP-MEM%m -Xms6000m -XX:+PrintGCDetails >>>>> -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps >>>>> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M >>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump" >>>>> >>>>> On Sat, Feb 10, 2018 at 12:16 PM, Jerry Peng < >>>>> jerry.boyang.p...@gmail.com> wrote: >>>>> >>>>>> Wuyang, >>>>>> >>>>>> Can you provide your whole storm.yaml ? >>>>>> >>>>>> Best, >>>>>> >>>>>> Jerry >>>>>> >>>>>> On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang < >>>>>> wuy...@winlab.rutgers.edu> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have the difficulty with the resource aware scheduler from the >>>>>>> version 1.1.1. >>>>>>> >>>>>>> As suggested by the website, I add the following lines to storm.yaml: >>>>>>> >>>>>>> *storm.scheduler: “org.apache.storm.scheduler.re >>>>>>> <http://org.apache.storm.scheduler.re>source.ResourceAwareScheduler”* >>>>>>> * supervisor.memory.capacity.mb: 2048.0* >>>>>>> * supervisor.cpu.capacity: 400.0* >>>>>>> >>>>>>> Afterwards, I rerun the storm and the nimbus process has been >>>>>>> killed. >>>>>>> The last log I found from nimbus.log is: >>>>>>> o.a.s.d.nimbus main [INFO] Using custom scheduler: “ >>>>>>> org.apache.storm.scheduler.resource.ResourceAwareScheduler”. >>>>>>> >>>>>>> I suppose something goes wrong when it runs the scheduler. >>>>>>> >>>>>>> Do I miss anything to enable this scheduler? >>>>>>> >>>>>>> Best, >>>>>>> Wuyang >>>>>>> ᐧ >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
