There were no issues. The variable was renamed to be more user friendly.
-Gour
On 4/10/15, 3:48 PM, "Nitin Aggarwal" <nitin3588.aggar...@gmail.com> wrote:
>My mistake, we are running slider version 0.50. I believe these configs
>were changed in 0.60 version.
>Also, were there any issues around port allocation that could fix this
>issue, so that I can back-port them to 0.50 ?
>
>On Wed, Apr 1, 2015 at 3:19 PM, Sumit Mohanty <smoha...@hortonworks.com>
>wrote:
>
>> "${SUPERVISOR.ALLOCATED_PORT}{DO_NOT_PROPAGATE}",
>>
>> should be changed to
>>
>> "${SUPERVISOR.ALLOCATED_PORT}{PER_CONTAINER}",
>>
>> As a reference -
>>
>>https://github.com/apache/incubator-slider/blob/develop/app-packages/stor
>>m/appConfig-default.json
>>
>> I think even DEF_ZK_PATH should be DEFAULT_ZK_PATH
>> ________________________________________
>> From: Nitin Aggarwal <nitin3588.aggar...@gmail.com>
>> Sent: Monday, March 30, 2015 10:38 AM
>> To: dev@slider.incubator.apache.org
>> Subject: Re: Invalid port 0 for storm instances
>>
>> Yes, storm package is built internally.
>>
>> App configuration:
>>
>> "appConf" :{
>> "schema" : "http://example.org/specification/v2.0.0";,
>> "metadata" : { },
>> "global" : {
>> "agent.conf" : "/apps/slider/agent/conf/agent.ini",
>> "application.def" :
>>"/apps/slider/app-packages/storm/storm_v0_9_4.zip",
>> "config_types" : "storm-site",
>> "create.default.zookeeper.node" : "true",
>> "env.MALLOC_ARENA_MAX" : "4",
>> "java_home" : "/usr/java/jdk1.7.0_40",
>> "package_list" : "files/storm-0.9.4-SNAPSHOT-bin.tar.gz",
>> "site.fs.default.name" : "hdfs://XXXXX/",
>> "site.fs.defaultFS" : "hdfs://XXXXXX:8020/",
>> "site.global.app_install_dir" : "${AGENT_WORK_ROOT}/app/install",
>> "site.global.app_log_dir" : "/srv/var/hadoop/logs/deathstar",
>> "site.global.app_pid_dir" : "${AGENT_WORK_ROOT}/app/run",
>> "site.global.app_root" :
>> "${AGENT_WORK_ROOT}/app/install/apache-storm-0.9.4-SNAPSHOT",
>> "site.global.app_user" : "yarn",
>> "site.global.ganglia_enabled" : "false",
>> "site.global.ganglia_server_host" : "${NN_HOST}",
>> "site.global.ganglia_server_id" : "Application2",
>> "site.global.ganglia_server_port" : "8668",
>> "site.global.hbase_instance_name" : "XXXXXX",
>> "site.global.opentsdb_server_host" : "XXXXX",
>> "site.global.opentsdb_server_port" : "4242",
>> "site.global.rest_api_admin_port" :
>>"${STORM_REST_API.ALLOCATED_PORT}",
>> "site.global.rest_api_port" : "${STORM_REST_API.ALLOCATED_PORT}",
>> "site.global.security_enabled" : "false",
>> "site.global.storm_instance_name" : "XXXXX",
>> "site.global.user_group" : "hadoop",
>> "site.storm-site.dev.zookeeper.path" :
>> "${AGENT_WORK_ROOT}/app/tmp/dev-storm-zookeeper",
>> "site.storm-site.drpc.childopts" : "-Xmx768m",
>> "site.storm-site.drpc.invocations.port" : "0",
>> "site.storm-site.drpc.port" : "0",
>> "site.storm-site.drpc.queue.size" : "128",
>> "site.storm-site.drpc.request.timeout.secs" : "600",
>> "site.storm-site.drpc.worker.threads" : "64",
>> "site.storm-site.java.library.path" :
>>
>>
>>"/etc/hadoop/conf:/usr/lib/hadoop/lib/native:/usr/local/lib:/opt/local/li
>>b:/usr/lib",
>> "site.storm-site.logviewer.appender.name" : "A1",
>> "site.storm-site.logviewer.childopts" : "-Xmx128m",
>> "site.storm-site.logviewer.port" :
>> "${SUPERVISOR.ALLOCATED_PORT}{DO_NOT_PROPAGATE}",
>> "site.storm-site.nimbus.childopts" : "-Xmx1024m",
>> "site.storm-site.nimbus.cleanup.inbox.freq.secs" : "600",
>> "site.storm-site.nimbus.file.copy.expiration.secs" : "600",
>> "site.storm-site.nimbus.host" : "${NIMBUS_HOST}",
>> "site.storm-site.nimbus.inbox.jar.expiration.secs" : "3600",
>> "site.storm-site.nimbus.monitor.freq.secs" : "10",
>> "site.storm-site.nimbus.reassign" : "true",
>> "site.storm-site.nimbus.supervisor.timeout.secs" : "60",
>> "site.storm-site.nimbus.task.launch.secs" : "120",
>> "site.storm-site.nimbus.task.timeout.secs" : "5",
>> "site.storm-site.nimbus.thrift.max_buffer_size" : "1048576",
>> "site.storm-site.nimbus.thrift.port" : "${NIMBUS.ALLOCATED_PORT}",
>> "site.storm-site.nimbus.topology.validator" :
>> "backtype.storm.nimbus.DefaultTopologyValidator",
>> "site.storm-site.storm.cluster.mode" : "distributed",
>> "site.storm-site.storm.local.dir" :
>>"${AGENT_WORK_ROOT}/app/tmp/storm",
>> "site.storm-site.storm.local.mode.zmq" : "false",
>> "site.storm-site.storm.messaging.netty.buffer_size" : "5242880",
>> "site.storm-site.storm.messaging.netty.client_worker_threads" : "1",
>> "site.storm-site.storm.messaging.netty.max_retries" : "300",
>> "site.storm-site.storm.messaging.netty.max_wait_ms" : "1000",
>> "site.storm-site.storm.messaging.netty.min_wait_ms" : "100",
>> "site.storm-site.storm.messaging.netty.server_worker_threads" : "1",
>> "site.storm-site.storm.messaging.transport" :
>> "backtype.storm.messaging.netty.Context",
>> "site.storm-site.storm.thrift.transport" :
>> "backtype.storm.security.auth.SimpleTransportPlugin",
>> "site.storm-site.storm.zookeeper.connection.timeout" : "15000",
>> "site.storm-site.storm.zookeeper.port" : "2181",
>> "site.storm-site.storm.zookeeper.retry.interval" : "1000",
>> "site.storm-site.storm.zookeeper.retry.intervalceiling.millis" :
>> "30000",
>> "site.storm-site.storm.zookeeper.retry.times" : "5",
>> "site.storm-site.storm.zookeeper.root" : "${DEF_ZK_PATH}",
>> "site.storm-site.storm.zookeeper.servers" : "['${ZK_HOST}']",
>> "site.storm-site.storm.zookeeper.session.timeout" : "20000",
>> "site.storm-site.supervisor.childopts" : "-Xms512m -Xmx512m
>> -XX:PermSize=64m -XX:MaxPermSize=64m",
>> "site.storm-site.supervisor.enable" : "true",
>> "site.storm-site.supervisor.heartbeat.frequency.secs" : "5",
>> "site.storm-site.supervisor.monitor.frequency.secs" : "3",
>> "site.storm-site.supervisor.slots.ports" :
>> "[${SUPERVISOR.ALLOCATED_PORT}{DO_NOT_PROPAGATE}]",
>> "site.storm-site.supervisor.worker.start.timeout.secs" : "120",
>> "site.storm-site.supervisor.worker.timeout.secs" : "5",
>> "site.storm-site.task.heartbeat.frequency.secs" : "3",
>> "site.storm-site.task.refresh.poll.secs" : "10",
>> "site.storm-site.topology.acker.executors" : "null",
>> "site.storm-site.topology.builtin.metrics.bucket.size.secs" : "60",
>> "site.storm-site.topology.debug" : "false",
>> "site.storm-site.topology.disruptor.wait.strategy" :
>> "com.lmax.disruptor.BlockingWaitStrategy",
>> "site.storm-site.topology.enable.message.timeouts" : "true",
>> "site.storm-site.topology.error.throttle.interval.secs" : "10",
>> "site.storm-site.topology.executor.receive.buffer.size" : "1024",
>> "site.storm-site.topology.executor.send.buffer.size" : "1024",
>> "site.storm-site.topology.fall.back.on.java.serialization" : "true",
>> "site.storm-site.topology.kryo.factory" :
>> "backtype.storm.serialization.DefaultKryoFactory",
>> "site.storm-site.topology.max.error.report.per.interval" : "5",
>> "site.storm-site.topology.max.spout.pending" : "null",
>> "site.storm-site.topology.max.task.parallelism" : "null",
>> "site.storm-site.topology.message.timeout.secs" : "30",
>> "site.storm-site.topology.optimize" : "true",
>> "site.storm-site.topology.receiver.buffer.size" : "8",
>> "site.storm-site.topology.skip.missing.kryo.registrations" :
>>"false",
>> "site.storm-site.topology.sleep.spout.wait.strategy.time.ms" : "1",
>> "site.storm-site.topology.spout.wait.strategy" :
>> "backtype.storm.spout.SleepSpoutWaitStrategy",
>> "site.storm-site.topology.state.synchronization.timeout.secs" :
>>"60",
>> "site.storm-site.topology.stats.sample.rate" : "0.05",
>> "site.storm-site.topology.tick.tuple.freq.secs" : "null",
>> "site.storm-site.topology.transfer.buffer.size" : "1024",
>> "site.storm-site.topology.trident.batch.emit.interval.millis" :
>>"500",
>> "site.storm-site.topology.tuple.serializer" :
>> "backtype.storm.serialization.types.ListDelegateSerializer",
>> "site.storm-site.topology.worker.childopts" : "null",
>> "site.storm-site.topology.worker.shared.thread.pool.size" : "4",
>> "site.storm-site.topology.workers" : "1",
>> "site.storm-site.transactional.zookeeper.port" : "null",
>> "site.storm-site.transactional.zookeeper.root" : "/transactional",
>> "site.storm-site.transactional.zookeeper.servers" : "null",
>> "site.storm-site.ui.port" : "${STORM_UI_SERVER.ALLOCATED_PORT}",
>> "site.storm-site.worker.childopts" : "-Xms2048m -Xmx2048m
>> -XX:PermSize=64m -XX:MaxPermSize=64m",
>> "site.storm-site.worker.heartbeat.frequency.secs" : "1",
>> "site.storm-site.zmq.hwm" : "0",
>> "site.storm-site.zmq.linger.millis" : "5000",
>> "site.storm-site.zmq.threads" : "1"
>>
>> },
>>
>> Also, it is working for some supervisors, which hints that parameter
>>should
>> be fine. I am not a lot familiar with slider codebase, but do we rely
>>that
>> nimbus should be up and running before we install supervisors
>> configuration, as we have to replace these markers with actual host and
>> port values ? Or supervisor agents poll these config parameters even
>>later
>> from some service, where nimbus published it after being started?
>>
>> Thanks
>> Nitin
>>
>>
>> On Mon, Mar 30, 2015 at 10:28 AM, Jon Maron <jma...@hortonworks.com>
>> wrote:
>>
>> > We transitioned from ³ALLOCATED_PORT² to ³PER_CONTAINER² - I just
>>can¹t
>> > recall whether that was in the 0.60 timeframe?
>> >
>> > ‹ Jon
>> >
>> > > On Mar 30, 2015, at 1:21 PM, Sumit Mohanty <sumit.moha...@gmail.com>
>> > wrote:
>> > >
>> > > Did you create the storm package yourself? Can you share the
>> > appConfig.json
>> > > you are using?
>> > >
>> > > On Mon, Mar 30, 2015 at 10:07 AM, Nitin Aggarwal <
>> > > nitin3588.aggar...@gmail.com> wrote:
>> > >
>> > >> It's a typo just in the mail. It is replaced correctly for some of
>>the
>> > >> supervisors configuration.
>> > >> I am running slider version 0.60.
>> > >>
>> > >> On Mon, Mar 30, 2015 at 10:04 AM, Sumit Mohanty <
>> > smoha...@hortonworks.com>
>> > >> wrote:
>> > >>
>> > >>> If the exact text was "ALOOCATED_PORT" then replacing them with
>> > >>> "ALLOCATED_PORT" might solve it.
>> > >>>
>> > >>> Otherwise, whats the version of Slider are you using?
>> > >>> ________________________________________
>> > >>> From: Nitin Aggarwal <nitin3588.aggar...@gmail.com>
>> > >>> Sent: Monday, March 30, 2015 9:51 AM
>> > >>> To: dev@slider.incubator.apache.org
>> > >>> Subject: Invalid port 0 for storm instances
>> > >>>
>> > >>> Hi,
>> > >>>
>> > >>> I am trying to run storm cluster with 200 instances, using Slider.
>> > While
>> > >>> submitting topologies to the cluster, some of the workers failed
>>to
>> > start
>> > >>> due to error.
>> > >>>
>> > >>> 2015-03-27T13:27:40.168-0400 b.s.event [ERROR] Error when
>>processing
>> > >> event
>> > >>> java.lang.IllegalArgumentException: invalid port: 0
>> > >>> at
>> > backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:54)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:47)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:43)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at
>> > >>>
>> > >>
>> >
>>
>>backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:3
>>6)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at backtype.storm.utils.Utils.downloadFromMaster(Utils.java:253)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at
>> backtype.storm.daemon.supervisor$fn__6900.invoke(supervisor.clj:482)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at clojure.lang.MultiFn.invoke(MultiFn.java:241)
>> > ~[clojure-1.5.1.jar:na]
>> > >>> at
>> > >>>
>> > >>>
>> > >>
>> >
>>
>>backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6820.inv
>>oke(supervisor.clj:371)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at
>>backtype.storm.event$event_manager$fn__2825.invoke(event.clj:40)
>> > >>> ~[storm-core-0.9.4-SNAPSHOT.jar:0.9.4-SNAPSHOT]
>> > >>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>> > >>> at java.lang.Thread.run(Thread.java:724) [na:1.7.0_40]
>> > >>>
>> > >>> I found that some of the supervisors don't have the correct
>> > >> configurations.
>> > >>> Their configuration, still have markers like ${NIMBUS_HOST},
>> > >>> ${NIMBUS.ALOOCATED_PORT}.
>> > >>>
>> > >>> Are these markers expected in supervisor storm configuration ?
>> > >>>
>> > >>> Thanks
>> > >>> Nitin
>> > >>>
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > thanks
>> > > Sumit
>> >
>> >
>>
