[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/433#issuecomment-76966697 @d2r I haved imported backtype.storm.generated.StormTopology in task.clj. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/433#issuecomment-76870913 @d2r I have added [STORM-554] to the title of this pull request and closed #312. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong closed the pull request at: https://github.com/apache/storm/pull/312 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-76868454 @d2r ok, i have closed this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-76647927 @d2r I create a new repository, and pull this request again. see #433. I have filed a JIRA, https://issues.apache.org/jira/browse/STORM-554. you can merge #433. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-74463198 @revans2 I create a new repository, and pull this request again. see https://github.com/apache/storm/pull/433. I have filed a JIRA, https://issues.apache.org/jira/browse/STORM-554. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
GitHub user BuDongDong opened a pull request: https://github.com/apache/storm/pull/433 Update "get-task-object" function, change the type of first param "topology" from ^TopologyContext to ^StormTopology Update "get-task-object" function, change the type of first param "topology" from ^TopologyContext to ^StormTopology. the "get-task-object" function in task.clj,the type of first param "topology" should be ^StormTopology not ^TopologyContext. the "get-task-object" is called by "mk-task-data" function in task.clj, "mk-task-data" is defined as following: (defn mk-task-data [executor-data task-id] (recursive-map :executor-data executor-data :task-id task-id :system-context (system-topology-context (:worker executor-data) executor-data task-id) :user-context (user-topology-context (:worker executor-data) executor-data task-id) :builtin-metrics (builtin-metrics/make-data (:type executor-data)) :tasks-fn (mk-tasks-fn <>) :object (get-task-object (.getRawTopology ^TopologyContext (:system-context <>)) (:component-id executor-data (:system-context <>) return TopologyContext instance, TopologyContext extends GeneralTopologyContext, the TopologyContext instance has StormTopology _topology. âgetRawTopologyâ method of TopologyContext must return StormTopology _topology. (.getRawTopology ^TopologyContext (:system-context <>) return StormTopology instance not TopologyContext instance. so the type of param topology is StormTopology not TopologyContext. at the same time, "get-task-object" call "get_spouts" and "get_bolts" of the param topology. you can find the "get_spouts" and "get_bolts" function are only defined in StormTopology, and StormTopology is not a subclass of TopologyContext. so i think the type of param topology is StormTopology not TopologyContext. You can merge this pull request into a Git repository by running: $ git pull https://github.com/BuDongDong/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #433 commit 0ee865750e8d54c00d2abac81c20c429483bc646 Author: zhangjinlong Date: 2015-02-16T05:49:06Z Update "get-task-object" function in task.clj Update "get-task-object" function, change the type of first param "topology" from ^TopologyContext to ^StormTopology. the "get-task-object" function in task.clj,the type of first param "topology" should be ^StormTopology not ^TopologyContext. the "get-task-object" is called by "mk-task-data" function in task.clj, "mk-task-data" is defined as following: (defn mk-task-data [executor-data task-id] (recursive-map :executor-data executor-data :task-id task-id :system-context (system-topology-context (:worker executor-data) executor-data task-id) :user-context (user-topology-context (:worker executor-data) executor-data task-id) :builtin-metrics (builtin-metrics/make-data (:type executor-data)) :tasks-fn (mk-tasks-fn <>) :object (get-task-object (.getRawTopology ^TopologyContext (:system-context <>)) (:component-id executor-data (:system-context <>) return TopologyContext instance, TopologyContext extends GeneralTopologyContext, the TopologyContext instance has StormTopology _topology. âgetRawTopologyâ method of TopologyContext must return StormTopology _topology. (.getRawTopology ^TopologyContext (:system-context <>) return StormTopology instance not TopologyContext instance. so the type of param topology is StormTopology not TopologyContext. at the same time, "get-task-object" call "get_spouts" and "get_bolts" of the param topology. you can find the "get_spouts" and "get_bolts" function are only defined in StormTopology, and StormTopology is not a subclass of TopologyContext. so i think the type of param topology is StormTopology not TopologyContext. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: update worker.clj-delete "missing-tasks" check...
Github user BuDongDong closed the pull request at: https://github.com/apache/storm/pull/288 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: update worker.clj-delete "missing-tasks" check...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/288#issuecomment-74460727 @revans2 ok, i will close this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: update worker.clj-delete "missing-tasks" check...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/288#issuecomment-69576596 @revans2 I have filed a JIRA(STORM-527) for this, please see https://issues.apache.org/jira/browse/STORM-527 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: update worker.clj-delete "missing-tasks" check...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/288#issuecomment-69574937 @revans2 thank you --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update stats.clj "rolling-window-set" function...
Github user BuDongDong closed the pull request at: https://github.com/apache/storm/pull/348 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update stats.clj "rolling-window-set" function...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/348#issuecomment-67004693 @nathanmarz Could you check this bug, thank you very much --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update stats.clj "rolling-window-set" function...
Github user BuDongDong commented on the pull request:
https://github.com/apache/storm/pull/348#issuecomment-66964691
(defrecord RollingWindow [updater merger extractor bucket-size-secs
num-buckets buckets])
I think that the "bucket-size-secs" of RollingWindow is 20, and the
"num-buckets" of RollingWindow is 30/540/4320. if not exchange the real
argument ânum-bucketsâ and "s" of ârolling-windowâ function, then the
"bucket-size-secs" of RollingWindow is 30/540/4320, and the "num-buckets" of
RollingWindow is 20.
creating CommonStats under not exchange the real argument
ânum-bucketsâ and "s" of ârolling-windowâ function:
(def NUM-STAT-BUCKETS 20)
;; 10 minutes, 3 hours, 1 day
(def STAT-BUCKETS [30 540 4320])
(defn- mk-common-stats
[rate]
(CommonStats.
(atom (apply keyed-counter-rolling-window-set NUM-STAT-BUCKETS
STAT-BUCKETS))
(atom (apply keyed-counter-rolling-window-set NUM-STAT-BUCKETS
STAT-BUCKETS))
rate))
"mk-common-stats" call " keyed-counter-rolling-window-set"
==>(apply keyed-counter-rolling-window-set 20 [30 540 4320])
(defn keyed-counter-rolling-window-set
[num-buckets & bucket-sizes]
(apply rolling-window-set incr-val (partial merge-with +) counter-extract
num-buckets bucket-sizes))
"keyed-counter-rolling-window-set" call "rolling-window-set"
==>(apply rolling-window-set incr-val (partial merge-with +)
counter-extract 20 [30 540 4320])
(defn rolling-window-set [updater merger extractor num-buckets &
bucket-sizes]
(RollingWindowSet. updater extractor (dofor [s bucket-sizes]
(rolling-window updater merger extractor s num-buckets)) nil)
)
"rolling-window-set" call constructor of "RollingWindowSet"
==>(RollingWindowSet. updater extractor (dofor [s [30 540 4320]]
(rolling-window updater merger extractor s 20)) nil)
constructor of "RollingWindowSet" call "rolling-window"
==>(rolling-window updater merger extractor 30 20)
==>(rolling-window updater merger extractor 540 20)
==>(rolling-window updater merger extractor 4320 20)
(defn rolling-window
[updater merger extractor bucket-size-secs num-buckets]
(RollingWindow. updater merger extractor bucket-size-secs num-buckets {}))
"rolling-window" call constructor of "RollingWindow"
==>(RollingWindow. updater merger extractor 30 20 {})
==>(RollingWindow. updater merger extractor 540 20 {})
==>(RollingWindow. updater merger extractor 4320 20 {})
if not exchange the real argument ânum-bucketsâ and "s" of
ârolling-windowâ function, then the "bucket-size-secs" of RollingWindow is
30/540/4320, and the "num-buckets" of RollingWindow is 20.
creating CommonStats under exchange the real argument ânum-bucketsâ
and "s" of ârolling-windowâ function:
==>(RollingWindow. updater merger extractor 20 30 {})
==>(RollingWindow. updater merger extractor 20 540 {})
==>(RollingWindow. updater merger extractor 20 4320 {})
I think the "bucket-size-secs" should represent the size of bucket not the
count of bucket; the ânum-bucketsâ should represent the count of bucket not
the size of bucket. so it is necessary to exchange the real argument
ânum-bucketsâ and "s" of ârolling-windowâ function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Update stats.clj "rolling-window-set" function...
GitHub user BuDongDong opened a pull request: https://github.com/apache/storm/pull/348 Update stats.clj "rolling-window-set" function, exchange the real argume... ...nt "num-buckets" and "s" of "rolling-window" function (defn rolling-window-set [updater merger extractor num-buckets & bucket-sizes] (RollingWindowSet. updater extractor (dofor [s bucket-sizes] (rolling-window updater merger extractor s num-buckets)) nil) ) (defrecord RollingWindow [updater merger extractor bucket-size-secs num-buckets buckets]) if not exchange the real argument ânum-bucketsâ and "s" of ârolling-windowâ function, then the "bucket-size-secs" of RollingWindow is 30/540/4320, and the "num-buckets" of RollingWindow is 20 I think that the "bucket-size-secs" of RollingWindow is 20, and the "num-buckets" of RollingWindow is 30/540/4320. You can merge this pull request into a Git repository by running: $ git pull https://github.com/BuDongDong/storm-1 master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/348.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #348 commit b827d660bf18f711da36012355c7d06e35b926e3 Author: zhangjinlong Date: 2014-12-15T08:50:50Z Update stats.clj "rolling-window-set" function, exchange the real argument "num-buckets" and "s" of "rolling-window" function (defn rolling-window-set [updater merger extractor num-buckets & bucket-sizes] (RollingWindowSet. updater extractor (dofor [s bucket-sizes] (rolling-window updater merger extractor s num-buckets)) nil) ) (defrecord RollingWindow [updater merger extractor bucket-size-secs num-buckets buckets]) if not exchange the real argument ânum-bucketsâ and "s" of ârolling-windowâ function, then the "bucket-size-secs" of RollingWindow is 30/540/4320, and the "num-buckets" of RollingWindow is 20 I think that the "bucket-size-secs" of RollingWindow is 20, and the "num-buckets" of RollingWindow is 30/540/4320. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-63250946 @xiaokang the type of real argument is "StormTopology", however the "get-task-object" function use a error type declaration "TopologyContext", clojure REPL will not raise error. you can see this example: ; SLIME 20100404 user> (defn get_date_time [^String date] (.getTime date)) #'user/get_date_time user> (def today (java.util.Date.)) #'user/today user> (get_date_time today) 1416187016658 user> (def ss (String. "today")) #'user/ss user> (get_date_time ss) the "get_date_time" function use a error type declaration "String" to param date, however, there is no âgetTimeâ method in class âStringâ. in fact, the "getTime" method is defined by "java.util.Date". (get_date_time today), clojure REPL will not raise error; but (get_date_time ss), clojure REPL must raise error "No matching field found: getTime for class java.lang.String". the question of "get-task-object" function is same as "get_date_time". because the type of real argument is "StormTopology", "get-task-object" calls "get_spouts" and "get_bolts" not raise error, but, if the type of real argument is "TopologyContext", "get-task-object" calls "get_spouts" and "get_bolts" must raise error. class TopologyContext do not have "get_spouts" and "get_bolts" method. in such a situation, clojure REPL type check will not raise error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-63206771 @harshach the "get-task-object" is called by "mk-task-data" function in task.clj, "mk-task-data" is defined as following: (defn mk-task-data [executor-data task-id] (recursive-map :executor-data executor-data :task-id task-id :system-context (system-topology-context (:worker executor-data) executor-data task-id) :user-context (user-topology-context (:worker executor-data) executor-data task-id) :builtin-metrics (builtin-metrics/make-data (:type executor-data)) :tasks-fn (mk-tasks-fn <>) :object (get-task-object (.getRawTopology ^TopologyContext (:system-context <>)) (:component-id executor-data (:system-context <>) return TopologyContext instance, TopologyContext extends GeneralTopologyContext, the TopologyContext instance has StormTopology _topology. âgetRawTopologyâ method of TopologyContext must return StormTopology _topology. (.getRawTopology ^TopologyContext (:system-context <>) return StormTopology instance not TopologyContext instance. so the type of param topology is StormTopology not TopologyContext. at the same time, "get-task-object" call "get_spouts" and "get_bolts" of the param topology. you can find the "get_spouts" and "get_bolts" function are only defined in StormTopology, and StormTopology is not a subclass of TopologyContext. so i think the type of param topology is StormTopology not TopologyContext. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-63165315 @harshach I have filed a JIRA, https://issues.apache.org/jira/browse/STORM-554. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
GitHub user BuDongDong opened a pull request: https://github.com/apache/storm/pull/312 Update "get-task-object" function, change the type of first param "topology" from ^TopologyContext to ^StormTopology update "get-task-object" function, change the type of first param "topology" from ^TopologyContext to ^StormTopology You can merge this pull request into a Git repository by running: $ git pull https://github.com/BuDongDong/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/312.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #312 commit 98a1c73d3a090635772131bccf621d384158c36c Author: zhangjinlong Date: 2014-11-13T04:02:10Z Update task.clj update "get-task-object" function, change the type of first param "topology" from ^TopologyContext to ^StormTopology --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update "get-task-object" function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-62844422 (defn- get-task-object [^TopologyContext topology component-id] ... ... ) i think the type of param topology is StormTopology not TopologyContext --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: update worker.clj-delete "missing-tasks" check...
GitHub user BuDongDong opened a pull request: https://github.com/apache/storm/pull/288 update worker.clj-delete "missing-tasks" checking missing-tasks set is created by two times filter my-assignment map, so i think keys(my-assignment) contains missing-tasks set. missing-tasks is always empty. (empty? missing-tasks) always return true. (let [missing-tasks (->> needed-tasks (filter (complement my-assignment)))] (when-not (empty? missing-tasks) (log-warn "Missing assignment for following tasks: " (pr-str missing-tasks)) )) You can merge this pull request into a Git repository by running: $ git pull https://github.com/BuDongDong/incubator-storm patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/288.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #288 commit 4b65d8152527f6770db6d103a7e2950f1a1f5f91 Author: zhangjinlong Date: 2014-10-09T09:48:00Z update worker.clj-delete "missing-tasks" checking missing-tasks set is created by two times filter my-assignment map, so i think keys(my-assignment) contains missing-tasks set. missing-tasks is always empty. (empty? missing-tasks) always return true. (let [missing-tasks (->> needed-tasks (filter (complement my-assignment)))] (when-not (empty? missing-tasks) (log-warn "Missing assignment for following tasks: " (pr-str missing-tasks)) )) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
