Re: Parallel SSH and system monitoring in Clojure
Hi Chris, Cool. Thanks for sharing with the group. I also worked on something similar: https://github.com/drsnyder/gantry. I was getting frustrated with our current Frankenstein version of capistrano and wanted to tinker with building a basic remote execution and deployment tool using clojure. Damon On Mar 15, 2:12 pm, Chris McBride cmm7...@gmail.com wrote: Hi, I releases two simple clojure libraries to help running commands via SSH on multiple servers. Hopefully someone will find it useful. http://info.rjmetrics.com/blog/bid/54114/Parallel-SSH-and-system-moni... https://github.com/RJMetrics/Parallel-SSH https://github.com/RJMetrics/Server-Stats Best, Chris McBride -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gantry system administration and deploy tool written in clojure
Hi Martin,
Thanks! Under the hood gantry uses rsync for the upload. See
https://github.com/drsnyder/gantry/blob/master/src/gantry/core.clj#L148.
Damon
On Jul 8, 1:30 am, martin_clausen martin.clau...@gmail.com wrote:
Looks very nice. You should consider supporting rsync for the upload
command.
On Jul 6, 10:36 pm, Damon Snyder drsny...@gmail.com wrote:
Hello,
I wanted to let everyone know about a tool I have been working on. I'm
calling it gantry (a type of crane that I see every day on my commute
on BART to SF and inspired by crane). I started working on it after
seeing crane (https://github.com/getwoven/crane) and seeing the
possibilities while also suffering the frustration of using capistrano
for deployment of a few non-rails apps (PHP to be specific). I was
longing for something simpler and found fabric (http://fabfile.org)
but I was disappointed when I learned that all commands are executed
sequentially (this has since been fixed I believe). I also wanted to
spend more time with clojure and see how it could be used for system
administration and scripting.
Gantry comes in two parts. The first is a command line tool and task
macro. You define tasks and the commands specified in those tasks
(provided by gantry.run) are run in parallel on each host using agents/
send-off. Here is a quick start:
$ curl -Lhttps://github.com/downloads/drsnyder/gantry/gantry
gantry; chmod +x gantry
$ cat gantryfile
(use 'gantry.run)
(task hello
(run (format echo \hello %s at $(hostname)\ (:name (get-args
(get-config))
^D
$ ./gantry -H one.myhost.com,two.myhost.com -t hello -s name=bob
INFO [one.myhost.com] hello bob at one.myhost.com
INFO [two.myhost.com] hello bob at two.myhost.com
The second part is a library for running commands remotely on one or
more hosts. Here are some examples:
(use 'gantry.run)
(remote host.com yum install -y atop)
(remote host.com yum install -y atop {:user deployer})
(remote* [host.com, host2.com] yum install -y atop {:user
deployer})
(upload host.com filea /tmp)
(upload host.com [filea, fileb] /tmp {:port 222})
(upload* [host.com, host2.com] [filea, fileb] /tmp {:id /
home/deployer/my-key})
The jar is in clojars. Use [gantry 0.0.1-SNAPSHOT] in your
dependencies. Seehttps://github.com/drsnyder/gantryforthe
documentation and source.
Feedback welcome!
Damon
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
ANN: Gantry system administration and deploy tool written in clojure
Hello,
I wanted to let everyone know about a tool I have been working on. I'm
calling it gantry (a type of crane that I see every day on my commute
on BART to SF and inspired by crane). I started working on it after
seeing crane (https://github.com/getwoven/crane) and seeing the
possibilities while also suffering the frustration of using capistrano
for deployment of a few non-rails apps (PHP to be specific). I was
longing for something simpler and found fabric (http://fabfile.org)
but I was disappointed when I learned that all commands are executed
sequentially (this has since been fixed I believe). I also wanted to
spend more time with clojure and see how it could be used for system
administration and scripting.
Gantry comes in two parts. The first is a command line tool and task
macro. You define tasks and the commands specified in those tasks
(provided by gantry.run) are run in parallel on each host using agents/
send-off. Here is a quick start:
$ curl -L https://github.com/downloads/drsnyder/gantry/gantry
gantry; chmod +x gantry
$ cat gantryfile
(use 'gantry.run)
(task hello
(run (format echo \hello %s at $(hostname)\ (:name (get-args
(get-config))
^D
$ ./gantry -H one.myhost.com,two.myhost.com -t hello -s name=bob
INFO [one.myhost.com] hello bob at one.myhost.com
INFO [two.myhost.com] hello bob at two.myhost.com
The second part is a library for running commands remotely on one or
more hosts. Here are some examples:
(use 'gantry.run)
(remote host.com yum install -y atop)
(remote host.com yum install -y atop {:user deployer})
(remote* [host.com, host2.com] yum install -y atop {:user
deployer})
(upload host.com filea /tmp)
(upload host.com [filea, fileb] /tmp {:port 222})
(upload* [host.com, host2.com] [filea, fileb] /tmp {:id /
home/deployer/my-key})
The jar is in clojars. Use [gantry 0.0.1-SNAPSHOT] in your
dependencies. See https://github.com/drsnyder/gantry for the
documentation and source.
Feedback welcome!
Damon
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: Best practice for distributing a standalone clojure command line utility
Hi Miki,
Thats an interesting idea that I had not thought of. It nicely
encapsulates everything in one package.
Thanks!
Damon
On Jun 16, 6:15 pm, Miki miki.teb...@gmail.com wrote:
One more option is to embed the jar in base64 encoding in a script, extract
the jar to a temp location and run it.
The following Perl script does that (it uses the base64 executable, but you
can probably use MIME::Base64 as well).
#!/usr/bin/env perl
$jar = /tmp/lp-$ENV{USER}.jar;
unless (-e $jar) {
open DECODE, |base64 -d $jar or die can't open base64 pipe;
print DECODE DATA;}
exec java, -jar, $jar, @ARGV
__DATA__
YOUR BASE64 ENCODED JAR GOES HERE
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Best practice for distributing a standalone clojure command line utility
Hi Everyone, I'm have a side project that I'm working on that I want to distribute as a standalone script. This is probably best illustrated as an example. What I would like to be able to do is give users a script, so they can do: gantry -H example.host.com -f examples/tasks.clj uptime Instead of java -jar gantry-0.0.1-SNAPSHOT-standalone.jar -H example.host.com - f examples/tasks.clj uptime What I'm curious about is what is the best practice within the community for doing this? Backing up for a minute, to me, the first invokation would be the preferred in my mind. Perhaps the second isn't so bad if you are used to interacting with jar files in this way. I think I would prefer that the details of running java be abstracted away from the command line user's experience. A few approaches I have thought about: 1) Upload the stand alone jar to github and have the gantry script pull down the standalone into ~/.gantry and construct the command line. 2) Create a cake wrapper (not preferred because it introduces a dependency). 3) Forget the wrapper and just distribute the jar. The project I am talking about is here: https://github.com/drsnyder/gantry. Thanks, Damon -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Properly including clojure-contrib
Hi cmn, I think if you add clojure.contrib.string to your use or simply add (:use clojure.contrib.string). I think that should fix it. Damon On Jun 16, 12:16 pm, octopusgrabbus octopusgrab...@gmail.com wrote: What is the proper way to :use clojure contrib so split resolves as a symbol? ns test-csv (:gen-class) (:import (java.io BufferedReader FileReader StringReader)) (:use clojure-csv.core) (:use [clojure.contrib.def])) (defn process-file [file-name] (with-open [br (BufferedReader. (FileReader. file-name))] (println (split , (line-seq br) (defn -main [ args] (process-file resultset.csv)) Thanks. cmn -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Question about when to use protocol+record and compilation warnings
Thanks everyone for all of the feedback. I think I have a solution to the warnings and if I understand deftype/defrecord, I should be able to replace defrecord with deftype in my implementation. I'll give it a try and report back when I have a chance. Thanks, Damon On Dec 26, 7:31 pm, David Nolen dnolen.li...@gmail.com wrote: On Sun, Dec 26, 2010 at 9:00 PM, Ken Wesson kwess...@gmail.com wrote: On Sun, Dec 26, 2010 at 9:25 PM, Alex Osborne a...@meshy.org wrote: Ken Wesson kwess...@gmail.com writes: Actually you don't need to AOT compile records or types. They work fine for interactive development. Eh. That's not what I saw written elsewhere. Or is it just protocols? Though usually those are used hand-in-hand with records. Perhaps you're thinking of gen-class? No. Protocols are also fine for interactive development. Someone here definitely said, recently and specifically, that at least one of the feature-complex around deftype/defprotocol/defrecord required AOT compilation. definterface/deftype/defprotocol/et al do not require AOT. structmaps are a legacy feature. defrecord is the way to go now. I've encountered some odd behavior at the SLIME REPL on occasion, but I haven't yet pinpointed a specific flow to recreate. It wasn't serious enough to impede interactive development. David -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Question about when to use protocol+record and compilation warnings
This worked as expected. I just replaced defrecord with deftype here https://github.com/drsnyder/beanstalk/blob/82f301f1f825bb05aa14d85a220ec57c1dea61b2/src/beanstalk/core.clj#L117 and re-ran my tests. The suggestion from Baishampayan to add (:refer-clojure :exclude [read peek use]) also worked. I added this to the core.clj and the tests and there are no warnings. Thanks! Damon On Dec 28, 9:45 pm, Damon Snyder drsny...@gmail.com wrote: Thanks everyone for all of the feedback. I think I have a solution to the warnings and if I understand deftype/defrecord, I should be able to replace defrecord with deftype in my implementation. I'll give it a try and report back when I have a chance. Thanks, Damon On Dec 26, 7:31 pm, David Nolen dnolen.li...@gmail.com wrote: On Sun, Dec 26, 2010 at 9:00 PM, Ken Wesson kwess...@gmail.com wrote: On Sun, Dec 26, 2010 at 9:25 PM, Alex Osborne a...@meshy.org wrote: Ken Wesson kwess...@gmail.com writes: Actually you don't need to AOT compile records or types. They work fine for interactive development. Eh. That's not what I saw written elsewhere. Or is it just protocols? Though usually those are used hand-in-hand with records. Perhaps you're thinking of gen-class? No. Protocols are also fine for interactive development. Someone here definitely said, recently and specifically, that at least one of the feature-complex around deftype/defprotocol/defrecord required AOT compilation. definterface/deftype/defprotocol/et al do not require AOT. structmaps are a legacy feature. defrecord is the way to go now. I've encountered some odd behavior at the SLIME REPL on occasion, but I haven't yet pinpointed a specific flow to recreate. It wasn't serious enough to impede interactive development. David -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Question about when to use protocol+record and compilation warnings
Hello, In an effort to learn a little more about clojure (and possibly introduce it at work) I decided to write a native client for the Beanstalk work queue. See http://kr.github.com/beanstalkd/ for more information about the queue and https://github.com/drsnyder/beanstalk for the client. One of the decisions I wasn't sure about was whether to use a protocol or a struct map for the (socket, reader, writer) tuple. I started using a struct-map and then switched over to defprotocol/defrecord. See https://github.com/drsnyder/beanstalk/blob/82f301f1f825bb05aa14d85a220ec57c1dea61b2/src/beanstalk/core.clj#L117 for the definition. Is one preferred over the other (struct vs record)? Using the protocol +record idiom maps really well to the queue protocol-- you end up with statements like (.put object params), (.reserve object) etc which are very clear and intuitive from the developers perspective. It creates a one-to-one mapping to the beanstalkd protocol commands. If I were using a struct-map I might have methods like (beanstalk-put object params) etc. But one issue I encountered with defprotocol is that there appears to be a possible symbol table/space issue. When I compile, I get warnings like the following: Warning: protocol #'beanstalk.core/BeanstalkObject is overwriting function read Warning: protocol #'beanstalk.core/BeanstalkObject is overwriting function peek Warning: protocol #'beanstalk.core/BeanstalkObject is overwriting function use WARNING: read already refers to: #'clojure.core/read in namespace: beanstalk.core, being replaced by: #'beanstalk.core/read WARNING: peek already refers to: #'clojure.core/peek in namespace: beanstalk.core, being replaced by: #'beanstalk.core/peek WARNING: use already refers to: #'clojure.core/use in namespace: beanstalk.core, being replaced by: #'beanstalk.core/use WARNING: read already refers to: #'clojure.core/read in namespace: beanstalk.core-test, being replaced by: #'beanstalk.core/read WARNING: peek already refers to: #'clojure.core/peek in namespace: beanstalk.core-test, being replaced by: #'beanstalk.core/peek WARNING: use already refers to: #'clojure.core/use in namespace: beanstalk.core-test, being replaced by: #'beanstalk.core/use Is there something I'm can do to prevent these collisions aside from using a different name for the method? Thanks, Damon -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Feedback on idiomatic clojure
Hi Meikel, Nicolas, and Justin,
Thank you for the great feedback! I learned a lot. I was puzzled about
(update-in (update-in)) and after doing that the - operator makes a
lot of sense. The reduce is clever and fits nicely as well.
I dropped the function that read in the lines of the file and used
read-lines instead. Below is what I put together after the feedback.
Thanks,
Damon
PS- I was mistaken before. The first version was taking over a minute.
This version takes about 15s.
(ns clojure.example.anagrams
(:require clojure.contrib.duck-streams)
(:gen-class))
(defn str-sort[string]
(when string
(apply str (sort string
(defn str-to-lower[string]
(when string
(.toLowerCase string)))
(defn anagram-add[anagrams akey word]
(if (contains? anagrams akey)
(- anagrams
(update-in [akey :count] inc)
(update-in [akey :words] conj word))
(assoc anagrams akey {:count 1 :words [word]})))
(def normalise (comp str-to-lower str-sort))
(defn build-anagrams[words]
(reduce #(anagram-add %1 (normalise %2) %2) {} words))
(defn print-anagram[v]
(println (str (:count (second v)) (first v) : (:words (second
v)
(defn print-anagrams[ana]
(doseq [v ana]
(print-anagram v)))
(defn anagram-key[elem]
(- (:count (second elem
;(def *words* (f-to-seq /usr/share/dict/web2))
;(def *anagrams* (sort-by anagram-key (build-anagrams *words*)))
;(print-anagrams (take 10 *anagrams*))
(defn -main[file]
(time (print-anagrams
(take 10
(sort-by anagram-key
(build-anagrams
(clojure.contrib.duck-streams/read-lines
file)))
On Aug 19, 5:41 am, Meikel Brandmeyer m...@kotka.de wrote:
Hi,
here my turn. Comments inline. Hope this helps.
Sincerely
Meikel
(defn f-to-seq
[file]
(with-open [rdr (java.io.BufferedReader.
(java.io.FileReader. file))]
(doall (line-seq rdr
; Yes. doall is required here. Alternatively you can wrap the whole
thing
; in the with-open. Then you can process also larger files without
keeping
; all data in memory. YMMV.
(defn str-sort
[string]
(when string
(apply str (sort string
; One can also skip the when here. Then nil input returns an empty
string.
; Whether that is better depends on the usage of the return value.
YMMV.
; (if (nil? x) x ...) is usually written as (when x ...).
(defn str-to-lower
[string]
(when string
(.toLowerCase string)))
(defn anagram-add
[anagrams akey word]
(if (contains? anagrams akey)
(- anagrams
(update-in [akey :count] inc)
(update-in [akey :words] conj word))
(assoc anagrams akey {:count 1 :words [word]})))
; I would prefer vectors over lists to store stuff. You get nice
; things for free, like O(1) appending, O(1) reverse, O(1) random
; access, ...
(defn word
[s]
(first s))
; or: (def word first)
; not needed for my solution
(def normalise (comp str-to-lower str-sort))
(defn build-anagrams
[words]
(reduce #(anagram-add %1 (normalise %2) %2) {} words))
; Looping with an accumulator, just returning the latter when the
; input is exhausted cries for reduce.
(defn print-anagram
[v]
(println (str (:count (second v)) (first v) : (:words (second
v)
; one could use destructuring to access things.
(defn print-anagram
[anagram]
(let [[normal-form {:keys [count words]}] anagram]
(println (str count normal-form : words
(defn print-anagrams
[ana]
(doseq [v ana]
(print-anagram v)))
; more descriptive names would be good, I guess
(defn anagram-key
[elem]
(- (:count (second elem
; the minus should take care of the reverse?
(def *words* (f-to-seq /usr/share/dict/web2))
(def *anagrams* (sort-by anagram-key (build-anagrams *words*)))
(print-anagrams (take 10 *anagrams*))
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
