Re: Reviewers needed for new Clojure book!

[Brian]

A bit late to the party but please include me in the list of possible
reviews.
thanks,
BDF.


On Tue, Aug 25, 2015 at 12:06 AM, Akhil Wali 
wrote:

> Thank you everyone for the great response!
> I'll notify Packt and they shall contact anyone who is shortlisted as a
> reviwer.
>
>
> On Monday, August 24, 2015 at 12:16:06 PM UTC+5:30, Akhil Wali wrote:
>>
>> If anyone is interested in being a reviewer for a new book "*Mastering
>> Clojure*" by Packt Publishing, please let me know.
>> Reviewers will be entitled to a 6 montn subscription of PacktLib
>> .
>>
>> Here's the list of topics covered in this title.
>>
>>-
>>- Working with Sequences and Types
>>- Orchestrating Concurrency and Parallelism
>>- Parallelization using Reducers
>>- Writing Macros
>>- Composing Transducers
>>- Using Functors and Monads
>>- Programming with Logic
>>- Asynchronous Programming
>>- Reactive Programming
>>- Working with Tests
>>- Troubleshooting and Best Practices
>>
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: D&D + Clojure?

[Brian]

I'm interested.

For my own uses I would be curious to see how difficult it would be to
update (degrade? retrograde? :) ) to 3.5/Pathfinder.


On Thu, Apr 5, 2018 at 11:04 AM, Davide Taviani  wrote:

> Hey!
> I'm a DM, a user of OrcPub2 and a full-time clojure + cljs developer (for
> data science stuff).
> I'm definitely interested, let me know!
> Also, I play with another colleague, so there's definitely a few more D&D
> + Clojure nerds out there.
>
> Davide
>
>
> On Tuesday, March 27, 2018 at 11:12:25 PM UTC+3, Larry Christensen wrote:
>>
>> I have a Dungeons & Dragons app built in Clojure and ClojureScript. With The
>> Original OrcPub  and The New OrcPub
>>  I have about 300K monthly users and have been
>> used by millions of people over the past few years. I'm looking for people
>> who might want to contribute to building some cool features for a fun app!
>>
>> -Larry Christensen a.k.a RedOrc
>> Supreme Leader
>> OrcPub
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help with idiomatic clojure.

[Brian]

Thanks Colin, Thanks Erik
Exactly what I was looking for.

I've updated the gist with Colin's suggestion and a bit of destructuring.
If this project gets any bigger I will definitely look at vlad
<https://github.com/logaan/vlad> and Prismatic Schema
<https://github.com/Prismatic/schema>.

BDF.

On Thu, Nov 12, 2015 at 12:02 PM, Erik Assum  wrote:

> There is also https://github.com/logaan/vlad which helps with validation.
>
> Erik.
> --
> i farta
>
> Den 12. nov. 2015 kl. 17.12 skrev Colin Yates :
>
> A nicer equivalent form would be:
>
> (cond-> []
>   this-error? (conj “It failed with this error”)
>   that-error? (conj “It failed with that error”))
>
> However, purely for validation there are a few utilities out there
> already. Checkout the ‘Validation’ section on
> http://www.clojure-toolbox.com
>
> Also, in terms of enforcing contracts - Prismatic Schema is highly
> recommended but hard to ‘englishify’ the errors. Failures are considered
> API failures rather than happy-case failures.
>
> On 12 Nov 2015, at 16:09, Brian Forester  wrote:
>
> I'm writing a very small REST application in clojure using compojure and 
> ring.  One problem is that I don't have anyone who can review my work or 
> provide feedback.
>
>
> I've written a small function to validate a simple JSON request.  I'm 
> validating the three values that are in the post and collecting the errors 
> for return.
>
> The core mechanic I've used
>
>  (swap! errors str "'grepString' must not be null or empty.\n"))
>
> does not seem to be a very idiomatic clojure way to solve this simple problem.
>
>
> https://gist.github.com/BDF/8e61daf8fe8b602a248a
>
>
> Any feedback is appreciated.
>
> BDF.
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help with idiomatic clojure.

[Brian]

I think I like 'seq' better than 'empty?'.I'm sure my opinions will
firm up after writing some more clojure.

> Also, in no-errors branch you probably want to return status: 200?
I  picked the function where I knew there was a better way.   This
validation function is called from this bit code.
(POST "/usagelog" req
   (let [errors (validate-request req)]
  (if (seq errors)
 errors
 (ring.util.response/response (grep-log-and-response req)))
   )
)


On Thu, Nov 12, 2015 at 4:08 PM, Colin Yates  wrote:

> One other minor point (if (seq some-sequence) true false) is preferred by
> some (I won’t say more idiomatic) than (if (empty? some-sequence) true
> false). Also, in no-errors branch you probably want to return status: 200?
>
>
> On 12 Nov 2015, at 19:44, Brian  wrote:
>
> Thanks Colin, Thanks Erik
> Exactly what I was looking for.
>
> I've updated the gist with Colin's suggestion and a bit of destructuring.
> If this project gets any bigger I will definitely look at vlad
> <https://github.com/logaan/vlad> and Prismatic Schema
> <https://github.com/Prismatic/schema>.
>
> BDF.
>
> On Thu, Nov 12, 2015 at 12:02 PM, Erik Assum  wrote:
>
>> There is also https://github.com/logaan/vlad which helps with
>> validation.
>>
>> Erik.
>> --
>> i farta
>>
>> Den 12. nov. 2015 kl. 17.12 skrev Colin Yates :
>>
>> A nicer equivalent form would be:
>>
>> (cond-> []
>>   this-error? (conj “It failed with this error”)
>>   that-error? (conj “It failed with that error”))
>>
>> However, purely for validation there are a few utilities out there
>> already. Checkout the ‘Validation’ section on
>> http://www.clojure-toolbox.com
>>
>> Also, in terms of enforcing contracts - Prismatic Schema is highly
>> recommended but hard to ‘englishify’ the errors. Failures are considered
>> API failures rather than happy-case failures.
>>
>> On 12 Nov 2015, at 16:09, Brian Forester 
>> wrote:
>>
>> I'm writing a very small REST application in clojure using compojure and 
>> ring.  One problem is that I don't have anyone who can review my work or 
>> provide feedback.
>>
>>
>> I've written a small function to validate a simple JSON request.  I'm 
>> validating the three values that are in the post and collecting the errors 
>> for return.
>>
>> The core mechanic I've used
>>
>>  (swap! errors str "'grepString' must not be null or empty.\n"))
>>
>> does not seem to be a very idiomatic clojure way to solve this simple 
>> problem.
>>
>>
>> https://gist.github.com/BDF/8e61daf8fe8b602a248a
>>
>>
>> Any feedback is appreciated.
>>
>> BDF.
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to c

REPL Namespace Display

[brian]

Hope this is the appropriate place for this... anyway, very trivial
issue.  I was playing around in an effort to begin to understand the
ns fns & managed to get the current/active ns & REPL displayed ns out
of synch via the following:

user> *ns*
#
user> (in-ns 'howdy)
#
howdy> (clojure.core/refer 'clojure.core)
nil
howdy> *ns*
#
howdy> (remove-ns 'howdy)
#
howdy> *ns*
#
howdy>
howdy> (in-ns 'user)
#
howdy>

Switching to another ns corrects it.

BTW l'm really looking forward to digging into this language as a long
time Java wanna-be Lispy programmer (so thx Rich)

-Brian

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to 
clojure+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: REPL Namespace Display

[brian]

That's correct.  And I couldn't reproduce with the java
clojure.lang.Repl either.  Guess I was thinking it was just a pass
through...  I'm an emacs & slime noob too ;)

-Brian

On Dec 18, 6:00 pm, Chouser  wrote:
> On Thu, Dec 18, 2008 at 6:02 PM, brian  wrote:
>
> > Hope this is the appropriate place for this... anyway, very trivial
> > issue.  I was playing around in an effort to begin to understand the
> > ns fns & managed to get the current/active ns & REPL displayed ns out
> > of synch via the following:
>
> > user> *ns*
> > #
> > user> (in-ns 'howdy)
> > #
> > howdy> (clojure.core/refer 'clojure.core)
> > nil
> > howdy> *ns*
> > #
> > howdy> (remove-ns 'howdy)
> > #
> > howdy> *ns*
> > #
> > howdy>
> > howdy> (in-ns 'user)
> > #
> > howdy>
>
> > Switching to another ns corrects it.
>
> We now have several Repls, and each acts slightly differently.  I
> couldn't reproduce this with clojure.lang.Repl.  I'm guessing from
> your prompt that you're using emacs/slime.  Is that correct?
>
> --Chouser
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to 
clojure+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: Is knowing Java a prerequisite for using Clojure?

[Brian]

As a Java programmer coming to Clojure I would be very interested in the
intended application.  How you approach that from a Clojure perspective is
what I'm interested in.  I've programmed in Lisp in the distant past
(Symbolics Lisp machines) and still have very found memories of that
environment.  Having programmed in C -> C++ -> Java since then ... well I
just miss the edit run test cycle that was so natural with those Lisp
machines.

My own concern is falling back on what I know best and using Java semantics
when they are not appropriate.

With the excellent Bruce Eckel recommendation and your background you might
be able to get by with going directly to the java language specification (
http://java.sun.com/docs/books/jvms/second_edition/html/VMSpecTOC.doc.html).

Brian F.

On Thu, Sep 17, 2009 at 10:59 AM, Terrance Davis
wrote:

>
> I'll add my two cents.
>
> You sound like you already program, so basic Java knowledge is useful
> with Clojure, but not necessary. You can pick up what you need to know
> as you learn Clojure. You will find hooks into Java are a lot more
> pleasant than interfacing with C and C++. I'm saying that one from
> experience.
>
> As for first books on Java,  Look for Bruce Eckel's "Thinking in
> Java" That is absolutely the best book currently available for Java. It
> is more written for beginner programmers, but would likely work for a
> more experienced programmer, too.
>
> -Terrance
>
> Hugh Aguilar wrote:
> > Thanks for the encouragement. I've already got the book.
> >
> > I suppose eventually I will have to learn Java. I have been putting it
> > off because I hear a lot of Java-bashing from programmers, and have
> > also noted that this is generally the impetus for the development of
> > languages such as Clojure and Scala and the dozens of others. On the
> > other hand, Java can't be any more difficult than C or C++ that I
> > already know. With languages such as Factor or Python I am relying on
> > the bindings to C and C++ programs, so with Clojure I would be relying
> > on the bindings to Java programs, which might be an improvement.
> >
> > Can you recommend a book or online resource for learning what I need
> > to know about Java? I saw the book "Learning Java" with the tigers on
> > the cover and it looked pretty good, but also pretty extensive ---
> > quite a lot to digest there, which might be why they put a large meat-
> > eater on the cover.
> >
> > If anybody is interested, I can tell you what my intended application
> > for Clojure is, and you can tell me about how you would go about such
> > a project. I will just be writing toy programs initially, but I do
> > actually have a specific purpose for Clojure in mind.
> >
> > On Sep 16, 11:34 pm, Krukow  wrote:
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: frameworks

[Brian]

I thought the question was looking for the Clojure equivalent to Ruby on
Rails (i.e.  Conjure ).

On Fri, Sep 18, 2009 at 9:54 AM, David Nolen  wrote:

> What do you mean by development frameworks? IDE support?
> On Fri, Sep 18, 2009 at 8:08 AM, demet8  wrote:
>
>>
>> Im new to Clojure. Are there any development frameworks for clojure
>> worth noting yet?
>>
>>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Microsoft drops Software Transactional Memory

[Brian]

Not directly related to Clojure but I thought this would be of interest to
folks:
http://www.infoq.com/news/2010/05/STM-Dropped

http://www.reddit.com/r/programming/comments/c4367/microsofts_experiments_with_software/

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Funding Clojure 2010

[brian]

I have  looked at hundreds of languages and flavors  more as a kind
of hobby these days. I know these comments are getting a  bit
tedious by now, but what I think clojure needs is higher visibility.
Right now only a handful of people know about it. The thought
occurred to me while I was installing another open source language
system today, I (but have only dabbled in clojure a bit so far), was
that they had a visually appealing website. Visual signals send a
strong message. This particular one has a lucious red strawberry 
(hint hint) motif, simple, clean, and appealing, and kind of
lightens up a  drab subject, makes it almost.. well I won't get into
it.  I'm coming at this as a bit of an outsider as have not been a
programmer in a long time since I got involved with start-ups. So
anyways, my point is, if you're looking for mass appeal, eveything
is pretty web-based packaging these days. Forget even the CD idea,
thats old, even for an old guy like me (I don't even have a CD drive
on my computer anymore !). Also a windows installer, and no asking
the user to compile things, make it as shrink wrapped as possible,
with a good online tutorial, and chat  help all bundled together in
a quick install.

Brian




Rich Hickey wrote:
> Funding Clojure 2010
>
> Background
> --
>
> It is important when using open source software that you consider who
> is paying for it, because someone is. There is no such thing as free
> software.
>
> Sometimes open source software is developed under a license with
> undesirable properties (e.g. the GPL), such that people are willing to
> pay for a (proprietary) version of it that is not subject to that
> license. Both Monty Widenius [1] and Richard Stallman [2] have argued
> for the necessity of such a mechanism to fund open source software,
> lest there be insufficient resources for its development. Clojure
> doesn't use the GPL, thus conveying more freedom to its users, but
> precluding me from funding it via dual licensing.
>
> Some companies develop technology as a component of a proprietary
> product or service, absorbing it as a necessary expense, only to
> decide that it is not a core, unique, or advantage-bearing business
> function. They can reduce their costs in ongoing development by open
> sourcing it, deriving benefit from community contributions and letting
> them focus on their core business [3]. It is important to note that
> the bulk of the costs are often in the original development, and are
> paid for by the proprietary product or service. That is not the case
> for Clojure.
>
> Some open source is the product of academic research, and is funded by
> the academic institution and/or research grants [4]. That is not the
> case for Clojure.
>
> Some open source software is (partially) funded by proprietary
> support. It is important to note that often the support income does
> not in fact make it to the people who create the software. Such income
> models work best for support sold to conservative enterprises [5].
> That is not the case for Clojure.
>
> Some companies 'fund' open source software by dedicating some of their
> employees' time, or making investments, in its development. There must
> be some business value to the company for doing so (e.g. it helps them
> sell hardware [6]), and thus is ultimately paid for by their
> proprietary products/services. That is not the case for Clojure.
>
> There *are* companies that make software themselves, whose consumers
> see a value in it and willingly pay to obtain that value. The money
> produced by this process pays the salaries of the people who are
> dedicated to making it, and some profit besides. It's called
> "proprietary software". People pay for proprietary software because
> they have to, but otherwise the scenario is very similar to open
> source - people make software, consumers get value from it. In fact,
> we often get a lot less with proprietary software - vendor lock-in, no
> source etc. Most alarmingly, this is the only model that associates
> value with software itself, and therefore with the people who make it.
>
> Why don't people pay for open source software? Primarily, because they
> don't *have to*. I think also, partially, it is because open source
> software often doesn't have a price tag. I think it should. I'd like
> to pay for open source, and know the money is going to those who
> create it. I'd like companies to *expect* to pay for it. I'd like to
> see people make a living (and even profit!) directly making open
> source, not as a side effect of some other proprietary process, to
> dedicate themselves to it, and not have it be hobb

newbie clojure/eclipse question

[brian]

Hi all,

I installed counterclockwise 0.43 in eclipse following the website 
instructions precisely . Everything seems fine, I  created a simple 
clojure  project, when I run it, I get:

Could not locate Clojure resource on classpath: x.clj

where x.clj is the file I am running (as REPL )

I looked in the run configuration,and x is in the default classpath.

thanks

Brian



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

newbie swank clojure emacs

[brian]

Hi,

I'm trying to follow the directions at:

http://riddell.us/tutorial/slime_swank/slime_swank.html

I'm doing this under cygwin.

I think I did everything correctly, but it seems not to find the 
clojure.jar file although I have it and the clojure-contrib.jar in my 
home directory as configured in the my .emacs file. I am using emacs 23.

I also have the clojure jar's  in my classpath.

this is my .emacs :

;; clojure-mode
(add-to-list 'load-path "~/.emacs.d/elpa/clojure-mode-1.6")
(require 'clojure-mode)

;; swank-clojure
(add-to-list 'load-path "~/opt/swank-clojure/src/emacs")
(add-to-list 'load-path "~/.emacs.d/elpa/swank-clojure-1.1.0")

(setq swank-clojure-jar-path "~/clojure.jar"
 swank-clojure-extra-classpaths (list
 "~/opt/swank-clojure/src/main/clojure"
 "~/clojure-contrib.jar"))

(require 'swank-clojure-autoload)

;; slime
(eval-after-load "slime"
 '(progn (slime-setup '(slime-repl

(add-to-list 'load-path "~/opt/slime")
(require 'slime)
(slime-setup)





this is the error when I call slime





(require 'swank.swank)

(swank.swank/ignore-protocol-version "2010-01-06")

(do (.. java.net.InetAddress getLocalHost getHostAddress) 
nil)(swank.swank/start-server 
"/cygdrive/c/DOCUME~1/brian/LOCALS~1/Temp/slime.2848" :encoding 
"iso-latin-1-unix")


java.lang.NoClassDefFoundError: clojure/main
Caused by: java.lang.ClassNotFoundException: clojure.main
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: clojure.main.  Program will exit.
Exception in thread "main"
Process inferior-lisp exited abnormally with code 1

Thanks
Brian


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: newbie swank clojure emacs

[brian]

Thanks for this, this is very cool indeed!

Eric Lavigne wrote:

Since you are using Windows, you may find Clojure Box easier to install.

http://clojure.bighugh.com/

I followed the riddell.us/.../slime_swank.html tutorial yesterday, and
can confirm that it works well for Ubuntu.


On Sun, Jan 10, 2010 at 5:58 AM, brian  wrote:
  

Hi,

I'm trying to follow the directions at:

http://riddell.us/tutorial/slime_swank/slime_swank.html

I'm doing this under cygwin.

I think I did everything correctly, but it seems not to find the clojure.jar
file although I have it and the clojure-contrib.jar in my home directory as
configured in the my .emacs file. I am using emacs 23.
I also have the clojure jar's  in my classpath.

this is my .emacs :

;; clojure-mode
(add-to-list 'load-path "~/.emacs.d/elpa/clojure-mode-1.6")
(require 'clojure-mode)

;; swank-clojure
(add-to-list 'load-path "~/opt/swank-clojure/src/emacs")
(add-to-list 'load-path "~/.emacs.d/elpa/swank-clojure-1.1.0")

(setq swank-clojure-jar-path "~/clojure.jar"
swank-clojure-extra-classpaths (list
"~/opt/swank-clojure/src/main/clojure"
"~/clojure-contrib.jar"))

(require 'swank-clojure-autoload)

;; slime
(eval-after-load "slime"
 '(progn (slime-setup '(slime-repl

(add-to-list 'load-path "~/opt/slime")
(require 'slime)
(slime-setup)





this is the error when I call slime





(require 'swank.swank)

(swank.swank/ignore-protocol-version "2010-01-06")

(do (.. java.net.InetAddress getLocalHost getHostAddress)
nil)(swank.swank/start-server
"/cygdrive/c/DOCUME~1/brian/LOCALS~1/Temp/slime.2848" :encoding
"iso-latin-1-unix")

java.lang.NoClassDefFoundError: clojure/main
Caused by: java.lang.ClassNotFoundException: clojure.main
  at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
  at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: clojure.main.  Program will exit.
Exception in thread "main"
Process inferior-lisp exited abnormally with code 1

Thanks
Brian



--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en




-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

clojure box "value as variable is void ?"

[brian]

Hi all,

I just installed Clojure Box, for me, marvelous! Everything works as 
described except one small problem though, following these directions 
from the readme.rtf:


"

You can also add jars or source directories from custom locations by 
adding them to your classpath in in your .emacs. To make 
c:/dev/project/src/a.clj and everything in c:/dev/my-lib.jar available, 
put this:


(setq swank-clojure-classpath

(list “c:/dev/project/src” “c:/dev/my-lib.jar”))

"

well, this is (all )my .emacs file, cut and pasted from that file:


(setq swank-clojure-classpath

(list “c:/dev/project/src” “c:/dev/my-lib.jar”))



and I get this error when I bring clojure box up:

Warning (initialization): An error occurred while loading `c:/Documents 
and Settings/brian/.emacs':


Symbol's value as variable is void: “c:/dev/project/src”


of course, these aren't my real paths, but I wanted to try this cut and 
paste to see if that was the problem.


Thanks!
Brian



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

trying to call jar files in clojure box

[brian]

Hi,

I am not sure if this is a Clojure Box problem or a compojure problem 
but in an attempt to try calling external jars files in clojure box,and 
wanting to try compojure, to eliminate any possible path problems, I've 
put compojure and all dependancies in the Clojure Box/lib  directory.


When I try to run

(use 'compojure)

   (defroutes my-app
 (GET "/"
   (html [:h1 "Hello World"]))
 (ANY "*"
   (page-not-found)))

   (run-server {:port 8080}
 "/*" (servlet my-app))


java.lang.Exception: Unable to resolve symbol: defroutes in this context 
(NO_SOURCE_FILE:1)

 [Thrown class clojure.lang.Compiler$CompilerException]

I get the same problem when I setq in my .emacs.


Thanks
Brian

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

classpath in clojure box

[brian]

Hi all,


I put the example code from Programming Clojure book
here:

C:\shcloj-code\code\examples

I verified that "introduction.clj"  is there.

This is cut and pasted from my .emacs file located in c:\program files\brian


(setq swank-clojure-classpath
(list "c:\shcloj-code\code\examples"))
 


but when I enter (require 'introduction)

I get this:

java.io.FileNotFoundException: Could not locate introduction__init.class 
or introduction.clj on classpath:  (NO_SOURCE_FILE:0)

 [Thrown class clojure.lang.Compiler$CompilerException]

Restarts:
0: [ABORT] Return to SLIME's top level.
1: [CAUSE] Throw cause of this exception

Backtrace:
 0: clojure.lang.Compiler.eval(Compiler.java:4543)
 1: clojure.core$eval__3990.invoke(core.clj:1728)
--more--

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: classpath in clojure box

[brian]

Shawn Hoover wrote:


On Tue, Jan 12, 2010 at 10:05 PM, brian <mailto:brw...@gmail.com>> wrote:


Hi all,


I put the example code from Programming Clojure book
here:

C:\shcloj-code\code\examples

I verified that "introduction.clj"  is there.

This is cut and pasted from my .emacs file located in c:\program
files\brian


(setq swank-clojure-classpath
   (list "c:\shcloj-code\code\examples")


Assuming that your HOME environment variable is set to c:\program 
files\brian


yes


and your .emacs is actually being loaded (I mention that because 
normally .emacs is under your Documents and Settings or Users directory,

c:\program files\brian

its here and i double checked it is the one i'm actually editing


but it should respect %HOME%), I think the problem is the \ in your 
classpath need to be escaped. Try converting them to / or \\.



tried both, they don't work

user> (require 'introduction)
nil

user> (use 'introduction)
; Evaluation aborted.


java.lang.Exception: namespace 'introduction' not found (NO_SOURCE_FILE:0)
 [Thrown class clojure.lang.Compiler$CompilerException]

Restarts:
0: [ABORT] Return to SLIME's top level.
1: [CAUSE] Throw cause of this exception

Backtrace:
 0: clojure.lang.Compiler.eval(Compiler.java:4543)
 1: clojure.core$eval__3990.invoke(core.clj:1728)



Shawn


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: classpath in clojure box

[brian]

I was wondering if this is the right statement for setting the external  
classpath?


setq swank-clojure-classpath



Shawn Hoover wrote:


On Tue, Jan 12, 2010 at 10:05 PM, brian <mailto:brw...@gmail.com>> wrote:


Hi all,


I put the example code from Programming Clojure book
here:

C:\shcloj-code\code\examples

I verified that "introduction.clj"  is there.

This is cut and pasted from my .emacs file located in c:\program
files\brian


(setq swank-clojure-classpath
   (list "c:\shcloj-code\code\examples")


Assuming that your HOME environment variable is set to c:\program 
files\brian and your .emacs is actually being loaded (I mention that 
because normally .emacs is under your Documents and Settings or Users 
directory, but it should respect %HOME%), I think the problem is the \ 
in your classpath need to be escaped. Try converting them to / or \\.


Shawn


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: classpath in clojure box

[brian]

Shawn Hoover wrote:


On Wed, Jan 13, 2010 at 8:38 PM, brian <mailto:brw...@gmail.com>> wrote:



I was wondering if this is the right statement for setting the
external  classpath?

setq swank-clojure-classpath


Another way is to use M-x swank-clojure-project. You enter a directory 
and it automatically adds that directory/src to the classpath. You 
could drop all of the shcloj-code\code into, say,

this didn't work

c:\projects\shcloj\src. M-x swank-clojure-project and at the prompt 
c:\projects\shcloj. Then try (require 'examples.introduction).


If that doesn't work, check the value of swank-clojure-classpath after 
the REPL starts: C-h v swank-clojure-classpath. Make sure it looks 
good and then fire up SysInternals ProcMon and filter on *.clj and see 
where the system is really looking.

Shawn,

The above didn't work, but apparently it doesn't pick up my .emacs file, 
which has


(setq swank-clojure-classpath
(list "c:/shcloj-code/code/examples"))

itknows where home is, when it boots up, the default is my home dir, 
where .emacs is



when i do C-h v

swank-clojure-classpath is a variable defined in `swank-clojure.el'.
Its value is
("c:/Program Files/Clojure Box/emacs/site-lisp/../../swank-clojure/src" 
"c:/shcloj-code/code/examples" "c:/Program Files/Clojure 
Box/lib/clojure-contrib.jar" "c:/Program Files/Clojure 
Box/lib/clojure.jar" "c:/Program Files/Clojure 
Box/lib/commons-codec-1.3.jar" "c:/Program Files/Clojure 
Box/lib/commons-fileupload-1.2.1.jar" "c:/Program Files/Clojure 
Box/lib/commons-io-1.4.jar" "c:/Program Files/Clojure 
Box/lib/compojure.jar" "c:/Program Files/Clojure 
Box/lib/grizzly-http-servlet-1.9.10.jar" "c:/Program Files/Clojure 
Box/lib/grizzly-http-webserver-1.9.10.jar" "c:/Program Files/Clojure 
Box/lib/jetty-6.1.15.jar" "c:/Program Files/Clojure 
Box/lib/jetty-util-6.1.15.jar" "c:/Program Files/Clojure 
Box/lib/servlet-api-2.5-20081211.jar")


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: classpath in clojure box

[brian]

Michael Wood wrote:

2010/1/14 brian :
  

Shawn Hoover wrote:


On Wed, Jan 13, 2010 at 8:38 PM, brian mailto:brw...@gmail.com>> wrote:


   I was wondering if this is the right statement for setting the
   external  classpath?

   setq swank-clojure-classpath


Another way is to use M-x swank-clojure-project. You enter a directory and
it automatically adds that directory/src to the classpath. You could drop
all of the shcloj-code\code into, say,
  

this didn't work



c:\projects\shcloj\src. M-x swank-clojure-project and at the prompt
c:\projects\shcloj. Then try (require 'examples.introduction).
  


Just a thought:

Add "c:/shcloj-code/code" (without the "examples") to your classpath
and then (use 'examples.introduction) as mentioned by Sean above.

I haven't tried this myself or looked at the code, but I suspect
that's what the problem is.

  


gee, i thought that would work, but nope


If that doesn't work, check the value of swank-clojure-classpath after the
REPL starts: C-h v swank-clojure-classpath. Make sure it looks good and then
fire up SysInternals ProcMon and filter on *.clj and see where the system is
really looking.
  

Shawn,

The above didn't work, but apparently it doesn't pick up my .emacs file,
which has

(setq swank-clojure-classpath
   (list "c:/shcloj-code/code/examples"))

itknows where home is, when it boots up, the default is my home dir, where
.emacs is


when i do C-h v

swank-clojure-classpath is a variable defined in `swank-clojure.el'.
Its value is
("c:/Program Files/Clojure Box/emacs/site-lisp/../../swank-clojure/src"
"c:/shcloj-code/code/examples" "c:/Program Files/Clojure
Box/lib/clojure-contrib.jar" "c:/Program Files/Clojure Box/lib/clojure.jar"
"c:/Program Files/Clojure Box/lib/commons-codec-1.3.jar" "c:/Program
Files/Clojure Box/lib/commons-fileupload-1.2.1.jar" "c:/Program
Files/Clojure Box/lib/commons-io-1.4.jar" "c:/Program Files/Clojure
Box/lib/compojure.jar" "c:/Program Files/Clojure
Box/lib/grizzly-http-servlet-1.9.10.jar" "c:/Program Files/Clojure
Box/lib/grizzly-http-webserver-1.9.10.jar" "c:/Program Files/Clojure
Box/lib/jetty-6.1.15.jar" "c:/Program Files/Clojure
Box/lib/jetty-util-6.1.15.jar" "c:/Program Files/Clojure
Box/lib/servlet-api-2.5-20081211.jar")



  


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: classpath in clojure box

[brian]

Michael Wood wrote:

2010/1/14 brian :
  

Shawn Hoover wrote:


On Wed, Jan 13, 2010 at 8:38 PM, brian mailto:brw...@gmail.com>> wrote:


   I was wondering if this is the right statement for setting the
   external  classpath?

   setq swank-clojure-classpath


Another way is to use M-x swank-clojure-project. You enter a directory and
it automatically adds that directory/src to the classpath. You could drop
all of the shcloj-code\code into, say,
  

this didn't work



c:\projects\shcloj\src. M-x swank-clojure-project and at the prompt
c:\projects\shcloj. Then try (require 'examples.introduction).
  


Just a thought:

Add "c:/shcloj-code/code" (without the "examples") to your classpath
and then (use 'examples.introduction) as mentioned by Sean above.

I haven't tried this myself or looked at the code, but I suspect
that's what the problem is.

  

I take that back,it did! hurray!

here I called (take 10 fibs) and worked , instead of  "(take 10 
introduction.fibs)" which is what the book says



If that doesn't work, check the value of swank-clojure-classpath after the
REPL starts: C-h v swank-clojure-classpath. Make sure it looks good and then
fire up SysInternals ProcMon and filter on *.clj and see where the system is
really looking.
  

Shawn,

The above didn't work, but apparently it doesn't pick up my .emacs file,
which has

(setq swank-clojure-classpath
   (list "c:/shcloj-code/code/examples"))

itknows where home is, when it boots up, the default is my home dir, where
.emacs is


when i do C-h v

swank-clojure-classpath is a variable defined in `swank-clojure.el'.
Its value is
("c:/Program Files/Clojure Box/emacs/site-lisp/../../swank-clojure/src"
"c:/shcloj-code/code/examples" "c:/Program Files/Clojure
Box/lib/clojure-contrib.jar" "c:/Program Files/Clojure Box/lib/clojure.jar"
"c:/Program Files/Clojure Box/lib/commons-codec-1.3.jar" "c:/Program
Files/Clojure Box/lib/commons-fileupload-1.2.1.jar" "c:/Program
Files/Clojure Box/lib/commons-io-1.4.jar" "c:/Program Files/Clojure
Box/lib/compojure.jar" "c:/Program Files/Clojure
Box/lib/grizzly-http-servlet-1.9.10.jar" "c:/Program Files/Clojure
Box/lib/grizzly-http-webserver-1.9.10.jar" "c:/Program Files/Clojure
Box/lib/jetty-6.1.15.jar" "c:/Program Files/Clojure
Box/lib/jetty-util-6.1.15.jar" "c:/Program Files/Clojure
Box/lib/servlet-api-2.5-20081211.jar")



  


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: [ANN] units.clj - unit conversion functions without repeating yourself

[Brian]

This is more java related but there is a java library and application (
http://futureboy.us/frinkdocs/) that provides a set of very powerful unit
conversion tools.


On Thu, Jan 14, 2010 at 10:49 AM, Scott Jaderholm wrote:

> Hi,
>
> I created a library that provides unit conversion functions[1] for
> several common units and allows you to define new units conversions
> with a single equation.
>
> The library does a few interesting things automatically:
>
> First, if you define inches-to-feet, it will create feet-to-inches for you.
> Second, if you define inches-to-feet and feet-to-meters, it will
> create inches-to-meters and meters-to-inches for you.
> Third, it will create the equivalent square and cubic functions as well.
>
> So for inches, feet, yards, meters, centimeters, millimeters, miles,
> and kilometers, if you specify 7 equations this library will define
> 168 conversion functions for you (56 each for length, area, and
> volume).
>
> The code is at http://gist.github.com/276662#file_units.clj
>
> I'd love to receive feedback.
>
> Thanks,
> Scott
>
> Notes
> [1] For a reliable alternative, see JScience.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: classpath in clojure box

[brian]

ok i see it

Brian

Shawn Hoover wrote:


On Thu, Jan 14, 2010 at 3:17 AM, brian <mailto:brw...@gmail.com>> wrote:


The above didn't work, but apparently it doesn't pick up my .emacs
file, which has

(setq swank-clojure-classpath
   (list "c:/shcloj-code/code/examples"))

itknows where home is, when it boots up, the default is my home
dir, where .emacs is


when i do C-h v

swank-clojure-classpath is a variable defined in `swank-clojure.el'.
Its value is
("c:/Program Files/Clojure
Box/emacs/site-lisp/../../swank-clojure/src"
"c:/shcloj-code/code/examples" "c:/Program Files/Clojure
Box/lib/clojure-contrib.jar" "c:/Program Files/Clojure
Box/lib/clojure.jar" "c:/Program Files/Clojure
Box/lib/commons-codec-1.3.jar" "c:/Program Files/Clojure
Box/lib/commons-fileupload-1.2.1.jar" "c:/Program Files/Clojure
Box/lib/commons-io-1.4.jar" "c:/Program Files/Clojure
Box/lib/compojure.jar" "c:/Program Files/Clojure
Box/lib/grizzly-http-servlet-1.9.10.jar" "c:/Program Files/Clojure
Box/lib/grizzly-http-webserver-1.9.10.jar" "c:/Program
Files/Clojure Box/lib/jetty-6.1.15.jar" "c:/Program Files/Clojure
Box/lib/jetty-util-6.1.15.jar" "c:/Program Files/Clojure
Box/lib/servlet-api-2.5-20081211.jar")


Actually it's there, buried between the swank-clojure/src and 
clojure-contrib.jar. It's Clojure Box's fault on the ordering, but it 
would only matter if you were trying to put your own version of 
swank-clojure on the classpath.


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Anyone on Google+ yet?

[Brian]

https://plus.google.com/100763770877733610336/posts


On Thu, Jul 14, 2011 at 3:15 PM, Jonathan Cardoso
wrote:

> https://plus.google.com/115729535012467854741
>
>  --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: ClojureScript Presentation - video

[Brian]

Here ya go:
http://blip.tv/clojure/rich-hickey-unveils-clojurescript-5399498

Brian


On Jul 21, 1:09 am, Harrison Maseko  wrote:
> Hi all,
> I did not watch the live cast of Rich's presentation on ClojureScript.
> Could someone please point me to a video or audio recording of the
> presentation if there's one somewhere on the Web?
> Thanks,
> -h.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure Conj extracurricular activities spreadsheet

[Brian]

Please sign me up for:

* ClojureScript
* core.logic / minikanren
* Go
* Jamming with Overtone
* The web and Clojure

On Oct 30, 6:10 am, Krukow  wrote:
> On Oct 25, 5:11 pm, Fogus  wrote:
>
> > This should be epic.
>
> Indeed. Please sign me up for:
>
> * ClojureScript
> * Pallet

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Meaningless Poll Time: Which JVM Language Is On Top?

[Brian]

http://java.dzone.com/polls/which-jvm-language-top-0

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Can we please deprecate the :use directive ?

[Brian Marick]

I find that having both `:use` and `:require` in my `ns` declarations can, with 
a little judicious formatting, communicate something about which packages are 
core and which are peripheral. Having to obfuscate that distinction with a 
`:refer :all` tacked - in a visually obscure way - onto `:require` forms would 
make me sad. Not wildly sad, but wistfully sad that people would prefer a 
breaking change to simply avoiding something they don't want to use.

Or:

On Aug 5, 2013, at 11:48 AM, Lee Spector  wrote:

> My point is just that the needs and priorities are diverse, and I wish people 
> wouldn't be so enthusiastic about deprecating features that are valued by 
> others in the community.


Latest book: /Functional Programming for the Object-Oriented Programmer/
https://leanpub.com/fp-oo

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

hashing binary data

[Brian Craft]

I'm loading data files of about 1-2G, which are composed of a bunch of 
numeric data blocks. I need to store the data blocks w/o storing 
duplicates. They arrive as vectors of floats, and are stored as primitive 
byte arrays.

I first tried memoizing the function that saves a block (returning an id), 
with the core memoize function. This failed because every block became a 
different key in the memoization, regardless of the content. It looks like 
clojure treats variables referencing primitive arrays as equal only if they 
refer to the same array. Note:

cavm.core=> ({[1 2 3] "foo"} [1 2 3])
"foo"
cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
nil
cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
"foo"



I next tried memoizing over the vector of floats, however performance 
became pathologically slow, and the process threw an OOM. I'm guessing this 
is due to the memory requirements of a clojure vector of floats vs. a 
primitive array of bytes holding the same data. Is there an easy way to 
compare the storage requirements?

Any suggestions on how better to handle this?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: hashing binary data

[Brian Craft]

On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote:
>
> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote:
>
>> I'm loading data files of about 1-2G, which are composed of a bunch of 
>> numeric data blocks. I need to store the data blocks w/o storing 
>> duplicates. They arrive as vectors of floats, and are stored as primitive 
>> byte arrays.
>>
>> I first tried memoizing the function that saves a block (returning an 
>> id), with the core memoize function. This failed because every block became 
>> a different key in the memoization, regardless of the content. It looks 
>> like clojure treats variables referencing primitive arrays as equal only if 
>> they refer to the same array. Note:
>>
>> cavm.core=> ({[1 2 3] "foo"} [1 2 3])
>> "foo"
>> cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
>> nil
>> cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
>> "foo"
>>
>>
>>
>> I next tried memoizing over the vector of floats, however performance 
>> became pathologically slow, and the process threw an OOM. I'm guessing this 
>> is due to the memory requirements of a clojure vector of floats vs. a 
>> primitive array of bytes holding the same data. Is there an easy way to 
>> compare the storage requirements?
>>
>> Any suggestions on how better to handle this?
>>
>
> You may want to use the :ndarray-float array implementation in the latest 
> version of core.matrix.
>
> This is effectively a wrapper over a raw Java float array: so your storage 
> requirement should be close to the size of the raw byte data (assuming the 
> data blocks are large enough that the size of the wrapper is negligible)
>

Ah, interesting.

> *matrix-implementation*
:vectorz
> ({(matrix [1 2 3 4]) "foo"} (matrix [1 2 3 4]))
"foo"

I don't otherwise need core.matrix at this point in the loader, but this is 
convenient. Why does that work?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: hashing binary data

[Brian Craft]

On Tuesday, September 3, 2013 10:03:00 PM UTC-7, Mikera wrote:
>
>
>
> On Wednesday, 4 September 2013 12:37:33 UTC+8, Brian Craft wrote:
>>
>>
>>
>> On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote:
>>>
>>> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote:
>>>
>>>> I'm loading data files of about 1-2G, which are composed of a bunch of 
>>>> numeric data blocks. I need to store the data blocks w/o storing 
>>>> duplicates. They arrive as vectors of floats, and are stored as primitive 
>>>> byte arrays.
>>>>
>>>> I first tried memoizing the function that saves a block (returning an 
>>>> id), with the core memoize function. This failed because every block 
>>>> became 
>>>> a different key in the memoization, regardless of the content. It looks 
>>>> like clojure treats variables referencing primitive arrays as equal only 
>>>> if 
>>>> they refer to the same array. Note:
>>>>
>>>> cavm.core=> ({[1 2 3] "foo"} [1 2 3])
>>>> "foo"
>>>> cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
>>>> nil
>>>> cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
>>>> "foo"
>>>>
>>>>
>>>>
>>>> I next tried memoizing over the vector of floats, however performance 
>>>> became pathologically slow, and the process threw an OOM. I'm guessing 
>>>> this 
>>>> is due to the memory requirements of a clojure vector of floats vs. a 
>>>> primitive array of bytes holding the same data. Is there an easy way to 
>>>> compare the storage requirements?
>>>>
>>>> Any suggestions on how better to handle this?
>>>>
>>>
>>> You may want to use the :ndarray-float array implementation in the 
>>> latest version of core.matrix.
>>>
>>> This is effectively a wrapper over a raw Java float array: so your 
>>> storage requirement should be close to the size of the raw byte data 
>>> (assuming the data blocks are large enough that the size of the wrapper is 
>>> negligible)
>>>
>>
>> Ah, interesting.
>>
>> > *matrix-implementation*
>> :vectorz
>> > ({(matrix [1 2 3 4]) "foo"} (matrix [1 2 3 4]))
>> "foo"
>>
>> I don't otherwise need core.matrix at this point in the loader, but this 
>> is convenient. Why does that work?
>>
>
> That works because Vectorz (the underlying Java lib) has a sane 
> implementation of .equals and .hashCode. It's pretty fast as well, though 
> it is still O(n) since it doesn't do hashcode caching.
>
> Note that the :vectorz implementation uses 8-byte doubles rather than 
> 4-byte floats though - so if you really need single precision to keep the 
> overall memory usage down then it might not be the best choice. I 
> personally never use 4-byte floats because the numerical errors soon become 
> problematic, but YMMV.
>
>
It looks like I can get this working for byte arrays as follows.

(deftype BAHashable [ba] 
  Object
  (equals [f g] (java.util.Arrays/equals ba (.ba g)))
  (hashCode [f] (java.util.Arrays/hashCode ba)))

({(BAHashable. (byte-array (map byte [1 2 3]))) "foo"} (BAHashable. 
(byte-array (map byte [1 2 3]
"foo"


I'm less certain of whether this is a good idea.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: hashing binary data

[Brian Craft]

On Wednesday, September 4, 2013 2:13:12 AM UTC-7, Brian Craft wrote:
>
>
>
> On Tuesday, September 3, 2013 10:03:00 PM UTC-7, Mikera wrote:
>>
>>
>>
>> On Wednesday, 4 September 2013 12:37:33 UTC+8, Brian Craft wrote:
>>>
>>>
>>>
>>> On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote:
>>>>
>>>> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote:
>>>>
>>>>> I'm loading data files of about 1-2G, which are composed of a bunch of 
>>>>> numeric data blocks. I need to store the data blocks w/o storing 
>>>>> duplicates. They arrive as vectors of floats, and are stored as primitive 
>>>>> byte arrays.
>>>>>
>>>>> I first tried memoizing the function that saves a block (returning an 
>>>>> id), with the core memoize function. This failed because every block 
>>>>> became 
>>>>> a different key in the memoization, regardless of the content. It looks 
>>>>> like clojure treats variables referencing primitive arrays as equal only 
>>>>> if 
>>>>> they refer to the same array. Note:
>>>>>
>>>>> cavm.core=> ({[1 2 3] "foo"} [1 2 3])
>>>>> "foo"
>>>>> cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
>>>>> nil
>>>>> cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
>>>>> "foo"
>>>>>
>>>>>
>>>>>
>>>>> I next tried memoizing over the vector of floats, however performance 
>>>>> became pathologically slow, and the process threw an OOM. I'm guessing 
>>>>> this 
>>>>> is due to the memory requirements of a clojure vector of floats vs. a 
>>>>> primitive array of bytes holding the same data. Is there an easy way to 
>>>>> compare the storage requirements?
>>>>>
>>>>> Any suggestions on how better to handle this?
>>>>>
>>>>
>>>> You may want to use the :ndarray-float array implementation in the 
>>>> latest version of core.matrix.
>>>>
>>>> This is effectively a wrapper over a raw Java float array: so your 
>>>> storage requirement should be close to the size of the raw byte data 
>>>> (assuming the data blocks are large enough that the size of the wrapper is 
>>>> negligible)
>>>>
>>>
>>> Ah, interesting.
>>>
>>> > *matrix-implementation*
>>> :vectorz
>>> > ({(matrix [1 2 3 4]) "foo"} (matrix [1 2 3 4]))
>>> "foo"
>>>
>>> I don't otherwise need core.matrix at this point in the loader, but this 
>>> is convenient. Why does that work?
>>>
>>
>> That works because Vectorz (the underlying Java lib) has a sane 
>> implementation of .equals and .hashCode. It's pretty fast as well, though 
>> it is still O(n) since it doesn't do hashcode caching.
>>
>> Note that the :vectorz implementation uses 8-byte doubles rather than 
>> 4-byte floats though - so if you really need single precision to keep the 
>> overall memory usage down then it might not be the best choice. I 
>> personally never use 4-byte floats because the numerical errors soon become 
>> problematic, but YMMV.
>>
>>
> It looks like I can get this working for byte arrays as follows.
>
> (deftype BAHashable [ba] 
>   Object
>   (equals [f g] (java.util.Arrays/equals ba (.ba g)))
>   (hashCode [f] (java.util.Arrays/hashCode ba)))
>
> ({(BAHashable. (byte-array (map byte [1 2 3]))) "foo"} (BAHashable. 
> (byte-array (map byte [1 2 3]
> "foo"
>
>
> I'm less certain of whether this is a good idea.
>


This gives me a number of reflection warnings, on field ba, on equals, and 
on hashCode. I can eliminate the one on hashCode by type hinting the ba 
parameter of BAHashable.

The one on (.ba g) and equals remains. Is there any way to hint these?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: hashing binary data

[Brian Craft]

On Wednesday, September 4, 2013 1:53:17 PM UTC-7, Karsten Schmidt wrote:
>
> On 4 September 2013 17:52, Brian Craft > 
> wrote: 
>
> > This gives me a number of reflection warnings, on field ba, on equals, 
> and 
> > on hashCode. I can eliminate the one on hashCode by type hinting the ba 
> > parameter of BAHashable. 
> > 
> > The one on (.ba g) and equals remains. Is there any way to hint these? 
>
> Brian, this seems to work (+ a bit more safe): 
>
> (set! *warn-on-reflection* true) 
>
> (deftype BAHashable [ba] 
>   Object 
>   (equals [this other] 
> (if (instance? BAHashable other) 
>   (java.util.Arrays/equals ^bytes ba ^bytes (.ba ^BAHashable other)) 
>   false)) 
>   (hashCode [this] 
> (java.util.Arrays/hashCode ^bytes ba))) 
> #_=>user.BAHashable 
>
> no more reflcopters! :) 
>


Thanks!

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

memory profile dominated by lazy seqs from dorun

[Brian Craft]

I'm using visualvm to generate a memory profile snapshot with tracebacks. 
The profile is dominated by clojure.lang.LazySeq. I'm confused that the 
allocation traces back to clojure.core$dorun.invoke.

Attaching a screen shot. The write_data() function is being called 
repeatedly, and does a dorun map over values it is passed.

Can someone explain how to interpret this? It looks like the dorun is 
eventually allocating a seq, and it's hanging around, which makes little 
sense to me.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
<>

finding retained head

[Brian Craft]

>From jmap output it's pretty clear I'm retaining the head of a seq somehow, 
but I have no idea how, or where. Is there any way to find the reference, 
besides staring at code waiting for enlightenment?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

Trying jhat now, reference chain here:

--> cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:)
--> clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:)
--> clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:)
--> clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:)
--> clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:)

This looks to me like the head is being retained as a parameter to a 
function. load_exp signature is like so:

defn load-exp [file timestamp filehash matrix]

But surely passing a seq to a function doesn't retain the head?

Here's the body of the function:

(defn load-exp [file timestamp filehash matrix]
  (kdb/transaction
(let [cid (merge-cohort file)
  exp (merge-exp file (format-timestamp timestamp) filehash cid)
  sids (load-samples cid (:samples (meta matrix)))]
  (load-exp-samples exp sids)
  (load-exp-matrix exp matrix


load-exp-matrix calls (dorun (map)) on matrix. There's one other reference 
here, where we pull the metadata. Does any of this retain the head?

On Tuesday, September 10, 2013 2:36:29 PM UTC-7, Brian Craft wrote:
>
> From jmap output it's pretty clear I'm retaining the head of a seq 
> somehow, but I have no idea how, or where. Is there any way to find the 
> reference, besides staring at code waiting for enlightenment?
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

It seems to be something about the jdbc/transaction macro. If I rewrite 
load_exp so it does nothing but walk the seq, it still consumes the heap:

(defn load-exp [file timestamp filehash matrix]
  (jdbc/transaction
(dorun (map identity matrix


If I remove the jdbc/transaction call, it's fine.

(defn load-exp [file timestamp filehash matrix]
  (dorun (map identity matrix)))



On Tuesday, September 10, 2013 3:24:29 PM UTC-7, Brian Craft wrote:
>
> Trying jhat now, reference chain here:
>
> --> cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:)
> --> clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:)
> --> clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:)
> --> clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:)
> --> clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:)
>
> This looks to me like the head is being retained as a parameter to a 
> function. load_exp signature is like so:
>
> defn load-exp [file timestamp filehash matrix]
>
> But surely passing a seq to a function doesn't retain the head?
>
> Here's the body of the function:
>
> (defn load-exp [file timestamp filehash matrix]
>   (kdb/transaction
> (let [cid (merge-cohort file)
>   exp (merge-exp file (format-timestamp timestamp) filehash cid)
>   sids (load-samples cid (:samples (meta matrix)))]
>   (load-exp-samples exp sids)
>   (load-exp-matrix exp matrix
>
>
> load-exp-matrix calls (dorun (map)) on matrix. There's one other reference 
> here, where we pull the metadata. Does any of this retain the head?
>
> On Tuesday, September 10, 2013 2:36:29 PM UTC-7, Brian Craft wrote:
>>
>> From jmap output it's pretty clear I'm retaining the head of a seq 
>> somehow, but I have no idea how, or where. Is there any way to find the 
>> reference, besides staring at code waiting for enlightenment?
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

Copying the pattern of jdbc/transaction*, I tried this:

(defn blah [func] (func))
(defn load-exp [file timestamp filehash matrix]
  (blah
(fn [] (dorun (map identity matrix)


which also consumes the heap.

Can anyone explain to me what's happening here? Something about creating a 
anonymous function?

On Tuesday, September 10, 2013 5:24:44 PM UTC-7, Brian Craft wrote:
>
> It seems to be something about the jdbc/transaction macro. If I rewrite 
> load_exp so it does nothing but walk the seq, it still consumes the heap:
>
> (defn load-exp [file timestamp filehash matrix]
>   (jdbc/transaction
> (dorun (map identity matrix
>
>
> If I remove the jdbc/transaction call, it's fine.
>
> (defn load-exp [file timestamp filehash matrix]
>   (dorun (map identity matrix)))
>
>
>
> On Tuesday, September 10, 2013 3:24:29 PM UTC-7, Brian Craft wrote:
>>
>> Trying jhat now, reference chain here:
>>
>> --> cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:)
>> --> clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:)
>> --> clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:)
>> --> clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:)
>> --> clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:)
>>
>> This looks to me like the head is being retained as a parameter to a 
>> function. load_exp signature is like so:
>>
>> defn load-exp [file timestamp filehash matrix]
>>
>> But surely passing a seq to a function doesn't retain the head?
>>
>> Here's the body of the function:
>>
>> (defn load-exp [file timestamp filehash matrix]
>>   (kdb/transaction
>> (let [cid (merge-cohort file)
>>   exp (merge-exp file (format-timestamp timestamp) filehash cid)
>>   sids (load-samples cid (:samples (meta matrix)))]
>>   (load-exp-samples exp sids)
>>   (load-exp-matrix exp matrix
>>
>>
>> load-exp-matrix calls (dorun (map)) on matrix. There's one other 
>> reference here, where we pull the metadata. Does any of this retain the 
>> head?
>>
>> On Tuesday, September 10, 2013 2:36:29 PM UTC-7, Brian Craft wrote:
>>>
>>> From jmap output it's pretty clear I'm retaining the head of a seq 
>>> somehow, but I have no idea how, or where. Is there any way to find the 
>>> reference, besides staring at code waiting for enlightenment?
>>>
>>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

Ah. So the root of my problem is that the jdbc/transaction macro creates a 
closure in my function..
(defmacro transaction


On Tuesday, September 10, 2013 7:34:32 PM UTC-7, Armando Blancas wrote:
>
> Can anyone explain to me what's happening here? Something about creating a 
>> anonymous function?
>>
>
> The problem is not creating the anonymous function but that it closes over 
> the matrix argument. The closure passed on to blah will keep a reference to 
> matrix until blah returns. This won't happen if the anonymous function 
> takes the matrix as a parameter:
>
> (defn blah [func v] (func v))
> (defn load-exp [matrix]
>   (blah
> (fn [m] (dorun (map identity m))) matrix))
>  
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

(defmacro transaction
  [& body]
  `(transaction* (fn [] ~@body)))

I'm not sure how to avoid that. The anonymous function created here doesn't 
take parameters.

On Tuesday, September 10, 2013 8:55:42 PM UTC-7, Brian Craft wrote:
>
> Ah. So the root of my problem is that the jdbc/transaction macro creates a 
> closure in my function..
> (defmacro transaction
>
>
> On Tuesday, September 10, 2013 7:34:32 PM UTC-7, Armando Blancas wrote:
>>
>> Can anyone explain to me what's happening here? Something about creating 
>>> a anonymous function?
>>>
>>
>> The problem is not creating the anonymous function but that it closes 
>> over the matrix argument. The closure passed on to blah will keep a 
>> reference to matrix until blah returns. This won't happen if the anonymous 
>> function takes the matrix as a parameter:
>>
>> (defn blah [func v] (func v))
>> (defn load-exp [matrix]
>>   (blah
>> (fn [m] (dorun (map identity m))) matrix))
>>  
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

ugh. Can't find documentation for this. What does it do?

On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote:
>
>
> On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft 
> > wrote:
>
>> (defmacro transaction
>>   [& body]
>>   `(transaction* (fn [] ~@body)))
>>
>> I'm not sure how to avoid that. The anonymous function created here 
>> doesn't take parameters.
>>
>
> The fix for this is:
> (defmacro transaction
>   [& body]
>   `(transaction* (^:once fn* [] ~@body)))
>
> It should be the default for all one-shot fns.
>
> Christophe
>
> -- 
> On Clojure http://clj-me.cgrand.net/
> Clojure Programming http://clojurebook.com
> Training, Consulting & Contracting http://lambdanext.eu/ 
>  

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

(started lein with LEIN_JVM_OPTS=-Xmx10M lein repl, to make it easy to see 
a full heap)

On Wednesday, September 11, 2013 8:59:50 AM UTC-7, Brian Craft wrote:
>
> This appears to have no effect on the problem. I tested with 
> jdbc/transaction, and with Sean's simple example:
>
> user=> (defn f [g] (g))
> #'user/f
> user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)
> #'user/t1
> user=> (t1 0 (range 100))
> java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)
> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
> #'user/t2
> user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)
> #'user/t1
> user=> (t1 0 (range 100))
> java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)
>
>
>
> On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote:
>>
>> ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs 
>> the commielr to clear closed-overs ASAP. It follows that you can't call 
>> such a function twice because it forgets its closed-overs.
>>
>>
>> On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft  wrote:
>>
>>> ugh. Can't find documentation for this. What does it do?
>>>
>>> On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand 
>>> wrote:
>>>
>>>>
>>>> On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft wrote:
>>>>
>>>>> (defmacro transaction
>>>>>   [& body]
>>>>>   `(transaction* (fn [] ~@body)))
>>>>>
>>>>> I'm not sure how to avoid that. The anonymous function created here 
>>>>> doesn't take parameters.
>>>>>
>>>>
>>>> The fix for this is:
>>>> (defmacro transaction
>>>>   [& body]
>>>>   `(transaction* (^:once fn* [] ~@body)))
>>>>
>>>> It should be the default for all one-shot fns.
>>>>
>>>> Christophe
>>>>
>>>> -- 
>>>> On Clojure http://clj-me.cgrand.net/
>>>> Clojure Programming http://clojurebook.com
>>>> Training, Consulting & Contracting http://lambdanext.eu/ 
>>>>  
>>>
>>
>>
>> -- 
>> On Clojure http://clj-me.cgrand.net/
>> Clojure Programming http://clojurebook.com
>> Training, Consulting & Contracting http://lambdanext.eu/ 
>>  
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

This appears to have no effect on the problem. I tested with 
jdbc/transaction, and with Sean's simple example:

user=> (defn f [g] (g))
#'user/f
user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)
#'user/t1
user=> (t1 0 (range 100))
java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)
user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
#'user/t2
user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)
#'user/t1
user=> (t1 0 (range 100))
java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)



On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote:
>
> ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs 
> the commielr to clear closed-overs ASAP. It follows that you can't call 
> such a function twice because it forgets its closed-overs.
>
>
> On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft 
> > wrote:
>
>> ugh. Can't find documentation for this. What does it do?
>>
>> On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote:
>>
>>>
>>> On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft  wrote:
>>>
>>>> (defmacro transaction
>>>>   [& body]
>>>>   `(transaction* (fn [] ~@body)))
>>>>
>>>> I'm not sure how to avoid that. The anonymous function created here 
>>>> doesn't take parameters.
>>>>
>>>
>>> The fix for this is:
>>> (defmacro transaction
>>>   [& body]
>>>   `(transaction* (^:once fn* [] ~@body)))
>>>
>>> It should be the default for all one-shot fns.
>>>
>>> Christophe
>>>
>>> -- 
>>> On Clojure http://clj-me.cgrand.net/
>>> Clojure Programming http://clojurebook.com
>>> Training, Consulting & Contracting http://lambdanext.eu/ 
>>>  
>>
>
>
> -- 
> On Clojure http://clj-me.cgrand.net/
> Clojure Programming http://clojurebook.com
> Training, Consulting & Contracting http://lambdanext.eu/ 
>  

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

I did start with visualvm. I posted a screenshot in an earlier thread. 
However I'm unable to make sense of its output. jhat pointed straight to 
the closure with the reference. visualvm gave me a thousand cascading 
widgets to expand with names that were meaningless to me, none of which 
pointed back to the closure. I'm looking at it again now, trying to find 
another leak. The references for a char[] start at "String" and go back to 
"cache (Java frame)", with no symbol anywhere that I can relate to the 
program being run. I don't know what to do with this. The Strings are from 
another seq: lines from a file via line-seq.

Unfortunately, jhat just hangs on this object, so I have no working 
instrumentation.

On Wednesday, September 11, 2013 1:08:33 AM UTC-7, David Powell wrote:
>
> jvisualvm has an innocuous button called "Dump Memory" or something.
> You'd expect it to write out a core dump or something, but actually it 
> opens up a GUI which lets you interactively explore all of the objects on 
> the heap.  It is pretty amazing.  Much better than jhat, which I've found 
> to be really flakey.
>
> Good for finding Classloader leaks too, or just generally finding where 
> all your memory has gone via the Compute Retained Sizes option.
>  

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

Correct, I forgot to paste that part. ;)

On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote:
>
> Just to confirm, (t2 0 (range 100)) -- using doseq instead of 
> dorun -- does NOT run out of memory, correct? 
>
> On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft 
> > 
> wrote: 
> > This appears to have no effect on the problem. I tested with 
> > jdbc/transaction, and with Sean's simple example: 
> > 
> > user=> (defn f [g] (g)) 
> > #'user/f 
> > user=> (defn t1 [n c] (f (fn [] (dorun (map identity c) 
> > #'user/t1 
> > user=> (t1 0 (range 100)) 
> > java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) 
> > user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) 
> > #'user/t2 
> > user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) 
> > #'user/t1 
> > user=> (t1 0 (range 100)) 
> > java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) 
> > 
> > 
> > 
> > On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand 
> wrote: 
> >> 
> >> ^:once on fn* (not fn, the fn macro doesn't propagate metadata) 
> instructs 
> >> the commielr to clear closed-overs ASAP. It follows that you can't call 
> such 
> >> a function twice because it forgets its closed-overs. 
> >> 
> >> 
> >> On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft  
> wrote: 
> >>> 
> >>> ugh. Can't find documentation for this. What does it do? 
> >>> 
> >>> On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand 
> >>> wrote: 
> >>>> 
> >>>> 
> >>>> On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft  
> wrote: 
> >>>>> 
> >>>>> (defmacro transaction 
> >>>>>   [& body] 
> >>>>>   `(transaction* (fn [] ~@body))) 
> >>>>> 
> >>>>> I'm not sure how to avoid that. The anonymous function created here 
> >>>>> doesn't take parameters. 
> >>>> 
> >>>> 
> >>>> The fix for this is: 
> >>>> (defmacro transaction 
> >>>>   [& body] 
> >>>>   `(transaction* (^:once fn* [] ~@body))) 
> >>>> 
> >>>> It should be the default for all one-shot fns. 
> >>>> 
> >>>> Christophe 
> >>>> 
> >>>> -- 
> >>>> On Clojure http://clj-me.cgrand.net/ 
> >>>> Clojure Programming http://clojurebook.com 
> >>>> Training, Consulting & Contracting http://lambdanext.eu/ 
> >> 
> >> 
> >> 
> >> 
> >> -- 
> >> On Clojure http://clj-me.cgrand.net/ 
> >> Clojure Programming http://clojurebook.com 
> >> Training, Consulting & Contracting http://lambdanext.eu/ 
> > 
> > -- 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "Clojure" group. 
> > To post to this group, send email to clo...@googlegroups.com 
> > Note that posts from new members are moderated - please be patient with 
> your 
> > first post. 
> > To unsubscribe from this group, send email to 
> > clojure+u...@googlegroups.com  
> > For more options, visit this group at 
> > http://groups.google.com/group/clojure?hl=en 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Clojure" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to clojure+u...@googlegroups.com . 
> > For more options, visit https://groups.google.com/groups/opt_out. 
>
>
>
> -- 
> Sean A Corfield -- (904) 302-SEAN 
> An Architect's View -- http://corfield.org/ 
> World Singles, LLC. -- http://worldsingles.com/ 
>
> "Perfection is the enemy of the good." 
> -- Gustave Flaubert, French realist novelist (1821-1880) 
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

Attaching a screenshot. Is this reference chain useful somehow? I can 
recognize the contents of the string from the input file. The input file is 
4G, and apparently lines from line-seq are lingering, so I'm still blowing 
the heap.

On Wednesday, September 11, 2013 11:20:18 AM UTC-7, Brian Craft wrote:
>
> I did start with visualvm. I posted a screenshot in an earlier thread. 
> However I'm unable to make sense of its output. jhat pointed straight to 
> the closure with the reference. visualvm gave me a thousand cascading 
> widgets to expand with names that were meaningless to me, none of which 
> pointed back to the closure. I'm looking at it again now, trying to find 
> another leak. The references for a char[] start at "String" and go back to 
> "cache (Java frame)", with no symbol anywhere that I can relate to the 
> program being run. I don't know what to do with this. The Strings are from 
> another seq: lines from a file via line-seq.
>
> Unfortunately, jhat just hangs on this object, so I have no working 
> instrumentation.
>
> On Wednesday, September 11, 2013 1:08:33 AM UTC-7, David Powell wrote:
>>
>> jvisualvm has an innocuous button called "Dump Memory" or something.
>> You'd expect it to write out a core dump or something, but actually it 
>> opens up a GUI which lets you interactively explore all of the objects on 
>> the heap.  It is pretty amazing.  Much better than jhat, which I've found 
>> to be really flakey.
>>
>> Good for finding Classloader leaks too, or just generally finding where 
>> all your memory has gone via the Compute Retained Sizes option.
>>  
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
<>

another game of exploding heap, via clojure.string/split

[Brian Craft]

After working around the seq + closure = death problem, I still had a 
severe memory leak in my code, which took many hours to find.

Holding a reference to a string returned by clojure.string/split is somehow 
retaining a reference to the original string. In my case I needed to hold 
the first column of each row in a tsv file that was 4G in size. This 
resulted in holding the entire 4G in memory.

Here's a demo. Function "data" returns a seq of lines that are about 1000 
bytes. The first column, however, is just a few bytes, and 10k of them 
should easily fit in 10M of heap space. But, no:

$ LEIN_JVM_OPTS=-Xmx10M lein repl
REPL started; server listening on localhost port 34955
user=> (defn data [] (for [i (range)] (str "row " i "\t" 
(clojure.string/join "" (repeat 1000 "x")
#'user/data
user=> (def x (vec (take 1 (map #(first (clojure.string/split % #"\t")) 
(data)
java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:4)
user=> 

If I copy the returned string with the String constructor, it's fine:

$ LEIN_JVM_OPTS=-Xmx10M lein repl
REPL started; server listening on localhost port 20587
user=> (defn data [] (for [i (range)] (str "row " i "\t" 
(clojure.string/join "" (repeat 1000 "x")
#'user/data
user=> (def x (vec (take 1 (map #(String. (first (clojure.string/split 
% #"\t"))) (data)
#'user/x
user=> (x 10)
"row 10"
user=> 

Two observations about this.

First, this behavior is very unexpected to me. I don't understand if it is 
a property of strings, collections, or string/split specifically that is 
causing it. Is there something in the docs that I overlooked, that would 
have warned of this?

Second, for tracking down problems like this, the available tooling is 
pathetic, to put it as politely as possible. jhat would not trace the the 
leaked strings. It consistently froze up when tracing them to GC roots. 
visualvm traced it back to CacheLRU, as in the screenshot I posted in the 
other thread, which was perfectly uninformative.

Without any usable tooling, the only workflow I found to narrow the problem 
was to iteratively stub out portions of code and re-run the program for 
several minutes to determine if the leak was active. Obviously, this is 
incredibly painful, slow, and tedious.

I'm hoping someone can tell me there's a better way.

Note that the leak did not appear in when exercising subsystems 
independently, because in that case no references were retained from one 
subsystem to the other. So, "try it in the repl" was not an effective 
strategy.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

or perhaps it's that macroexpand drops the ^:once

On Thursday, September 12, 2013 10:55:28 AM UTC-7, Brian Craft wrote:
>
> I think my monkey-patch of jdbc/transaction didn't take. Doing 
> user/sourcery on it shows your version, but doing a macroexpand-all shows 
> the original.
>
> My code calls via korma as kdb:
>
> => (clojure.walk/macroexpand-all '(kdb/transaction nil))
> (if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* 
> (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection 
> (clojure.core/deref korma.db/_default)) (fn* ([] 
> (clojure.java.jdbc/transaction* (fn* [] nil))
>
> I thought I could get it to take by doing a (require 'clojure.java.jdbc) 
> and applying the patch before doing a require on korma, but that doesn't 
> seem to help.
>
> (require 'clojure.java.jdbc)
> ; XXX monkey-patch jdbc to avoid head retention bug
> (in-ns 'clojure.java.jdbc)
> (defmacro transaction
>   [& body]
>   `(transaction* (^:once fn* [] ~@body)))
>
> (ns   (:require [korma.db as kdb]))
>
> .. and restart the repl, but macroexpand-all still gives me the result 
> above.
>
> On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote:
>>
>> I don't get the same results:
>>
>> $ LEIN_JVM_OPTS=-Xmx20M lein repl
>> nREPL server started on port 61221 on host 127.0.0.1
>> REPL-y 0.2.1
>> Clojure 1.5.1
>> Docs: (doc function-name-here)
>>   (find-doc "part-of-name-here")
>>   Source: (source function-name-here)
>>  Javadoc: (javadoc java-object-or-class-here)
>> Exit: Control+D or (exit) or (quit)
>>
>> user=> (defn f [g] (g))
>> #'user/f
>> user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)
>> #'user/t1
>> user=> (t1 0 (range 100))
>>
>> OutOfMemoryError GC overhead limit exceeded  java.lang.Long.valueOf 
>> (Long.java:577)
>> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
>> #'user/t2
>> user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)
>> #'user/t1
>> user=>  (t1 0 (range 100))
>> nil
>> user=> (t2 0 (range 100))
>>
>> OutOfMemoryError GC overhead limit exceeded  
>> clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29)
>>
>> BUT this is because the previous OOM left the JVM in a dirty state: try 
>> to reorder your expressions:
>>
>> $ LEIN_JVM_OPTS=-Xmx20M lein repl
>> nREPL server started on port 61245 on host 127.0.0.1
>> REPL-y 0.2.1
>> Clojure 1.5.1
>> Docs: (doc function-name-here)
>>   (find-doc "part-of-name-here")
>>   Source: (source function-name-here)
>>  Javadoc: (javadoc java-object-or-class-here)
>> Exit: Control+D or (exit) or (quit)
>>
>> user=> (defn f [g] (g))
>> #'user/f
>> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
>> #'user/t2
>> user=> (t2 0 (range 100))
>> nil
>>
>> ^^this last one failed in the previous run.
>>
>> I'm not quite sure about why the doseq version works -- I would have to 
>> research a bit. My gut feeling is that doseq is based on loop and loops are 
>> lifted into ^:once fn* by the compiler in some cases. 
>> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951
>>
>>
>>
>> On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft  wrote:
>>
>>> Correct, I forgot to paste that part. ;)
>>>
>>>
>>> On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote:
>>>
>>>> Just to confirm, (t2 0 (range 100)) -- using doseq instead of 
>>>> dorun -- does NOT run out of memory, correct? 
>>>>
>>>> On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft  
>>>> wrote: 
>>>> > This appears to have no effect on the problem. I tested with 
>>>> > jdbc/transaction, and with Sean's simple example: 
>>>> > 
>>>> > user=> (defn f [g] (g)) 
>>>> > #'user/f 
>>>> > user=> (defn t1 [n c] (f (fn [] (dorun (map identity c) 
>>>> > #'user/t1 
>>>> > user=> (t1 0 (range 100)) 
>>>> > java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) 
>>>> > user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) 
>>>> > #'user/t2 
>>>> > user=> (defn t1 [n c] (f (^:once fn* [] (dorun

Re: finding retained head

[Brian Craft]

After patching both transaction and with-connection (which is also used by 
korma.db/transaction, and also creates a closure), I can pass in the seq 
w/o leaking.

Thanks, Christophe.

On Thursday, September 12, 2013 11:20:11 AM UTC-7, Brian Craft wrote:
>
> or perhaps it's that macroexpand drops the ^:once
>
> On Thursday, September 12, 2013 10:55:28 AM UTC-7, Brian Craft wrote:
>>
>> I think my monkey-patch of jdbc/transaction didn't take. Doing 
>> user/sourcery on it shows your version, but doing a macroexpand-all shows 
>> the original.
>>
>> My code calls via korma as kdb:
>>
>> => (clojure.walk/macroexpand-all '(kdb/transaction nil))
>> (if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* 
>> (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection 
>> (clojure.core/deref korma.db/_default)) (fn* ([] 
>> (clojure.java.jdbc/transaction* (fn* [] nil))
>>
>> I thought I could get it to take by doing a (require 'clojure.java.jdbc) 
>> and applying the patch before doing a require on korma, but that doesn't 
>> seem to help.
>>
>> (require 'clojure.java.jdbc)
>> ; XXX monkey-patch jdbc to avoid head retention bug
>> (in-ns 'clojure.java.jdbc)
>> (defmacro transaction
>>   [& body]
>>   `(transaction* (^:once fn* [] ~@body)))
>>
>> (ns   (:require [korma.db as kdb]))
>>
>> .. and restart the repl, but macroexpand-all still gives me the result 
>> above.
>>
>> On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote:
>>>
>>> I don't get the same results:
>>>
>>> $ LEIN_JVM_OPTS=-Xmx20M lein repl
>>> nREPL server started on port 61221 on host 127.0.0.1
>>> REPL-y 0.2.1
>>> Clojure 1.5.1
>>> Docs: (doc function-name-here)
>>>   (find-doc "part-of-name-here")
>>>   Source: (source function-name-here)
>>>  Javadoc: (javadoc java-object-or-class-here)
>>> Exit: Control+D or (exit) or (quit)
>>>
>>> user=> (defn f [g] (g))
>>> #'user/f
>>> user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)
>>> #'user/t1
>>> user=> (t1 0 (range 100))
>>>
>>> OutOfMemoryError GC overhead limit exceeded  java.lang.Long.valueOf 
>>> (Long.java:577)
>>> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
>>> #'user/t2
>>> user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)
>>> #'user/t1
>>> user=>  (t1 0 (range 100))
>>> nil
>>> user=> (t2 0 (range 100))
>>>
>>> OutOfMemoryError GC overhead limit exceeded  
>>> clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29)
>>>
>>> BUT this is because the previous OOM left the JVM in a dirty state: try 
>>> to reorder your expressions:
>>>
>>> $ LEIN_JVM_OPTS=-Xmx20M lein repl
>>> nREPL server started on port 61245 on host 127.0.0.1
>>> REPL-y 0.2.1
>>> Clojure 1.5.1
>>> Docs: (doc function-name-here)
>>>   (find-doc "part-of-name-here")
>>>   Source: (source function-name-here)
>>>  Javadoc: (javadoc java-object-or-class-here)
>>> Exit: Control+D or (exit) or (quit)
>>>
>>> user=> (defn f [g] (g))
>>> #'user/f
>>> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
>>> #'user/t2
>>> user=> (t2 0 (range 100))
>>> nil
>>>
>>> ^^this last one failed in the previous run.
>>>
>>> I'm not quite sure about why the doseq version works -- I would have to 
>>> research a bit. My gut feeling is that doseq is based on loop and loops are 
>>> lifted into ^:once fn* by the compiler in some cases. 
>>> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951
>>>
>>>
>>>
>>> On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft  wrote:
>>>
>>>> Correct, I forgot to paste that part. ;)
>>>>
>>>>
>>>> On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote:
>>>>
>>>>> Just to confirm, (t2 0 (range 100)) -- using doseq instead of 
>>>>> dorun -- does NOT run out of memory, correct? 
>>>>>
>>>>> On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft  
>>>>> wrote: 
>>>>> > This ap

Re: docs for ^:once

[Brian Craft]

Hm, I think I'm wrong about that workaround.

On Thursday, September 12, 2013 12:10:44 PM UTC-7, Brian Craft wrote:
>
> Thanks.
>
> In my case, I started with code something like
>
> (defn load-exp [myseq]
>   (kdb/transaction body))
>
> where body has references to myseq. Christophe's patch of jdbc/transaction 
> works. I can also work around it like
>
> (defn load-exp [& args]
>   (apply (^:once fn [myseq] (kdb/transaction body))
>  args))
>
> at which point I start wondering if it makes sense to apply ^:once to 
> load-exp, but I don't really understand it.
>
>
> On Thursday, September 12, 2013 11:59:15 AM UTC-7, Christian Sperandio 
> wrote:
>>
>> You'll find a case of use of this term here:
>>
>> http://clj-me.cgrand.net/2013/09/11/macros-closures-and-unexpected-object-retention/
>>  Le 12 sept. 2013 20:56, "Brian Craft"  a écrit :
>>
>>> I still don't really understand ^:once. Are there docs anywhere? Is 
>>> there a way to inspect a function for this property?
>>>
>>>  -- 
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to clojure+u...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

docs for ^:once

[Brian Craft]

I still don't really understand ^:once. Are there docs anywhere? Is there a 
way to inspect a function for this property?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: finding retained head

[Brian Craft]

I think my monkey-patch of jdbc/transaction didn't take. Doing 
user/sourcery on it shows your version, but doing a macroexpand-all shows 
the original.

My code calls via korma as kdb:

=> (clojure.walk/macroexpand-all '(kdb/transaction nil))
(if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* 
(fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection 
(clojure.core/deref korma.db/_default)) (fn* ([] 
(clojure.java.jdbc/transaction* (fn* [] nil))

I thought I could get it to take by doing a (require 'clojure.java.jdbc) 
and applying the patch before doing a require on korma, but that doesn't 
seem to help.

(require 'clojure.java.jdbc)
; XXX monkey-patch jdbc to avoid head retention bug
(in-ns 'clojure.java.jdbc)
(defmacro transaction
  [& body]
  `(transaction* (^:once fn* [] ~@body)))

(ns   (:require [korma.db as kdb]))

.. and restart the repl, but macroexpand-all still gives me the result 
above.

On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote:
>
> I don't get the same results:
>
> $ LEIN_JVM_OPTS=-Xmx20M lein repl
> nREPL server started on port 61221 on host 127.0.0.1
> REPL-y 0.2.1
> Clojure 1.5.1
> Docs: (doc function-name-here)
>   (find-doc "part-of-name-here")
>   Source: (source function-name-here)
>  Javadoc: (javadoc java-object-or-class-here)
> Exit: Control+D or (exit) or (quit)
>
> user=> (defn f [g] (g))
> #'user/f
> user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)
> #'user/t1
> user=> (t1 0 (range 100))
>
> OutOfMemoryError GC overhead limit exceeded  java.lang.Long.valueOf 
> (Long.java:577)
> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
> #'user/t2
> user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)
> #'user/t1
> user=>  (t1 0 (range 100))
> nil
> user=> (t2 0 (range 100))
>
> OutOfMemoryError GC overhead limit exceeded  
> clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29)
>
> BUT this is because the previous OOM left the JVM in a dirty state: try to 
> reorder your expressions:
>
> $ LEIN_JVM_OPTS=-Xmx20M lein repl
> nREPL server started on port 61245 on host 127.0.0.1
> REPL-y 0.2.1
> Clojure 1.5.1
> Docs: (doc function-name-here)
>   (find-doc "part-of-name-here")
>   Source: (source function-name-here)
>  Javadoc: (javadoc java-object-or-class-here)
> Exit: Control+D or (exit) or (quit)
>
> user=> (defn f [g] (g))
> #'user/f
> user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)
> #'user/t2
> user=> (t2 0 (range 100))
> nil
>
> ^^this last one failed in the previous run.
>
> I'm not quite sure about why the doseq version works -- I would have to 
> research a bit. My gut feeling is that doseq is based on loop and loops are 
> lifted into ^:once fn* by the compiler in some cases. 
> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951
>
>
>
> On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft 
> > wrote:
>
>> Correct, I forgot to paste that part. ;)
>>
>>
>> On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote:
>>
>>> Just to confirm, (t2 0 (range 100)) -- using doseq instead of 
>>> dorun -- does NOT run out of memory, correct? 
>>>
>>> On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft  
>>> wrote: 
>>> > This appears to have no effect on the problem. I tested with 
>>> > jdbc/transaction, and with Sean's simple example: 
>>> > 
>>> > user=> (defn f [g] (g)) 
>>> > #'user/f 
>>> > user=> (defn t1 [n c] (f (fn [] (dorun (map identity c) 
>>> > #'user/t1 
>>> > user=> (t1 0 (range 100)) 
>>> > java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) 
>>> > user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) 
>>> > #'user/t2 
>>> > user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) 
>>> > #'user/t1 
>>> > user=> (t1 0 (range 100)) 
>>> > java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) 
>>> > 
>>> > 
>>> > 
>>> > On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand 
>>> wrote: 
>>> >> 
>>> >> ^:once on fn* (not fn, the fn macro doesn't propagate metadata) 
>>> instructs 
>>> >> the commielr to clear closed-overs ASAP. It follows that you can't 
>&g

Re: docs for ^:once

[Brian Craft]

Thanks.

In my case, I started with code something like

(defn load-exp [myseq]
  (kdb/transaction body))

where body has references to myseq. Christophe's patch of jdbc/transaction 
works. I can also work around it like

(defn load-exp [& args]
  (apply (^:once fn [myseq] (kdb/transaction body))
 args))

at which point I start wondering if it makes sense to apply ^:once to 
load-exp, but I don't really understand it.


On Thursday, September 12, 2013 11:59:15 AM UTC-7, Christian Sperandio 
wrote:
>
> You'll find a case of use of this term here:
>
> http://clj-me.cgrand.net/2013/09/11/macros-closures-and-unexpected-object-retention/
>  Le 12 sept. 2013 20:56, "Brian Craft" > 
> a écrit :
>
>> I still don't really understand ^:once. Are there docs anywhere? Is there 
>> a way to inspect a function for this property?
>>
>>  -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: another game of exploding heap, via clojure.string/split

[Brian Craft]

Ouch. Thanks for the explanation.

On Thursday, September 12, 2013 9:46:47 AM UTC-7, Andy Fingerhut wrote:
>
> Clojure's substr, and many other functions that return substrings of a 
> larger one (e.g. re-find, re-seq, etc) are based on Java's 
> java.lang.String/substring() method behavior.
>
> Before Java version 7u6 or thereabouts, this was implemented in O(1) time 
> by creating a String object that referred to an offset and length within 
> the original String object, thus retaining a reference to it as long as the 
> substrings were referenced.
>
> Around Java version 7u6, Java's substring() method behavior changed to 
> copy the desired substring into a new String object, so no references are 
> kept to the original.
>
> 
> http://www.javaadvent.com/2012/12/changes-to-stringsubstring-in-java-7.html
>
> Fun, eh?  And no, this was not obvious to me until I ran across the issue 
> some time back.  Mark Engelberg encountered this issue while doing 
> performance tuning on his Instaparse library: 
> https://github.com/Engelberg/instaparse
>
> If you know you are deploying on a Java version that is earlier than 7u6, 
> you can using the String constructor, e.g. (String. s) from Clojure to 
> force the copying of the string.  You could even get fancier and write code 
> that depends upon the Java version you are running upon, if that interests 
> you.
>
> Andy
>
>
> On Thu, Sep 12, 2013 at 9:08 AM, Brian Craft 
> > wrote:
>
>> After working around the seq + closure = death problem, I still had a 
>> severe memory leak in my code, which took many hours to find.
>>
>> Holding a reference to a string returned by clojure.string/split is 
>> somehow retaining a reference to the original string. In my case I needed 
>> to hold the first column of each row in a tsv file that was 4G in size. 
>> This resulted in holding the entire 4G in memory.
>>
>> Here's a demo. Function "data" returns a seq of lines that are about 1000 
>> bytes. The first column, however, is just a few bytes, and 10k of them 
>> should easily fit in 10M of heap space. But, no:
>>
>> $ LEIN_JVM_OPTS=-Xmx10M lein repl
>> REPL started; server listening on localhost port 34955
>> user=> (defn data [] (for [i (range)] (str "row " i "\t" 
>> (clojure.string/join "" (repeat 1000 "x")
>> #'user/data
>> user=> (def x (vec (take 1 (map #(first (clojure.string/split % 
>> #"\t")) (data)
>> java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:4)
>> user=> 
>>
>> If I copy the returned string with the String constructor, it's fine:
>>
>> $ LEIN_JVM_OPTS=-Xmx10M lein repl
>> REPL started; server listening on localhost port 20587
>> user=> (defn data [] (for [i (range)] (str "row " i "\t" 
>> (clojure.string/join "" (repeat 1000 "x")
>> #'user/data
>> user=> (def x (vec (take 1 (map #(String. (first 
>> (clojure.string/split % #"\t"))) (data)
>> #'user/x
>> user=> (x 10)
>> "row 10"
>> user=> 
>>
>> Two observations about this.
>>
>> First, this behavior is very unexpected to me. I don't understand if it 
>> is a property of strings, collections, or string/split specifically that is 
>> causing it. Is there something in the docs that I overlooked, that would 
>> have warned of this?
>>
>> Second, for tracking down problems like this, the available tooling is 
>> pathetic, to put it as politely as possible. jhat would not trace the the 
>> leaked strings. It consistently froze up when tracing them to GC roots. 
>> visualvm traced it back to CacheLRU, as in the screenshot I posted in the 
>> other thread, which was perfectly uninformative.
>>
>> Without any usable tooling, the only workflow I found to narrow the 
>> problem was to iteratively stub out portions of code and re-run the program 
>> for several minutes to determine if the leak was active. Obviously, this is 
>> incredibly painful, slow, and tedious.
>>
>> I'm hoping someone can tell me there's a better way.
>>
>> Note that the leak did not appear in when exercising subsystems 
>> independently, because in that case no references were retained from one 
>> subsystem to the other. So, "try it in the repl" was not an effective 
>> strategy.
>>  
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>>

Re: another game of exploding heap, via clojure.string/split

[Brian Craft]

Very interesting. Thanks.

On Saturday, September 14, 2013 5:48:57 AM UTC-7, TheBusby wrote:
>
> If you're working with large text files, tsv especially, you may encounter 
> Java's memory overhead for Strings as well.
> I remember parsing a 1GB TSV file into EDN, and watching it consume 10GB 
> of RAM.
>
> I'd recommend taking a look at the Iota library to see if it would be of 
> any help to you,
> https://github.com/thebusby/iota
>
>
>
> On Fri, Sep 13, 2013 at 11:47 AM, Cedric Greevey 
> 
> > wrote:
>
>> On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut 
>> 
>> > wrote:
>>
>>> I have just added some discussion of this on ClojureDocs.org for the 
>>> function clojure.core/subs, and references to that discussion for several 
>>> other Clojure functions that I am pretty sure are affected, e.g. re-find, 
>>> re-seq, re-matches, clojure.string/split, replace, replace-first
>>>
>>
>> We know with certainty that clojure.string/split is affected. Also, the 
>> OP's question about how to use tooling to track down similar leaks in the 
>> future does not appear to have been satisfactorily answered as of yet.
>>  
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Memory leak problem

[Brian Craft]

A symptom of this would be jmap or visualvm reporting [C or char[] as the 
largest allocation, by class.


On Monday, September 16, 2013 8:40:27 AM UTC-7, Andy Fingerhut wrote:
>
> Are you using a version of Java earlier than 7u6?  If so, this *might* be 
> related to the conversation from a few days ago about functions like subs, 
> re-find, etc. returning short strings that keep references to the longer 
> strings they were created from:
>
> https://groups.google.com/forum/#!topic/clojure/PeHTCWNgrL0
>
> Even if you are using such a Java version, I have not confirmed whether 
> the function clojure.tools.reader.edn/read-string returns substrings of the 
> input string.
>
> One quick thing to try that might help is to call 
> clojure.tools.reader.edn/read on the File object f directly, without 
> calling slurp.  That will at least avoid the behavior of creating big 
> strings with the entire contents of each file.  It should also avoid the 
> "small substrings holding references to large original strings", if that is 
> indeed what you are experiencing.
>
> Andy
>
>
> On Mon, Sep 16, 2013 at 8:17 AM, Joachim De Beule 
> 
> > wrote:
>
>> Dear List,
>>
>> I'm experiencing a memory leak and I don't understand why.
>>
>> I have a bunch of 50 files on disk called "data-1.edn" through 
>> "data-50.edn". I perform the following code:
>>
>> (def all-processed-data (reduce (fn [ret f] (merge ret (process-data 
>> (clojure.tools.reader.edn/read-string (slurp f) {} file-list))
>>
>> "file-list" is a sequence of java.io.File objects pointing towards the 50 
>> data files.
>> "process-data" is a function that produces a map from the data read.
>>
>> Although each individual datafile is about 100 Megabytes, the result of 
>> processing a datafile is a map of about 200k only, so that the map returned 
>> by the entire above expression is only about 10 Megabytes. Nevertheless, 
>> executing the above code fills up memory and eventually stops clojure from 
>> functioning. Why?
>>
>> Thanks a lot for any suggestions!
>>
>> Joachim.
>>  
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: another game of exploding heap, via clojure.string/split

[Brian Craft]

On Thursday, September 12, 2013 7:47:02 PM UTC-7, Cedric Greevey wrote:
>
> On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut 
> 
> > wrote:
>
>> I have just added some discussion of this on ClojureDocs.org for the 
>> function clojure.core/subs, and references to that discussion for several 
>> other Clojure functions that I am pretty sure are affected, e.g. re-find, 
>> re-seq, re-matches, clojure.string/split, replace, replace-first
>>
>
> We know with certainty that clojure.string/split is affected. Also, the 
> OP's question about how to use tooling to track down similar leaks in the 
> future does not appear to have been satisfactorily answered as of yet.
>

cricket, cricket, cricket... 

;)

Is there really no working tooling for the jvm?

The string thing bothers me less than the problem of seq heads. It is 
ridiculously easy to create a memory leak with a seq, and desperately hard 
to track one down. I would be surprised if most clojure apps were not 
leaking memory somewhere, in places that go unnoticed until a sufficiently 
large input fills the heap.

I wonder if a static analysis approach could identify code that appears to 
retain a seq head to no effect.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: another game of exploding heap, via clojure.string/split

[Brian Craft]

Awesome, thanks! I'd forgotten about the hits for MAT, because I don't use 
eclipse, and I had not found YourKit.

On Tuesday, September 17, 2013 9:13:17 AM UTC-7, Andy Fingerhut wrote:
>
> I didn't notice before posting my previous message that YourKit also has a 
> free "for open source project use only" license for their tool.  Click on 
> the "Open Source" or "License Comparison" tabs on this page:
>
> http://www.yourkit.com/purchase/index.jsp
>
>
> On Tue, Sep 17, 2013 at 8:56 AM, Andy Fingerhut 
> 
> > wrote:
>
>> Some of the hits point at commercial tools, which you didn't mention.
>>
>> I've heard positive comments about YourKit in the past.  It is 
>> commercial, but it looks pretty easy to get a 15-day evaluation license.  I 
>> haven't used it, but it claims to have some features to aid in detecting 
>> and analyzing memory leaks: 
>> http://www.yourkit.com/docs/80/help/memory_leaks.jsp
>>
>> I also saw hits for a tool called plumbr.  I have no information about 
>> it, good or bad.
>>
>> Andy
>>
>>
>>
>>
>> On Tue, Sep 17, 2013 at 8:45 AM, Brian Craft 
>> > wrote:
>>
>>> I did, of course, spend a lot of time with google before posting. All of 
>>> the hits point to jconsole, jmap, and visualvm. None of these tools work 
>>> reliably. They hang, they crash, they spit up errors, they generate useless 
>>> results. You'll note in another thread this morning another developer 
>>> having jmap and visualvm barf on them. It's not an isolated incident.
>>>
>>>
>>> On Tuesday, September 17, 2013 8:21:14 AM UTC-7, Andy Fingerhut wrote:
>>>
>>>> Another possibility: The people who know aren't reading this thread.
>>>>
>>>> I'd tell you if I knew, but I haven't needed to track down a problem 
>>>> like this for several years, and forgotten whatever tool I used at the 
>>>> time 
>>>> (it was probably jmap).
>>>>
>>>> Suggestion: Google search "java memory leak" and see what tools and 
>>>> techniques people suggest in articles they write on the topic.
>>>>
>>>> Andy
>>>>
>>>>
>>>> On Tue, Sep 17, 2013 at 8:07 AM, Brian Craft wrote:
>>>>
>>>>>  
>>>>>
>>>>> On Thursday, September 12, 2013 7:47:02 PM UTC-7, Cedric Greevey wrote:
>>>>>
>>>>>> On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut >>>>> > wrote:
>>>>>>
>>>>>>> I have just added some discussion of this on ClojureDocs.org for the 
>>>>>>> function clojure.core/subs, and references to that discussion for 
>>>>>>> several 
>>>>>>> other Clojure functions that I am pretty sure are affected, e.g. 
>>>>>>> re-find, 
>>>>>>> re-seq, re-matches, clojure.string/split, replace, replace-first
>>>>>>>
>>>>>>
>>>>>> We know with certainty that clojure.string/split is affected. Also, 
>>>>>> the OP's question about how to use tooling to track down similar leaks 
>>>>>> in 
>>>>>> the future does not appear to have been satisfactorily answered as of 
>>>>>> yet.
>>>>>>
>>>>>
>>>>> cricket, cricket, cricket... 
>>>>>
>>>>> ;)
>>>>>
>>>>> Is there really no working tooling for the jvm?
>>>>>
>>>>> The string thing bothers me less than the problem of seq heads. It is 
>>>>> ridiculously easy to create a memory leak with a seq, and desperately 
>>>>> hard 
>>>>> to track one down. I would be surprised if most clojure apps were not 
>>>>> leaking memory somewhere, in places that go unnoticed until a 
>>>>> sufficiently 
>>>>> large input fills the heap.
>>>>>
>>>>> I wonder if a static analysis approach could identify code that 
>>>>> appears to retain a seq head to no effect.
>>>>>
>>>>>  -- 
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clo...@googlegroups.com
>>>>>
>>>>> Note tha

Re: another game of exploding heap, via clojure.string/split

[Brian Craft]

I did, of course, spend a lot of time with google before posting. All of 
the hits point to jconsole, jmap, and visualvm. None of these tools work 
reliably. They hang, they crash, they spit up errors, they generate useless 
results. You'll note in another thread this morning another developer 
having jmap and visualvm barf on them. It's not an isolated incident.

On Tuesday, September 17, 2013 8:21:14 AM UTC-7, Andy Fingerhut wrote:
>
> Another possibility: The people who know aren't reading this thread.
>
> I'd tell you if I knew, but I haven't needed to track down a problem like 
> this for several years, and forgotten whatever tool I used at the time (it 
> was probably jmap).
>
> Suggestion: Google search "java memory leak" and see what tools and 
> techniques people suggest in articles they write on the topic.
>
> Andy
>
>
> On Tue, Sep 17, 2013 at 8:07 AM, Brian Craft 
> > wrote:
>
>>
>>
>> On Thursday, September 12, 2013 7:47:02 PM UTC-7, Cedric Greevey wrote:
>>
>>> On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut wrote:
>>>
>>>> I have just added some discussion of this on ClojureDocs.org for the 
>>>> function clojure.core/subs, and references to that discussion for several 
>>>> other Clojure functions that I am pretty sure are affected, e.g. re-find, 
>>>> re-seq, re-matches, clojure.string/split, replace, replace-first
>>>>
>>>
>>> We know with certainty that clojure.string/split is affected. Also, the 
>>> OP's question about how to use tooling to track down similar leaks in the 
>>> future does not appear to have been satisfactorily answered as of yet.
>>>
>>
>> cricket, cricket, cricket... 
>>
>> ;)
>>
>> Is there really no working tooling for the jvm?
>>
>> The string thing bothers me less than the problem of seq heads. It is 
>> ridiculously easy to create a memory leak with a seq, and desperately hard 
>> to track one down. I would be surprised if most clojure apps were not 
>> leaking memory somewhere, in places that go unnoticed until a sufficiently 
>> large input fills the heap.
>>
>> I wonder if a static analysis approach could identify code that appears 
>> to retain a seq head to no effect.
>>
>>  -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Memory leak problem

[Brian Craft]

I'm learning that the tooling for the jvm covers a spectrum from 
pathetically broken to non-existent.

I've had some luck running jmap like "jmap -histo:live ". Pipe it 
through head and run with watch, and you have a crude real-time monitor. 
E.g. running against the processing running from a SNAPSHOT uberjar:

watch jmap -histo:live $(ps axu | grep SNA[P] | sed -e 's/ \+/\t/g' | 
cut -f2) \| head -40





On Tuesday, September 17, 2013 2:14:20 AM UTC-7, Joachim De Beule wrote:
>
> Dear Brian,
>
> Thanks for your reply. I tried to use jmap, unfortunately it fails:
>
> bash-3.2$ jmap -F -dump:file=heap.bin 58708
> Attaching to process ID 58708, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.51-b01-457
> Dumping heap to heap.bin ...
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jmap.JMap.runTool(JMap.java:179)
> at sun.tools.jmap.JMap.main(JMap.java:110)
> Caused by: sun.jvm.hotspot.oops.UnknownOopException
>
> As for visualvm: it never seems to finish producing a heap dump :( 
>
> Joachim. 
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

*db* binding with jetty

[Brian Craft]

Trying to get a small app off the ground with noir/jdbc/jetty (jdbc 0.2.2, 
I think), I'm getting a "No valid DB connection selected" error.

I suspect this is because I start it something like

(with-connection mydb
  (server/start))

which creates a thread-local binding of *db* in jdbc, then starts jetty in 
a different thread, with no binding for *db*.

Does that sound correct?

If so, I'm not sure how to work around it.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: *db* binding with jetty

[Brian Craft]

Even in that case you have to get the db info to the handlers somehow. A 
lot of demo code I see just hard-codes the db stuff, which is not useful.

I can make it work via middleware, like

(partial db db-middleware)

then add db-middleware to ring before starting jetty.

I'm not sure if this is a good plan, or how the db stuff will handle 
multiple threads.



On Monday, September 23, 2013 11:42:59 AM UTC-7, Andy Fingerhut wrote:
>
> Caveat: I am experienced with Clojure, but a newbie at web development in 
> *any* language.  I have recently begun going through the beta version of 
> the book "Web Development with Clojure"
>
> http://pragprog.com/book/dswdcloj/web-development-with-clojure
>
> Hopefully others with more web dev experience will respond with more 
> experience and knowledge to back up their answers.
>
> In the simple web server code I've been working with in the book, it does 
> not make a connection to the database when jetty is started.  Instead, a 
> new connection is made to the database every time a request is handled, and 
> the connection is closed before the response is sent back.  All of the 
> examples I have seen so far handle all requests within a single thread, so 
> it does not hit the issue you have run into.
>
> There are probably fancier techniques used when people are trying to 
> squeeze more performance out of their database interaction.
>
> Andy
>
>
> On Mon, Sep 23, 2013 at 11:25 AM, Brian Craft 
> > wrote:
>
>> Trying to get a small app off the ground with noir/jdbc/jetty (jdbc 
>> 0.2.2, I think), I'm getting a "No valid DB connection selected" error.
>>
>> I suspect this is because I start it something like
>>
>> (with-connection mydb
>>   (server/start))
>>
>> which creates a thread-local binding of *db* in jdbc, then starts jetty 
>> in a different thread, with no binding for *db*.
>>
>> Does that sound correct?
>>
>> If so, I'm not sure how to work around it.
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: *db* binding with jetty

[Brian Craft]

My question is more about how to pass something to the app without sticking 
it in a global. This might be a noir question. Handlers are called without 
any passed context. I'm not sure if that's peculiar to noir, or something 
inherited from ring. So, the only way I know to get context to the app is 
to merge it into the request context with middleware.

(defn db-middleware [db app]
  (fn [req]
(app (assoc req :db db


where db could be the connection pool, or db credentials, or whatever else 
is required to make a connection.

On Monday, September 23, 2013 12:35:26 PM UTC-7, Sean Corfield wrote:
>
> Take a look at 
> http://clojure-doc.org/articles/ecosystem/java_jdbc/connection_pooling.html 
>
> You would want to set up a connection pool and pass that into your app 
> as part of its context. 
>
> Does that help? 
>
> Sean 
>
> On Mon, Sep 23, 2013 at 11:25 AM, Brian Craft 
> > 
> wrote: 
> > Trying to get a small app off the ground with noir/jdbc/jetty (jdbc 
> 0.2.2, I 
> > think), I'm getting a "No valid DB connection selected" error. 
> > 
> > I suspect this is because I start it something like 
> > 
> > (with-connection mydb 
> >   (server/start)) 
> > 
> > which creates a thread-local binding of *db* in jdbc, then starts jetty 
> in a 
> > different thread, with no binding for *db*. 
> > 
> > Does that sound correct? 
> > 
> > If so, I'm not sure how to work around it. 
> > 
> > -- 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "Clojure" group. 
> > To post to this group, send email to clo...@googlegroups.com 
> > Note that posts from new members are moderated - please be patient with 
> your 
> > first post. 
> > To unsubscribe from this group, send email to 
> > clojure+u...@googlegroups.com  
> > For more options, visit this group at 
> > http://groups.google.com/group/clojure?hl=en 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Clojure" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to clojure+u...@googlegroups.com . 
> > For more options, visit https://groups.google.com/groups/opt_out. 
>
>
>
> -- 
> Sean A Corfield -- (904) 302-SEAN 
> An Architect's View -- http://corfield.org/ 
> World Singles, LLC. -- http://worldsingles.com/ 
>
> "Perfection is the enemy of the good." 
> -- Gustave Flaubert, French realist novelist (1821-1880) 
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Parallelising over a lazy sequence - request for help

[Brian Craft]

On the other hand it is 2013, not 2003. 40G is small in terms of modern 
hardware. Terabyte ram servers have been available for awhile, at prices 
within the reach of many projects. "Large" data in this decade is measured 
in petabytes, at least.


On Sunday, September 29, 2013 5:13:14 PM UTC-7, Paul Mooser wrote:
>
> Thanks - when I said "small", I was referring to the fact that your tests 
> were using the first 1 pages, as opposed to the entire data dump. Sorry 
> if I was unclear or misunderstood. 
>
> On Sunday, September 29, 2013 3:20:38 PM UTC-7, Paul Butcher wrote:
>>
>> The dataset I'm using is a Wikipedia dump, which hardly counts as "small" 
>> :-)
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

streams, reading bytes

[Brian Craft]

I'm struggling with how to gzip/gunzip byte arrays from clojure. I don't 
really get how clojure.java.io streams supposed to be used.

This appears to gzip a string:

(let [baos (java.io.ByteArrayOutputStream.)
  gzos (java.util.zip.GZIPOutputStream. baos)]
(.write gzos (.getBytes "foo"))
(.finish gzos)
(.toByteArray baos))


and I can expand on this to gunzip the string afterwards:

(let [baos (java.io.ByteArrayOutputStream.)
  gzos (java.util.zip.GZIPOutputStream. baos)]
  (.write gzos (.getBytes "foo"))
  (.finish gzos)
  (let [gzbytes (.toByteArray baos)
bais (java.io.ByteArrayInputStream. gzbytes)
gzis (java.util.zip.GZIPInputStream. bais)]
(slurp gzis)))

But slurp builds strings. I need to do this with byte arrays.  Do I have to 
rewrite slurp to build a byte array? Is there any simpler way to do this?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: streams, reading bytes

[Brian Craft]

I like the blurb:

"This library is a Rosetta stone for all the byte representations Java has 
to offer, and gives you the freedom to forget all the APIs you never wanted 
to know in the first place."

My feelings, exactly. ;)

Thanks.

On Thursday, October 10, 2013 6:44:32 PM UTC-7, Ben Mabey wrote:
>
> On 10/10/13 7:21 PM, Brian Craft wrote: 
> > I'm struggling with how to gzip/gunzip byte arrays from clojure. I 
> > don't really get how clojure.java.io streams supposed to be used. 
> > 
> > This appears to gzip a string: 
> > 
> > (let [baos (java.io.ByteArrayOutputStream.) 
> >   gzos (java.util.zip.GZIPOutputStream. baos)] 
> > (.write gzos (.getBytes "foo")) 
> > (.finish gzos) 
> > (.toByteArray baos)) 
> > 
> > 
> > and I can expand on this to gunzip the string afterwards: 
> > 
> > (let [baos (java.io.ByteArrayOutputStream.) 
> >   gzos (java.util.zip.GZIPOutputStream. baos)] 
> >   (.write gzos (.getBytes "foo")) 
> >   (.finish gzos) 
> >   (let [gzbytes (.toByteArray baos) 
> > bais (java.io.ByteArrayInputStream. gzbytes) 
> > gzis (java.util.zip.GZIPInputStream. bais)] 
> > (slurp gzis))) 
> > 
> > But slurp builds strings. I need to do this with byte arrays.  Do I 
> > have to rewrite slurp to build a byte array? Is there any simpler way 
> > to do this? 
>
> Zach Tellman's byte-streams library is very handy for these kinds of 
> conversions: 
>
> https://github.com/ztellman/byte-streams 
>
> -Ben 
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

with-open

[Brian Craft]

I'm working on module that may use a number of different backends, some of 
which may keep open files across a number of API calls. Other backends may 
not use files at all. Like

(backend-init) ; opens some files, opaque to the caller
(backend-do-something); uses the open files
(backend-finish); close the files

Off-hand, I don't see a way to structure code like this to use with-open, 
since the backend would lose its open files when returning from any 
with-open it might do.

Am I approaching this the wrong way? Is there a standard way of structuring 
code like this? It's a bit like the backend is itself the resource to be 
opened (under the hood would be opening multiple files). I saw a thread 
about making a "close" multimethod that sounded related.

Also, when the backend is opening files it will need to catch errors and 
close any opened files (e.g. if the second open fails, close the first), 
which is a bit like with-open, except it would call "close" only on error. 
Is this an existing method or idiom for this?

I guess one alternative is to pass a body to be executed to the backend, so 
the backend can wrap it in with-open.


-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

making a lot of small calcs parallel

[Brian Craft]

I'm walking a seq of many millions of floats, encoding them for the 
persistence layer, and getting sequence ids from the db. So, conceptually 
there are two parts: the slow part, and the side-effecting part. Vaguely 
like

(map get-ids (map encode float-seq))

which is later reduced while writing to disk. In the get-ids step the order 
matters. My first attempt to make the slow part parallel was to use pmap,

(map get-ids (pmap encode float-seq))

However that's actually slower. I expect this is because even though 
"encode" is the bottleneck, it's still faster than the overhead of pmap. I 
next tried pmap over groups of floats, a bit like

(map get-ids (flatten (pmap #(map encode %) (partition-all 2 
float-seq

(sorry for any typos, I'm just pseudo-coding here) This was still slower, 
which surprised me. I understand the first pmap result, but this one is 
puzzling to me. Even if I partition half the length of the seq (so in 
theory it can run two threads, each of which will run five or six seconds), 
it's no faster than map.  Part of this seems to be the overhead of creating 
more intermediate seqs. Perhaps I'm misunderstanding what's happening 
during partition-all.

Is there some obvious way to approach this scenario? I looked briefly at 
the reducers library, however it was unclear to me how to deal with the 
side-effecting portion of the operation. The second (fast) map operation 
needs to be done in order.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

avoiding generators

[Brian Craft]

I need to pull sequence ids from a db, which is a lot like using a 
generator. I found this thread about generators:

https://groups.google.com/forum/?fromgroups=#!searchin/clojure/generator/clojure/IL_ONPb52-0/S_2p2-vjVCYJ

which suggests using seqs, like so:

(defn sines [] 
   (cycle (map #(Math/sin %) (range 0 6.28 0.01 

(doseq [s (take num-iters (sines))] 
  (println s)) 

However my case is more complex, because there's not a single call (like 
doseq, above) consuming the seq. I'm processing structured data, and need 
to get ids for the leaf nodes. So it looks more like

(defn process-row [row]
  (map #(process-value (get-id) %) row))

(map process-row rows)

That is, process-row gets called multiple times, and needs to retrieve ids. 
If I pass in a seq of ids, I'll get the same ids each time. If I try to 
work around this by, say, have process-row return the number of ids 
consumed, and doing a "drop n" when calling process-row the next time, I'm 
holding the seq head (which I can't afford).

Is there some other way to do this?


-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: making a lot of small calcs parallel

[Brian Craft]

Hm. float-seq may not fit in memory. Perhaps I can read it in blocks.

On Monday, October 14, 2013 11:01:52 AM UTC-7, Herwig Hochleitner wrote:
>
> + make sure to pour float-seq into a vector before r/map, to make full use 
> of parallel folding
>
>
> 2013/10/14 Herwig Hochleitner >
>
>> Try
>>
>> (require '[clojure.core.reducers :as r])
>> (reduce (fn [res val] (get-ids val))
>> nil (r/map encode float-seq))
>>
>> This should parallel fold encode over float-seq (r/map) and then map 
>> get-ids in order, but without allocation.
>>
>>
>> 2013/10/14 Brian Craft >
>>
>>> I'm walking a seq of many millions of floats, encoding them for the 
>>> persistence layer, and getting sequence ids from the db. So, conceptually 
>>> there are two parts: the slow part, and the side-effecting part. Vaguely 
>>> like
>>>
>>> (map get-ids (map encode float-seq))
>>>
>>> which is later reduced while writing to disk. In the get-ids step the 
>>> order matters. My first attempt to make the slow part parallel was to 
>>> use pmap,
>>>
>>> (map get-ids (pmap encode float-seq))
>>>
>>> However that's actually slower. I expect this is because even though 
>>> "encode" is the bottleneck, it's still faster than the overhead of pmap. I 
>>> next tried pmap over groups of floats, a bit like
>>>
>>> (map get-ids (flatten (pmap #(map encode %) (partition-all 2 
>>> float-seq
>>>
>>> (sorry for any typos, I'm just pseudo-coding here) This was still 
>>> slower, which surprised me. I understand the first pmap result, but this 
>>> one is puzzling to me. Even if I partition half the length of the seq (so 
>>> in theory it can run two threads, each of which will run five or six 
>>> seconds), it's no faster than map.  Part of this seems to be the 
>>> overhead of creating more intermediate seqs. Perhaps I'm misunderstanding 
>>> what's happening during partition-all.
>>>
>>> Is there some obvious way to approach this scenario? I looked briefly at 
>>> the reducers library, however it was unclear to me how to deal with the 
>>> side-effecting portion of the operation. The second (fast) map operation 
>>> needs to be done in order.
>>>  
>>> -- 
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com 
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to clojure+u...@googlegroups.com .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

finding a call to longCast

[Brian Craft]

Profiling shows clojure.lang.RT.longCast is currently 25% of the run time 
of the code I'm working on. It's being called from a routine that 
manipulates primitive arrays while building a hex string, with aget, aset, 
inc, and bitwise operations. Looking through the source for those, I can't 
see where it's being called. How can I find it?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: making a lot of small calcs parallel

[Brian Craft]

Answering my own question, my grouping example failed because of the 
laziness of the expressions being computed in the pmap: the threads were 
not evaluating the expressions until they were evaluated (sequentially) by 
the main thread. Adding a doall gives the result I was hoping for.

On Sunday, October 13, 2013 4:10:16 PM UTC-7, Brian Craft wrote:
>
> I'm walking a seq of many millions of floats, encoding them for the 
> persistence layer, and getting sequence ids from the db. So, conceptually 
> there are two parts: the slow part, and the side-effecting part. Vaguely 
> like
>
> (map get-ids (map encode float-seq))
>
> which is later reduced while writing to disk. In the get-ids step the 
> order matters. My first attempt to make the slow part parallel was to use 
> pmap,
>
> (map get-ids (pmap encode float-seq))
>
> However that's actually slower. I expect this is because even though 
> "encode" is the bottleneck, it's still faster than the overhead of pmap. I 
> next tried pmap over groups of floats, a bit like
>
> (map get-ids (flatten (pmap #(map encode %) (partition-all 2 
> float-seq
>
> (sorry for any typos, I'm just pseudo-coding here) This was still slower, 
> which surprised me. I understand the first pmap result, but this one is 
> puzzling to me. Even if I partition half the length of the seq (so in 
> theory it can run two threads, each of which will run five or six seconds), 
> it's no faster than map.  Part of this seems to be the overhead of 
> creating more intermediate seqs. Perhaps I'm misunderstanding what's 
> happening during partition-all.
>
> Is there some obvious way to approach this scenario? I looked briefly at 
> the reducers library, however it was unclear to me how to deal with the 
> side-effecting portion of the operation. The second (fast) map operation 
> needs to be done in order.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: making a lot of small calcs parallel

[Brian Craft]

No, it's still lazy. Partitions are handed off to threads as they are 
needed, and the results are collected in the main thread. I found the same 
code in the O'Reilly book after I figured it out, in the "Parallelism on 
the Cheap" section. It points out the problem with pmap over small 
computations, and has the workaround with partition-all and doall. 
Apparently I should have started there. :-p

On Monday, October 14, 2013 10:13:09 PM UTC-7, Sean Corfield wrote:
>
> I nearly suggested that but it sounded so counter-intuitive and I 
> didn't have time to construct a test-bed for it... glad you figured it 
> out. That means that your float-seq has to be fully realized in memory 
> tho', right? 
>
> On Mon, Oct 14, 2013 at 9:58 PM, Brian Craft 
> > 
> wrote: 
> > Answering my own question, my grouping example failed because of the 
> > laziness of the expressions being computed in the pmap: the threads were 
> not 
> > evaluating the expressions until they were evaluated (sequentially) by 
> the 
> > main thread. Adding a doall gives the result I was hoping for. 
> > 
> > 
> > On Sunday, October 13, 2013 4:10:16 PM UTC-7, Brian Craft wrote: 
> >> 
> >> I'm walking a seq of many millions of floats, encoding them for the 
> >> persistence layer, and getting sequence ids from the db. So, 
> conceptually 
> >> there are two parts: the slow part, and the side-effecting part. 
> Vaguely 
> >> like 
> >> 
> >> (map get-ids (map encode float-seq)) 
> >> 
> >> which is later reduced while writing to disk. In the get-ids step the 
> >> order matters. My first attempt to make the slow part parallel was to 
> use 
> >> pmap, 
> >> 
> >> (map get-ids (pmap encode float-seq)) 
> >> 
> >> However that's actually slower. I expect this is because even though 
> >> "encode" is the bottleneck, it's still faster than the overhead of 
> pmap. I 
> >> next tried pmap over groups of floats, a bit like 
> >> 
> >> (map get-ids (flatten (pmap #(map encode %) (partition-all 2 
> >> float-seq 
> >> 
> >> (sorry for any typos, I'm just pseudo-coding here) This was still 
> slower, 
> >> which surprised me. I understand the first pmap result, but this one is 
> >> puzzling to me. Even if I partition half the length of the seq (so in 
> theory 
> >> it can run two threads, each of which will run five or six seconds), 
> it's no 
> >> faster than map.  Part of this seems to be the overhead of creating 
> more 
> >> intermediate seqs. Perhaps I'm misunderstanding what's happening during 
> >> partition-all. 
> >> 
> >> Is there some obvious way to approach this scenario? I looked briefly 
> at 
> >> the reducers library, however it was unclear to me how to deal with the 
> >> side-effecting portion of the operation. The second (fast) map 
> operation 
> >> needs to be done in order. 
> > 
> > -- 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "Clojure" group. 
> > To post to this group, send email to clo...@googlegroups.com 
> > Note that posts from new members are moderated - please be patient with 
> your 
> > first post. 
> > To unsubscribe from this group, send email to 
> > clojure+u...@googlegroups.com  
> > For more options, visit this group at 
> > http://groups.google.com/group/clojure?hl=en 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Clojure" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to clojure+u...@googlegroups.com . 
> > For more options, visit https://groups.google.com/groups/opt_out. 
>
>
>
> -- 
> Sean A Corfield -- (904) 302-SEAN 
> An Architect's View -- http://corfield.org/ 
> World Singles, LLC. -- http://worldsingles.com/ 
>
> "Perfection is the enemy of the good." 
> -- Gustave Flaubert, French realist novelist (1821-1880) 
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: is PG's "imperative outside-in" advice any good?

[Brian Hurt]

Lifting subexpressions up into lets is actually something I do a lot- for
one very important reason: it lets me insert print statements (or logging
statements) showing the value of the subexpression.  So I'll do;
(let [ x (subexpression) ]
(main-expression))

because it lets me do:
(let [ x (subexpression) ]
(println "The value of x is" x)
(main-expression))

If fact, a lot of times I'll do;
(let [ x (subexpression)
res (main-expression) ]
res)

because it lets me do:
(let [ x (subexpression)
_ (println "The value of x is" x)
res (main-expression) ]
(println "The value of the whole expression is" res)
res)

This is of great value in debugging.

Brian



On Tue, Oct 15, 2013 at 9:56 AM, Mikera wrote:

> I certainly prefer giving names to intermediate results with a "let"
> block: having good names and breaking the computation up into logical
> chunks makes the code much easier to understand and maintain when you come
> back to it later.
>
> PG's example though is bad for different reasons - this is actually
> mutating variables in an imperative style, which is definitely "bad style"
> - both in Lisp and Clojure I think. The Clojure equivalent would be to use
> atoms (or vars) and mutating them.
>
> "let" on its own is purely functional, and doesn't have this problem.
>
>
> On Tuesday, 15 October 2013 20:29:29 UTC+8, Daniel Higginbotham wrote:
>>
>> I've been going through On Lisp by Paul Graham and on page 33 he
>> recommends against performing "intermediate" bindings. Does this advice
>> hold for Clojure? Here are a couple examples:
>>
>> ;; Common Lisp (from the book)
>> (defun bad (x)
>>  (let (y sqr)
>>(setq y (car x))
>>(setq sqr (expt y 2))
>>(list 'a sqr)))
>>
>> (defun good (x)
>>  (list 'a (expt (car x) 2)))
>>
>> ;; Clojure
>> (defn bad [x]
>>  (let [y (first x)
>>sqr (expt y 2)]
>>(list 'a sqr)))
>>
>> (defn good [x]
>>  (list 'a (expt (first x) 2)))
>>
>> Paul Graham explains:
>>
>> "The final result is shorter than what we began with, and easier to
>> understand. In the original code, we’re faced with the final expression
>> (list 'a sqr), and it’s not immediately clear where the value of sqr comes
>> from. Now the source of the return value is laid out for us like a road
>> map.
>>
>> The example in this section was a short one, but the technique scales up.
>> Indeed, it becomes more valuable as it is applied to larger functions."
>>
>> In clojure you can't do setq of course but I find myself going against
>> this advice all the time, and I find that it's more important to do so when
>> working with larger functions. I think introducing names makes code
>> clearer. Here's an example from my own code:
>>
>> (defn create-topic
>>  [params]
>>  (let [params (merge params (db/tempids :topic-id :post-id :watch-id))
>>topic (remove-nils-from-map (c/mapify params mr/topic->txdata))
>>watch (c/mapify params mr/watch->txdata)
>>post (c/mapify params mr/post->txdata)]
>>{:result (db/t [topic post watch])
>> :tempid (:topic-id params)}))
>>
>> To my mind, creating bindings for "topic", "watch", and "post" makes the
>> code easier to understand. When you get to "(db/t [topic post watch])" you
>> don't have to deal with as much visual noise to understand exactly what's
>> going into the transaction.
>>
>> So, is PG's advice any good?
>>
>> Thanks!
>> Daniel
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

reflection warning with threading macro

[Brian Craft]

This gives me a reflection warning:

(.write out ^String (-> slist meta :hex))

This does not:

(.write out ^String (:hex (meta slist)))

=> (clojure.walk/macroexpand-all '(-> slist meta :hex))
(:hex (meta slist))

What's going on? Is there some other way to type hint the case with the 
threading macro?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: doseq vs dorun

[Brian Craft]

I have the same use case: walking a seq of an input file, and doing file/db 
operations for each row. pmap is working very well, but it has required a 
lot of attention to the data flow, to make sure that no significant compute 
is done in the main thread. Otherwise IO blocks the compute.

I briefly tried working with the reducers library, which generally made 
things 2-3 times slower, presumably because I'm using it incorrectly. I 
would really like to see more reducers examples, e.g. for this case: 
reading a seq larger than memory, doing transforms on the data, and then 
executing side effects.

On Thursday, October 17, 2013 4:04:51 AM UTC-7, Mikera wrote:
>
> On Thursday, 17 October 2013 10:34:18 UTC+8, Pradeep Gollakota wrote:
>
>> Hi All,
>>
>> I’m (very) new to clojure (and loving it)… and I’m trying to wrap my head 
>> around how to correctly choose doseq vs dorun for my particular use case. 
>> I’ve read this earlier post 
>> https://groups.google.com/forum/#!msg/clojure/8ebJsllH8UY/mXtixH3CRRsJand I 
>> had a clarifying question.
>>
>> From what I gathered in the above post, it’s more efficient to use doseq 
>> instead of dorun since map creates another seq. However, if the fn you want 
>> to apply on the seq can be parallelized, doseq wouldn’t give you the 
>> ability to parallelize. With dorun you can use pmap instead of map and get 
>> parallelization.
>>
>> (doseq [i some-lazy-seq] side-effect-fn)
>> (dorun (pmap side-effect-fn some-lazy-seq))
>>
>> What is the idiomatic way of parallelizing a computation on a lazy seq?
>>
> I don't think there is a single idiomatic way. It depends on lots of 
> things, e.g.:
> - How expensive is each side-effect-fn? If it is cheap, then the ovehead 
> of making things parallel may not be worth it
> - Do you want to constrain the thread pool or have a separate thread for 
> each element? For the later, futures are an option
> - Where is the actual bottleneck? If an external resource is constrained, 
> CPU parallelization may not help you at all.
> - How is the lazy sequence being produced? Is it already realised, or 
> being computed on the fly?
> - Is there any concern about ordering / concurrent access to resources / 
> race conditions?
>
> Assuming that side-effect-fn is relatively CPU-expensive and that the 
> runtimes of each call to it are reasonably similar, then I'd say that your 
> (dorun (pmap .)) version is a decent choice. Otherwise you make want to 
> take a look at the "reducers" library - the Fork/Join capabilities are very 
> impressive and should do what you need.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

getting a stack trace

[Brian Craft]

In trying to understand how threads work I'd like to dump a stack trace at 
various places. All the stack trace calls take an exception. Is there some 
simpler way, or should I throw & catch an except just to get a stack trace?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: getting a stack trace

[Brian Craft]

ah, sorry, that's a shell tool? I meant dumping a stack trace from code. 
Throwing & catching works, it's just a bit goofy. Like

(defn- stack-trace [msg]
  (try
(throw (IllegalArgumentException. msg))
(catch IllegalArgumentException e
  (stacktrace/print-stack-trace e


On Friday, October 18, 2013 10:17:43 AM UTC-7, Alex Baranosky wrote:
>
> > jstack 
>
>
> On Fri, Oct 18, 2013 at 9:44 AM, Brian Craft 
> > wrote:
>
>> In trying to understand how threads work I'd like to dump a stack trace 
>> at various places. All the stack trace calls take an exception. Is there 
>> some simpler way, or should I throw & catch an except just to get a stack 
>> trace?
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: getting a stack trace

[Brian Craft]

Yes, thanks!

On Friday, October 18, 2013 12:42:00 PM UTC-7, Stanislav Sedov wrote:
>
>
> On Oct 18, 2013, at 12:35 PM, Brian Craft > 
> wrote: 
>
> > ah, sorry, that's a shell tool? I meant dumping a stack trace from code. 
> Throwing & catching works, it's just a bit goofy. Like 
> > 
> > (defn- stack-trace [msg] 
> >   (try 
> > (throw (IllegalArgumentException. msg)) 
> > (catch IllegalArgumentException e 
> >   (stacktrace/print-stack-trace e 
>
> Seems like what you want is 
> ,--- 
> | (Thread/dumpStack) 
> `--- 
> . 
>
> -- 
> ST4096-RIPE 
>
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

binding and lazy seqs

[Brian Craft]

I just came across this example in "Clojure Programming", chapt 4:

(def ^:dynamic *max-value* 255)
(defn valid-value? [v] (<= v *max-value*))

(binding [*max-value* 500] (map valid-value? [299]))
;= (false)

It's not really explained in the text. I'm guessing this happens because 
when the (binding...) expression is evaluated, it merely returns the head 
of the seq. The values of the seq aren't evaluated until the repl 
implicitly realizes the return value. (I can see the point of this repl 
behavior, but it has often misled me by giving me different results than I 
get when running code outside the repl).

The workaround in the book is to move the "binding" into the map function, 
though I would think "doall" would be a better solution.

(binding [*max-value* 500] (doall (map valid-value? [299])))
;= (true)

This seems like the same issue that comes up with libraries that implement 
a with- API: with-db, and so-forth. You have to walk any seqs within 
the binding form if the binding is to have any affect on the values in the 
seqs.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

structuring a db load, back pressure

[Brian Craft]

I'm doing a load to db, which looks roughly like "read, transform, insert, 
repeat".

Blocking on the inserts leaves the cores sitting cold while they could be 
doing the next read/transform. It's tempting to try an agent for the 
inserts. If I understand them correctly, that would queue the inserts as 
they are ready, allowing the read/transform to continue. I'm concerned that 
this would just fill memory, as the inserts can't keep up with the reads.

Is there some other obvious way to structure this? The only relevant hits 
I've found on this list are about the async lib, which I gather can handle 
this by keeping a bounded queue that can block the producer. I was hoping 
to put off learning the async stuff until later. ;)

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: What non-deprecated Clojure Web libraries to use?

[Brian Craft]

Try create-db.

On Monday, October 28, 2013 1:06:49 AM UTC-7, Manuel Paccagnella wrote:
>
> Il giorno lunedì 28 ottobre 2013 04:12:50 UTC+1, Christopher Allen ha 
> scritto:
>
> You can use Korma with Stuart Sierra's workflow just fine.
>>
>> Really? Nice! Last time I tried I didn’t managed to get it to work 
> properly. 
>
> What happens to an open connection binded to a Var (via defdb) when the 
> ns gets reloaded? Or maybe I should use korma.db/get-connectionexplicitly in 
> the system starting phase?
>
> Thank you for your feedback.
>
> On Sunday, October 27, 2013 5:07:02 PM UTC-7, Manuel Paccagnella wrote:
>>>
>>> Il giorno lunedì 28 ottobre 2013 00:30:06 UTC+1, Alexander Hudek ha 
>>> scritto:
>>>
>>> http://www.luminusweb.net/ gives a reasonable starting setup. The only 
 thing I would recommend doing differently is to use clojure/java.jdbc or 
 honeysql instead of korma for an sql dsl.

>>> I agree. Korma is quite interesting as a DSL, but currently 
>>> usesVars 
>>> under the hood to manage connections and this doesn’t play well with 
>>> dynamic workflows as the 
>>> oneby 
>>> Stuart Sierra.
>>>
>>> Personally I’ve used just 
>>> clojure.java.jdbc(
>>> here  is a micro-tutorial that 
>>> I’ve written for it) that is going under an API overhaul to make 
>>> connections management more explicit and functional. If you need an SQL 
>>> generation library you could look at Honey 
>>> SQL. 
>>>
>>>
>>> Hope it helps,
>>>
>>> Manuel 
>>>
>>> PS: You could also take a look at the new book Web Development in 
>>> Clojure by 
>>> the author of 
>>> Luminus .
>>>
>>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: What non-deprecated Clojure Web libraries to use?

[Brian Craft]

On Monday, October 28, 2013 4:51:54 PM UTC-7, Alexander Hudek wrote:
>
>
> It is incredibly hard to write a clean sql dsl due to differences in how 
> various database drivers work, and also due to how complex sql itself is. 
> It's worth noting that you can always selectively fall back to jdbc if you 
> encounter any of these issues. Korma uses jdbc itself and works well with 
> it. We instead mix jdbc with light honeysql usage.
>

Of course, this is one of the big reasons an abstraction layer is critical 
for any project larger than a toy: it makes absolutely no sense for every 
application developer to independently discover every fiddling little 
difference between every sql implementation, and to work around every 
complexity in structuring sql queries for boilerplate tables. It's a 
completely waste of time. It only makes sense to dive into that when 
dealing with data that do not fit the 99% use case of simple tables, with 
standard relations (1-1, 1-m, m-m), and low performance requirements. 
 Application engineering time should be spent on the problems that aren't 
already solved, e.g. the petabyte scale data, the low-latency data, the 
data that doesn't fit in squares, etc.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: What non-deprecated Clojure Web libraries to use?

[Brian Craft]

On Monday, October 28, 2013 4:36:56 PM UTC-7, Chris Kuttruff wrote:
>
> Separate from DSLs like Korma, etc.  I have written a simple library for 
> doing database migrations with clojure (clj-sql-up ( 
> https://github.com/ckuttruff/clj-sql-up )).  There are also other 
> libraries still maintained along these lines (drift, migratus, ragtime, 
> etc.)
>
>
It's unfortunate that these are separate, because you need the schema 
information not just for migrations, but also for query abstraction (sql 
dsl, etc.). The argument for small, composable libraries only works if they 
can actually be meaningfully composed: if, in this case, a declared schema 
can be used for migrations, and query abstraction, and administrative UI, 
and anything else that requires it. So far there's not much like this in 
clojure that I've found.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: What non-deprecated Clojure Web libraries to use?

[Brian Craft]

In general, my point is that libraries don't compose if they have 
incompatible or hidden representations of the data structures over which 
they operate, which is the default condition if no one has thought about 
how the libraries might be used together. A consequence of this is that a 
framework is much, much greater than the sum of its parts.

A dsl that can abstract away details of building queries (e.g. joins) 
requires a declared schema. In contrast, the clojure migration tools 
describe a schema as a series of functions, or sql string literals. It's 
hard to derive one from the other. You wouldn't want to start trying to 
parse the sql to deduce the schema, for instance. Consequently you end up 
repeating the schema. Then you add an administrative UI, and you have the 
same problem: the pages and forms for the admin depend on the schema. You 
end up repeating the schema a third time. And so forth. It quickly becomes 
unmanageable.

For this case, migrations, it's easier to derive the sql for the migrations 
from a declared schema (doing a diff against the previous version) than the 
other way around (parsing sql to find the schema). This is how django-south 
works (it automatically generates the sql for the migrations, in most 
cases), but there's nothing like it for clojure that I'm aware of. Also, 
the sql dsls in clojure that I've seen cover very little of sql for data 
model creation, so you can't actually compose them with the migration tools 
as you suggest: they can't represent migrations.

Having a declared schema also makes the code more maintainable. It can be 
bewildering to work on code where the schema is written as a series of 
"alter table" statements. Any non-trivial project will have a dozen or two 
boilerplate tables (users, sessions, settings, etc.).  If they are all 
written as a series of "alter table" statements, there is a huge cognitive 
load just in figuring out what the tables are, and how they are related.

On Tuesday, October 29, 2013 4:09:33 PM UTC-7, Chris Kuttruff wrote:
>
> Well things were kept separate intentionally.  If someone wants to use 
> Korma or some other DSL within their migrations, they can augment their 
> migration file to use that to generate the SQL, but having the migrations 
> set up such that instructions to jdbc are simple clojure strings is very 
> intentional.  This way I don't limit anyone's decision about what other 
> libraries they use, but complicated migrations can easily be dynamically 
> generated (since they are being picked up within the context of a clojure 
> file).
>
> Not sure I fully understand your point, but this seems like a reasonable 
> case for modularity.
>
>
> On Tuesday, October 29, 2013 8:49:55 AM UTC-7, Brian Craft wrote:
>>
>>
>>
>> On Monday, October 28, 2013 4:36:56 PM UTC-7, Chris Kuttruff wrote:
>>>
>>> Separate from DSLs like Korma, etc.  I have written a simple library for 
>>> doing database migrations with clojure (clj-sql-up ( 
>>> https://github.com/ckuttruff/clj-sql-up )).  There are also other 
>>> libraries still maintained along these lines (drift, migratus, ragtime, 
>>> etc.)
>>>
>>>
>> It's unfortunate that these are separate, because you need the schema 
>> information not just for migrations, but also for query abstraction (sql 
>> dsl, etc.). The argument for small, composable libraries only works if they 
>> can actually be meaningfully composed: if, in this case, a declared schema 
>> can be used for migrations, and query abstraction, and administrative UI, 
>> and anything else that requires it. So far there's not much like this in 
>> clojure that I've found.
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: SQL to logic rule mapping

[Brian Craft]

On Wednesday, October 30, 2013 7:56:46 AM UTC-7, ArturoH wrote:
>
>  I think the shortcoming of ORM tools is that they bring a higher level 
> technology like SQL and lower it down to the low level of imperative 
> languages.
>

Can you give an example?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: SQL to logic rule mapping

[Brian Craft]

I did that recently, as well. Found this:

https://github.com/ghxiao/nrdatalog2sql

Haven't done more than glance at the code.

On Wednesday, October 30, 2013 4:29:33 PM UTC-7, Mark wrote:
>
>
>
> On Wednesday, October 30, 2013 4:27:31 PM UTC-7, Christopher Allen wrote:
>>
>> If you just want to build up and apply constraints, Korma can do that.
>>
>> If you want something closer to Datalog with unification, then a Datalog 
>> to SQL bridge is the most practical of largely impractical choices.
>>
>
> Are you aware of a Datalog -> SQL implementation?  I went looking for one 
> about a year ago and could only find a paper on how one might achieve such 
> a thing.  I was hoping for a more out-of-the-box solution. 
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: SQL to logic rule mapping

[Brian Craft]

On Wednesday, October 30, 2013 8:04:25 PM UTC-7, ArturoH wrote:
>
> On Wednesday, October 30, 2013 12:01:02 PM UTC-5, Brian Craft wrote
>
>> On Wednesday, October 30, 2013 7:56:46 AM UTC-7, ArturoH wrote:
>>>
>>>  I think the shortcoming of ORM tools is that they bring a higher level 
>>> technology like SQL and lower it down to the low level of imperative 
>>> languages.
>>>
>>
>> Can you give an example?
>>
>
> In SQL you can nest statements use views, outer joins. Some new features 
> include window functions. In a single SQL statement you can get a lot of 
> functionality that would take many more lines of code in an imperative 
> language. Plus the database has statistics on data distribution that is 
> used to order the file access. Once your data makes it to the imperative 
> language. The order of execution of for example, looping constructs, is 
> hard-coded. And I could go on. I have to say ORM writers have done wonders 
> with imperative languages. They are perfectly usable and in production in 
> many places. I just want something even better.  
>

My experience with ORMs is that they make these things more declarative, 
not more imperative. For example, sql requires a lot of low-level 
information in how to perform a query: which tables to join, which fields 
to join on, what order to join, whether to do a join or a subselect, etc., 
etc., etc. The ORMs instead present a more declarative API, like 
user.address.city, abstracting all the low-level details of following 
relations.

 

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: SQL to logic rule mapping

[Brian Craft]

On Wednesday, October 30, 2013 8:37:51 PM UTC-7, ArturoH wrote:
>
> One other thing I did not know is that there are Datalog to SQL bridges 
> that is good to know.
>
>
 
That python script is the only one I've found. I'd be curious to hear of 
any others.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: What non-deprecated Clojure Web libraries to use?

[Brian Craft]

Yes, exactly. Currently, korma's data modeling occurs via macros that 
create data structures which aren't exactly part of the public API. So, 
your options are to wrap all the macros in a way that exposes the data 
model, or to try to interpret the data structures that it creates, without 
any guarantee that they'll be the same in the next release.

On Wednesday, October 30, 2013 4:22:33 PM UTC-7, Manuel Paccagnella wrote:
>
> A probably simplistic consideration: maybe there should be a data model 
> expressed as a data structure so that it can be leveraged by arbitrary 
> libs. This way there would be a single representation, but no explicit 
> dependencies between single libs. Here probably Datomic could be an example.
>
> Il giorno mercoledì 30 ottobre 2013 01:12:25 UTC+1, Brian Craft ha scritto:
>>
>> In general, my point is that libraries don't compose if they have 
>> incompatible or hidden representations of the data structures over which 
>> they operate, which is the default condition if no one has thought about 
>> how the libraries might be used together. A consequence of this is that a 
>> framework is much, much greater than the sum of its parts.
>>
>> A dsl that can abstract away details of building queries (e.g. joins) 
>> requires a declared schema. In contrast, the clojure migration tools 
>> describe a schema as a series of functions, or sql string literals. It's 
>> hard to derive one from the other. You wouldn't want to start trying to 
>> parse the sql to deduce the schema, for instance. Consequently you end up 
>> repeating the schema. Then you add an administrative UI, and you have the 
>> same problem: the pages and forms for the admin depend on the schema. You 
>> end up repeating the schema a third time. And so forth. It quickly becomes 
>> unmanageable.
>>
>> For this case, migrations, it's easier to derive the sql for the 
>> migrations from a declared schema (doing a diff against the previous 
>> version) than the other way around (parsing sql to find the schema). This 
>> is how django-south works (it automatically generates the sql for the 
>> migrations, in most cases), but there's nothing like it for clojure that 
>> I'm aware of. Also, the sql dsls in clojure that I've seen cover very 
>> little of sql for data model creation, so you can't actually compose them 
>> with the migration tools as you suggest: they can't represent migrations.
>>
>> Having a declared schema also makes the code more maintainable. It can be 
>> bewildering to work on code where the schema is written as a series of 
>> "alter table" statements. Any non-trivial project will have a dozen or 
>> two boilerplate tables (users, sessions, settings, etc.).  If they are all 
>> written as a series of "alter table" statements, there is a huge 
>> cognitive load just in figuring out what the tables are, and how they are 
>> related.
>>
>> On Tuesday, October 29, 2013 4:09:33 PM UTC-7, Chris Kuttruff wrote:
>>>
>>> Well things were kept separate intentionally.  If someone wants to use 
>>> Korma or some other DSL within their migrations, they can augment their 
>>> migration file to use that to generate the SQL, but having the migrations 
>>> set up such that instructions to jdbc are simple clojure strings is very 
>>> intentional.  This way I don't limit anyone's decision about what other 
>>> libraries they use, but complicated migrations can easily be dynamically 
>>> generated (since they are being picked up within the context of a clojure 
>>> file).
>>>
>>> Not sure I fully understand your point, but this seems like a reasonable 
>>> case for modularity.
>>>
>>>
>>> On Tuesday, October 29, 2013 8:49:55 AM UTC-7, Brian Craft wrote:
>>>>
>>>>
>>>>
>>>> On Monday, October 28, 2013 4:36:56 PM UTC-7, Chris Kuttruff wrote:
>>>>>
>>>>> Separate from DSLs like Korma, etc.  I have written a simple library 
>>>>> for doing database migrations with clojure (clj-sql-up ( 
>>>>> https://github.com/ckuttruff/clj-sql-up )).  There are also other 
>>>>> libraries still maintained along these lines (drift, migratus, ragtime, 
>>>>> etc.)
>>>>>
>>>>>
>>>> It's unfortunate that these are separate, because you need the schema 
>>>> information not just for migrations, but also for query abstraction (sql 
>>>> dsl, etc.). The argument for small, composable libraries onl

Re: [ANN] Yesql 0.2.1 - Clojure & SQL queries rethought.

[Brian Craft]

On Monday, November 11, 2013 12:57:55 PM UTC-8, Luc wrote:
>
> Ha ! Ha ! Never thought of this approach and frankly it's the simplest 
> one. 
>
> We create various db adapters and we routinely have to switch between 
> raw SQL statements and express them to run from Clojure and it's been an 
> extra 
> step that yields zero value. 
>
> Not as painfull as using an ORM but still annoying. 
>
>
I'm curious about your ORM experience. I'm certainly finding that the 
clojure sql dsls are adding little, and I must constantly fall back to raw 
sql.  However my experience with ORMs has been exactly the opposite. 
They've cost us about as much developer time as our power supplies: they 
just work, and provide an incredible amount of functionality.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] Yesql 0.2.1 - Clojure & SQL queries rethought.

[Brian Craft]

What prevents you from formatting the example with the clojure string of 
sql like the example with the separate file of sql? Is the separate file 
just for the sake of syntax highlighting? 

On Monday, November 11, 2013 8:52:01 AM UTC-8, Kris Jenkins wrote:
>
> Hi Mamun,
>
> Hmm...I hadn't planned on it, but I could see it would suit some people's 
> development style. (It could get messy, but so can Clojure namespaces if 
> you're undisciplined. The developer should be allowed to take 
> responsibility.)
>
> The tricky parts would be:
>
>- How would you delimit the various queries? In a clear, portable way 
>that allowed for docstrings? 
>- How would you refer to the individual queries? Some kind of 
>mandatory '-- NAME: foo-query' syntax?
>
> Any thoughts?
>
> Kris
>
> On Monday, 11 November 2013 14:46:55 UTC, Mamun wrote:
>>
>> Hi Kris,
>>
>> It's look fine. Is is possible to to add more query in one sql file? I 
>> mean- I'd like to create one sql file and store all query on that file.
>>
>> BR,
>> Mamun
>>
>>
>>
>> On Monday, November 11, 2013 2:38:27 PM UTC+1, Kris Jenkins wrote:
>>>
>>> Thanks David, that's nice of you to say. I know exactly what you mean - 
>>> I didn't want to be the guy to write yet another Clojure/SQL library - but 
>>> when I figured out why none of the existing ones was working for me, I had 
>>> no choice. :-D
>>>
>>> There's still plenty that can be done with the implementation, but 
>>> hopefully the world will look kindly on the design, and it can evolve from 
>>> there...
>>>
>>> Kris
>>>
>>> On Monday, 11 November 2013 11:45:21 UTC, David Della Costa wrote:

 I was about to be like, "oh no, not another one!" and then I read the 
 README and I thought, "oh, interesting... 

 So, kudos on thinking outside the box.  I certainly agree with a lot of 
 the points you've made.  I'll definitely be playing around with this. 

 Cheers, 
 DD 

 (2013/11/11 20:10), Kris Jenkins wrote: 
 > https://github.com/krisajenkins/yesql 
 > 
 > Yesql is a simple library for blending SQL & Clojure together, 
 cleanly. 
 > Here's how it works , 

 > and how to use it <
 https://github.com/krisajenkins/yesql#example-usage>. 
 > 
 > Feedback welcomed, 
 > Kris 
 > 
 > -- 
 > -- 
 > You received this message because you are subscribed to the Google 
 > Groups "Clojure" group. 
 > To post to this group, send email to clo...@googlegroups.com 
 > Note that posts from new members are moderated - please be patient 
 with 
 > your first post. 
 > To unsubscribe from this group, send email to 
 > clojure+u...@googlegroups.com 
 > For more options, visit this group at 
 > http://groups.google.com/group/clojure?hl=en 
 > --- 
 > You received this message because you are subscribed to the Google 
 > Groups "Clojure" group. 
 > To unsubscribe from this group and stop receiving emails from it, 
 send 
 > an email to clojure+u...@googlegroups.com. 
 > For more options, visit https://groups.google.com/groups/opt_out. 

>>>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Releasing Caribou today: Open Source Clojure Web Ecosystem

[Brian Craft]

Looks very cool. I'm happy to see that data modeling is taken seriously, 
which in my experience is the biggest piece lacking in other clojure web 
tools.

The docs have a lot of layout problems with words running together, like 
so: "data from oneenvironment". Looks like a string joining operation 
that's not quite right.

On Tuesday, November 12, 2013 3:52:10 PM UTC-8, Ryan Spangler wrote:
>
> Hello Clojure,
>
> Excited to announce today the release of Caribou!  http://let-caribou.in/
>
> We have been building web sites and web applications with it for over two 
> years now and improving it every day.  Currently we have four people 
> working on it and another ten using it to build things, so it is getting a 
> lot of real world testing.
>
> It has been designed as a collection of independent libraries that could 
> each be useful on their own, but which come together as a meaningful whole.
>
> We have been spending the last couple months getting it ready for a full 
> open source release, and I am happy to say it is finally ready.  Funded and 
> supported by Instrument in Portland, OR:  http://weareinstrument.com/  We 
> have four projects using it in production, and several more about to be 
> launched (as well as over a dozen internal things).
>
> Documentation is here:  http://caribou.github.io/caribou/docs/outline.html
>
> Source is here:  http://github.com/caribou/caribou (use this for issues, 
> you don't actually need the source as it is installed through a lein 
> template).
>
> Some of the independently useful libraries Caribou is built on are:
>
> * Polaris -- Routing with data (not macros) and reverse routing! :  
> https://github.com/caribou/polaris
> * Lichen -- Image resizing to and from s3 or on disk: 
> https://github.com/caribou/lichen
> * Schmetterling -- Debugging Clojure processes from the browser:  
> https://github.com/prismofeverything/schmetterling
> * Antlers -- Useful extensions to mustache templating (helpers and blocks, 
> among other things):  https://github.com/caribou/antlers
> * Groundhog -- Replay http requests: 
> https://github.com/noisesmith/groundhog
>
> And many others.
>
> Basically this is an Alpha release, and I am announcing it here first in 
> order to get as much feedback from the community as possible.  We have made 
> it as useful as we can for our purposes and recognize that for it to 
> improve from here, we really need as many people using it and building 
> things with it as possible.  The documentation also needs to be put through 
> its paces:  we need to see how well people are able to use it who know 
> nothing about it, based only on the existing docs.
>
> All feedback welcome!  
>
> Thanks for reading!  I hope you find it useful.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

On Wednesday, November 13, 2013 3:23:52 PM UTC-8, Justin Smith wrote:
>
>
> Clojure provides execution efficiency that Ruby or Python cannot match. 
> This translates to lowered hosting costs.
>
>>
>>
It should be noted that the go-to solution for performance in python is to 
drop to native code for bottlenecks, which is pretty easy, and which still 
out-performs the jvm. Whether clojure runs faster than python in a 
particular application is not a given.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

I also work in a python/django shop, and have been experimenting with 
clojure for about nine months. Before yesterday I would have told you that 
clojure web tooling does not come remotely close to the power of django. 
With a large amount of effort in piecing different libraries together, you 
might cover 30 or 40% of what django provides out of the box. I will have 
to reevaluate that with the announcement of Caribou, which looks good at 
first glance.

As a language, clojure is much more enjoyable. Python may seem clumsy and 
ineffective after writing clojure for awhile.


On Wednesday, November 13, 2013 2:38:49 PM UTC-8, Marcus Blankenship wrote:
>
> Hi Folks,
>
> We’re a Python / Django shop, and some folks are getting excited about 
> using Clojure for building web apps.  Certainly there are numerous 
> open-source options to assist us (Pedastal, Ring, Compojure, Caribou, etc), 
> but I think it begs a larger question: *as a rule, do web applications 
> need the power that Clojure brings to the table?*
>
> Other folks on my team are telling me that solutions built with Python / 
> Django (or even RubyOnRails) fully satisfy the needs of 99% of the web apps 
> we have built, and that Clojure offers nothing new to this problem space.  
>
> So, here’s the question: *How are you are actually using Clojure, and why 
> did you choose to use it, particularly in the “web application” space? * 
>
> Thanks,
> Marcus
>
>
>
> marcus blankenship
> \\\ Partner, Problem Solver, Linear Thinker
> \\\ 541.805.2736 \ @justzeros \ skype:marcuscreo
>  
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

On Thursday, November 14, 2013 1:01:33 AM UTC-8, Ryan Spangler wrote:
>
> I've also used Python extensively (and we still have some projects in it), 
> but there is a lot of complexity in just dealing with the environments and 
> dependencies (virtualenv is a standard practice now?)  Once people have 
> lein installed, they can download one of our Clojure projects and we know 
> everyone is using the same version of everything without any hassle.  It 
> just works.  
>

I find virtualenv/pip much easier to use than lein. The search feature in 
lein is unusable, and having to edit a file to pull in dependencies is 
awkward. I'd love to have something as easy as pip for clojure.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

On Thursday, November 14, 2013 2:28:51 AM UTC-8, Sean Johnson wrote:
>
> Framework vs. ecosystem of interoperable libraries. The monolithic 
> framework (see Rails and Django) is not the way web development is done in 
> Clojure. Instead imagine the Clojure world as a set of legos. You get to 
> build whatever you want with your legos, selecting just the blocks you 
> need, in the colors and sizes you want, and they all snap together [1]. The 
> benefit to this approach is simplicity. There is a LOT to know about the 
> Rails and Django frameworks, as they are big, mature things that have grown 
> to solve all the common web development needs, and you start your 
> development with ALL of that framework. Your starting point with Clojure on 
> the other hand is much simpler. It will seem too simple... at first you'll 
> be wondering... where is all the "stuff"? You start with maybe just ring 
> and compojure (web middleware and routing) and a few lines of your code and 
> your first couple of stories are already complete. You add libraries as you 
> find you need them as you build out your app over time, but at all times 
> your app is as simple as it can be, and only has the "stuff" it needs. It's 
> therefore much easier to understand and less a "big ball of mud".
>

I don't believe the legos analogy is very accurate for clojure. Or, rather, 
it's more of a vision than a reality. I'm unaware of any libraries in 
clojure that you can piece together to give you the features of 
django-south, django admin, and the forms/validation/db layers, for 
example.  Today, clojure web libraries are more like an auto parts store: 
your chance of putting together a complete car from the inventory is slim 
indeed, and your chance of doing it in a timely fashion is exactly zero. 
 There are about a dozen migration libraries for clojure, for example, but 
none of them integrate very well with the other libraries because there 
isn't much in the way of a data modeling framework for the different 
components to leverage. Also, I don't think any of them provide the 
simplicity of south, where your data model is declared in one place (rather 
than in a sequence of sql commands that you have to interpret to deduce the 
final data model), and migrations are derived from changes in the data 
model.

The putting of things together is the hard part of software engineering. This 
makes a huge difference in development time.  Clean algorithms are easy in 
comparison. Rich talks about the importance of taking things apart, but you 
can't take things apart that never fit together in the first place. IIRC he 
actually suggested that one take things apart such that they fit together 
again. I agree with this, but I don't think it describes much of clojure 
web tooling today.

Also, the overwhelming majority of db and web work is boilerplate. That's 
what django provides: all the mindless boilerplate that is completely 
uninteresting to your problem domain, but necessary to launch a web site. 
With the current clojure web tools, there is no solution for most of that 
boilerplate, which can easily add an order of magnitude to your development 
time.

And again, I will have to reevaluate all of this in light of caribou, which 
appears to address much of it. Also, I'd be very happy to hear that I'm 
wrong, if anyone has example projects to show, or libraries I've missed.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

On Thursday, November 14, 2013 9:17:32 AM UTC-8, James Reeves wrote:
>
> On 14 November 2013 16:22, Brian Craft  >wrote:
>
>>
>> I don't believe the legos analogy is very accurate for clojure. Or, 
>> rather, it's more of a vision than a reality. I'm unaware of any libraries 
>> in clojure that you can piece together to give you the features of 
>> django-south, django admin, and the forms/validation/db layers, for 
>> example.  Today, clojure web libraries are more like an auto parts store: 
>> your chance of putting together a complete car from the inventory is slim 
>> indeed, and your chance of doing it in a timely fashion is exactly zero.
>>
>
> Except people *are* developing web applications in Clojure, so clearly 
> this isn't an accurate statement.
>

I haven't seen a web application in clojure with the functionality of a 
django app, with the exception of polyglot applications that are using 
non-clojure frameworks to fill the gaps. I would very much like to see one.


>
> Also, the overwhelming majority of db and web work is boilerplate. That's 
>> what django provides: all the mindless boilerplate that is completely 
>> uninteresting to your problem domain, but necessary to launch a web site.
>
>  
> This certainly hasn't been my experience.
>
> Can you be certain this isn't a symptom of the tool you're using? 
> Frameworks like Django and Rails are designed to build web applications in 
> a very specific way, and this involves generating a lot of database and 
> HTML boilerplate.
>
> I can't say for sure whether it's because of Clojure, or because I'm 
> working on different problems these days, but I don't tend to deal with the 
> same issues I did when I used Rails. Nowadays I find myself building 
> systems out of small, isolated components, which seem to eliminate a lot of 
> issues Rails was designed to work around.
>
> For instance, database migrations. This was something I thought about a 
> lot a few years ago, until I gradually realised that I wasn't finding 
> myself in any situations where I needed them.
>
>

I don't really understand this argument. What database tables would we not 
require if we used clojure? User passwords? Page content? User settings? 
Admin privileges? Of those tables, which ones would we not need admin 
interfaces for if we used clojure? I don't understand how choice of 
framework has any bearing on what data the app must store, what admin 
interfaces are required, what validation layers must be present, etc. You 
must validate all the user input. You must store all the data required by 
the app. You must have admin backends for all of the data used by the app. 
Using clojure doesn't change any of those requirements.

And when we have changes to the data model, for example when finding that a 
new feature requires that we track an additional field for each user, or 
finding that a new clinical data set requires a more complex model of 
clinical samples, how would clojure allow us to avoid migrating the data 
model in the database, or update the model in a reliable way across several 
installs without using migrations? Again, I don't understand how the 
framework has anything to do with the necessity of updating a data model 
when new features or new data demand it.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

On Thursday, November 14, 2013 9:42:52 AM UTC-8, Jonathan Irving wrote:
>
> I agree with much of what you write James - I'm paid to write rails and 
> node.js code, and I'm finding that node is encouraging me to compose 
> small components and basically sidestep a lot of the issues that rails 
> is designed to address. 
>

Can you give a concrete example? 

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Do web apps need Clojure?

[Brian Craft]

On Thursday, November 14, 2013 10:50:48 AM UTC-8, Jonathan Irving wrote:
>
> * Brian Craft > [2013-11-14 10:24 
> -0800]: 
> > On Thursday, November 14, 2013 9:42:52 AM UTC-8, Jonathan Irving wrote: 
> > > 
> > > I agree with much of what you write James - I'm paid to write rails 
> and 
> > > node.js code, and I'm finding that node is encouraging me to compose 
> > > small components and basically sidestep a lot of the issues that rails 
> > > is designed to address. 
> > > 
> > 
> > Can you give a concrete example? 
>
> Tricky, because my work doesn't belong to me. 
>
> I have a component which collects weather forecasts from a public API 
> for all of the US. This is isolated from other parts of the app. It 
> acquires the data in one form, transcodes and strips it down, and puts 
> it into a mongodb. 
>
> Another component runs async map reduce ops on the data, creates 
> aggregated and otherwise reduced data. 
>
> Finally, there is a service endpoint which serves the product to another 
> component, which (irrelevantly) is a rails app. 
>
> Managing the data model for these components is trivial, because they 
> only touch one kind of data, and they are each very small. What the 
> rails app sees is a service endpoint with a well defined API. I use no 
> ODM or ORM, just mongodb queries. 
>

Doesn't this reflect more on your problem domain than on your choice of 
tools? The first two components are roughly ETL, and the last is data API? 
How would this have been different in rails? I've done this in django, and 
it's the same as you describe here.

But more to the point, the pieces you've mentioned can't be composed to 
build a web app with a UI, and you can't decompose a web app with a UI into 
just the service layer bits. You also need the UI bits: the forms layer, 
the validation layer, the admin, etc., etc. To be maintainable, those 
layers should be common for different components in the application: it 
does not scale to require devs to learn multiple template languages, 
multiple migration tools, multiple admin interfaces, etc.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.