Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
I've faced this issue before, and I'd hazard a guess that like me, it could be related to the IOPS settings. AWS' IO performance is highly variable, and you get good performance in bursts, in very low volumes, and generally only initially for long runs. Which kind of makes sense, unless you reserve. durable-queue has a good amount of round-tripping for each 'task' (depending on how often it syncs too). Combine that with your general applications logging (debug mode? info mode?) and it simply adds up overwhelming the entire IO system. Our volumes were large (both data-, and number of instances-wise). My workaround was to create a RAM-based filesystem and hold the durable-queue on it. Which is okay for our use-case because if the machine were to be killed, we didn't care about the "durability" across instance reboots as such. Our perf issues as far as durable-queues go vanished! As an aside, reducing the log-levels too affected app performance on AWS, for the default config instances. Memory consumption, though an indicator of problems, isn't necessarily a direct indicator of things. The IO-backlog's backpressure on the entire JVM runtime is also a possible reason. At least in my case, it went away once the durable-queue was moved to the RAM-FS. Good luck! -- jaju On Fri, Oct 13, 2017 at 10:05 AM, <lawrence.krub...@gmail.com> wrote: > Following Daniel Compton's suggestion, I turned on logging for GC. I don't > see it happening more often, but the slow down does seem related to the > moment when the app hits the maximum memory allowed. It had been running > with 4G, so I increased that to 7G, so it goes longer now before it hits > 98% memory usage, but it does hit it eventually and then everything crawls > to a very slow speed. Not sure how much memory I would have to use to avoid > using up almost all of the memory. I suppose I'll figure that out via trial > and error. Until I can figure that out, nearly all other performance tricks > seems a bit besides the point. > > > > On Thursday, October 12, 2017 at 9:01:23 PM UTC-4, Nathan Fisher wrote: >> >> Hi! >> >> Can you change one of the variables? Specifically can you replicate this >> on your local machine? If it happens locally then I would focus on >> something in the JVM eco-system. >> >> If you can't replicate it locally then it's possibly AWS specific. It >> sounds like you're using a t2.large or m4.xlarge. If it's the prior you may >> very well be contending between with your network bandwidth. EC2's host >> drive (EBS) is a networked drive which is split between your standard >> network traffic and the drive volume. If that's the issue then you might >> need to look at provisioned IOPs. A quick(ish) way to test that hypothesis >> is to provision a host with high networking performance and provisioned >> IOPs. >> >> Cheers, >> Nathan >> >> On Fri, 13 Oct 2017 at 00:05 <lawrence...@gmail.com> wrote: >> >>> Daniel Compton, good suggestion. I've increased the memory to see if I >>> can postpone the GCs, and I'll log that more carefully. >>> >>> >>> On Wednesday, October 11, 2017 at 8:35:44 PM UTC-4, Daniel Compton wrote: >>> >>>> Without more information it's hard to tell, but this looks a like it >>>> could be a garbage collection issue. Can you run your test again and add >>>> some logging/monitoring to show each garbage collection? If my hunch is >>>> right, you'll see garbage collections getting more and more frequent until >>>> they take up nearly all the CPU time, preventing much forward progress >>>> writing to the queue. >>>> >>>> If it's AWS based throttling, then CloudWatch monitoring >>>> http://docs.aws.amazon.com/AWSEC2/latest/UserGuid >>>> e/monitoring-volume-status.html#using_cloudwatch_ebs might show you >>>> some hints. You could also test with an NVMe drive attached, just to see if >>>> disk bandwidth is the issue. >>>> >>>> On Thu, Oct 12, 2017 at 11:58 AM Justin Smith <noise...@gmail.com> >>>> wrote: >>>> >>> a small thing here, if memory usage is important you should be building >>>>> and running an uberjar instead of using lein on the server (this also has >>>>> other benefits), and if you are doing that your project.clj jvm-opts are >>>>> not used, you have to configure your java command line in aws instead >>>>> >>>>> On Wed, Oct 11, 2017 at 3:52 PM <lawrence...@gmail.com> wrote: >>>>> >>>> I can't figure out if this is a Clojure question or an AWS question. >>>&g
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
It sounds like you have a memory leak. I would look at addressing that before any performance tricks. On Fri, 13 Oct 2017 at 05:35, <lawrence.krub...@gmail.com> wrote: > Following Daniel Compton's suggestion, I turned on logging for GC. I don't > see it happening more often, but the slow down does seem related to the > moment when the app hits the maximum memory allowed. It had been running > with 4G, so I increased that to 7G, so it goes longer now before it hits > 98% memory usage, but it does hit it eventually and then everything crawls > to a very slow speed. Not sure how much memory I would have to use to avoid > using up almost all of the memory. I suppose I'll figure that out via trial > and error. Until I can figure that out, nearly all other performance tricks > seems a bit besides the point. > > > > On Thursday, October 12, 2017 at 9:01:23 PM UTC-4, Nathan Fisher wrote: > >> Hi! >> >> Can you change one of the variables? Specifically can you replicate this >> on your local machine? If it happens locally then I would focus on >> something in the JVM eco-system. >> >> If you can't replicate it locally then it's possibly AWS specific. It >> sounds like you're using a t2.large or m4.xlarge. If it's the prior you may >> very well be contending between with your network bandwidth. EC2's host >> drive (EBS) is a networked drive which is split between your standard >> network traffic and the drive volume. If that's the issue then you might >> need to look at provisioned IOPs. A quick(ish) way to test that hypothesis >> is to provision a host with high networking performance and provisioned >> IOPs. >> >> Cheers, >> Nathan >> >> On Fri, 13 Oct 2017 at 00:05 <lawrence...@gmail.com> wrote: >> >>> Daniel Compton, good suggestion. I've increased the memory to see if I >>> can postpone the GCs, and I'll log that more carefully. >>> >>> >>> On Wednesday, October 11, 2017 at 8:35:44 PM UTC-4, Daniel Compton wrote: >>> >>>> Without more information it's hard to tell, but this looks a like it >>>> could be a garbage collection issue. Can you run your test again and add >>>> some logging/monitoring to show each garbage collection? If my hunch is >>>> right, you'll see garbage collections getting more and more frequent until >>>> they take up nearly all the CPU time, preventing much forward progress >>>> writing to the queue. >>>> >>>> If it's AWS based throttling, then CloudWatch monitoring >>>> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html#using_cloudwatch_ebs >>>> might >>>> show you some hints. You could also test with an NVMe drive attached, just >>>> to see if disk bandwidth is the issue. >>>> >>>> On Thu, Oct 12, 2017 at 11:58 AM Justin Smith <noise...@gmail.com> >>>> wrote: >>>> >>> a small thing here, if memory usage is important you should be building >>>>> and running an uberjar instead of using lein on the server (this also has >>>>> other benefits), and if you are doing that your project.clj jvm-opts are >>>>> not used, you have to configure your java command line in aws instead >>>>> >>>>> On Wed, Oct 11, 2017 at 3:52 PM <lawrence...@gmail.com> wrote: >>>>> >>>> I can't figure out if this is a Clojure question or an AWS question. >>>>>> And if it is a Clojure question, I can't figure out if it is more of a >>>>>> general JVM question, or if it is specific to some library such as >>>>>> durable-queue. I can redirect my question elsewhere, if people think this >>>>>> is an AWS question. >>>>>> >>>>>> In my project.clj, I try to give my app a lot of memory: >>>>>> >>>>>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"]) >>>>>> >>>>>> And the app starts off pulling data from MySQL and writing it to >>>>>> Durable-Queue at a rapid rate. ( >>>>>> https://github.com/Factual/durable-queue ) >>>>>> >>>>>> I have some logging set up to report every 30 seconds. >>>>>> >>>>>> :enqueued 370137, >>>>>> >>>>>> 30 seconds later: >>>>>> >>>>>> :enqueued 608967, >>>>>> >>>>>> 30 seconds later: >>>>>>
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
Following Daniel Compton's suggestion, I turned on logging for GC. I don't see it happening more often, but the slow down does seem related to the moment when the app hits the maximum memory allowed. It had been running with 4G, so I increased that to 7G, so it goes longer now before it hits 98% memory usage, but it does hit it eventually and then everything crawls to a very slow speed. Not sure how much memory I would have to use to avoid using up almost all of the memory. I suppose I'll figure that out via trial and error. Until I can figure that out, nearly all other performance tricks seems a bit besides the point. On Thursday, October 12, 2017 at 9:01:23 PM UTC-4, Nathan Fisher wrote: > > Hi! > > Can you change one of the variables? Specifically can you replicate this > on your local machine? If it happens locally then I would focus on > something in the JVM eco-system. > > If you can't replicate it locally then it's possibly AWS specific. It > sounds like you're using a t2.large or m4.xlarge. If it's the prior you may > very well be contending between with your network bandwidth. EC2's host > drive (EBS) is a networked drive which is split between your standard > network traffic and the drive volume. If that's the issue then you might > need to look at provisioned IOPs. A quick(ish) way to test that hypothesis > is to provision a host with high networking performance and provisioned > IOPs. > > Cheers, > Nathan > > On Fri, 13 Oct 2017 at 00:05 <lawrence...@gmail.com > wrote: > >> Daniel Compton, good suggestion. I've increased the memory to see if I >> can postpone the GCs, and I'll log that more carefully. >> >> >> On Wednesday, October 11, 2017 at 8:35:44 PM UTC-4, Daniel Compton wrote: >> >>> Without more information it's hard to tell, but this looks a like it >>> could be a garbage collection issue. Can you run your test again and add >>> some logging/monitoring to show each garbage collection? If my hunch is >>> right, you'll see garbage collections getting more and more frequent until >>> they take up nearly all the CPU time, preventing much forward progress >>> writing to the queue. >>> >>> If it's AWS based throttling, then CloudWatch monitoring >>> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html#using_cloudwatch_ebs >>> might >>> show you some hints. You could also test with an NVMe drive attached, just >>> to see if disk bandwidth is the issue. >>> >>> On Thu, Oct 12, 2017 at 11:58 AM Justin Smith <noise...@gmail.com> >>> wrote: >>> >> a small thing here, if memory usage is important you should be building >>>> and running an uberjar instead of using lein on the server (this also has >>>> other benefits), and if you are doing that your project.clj jvm-opts are >>>> not used, you have to configure your java command line in aws instead >>>> >>>> On Wed, Oct 11, 2017 at 3:52 PM <lawrence...@gmail.com> wrote: >>>> >>> I can't figure out if this is a Clojure question or an AWS question. And >>>>> if it is a Clojure question, I can't figure out if it is more of a >>>>> general >>>>> JVM question, or if it is specific to some library such as durable-queue. >>>>> I >>>>> can redirect my question elsewhere, if people think this is an AWS >>>>> question. >>>>> >>>>> In my project.clj, I try to give my app a lot of memory: >>>>> >>>>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"]) >>>>> >>>>> And the app starts off pulling data from MySQL and writing it to >>>>> Durable-Queue at a rapid rate. ( >>>>> https://github.com/Factual/durable-queue ) >>>>> >>>>> I have some logging set up to report every 30 seconds. >>>>> >>>>> :enqueued 370137, >>>>> >>>>> 30 seconds later: >>>>> >>>>> :enqueued 608967, >>>>> >>>>> 30 seconds later: >>>>> >>>>> :enqueued 828950, >>>>> >>>>> It's a dramatic slowdown. The app is initially writing to the queue at >>>>> faster than 10,000 documents a second, but it slows steadily, and after >>>>> 10 >>>>> minutes it writes less than 1,000 documents per second. Since I have to >>>>> write a few million documents, 10,000 a second is the slo
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
Hi! Can you change one of the variables? Specifically can you replicate this on your local machine? If it happens locally then I would focus on something in the JVM eco-system. If you can't replicate it locally then it's possibly AWS specific. It sounds like you're using a t2.large or m4.xlarge. If it's the prior you may very well be contending between with your network bandwidth. EC2's host drive (EBS) is a networked drive which is split between your standard network traffic and the drive volume. If that's the issue then you might need to look at provisioned IOPs. A quick(ish) way to test that hypothesis is to provision a host with high networking performance and provisioned IOPs. Cheers, Nathan On Fri, 13 Oct 2017 at 00:05 <lawrence.krub...@gmail.com> wrote: > Daniel Compton, good suggestion. I've increased the memory to see if I can > postpone the GCs, and I'll log that more carefully. > > > On Wednesday, October 11, 2017 at 8:35:44 PM UTC-4, Daniel Compton wrote: > >> Without more information it's hard to tell, but this looks a like it >> could be a garbage collection issue. Can you run your test again and add >> some logging/monitoring to show each garbage collection? If my hunch is >> right, you'll see garbage collections getting more and more frequent until >> they take up nearly all the CPU time, preventing much forward progress >> writing to the queue. >> >> If it's AWS based throttling, then CloudWatch monitoring >> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html#using_cloudwatch_ebs >> might >> show you some hints. You could also test with an NVMe drive attached, just >> to see if disk bandwidth is the issue. >> >> On Thu, Oct 12, 2017 at 11:58 AM Justin Smith <noise...@gmail.com> wrote: >> > a small thing here, if memory usage is important you should be building >>> and running an uberjar instead of using lein on the server (this also has >>> other benefits), and if you are doing that your project.clj jvm-opts are >>> not used, you have to configure your java command line in aws instead >>> >>> On Wed, Oct 11, 2017 at 3:52 PM <lawrence...@gmail.com> wrote: >>> >> I can't figure out if this is a Clojure question or an AWS question. And >>>> if it is a Clojure question, I can't figure out if it is more of a general >>>> JVM question, or if it is specific to some library such as durable-queue. I >>>> can redirect my question elsewhere, if people think this is an AWS >>>> question. >>>> >>>> In my project.clj, I try to give my app a lot of memory: >>>> >>>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"]) >>>> >>>> And the app starts off pulling data from MySQL and writing it to >>>> Durable-Queue at a rapid rate. ( >>>> https://github.com/Factual/durable-queue ) >>>> >>>> I have some logging set up to report every 30 seconds. >>>> >>>> :enqueued 370137, >>>> >>>> 30 seconds later: >>>> >>>> :enqueued 608967, >>>> >>>> 30 seconds later: >>>> >>>> :enqueued 828950, >>>> >>>> It's a dramatic slowdown. The app is initially writing to the queue at >>>> faster than 10,000 documents a second, but it slows steadily, and after 10 >>>> minutes it writes less than 1,000 documents per second. Since I have to >>>> write a few million documents, 10,000 a second is the slowest speed I can >>>> live with. >>>> >>>> The queues are in the /tmp folder of an AWS instance that has plenty of >>>> disk space, 4 CPUs, and 16 gigs of RAM. >>>> >>>> Why does the app slow down so much? I had 4 thoughts: >>>> >>>> 1.) the app struggles as it hits a memory limit >>>> >>>> 2.) memory bandwidth is the problem >>>> >>>> 3.) AWS is enforcing some weird IOPS limit >>>> >>>> 4.) durable-queue is misbehaving >>>> >>>> As to possibility #1, I notice the app starts like this: >>>> >>>> Memory in use (percentage/used/max-heap): (\"66%\" \"2373M\" \"3568M\") >>>> >>>> but 60 seconds later I see: >>>> >>>> Memory in use (percentage/used/max-heap): (\"94%\" \"3613M\" \"3819M\") >>>> >>>> So I've run out of allowed memory. But why is that? I thought I gave >>>> this app 7 gigs: >
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
Daniel Compton, good suggestion. I've increased the memory to see if I can postpone the GCs, and I'll log that more carefully. On Wednesday, October 11, 2017 at 8:35:44 PM UTC-4, Daniel Compton wrote: > > Without more information it's hard to tell, but this looks a like it could > be a garbage collection issue. Can you run your test again and add some > logging/monitoring to show each garbage collection? If my hunch is right, > you'll see garbage collections getting more and more frequent until they > take up nearly all the CPU time, preventing much forward progress writing > to the queue. > > If it's AWS based throttling, then CloudWatch monitoring > http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html#using_cloudwatch_ebs > might > show you some hints. You could also test with an NVMe drive attached, just > to see if disk bandwidth is the issue. > > On Thu, Oct 12, 2017 at 11:58 AM Justin Smith <noise...@gmail.com > > wrote: > >> a small thing here, if memory usage is important you should be building >> and running an uberjar instead of using lein on the server (this also has >> other benefits), and if you are doing that your project.clj jvm-opts are >> not used, you have to configure your java command line in aws instead >> >> On Wed, Oct 11, 2017 at 3:52 PM <lawrence...@gmail.com > >> wrote: >> >>> I can't figure out if this is a Clojure question or an AWS question. And >>> if it is a Clojure question, I can't figure out if it is more of a general >>> JVM question, or if it is specific to some library such as durable-queue. I >>> can redirect my question elsewhere, if people think this is an AWS >>> question. >>> >>> In my project.clj, I try to give my app a lot of memory: >>> >>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"]) >>> >>> And the app starts off pulling data from MySQL and writing it to >>> Durable-Queue at a rapid rate. ( >>> https://github.com/Factual/durable-queue ) >>> >>> I have some logging set up to report every 30 seconds. >>> >>> :enqueued 370137, >>> >>> 30 seconds later: >>> >>> :enqueued 608967, >>> >>> 30 seconds later: >>> >>> :enqueued 828950, >>> >>> It's a dramatic slowdown. The app is initially writing to the queue at >>> faster than 10,000 documents a second, but it slows steadily, and after 10 >>> minutes it writes less than 1,000 documents per second. Since I have to >>> write a few million documents, 10,000 a second is the slowest speed I can >>> live with. >>> >>> The queues are in the /tmp folder of an AWS instance that has plenty of >>> disk space, 4 CPUs, and 16 gigs of RAM. >>> >>> Why does the app slow down so much? I had 4 thoughts: >>> >>> 1.) the app struggles as it hits a memory limit >>> >>> 2.) memory bandwidth is the problem >>> >>> 3.) AWS is enforcing some weird IOPS limit >>> >>> 4.) durable-queue is misbehaving >>> >>> As to possibility #1, I notice the app starts like this: >>> >>> Memory in use (percentage/used/max-heap): (\"66%\" \"2373M\" \"3568M\") >>> >>> but 60 seconds later I see: >>> >>> Memory in use (percentage/used/max-heap): (\"94%\" \"3613M\" \"3819M\") >>> >>> So I've run out of allowed memory. But why is that? I thought I gave >>> this app 7 gigs: >>> >>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"]) >>> >>> As to possibility #2, I found this old post on the Clojure mailist: >>> >>> Andy Fingerhut wrote, "one thing I've found in the past on a 2-core >>> machine that was achieving much less than 2x speedup was memory bandwidth >>> being the limiting factor." >>> >>> >>> https://groups.google.com/forum/#!searchin/clojure/xmx$20xms$20maximum%7Csort:relevance/clojure/48W2eff3caU/HS6u547gtrAJ >>> >>> But I am not sure how to test this. >>> >>> As to possibility #3, I'm not sure how AWS enforces its IOPS limits. If >>> people think this is the most likely possibility, then I will repost my >>> question in an AWS forum. >>> >>> As to possibility #4, durable-queue is well-tested and used in a lot of >>> projects, and Zach Tellman is smart and makes few
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
Justin Smith, thanks, I've created an Uberjar that I now run under
Supervisord on an EC2 instance, and I set the JVM options via the command
that Supervisord calls.
On Wednesday, October 11, 2017 at 6:58:52 PM UTC-4, Justin Smith wrote:
>
> a small thing here, if memory usage is important you should be building
> and running an uberjar instead of using lein on the server (this also has
> other benefits), and if you are doing that your project.clj jvm-opts are
> not used, you have to configure your java command line in aws instead
>
> On Wed, Oct 11, 2017 at 3:52 PM <lawrence...@gmail.com >
> wrote:
>
>> I can't figure out if this is a Clojure question or an AWS question. And
>> if it is a Clojure question, I can't figure out if it is more of a general
>> JVM question, or if it is specific to some library such as durable-queue. I
>> can redirect my question elsewhere, if people think this is an AWS
>> question.
>>
>> In my project.clj, I try to give my app a lot of memory:
>>
>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
>>
>> And the app starts off pulling data from MySQL and writing it to
>> Durable-Queue at a rapid rate. ( https://github.com/Factual/durable-queue
>> )
>>
>> I have some logging set up to report every 30 seconds.
>>
>> :enqueued 370137,
>>
>> 30 seconds later:
>>
>> :enqueued 608967,
>>
>> 30 seconds later:
>>
>> :enqueued 828950,
>>
>> It's a dramatic slowdown. The app is initially writing to the queue at
>> faster than 10,000 documents a second, but it slows steadily, and after 10
>> minutes it writes less than 1,000 documents per second. Since I have to
>> write a few million documents, 10,000 a second is the slowest speed I can
>> live with.
>>
>> The queues are in the /tmp folder of an AWS instance that has plenty of
>> disk space, 4 CPUs, and 16 gigs of RAM.
>>
>> Why does the app slow down so much? I had 4 thoughts:
>>
>> 1.) the app struggles as it hits a memory limit
>>
>> 2.) memory bandwidth is the problem
>>
>> 3.) AWS is enforcing some weird IOPS limit
>>
>> 4.) durable-queue is misbehaving
>>
>> As to possibility #1, I notice the app starts like this:
>>
>> Memory in use (percentage/used/max-heap): (\"66%\" \"2373M\" \"3568M\")
>>
>> but 60 seconds later I see:
>>
>> Memory in use (percentage/used/max-heap): (\"94%\" \"3613M\" \"3819M\")
>>
>> So I've run out of allowed memory. But why is that? I thought I gave this
>> app 7 gigs:
>>
>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
>>
>> As to possibility #2, I found this old post on the Clojure mailist:
>>
>> Andy Fingerhut wrote, "one thing I've found in the past on a 2-core
>> machine that was achieving much less than 2x speedup was memory bandwidth
>> being the limiting factor."
>>
>>
>> https://groups.google.com/forum/#!searchin/clojure/xmx$20xms$20maximum%7Csort:relevance/clojure/48W2eff3caU/HS6u547gtrAJ
>>
>> But I am not sure how to test this.
>>
>> As to possibility #3, I'm not sure how AWS enforces its IOPS limits. If
>> people think this is the most likely possibility, then I will repost my
>> question in an AWS forum.
>>
>> As to possibility #4, durable-queue is well-tested and used in a lot of
>> projects, and Zach Tellman is smart and makes few mistakes, so I'm doubtful
>> that it is to blame, but I do notice that it starts off with 4 active slabs
>> and then after 120 seconds, it is only using 1 slab. Is that expected? If
>> people think this is the possible problem then I'll ask somewhere specific
>> to durable-queue
>>
>> Overall, my log information looks like this:
>>
>> ("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs
>> 3, :num-active-slabs 2, :enqueued 370137, :retried 0, :completed 369934,
>> :in-progress 10}})
>>
>> ("\nResource usage: " "Memory in use (percentage/used/max-heap):
>> (\"66%\" \"2373M\" \"3568M\")\n\nCPU usage (how-many-cpu's/load-average):
>> [4 5.05]\n\nFree memory in jvm: [1171310752]")
>>
>> 30 seconds later
>>
>> ("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs
>> 4, :num-active-slabs 4, :enqueued 608967, :retried 0, :completed 608511,
>> :in-
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
Without more information it's hard to tell, but this looks a like it could
be a garbage collection issue. Can you run your test again and add some
logging/monitoring to show each garbage collection? If my hunch is right,
you'll see garbage collections getting more and more frequent until they
take up nearly all the CPU time, preventing much forward progress writing
to the queue.
If it's AWS based throttling, then CloudWatch monitoring
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html#using_cloudwatch_ebs
might
show you some hints. You could also test with an NVMe drive attached, just
to see if disk bandwidth is the issue.
On Thu, Oct 12, 2017 at 11:58 AM Justin Smith <noisesm...@gmail.com> wrote:
> a small thing here, if memory usage is important you should be building
> and running an uberjar instead of using lein on the server (this also has
> other benefits), and if you are doing that your project.clj jvm-opts are
> not used, you have to configure your java command line in aws instead
>
> On Wed, Oct 11, 2017 at 3:52 PM <lawrence.krub...@gmail.com> wrote:
>
>> I can't figure out if this is a Clojure question or an AWS question. And
>> if it is a Clojure question, I can't figure out if it is more of a general
>> JVM question, or if it is specific to some library such as durable-queue. I
>> can redirect my question elsewhere, if people think this is an AWS
>> question.
>>
>> In my project.clj, I try to give my app a lot of memory:
>>
>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
>>
>> And the app starts off pulling data from MySQL and writing it to
>> Durable-Queue at a rapid rate. ( https://github.com/Factual/durable-queue
>> )
>>
>> I have some logging set up to report every 30 seconds.
>>
>> :enqueued 370137,
>>
>> 30 seconds later:
>>
>> :enqueued 608967,
>>
>> 30 seconds later:
>>
>> :enqueued 828950,
>>
>> It's a dramatic slowdown. The app is initially writing to the queue at
>> faster than 10,000 documents a second, but it slows steadily, and after 10
>> minutes it writes less than 1,000 documents per second. Since I have to
>> write a few million documents, 10,000 a second is the slowest speed I can
>> live with.
>>
>> The queues are in the /tmp folder of an AWS instance that has plenty of
>> disk space, 4 CPUs, and 16 gigs of RAM.
>>
>> Why does the app slow down so much? I had 4 thoughts:
>>
>> 1.) the app struggles as it hits a memory limit
>>
>> 2.) memory bandwidth is the problem
>>
>> 3.) AWS is enforcing some weird IOPS limit
>>
>> 4.) durable-queue is misbehaving
>>
>> As to possibility #1, I notice the app starts like this:
>>
>> Memory in use (percentage/used/max-heap): (\"66%\" \"2373M\" \"3568M\")
>>
>> but 60 seconds later I see:
>>
>> Memory in use (percentage/used/max-heap): (\"94%\" \"3613M\" \"3819M\")
>>
>> So I've run out of allowed memory. But why is that? I thought I gave this
>> app 7 gigs:
>>
>> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
>>
>> As to possibility #2, I found this old post on the Clojure mailist:
>>
>> Andy Fingerhut wrote, "one thing I've found in the past on a 2-core
>> machine that was achieving much less than 2x speedup was memory bandwidth
>> being the limiting factor."
>>
>>
>> https://groups.google.com/forum/#!searchin/clojure/xmx$20xms$20maximum%7Csort:relevance/clojure/48W2eff3caU/HS6u547gtrAJ
>>
>> But I am not sure how to test this.
>>
>> As to possibility #3, I'm not sure how AWS enforces its IOPS limits. If
>> people think this is the most likely possibility, then I will repost my
>> question in an AWS forum.
>>
>> As to possibility #4, durable-queue is well-tested and used in a lot of
>> projects, and Zach Tellman is smart and makes few mistakes, so I'm doubtful
>> that it is to blame, but I do notice that it starts off with 4 active slabs
>> and then after 120 seconds, it is only using 1 slab. Is that expected? If
>> people think this is the possible problem then I'll ask somewhere specific
>> to durable-queue
>>
>> Overall, my log information looks like this:
>>
>> ("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs
>> 3, :num-active-slabs 2, :enqueued 370137, :retried 0, :completed 369934,
>> :in-progress 10}})
>>
>> ("\nResource usage: " "Memory in
Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
a small thing here, if memory usage is important you should be building and
running an uberjar instead of using lein on the server (this also has other
benefits), and if you are doing that your project.clj jvm-opts are not
used, you have to configure your java command line in aws instead
On Wed, Oct 11, 2017 at 3:52 PM <lawrence.krub...@gmail.com> wrote:
> I can't figure out if this is a Clojure question or an AWS question. And
> if it is a Clojure question, I can't figure out if it is more of a general
> JVM question, or if it is specific to some library such as durable-queue. I
> can redirect my question elsewhere, if people think this is an AWS
> question.
>
> In my project.clj, I try to give my app a lot of memory:
>
> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
>
> And the app starts off pulling data from MySQL and writing it to
> Durable-Queue at a rapid rate. ( https://github.com/Factual/durable-queue
> )
>
> I have some logging set up to report every 30 seconds.
>
> :enqueued 370137,
>
> 30 seconds later:
>
> :enqueued 608967,
>
> 30 seconds later:
>
> :enqueued 828950,
>
> It's a dramatic slowdown. The app is initially writing to the queue at
> faster than 10,000 documents a second, but it slows steadily, and after 10
> minutes it writes less than 1,000 documents per second. Since I have to
> write a few million documents, 10,000 a second is the slowest speed I can
> live with.
>
> The queues are in the /tmp folder of an AWS instance that has plenty of
> disk space, 4 CPUs, and 16 gigs of RAM.
>
> Why does the app slow down so much? I had 4 thoughts:
>
> 1.) the app struggles as it hits a memory limit
>
> 2.) memory bandwidth is the problem
>
> 3.) AWS is enforcing some weird IOPS limit
>
> 4.) durable-queue is misbehaving
>
> As to possibility #1, I notice the app starts like this:
>
> Memory in use (percentage/used/max-heap): (\"66%\" \"2373M\" \"3568M\")
>
> but 60 seconds later I see:
>
> Memory in use (percentage/used/max-heap): (\"94%\" \"3613M\" \"3819M\")
>
> So I've run out of allowed memory. But why is that? I thought I gave this
> app 7 gigs:
>
> :jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
>
> As to possibility #2, I found this old post on the Clojure mailist:
>
> Andy Fingerhut wrote, "one thing I've found in the past on a 2-core
> machine that was achieving much less than 2x speedup was memory bandwidth
> being the limiting factor."
>
>
> https://groups.google.com/forum/#!searchin/clojure/xmx$20xms$20maximum%7Csort:relevance/clojure/48W2eff3caU/HS6u547gtrAJ
>
> But I am not sure how to test this.
>
> As to possibility #3, I'm not sure how AWS enforces its IOPS limits. If
> people think this is the most likely possibility, then I will repost my
> question in an AWS forum.
>
> As to possibility #4, durable-queue is well-tested and used in a lot of
> projects, and Zach Tellman is smart and makes few mistakes, so I'm doubtful
> that it is to blame, but I do notice that it starts off with 4 active slabs
> and then after 120 seconds, it is only using 1 slab. Is that expected? If
> people think this is the possible problem then I'll ask somewhere specific
> to durable-queue
>
> Overall, my log information looks like this:
>
> ("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs
> 3, :num-active-slabs 2, :enqueued 370137, :retried 0, :completed 369934,
> :in-progress 10}})
>
> ("\nResource usage: " "Memory in use (percentage/used/max-heap):
> (\"66%\" \"2373M\" \"3568M\")\n\nCPU usage (how-many-cpu's/load-average):
> [4 5.05]\n\nFree memory in jvm: [1171310752]")
>
> 30 seconds later
>
> ("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs
> 4, :num-active-slabs 4, :enqueued 608967, :retried 0, :completed 608511,
> :in-progress 10}})
>
> ("\nResource usage: " "Memory in use (percentage/used/max-heap):
> (\"76%\" \"2752M\" \"3611M\")\n\nCPU usage (how-many-cpu's/load-average):
> [4 5.87]\n\nFree memory in jvm: [901122456]")
>
> 30 seconds later
>
> ("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs
> 4, :num-active-slabs 3, :enqueued 828950, :retried 0, :completed 828470,
> :in-progress 10}})
>
> ("\nResource usage: " "Memory in use (percentage/used/max-heap):
> (\"94%\" \"3613M\" \"3819M\")\n\nCPU usage (how-many-cpu's/load-average)
possibly a Clojure question or possibly an AWS question: slow writes to durable-queue
I can't figure out if this is a Clojure question or an AWS question. And if
it is a Clojure question, I can't figure out if it is more of a general JVM
question, or if it is specific to some library such as durable-queue. I can
redirect my question elsewhere, if people think this is an AWS question.
In my project.clj, I try to give my app a lot of memory:
:jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
And the app starts off pulling data from MySQL and writing it to
Durable-Queue at a rapid rate. ( https://github.com/Factual/durable-queue )
I have some logging set up to report every 30 seconds.
:enqueued 370137,
30 seconds later:
:enqueued 608967,
30 seconds later:
:enqueued 828950,
It's a dramatic slowdown. The app is initially writing to the queue at
faster than 10,000 documents a second, but it slows steadily, and after 10
minutes it writes less than 1,000 documents per second. Since I have to
write a few million documents, 10,000 a second is the slowest speed I can
live with.
The queues are in the /tmp folder of an AWS instance that has plenty of
disk space, 4 CPUs, and 16 gigs of RAM.
Why does the app slow down so much? I had 4 thoughts:
1.) the app struggles as it hits a memory limit
2.) memory bandwidth is the problem
3.) AWS is enforcing some weird IOPS limit
4.) durable-queue is misbehaving
As to possibility #1, I notice the app starts like this:
Memory in use (percentage/used/max-heap): (\"66%\" \"2373M\" \"3568M\")
but 60 seconds later I see:
Memory in use (percentage/used/max-heap): (\"94%\" \"3613M\" \"3819M\")
So I've run out of allowed memory. But why is that? I thought I gave this
app 7 gigs:
:jvm-opts ["-Xms7g" "-Xmx7g" "-XX:-UseCompressedOops"])
As to possibility #2, I found this old post on the Clojure mailist:
Andy Fingerhut wrote, "one thing I've found in the past on a 2-core machine
that was achieving much less than 2x speedup was memory bandwidth being the
limiting factor."
https://groups.google.com/forum/#!searchin/clojure/xmx$20xms$20maximum%7Csort:relevance/clojure/48W2eff3caU/HS6u547gtrAJ
But I am not sure how to test this.
As to possibility #3, I'm not sure how AWS enforces its IOPS limits. If
people think this is the most likely possibility, then I will repost my
question in an AWS forum.
As to possibility #4, durable-queue is well-tested and used in a lot of
projects, and Zach Tellman is smart and makes few mistakes, so I'm doubtful
that it is to blame, but I do notice that it starts off with 4 active slabs
and then after 120 seconds, it is only using 1 slab. Is that expected? If
people think this is the possible problem then I'll ask somewhere specific
to durable-queue
Overall, my log information looks like this:
("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs 3,
:num-active-slabs 2, :enqueued 370137, :retried 0, :completed 369934,
:in-progress 10}})
("\nResource usage: " "Memory in use (percentage/used/max-heap):
(\"66%\" \"2373M\" \"3568M\")\n\nCPU usage (how-many-cpu's/load-average):
[4 5.05]\n\nFree memory in jvm: [1171310752]")
30 seconds later
("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs 4,
:num-active-slabs 4, :enqueued 608967, :retried 0, :completed 608511,
:in-progress 10}})
("\nResource usage: " "Memory in use (percentage/used/max-heap):
(\"76%\" \"2752M\" \"3611M\")\n\nCPU usage (how-many-cpu's/load-average):
[4 5.87]\n\nFree memory in jvm: [901122456]")
30 seconds later
("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs 4,
:num-active-slabs 3, :enqueued 828950, :retried 0, :completed 828470,
:in-progress 10}})
("\nResource usage: " "Memory in use (percentage/used/max-heap):
(\"94%\" \"3613M\" \"3819M\")\n\nCPU usage (how-many-cpu's/load-average):
[4 6.5]\n\nFree memory in jvm: [216459664]")
30 seconds later
("\nStats about from-mysql-to-tables-queue: " {"message" {:num-slabs 1,
:num-active-slabs 1, :enqueued 1051974, :retried 0, :completed 1051974,
:in-progress 0}})
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Joy of Clojure question
Bridget and Guru are both right about the reason that \3 is found at position 13 in the string. However, the code you typed isn't valid Clojure, since the pos function expects pred to be a function of v. Thus, your pos call would need to look like this: (pos #(= % \3) :a 4 :b 1 :c 3 :d 4) = (13) ;; note that pos returns a list of all indeces matching pred Alternatively, you could have defined pos to simply perform equality matching on its first argument like so: (defn pos [val coll] (for [[i v] (index coll) :when (= val v)] i)) (pos \3 :a 4 :b 1 :c 3 :d 4) = (13) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Joy of Clojure question
Haha, right on Gary! I used the later version of the original function, but the earlier call to the original function. Here is what it looked like in the beginning. This was an example of how not to write a function that looks up indices by value. The example code I posted above was the supposed better way. (defn pos [e coll] (let [cmp (if (map? coll) #(= (second %1) %2) #(= %1 %2))] (loop [s coll idx 0] (when (seq s) (if (cmp (first s) e) (if (map? coll) (first (first s)) idx) (recur (next s) (inc idx))) On Thursday, May 15, 2014 1:31:34 AM UTC+9, Gary Johnson wrote: Bridget and Guru are both right about the reason that \3 is found at position 13 in the string. However, the code you typed isn't valid Clojure, since the pos function expects pred to be a function of v. Thus, your pos call would need to look like this: (pos #(= % \3) :a 4 :b 1 :c 3 :d 4) = (13) ;; note that pos returns a list of all indeces matching pred Alternatively, you could have defined pos to simply perform equality matching on its first argument like so: (defn pos [val coll] (for [[i v] (index coll) :when (= val v)] i)) (pos \3 :a 4 :b 1 :c 3 :d 4) = (13) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Joy of Clojure question
I am reading The Joy of Clojure now and am finishing chapter 5 on sequence abstractions. There is an example given that demonstrates how to locate the index of an element in a sequence by value, but I don't understand why the character lookup here is returning 13. Can somebody please explain this to me? (defn index [coll] (cond (map? coll) (seq coll) (set? coll) (map vector coll coll) :else (map vector (iterate inc 0) coll))) (defn pos [pred coll] (for [[i v] (index coll) :when (pred v)] i)) (pos \3 :a 4 :b 1 :c 3 :d 4) = 13 J -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Joy of Clojure question
\3 is a character literal, so you're looking up the character 3. You're looking it up in a collection which is a string. Count from the first position, 0, a colon, then 1, the letter a, and so on. 3 is at the 13th position. On Tuesday, May 13, 2014 9:44:38 PM UTC-4, gamma235 wrote: I am reading The Joy of Clojure now and am finishing chapter 5 on sequence abstractions. There is an example given that demonstrates how to locate the index of an element in a sequence by value, but I don't understand why the character lookup here is returning 13. Can somebody please explain this to me? (defn index [coll] (cond (map? coll) (seq coll) (set? coll) (map vector coll coll) :else (map vector (iterate inc 0) coll))) (defn pos [pred coll] (for [[i v] (index coll) :when (pred v)] i)) (pos \3 :a 4 :b 1 :c 3 :d 4) = 13 J -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Joy of Clojure question
The call to (index) returns this vector ([0 \:] [1 \a] [2 \space] [3 \4] [4 \space] [5 \:] [6 \b] [7 \space] [8 \1] [9 \space] [10 \:] [11 \c] [12 \space] [13 \3] [14 \space] [15 \:] [16 \d] [17 \space] [18 \4]) And, 3 here is in the 13th position. Your (cond) in index function is picking up the :else branch. On Tue, May 13, 2014 at 6:44 PM, gamma235 jesus.diama...@gmail.com wrote: I am reading The Joy of Clojure now and am finishing chapter 5 on sequence abstractions. There is an example given that demonstrates how to locate the index of an element in a sequence by value, but I don't understand why the character lookup here is returning 13. Can somebody please explain this to me? (defn index [coll] (cond (map? coll) (seq coll) (set? coll) (map vector coll coll) :else (map vector (iterate inc 0) coll))) (defn pos [pred coll] (for [[i v] (index coll) :when (pred v)] i)) (pos \3 :a 4 :b 1 :c 3 :d 4) = 13 J -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Joy of Clojure question
Okay, both these answers make lots of sense. thank you for your help. J On Wednesday, May 14, 2014 11:47:51 AM UTC+9, Guru Devanla wrote: The call to (index) returns this vector ([0 \:] [1 \a] [2 \space] [3 \4] [4 \space] [5 \:] [6 \b] [7 \space] [8 \1] [9 \space] [10 \:] [11 \c] [12 \space] [13 \3] [14 \space] [15 \:] [16 \d] [17 \space] [18 \4]) And, 3 here is in the 13th position. Your (cond) in index function is picking up the :else branch. On Tue, May 13, 2014 at 6:44 PM, gamma235 jesus.d...@gmail.comjavascript: wrote: I am reading The Joy of Clojure now and am finishing chapter 5 on sequence abstractions. There is an example given that demonstrates how to locate the index of an element in a sequence by value, but I don't understand why the character lookup here is returning 13. Can somebody please explain this to me? (defn index [coll] (cond (map? coll) (seq coll) (set? coll) (map vector coll coll) :else (map vector (iterate inc 0) coll))) (defn pos [pred coll] (for [[i v] (index coll) :when (pred v)] i)) (pos \3 :a 4 :b 1 :c 3 :d 4) = 13 J -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.comjavascript: Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com javascript: For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Clojure question
ng...@students.rowan.edu writes: Hello, I'm doing a school paper on Clojure but there are two questions that i just can't find answers to anywhere they are: Are data types bound to variables and parameters at compile-time? run-time? a combination? and also, How are parameters passed? (Pass by value? Pass by reference? Pass by value-result? etc.) If anybody knows the answers to these two questions I would greatly appreciate it! If you can't find the answers to these questions anywhere, you either have not looked, or don't understand the questions. Either way, you will learn nothing from the answers you are getting. You probably wouldn't be able to tell the difference between a correct answer and one which is just a wind-up. Which category do you think Marko's answer comes into? Phil -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Clojure question
Hello, I'm doing a school paper on Clojure but there are two questions that i just can't find answers to anywhere they are: Are data types bound to variables and parameters at compile-time? run-time? a combination? and also, How are parameters passed? (Pass by value? Pass by reference? Pass by value-result? etc.) If anybody knows the answers to these two questions I would greatly appreciate it! -Matt -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Clojure question
Are data types bound to variables and parameters at compile-time? run-time? a combination? Clojure, like LISPs in general, is primarily a dynamically-typed language, so variables/parameters don't have a type assigned to them. However, on JVM Clojure there are optional type hints, which constrain the type at compile time. How are parameters passed? (Pass by value? Pass by reference? Pass by value-result? etc.) Clojure inherits the argument-passing semantics from Java. So it is pass-by-value, where the value passed is an object reference. In addition there are optimization facilities that enable the passing of primitive-typed values. -marko -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
PCL - Clojure: question about variables
I am new to Clojure and have been working through some of the examples on the PCL - Clojure blog. Currently I am working through Chapter 6 (variables). The blog says Nothing stops multiple functions from closing on the same variables. Here is a function that returns an incrementer, decrementer, and accessor, all sharing the same counter: (defn counters [] (let [count (ref 0)] (list #(dosync (alter count inc)) #(dosync (alter count dec)) #(deref count I tried to this example (with Clojure 1.3.0 and 1.2.1) and found that it does not appear to work, or I am totally missing something. Here is some output from a REPL session using this function. Could you help me understand the output or, if there is a mistake in the example, could you explain what it is? Question 1: I expected these two statements to produce output 1, 2 user= ((nth (counters) 0)) 1 user= ((nth (counters) 0)) 1 Question 2: So then I did the following: user= (def a (nth (counters) 0)) #'user/a user= (def b (nth (counters) 2)) #'user/b user= (a) 1 user= (a) 2 user= (a) 3 user= (b) 0 Now the incrementer appears to be working, but then when I try to use the accessor to inspect the value of count, it is still 0. So it appears that the 3 anonymous functions in the list are not closing over count. Could somebody help me understand this behavior? Thank you for your help. matthew -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: PCL - Clojure: question about variables
Hi, with each call to counters you get new incrementers, decrementers and accessors referecing different counters underneath. Capture the return value of the counters call. (def fns (counters)) ((nth fns 0)) ((nth fns 0)) ((nth fns 2)) Hope this helps. Sincerely Meikel -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
beginner clojure question: OutOfMemory error processing (slightly) large data file
Hi, I (still) consider myself new to clojure. I am trying to read a 37Mb file that will grow 500k every 2 days. I don't consider this to be input large enough file to merit using Hadoop and I'd like to process it in Clojure in an efficient, speedy, and idiomatic way. I simply want something akin to a transpose, where the input looks like this: ( [ a1 b1 c1 d1 ] [ a2 b2 c2 d2 ] [ a3 b3 c3 d3 ]) …and the output looks like this: [ [ a1 a2 a3 ] [ b1 b2 b3 ] [ c1 c2 c3 ] [ d1 d2 d3 ] ] Gleaning what I can from various sources and cobbling them together, I have the following below, which works for small input but not for the intended file sizes (and larger) I'd like it to be able to handle. (use 'clojure.contrib.io) (require 'clojure.string) (def tabfn /Users/avram/data/testdata.tab) (defn is-comment? Checks if argument is a comment (i.e. starts with a '#'). Returns: boolean. [line] (= \# (first line))) (defn data-lines Returns data lines in file (i.e. all lines that do not start with '#') Returns: sequence containing data lines [filename] (drop-while is-comment? (line-seq (reader filename (defn parsed-data-lines [filename] (map #(clojure.string/split % #\t) (data-lines filename))) (def signals (vec (apply map vector (parsed-data-lines tabfn user= (def signals (vec (apply map vector (parsed-data-lines tabfn java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:68) How can I avoid the OutOfMemoryError? Is there a Leiningen setting where I can increase the memory or is there a more efficient way to achieve this? Also, I'd prefer to read in gzip'd tab-delimited files instead of uncompressed tab-delimited files. What is the idiomatic clojure way to do this? Comments on improvements and criticisms welcome :) Thanks, Avram -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: beginner clojure question: OutOfMemory error processing (slightly) large data file
On Tue, Mar 22, 2011 at 4:00 PM, Avram aav...@me.com wrote: Hi, I (still) consider myself new to clojure. I am trying to read a 37Mb file that will grow 500k every 2 days. I don't consider this to be input large enough file to merit using Hadoop and I'd like to process it in Clojure in an efficient, speedy, and idiomatic way. I simply want something akin to a transpose, where the input looks like this: ( [ a1 b1 c1 d1 ] [ a2 b2 c2 d2 ] [ a3 b3 c3 d3 ]) …and the output looks like this: [ [ a1 a2 a3 ] [ b1 b2 b3 ] [ c1 c2 c3 ] [ d1 d2 d3 ] ] Gleaning what I can from various sources and cobbling them together, I have the following below, which works for small input but not for the intended file sizes (and larger) I'd like it to be able to handle. You'll need to avoid holding onto the head of your line-seq, which means you'll need to make multiple passes over the data, one for the as, one for the bs, and etc., with the output a lazy seq of lazy seqs. (defn data-lines Returns data lines in file (i.e. all lines that do not start with '#') Returns: sequence containing data lines [filename] (drop-while is-comment? (line-seq (reader filename The description doesn't match the function, unless it's guaranteed that no line will start with # after the first line that doesn't do so. You may want remove instead of drop-while here, or to change the doc string. Also, I'd prefer to read in gzip'd tab-delimited files instead of uncompressed tab-delimited files. What is the idiomatic clojure way to do this? There are zip functions in the Java standard library. I don't know if they can handle gzip, or just pkzip. In the worst case, you'd have no library you could use. Even then, it could be done in at least two ways. 1. Use Runtime/exec to call shell tools to gunzip the file to a temporary file for processing. 2. Read at wikipedia and implement gunzip in Clojure, using byte arrays and whatever other tools you'd need to work with binary data at a low level, and/or Java's ByteBuffer and related classes. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: beginner clojure question: OutOfMemory error processing (slightly) large data file
Hi Avram, Assuming you're using the Sun/Oracle JDK, you can increase the size of the Java heap with the -Xmx command-line option. For example: java -Xmx512mb -cp clojure.jar:your-source-dir clojure.main Will run Java with a 512 MB heap. This increases the amount of memory available to your program. Obviously, you don't want the heap to be larger than the available RAM. With Leiningen, you can add the :jvm-opts option in project.clj, as shown here: https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L142 More generally, this line: (def signals (vec ...)) says that you want the entire result, as a vector, stored as the value of the Var `signals`. That means your entire result data must fit in the Java heap. For a 37 MB file, that's not unreasonable, but as soon as your file gets larger than your available RAM, you'll have to come up with an alternate approach. -Stuart Sierra clojure.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: beginner clojure question: OutOfMemory error processing (slightly) large data file
Oh, and the standard JDK class java.util.zip.GZIPInputStream implements gzip decompression. -Stuart Sierra clojure.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: beginner clojure question: OutOfMemory error processing (slightly) large data file
Thanks, Stuart. With Leiningen, you can add the :jvm-opts option in project.clj, Cool, this is what I was looking for :) (def signals (vec ...)) says that you want the entire result, as a vector, stored as the value of the Var `signals`. That means your entire result data must fit in the Java heap. Yes, this is what I want. I suppose that if it grows too large, I can figure out a way to write it to a file somehow instead of creating a vector. The end result will likely be a JSON representation for each of the signals a, b, c, and d written to a file. ~A -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Swing Java example to Clojure - Question about better translation
Hello, I've started dabbling into Swing Clojure, and for fun decided to translate one of the Sun examples (requires Java SE 1.6 though) It's located here: http://java.sun.com/docs/books/tutorial/uiswing/examples/learn/index.html#CelsiusConverter and the java code here: http://java.sun.com/docs/books/tutorial/uiswing/examples/learn/CelsiusConverterProject/src/learn/CelsiusConverterGUI.java I've translated the above java code to clojure here: http://paste.lisp.org/display/79061 But I feel I did not do good job on it. The hard part was this convention of creating object inline, and calling it's method to further add elements/extend the system. It reminds of TurboVision from the glory TurboPascal days: layout.setHorizontalGroup( layout.createParallelGroup (javax.swing.GroupLayout.Alignment.LEADING) .addGroup(layout.createSequentialGroup() .addContainerGap() .addGroup(layout.createParallelGroup (javax.swing.GroupLayout.Alignment.LEADING) .addGroup(layout.createSequentialGroup() . // more code omitted .addContainerGap(27, Short.MAX_VALUE)) ); I've did my best: (doto layout (.setHorizontalGroup (.. layout (createParallelGroup GroupLayout$Alignment/LEADING) (addGroup (.. layout (createSequentialGroup) (addContainerGap) (addGroup (.. layout (createParallelGroup GroupLayout$Alignment/LEADING) (addGroup ;; more code omitted (addComponentfahrenheit-label) (addContainerGap 27 Short/MAX_VALUE) But I feel I can do it better (some fancy macro?). Any ideas how I can improve it. I understand from the Java comments that the layout code was autogenerated by a tool, but still it's an interresting question. Granted the .. sugary macro helped a lot, but my brain melted a bit while I was translating it :) Here is the full code, in case the paste.lisp.org doesn't work: (import '(javax.swing JFrame JLabel JTextField JButton GroupLayout GroupLayout$Alignment SwingConstants LayoutStyle LayoutStyle $ComponentPlacement) '(java.awt.event ActionListener) '(java.awt Component)) (defn celsius-converter-gui Translation of the simple Celsius Converter GUI Swing Example from Java to Clojure (requires Java SE 1.6) http://java.sun.com/docs/books/tutorial/uiswing/examples/learn/CelsiusConverterProject/src/learn/CelsiusConverterGUI.java; [] (let [frame(new JFrame Celsius Converter) pane (. frame getContentPane) layout (new GroupLayout pane) temp-text-field (new JTextField) celsius-label(new JLabel Celsius) convert-button (new JButton Convert) fahrenheit-label (new JLabel Fahrenheit)] (. convert-button addActionListener (proxy [ActionListener] [] (actionPerformed [event] (. fahrenheit-label setText (str (+ 32.0 (* 1.8 (Double/parseDouble (. temp-text-field getText Fahrenheit) (doto layout (.setHorizontalGroup (.. layout (createParallelGroup GroupLayout$Alignment/LEADING) (addGroup (.. layout (createSequentialGroup) (addContainerGap) (addGroup (.. layout (createParallelGroup GroupLayout$Alignment/LEADING) (addGroup (.. layout (createSequentialGroup) (addComponenttemp-text-field GroupLayout/PREFERRED_SIZE GroupLayout/DEFAULT_SIZE GroupLayout/PREFERRED_SIZE) (addPreferredGap LayoutStyle$ComponentPlacement/RELATED) (addComponentcelsius-label))) (addGroup (.. layout (createSequentialGroup) (addComponentconvert-button) (addPreferredGap LayoutStyle$ComponentPlacement/RELATED) (addComponentfahrenheit-label) (addContainerGap 27 Short/MAX_VALUE) (.linkSize SwingConstants/HORIZONTAL (into-array Component [convert-button temp-text-field])) (.setVerticalGroup (.. layout (createParallelGroup GroupLayout$Alignment/LEADING) (addGroup (.. layout (createSequentialGroup) (addContainerGap) (addGroup (.. layout (createParallelGroup GroupLayout$Alignment/BASELINE)
Re: Having struggle in understanding FP (and Clojure question)
On Fri, Nov 7, 2008 at 12:25 AM, cwyang [EMAIL PROTECTED] wrote: My expectation is these: 1) For C10K problem (or C100K), application must not use native threads. Big stack size means low concurrency Hi, I assume you mean 'new native thread per request' is bad for CnK. Clojure's thread-pooling is not very different from the N:M threading model that Erlang-OTP/Yaws uses, in a general sense. 4) At the same time, there must be ways for connecting conceptual gap between 2) and 3). In other words, the way for suspending current execution of function, saving current execution snapshot (normally native thread stack, but may be different), and switching to other functions are needed when the request for I/O should be blocked. It's important that the size of current execution snapshot should be small, since it determine the degree of concurrency. Sounds like you're looking for first-class continuations here.You won't find that on any JVM language, there's no JVM support for them. (It is true that JVM-based Scheme languages have continuations, but they are severely limited by the constraints of the JVM). If you really believe you need this, you might be interested in Termite, an Erlang-style actor system written for Gambit Scheme (which has serializable first-class continuations and very lightweight threads, but no SMP support if I remember correctly). An alternative would be to use non-blocking I/O multiplexing -- an event-driven model like you would write with select() or epoll(). I don't know the Java universe very well, but I know JVM has asynchronous I/O, so this must be an available option. Writing such programs can sometimes lead to hard-to-understand, fragmented application code, but Clojure's flexible syntax might be able to help unify the fragments into a more readable form. Best, Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On 05.11.2008, at 17:16, Mark H. wrote: and replace copies with destructive writes. I haven't seen a purely functional formulation of LU factorization but it could be done without too much trouble. Of course there's no reason to go through that effort because people spend so much time optimizing LU and its constituent components that you would be better off reusing their work. For the immediate future, yes. But with changing computer architectures, the existing algorithms and routines may lose much of their interest in the future. To me the more interesting and rewarding task is to figure out how to splice existing HPC libraries into a functional framework, without losing the ability to reason functionally about the components. Unfortunately, it is already a bit of a pain to link existing HPC libraries (written in Fortran, C, or C++) to functional code in any decent language. Clojure won't help there, as there are still very few HPC libraries for the JVM and JNI adds too much of an overhead. Definitely! We've got at least one fellow here who uses Common Lisp to generate stencil codes. He's been thinking about switching to Clojure ever since he and I worked on a thorny Lisp problem together ;-) I am looking forward to a nice Clojure library then :-) Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
Thanks everyone. I've read all the replies, and I might understand something. (Though it seems that discussion goes to beyond my current understanding in the middle of this discussion threads) To explore further, I become another owner of 'Programming Clojure' :-) For my second question, I'll keep thinking and repost question. I'm really concern about concurrency of Clojure, since the concurrency in my world means _the C10K problem_, that is supporting simultaneous 1 connections or over for networking applications. (http://www.kegel.com/c10k.html) In YAWS, Erlang have shown one solution, so I'm curios that Clojure could be another viable alternative, and that how well-suited Clojure is for that kind of concurrency. My expectation is these: 1) For C10K problem (or C100K), application must not use native threads. Big stack size means low concurrency 2) For application developer, thread model is easy to develop and follow. 3) Therefore, underlying system should support concurrency facility like micro-thread, lightweight process (in Erlang) , agent (in Clojure), or whatever. In this case, underlying facility should have high-performance. 4) At the same time, there must be ways for connecting conceptual gap between 2) and 3). In other words, the way for suspending current execution of function, saving current execution snapshot (normally native thread stack, but may be different), and switching to other functions are needed when the request for I/O should be blocked. It's important that the size of current execution snapshot should be small, since it determine the degree of concurrency. After I invest some time to learn Clojure, I'll repost. Thank you. - cw --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Nov 6, 11:58 am, Konrad Hinsen [EMAIL PROTECTED] wrote: On 05.11.2008, at 17:16, Mark H. wrote: For the immediate future, yes. But with changing computer architectures, the existing algorithms and routines may lose much of their interest in the future. Haha, yes, we're working on this ;-) (a number of my colleagues are, at least). To me the more interesting and rewarding task is to figure out how to splice existing HPC libraries into a functional framework, without losing the ability to reason functionally about the components. Unfortunately, it is already a bit of a pain to link existing HPC libraries (written in Fortran, C, or C++) to functional code in any decent language. Clojure won't help there, as there are still very few HPC libraries for the JVM and JNI adds too much of an overhead. Does the JNI require copy-in / copy-out for arrays of floating-point numbers? I know Java has messed-up floating-point (Sun stupidly took away x87's 80-bit temps and other good things) but I'd at least like to avoid copy overhead for matrix and vector ops. I don't mind the extra function call out of JVM space as long as I can amortize it over the cost of a big matrix operation. (No way I'm writing loops in Clojure for that.) There are a number of Python-based projects to do what you mentioned (link HPC libraries in Fortran or C to a higher-level language), so it's not impossible. It's just a lot of tedious work for some unlucky coders. Definitely! We've got at least one fellow here who uses Common Lisp to generate stencil codes. He's been thinking about switching to Clojure ever since he and I worked on a thorny Lisp problem together ;-) I am looking forward to a nice Clojure library then :-) We'll see -- he's got it working in CL now and he's got a deadline, so he may not bother migrating it. Plus he likes the ECL / C linkup; Java may not be so helpful for him. mfh --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
Hi, On 5 Nov., 08:31, Konrad Hinsen [EMAIL PROTECTED] wrote: That's exactly my point. Multimethods may well be sufficient or even superior for implementing OO concepts useful in Clojure. We will see when someone actually uses them this way (or has it already happened?). But as long as the most important interfaces (sequences, maps, numbers), which happen to be the ones also used by the built-in types, don't use the same mechanism, there will always be first-class and second-class citizens in Clojure's datatype universe. I don't think that there are first-class and second-class citizens. They just have a different aspect. Having pure Clojure classes with multimethods might well be integrated and nice, but you get problems when you have to interface to Java. On the other hand using Java classes gives you easy interface, but you have to live with gen-class. So which one is first, which one second class? I think each solution has just different Pros and Cons. What would interesting, is the question, whether both approaches could be combined. We make a Java class (via gen-class), which gets a default implementation for each method, which references a similar named multimethod instead of the Class-methodName. Clojure classes deriving from that class do not provide an a Java method themselves, but register with the multimethod. The Clojure code could use the multimethods, while there are still the methods available for a Java interface. So the downside would be that one can only inherit from one such prototype class (since it needs to be a class, not an interface, for the default implementation of the methods) or one has to respecify the method - multimethod translation in all sub classes. Maybe a modified gen-class could support? (These are just random thoughts. Maybe they make sense, maybe not.) For the built-in datatypes: I implemented nth and get as multimethods (as some kind of proof-of-concept), but Rich was not very interested due to performance issues and the fact, that the datatypes are used internally in very low-level areas. I don't really follow the argument about monkey patching. I think it's not monkey patching at all, since the multimethods only define the interface. So I don't modify a thing which is returned by hash-map. That's still a hash map using the hash-map implementation. - http://clojure-log.n01se.net/date/2008-10-06.html#13:18 Sincerely Meikel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Nov 5, 2008, at 11:12, mb wrote: I don't think that there are first-class and second-class citizens. They just have a different aspect. Right, one can't say one is superior to the other. There are just two separate worlds, each having its own characteristics. Having pure Clojure classes with multimethods might well be integrated and nice, but you get problems when you have to interface to Java. On the other hand using Java classes gives you easy interface, but you have to live with gen-class. But does gen-class have to look the way it does? Couldn't the same functionality be provided in a way that looks more like a proper part of the language? What would interesting, is the question, whether both approaches could be combined. We make a Java class (via gen-class), which gets a default implementation for each method, which references a similar named multimethod instead of the Class-methodName. That sounds like an interesting idea. For the built-in datatypes: I implemented nth and get as multimethods (as some kind of proof-of-concept), but Rich was not very interested due to performance issues and the fact, that the datatypes are used internally in very low-level areas. I can understand performance arguments, of course. And I don't really want to modify the built-in types in any way, I just want to be able to use the same interfaces in completely independent datatype implementations. The current implementation of nth provides a special implementation for each kind of datatype that it knows about, and fails for unknown types. Shouldn't it be possible to add a default case at the end that calls a Clojure multimethod? That shouldn't have any performance impact on the built-in datatypes. I don't really follow the argument about monkey patching. I think it's not monkey patching at all, since the multimethods only define the interface. I agree. It's no more monkey-patching than any use of multimethods would be monkey-patching. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
Hi, On 5 Nov., 15:40, Konrad Hinsen [EMAIL PROTECTED] wrote: But does gen-class have to look the way it does? Couldn't the same functionality be provided in a way that looks more like a proper part of the language? I'm not sure about the form itself. I had look at a CLOS tutorial and that defclass doesn't look to different in the form the function is called, a mix of keywords and arguments. My main concern with gen-class is that it doesn't use the namespace itself. I posted a patch[1] yesterday which addresses the namespace and -/_ translation. I rose this also some weeks ago on the list but there was no response. So the general interest seems to be pretty low. On the other hand Rich, seems to have something like this on his TODO list[2]. But maybe everything is different again because of AOT. So I'm not sure, what his plans are for gen-class right now. [1]: http://groups.google.com/group/clojure/browse_thread/thread/386d90a8b757aee4 [2]: http://richhickey.backpackit.com/pub/1597914 The current implementation of nth provides a special implementation for each kind of datatype that it knows about, and fails for unknown types. Shouldn't it be possible to add a default case at the end that calls a Clojure multimethod? That shouldn't have any performance impact on the built-in datatypes. That is a good idea. It would be really cool. Sincerely Meikel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Nov 4, 11:46 pm, Konrad Hinsen [EMAIL PROTECTED] wrote: Long answer: SISAL is an example of a functional parallel language Ah, right, there was SISAL... unfortunately long forgotten. I saw a retrospective presentation on it at SIAM PP this spring by one of the Livermore folks who promoted it. Pretty neat! One could express solving linear systems (which is what I presume you mean by matrix inversion, unless you really want the entries of the inverse) in a purely functional way using a language like SISAL with a compiler that optimizes away temporaries, and the resulting Could one? That was actually the core of my question. Of course HPC applications would require smart compilers that, but first of all there must be a purely functional algorithm that can be transformed automatically into an efficient one. Do you know any references to such algorithms in linear algebra? There's probably a big difference between in theory and in practice in this case ;-) SISAL didn't have multidimensional arrays so I doubt very much that it could have optimized away the creation of matrix temporaries. The usual trick to make things look functional is to index vector variables with the current iteration, so that you aren't overwriting anything. Then if the compiler can prove that there are no read-after- write conflicts, it can collapse the vector variables into one vector and replace copies with destructive writes. I haven't seen a purely functional formulation of LU factorization but it could be done without too much trouble. Of course there's no reason to go through that effort because people spend so much time optimizing LU and its constituent components that you would be better off reusing their work. Of course, HPC applications dealing with large data sets would always want algorithms that modify matrices in place instead of allocating more memory. I don't expect a purely OO HPC world any time soon. But it would still be interesting to know how many of the traditional CPU- hungry algorithms already have known efficient functional equivalents. To me the more interesting and rewarding task is to figure out how to splice existing HPC libraries into a functional framework, without losing the ability to reason functionally about the components. Me too. HPC is only one aspect of my work. And I think languages like Clojure can be useful for generating specialized HPC code as well, just like FFTW uses Caml code for generating C routines. Definitely! We've got at least one fellow here who uses Common Lisp to generate stencil codes. He's been thinking about switching to Clojure ever since he and I worked on a thorny Lisp problem together ;-) mfh --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Tuesday 04 November 2008 06:03, Konrad Hinsen wrote: ... As an illustration of the two approaches, consider a program to sort data. In OOP, one would define an abstract class comparable with a method sort that works by calling methods such as greater and equal implemented in concrete subclasses. In FP, one would write a function sort that takes as arguments a list of things to sort plus a function to do the comparisons. At the top level of the program, you'd see interface comparable in the OOP version and function sort in the FP version. A mixed OOP-FP program might call the FP function sort and pass the method compare of a subclass of comparable as the comparison function. Even the current Java libraries and object model belie this comparison. An array of intrisically comparable instances (those that implement Comparable) can be sorted without supplying a Comparator. But an array of arbitrary Objects can be sorted in arbitrary and flexible ways by supplying a Comparable that accepts the types submitted to it. Konrad. Randall Schulz --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Nov 4, 2008, at 15:20, mb wrote: And as an effect of not forcing the sorting logic into some class, one can easily sort the same data in different ways in the FP style. While implementing an interface Comparable defines *one* way of sorting, the FP style separates the functions from the data. So it is eg. easily possible to sort a list of email objects by subject, author or date, or any combination thereof. Sorting is indeed an example where the FP approach is superior. But then, it is not representative in size and complexity of a typical program or library. One could easily cite other examples where OOP would be the better choice. My ideal language would support both as much as possible. An example of where Clojure lacks OO features, in my opinion, is your lazy-map library. It provides an additional implementation for an existing interface, which is a typical OO approach, and a very useful one. But although all your code is written Clojure, it looks like a kludge in having to use a feature (genclass) intended for Java compatibility and forcing an unnatural file structure (one file for the methods, one for the code generation, and one for the end-user wrapper) on the implementation. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Nov 4, 2008, at 3:44, cwyang wrote: So, I understand as follows: - OOP: keeping objects, which has states and methods. methods are encapuslated to the corresponding object. - FP: keeping objects(or structs or variables, whatever it is called) and functions on the objects. functions does not encapsulated to specific objects, for some reasons. Other characteristics such as function as first-order data structure and enforcing functional way (discourated side effects and immutable variables) can be important, but not in modeling the world. I think the difference is more fundamental than what you describe. On the other hand, I don't think that OOP and FP are contradictory. Here is my view of things: OOP = data abstraction. The program structure is defined by the data that the program works on. Important data structures are identified and implemented together with their associated operations as classes. FP = algorithmic abstraction. The program structure is defined by the algorithms that are used. Important algorithmic patterns are identified and implemented as functions that take functional arguments, or as macros. Functional arguments make it possible to insert specific actions into a skeleton. There are additional properties typically associated with OOP (polymorphism, inheritance, ...) and FP (referential transparency, immutable data structures, ...), but I think the approch to program structure is the aspect where the two methods differ most. On the other hand, there is no reason not to use both data abstraction and algorithmic abstraction in a program. In that sense OOP and FP are orthogonal and combinable. As an illustration of the two approaches, consider a program to sort data. In OOP, one would define an abstract class comparable with a method sort that works by calling methods such as greater and equal implemented in concrete subclasses. In FP, one would write a function sort that takes as arguments a list of things to sort plus a function to do the comparisons. At the top level of the program, you'd see interface comparable in the OOP version and function sort in the FP version. A mixed OOP-FP program might call the FP function sort and pass the method compare of a subclass of comparable as the comparison function. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On Nov 4, 2008, at 17:26, Mark H. wrote: Of course sometimes you need side effects in order to accomplish useful work, and a good FP language will provide abstractions that help you limit side effects and reason about them. Clojure does this by means of refs, agents, and transactional updates. refs are different than regular values so that you know they are mutable. Vars are also mutable, even if that mutability is local to a thread. I guess one could write state-modifying code in Clojure just like in most imperative languages, it is merely discouraged and made hard to hide. That looks like a very reasonable compromise to me. An object-oriented program may also use FP. OOP and FP are not opposites. However, a lot of OO programming these days involves changing the state of opaque objects (therefore, not purely functional) via member functions. I've heard this characterized as the new spaghetti code because it's hard to reason about thread- safety when objects are being modified all the time, and when these modifications aren't syntactically obvious. Even in the absence of threads, changing state can be a cause of trouble, in particular if it is well hidden under a few layers of nice-looking APIs. On the other hand, I am not sure that all important algorithms already have purely functional equivalents that are sufficiently efficient for real life. Is there an efficient purely functional algorithm for matrix inversion, for example? Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
Hi, Am 04.11.2008 um 16:06 schrieb Konrad Hinsen: An example of where Clojure lacks OO features, in my opinion, is your lazy-map library. It provides an additional implementation for an existing interface, which is a typical OO approach, and a very useful one. But although all your code is written Clojure, it looks like a kludge in having to use a feature (genclass) intended for Java compatibility and forcing an unnatural file structure (one file for the methods, one for the code generation, and one for the end-user wrapper) on the implementation. I disagree. Clojure brings everything needed: multimethods. One defines a set of multimethods (the interface) and different arguments may have different implementations. Inheritance can also be replaced with delegation without problems. And in fact lazy-map does not inherit from any maptype. Instead it delegates to another instance without caring for whether it's a struct map, hash map or sorted map. With inheritance each of this types would have needed an own lazy-x-map class. This is again some style of functional OO programming. Suppose get, contains?, etc. were multimethods. I would not have needed gen-class. I just would register my own implementations for the lazy- map. But the problem is: Clojure's map interface is not implemented with multimethods. They have a Java side. If I want to provide a drop-in replacement I have to also serve that side. Hence I need gen-class. Or Java itself for that matter in case gen-class is too kludgy. I don't think that Clojure is that bad for object-oriented programming. It just looks a bit different than usual. Sincerely Meikel smime.p7s Description: S/MIME cryptographic signature
Re: Having struggle in understanding FP (and Clojure question)
On Nov 4, 10:01 am, Konrad Hinsen [EMAIL PROTECTED] wrote: On the other hand, I am not sure that all important algorithms already have purely functional equivalents that are sufficiently efficient for real life. Is there an efficient purely functional algorithm for matrix inversion, for example? Short answer: No ;-) Long answer: SISAL is an example of a functional parallel language designed for performance. Its compiler could optimize away temporaries in purely functional code (the equivalent of loop/recur). Optimizing away temporaries (esp. temp vectors and submatrices turns out to be important for several reasons: * Parallel GC is scary (not impossible, though) * Memory usage is tight for big problems * A lot of HPC code is limited by memory bandwidth so creating temporary vectors (for example) rather than overwriting existing ones is slow One could express solving linear systems (which is what I presume you mean by matrix inversion, unless you really want the entries of the inverse) in a purely functional way using a language like SISAL with a compiler that optimizes away temporaries, and the resulting implementation may very well be highly efficient. Krylov methods like conjugate gradient for sparse linear systems would be even easier for the compiler to optimize since they don't modify the matrix, only vectors. HPC coders tend to resist functional implementations because * they like to control memory allocation themselves * the existing code infrastructure is not functional * for cultural reasons That was basically why SISAL didn't take off at Livermore. HPC coders do, however, write a lot of control code, scripts, etc. that can benefit from FP. This is why I follow Clojure (for example). mfh --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On 04.11.2008, at 19:08, Meikel Brandmeyer wrote: Suppose get, contains?, etc. were multimethods. I would not have needed gen-class. I just would register my own implementations for the lazy-map. But the problem is: Clojure's map interface is not implemented with multimethods. They have a Java side. That's exactly my point. Multimethods may well be sufficient or even superior for implementing OO concepts useful in Clojure. We will see when someone actually uses them this way (or has it already happened?). But as long as the most important interfaces (sequences, maps, numbers), which happen to be the ones also used by the built-in types, don't use the same mechanism, there will always be first-class and second-class citizens in Clojure's datatype universe. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Having struggle in understanding FP (and Clojure question)
On 04.11.2008, at 20:07, Mark H. wrote: Long answer: SISAL is an example of a functional parallel language Ah, right, there was SISAL... unfortunately long forgotten. One could express solving linear systems (which is what I presume you mean by matrix inversion, unless you really want the entries of the inverse) in a purely functional way using a language like SISAL with a compiler that optimizes away temporaries, and the resulting Could one? That was actually the core of my question. Of course HPC applications would require smart compilers that, but first of all there must be a purely functional algorithm that can be transformed automatically into an efficient one. Do you know any references to such algorithms in linear algebra? Of course, HPC applications dealing with large data sets would always want algorithms that modify matrices in place instead of allocating more memory. I don't expect a purely OO HPC world any time soon. But it would still be interesting to know how many of the traditional CPU- hungry algorithms already have known efficient functional equivalents. That was basically why SISAL didn't take off at Livermore. HPC coders do, however, write a lot of control code, scripts, etc. that can benefit from FP. This is why I follow Clojure (for example). Me too. HPC is only one aspect of my work. And I think languages like Clojure can be useful for generating specialized HPC code as well, just like FFTW uses Caml code for generating C routines. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
newb slime/clojure question...
I've been trying to get clojure working with slime (on windows). I can run ants.clj from the command line fine. I can run simple expressions under slime (e.g. (+ 2 3)). However, when I try to run ants.clj from within slime, I get the following warning in the inferior lisp buffer, and nothing happens. Reflection warning, line: 280 - call to drawImage can't be resolved. Reflection warning, line: 283 - call to setPreferredSize can't be resolved. Looking at the sun.boot.class.path property for both executables seems the same. When I look at the system classpath using the invocation I found here: http://paste.lisp.org/display/69257 I get the clojure.jar, and ~/.clojure/*. From the command line directly, I just get clojure.jar on the classpath. Anybody have any advice on how I can fix this? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
