Re: using durable-queue, works locally, get :time-out when moving to an EC2
> Is the problem possibly a difference between your
> compilation environment and your deploy env?
This seems to be proven by the fact that upgrading Java on the EC2 instance
fixed the problem.
On Wednesday, January 16, 2019 at 3:49:41 PM UTC-5, Chris Nuernberger wrote:
>
> Are you using aot?
>
> We have used that durable queue with 1.8. In fact, we have a
> compatibility later that allows you to move from the durable queue to an
> Aws queue:
> (mostly undocumented)
>
> https://github.com/techascent/tech.queue
>
> Is the problem possibly a difference between your compilation environment
> and your deploy env?
>
>
>
> On Wed, Jan 16, 2019, 1:29 PM wrote:
>
>> So, I upgraded to Java 11, and now everything works. So I guess this was
>> a version conflict.
>>
>> Just curious, but is there a way for Factual to make durable-queue to
>> tell Leiningen that Java 11 is necessary?
>>
>>
>>
>> On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com
>> wrote:
>>>
>>> On the new EC2 instance, running Ubuntu:
>>>
>>> java -version
>>> openjdk version "1.8.0_191"
>>> OpenJDK Runtime Environment (build
>>> 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
>>> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>>>
>>> Is it possible this version does not have the checksum signature that
>>> durable-queue is looking for?
>>>
>>> I'm thinking this is some kind of version problem.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5,
>>> lawrence...@gmail.com wrote:
Sorry, I'm an idiot. The real error was when I called put!
I don't understand this error:
INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
at durable_queue$checksum.invokeStatic (durable_queue.clj:64)
durable_queue$checksum.invokePrim (durable_queue.clj:-1)
durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314)
durable_queue$queues$reify__6549$slab_BANG___6570.invoke
(durable_queue.clj:702)
durable_queue$queues$reify__6549$fn__6575.invoke
(durable_queue.clj:719)
durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717)
durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732)
humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke
(core.clj:665)
humongorous_nlp.core$cycle_to_database$fn__13673.invoke
(core.clj:663)
clojure.lang.AFn.run (AFn.java:22)
java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:511)
java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301
(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
java.lang.Thread.run (Thread.java:748)
On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5,
lawrence...@gmail.com wrote:
>
> I was away from Clojure for a year and I missed it. I am pleased to be
> back. But I've forgotten certain common errors. I feel like this is
> something I used to know but now I've lost the knowledge.
>
> I'm using Factual's durable-queue to put a step inbetween the import
> of large JSON files, and their writes to the database. This works fine on
> my local MacBook Pro, but when I move to an EC2 instance, I'm instead
> getting time-outs when durable-queue tries to read from the queue.
>
> At start up the app creates a few of these workers, which run for as
> long as the app is running:
>
> (defn worker
> [from-topics-to-persistence-queue current-database-connection]
> (slingshot/try+
>(loop [message (durable/take! from-topics-to-persistence-queue
> :message 6 :timed-out!)]
> (slingshot/try+
> (log-seq " the message in the work function " message)
> (when (= (type message) durable_queue.Task)
> (advance message current-database-connection))
> (catch Object o
> (log-seq "error in worker function")
> (durable/retry! message)
> (log-seq o)))
> (recur (durable/take! from-topics-to-persistence-queue :message
> 6 :timed-out!)))
>(catch Object o
> (error o)
> (slingshot/throw+ {
> :type worker
> :error o
> :from-topics-to-persistence-queue
> from-topics-to-persistence-queue
>
Re: using durable-queue, works locally, get :time-out when moving to an EC2
Are you using aot?
We have used that durable queue with 1.8. In fact, we have a compatibility
later that allows you to move from the durable queue to an Aws queue:
(mostly undocumented)
https://github.com/techascent/tech.queue
Is the problem possibly a difference between your compilation environment
and your deploy env?
On Wed, Jan 16, 2019, 1:29 PM So, I upgraded to Java 11, and now everything works. So I guess this was a
> version conflict.
>
> Just curious, but is there a way for Factual to make durable-queue to tell
> Leiningen that Java 11 is necessary?
>
>
>
> On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com
> wrote:
>>
>> On the new EC2 instance, running Ubuntu:
>>
>> java -version
>> openjdk version "1.8.0_191"
>> OpenJDK Runtime Environment (build
>> 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
>> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>>
>> Is it possible this version does not have the checksum signature that
>> durable-queue is looking for?
>>
>> I'm thinking this is some kind of version problem.
>>
>>
>>
>>
>>
>>
>> On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com
>> wrote:
>>>
>>> Sorry, I'm an idiot. The real error was when I called put!
>>>
>>> I don't understand this error:
>>>
>>> INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
>>> java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
>>> at durable_queue$checksum.invokeStatic (durable_queue.clj:64)
>>> durable_queue$checksum.invokePrim (durable_queue.clj:-1)
>>> durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314)
>>> durable_queue$queues$reify__6549$slab_BANG___6570.invoke
>>> (durable_queue.clj:702)
>>> durable_queue$queues$reify__6549$fn__6575.invoke
>>> (durable_queue.clj:719)
>>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717)
>>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732)
>>> humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke
>>> (core.clj:665)
>>> humongorous_nlp.core$cycle_to_database$fn__13673.invoke
>>> (core.clj:663)
>>> clojure.lang.AFn.run (AFn.java:22)
>>> java.util.concurrent.Executors$RunnableAdapter.call
>>> (Executors.java:511)
>>> java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308)
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301
>>> (ScheduledThreadPoolExecutor.java:180)
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
>>> (ScheduledThreadPoolExecutor.java:294)
>>> java.util.concurrent.ThreadPoolExecutor.runWorker
>>> (ThreadPoolExecutor.java:1149)
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run
>>> (ThreadPoolExecutor.java:624)
>>> java.lang.Thread.run (Thread.java:748)
>>>
>>>
>>>
>>>
>>> On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5,
>>> lawrence...@gmail.com wrote:
I was away from Clojure for a year and I missed it. I am pleased to be
back. But I've forgotten certain common errors. I feel like this is
something I used to know but now I've lost the knowledge.
I'm using Factual's durable-queue to put a step inbetween the import of
large JSON files, and their writes to the database. This works fine on my
local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
time-outs when durable-queue tries to read from the queue.
At start up the app creates a few of these workers, which run for as
long as the app is running:
(defn worker
[from-topics-to-persistence-queue current-database-connection]
(slingshot/try+
(loop [message (durable/take! from-topics-to-persistence-queue
:message 6 :timed-out!)]
(slingshot/try+
(log-seq " the message in the work function " message)
(when (= (type message) durable_queue.Task)
(advance message current-database-connection))
(catch Object o
(log-seq "error in worker function")
(durable/retry! message)
(log-seq o)))
(recur (durable/take! from-topics-to-persistence-queue :message
6 :timed-out!)))
(catch Object o
(error o)
(slingshot/throw+ {
:type worker
:error o
:from-topics-to-persistence-queue
from-topics-to-persistence-queue
:current-database-connection
current-database-connection
}
On the EC2 instance, I see in the output:
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO: the message in the work function
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO:
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO: :timed-out!
I'm using an environme
Re: using durable-queue, works locally, get :time-out when moving to an EC2
So, I upgraded to Java 11, and now everything works. So I guess this was a
version conflict.
Just curious, but is there a way for Factual to make durable-queue to tell
Leiningen that Java 11 is necessary?
On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com
wrote:
>
> On the new EC2 instance, running Ubuntu:
>
> java -version
> openjdk version "1.8.0_191"
> OpenJDK Runtime Environment (build
> 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>
> Is it possible this version does not have the checksum signature that
> durable-queue is looking for?
>
> I'm thinking this is some kind of version problem.
>
>
>
>
>
>
> On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com
> wrote:
>>
>> Sorry, I'm an idiot. The real error was when I called put!
>>
>> I don't understand this error:
>>
>> INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
>> java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
>> at durable_queue$checksum.invokeStatic (durable_queue.clj:64)
>> durable_queue$checksum.invokePrim (durable_queue.clj:-1)
>> durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314)
>> durable_queue$queues$reify__6549$slab_BANG___6570.invoke
>> (durable_queue.clj:702)
>> durable_queue$queues$reify__6549$fn__6575.invoke
>> (durable_queue.clj:719)
>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717)
>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732)
>> humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke
>> (core.clj:665)
>> humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663)
>> clojure.lang.AFn.run (AFn.java:22)
>> java.util.concurrent.Executors$RunnableAdapter.call
>> (Executors.java:511)
>> java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308)
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301
>>
>> (ScheduledThreadPoolExecutor.java:180)
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
>> (ScheduledThreadPoolExecutor.java:294)
>> java.util.concurrent.ThreadPoolExecutor.runWorker
>> (ThreadPoolExecutor.java:1149)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:624)
>> java.lang.Thread.run (Thread.java:748)
>>
>>
>>
>>
>> On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com
>> wrote:
>>>
>>> I was away from Clojure for a year and I missed it. I am pleased to be
>>> back. But I've forgotten certain common errors. I feel like this is
>>> something I used to know but now I've lost the knowledge.
>>>
>>> I'm using Factual's durable-queue to put a step inbetween the import of
>>> large JSON files, and their writes to the database. This works fine on my
>>> local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
>>> time-outs when durable-queue tries to read from the queue.
>>>
>>> At start up the app creates a few of these workers, which run for as
>>> long as the app is running:
>>>
>>> (defn worker
>>> [from-topics-to-persistence-queue current-database-connection]
>>> (slingshot/try+
>>>(loop [message (durable/take! from-topics-to-persistence-queue
>>> :message 6 :timed-out!)]
>>> (slingshot/try+
>>> (log-seq " the message in the work function " message)
>>> (when (= (type message) durable_queue.Task)
>>> (advance message current-database-connection))
>>> (catch Object o
>>> (log-seq "error in worker function")
>>> (durable/retry! message)
>>> (log-seq o)))
>>> (recur (durable/take! from-topics-to-persistence-queue :message
>>> 6 :timed-out!)))
>>>(catch Object o
>>> (error o)
>>> (slingshot/throw+ {
>>> :type worker
>>> :error o
>>> :from-topics-to-persistence-queue
>>> from-topics-to-persistence-queue
>>> :current-database-connection
>>> current-database-connection
>>> }
>>>
>>>
>>> On the EC2 instance, I see in the output:
>>>
>>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
>>> INFO: the message in the work function
>>>
>>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
>>> INFO:
>>>
>>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
>>> INFO: :timed-out!
>>>
>>> I'm using an environment var to set the path to the queue:
>>>
>>> PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue
>>>
>>> I feel like I ran into this problem 2 years ago, but I can't recall the
>>> solution. Is the problem specific to AWS?
>>>
>>> An important clue, I think, is that I'm not getting an error on the
>>> put!, only on the take! Why would that be?
>>>
>>>
>>> --- lawrence
>>>
>>>
>>>
>>>
--
You received this messag
Re: using durable-queue, works locally, get :time-out when moving to an EC2
On the new EC2 instance, running Ubuntu:
java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
Is it possible this version does not have the checksum signature that
durable-queue is looking for?
I'm thinking this is some kind of version problem.
On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com
wrote:
>
> Sorry, I'm an idiot. The real error was when I called put!
>
> I don't understand this error:
>
> INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
> java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
> at durable_queue$checksum.invokeStatic (durable_queue.clj:64)
> durable_queue$checksum.invokePrim (durable_queue.clj:-1)
> durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314)
> durable_queue$queues$reify__6549$slab_BANG___6570.invoke
> (durable_queue.clj:702)
> durable_queue$queues$reify__6549$fn__6575.invoke
> (durable_queue.clj:719)
> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717)
> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732)
> humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke
> (core.clj:665)
> humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663)
> clojure.lang.AFn.run (AFn.java:22)
> java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301
>
> (ScheduledThreadPoolExecutor.java:180)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
> (ScheduledThreadPoolExecutor.java:294)
> java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> java.lang.Thread.run (Thread.java:748)
>
>
>
>
> On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com
> wrote:
>>
>> I was away from Clojure for a year and I missed it. I am pleased to be
>> back. But I've forgotten certain common errors. I feel like this is
>> something I used to know but now I've lost the knowledge.
>>
>> I'm using Factual's durable-queue to put a step inbetween the import of
>> large JSON files, and their writes to the database. This works fine on my
>> local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
>> time-outs when durable-queue tries to read from the queue.
>>
>> At start up the app creates a few of these workers, which run for as long
>> as the app is running:
>>
>> (defn worker
>> [from-topics-to-persistence-queue current-database-connection]
>> (slingshot/try+
>>(loop [message (durable/take! from-topics-to-persistence-queue
>> :message 6 :timed-out!)]
>> (slingshot/try+
>> (log-seq " the message in the work function " message)
>> (when (= (type message) durable_queue.Task)
>> (advance message current-database-connection))
>> (catch Object o
>> (log-seq "error in worker function")
>> (durable/retry! message)
>> (log-seq o)))
>> (recur (durable/take! from-topics-to-persistence-queue :message
>> 6 :timed-out!)))
>>(catch Object o
>> (error o)
>> (slingshot/throw+ {
>> :type worker
>> :error o
>> :from-topics-to-persistence-queue
>> from-topics-to-persistence-queue
>> :current-database-connection
>> current-database-connection
>> }
>>
>>
>> On the EC2 instance, I see in the output:
>>
>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
>> INFO: the message in the work function
>>
>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
>> INFO:
>>
>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
>> INFO: :timed-out!
>>
>> I'm using an environment var to set the path to the queue:
>>
>> PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue
>>
>> I feel like I ran into this problem 2 years ago, but I can't recall the
>> solution. Is the problem specific to AWS?
>>
>> An important clue, I think, is that I'm not getting an error on the put!,
>> only on the take! Why would that be?
>>
>>
>> --- lawrence
>>
>>
>>
>>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subsc
Re: using durable-queue, works locally, get :time-out when moving to an EC2
Sorry, I'm an idiot. The real error was when I called put!
I don't understand this error:
INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V
at durable_queue$checksum.invokeStatic (durable_queue.clj:64)
durable_queue$checksum.invokePrim (durable_queue.clj:-1)
durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314)
durable_queue$queues$reify__6549$slab_BANG___6570.invoke
(durable_queue.clj:702)
durable_queue$queues$reify__6549$fn__6575.invoke (durable_queue.clj:719)
durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717)
durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732)
humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke
(core.clj:665)
humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663)
clojure.lang.AFn.run (AFn.java:22)
java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301
(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:294)
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
java.lang.Thread.run (Thread.java:748)
On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com
wrote:
>
> I was away from Clojure for a year and I missed it. I am pleased to be
> back. But I've forgotten certain common errors. I feel like this is
> something I used to know but now I've lost the knowledge.
>
> I'm using Factual's durable-queue to put a step inbetween the import of
> large JSON files, and their writes to the database. This works fine on my
> local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
> time-outs when durable-queue tries to read from the queue.
>
> At start up the app creates a few of these workers, which run for as long
> as the app is running:
>
> (defn worker
> [from-topics-to-persistence-queue current-database-connection]
> (slingshot/try+
>(loop [message (durable/take! from-topics-to-persistence-queue :message
> 6 :timed-out!)]
> (slingshot/try+
> (log-seq " the message in the work function " message)
> (when (= (type message) durable_queue.Task)
> (advance message current-database-connection))
> (catch Object o
> (log-seq "error in worker function")
> (durable/retry! message)
> (log-seq o)))
> (recur (durable/take! from-topics-to-persistence-queue :message 6
> :timed-out!)))
>(catch Object o
> (error o)
> (slingshot/throw+ {
> :type worker
> :error o
> :from-topics-to-persistence-queue
> from-topics-to-persistence-queue
> :current-database-connection
> current-database-connection
> }
>
>
> On the EC2 instance, I see in the output:
>
> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
> INFO: the message in the work function
>
> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
> INFO:
>
> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
> INFO: :timed-out!
>
> I'm using an environment var to set the path to the queue:
>
> PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue
>
> I feel like I ran into this problem 2 years ago, but I can't recall the
> solution. Is the problem specific to AWS?
>
> An important clue, I think, is that I'm not getting an error on the put!,
> only on the take! Why would that be?
>
>
> --- lawrence
>
>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
using durable-queue, works locally, get :time-out when moving to an EC2
I was away from Clojure for a year and I missed it. I am pleased to be
back. But I've forgotten certain common errors. I feel like this is
something I used to know but now I've lost the knowledge.
I'm using Factual's durable-queue to put a step inbetween the import of
large JSON files, and their writes to the database. This works fine on my
local MacBook Pro, but when I move to an EC2 instance, I'm instead getting
time-outs when durable-queue tries to read from the queue.
At start up the app creates a few of these workers, which run for as long
as the app is running:
(defn worker
[from-topics-to-persistence-queue current-database-connection]
(slingshot/try+
(loop [message (durable/take! from-topics-to-persistence-queue :message
6 :timed-out!)]
(slingshot/try+
(log-seq " the message in the work function " message)
(when (= (type message) durable_queue.Task)
(advance message current-database-connection))
(catch Object o
(log-seq "error in worker function")
(durable/retry! message)
(log-seq o)))
(recur (durable/take! from-topics-to-persistence-queue :message 6
:timed-out!)))
(catch Object o
(error o)
(slingshot/throw+ {
:type worker
:error o
:from-topics-to-persistence-queue
from-topics-to-persistence-queue
:current-database-connection
current-database-connection
}
On the EC2 instance, I see in the output:
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO: the message in the work function
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO:
Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke
INFO: :timed-out!
I'm using an environment var to set the path to the queue:
PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue
I feel like I ran into this problem 2 years ago, but I can't recall the
solution. Is the problem specific to AWS?
An important clue, I think, is that I'm not getting an error on the put!,
only on the take! Why would that be?
--- lawrence
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
