Re: using durable-queue, works locally, get :time-out when moving to an EC2
> Is the problem possibly a difference between your > compilation environment and your deploy env? This seems to be proven by the fact that upgrading Java on the EC2 instance fixed the problem. On Wednesday, January 16, 2019 at 3:49:41 PM UTC-5, Chris Nuernberger wrote: > > Are you using aot? > > We have used that durable queue with 1.8. In fact, we have a > compatibility later that allows you to move from the durable queue to an > Aws queue: > (mostly undocumented) > > https://github.com/techascent/tech.queue > > Is the problem possibly a difference between your compilation environment > and your deploy env? > > > > On Wed, Jan 16, 2019, 1:29 PM wrote: > >> So, I upgraded to Java 11, and now everything works. So I guess this was >> a version conflict. >> >> Just curious, but is there a way for Factual to make durable-queue to >> tell Leiningen that Java 11 is necessary? >> >> >> >> On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com >> wrote: >>> >>> On the new EC2 instance, running Ubuntu: >>> >>> java -version >>> openjdk version "1.8.0_191" >>> OpenJDK Runtime Environment (build >>> 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12) >>> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) >>> >>> Is it possible this version does not have the checksum signature that >>> durable-queue is looking for? >>> >>> I'm thinking this is some kind of version problem. >>> >>> >>> >>> >>> >>> >>> On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, >>> lawrence...@gmail.com wrote: Sorry, I'm an idiot. The real error was when I called put! I don't understand this error: INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V at durable_queue$checksum.invokeStatic (durable_queue.clj:64) durable_queue$checksum.invokePrim (durable_queue.clj:-1) durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314) durable_queue$queues$reify__6549$slab_BANG___6570.invoke (durable_queue.clj:702) durable_queue$queues$reify__6549$fn__6575.invoke (durable_queue.clj:719) durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717) durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732) humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke (core.clj:665) humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663) clojure.lang.AFn.run (AFn.java:22) java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511) java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301 (ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) java.lang.Thread.run (Thread.java:748) On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com wrote: > > I was away from Clojure for a year and I missed it. I am pleased to be > back. But I've forgotten certain common errors. I feel like this is > something I used to know but now I've lost the knowledge. > > I'm using Factual's durable-queue to put a step inbetween the import > of large JSON files, and their writes to the database. This works fine on > my local MacBook Pro, but when I move to an EC2 instance, I'm instead > getting time-outs when durable-queue tries to read from the queue. > > At start up the app creates a few of these workers, which run for as > long as the app is running: > > (defn worker > [from-topics-to-persistence-queue current-database-connection] > (slingshot/try+ >(loop [message (durable/take! from-topics-to-persistence-queue > :message 6 :timed-out!)] > (slingshot/try+ > (log-seq " the message in the work function " message) > (when (= (type message) durable_queue.Task) > (advance message current-database-connection)) > (catch Object o > (log-seq "error in worker function") > (durable/retry! message) > (log-seq o))) > (recur (durable/take! from-topics-to-persistence-queue :message > 6 :timed-out!))) >(catch Object o > (error o) > (slingshot/throw+ { > :type worker > :error o > :from-topics-to-persistence-queue > from-topics-to-persistence-queue >
Re: using durable-queue, works locally, get :time-out when moving to an EC2
Are you using aot? We have used that durable queue with 1.8. In fact, we have a compatibility later that allows you to move from the durable queue to an Aws queue: (mostly undocumented) https://github.com/techascent/tech.queue Is the problem possibly a difference between your compilation environment and your deploy env? On Wed, Jan 16, 2019, 1:29 PM So, I upgraded to Java 11, and now everything works. So I guess this was a > version conflict. > > Just curious, but is there a way for Factual to make durable-queue to tell > Leiningen that Java 11 is necessary? > > > > On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com > wrote: >> >> On the new EC2 instance, running Ubuntu: >> >> java -version >> openjdk version "1.8.0_191" >> OpenJDK Runtime Environment (build >> 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12) >> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) >> >> Is it possible this version does not have the checksum signature that >> durable-queue is looking for? >> >> I'm thinking this is some kind of version problem. >> >> >> >> >> >> >> On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com >> wrote: >>> >>> Sorry, I'm an idiot. The real error was when I called put! >>> >>> I don't understand this error: >>> >>> INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V >>> java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V >>> at durable_queue$checksum.invokeStatic (durable_queue.clj:64) >>> durable_queue$checksum.invokePrim (durable_queue.clj:-1) >>> durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314) >>> durable_queue$queues$reify__6549$slab_BANG___6570.invoke >>> (durable_queue.clj:702) >>> durable_queue$queues$reify__6549$fn__6575.invoke >>> (durable_queue.clj:719) >>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717) >>> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732) >>> humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke >>> (core.clj:665) >>> humongorous_nlp.core$cycle_to_database$fn__13673.invoke >>> (core.clj:663) >>> clojure.lang.AFn.run (AFn.java:22) >>> java.util.concurrent.Executors$RunnableAdapter.call >>> (Executors.java:511) >>> java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308) >>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301 >>> (ScheduledThreadPoolExecutor.java:180) >>> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run >>> (ScheduledThreadPoolExecutor.java:294) >>> java.util.concurrent.ThreadPoolExecutor.runWorker >>> (ThreadPoolExecutor.java:1149) >>> java.util.concurrent.ThreadPoolExecutor$Worker.run >>> (ThreadPoolExecutor.java:624) >>> java.lang.Thread.run (Thread.java:748) >>> >>> >>> >>> >>> On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, >>> lawrence...@gmail.com wrote: I was away from Clojure for a year and I missed it. I am pleased to be back. But I've forgotten certain common errors. I feel like this is something I used to know but now I've lost the knowledge. I'm using Factual's durable-queue to put a step inbetween the import of large JSON files, and their writes to the database. This works fine on my local MacBook Pro, but when I move to an EC2 instance, I'm instead getting time-outs when durable-queue tries to read from the queue. At start up the app creates a few of these workers, which run for as long as the app is running: (defn worker [from-topics-to-persistence-queue current-database-connection] (slingshot/try+ (loop [message (durable/take! from-topics-to-persistence-queue :message 6 :timed-out!)] (slingshot/try+ (log-seq " the message in the work function " message) (when (= (type message) durable_queue.Task) (advance message current-database-connection)) (catch Object o (log-seq "error in worker function") (durable/retry! message) (log-seq o))) (recur (durable/take! from-topics-to-persistence-queue :message 6 :timed-out!))) (catch Object o (error o) (slingshot/throw+ { :type worker :error o :from-topics-to-persistence-queue from-topics-to-persistence-queue :current-database-connection current-database-connection } On the EC2 instance, I see in the output: Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke INFO: the message in the work function Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke INFO: Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke INFO: :timed-out! I'm using an environme
Re: using durable-queue, works locally, get :time-out when moving to an EC2
So, I upgraded to Java 11, and now everything works. So I guess this was a version conflict. Just curious, but is there a way for Factual to make durable-queue to tell Leiningen that Java 11 is necessary? On Wednesday, January 16, 2019 at 3:17:49 PM UTC-5, lawrence...@gmail.com wrote: > > On the new EC2 instance, running Ubuntu: > > java -version > openjdk version "1.8.0_191" > OpenJDK Runtime Environment (build > 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12) > OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) > > Is it possible this version does not have the checksum signature that > durable-queue is looking for? > > I'm thinking this is some kind of version problem. > > > > > > > On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com > wrote: >> >> Sorry, I'm an idiot. The real error was when I called put! >> >> I don't understand this error: >> >> INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V >> java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V >> at durable_queue$checksum.invokeStatic (durable_queue.clj:64) >> durable_queue$checksum.invokePrim (durable_queue.clj:-1) >> durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314) >> durable_queue$queues$reify__6549$slab_BANG___6570.invoke >> (durable_queue.clj:702) >> durable_queue$queues$reify__6549$fn__6575.invoke >> (durable_queue.clj:719) >> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717) >> durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732) >> humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke >> (core.clj:665) >> humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663) >> clojure.lang.AFn.run (AFn.java:22) >> java.util.concurrent.Executors$RunnableAdapter.call >> (Executors.java:511) >> java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308) >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301 >> >> (ScheduledThreadPoolExecutor.java:180) >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run >> (ScheduledThreadPoolExecutor.java:294) >> java.util.concurrent.ThreadPoolExecutor.runWorker >> (ThreadPoolExecutor.java:1149) >> java.util.concurrent.ThreadPoolExecutor$Worker.run >> (ThreadPoolExecutor.java:624) >> java.lang.Thread.run (Thread.java:748) >> >> >> >> >> On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com >> wrote: >>> >>> I was away from Clojure for a year and I missed it. I am pleased to be >>> back. But I've forgotten certain common errors. I feel like this is >>> something I used to know but now I've lost the knowledge. >>> >>> I'm using Factual's durable-queue to put a step inbetween the import of >>> large JSON files, and their writes to the database. This works fine on my >>> local MacBook Pro, but when I move to an EC2 instance, I'm instead getting >>> time-outs when durable-queue tries to read from the queue. >>> >>> At start up the app creates a few of these workers, which run for as >>> long as the app is running: >>> >>> (defn worker >>> [from-topics-to-persistence-queue current-database-connection] >>> (slingshot/try+ >>>(loop [message (durable/take! from-topics-to-persistence-queue >>> :message 6 :timed-out!)] >>> (slingshot/try+ >>> (log-seq " the message in the work function " message) >>> (when (= (type message) durable_queue.Task) >>> (advance message current-database-connection)) >>> (catch Object o >>> (log-seq "error in worker function") >>> (durable/retry! message) >>> (log-seq o))) >>> (recur (durable/take! from-topics-to-persistence-queue :message >>> 6 :timed-out!))) >>>(catch Object o >>> (error o) >>> (slingshot/throw+ { >>> :type worker >>> :error o >>> :from-topics-to-persistence-queue >>> from-topics-to-persistence-queue >>> :current-database-connection >>> current-database-connection >>> } >>> >>> >>> On the EC2 instance, I see in the output: >>> >>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke >>> INFO: the message in the work function >>> >>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke >>> INFO: >>> >>> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke >>> INFO: :timed-out! >>> >>> I'm using an environment var to set the path to the queue: >>> >>> PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue >>> >>> I feel like I ran into this problem 2 years ago, but I can't recall the >>> solution. Is the problem specific to AWS? >>> >>> An important clue, I think, is that I'm not getting an error on the >>> put!, only on the take! Why would that be? >>> >>> >>> --- lawrence >>> >>> >>> >>> -- You received this messag
Re: using durable-queue, works locally, get :time-out when moving to an EC2
On the new EC2 instance, running Ubuntu: java -version openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) Is it possible this version does not have the checksum signature that durable-queue is looking for? I'm thinking this is some kind of version problem. On Wednesday, January 16, 2019 at 2:50:14 PM UTC-5, lawrence...@gmail.com wrote: > > Sorry, I'm an idiot. The real error was when I called put! > > I don't understand this error: > > INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V > java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V > at durable_queue$checksum.invokeStatic (durable_queue.clj:64) > durable_queue$checksum.invokePrim (durable_queue.clj:-1) > durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314) > durable_queue$queues$reify__6549$slab_BANG___6570.invoke > (durable_queue.clj:702) > durable_queue$queues$reify__6549$fn__6575.invoke > (durable_queue.clj:719) > durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717) > durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732) > humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke > (core.clj:665) > humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663) > clojure.lang.AFn.run (AFn.java:22) > java.util.concurrent.Executors$RunnableAdapter.call > (Executors.java:511) > java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308) > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301 > > (ScheduledThreadPoolExecutor.java:180) > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run > (ScheduledThreadPoolExecutor.java:294) > java.util.concurrent.ThreadPoolExecutor.runWorker > (ThreadPoolExecutor.java:1149) > java.util.concurrent.ThreadPoolExecutor$Worker.run > (ThreadPoolExecutor.java:624) > java.lang.Thread.run (Thread.java:748) > > > > > On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com > wrote: >> >> I was away from Clojure for a year and I missed it. I am pleased to be >> back. But I've forgotten certain common errors. I feel like this is >> something I used to know but now I've lost the knowledge. >> >> I'm using Factual's durable-queue to put a step inbetween the import of >> large JSON files, and their writes to the database. This works fine on my >> local MacBook Pro, but when I move to an EC2 instance, I'm instead getting >> time-outs when durable-queue tries to read from the queue. >> >> At start up the app creates a few of these workers, which run for as long >> as the app is running: >> >> (defn worker >> [from-topics-to-persistence-queue current-database-connection] >> (slingshot/try+ >>(loop [message (durable/take! from-topics-to-persistence-queue >> :message 6 :timed-out!)] >> (slingshot/try+ >> (log-seq " the message in the work function " message) >> (when (= (type message) durable_queue.Task) >> (advance message current-database-connection)) >> (catch Object o >> (log-seq "error in worker function") >> (durable/retry! message) >> (log-seq o))) >> (recur (durable/take! from-topics-to-persistence-queue :message >> 6 :timed-out!))) >>(catch Object o >> (error o) >> (slingshot/throw+ { >> :type worker >> :error o >> :from-topics-to-persistence-queue >> from-topics-to-persistence-queue >> :current-database-connection >> current-database-connection >> } >> >> >> On the EC2 instance, I see in the output: >> >> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke >> INFO: the message in the work function >> >> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke >> INFO: >> >> Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke >> INFO: :timed-out! >> >> I'm using an environment var to set the path to the queue: >> >> PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue >> >> I feel like I ran into this problem 2 years ago, but I can't recall the >> solution. Is the problem specific to AWS? >> >> An important clue, I think, is that I'm not getting an error on the put!, >> only on the take! Why would that be? >> >> >> --- lawrence >> >> >> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subsc
Re: using durable-queue, works locally, get :time-out when moving to an EC2
Sorry, I'm an idiot. The real error was when I called put! I don't understand this error: INFO: java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V at durable_queue$checksum.invokeStatic (durable_queue.clj:64) durable_queue$checksum.invokePrim (durable_queue.clj:-1) durable_queue.TaskSlab.append_to_slab_BANG_ (durable_queue.clj:314) durable_queue$queues$reify__6549$slab_BANG___6570.invoke (durable_queue.clj:702) durable_queue$queues$reify__6549$fn__6575.invoke (durable_queue.clj:719) durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:717) durable_queue$queues$reify__6549.put_BANG_ (durable_queue.clj:732) humongorous_nlp.core$cycle_to_database$fn__13673$fn__13693.invoke (core.clj:665) humongorous_nlp.core$cycle_to_database$fn__13673.invoke (core.clj:663) clojure.lang.AFn.run (AFn.java:22) java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511) java.util.concurrent.FutureTask.runAndReset (FutureTask.java:308) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301 (ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:294) java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) java.lang.Thread.run (Thread.java:748) On Wednesday, January 16, 2019 at 1:47:21 PM UTC-5, lawrence...@gmail.com wrote: > > I was away from Clojure for a year and I missed it. I am pleased to be > back. But I've forgotten certain common errors. I feel like this is > something I used to know but now I've lost the knowledge. > > I'm using Factual's durable-queue to put a step inbetween the import of > large JSON files, and their writes to the database. This works fine on my > local MacBook Pro, but when I move to an EC2 instance, I'm instead getting > time-outs when durable-queue tries to read from the queue. > > At start up the app creates a few of these workers, which run for as long > as the app is running: > > (defn worker > [from-topics-to-persistence-queue current-database-connection] > (slingshot/try+ >(loop [message (durable/take! from-topics-to-persistence-queue :message > 6 :timed-out!)] > (slingshot/try+ > (log-seq " the message in the work function " message) > (when (= (type message) durable_queue.Task) > (advance message current-database-connection)) > (catch Object o > (log-seq "error in worker function") > (durable/retry! message) > (log-seq o))) > (recur (durable/take! from-topics-to-persistence-queue :message 6 > :timed-out!))) >(catch Object o > (error o) > (slingshot/throw+ { > :type worker > :error o > :from-topics-to-persistence-queue > from-topics-to-persistence-queue > :current-database-connection > current-database-connection > } > > > On the EC2 instance, I see in the output: > > Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke > INFO: the message in the work function > > Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke > INFO: > > Jan 16, 2019 6:43:23 PM humongorous-nlp.core invoke > INFO: :timed-out! > > I'm using an environment var to set the path to the queue: > > PATH_TO_DURABLE_QUEUE_S3_TO_DATABASE=/tmp/s3_to_database_queue > > I feel like I ran into this problem 2 years ago, but I can't recall the > solution. Is the problem specific to AWS? > > An important clue, I think, is that I'm not getting an error on the put!, > only on the take! Why would that be? > > > --- lawrence > > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.