[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 I also went ahead and added a bit better logging if the CME returns, we should get a sense of what the other threads are doing. Definitely would like to see if you see any CMEs with the current commit. If so, it'd be great to check the error log, the full thread dump for that process should be there at the point of the CME. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 @dlyle65535 any chance you can run that again with this commit? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 Just for those poor souls who are debugging this on the internet, this is the sister exception (from the other thread): > 2016-06-10 13:34:44.687 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.util.ConcurrentModificationException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.daemon.executor$fn__5492$fn__5505$fn__5556.invoke(executor.clj:813) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.util$async_loop$fn__644.invoke(util.clj:479) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51] Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) ~[?:1.8.0_51] at java.util.HashMap$EntryIterator.next(HashMap.java:1463) ~[?:1.8.0_51] at java.util.HashMap$EntryIterator.next(HashMap.java:1461) ~[?:1.8.0_51] at java.util.HashMap.putMapEntries(HashMap.java:511) ~[?:1.8.0_51] at java.util.HashMap.clone(HashMap.java:1324) ~[?:1.8.0_51] at org.apache.metron.enrichment.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:66) ~[stormjar.jar:?] at backtype.storm.daemon.executor$fn__5492$tuple_action_fn__5494.invoke(executor.clj:684) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.daemon.executor$mk_task_receiver$fn__5415.invoke(executor.clj:431) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.disruptor$clojure_handler$reify__4991.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] ... 6 more --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 So, that's the other side of the concurrent modification exception, @dlyle65535. Somehow we are modifying the JSONObject message while it's deserializing, like in https://groups.google.com/forum/#!topic/storm-user/67ycy_WLqEQ except while deserializing, not serializing. This exception is the CME from the deserialization code's view. The trick here is that I'm not sure how it's deserializing while we're using it. I'll have to consider this more. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/149 Overall ran well. Increasing the indexWriter executors to 2 didn't create the ConcurrentModificationException. Increasing the kafkaSpout executors to 2 gave me the following exception, but the topology recovered. I'm +1 on the patch, but I've left my test rig up in case you wish to do some additional debugging. Trace: 2016-06-10 22:23:36.438 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: java.io.OptionalDataException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.daemon.executor$fn__5495$fn__5508$fn__5559.invoke(executor.clj:808) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] Caused by: java.lang.RuntimeException: java.io.OptionalDataException at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:?] at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] ... 6 more Caused by: java.io.OptionalDataException at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1371) ~[?:1.8.0_40] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[?:1.8.0_40] at java.util.HashMap.readObject(HashMap.java:1394) ~[?:1.8.0_40] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_40] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_40] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40] at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40] at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) ~[?:1.8.0_40] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) ~[?:1.8.0_40] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) ~[?:1.8.0_40] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) ~[?:1.8.0_40] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[?:1.8.0_40] at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:?] at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4] at backtype.stor
[GitHub] incubator-metron pull request #147: METRON-208 Add Google Analytics tags to ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/147 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/149 +1 from me pending EC2 test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/149 + 1from me. great job --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #147: METRON-208 Add Google Analytics tags to Metron ...
Github user iraghumitra commented on the issue: https://github.com/apache/incubator-metron/pull/147 Corrected Ryan's apache id --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #147: METRON-208 Add Google Analytics tags to Metron ...
Github user iraghumitra commented on the issue: https://github.com/apache/incubator-metron/pull/147 Corrected apache id of Ryan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #147: METRON-208 Add Google Analytics tags to Metron ...
Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/147 My apache id is incorrect. After @iraghumitra makes the change +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #149: METRON-217: Found a grabbag of bugs
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/149 METRON-217: Found a grabbag of bugs Doing some scale testing, I found a grabbag of bugs: * The elasticsearch writer should allow multiple ES hosts to be passed in via a List. This should be backwards compatible. * There is a concurrent modification exception while cloning the message field * The parser topology numSpoutTasks and numParserTasks are swapped * The grok parser should throw an exception if it cannot parse so the message that could not be parsed is sent to the error queue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron bug_squash_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #149 commit 3738df5112bf7514fb1a9e9b6c7d812ce97a8a69 Author: cstella Date: 2016-06-10T19:17:57Z Fixing a grabbag of bugs. commit 08609a91483683c241e7f3e706630f79e25bffcb Author: cstella Date: 2016-06-10T19:28:11Z Updating writer bolt. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
