One more thing, just for posterity here, it always freezes at 6 records
written to HDFS.  That's the reason I thought it was a flushing issue.

On Mon, Jan 23, 2017 at 3:38 PM, Casey Stella <[email protected]> wrote:

> Ok, so now I'm concerned that this isn't a fluke.  Here's an excerpt from
> the failing logs on travis for my PR with substantially longer timeouts (
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/194575474/log.txt)
>
> Running org.apache.metron.solr.integration.SolrIndexingIntegrationTest
> 0 vs 10 vs 0
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 317.056 sec 
> <<< FAILURE!
> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)  Time 
> elapsed: 316.949 sec  <<< ERROR!
> java.lang.RuntimeException: Took too long to complete: 300783 > 300000
>       at 
> org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:131)
>       at 
> org.apache.metron.indexing.integration.IndexingIntegrationTest.test(IndexingIntegrationTest.java:173)
>       
>
> I'm getting the impression that this isn't the timeout and we have a mystery 
> on our hands.  Each of those lines "10 vs 10 vs 6" happen 15 seconds apart.  
> That line means that it read 10 entries from kafka, 10 entries from the 
> indexed data and 6 entries from HDFS.  It's that 6 entries that is the 
> problem.   Also of note, this does not seem to happen to me locally AND it's 
> not consistent on Travis.  Given all that I'd say that it's a problem with 
> the HDFS Writer not getting flushed, but I verified that it is indeed flushed 
> per message.
>
>
> Anyway, tl;dr we have a mystery unit test bug that isn't deterministic wrt 
> the unit tests and may or may not manifest itself outside of the unit tests.  
> So, yeah, I'll be looking at it, but would appreciate others taking a gander 
> too.
>
>
> Casey
>
>
> On Mon, Jan 23, 2017 at 2:09 PM, Casey Stella <[email protected]> wrote:
>
>> Yeah, I adjusted the timeout on the indexing integration tests as part of
>> https://github.com/apache/incubator-metron/pull/420 which I'll merge in
>> today.
>>
>> On Mon, Jan 23, 2017 at 2:01 PM, [email protected] <[email protected]>
>> wrote:
>>
>>> Okay, now we have back to back failures, and it looks like it may have
>>> been
>>> a timeout issue?
>>>  `test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest):
>>> Took too long to complete: 150582 > 150000`, more details below:
>>>
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>>> 166.167 sec <<< FAILURE!
>>>
>>> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
>>> Time elapsed: 166.071 sec  <<< ERROR!
>>>
>>> java.lang.RuntimeException: Took too long to complete: 150582 > 150000
>>>
>>>         at org.apache.metron.integration.ComponentRunner.process(Compon
>>> entRunner.java:131)
>>>
>>>         at org.apache.metron.indexing.integration.IndexingIntegrationTe
>>> st.test(IndexingIntegrationTest.java:173)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>>
>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:483)
>>>
>>>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
>>> FrameworkMethod.java:50)
>>>
>>>         at org.junit.internal.runners.model.ReflectiveCallable.run(Refl
>>> ectiveCallable.java:12)
>>>
>>>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(Fr
>>> ameworkMethod.java:47)
>>>
>>>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(
>>> InvokeMethod.java:17)
>>>
>>>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>>>
>>>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit
>>> 4ClassRunner.java:78)
>>>
>>>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit
>>> 4ClassRunner.java:57)
>>>
>>>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>>>
>>>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:
>>> 71)
>>>
>>>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java
>>> :288)
>>>
>>>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:
>>> 58)
>>>
>>>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:
>>> 268)
>>>
>>>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>>>
>>>         at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUni
>>> t4Provider.java:252)
>>>
>>>         at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestS
>>> et(JUnit4Provider.java:141)
>>>
>>>         at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit
>>> 4Provider.java:112)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>>
>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:483)
>>>
>>>         at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodW
>>> ithArray(ReflectionUtils.java:189)
>>>
>>>         at org.apache.maven.surefire.booter.ProviderFactory$ProviderPro
>>> xy.invoke(ProviderFactory.java:165)
>>>
>>>         at org.apache.maven.surefire.booter.ProviderFactory.invokeProvi
>>> der(ProviderFactory.java:85)
>>>
>>>         at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInPro
>>> cess(ForkedBooter.java:115)
>>>
>>>         at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBoo
>>> ter.java:75)
>>>
>>>
>>> Jon
>>>
>>> On Thu, Jan 19, 2017 at 2:49 PM [email protected] <[email protected]>
>>> wrote:
>>>
>>> > The build has been showing as failing
>>> > <https://github.com/apache/incubator-metron> for a little while now.
>>> I
>>> > know we recently updated the language around Merge Requirements
>>> > <https://cwiki.apache.org/confluence/pages/viewpage.action?p
>>> ageId=61332235>,
>>> > but if I recall properly our current issue is simply a Travis CI
>>> overload
>>> > issue.  Is there a way we can update the wiki doc to account for
>>> situations
>>> > like this?
>>> >
>>> > Jon
>>> > --
>>> >
>>> > Jon
>>> >
>>> > Sent from my mobile device
>>> >
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>>
>>
>

Reply via email to