[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516573#comment-16516573 ] David Smiley commented on SOLR-12386: - No; SOLR-12021 does not fix it; I've seen this error several times well after that was committed. As Hoss says in his first comment here: bq. I wonder if the root cause here is similar to what I found in that jira? "similar" (not same) > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516017#comment-16516017 ] Erick Erickson commented on SOLR-12386: --- [~dsmiley] Looking at BadApple'd tests, ConcurrentCreateRoutedAliasTest is AwaitsFix on this. This test hasn't failed in 4 weeks, WDYT about un-annotating it on the theory that it's fixed by SOLR-12021? > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489821#comment-16489821 ] David Smiley commented on SOLR-12386: - Interesting. I'm not sure how this would be an issue for the SolrCloud tests for configSet resources since they are in ZooKeeper. Or am I missing something? > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[
https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488172#comment-16488172
]
Hoss Man commented on SOLR-12386:
-
BTW...
{quote}IMHO, the file handle leak in the mentioned commit could habe been fixed
by just using try-with-resources around Class(Loader)#getResourceAsStream() ...
{quote}
...that would have solved that particular leak, but it would not have fixed the
error message returned in the event that the {{getResourceAsStream()}} _in that
line of code_ returned 'null' down the road because some _other_ place in the
code had a different resource leak ... we would still have the possibility of
the caller throwing confusing {{invalid API spec:
apispec/core.config.Commands.json}} exceptions because the method just returned
'null' and it couldn't distinguish "resource doesn't exist" from "OS wouldn't
let the JVM open that resource"
> Test fails for "Can't find resource" for files in the _default configset
>
>
> Key: SOLR-12386
> URL: https://issues.apache.org/jira/browse/SOLR-12386
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Priority: Minor
> Attachments: cant find resource, stacktrace.txt
>
>
> Some tests, especially ConcurrentCreateRoutedAliasTest, have failed
> sporadically failed with the message "Can't find resource" pertaining to a
> file that is in the default ConfigSet yet mysteriously can't be found. This
> happens when a collection is being created that ultimately fails for this
> reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[
https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488164#comment-16488164
]
Hoss Man commented on SOLR-12386:
-
bq. Unfortunately there is no Class#getResources(), it's only on classloader.
Why does that matter? Doesn't {{Class.getResourceAsStream()}} just call
{{Class.getClassLoader().getResourceAsStream()}} ? we could still replace it
with a helper utility like you're describing by passing
{{Class.getClassLoader()}}
bq. But we should fix the underlying issue (the leaks first), then think about
improving that situation.
My point before is that there may not actually *be* a leak -- it may very well
be that all streams are getting closed properly, but that some tests are
opening just enough resources that (depending on what other tests ran in the
same JVM and what classes got loaded) they are hitting the ulimit for open
files -- but instead of a clear error to that effect, we're getting "null" from
{{getResourceAsStream()}}
ie: i agree with you that if there is a file handle leak we should fix it, but
that is an indepenent possibility from the fact that we can/should "fix" the
code we have which opens Resources to better report/propogate when we hit
'IOException: Too many open files' under the covers so people aren't baffled
and confused by "Can't find resource" for files that definitely exist.
> Test fails for "Can't find resource" for files in the _default configset
>
>
> Key: SOLR-12386
> URL: https://issues.apache.org/jira/browse/SOLR-12386
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Priority: Minor
> Attachments: cant find resource, stacktrace.txt
>
>
> Some tests, especially ConcurrentCreateRoutedAliasTest, have failed
> sporadically failed with the message "Can't find resource" pertaining to a
> file that is in the default ConfigSet yet mysteriously can't be found. This
> happens when a collection is being created that ultimately fails for this
> reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488143#comment-16488143 ] Uwe Schindler commented on SOLR-12386: -- bq. And now that i look at it again: if we used Enumeration getResources(String name) instead of URL getResource(String) wouldn't that ensure we would get an IOException instead of "null" in the case you're talking about where even the lookup of the name failed because of too many filehandles? This could work if we create a static utility method taking Class or ClassLoader and a resource name, returning an InputStream. Unfortunately there is no Class#getResources(), it's only on classloader. We still have the problem outside of Solr's code that after running out of file handles, loading of resources fails - and sometimes fails with NPE! IMHO, the file handle leak in the mentioned commit could habe been fixed by just using try-with-resources around Class(Loader)#getResourceAsStream(). But we should fix the underlying issue (the leaks first), then think about improving that situation. > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[
https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488133#comment-16488133
]
Hoss Man commented on SOLR-12386:
-
bq. GetResource() may also return null ...
But at least then you can tell the difference between "getResource() returned
null meaning we didn't locate the file resource name" and "openConnection()
threw 'IOException: Too many open files'"
And now that i look at it again: if we used {{Enumeration
getResources(String name)}} instead of {{URL getResource(String)}} wouldn't
that ensure we would get an IOException instead of "null" in the case you're
talking about where even the lookup of the name failed because of too many
filehandles?
> Test fails for "Can't find resource" for files in the _default configset
>
>
> Key: SOLR-12386
> URL: https://issues.apache.org/jira/browse/SOLR-12386
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Priority: Minor
> Attachments: cant find resource, stacktrace.txt
>
>
> Some tests, especially ConcurrentCreateRoutedAliasTest, have failed
> sporadically failed with the message "Can't find resource" pertaining to a
> file that is in the default ConfigSet yet mysteriously can't be found. This
> happens when a collection is being created that ultimately fails for this
> reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488104#comment-16488104 ] Uwe Schindler commented on SOLR-12386: -- GetResource() may also return null, if the lookup of a file name does not work. Happens easily when you run out of file handles and the JAR file to be search in was closed by another call before. > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[
https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488095#comment-16488095
]
Hoss Man commented on SOLR-12386:
-
{quote}All other classloader methods are behaving the same way, so how to load
resources then?
{quote}
Not true – getResourceAsStream explicitly swallows any IOExceptions (which
might be thrown if there are too many open files) and returns "null" if they
are encountered – we can do the same thing but actually catch & wrap/rethrow
the IOExceptions by calling {{ClassLoader.getResource() +
URL.openConnection()}} instead.
As i said: see SOLR-12021...
https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=blobdiff;f=solr/solrj/src/java/org/apache/solr/common/util/Utils.java;h=d35486e6c7688c4b32d8bd6840e590a36b4a5ab2;hp=4ab24d2be3e2aaf39b041c2a3676f456040b5e58;hb=9e0e301;hpb=df0f141907b0701d7b1f1fc297ae33ef901844a0
> Test fails for "Can't find resource" for files in the _default configset
>
>
> Key: SOLR-12386
> URL: https://issues.apache.org/jira/browse/SOLR-12386
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Priority: Minor
> Attachments: cant find resource, stacktrace.txt
>
>
> Some tests, especially ConcurrentCreateRoutedAliasTest, have failed
> sporadically failed with the message "Can't find resource" pertaining to a
> file that is in the default ConfigSet yet mysteriously can't be found. This
> happens when a collection is being created that ultimately fails for this
> reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488088#comment-16488088 ] Uwe Schindler commented on SOLR-12386: -- You can't make getResourceAsStream a forbidden-api, because this method(s) is basically all Java resource handling is going through. All other classloader methods are behaving the same way, so how to load resources then? > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[
https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488082#comment-16488082
]
Hoss Man commented on SOLR-12386:
-
This overall symptom sounds really familar – see SOLR-12021.
I wonder if the root cause here is similar to what I found in that jira?
{quote}[resource files] are being loaded with
{{...getClassLoader().getResourceAsStream(resourceName)}} – but nothing is ever
closing the stream, so it can eventually (depending on what test classes run in
each JVM and how many files they try to open like this) cause the JVM to hit
the ulimit for open file handles – but that specific cause of the failure is
never reported, because {{ClassLoader.getResourseAsStream(...)}} is explicity
designed to swallow any IOExceptions encountered and just returns "null"...
{quote}
...allthough there was definitely some leaked InputStreams in that jira, there
wouldn't neccessarily even have to be a resource leak to see similar problems:
if individual tests are opening a lot of cores concurrently, they could be
hitting the ulimit on jenkins ("randomly" depending on what other tests were
run in the same JVM causing a variable number of open file handles to various
class files held open by the current system classloader) but instead of
throwing a clean error to that effect getSourceAsStream just returns null and
causes the SolrResourceLoader to assume it doesn't exist ... maybe?
(we should probably consider making {{ClassLoader.getResourceAsStream}} a
forbidden API to prevent this risk even if it's *not* the cause of the current
failures.)
> Test fails for "Can't find resource" for files in the _default configset
>
>
> Key: SOLR-12386
> URL: https://issues.apache.org/jira/browse/SOLR-12386
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Priority: Minor
> Attachments: cant find resource, stacktrace.txt
>
>
> Some tests, especially ConcurrentCreateRoutedAliasTest, have failed
> sporadically failed with the message "Can't find resource" pertaining to a
> file that is in the default ConfigSet yet mysteriously can't be found. This
> happens when a collection is being created that ultimately fails for this
> reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484591#comment-16484591 ] David Smiley commented on SOLR-12386: - Analysis strategy: * Enumerate each test that has failed this way (share here). What do they have in common? * Try to make a test that easily reproduces; maybe with _some_ beasting. * ? go digging; come up with hair brained theories ? > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484582#comment-16484582 ] ASF subversion and git services commented on SOLR-12386: Commit 982268efd14147ab99ab5b3e152fd4106e6581f1 in lucene-solr's branch refs/heads/branch_7x from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=982268e ] SOLR-12386: Apply AwaitsFix to ConcurrentCreateRoutedAliasTest (cherry picked from commit af59c46) > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12386) Test fails for "Can't find resource" for files in the _default configset
[ https://issues.apache.org/jira/browse/SOLR-12386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484575#comment-16484575 ] ASF subversion and git services commented on SOLR-12386: Commit af59c46363f3497d44548021e4ff15d924ddbec3 in lucene-solr's branch refs/heads/master from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=af59c46 ] SOLR-12386: Apply AwaitsFix to ConcurrentCreateRoutedAliasTest > Test fails for "Can't find resource" for files in the _default configset > > > Key: SOLR-12386 > URL: https://issues.apache.org/jira/browse/SOLR-12386 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Attachments: cant find resource, stacktrace.txt > > > Some tests, especially ConcurrentCreateRoutedAliasTest, have failed > sporadically failed with the message "Can't find resource" pertaining to a > file that is in the default ConfigSet yet mysteriously can't be found. This > happens when a collection is being created that ultimately fails for this > reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
