SecureRandom algorithm 'NativePRNG'

[Isabelle Giguere]

Hi;

Interagtion test run using Jenkins is failing with this error message:

SecureRandom algorithm 'NativePRNG' is in use by your JVM, which is a 
potentially blocking algorithm on some environments. Please report the details 
of this failure (and your JVM vendor/version) to solr-user@lucene.apache.org. 
You can try to run your tests with -Djava.security.egd=file:/dev/./urandom or 
bypass this check using -Dtest.solr.allowed.securerandom=NativePRNG as a JVM 
option when running tests.

Relevant system information:
java.runtime.version : 1.8.0_161-b12
java.vm.name : Java HotSpot(TM) 64-Bit Server VM
java.vm.vendor : Oracle Corporation
java.vm.version : 25.161-b12
os.arch : amd64
os.name : Linux
os.version : 3.13.0-110-generic

Complete stack trace attached.

Thank you;

Isabelle Giguère
Computational Linguist and Java Developer  |  Semantic Technologies R
Linguiste informaticienne et développeur Java  |  Semantic Technologies R
Phone: 514-908-5406 #225
Website:  www.opentext.com

[http://mimage.opentext.com/alt_content/binary/images/emailsupport-logo-opentext.gif]

This email message is confidential, may be privileged, and is intended for the 
exclusive use of the addressee. Any other person is strictly prohibited from 
disclosing or reproducing it. If the addressee cannot be reached or is unknown 
to you, please inform the sender by return email and delete this email message 
and all copies immediately.
---
Ce message s'adresse uniquement à la personne ou à l'entité indiquée et peut 
contenir des renseignements confidentiels ou privilégiés. Toute consultation, 
retransmission, diffusion ou tout autre usage de l'information contenue dans ce 
message ainsi que toute adoption de mesures fondée sur celle-ci, par des 
personnes ou des entités autres que le destinataire indiqué est interdit. Si ce 
message vous a été transmis par erreur, veuillez en informer l'envoyeur en le 
lui retournant sur-le-champ et supprimer ensuite immédiatement le message, 
ainsi que toutes les pièces jointes, sans le copier, le diffuser ou le 
divulguer.

Error Message

SecureRandom algorithm 'NativePRNG' is in use by your JVM, which is a 
potentially blocking algorithm on some environments. Please report the details 
of this failure (and your JVM vendor/version) to solr-user@lucene.apache.org. 
You can try to run your tests with -Djava.security.egd=file:/dev/./urandom or 
bypass this check using -Dtest.solr.allowed.securerandom=NativePRNG as a JVM 
option when running tests.

Stacktrace

java.lang.AssertionError: SecureRandom algorithm 'NativePRNG' is in use by your 
JVM, which is a potentially blocking algorithm on some environments. Please 
report the details of this failure (and your JVM vendor/version) to 
solr-user@lucene.apache.org. You can try to run your tests with 
-Djava.security.egd=file:/dev/./urandom or bypass this check using 
-Dtest.solr.allowed.securerandom=NativePRNG as a JVM option when running tests.
at __randomizedtesting.SeedInfo.seed([138DC08512C1C206]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at 
org.apache.solr.SolrTestCaseJ4.assertNonBlockingRandomGeneratorAvailable(SolrTestCaseJ4.java:2682)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

Re: Storing & using feature vectors

[Doug Turnbull]

This is a pretty big hole in Lucene-based search right now that many
practitioners have struggled with

I know a couple of people who have worked on solutions. And I've used a
couple of hacks:

- You can hack together something that does cosine similarity using the
term frequency & query boosts DelimitedTermFreqFilterFactory. Basically the
term frequency becomes a feature weight on the document. Boosts become the
query weight. If you massage things correctly with the similarity, the
resulting boolean similarity is a dot product...

- Erik Hatcher has done some great work with payloads which you might want
to check out. See the delimited payload filter factory, and payload score
function queries

- Simon Hughes Activate Talk (slides/video not yet posted) covers this
topic in some depth

- Rene Kriegler's Haystack Talk discusses encoding Inception model
vectorizations of images:
https://opensourceconnections.com/events/haystack-single/haystack-relevance-scoring/

If this is a huge importance to you, I might also suggest looking at vespa,
which makes tensors a first-class citizen and makes matrix-math pretty
seamless: http://vespa.ai

Hope that helps
-Doug

On Fri, Oct 19, 2018 at 12:50 PM Ken Krugler 
wrote:

> Hi all,
>
> [I posted on the Lucene list two days ago, but didn’t see any response -
> checking here for completeness]
>
> I’ve been looking at directly storing feature vectors and providing
> scoring/filtering support.
>
> This is for vectors consisting of (typically 300 - 2048) floats or doubles.
>
> It’s following the same pattern as geospatial support - so a new field
> type and query/parser, plus plumbing to hook it into Solr.
>
> Before I go much further, is there anything like this already done, or in
> the works?
>
> Thanks,
>
> — Ken
>
> --
> Ken Krugler
> +1 530-210-6378 <(530)%20210-6378>
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
> --
CTO, OpenSource Connections
Author, Relevant Search
http://o19s.com/doug

Storing & using feature vectors

[Ken Krugler]

Hi all,

[I posted on the Lucene list two days ago, but didn’t see any response - 
checking here for completeness]
 
I’ve been looking at directly storing feature vectors and providing 
scoring/filtering support.

This is for vectors consisting of (typically 300 - 2048) floats or doubles.

It’s following the same pattern as geospatial support - so a new field type and 
query/parser, plus plumbing to hook it into Solr.

Before I go much further, is there anything like this already done, or in the 
works?

Thanks,

— Ken

--
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra

Re: Response time creep in Solr

[Shawn Heisey]

On 10/19/2018 7:57 AM, Roopa Rao wrote:

 From the past few months there has been a steady increase in the Solr
response time in our application, yes there are enhancements and index size
increase.
How to approach this issue to find the root cause for this slow and
constant increase? What parameters to check?


Please be aware of this:

https://home.apache.org/~hossman/#threadhijack

There's not a lot of information here.  Without certain pieces of 
information, there's little help we can offer.


Let's start with this:

https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue

There may be more questions after that info is provided.

Thanks,
Shawn

Response time creep in Solr

[Roopa Rao]

Hi All,

>From the past few months there has been a steady increase in the Solr
response time in our application, yes there are enhancements and index size
increase.
How to approach this issue to find the root cause for this slow and
constant increase? What parameters to check?

Any pointers on this is appreciated!

Thank you,
Roopa