[GitHub] [lucene-solr] gunasekhardora opened a new pull request #1048: SOLR-13806: change type of QueryResponse._explainMap
gunasekhardora opened a new pull request #1048: SOLR-13806: change type of QueryResponse._explainMap URL: https://github.com/apache/lucene-solr/pull/1048 # Description Please provide a short description of the changes you're making with this pull request. # Solution Please provide a short description of the approach taken to implement your solution. # Tests Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem. # Checklist Please review the following and check all that apply: - [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I am authorized to contribute this code to the ASF and have removed any code I do not have a license to distribute. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [x] I have developed this patch against the `master` branch. - [ ] I have run `ant precommit` and the appropriate test suite. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985481#comment-16985481 ] Robert Muir commented on SOLR-13986: With the attached patch (containing numerous ugly hadoop hacks), all tests pass. Basically the hacks still allow hadoop to do its crazy process execution in the specific places that are needed for it to work. > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch, > SOLR-13986.patch, SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated SOLR-13986: --- Attachment: SOLR-13986.patch > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch, > SOLR-13986.patch, SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] praste commented on a change in pull request #153: Fix issues reported by findbugs
praste commented on a change in pull request #153: Fix issues reported by
findbugs
URL: https://github.com/apache/lucene-solr/pull/153#discussion_r352319352
##
File path:
solr/contrib/extraction/src/java/org/apache/solr/handler/extraction/XLSXResponseWriter.java
##
@@ -357,7 +357,7 @@ public void writeArray(String name, Iterator val) throws
IOException {
if (v instanceof IndexableField) {
IndexableField f = (IndexableField)v;
if (v instanceof Date) {
- output.append(((Date) val).toInstant().toString() + "; ");
+ output.append(((Date) v).toInstant().toString() + "; ");
Review comment:
I have not seen this in the logs.
I hit some weird error a few years back and I couldn’t believe how that we
have such blatant bugs in the code all over that I ran findbugs and started
wack-a-mole game for all the bugs reported.
I don’t remember what the original bug was anymore. I will give this a
second shot and I will put small patches for each type of error reported by
findbugs
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13897) Unsafe publication of Terms object in ZkShardTerms
[ https://issues.apache.org/jira/browse/SOLR-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar reassigned SOLR-13897: Assignee: Shalin Shekhar Mangar > Unsafe publication of Terms object in ZkShardTerms > -- > > Key: SOLR-13897 > URL: https://issues.apache.org/jira/browse/SOLR-13897 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.2, 8.3 >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.4 > > Attachments: SOLR-13897.patch, SOLR-13897.patch, SOLR-13897.patch > > > The Terms object in ZkShardTerms is written using a write lock but reading is > allowed freely. This is not safe and can cause visibility issues and > associated race conditions under contention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13897) Unsafe publication of Terms object in ZkShardTerms
[ https://issues.apache.org/jira/browse/SOLR-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985479#comment-16985479 ] Shalin Shekhar Mangar commented on SOLR-13897: -- This patch adds registerTerm inside ZkCollectionTerms so that it is called after synchronizing on the same terms object as that used for the remove. I couldn't quite make out a condition where both could happen concurrently but it makes me sleep better knowing that they absolutely cannot. > Unsafe publication of Terms object in ZkShardTerms > -- > > Key: SOLR-13897 > URL: https://issues.apache.org/jira/browse/SOLR-13897 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.2, 8.3 >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.4 > > Attachments: SOLR-13897.patch, SOLR-13897.patch, SOLR-13897.patch > > > The Terms object in ZkShardTerms is written using a write lock but reading is > allowed freely. This is not safe and can cause visibility issues and > associated race conditions under contention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13897) Unsafe publication of Terms object in ZkShardTerms
[ https://issues.apache.org/jira/browse/SOLR-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-13897: - Status: Patch Available (was: Open) > Unsafe publication of Terms object in ZkShardTerms > -- > > Key: SOLR-13897 > URL: https://issues.apache.org/jira/browse/SOLR-13897 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.2, 8.3 >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.4 > > Attachments: SOLR-13897.patch, SOLR-13897.patch, SOLR-13897.patch > > > The Terms object in ZkShardTerms is written using a write lock but reading is > allowed freely. This is not safe and can cause visibility issues and > associated race conditions under contention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13897) Unsafe publication of Terms object in ZkShardTerms
[ https://issues.apache.org/jira/browse/SOLR-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-13897: - Attachment: SOLR-13897.patch > Unsafe publication of Terms object in ZkShardTerms > -- > > Key: SOLR-13897 > URL: https://issues.apache.org/jira/browse/SOLR-13897 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.2, 8.3 >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.4 > > Attachments: SOLR-13897.patch, SOLR-13897.patch, SOLR-13897.patch > > > The Terms object in ZkShardTerms is written using a write lock but reading is > allowed freely. This is not safe and can cause visibility issues and > associated race conditions under contention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13989) Move all hadoop related code to a contrib module
Shalin Shekhar Mangar created SOLR-13989:
Summary: Move all hadoop related code to a contrib module
Key: SOLR-13989
URL: https://issues.apache.org/jira/browse/SOLR-13989
Project: Solr
Issue Type: Task
Security Level: Public (Default Security Level. Issues are Public)
Components: Hadoop Integration
Reporter: Shalin Shekhar Mangar
Fix For: master (9.0)
Spin off from SOLR-13986:
{quote}
It seems really important to move or remove this hadoop shit out of the solr
core: It is really unreasonable that solr core depends on hadoop. that's gonna
simply block any progress improving its security, because solr code will get
dragged down by hadoop's code.
{quote}
We should move all hadoop related dependencies to a separate contrib module
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985477#comment-16985477 ] Shalin Shekhar Mangar commented on SOLR-13986: -- bq. Unrelated to these specific problems, It seems really important to move or remove this hadoop shit out of the solr core: It is really unreasonable that solr core depends on hadoop. that's gonna simply block any progress improving its security, because solr code will get dragged down by hadoop's code. I agree that hadoop specific code should live in a contrib. I'll open an issue to do that. > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch, > SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985475#comment-16985475 ] Robert Muir commented on SOLR-13986: I can't hack around all the broken hadoop code here. The last bad guy is "DF" class which should not even exist. But it executes and parses output of "df" command, instead of using e.g. java's FileStore API. And there is definitely no exception i can send to it, to make it go away. The hadoop code is just too broken. We can shove it under the rug, disable 2 or 3 hdfs tests (which from what i can tell, weren't working correctly anyway), until hadoop shit can be fixed. Personally, I'm not motivated to try to fix hadoop code in order to prevent RCE in solr. And this weekend is coming to an end and my time will be up to try to get things moving. Unrelated to these specific problems, It seems really important to move or remove this hadoop shit out of the solr core: It is really unreasonable that solr core depends on hadoop. that's gonna simply block any progress improving its security, because solr code will get dragged down by hadoop's code. > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch, > SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated SOLR-13986: --- Attachment: SOLR-13986.patch Status: Open (was: Open) I added a hack for their equally broken "canWrite" method because i'm here already. but really this hadoop code should get fixed. > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch, > SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985471#comment-16985471 ] Robert Muir commented on SOLR-13986: These problems are already present in tests today with MiniDFSCluster, not even caused by my changes. For example the same exact issue happens in checkWrite(). I guess i'll workaround it here, even though its not related to this issue at all, but its surprising to me. Nobody must be looking at hdfs test failures at all... > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985468#comment-16985468 ] Robert Muir commented on SOLR-13986: I implemented patch with my workaround from before. Note that 2 tests still fail: HdfsThreadLeakTest, TestHdfsCloudBackupRestore The problem is hadoop test code (MiniDFSCluster.createPermissionsDiagnosisString): https://github.com/apache/hadoop/blob/7a3188d054481b9bd563e337901e93476303ce7f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L1260 It calls File.canExecute, but doesn't handle SecurityException. This is clearly documented in the java.io.File api: https://docs.oracle.com/javase/7/docs/api/java/io/File.html#canExecute() I think I can hack around it but someone should really do something about this hadoop code. > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated SOLR-13986: --- Attachment: SOLR-13986.patch > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch, SOLR-13986.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13897) Unsafe publication of Terms object in ZkShardTerms
[ https://issues.apache.org/jira/browse/SOLR-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-13897: - Attachment: SOLR-13897.patch > Unsafe publication of Terms object in ZkShardTerms > -- > > Key: SOLR-13897 > URL: https://issues.apache.org/jira/browse/SOLR-13897 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.2, 8.3 >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.4 > > Attachments: SOLR-13897.patch, SOLR-13897.patch > > > The Terms object in ZkShardTerms is written using a write lock but reading is > allowed freely. This is not safe and can cause visibility issues and > associated race conditions under contention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13897) Unsafe publication of Terms object in ZkShardTerms
[ https://issues.apache.org/jira/browse/SOLR-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985459#comment-16985459 ] Shalin Shekhar Mangar commented on SOLR-13897: -- The onTermUpdates might receive updates out of order (i.e. monotonic term versions are not guaranteed inside onTermUpdates) but it is not a problem in the default RecoveringCoreTermWatcher implementation because it tracks the last term that triggered recovery and returns if it is greater (or equal) to the current term. This patch adds javadocs to the CoreTermWatcher interface and calls out the behavior of these invocations. > Unsafe publication of Terms object in ZkShardTerms > -- > > Key: SOLR-13897 > URL: https://issues.apache.org/jira/browse/SOLR-13897 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.2, 8.3 >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (9.0), 8.4 > > Attachments: SOLR-13897.patch, SOLR-13897.patch > > > The Terms object in ZkShardTerms is written using a write lock but reading is > allowed freely. This is not safe and can cause visibility issues and > associated race conditions under contention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant commented on a change in pull request #1043: LUCENE-9071: Speed up BM25 scores.
bruno-roustant commented on a change in pull request #1043: LUCENE-9071: Speed
up BM25 scores.
URL: https://github.com/apache/lucene-solr/pull/1043#discussion_r352307646
##
File path:
lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java
##
@@ -221,8 +251,8 @@ public final SimScorer scorer(float boost,
CollectionStatistics collectionStats,
@Override
public float score(float freq, long encodedNorm) {
- double norm = cache[((byte) encodedNorm) & 0xFF];
- return weight * (float) (freq / (freq + norm));
+ float norm = cache[((byte) encodedNorm) & 0xFF];
+ return weight * tf(freq, norm);
Review comment:
Thanks, very good point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] original-brownbear opened a new pull request #1047: MINOR: Fix Incorrect Constant Name in Codec Docs
original-brownbear opened a new pull request #1047: MINOR: Fix Incorrect Constant Name in Codec Docs URL: https://github.com/apache/lucene-solr/pull/1047 The name was wrong here. Also, added a link to make this doc more fun to navigate in HTML form and make sure it doesn't go bad again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] rmuir commented on a change in pull request #1043: LUCENE-9071: Speed up BM25 scores.
rmuir commented on a change in pull request #1043: LUCENE-9071: Speed up BM25
scores.
URL: https://github.com/apache/lucene-solr/pull/1043#discussion_r352303377
##
File path:
lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java
##
@@ -221,8 +251,8 @@ public final SimScorer scorer(float boost,
CollectionStatistics collectionStats,
@Override
public float score(float freq, long encodedNorm) {
- double norm = cache[((byte) encodedNorm) & 0xFF];
- return weight * (float) (freq / (freq + norm));
+ float norm = cache[((byte) encodedNorm) & 0xFF];
+ return weight * tf(freq, norm);
Review comment:
the crazy casting is not an optimization but instead to ensure that scores
dont go backwards with increasing tf, decreasing norm, etc. it has to behave
"semimonotonic" (like described in java Math api docs) so that wand type
optimizations are correct. so if you changed the api to be all doubles, it
would mean we'd have to use BigDecimal which would br slower.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant commented on a change in pull request #1043: LUCENE-9071: Speed up BM25 scores.
bruno-roustant commented on a change in pull request #1043: LUCENE-9071: Speed
up BM25 scores.
URL: https://github.com/apache/lucene-solr/pull/1043#discussion_r352157819
##
File path:
lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java
##
@@ -221,8 +251,8 @@ public final SimScorer scorer(float boost,
CollectionStatistics collectionStats,
@Override
public float score(float freq, long encodedNorm) {
- double norm = cache[((byte) encodedNorm) & 0xFF];
- return weight * (float) (freq / (freq + norm));
+ float norm = cache[((byte) encodedNorm) & 0xFF];
+ return weight * tf(freq, norm);
Review comment:
As I understand, this is the line that optimizes. Indeed casts to double and
then to float cost. I'm surprised that it matters on the overall query
throughput. It is in the order of a couple of ns, so the impact is visible for
lots of scores (millions), yes.
If freq, norm and score were double, then we wouldn't have casts and the
speed would be the good one on 64 bits machines and we wouldn't need this
optimization. Could this API be changed in 9.x to have double instead of float?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13988) Harden CreateCollectionCleanupTest
[
https://issues.apache.org/jira/browse/SOLR-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson updated SOLR-13988:
--
Priority: Minor (was: Major)
> Harden CreateCollectionCleanupTest
> --
>
> Key: SOLR-13988
> URL: https://issues.apache.org/jira/browse/SOLR-13988
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
> Attachments: SOLR-13988.patch
>
>
> This test counts on being unable to create
> {code}
> /some_invalid_dir/foo
> {code}
> And if it does somehow succeed, it'll affect the filesystem (which it
> apparently can at least on Windows under some circumstances and maybe Unix if
> you're super-user).
> It seems safer to create a temp dir and explicitly disable write permissions
> on it and use that instead. See attached patch. While I don't have a Windows
> machine to test on, the javadocs assure me that I can disable write perms
> this way on Windows...
> [~anshum][~tflobbe] Does this seem OK to you?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13988) Harden CreateCollectionCleanupTest
[
https://issues.apache.org/jira/browse/SOLR-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson updated SOLR-13988:
--
Status: Patch Available (was: Open)
> Harden CreateCollectionCleanupTest
> --
>
> Key: SOLR-13988
> URL: https://issues.apache.org/jira/browse/SOLR-13988
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-13988.patch
>
>
> This test counts on being unable to create
> {code}
> /some_invalid_dir/foo
> {code}
> And if it does somehow succeed, it'll affect the filesystem (which it
> apparently can at least on Windows under some circumstances and maybe Unix if
> you're super-user).
> It seems safer to create a temp dir and explicitly disable write permissions
> on it and use that instead. See attached patch. While I don't have a Windows
> machine to test on, the javadocs assure me that I can disable write perms
> this way on Windows...
> [~anshum][~tflobbe] Does this seem OK to you?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13988) Harden CreateCollectionCleanupTest
[
https://issues.apache.org/jira/browse/SOLR-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson updated SOLR-13988:
--
Attachment: SOLR-13988.patch
> Harden CreateCollectionCleanupTest
> --
>
> Key: SOLR-13988
> URL: https://issues.apache.org/jira/browse/SOLR-13988
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-13988.patch
>
>
> This test counts on being unable to create
> {code}
> /some_invalid_dir/foo
> {code}
> And if it does somehow succeed, it'll affect the filesystem (which it
> apparently can at least on Windows under some circumstances and maybe Unix if
> you're super-user).
> It seems safer to create a temp dir and explicitly disable write permissions
> on it and use that instead. See attached patch. While I don't have a Windows
> machine to test on, the javadocs assure me that I can disable write perms
> this way on Windows...
> [~anshum][~tflobbe] Does this seem OK to you?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13988) Harden CreateCollectionCleanupTest
Erick Erickson created SOLR-13988:
-
Summary: Harden CreateCollectionCleanupTest
Key: SOLR-13988
URL: https://issues.apache.org/jira/browse/SOLR-13988
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Erick Erickson
Assignee: Erick Erickson
This test counts on being unable to create
{code}
/some_invalid_dir/foo
{code}
And if it does somehow succeed, it'll affect the filesystem (which it
apparently can at least on Windows under some circumstances and maybe Unix if
you're super-user).
It seems safer to create a temp dir and explicitly disable write permissions on
it and use that instead. See attached patch. While I don't have a Windows
machine to test on, the javadocs assure me that I can disable write perms this
way on Windows...
[~anshum][~tflobbe] Does this seem OK to you?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13982) set security-related http response headers by default
[
https://issues.apache.org/jira/browse/SOLR-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985418#comment-16985418
]
Robert Muir commented on SOLR-13982:
I made a followup issue to address this {{unsafe-eval}}: SOLR-13987
I'd like for us to "get started" with something similar to the patch here. It
is better than nothing!
> set security-related http response headers by default
> -
>
> Key: SOLR-13982
> URL: https://issues.apache.org/jira/browse/SOLR-13982
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13982.patch
>
>
> Solr server should set some best practice http security response headers, to
> e.g. protect users of the admin ui against XSS/injection/clickjacking/etc
> * Content-Security-Policy
> * X-Content-Type-Options
> * X-XSS-Protection
> * X-Frame-Options
> Disabling inline javascript is important, so that e.g. if there is a bug then
> injected code doesn't get executed. Unfortunately the current admin UI
> dangerously relies on {{eval}}, so for now {{unsafe-eval}} must be allowed so
> that everything still works. This should really be cleaned up, i have the
> feeling as long as it works this way, that you can still execute stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13987) fix admin UI to not rely on javascript eval()
Robert Muir created SOLR-13987: -- Summary: fix admin UI to not rely on javascript eval() Key: SOLR-13987 URL: https://issues.apache.org/jira/browse/SOLR-13987 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Robert Muir Followup from SOLR-13982: currently any CSP is weak because it must allow this eval: means arbitrary javascript can still be executed. Let's fix the admin UI to not require eval so it can be disabled by the browser. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13986) remove "execute" permission from solr-tests.policy
[
https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985413#comment-16985413
]
Robert Muir commented on SOLR-13986:
Attached "notyet" patch disables the execution, but triggers some
hadoop-related test failures. For example:
{noformat}
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=99890C3603287BF
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=lkt
-Dtests.timezone=Europe/Podgorica -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J3 | TestImpersonationWithHadoopAuth (suite) <<<
[junit4]> Throwable #1:
com.google.common.util.concurrent.UncheckedExecutionException:
java.security.AccessControlException: access denied ("java.io.FilePermission"
"<>" "execute")
[junit4]>at
__randomizedtesting.SeedInfo.seed([99890C3603287BF]:0)
[junit4]>at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
[junit4]>at
com.google.common.cache.LocalCache.get(LocalCache.java:3951)
[junit4]>at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973)
[junit4]>at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957)
[junit4]>at
org.apache.hadoop.security.Groups.getGroups(Groups.java:228)
[junit4]>at
org.apache.solr.security.hadoop.ImpersonationUtil.getUsersFirstGroup(ImpersonationUtil.java:42)
[junit4]>at
org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth.setupClass(TestImpersonationWithHadoopAuth.java:74)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:567)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:830)
[junit4]> Caused by: java.security.AccessControlException: access denied
("java.io.FilePermission" "<>" "execute")
[junit4]>at
java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
[junit4]>at
java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
[junit4]>at
java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
[junit4]>at
java.base/java.lang.SecurityManager.checkExec(SecurityManager.java:658)
[junit4]>at
java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1096)
[junit4]>at
java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
[junit4]>at
org.apache.hadoop.util.Shell.runCommand(Shell.java:938)
[junit4]>at org.apache.hadoop.util.Shell.run(Shell.java:901)
[junit4]>at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
[junit4]>at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:200)
[junit4]>at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
[junit4]>at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
[junit4]>at
org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:387)
[junit4]>at
org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:321)
[junit4]>at
org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:270)
[junit4]>at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
[junit4]>at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276)
[junit4]>at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
[junit4]>at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
[junit4]>... 29 more
{noformat}
This is what I expected would happen, hadoop has really bad code here. In
general it expects to run a bunch of shell commands and parse output (instead
of e.g. invoking proper java apis). When it runs these shell commands, it only
handles IOException and not SecurityException, so it fails completely.
One approach I used to tame this hadoop code before, is a custom Policy
subclass,that sneaky-rethrows an IOException to this specific hadoop code
instead of a SecurityException:
https://github.com/elastic/e
[jira] [Updated] (SOLR-13986) remove "execute" permission from solr-tests.policy
[ https://issues.apache.org/jira/browse/SOLR-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated SOLR-13986: --- Attachment: SOLR-13986-notyet.patch > remove "execute" permission from solr-tests.policy > -- > > Key: SOLR-13986 > URL: https://issues.apache.org/jira/browse/SOLR-13986 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13986-notyet.patch > > > If we don't really need to execute processes, we can take the permission > away. That way any attempt to execute something results in a > SecurityException rather than running a process. > It is necessary to first fix the tests policy before thinking about > supporting securitymanager in solr. This way we can ensure functionality does > not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13986) remove "execute" permission from solr-tests.policy
Robert Muir created SOLR-13986: -- Summary: remove "execute" permission from solr-tests.policy Key: SOLR-13986 URL: https://issues.apache.org/jira/browse/SOLR-13986 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Robert Muir If we don't really need to execute processes, we can take the permission away. That way any attempt to execute something results in a SecurityException rather than running a process. It is necessary to first fix the tests policy before thinking about supporting securitymanager in solr. This way we can ensure functionality does not break via our tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13984) Solr should run inside a SecurityManager
[
https://issues.apache.org/jira/browse/SOLR-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985411#comment-16985411
]
Robert Muir commented on SOLR-13984:
Hi Ishan: what do you think about narrowing the scope of this first issue to
disabling process execution (e.g. RCE). I think this may be challenging enough.
Implementing this "fully", to do things like protect filesystem access to
defend against other attacks like directory traversal, will likely involve
major changes. For example if users can do HTTP requests that spin up new cores
on arbitrary filesystem locations, and that model must be supported, then its
impossible to really limit filesystem access. And if components try to allow
arbitrary execution of scripts or similar, it gets heavy.
Essentially to secure the application, it is more than just adding some
'sandbox feature', usually code has to be refactored around principle of least
priv and code with security issues has to be fixed. SOLR-13982 is a good
example of this: browser's 'sandbox feature' is not fully effective until we
actually fix the underlying code to be more secure.
But I think we should special case RCE: disable process execution and try to
make it impossible as a first step.
> Solr should run inside a SecurityManager
>
>
> Key: SOLR-13984
> URL: https://issues.apache.org/jira/browse/SOLR-13984
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
>
> To reduce the effect of attacks, esp. RCE, Solr should run inside a
> SecurityManager.
> Quoting Uwe here:
> {quote}
> The correct way to fix all issues we have seen the last time is very simple:
> LET'S RUN SOLR INSIDE A SECURITY MANAGER IN PRODUCTION (like in tests).
> Elasticsearch is doing this, so please please let's do this instead. But this
> requires to finally get rid of the webapplication and start.jar and add our
> own bootstrapping (like in tests) that configure Jetty and Security Manager
> from our own org.apache.solr.bootstrap.Main.java (or similar).
> {quote}
> https://jira.apache.org/jira/browse/SOLR-12316?focusedCommentId=16465038&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16465038
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13983) remove or replace process execution in SystemInfoHandler
[
https://issues.apache.org/jira/browse/SOLR-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985403#comment-16985403
]
Robert Muir commented on SOLR-13983:
The only use of the e.g. {{uptime}} command run here seemed to be the admin UI,
which would then parse load averages from it (if it was available). I changed
it to display the load average from the OSMXBean instead (which is already in
the response)
> remove or replace process execution in SystemInfoHandler
>
>
> Key: SOLR-13983
> URL: https://issues.apache.org/jira/browse/SOLR-13983
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13983.patch
>
>
> SystemInfoHandler is the only place in solr code executing processes.
> Since solr is a server/long running process listening to HTTP, ideally
> process execution could be disabled (e.g. with security manager). But first
> this code needs to be removed or replaced, so that there is no legitimate use
> of it:
> {noformat}
> try {
> if (!Constants.WINDOWS) {
> info.add( "uname", execute( "uname -a" ) );
> info.add( "uptime", execute( "uptime" ) );
> }
> } catch( Exception ex ) {
> log.warn("Unable to execute command line tools to get operating system
> properties.", ex);
> }
> return info;
> {noformat}
> It already looks like its getting data from OS MXbean here, so maybe this
> logic is simply outdated or not needed. It seems to be "best-effort" anyway.
> Alternatively similar stuff could be fetched by reading from e.g. /proc file
> system location if needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13983) remove or replace process execution in SystemInfoHandler
[
https://issues.apache.org/jira/browse/SOLR-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated SOLR-13983:
---
Attachment: SOLR-13983.patch
> remove or replace process execution in SystemInfoHandler
>
>
> Key: SOLR-13983
> URL: https://issues.apache.org/jira/browse/SOLR-13983
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13983.patch
>
>
> SystemInfoHandler is the only place in solr code executing processes.
> Since solr is a server/long running process listening to HTTP, ideally
> process execution could be disabled (e.g. with security manager). But first
> this code needs to be removed or replaced, so that there is no legitimate use
> of it:
> {noformat}
> try {
> if (!Constants.WINDOWS) {
> info.add( "uname", execute( "uname -a" ) );
> info.add( "uptime", execute( "uptime" ) );
> }
> } catch( Exception ex ) {
> log.warn("Unable to execute command line tools to get operating system
> properties.", ex);
> }
> return info;
> {noformat}
> It already looks like its getting data from OS MXbean here, so maybe this
> logic is simply outdated or not needed. It seems to be "best-effort" anyway.
> Alternatively similar stuff could be fetched by reading from e.g. /proc file
> system location if needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13982) set security-related http response headers by default
[
https://issues.apache.org/jira/browse/SOLR-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985398#comment-16985398
]
Robert Muir commented on SOLR-13982:
It seems the angular framework used here can be coerced into behaving securely,
but I imagine all this inline stuff has to be cleaned up or precompiled or
whatever: https://docs.angularjs.org/api/ng/directive/ngCsp
> set security-related http response headers by default
> -
>
> Key: SOLR-13982
> URL: https://issues.apache.org/jira/browse/SOLR-13982
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13982.patch
>
>
> Solr server should set some best practice http security response headers, to
> e.g. protect users of the admin ui against XSS/injection/clickjacking/etc
> * Content-Security-Policy
> * X-Content-Type-Options
> * X-XSS-Protection
> * X-Frame-Options
> Disabling inline javascript is important, so that e.g. if there is a bug then
> injected code doesn't get executed. Unfortunately the current admin UI
> dangerously relies on {{eval}}, so for now {{unsafe-eval}} must be allowed so
> that everything still works. This should really be cleaned up, i have the
> feeling as long as it works this way, that you can still execute stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13982) set security-related http response headers by default
[
https://issues.apache.org/jira/browse/SOLR-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985397#comment-16985397
]
Robert Muir commented on SOLR-13982:
You can see the bad guy if you remove {{unsafe-eval}} from the CSP here, hit
the admin UI, and look at browser console:
{noformat}
Content Security Policy: The page’s settings blocked the loading of a resource
at eval (“script-src”). angular.min.js:formatted:4889
{noformat}
> set security-related http response headers by default
> -
>
> Key: SOLR-13982
> URL: https://issues.apache.org/jira/browse/SOLR-13982
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13982.patch
>
>
> Solr server should set some best practice http security response headers, to
> e.g. protect users of the admin ui against XSS/injection/clickjacking/etc
> * Content-Security-Policy
> * X-Content-Type-Options
> * X-XSS-Protection
> * X-Frame-Options
> Disabling inline javascript is important, so that e.g. if there is a bug then
> injected code doesn't get executed. Unfortunately the current admin UI
> dangerously relies on {{eval}}, so for now {{unsafe-eval}} must be allowed so
> that everything still works. This should really be cleaned up, i have the
> feeling as long as it works this way, that you can still execute stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13982) set security-related http response headers by default
[
https://issues.apache.org/jira/browse/SOLR-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985392#comment-16985392
]
Robert Muir commented on SOLR-13982:
{quote}
How extensively do we use unsafe evaluations in the admin UI? Can we remove
those functionality from the admin UI that use such evaluations?
{quote}
It seems extensive: for example if you open up {{solr/webapp/web/index.html}}
you see elements like this:
{noformat}
{noformat}
the stuff in the {{ng-style}} attribute looks a lot like javascript code, I am
afraid somehow things are organized such that javascript code can be placed in
attributes like this and it gets {{eval}}'d at runtime.
So with this patch, if i find an injection, while i may not be able to simply
inject inline JS directly anymore, i can probably just encode my payload
instead into an attribute like this ng-style and it will get eval'ed for me:
its not gonna be that much safer until we address this crazy eval'ing.
I don't know enough about the frameworks being used here to know if we are just
using a very naive insecure framework, and it needs to be swapped out for
something else, or if we are just using it in an insecure way and there is a
simple fix. Either way, seems like a biggish change to fix, and we should start
somewhere IMO.
> set security-related http response headers by default
> -
>
> Key: SOLR-13982
> URL: https://issues.apache.org/jira/browse/SOLR-13982
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13982.patch
>
>
> Solr server should set some best practice http security response headers, to
> e.g. protect users of the admin ui against XSS/injection/clickjacking/etc
> * Content-Security-Policy
> * X-Content-Type-Options
> * X-XSS-Protection
> * X-Frame-Options
> Disabling inline javascript is important, so that e.g. if there is a bug then
> injected code doesn't get executed. Unfortunately the current admin UI
> dangerously relies on {{eval}}, so for now {{unsafe-eval}} must be allowed so
> that everything still works. This should really be cleaned up, i have the
> feeling as long as it works this way, that you can still execute stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13982) set security-related http response headers by default
[
https://issues.apache.org/jira/browse/SOLR-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985386#comment-16985386
]
Ishan Chattopadhyaya commented on SOLR-13982:
-
Looks good, Robert. Thanks!
bq. Disabling inline javascript is important, so that e.g. if there is a bug
then injected code doesn't get executed. Unfortunately the current admin UI
dangerously relies on eval, so for now unsafe-eval must be allowed so that
everything still works. This should really be cleaned up, i have the feeling as
long as it works this way, that you can still execute stuff.
How extensively do we use unsafe evaluations in the admin UI? Can we remove
those functionality from the admin UI that use such evaluations?
> set security-related http response headers by default
> -
>
> Key: SOLR-13982
> URL: https://issues.apache.org/jira/browse/SOLR-13982
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Robert Muir
>Priority: Major
> Attachments: SOLR-13982.patch
>
>
> Solr server should set some best practice http security response headers, to
> e.g. protect users of the admin ui against XSS/injection/clickjacking/etc
> * Content-Security-Policy
> * X-Content-Type-Options
> * X-XSS-Protection
> * X-Frame-Options
> Disabling inline javascript is important, so that e.g. if there is a bug then
> injected code doesn't get executed. Unfortunately the current admin UI
> dangerously relies on {{eval}}, so for now {{unsafe-eval}} must be allowed so
> that everything still works. This should really be cleaned up, i have the
> feeling as long as it works this way, that you can still execute stuff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13985) bind to localhost by default
[ https://issues.apache.org/jira/browse/SOLR-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated SOLR-13985: --- Attachment: SOLR-13985.patch > bind to localhost by default > > > Key: SOLR-13985 > URL: https://issues.apache.org/jira/browse/SOLR-13985 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Robert Muir >Priority: Major > Attachments: SOLR-13985.patch > > > Currently solr binds to all interfaces by default. > The default should be safer, so that e.g. the user is not exposed to the > internet until they make an explicit step to do so. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13985) bind to localhost by default
Robert Muir created SOLR-13985: -- Summary: bind to localhost by default Key: SOLR-13985 URL: https://issues.apache.org/jira/browse/SOLR-13985 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Robert Muir Currently solr binds to all interfaces by default. The default should be safer, so that e.g. the user is not exposed to the internet until they make an explicit step to do so. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13984) Solr should run inside a SecurityManager
[
https://issues.apache.org/jira/browse/SOLR-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985382#comment-16985382
]
Ishan Chattopadhyaya commented on SOLR-13984:
-
Can someone with the expertise on this please take this up? Also, should we
mark this as a release blocker for 9.0?
> Solr should run inside a SecurityManager
>
>
> Key: SOLR-13984
> URL: https://issues.apache.org/jira/browse/SOLR-13984
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Priority: Major
>
> To reduce the effect of attacks, esp. RCE, Solr should run inside a
> SecurityManager.
> Quoting Uwe here:
> {quote}
> The correct way to fix all issues we have seen the last time is very simple:
> LET'S RUN SOLR INSIDE A SECURITY MANAGER IN PRODUCTION (like in tests).
> Elasticsearch is doing this, so please please let's do this instead. But this
> requires to finally get rid of the webapplication and start.jar and add our
> own bootstrapping (like in tests) that configure Jetty and Security Manager
> from our own org.apache.solr.bootstrap.Main.java (or similar).
> {quote}
> https://jira.apache.org/jira/browse/SOLR-12316?focusedCommentId=16465038&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16465038
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13984) Solr should run inside a SecurityManager
Ishan Chattopadhyaya created SOLR-13984:
---
Summary: Solr should run inside a SecurityManager
Key: SOLR-13984
URL: https://issues.apache.org/jira/browse/SOLR-13984
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Ishan Chattopadhyaya
To reduce the effect of attacks, esp. RCE, Solr should run inside a
SecurityManager.
Quoting Uwe here:
{quote}
The correct way to fix all issues we have seen the last time is very simple:
LET'S RUN SOLR INSIDE A SECURITY MANAGER IN PRODUCTION (like in tests).
Elasticsearch is doing this, so please please let's do this instead. But this
requires to finally get rid of the webapplication and start.jar and add our own
bootstrapping (like in tests) that configure Jetty and Security Manager from
our own org.apache.solr.bootstrap.Main.java (or similar).
{quote}
https://jira.apache.org/jira/browse/SOLR-12316?focusedCommentId=16465038&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16465038
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13759) Optimize Queries when query filtering by TRA router.field
[ https://issues.apache.org/jira/browse/SOLR-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck reassigned SOLR-13759: --- Assignee: Gus Heck > Optimize Queries when query filtering by TRA router.field > - > > Key: SOLR-13759 > URL: https://issues.apache.org/jira/browse/SOLR-13759 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Assignee: Gus Heck >Priority: Minor > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries will be by filtering query by date range, by > that we will be able to querying the specific TRA collections taking > advantage of the TRA mechanism of partitioning data based on date. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13759) Optimize Queries when query filtering by TRA router.field
[ https://issues.apache.org/jira/browse/SOLR-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985380#comment-16985380 ] Gus Heck commented on SOLR-13759: - [~moshebla], you seem to have removed your patch? Was that on purpose? Should we set the status back until it's ready? > Optimize Queries when query filtering by TRA router.field > - > > Key: SOLR-13759 > URL: https://issues.apache.org/jira/browse/SOLR-13759 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Priority: Minor > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries will be by filtering query by date range, by > that we will be able to querying the specific TRA collections taking > advantage of the TRA mechanism of partitioning data based on date. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13759) Optimize Queries when query filtering by TRA router.field
[ https://issues.apache.org/jira/browse/SOLR-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mosh updated SOLR-13759: Attachment: SOLR-13759.patch > Optimize Queries when query filtering by TRA router.field > - > > Key: SOLR-13759 > URL: https://issues.apache.org/jira/browse/SOLR-13759 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Priority: Minor > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries will be by filtering query by date range, by > that we will be able to querying the specific TRA collections taking > advantage of the TRA mechanism of partitioning data based on date. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13759) Optimize Queries when query filtering by TRA router.field
[ https://issues.apache.org/jira/browse/SOLR-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mosh updated SOLR-13759: Attachment: (was: SOLR-13759.patch) > Optimize Queries when query filtering by TRA router.field > - > > Key: SOLR-13759 > URL: https://issues.apache.org/jira/browse/SOLR-13759 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Priority: Minor > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries will be by filtering query by date range, by > that we will be able to querying the specific TRA collections taking > advantage of the TRA mechanism of partitioning data based on date. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13808) Query DSL should let to cache filter
[
https://issues.apache.org/jira/browse/SOLR-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985374#comment-16985374
]
Mikhail Khludnev commented on SOLR-13808:
-
[~caomanhdat], what's your opinion? Can I push it into master for forthcoming
8.4?
> Query DSL should let to cache filter
>
>
> Key: SOLR-13808
> URL: https://issues.apache.org/jira/browse/SOLR-13808
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-13808.patch
>
>
> Query DSL let to express Lucene BQ's filter
>
> {code:java}
> { query: {bool: { filter: {term: {f:name,query:"foo bar"}}} }}{code}
> However, it might easily catch the need in caching it in filter cache. This
> might rely on ExtensibleQuery and QParser:
> {code:java}
> { query: {bool: { filter: {term: {f:name,query:"foo bar", cache:true}}} }}
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9031) UnsupportedOperationException on highlighting Interval Query
[
https://issues.apache.org/jira/browse/LUCENE-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985372#comment-16985372
]
Mikhail Khludnev commented on LUCENE-9031:
--
{quote}I'd settle on looping as better than the total duplication we have now
{quote}
[~romseygeek], please reconsider. If you are ok with loops, I'd bring them
back.
> UnsupportedOperationException on highlighting Interval Query
>
>
> Key: LUCENE-9031
> URL: https://issues.apache.org/jira/browse/LUCENE-9031
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/queries
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: 8.4
>
> Attachments: LUCENE-9031.patch, LUCENE-9031.patch, LUCENE-9031.patch,
> LUCENE-9031.patch, LUCENE-9031.patch, LUCENE-9031.patch, LUCENE-9031.patch,
> LUCENE-9031.patch, LUCENE-9031.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> When UnifiedHighlighter highlights Interval Query it encounters
> UnsupportedOperationException.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9073) IntervalQuery to respond field on toString and explain
[ https://issues.apache.org/jira/browse/LUCENE-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Khludnev updated LUCENE-9073: - Attachment: LUCENE-9073.patch > IntervalQuery to respond field on toString and explain > -- > > Key: LUCENE-9073 > URL: https://issues.apache.org/jira/browse/LUCENE-9073 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/queries >Reporter: Mikhail Khludnev >Priority: Minor > Attachments: LUCENE-9073.patch > > > It seems like IntervalQuery doesnt' expose the field whether on toString() > nor via explain(). I think it's worth to return field for better visibility. > WDYT, [~romseygeek] ? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13904) Make Analytics component aware of timeAllowed
[
https://issues.apache.org/jira/browse/SOLR-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985367#comment-16985367
]
Lucene/Solr QA commented on SOLR-13904:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
6s{color} | {color:green} analytics in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13904 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987050/SOLR-13904.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP
Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 5a69734 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/606/testReport/ |
| modules | C: solr/contrib/analytics U: solr/contrib/analytics |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/606/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Make Analytics component aware of timeAllowed
> -
>
> Key: SOLR-13904
> URL: https://issues.apache.org/jira/browse/SOLR-13904
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mikhail Khludnev
>Assignee: Houston Putman
>Priority: Major
> Attachments: SOLR-13904.patch, SOLR-13904.patch, SOLR-13904.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13806) SolrJ QueryResponse._explainMap is incorrectly typed
[
https://issues.apache.org/jira/browse/SOLR-13806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985349#comment-16985349
]
Guna Sekhar Dora commented on SOLR-13806:
-
[~ab] May I pick this up?
> SolrJ QueryResponse._explainMap is incorrectly typed
>
>
> Key: SOLR-13806
> URL: https://issues.apache.org/jira/browse/SOLR-13806
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Priority: Major
> Labels: newdev
>
> This field, and the corresponding getter, and the code that extracts debug
> information from the NamedList response in {{extractDebugInfo}}, all use a
> {{Map}} type.
> However, when {{debug.explain.structured=true}} is used the values returned
> in response are not String-s, instead they are {{SimpleOrderedMap}}-s. This
> causes the following exception to be thrown:
> {code}
> java.lang.ClassCastException: class
> org.apache.solr.common.util.SimpleOrderedMap cannot be cast to class
> java.lang.String (org.apache.solr.common.util.SimpleOrderedMap is in unnamed
> module of loader 'app'; java.lang.String is in module java.base of loader
> 'bootstrap')
> at
> __randomizedtesting.SeedInfo.seed([1D6FB4036A639051:173F5FF19E860E6F]:0)
> at
> org.apache.solr.client.solrj.response.QueryResponse.extractDebugInfo(QueryResponse.java:246)
> at
> org.apache.solr.client.solrj.response.QueryResponse.setResponse(QueryResponse.java:143)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:987)
> ...
> {code}
> The simple fix is to change the type of this field to {{Map}}
> but this would change the public API of {{QueryResponse.getExplainMap()}} in
> incompatible way. Still, I would argue it's worth to change it - AFAIK this
> getter is not used anywhere in the Solr's codebase, and the change makes it
> more consistent with other getters.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13759) Optimize Queries when query filtering by TRA router.field
[ https://issues.apache.org/jira/browse/SOLR-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mosh updated SOLR-13759: Status: Patch Available (was: Open) > Optimize Queries when query filtering by TRA router.field > - > > Key: SOLR-13759 > URL: https://issues.apache.org/jira/browse/SOLR-13759 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Priority: Minor > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries will be by filtering query by date range, by > that we will be able to querying the specific TRA collections taking > advantage of the TRA mechanism of partitioning data based on date. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-11960) Add collection level properties
[ https://issues.apache.org/jira/browse/SOLR-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985301#comment-16985301 ] Mia Parker edited comment on SOLR-11960 at 11/30/19 9:35 AM: - I like your content very much. I wanted this only. I have seen it many times, I used to get the answer on your website. Well! let me tell you. I work at [Allegiant Airlines|https://airlinesreservation.org/allegiant-airlines/] If you ever want to book tickets in the future, Contact Us. was (Author: airlinesreservation): I like your content very much. I wanted this only. I have seen it many times, I used to get the answer on your website. Well! let me tell you. I work at [[https://airlinesreservation.org/allegiant-airlines/] Allegiant Airlines]. If you ever want to book tickets in the future, Contact Us. > Add collection level properties > --- > > Key: SOLR-11960 > URL: https://issues.apache.org/jira/browse/SOLR-11960 > Project: Solr > Issue Type: New Feature >Reporter: Peter Rusko >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Blocker > Fix For: 7.3, 8.0 > > Attachments: SOLR-11960.patch, SOLR-11960.patch, SOLR-11960.patch, > SOLR-11960.patch, SOLR-11960.patch, SOLR-11960_2.patch > > > Solr has cluster properties, but no easy and extendable way of defining > properties that affect a single collection. Collection properties could be > stored in a single zookeeper node per collection, making it possible to > trigger zookeeper watchers for only those Solr nodes that have cores of that > collection. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11960) Add collection level properties
[ https://issues.apache.org/jira/browse/SOLR-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985301#comment-16985301 ] Mia Parker commented on SOLR-11960: --- I like your content very much. I wanted this only. I have seen it many times, I used to get the answer on your website. Well! let me tell you. I work at [[https://airlinesreservation.org/allegiant-airlines/] Allegiant Airlines]. If you ever want to book tickets in the future, Contact Us. > Add collection level properties > --- > > Key: SOLR-11960 > URL: https://issues.apache.org/jira/browse/SOLR-11960 > Project: Solr > Issue Type: New Feature >Reporter: Peter Rusko >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Blocker > Fix For: 7.3, 8.0 > > Attachments: SOLR-11960.patch, SOLR-11960.patch, SOLR-11960.patch, > SOLR-11960.patch, SOLR-11960.patch, SOLR-11960_2.patch > > > Solr has cluster properties, but no easy and extendable way of defining > properties that affect a single collection. Collection properties could be > stored in a single zookeeper node per collection, making it possible to > trigger zookeeper watchers for only those Solr nodes that have cores of that > collection. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11960) Add collection level properties
[ https://issues.apache.org/jira/browse/SOLR-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985300#comment-16985300 ] Mia Parker commented on SOLR-11960: --- I like this.. > Add collection level properties > --- > > Key: SOLR-11960 > URL: https://issues.apache.org/jira/browse/SOLR-11960 > Project: Solr > Issue Type: New Feature >Reporter: Peter Rusko >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Blocker > Fix For: 7.3, 8.0 > > Attachments: SOLR-11960.patch, SOLR-11960.patch, SOLR-11960.patch, > SOLR-11960.patch, SOLR-11960.patch, SOLR-11960_2.patch > > > Solr has cluster properties, but no easy and extendable way of defining > properties that affect a single collection. Collection properties could be > stored in a single zookeeper node per collection, making it possible to > trigger zookeeper watchers for only those Solr nodes that have cores of that > collection. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9066) Modal Strategy In Concurrent Query Execution
[ https://issues.apache.org/jira/browse/LUCENE-9066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985291#comment-16985291 ] Adrien Grand commented on LUCENE-9066: -- The high level idea makes sense to me: increasing the number of slices usually improves latency, but also hurts throughput. So when the queue fills up and you have to wait for other queries to complete before starting anyway, it probably makes sense to decrease the number of slices as an attempt to get the number of tasks in the queue low again to get latency benefits back? I wonder that getting system statistics might not be necessary: if the system gets slow then the queue will fill up naturally anyway? > Modal Strategy In Concurrent Query Execution > > > Key: LUCENE-9066 > URL: https://issues.apache.org/jira/browse/LUCENE-9066 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > When executing a query concurrently today, we do not take any sort of system > statistics into account. For e.g. if the node is under high pressure, it is > not advisable to spawn a large number of threads for a query since they are > most likely to be blocked waiting for CPU to be available. However, the > converse is that for a lightly loaded cluster, the query can consume as many > threads as required. > > This Jira tracks high level efforts in this direction. The first idea is to > account the Executor's wait queue's size as a factor when allocating slices > to a query's segments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
