[GitHub] [lucene-solr] atris commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
atris commented on a change in pull request #1906: URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r493238083 ## File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.solr.servlet; Review comment: RequestRateLimiter lives within this package -- and since they directly and only work with SDF, should they not belong here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
atris commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r493214015
##
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj> obj) {
// Need to reset to null first otherwise the mappings in
placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the
config, that's all we do.
clusterProperties.setClusterProperties(
-
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+if (!unset) {
Review comment:
I believe that if we dont set it to null before writing, the new
properties will be appended to existing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
atris commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r493207484
##
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj> obj) {
// Need to reset to null first otherwise the mappings in
placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the
config, that's all we do.
clusterProperties.setClusterProperties(
-
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+if (!unset) {
+ clusterProperties.setClusterProperties(
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
placementPluginConfig));
+}
+
+ } catch (Exception e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error
in API", e);
+ }
+}
+
+@Command(name = "set-ratelimiters")
+public void setRateLimiters(PayloadObj> obj) {
+ Map rateLimiterConfig = obj.getDataMap();
+ final boolean unset = rateLimiterConfig.isEmpty();
Review comment:
I actually prefer {}, seems cleaner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493112985
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException {
[GitHub] [lucene-solr] mocobeta commented on pull request #1836: LUCENE-9317: Clean up split package in analyzers-common
mocobeta commented on pull request #1836: URL: https://github.com/apache/lucene-solr/pull/1836#issuecomment-696524047 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob closed pull request #1176: LUCENE-9143 Add error-prone checks to build, but disabled
madrob closed pull request #1176: URL: https://github.com/apache/lucene-solr/pull/1176 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul closed pull request #1880: SOLR-14151: Do not reload core for schema changes
noblepaul closed pull request #1880: URL: https://github.com/apache/lucene-solr/pull/1880 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on a change in pull request #1905: LUCENE-9488 Release with Gradle Part 2
madrob commented on a change in pull request #1905:
URL: https://github.com/apache/lucene-solr/pull/1905#discussion_r492937930
##
File path: lucene/build.gradle
##
@@ -15,8 +15,56 @@
* limitations under the License.
*/
+// Should we do this as :lucene:packaging similar to how Solr does it?
+// Or is this fine here?
+
+plugins {
+ id 'distribution'
+}
+
description = 'Parent project for Apache Lucene Core'
subprojects {
group "org.apache.lucene"
-}
\ No newline at end of file
+}
+
+distributions {
+ main {
+ // This is empirically wrong, but it is mostly a copy from `ant
package-zip`
Review comment:
I tried looking at the solr packaging for figuring this out, but kept
getting duplicate entries in the archive. I left the TODOs for what still
remains to be added to the zip, can you give me a little more direction on what
the implementation would look like?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] sigram commented on a change in pull request #1864: SOLR-14850 ExactStatsCache NullPointerException when shards.tolerant=true
sigram commented on a change in pull request #1864:
URL: https://github.com/apache/lucene-solr/pull/1864#discussion_r492706938
##
File path: solr/core/src/java/org/apache/solr/search/stats/ExactStatsCache.java
##
@@ -94,6 +94,12 @@ protected ShardRequest
doRetrieveStatsRequest(ResponseBuilder rb) {
protected void doMergeToGlobalStats(SolrQueryRequest req,
List responses) {
Set allTerms = new HashSet<>();
for (ShardResponse r : responses) {
+ if
("true".equalsIgnoreCase(req.getParams().get(ShardParams.SHARDS_TOLERANT)) &&
r.getException() != null) {
Review comment:
@Hronom I'd like to merge this PR soon, can you add a couple lines to
`TestDefaultStatsCache` to verify this fix works?
`BaseDistributedSearchTestCase` already creates 3 "dead servers", which you can
explicitly add to the query params; `shards` parameter using its
`setDistributedParams(params)` method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on a change in pull request #1877:
URL: https://github.com/apache/lucene-solr/pull/1877#discussion_r492990280
##
File path:
solr/core/src/test/org/apache/solr/request/macro/TestMacroExpander.java
##
@@ -143,15 +142,31 @@ public void testMapExprExpandOn() {
String oldVal = System.getProperty("StreamingExpressionMacros","false");
System.setProperty("StreamingExpressionMacros", "true");
try {
- @SuppressWarnings({"rawtypes"})
- Map expanded = MacroExpander.expand(request);
- assertEquals("zero", ((String[])expanded.get("fq"))[0]);
- assertEquals("one", ((String[])expanded.get("fq"))[1]);
- assertEquals("two", ((String[]) expanded.get("fq"))[2]);
- assertEquals("three", ((String[]) expanded.get("fq"))[3]);
- assertEquals("one", ((String[])expanded.get("expr"))[0]);
+ Map expanded = MacroExpander.expand(request);
+ assertEquals("zero", expanded.get("fq")[0]);
+ assertEquals("one", expanded.get("fq")[1]);
+ assertEquals("two", expanded.get("fq")[2]);
+ assertEquals("three", expanded.get("fq")[3]);
+ assertEquals("one", expanded.get("expr")[0]);
} finally {
System.setProperty("StreamingExpressionMacros", oldVal);
}
}
+
+ @Test
+ public void testUnbalanced() { // SOLR-13181
+final MacroExpander meSkipOnMissingParams = new
MacroExpander(Collections.emptyMap());
+final MacroExpander meFailOnMissingParams = new
MacroExpander(Collections.emptyMap(), true);
+assertEquals("${noClose", meSkipOnMissingParams.expand("${noClose"));
Review comment:
Ah; thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] arafalov commented on pull request #1900: SOLR-14036: Remove explicit distrib=false from /terms handler
arafalov commented on pull request #1900: URL: https://github.com/apache/lucene-solr/pull/1900#issuecomment-696402318 > @arafalov > There is one already which I have modified(DistributedTermsComponentTest) - https://github.com/apache/lucene-solr/pull/1900/files#diff-9b9ccbcf271c6320902d92479615f4fbR73 Thank you for clarification. I saw that code change, but did not realize it answered my question. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley merged pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley merged pull request #1877: URL: https://github.com/apache/lucene-solr/pull/1877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
mikemccand commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493041886
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException
[GitHub] [lucene-solr] noblepaul commented on pull request #1898: SOLR-14882
noblepaul commented on pull request #1898: URL: https://github.com/apache/lucene-solr/pull/1898#issuecomment-696676802 can you please add a title? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1909: LUCENE-9539: Remove caches from SortingCodecReader
jimczi commented on a change in pull request #1909:
URL: https://github.com/apache/lucene-solr/pull/1909#discussion_r492921270
##
File path: lucene/core/src/java/org/apache/lucene/index/SortingCodecReader.java
##
@@ -41,21 +41,13 @@
* {@link Sort}. This can be used to re-sort and index after it's been created
by wrapping all
* readers of the index with this reader and adding it to a fresh IndexWriter
via
* {@link IndexWriter#addIndexes(CodecReader...)}.
+ * NOTE: This reader should only be used for merging. Pulling fields from this
ready might be very costly and memory
Review comment:
nit:
```suggestion
* NOTE: This reader should only be used for merging. Pulling fields from
this reader might be very costly and memory
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1733: LUCENE-9450 Use BinaryDocValues in the taxonomy writer
mikemccand commented on pull request #1733: URL: https://github.com/apache/lucene-solr/pull/1733#issuecomment-696781342 > So I propose we get rid of the fullPathField altogether. Wow, +1, this looks like it is (pre-existingly?) double-indexed? Maybe we should do this as a separate pre-cursor PR to this one (switch to `StoredField` when indexing the `fullPathField`)? > For maintaining backwards compatibility, we can read facet labels from new BinaryDocValues field, falling back to old StoredField if BinaryDocValues field does not exist or has no value for the docId. The performance penalty of doing so should be acceptable. Yeah +1 to, on a hit by hit basis, try `BinaryDocValues` first, and then fallback to the `StoredField`. This is the cost of backwards compatibility ... though, for a fully new (all `BinaryDocValues`) index, the performance should be fine. Also, note that in Lucene 10.x we can remove that back-compat fallback. > Alternatively we can implement a special merge policy that takes care of moving data from old Stored field to BinaryDocValues field at the time of merge but that might be tricky to implement. I think this would indeed be tricky. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] cpoerschke commented on a change in pull request #1877: SOLR-13181: param macro expansion could throw
cpoerschke commented on a change in pull request #1877:
URL: https://github.com/apache/lucene-solr/pull/1877#discussion_r492843243
##
File path:
solr/core/src/test/org/apache/solr/request/macro/TestMacroExpander.java
##
@@ -143,15 +142,31 @@ public void testMapExprExpandOn() {
String oldVal = System.getProperty("StreamingExpressionMacros","false");
System.setProperty("StreamingExpressionMacros", "true");
try {
- @SuppressWarnings({"rawtypes"})
- Map expanded = MacroExpander.expand(request);
- assertEquals("zero", ((String[])expanded.get("fq"))[0]);
- assertEquals("one", ((String[])expanded.get("fq"))[1]);
- assertEquals("two", ((String[]) expanded.get("fq"))[2]);
- assertEquals("three", ((String[]) expanded.get("fq"))[3]);
- assertEquals("one", ((String[])expanded.get("expr"))[0]);
+ Map expanded = MacroExpander.expand(request);
+ assertEquals("zero", expanded.get("fq")[0]);
+ assertEquals("one", expanded.get("fq")[1]);
+ assertEquals("two", expanded.get("fq")[2]);
+ assertEquals("three", expanded.get("fq")[3]);
+ assertEquals("one", expanded.get("expr")[0]);
} finally {
System.setProperty("StreamingExpressionMacros", oldVal);
}
}
+
+ @Test
+ public void testUnbalanced() { // SOLR-13181
+final MacroExpander meSkipOnMissingParams = new
MacroExpander(Collections.emptyMap());
+final MacroExpander meFailOnMissingParams = new
MacroExpander(Collections.emptyMap(), true);
+assertEquals("${noClose", meSkipOnMissingParams.expand("${noClose"));
Review comment:
`${goodAnswer} ${noClose` is what I had in mind. Pushed a corresponding
commit to the PR's branch, hope you don't mind.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] arafalov merged pull request #1896: SOLR-14880: Support coreRootDirectory setting when create new cores from command line, in standalone mode
arafalov merged pull request #1896: URL: https://github.com/apache/lucene-solr/pull/1896 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob merged pull request #1910: Use github actions cache
madrob merged pull request #1910: URL: https://github.com/apache/lucene-solr/pull/1910 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1908: LUCENE-9539: Use more compact datastructures for sorting doc-values
dweiss commented on pull request #1908: URL: https://github.com/apache/lucene-solr/pull/1908#issuecomment-696652609 Nice. +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on pull request #1903: Fix bug in sort optimization
mayya-sharipova commented on pull request #1903: URL: https://github.com/apache/lucene-solr/pull/1903#issuecomment-696373592 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #1880: SOLR-14151: Do not reload core for schema changes
noblepaul commented on pull request #1880: URL: https://github.com/apache/lucene-solr/pull/1880#issuecomment-696677088 Not fixing right now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9444) Need an API to easily fetch facet labels for a field in a document
[
https://issues.apache.org/jira/browse/LUCENE-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200529#comment-17200529
]
Ankur edited comment on LUCENE-9444 at 9/23/20, 4:34 AM:
-
Thanks [~mikemccand], I incorporated the code review feedback and
* Replaced {{assert} with {{IllegalArgumentException}} for invalid inputs
* Added javadoc notes explaining that returned _FacetLabels_ may not be in the
same order in which they were indexed.
* Enhanced {{TestTaxonomyFacetCount.testRandom()}} method to exercise the API
to get facet labels for each matching document.
* Here is the updated PR -
https://github.com/apache/lucene-solr/pull/1893/commits/75ff251ebac9034c93edbb43dcf5d8dd0f1058ae
was (Author: goankur):
[~mikemccand], I incorporated the code review feedback and
* Replaced {{assert} with {{IllegalArgumentException}} for invalid inputs
* Added javadoc notes explaining that returned _FacetLabels_ may not be in the
same order in which they were indexed.
* Enhanced {{TestTaxonomyFacetCount.testRandom()}} method to exercise the API
to get facet labels for each matching document.
* Here is the updated PR -
https://github.com/apache/lucene-solr/pull/1893/commits/75ff251ebac9034c93edbb43dcf5d8dd0f1058ae
> Need an API to easily fetch facet labels for a field in a document
> --
>
> Key: LUCENE-9444
> URL: https://issues.apache.org/jira/browse/LUCENE-9444
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/facet
>Affects Versions: 8.6
>Reporter: Ankur
>Priority: Major
> Labels: facet
> Attachments: LUCENE-9444.patch, LUCENE-9444.patch,
> LUCENE-9444.v2.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> A facet field may be included in the list of fields whose values are to be
> returned for each hit.
> In order to get the facet labels for each hit we need to
> # Create an instance of _DocValuesOrdinalsReader_ and invoke
> _getReader(LeafReaderContext context)_ method to obtain an instance of
> _OrdinalsSegmentReader()_
> # _OrdinalsSegmentReader.get(int docID, IntsRef ordinals)_ method is then
> used to fetch and decode the binary payload in the document's BinaryDocValues
> field. This provides the ordinals that refer to facet labels in the
> taxonomy.**
> # Lastly TaxonomyReader.getPath(ord) is used to fetch the labels to be
> returned.
>
> Ideally there should be a simple API - *String[] getLabels(docId)* that hides
> all the above details and gives us the string labels. This can be part of
> *TaxonomyFacets* but that's just one idea.
> I am opening this issue to get community feedback and suggestions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9444) Need an API to easily fetch facet labels for a field in a document
[
https://issues.apache.org/jira/browse/LUCENE-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200529#comment-17200529
]
Ankur commented on LUCENE-9444:
---
[~mikemccand], I incorporated the code review feedback and
* Replaced {{assert} with {{IllegalArgumentException}} for invalid inputs
* Added javadoc notes explaining that returned _FacetLabels_ may not be in the
same order in which they were indexed.
* Enhanced {{TestTaxonomyFacetCount.testRandom()}} method to exercise the API
to get facet labels for each matching document.
* Here is the updated PR -
https://github.com/apache/lucene-solr/pull/1893/commits/75ff251ebac9034c93edbb43dcf5d8dd0f1058ae
> Need an API to easily fetch facet labels for a field in a document
> --
>
> Key: LUCENE-9444
> URL: https://issues.apache.org/jira/browse/LUCENE-9444
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/facet
>Affects Versions: 8.6
>Reporter: Ankur
>Priority: Major
> Labels: facet
> Attachments: LUCENE-9444.patch, LUCENE-9444.patch,
> LUCENE-9444.v2.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> A facet field may be included in the list of fields whose values are to be
> returned for each hit.
> In order to get the facet labels for each hit we need to
> # Create an instance of _DocValuesOrdinalsReader_ and invoke
> _getReader(LeafReaderContext context)_ method to obtain an instance of
> _OrdinalsSegmentReader()_
> # _OrdinalsSegmentReader.get(int docID, IntsRef ordinals)_ method is then
> used to fetch and decode the binary payload in the document's BinaryDocValues
> field. This provides the ordinals that refer to facet labels in the
> taxonomy.**
> # Lastly TaxonomyReader.getPath(ord) is used to fetch the labels to be
> returned.
>
> Ideally there should be a simple API - *String[] getLabels(docId)* that hides
> all the above details and gives us the string labels. This can be part of
> *TaxonomyFacets* but that's just one idea.
> I am opening this issue to get community feedback and suggestions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1836: LUCENE-9317: Clean up split package in analyzers-common
dweiss commented on pull request #1836: URL: https://github.com/apache/lucene-solr/pull/1836#issuecomment-696537432 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley merged pull request #1838: SOLR-14768: Fix multipart POST to Solr.
dsmiley merged pull request #1838: URL: https://github.com/apache/lucene-solr/pull/1838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] arafalov commented on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
arafalov commented on pull request #1863: URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-696704951 @noblepaul Good point: 1. **bin/solr start** 2. Edit the default configset's add-schema-fields definition to - Replace AddSchema... with GuessSchema... URP - Remove _default_ parameter (code automatically widens to "String" as last step) - Remove _Integer_ mapping, where _Long_ already exists (code automatically widens to nearest acceptable definition for numerics) 3. **bin/solr create -c guess** 4. **bin/post -c guess -params "guess-schema=true" example/exampledocs/*.xml** 5. As post does indexing and then commit, it triggers both phases of the URP 6. At the point, the schema is created (based on all data) but nothing is actually indexed; the user can review and adjust the schema without any issues 7. Flag value can actually be anything for now, in the future I was thinking that maybe this could be a tag to mark new fields (see SOLR-14855) 8. **bin/post -c guess example/exampledocs/*.xml** 9. Now the data is entered and can be searched 10. **bin/post -c guess -params "guess-schema=true" example/films/films.json** 11. **bin/post -c guess example/films/films.json** 12. This shows that we don't need the "extra steps" instructions, the films example current requires. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
noblepaul commented on pull request #1863: URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-696679556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
noblepaul commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r492668757
##
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj> obj) {
// Need to reset to null first otherwise the mappings in
placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the
config, that's all we do.
clusterProperties.setClusterProperties(
-
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+if (!unset) {
+ clusterProperties.setClusterProperties(
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
placementPluginConfig));
+}
+
+ } catch (Exception e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error
in API", e);
+ }
+}
+
+@Command(name = "set-ratelimiters")
+public void setRateLimiters(PayloadObj> obj) {
+ Map rateLimiterConfig = obj.getDataMap();
+ final boolean unset = rateLimiterConfig.isEmpty();
Review comment:
wouldn't you do `set-ratelimters : null` instead of `set-ratelimters :
{}`
##
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj> obj) {
// Need to reset to null first otherwise the mappings in
placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the
config, that's all we do.
clusterProperties.setClusterProperties(
-
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+if (!unset) {
+ clusterProperties.setClusterProperties(
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
placementPluginConfig));
+}
+
+ } catch (Exception e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error
in API", e);
+ }
+}
+
+@Command(name = "set-ratelimiters")
Review comment:
why plural? are there multiple rate limiters?
##
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj> obj) {
// Need to reset to null first otherwise the mappings in
placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the
config, that's all we do.
clusterProperties.setClusterProperties(
-
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
null));
+if (!unset) {
+ clusterProperties.setClusterProperties(
+
Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY,
placementPluginConfig));
+}
+
+ } catch (Exception e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error
in API", e);
+ }
+}
+
+@Command(name = "set-ratelimiters")
+public void setRateLimiters(PayloadObj> obj) {
Review comment:
Why not have a strongly typed java object? why are you using a loose
`Map` object
##
File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.servlet;
Review comment:
why is this in the `o.a.s.servlet` package? It's not a servlet/filter
##
File path: solr/solr-ref-guide/src/rate-limiters.adoc
##
@@ -31,67 +31,44 @@ pronounced under high stress in production workloads. The
current implementation
resources for indexing.
[GitHub] [lucene-solr] TomMD commented on a change in pull request #1901: SOLR-14883 Add a Muse (Continuous assurance platform) configuration
TomMD commented on a change in pull request #1901: URL: https://github.com/apache/lucene-solr/pull/1901#discussion_r492995357 ## File path: .muse/config.toml ## @@ -0,0 +1 @@ +jdk11 = true Review comment: I've added the comment to the file as well now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #1861: SOLR-10391: Add overwrite option to UPLOAD ConfigSet action
tflobbe commented on a change in pull request #1861:
URL: https://github.com/apache/lucene-solr/pull/1861#discussion_r492423507
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/ConfigSetsHandler.java
##
@@ -170,21 +176,90 @@ private void handleConfigUploadRequest(SolrQueryRequest
req, SolrQueryResponse r
// Create a node for the configuration in zookeeper
boolean trusted = getTrusted(req);
-zkClient.makePath(configPathInZk, ("{\"trusted\": " +
Boolean.toString(trusted) + "}").
-getBytes(StandardCharsets.UTF_8), true);
+Set filesToDelete = Collections.emptySet();
+if (overwritesExisting) {
+ if (!trusted) {
+ensureOverwritingUntrustedConfigSet(zkClient, configPathInZk);
+ }
+ if (req.getParams().getBool(ConfigSetParams.CLEANUP, false)) {
+filesToDelete = getAllConfigsetFiles(zkClient, configPathInZk);
+ }
+} else {
+ zkClient.makePath(configPathInZk, ("{\"trusted\": " +
Boolean.toString(trusted) + "}").
+ getBytes(StandardCharsets.UTF_8), true);
+}
ZipInputStream zis = new ZipInputStream(inputStream,
StandardCharsets.UTF_8);
ZipEntry zipEntry = null;
while ((zipEntry = zis.getNextEntry()) != null) {
String filePathInZk = configPathInZk + "/" + zipEntry.getName();
+ if (filePathInZk.endsWith("/")) {
+filesToDelete.remove(filePathInZk.substring(0, filePathInZk.length()
-1));
+ } else {
+filesToDelete.remove(filePathInZk);
+ }
if (zipEntry.isDirectory()) {
-zkClient.makePath(filePathInZk, true);
+zkClient.makePath(filePathInZk, false, true);
Review comment:
Yes, I don't think this would be needed at this point. No API would set
this, and no Solr component would read it. I guess the only case would be a
custom component that writes and reads directly to ZooKeeper.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on pull request #1877: URL: https://github.com/apache/lucene-solr/pull/1877#issuecomment-696943683 @gus-asf (in reference to you're fix in SOLR-12891): I can appreciate you didn't want to break back-compat. In this issue/PR, I'm not getting at that matter at all, it's a separate bug fix when the syntax is given wrong. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson commented on pull request #1733: LUCENE-9450 Use BinaryDocValues in the taxonomy writer
ErickErickson commented on pull request #1733: URL: https://github.com/apache/lucene-solr/pull/1733#issuecomment-696828850 > On Sep 22, 2020, at 11:04 AM, Michael McCandless wrote: > > > So I propose we get rid of the fullPathField altogether. > > Wow, +1, this looks like it is (pre-existingly?) double-indexed? Maybe we should do this as a separate pre-cursor PR to this one (switch to StoredField when indexing the fullPathField)? > > For maintaining backwards compatibility, we can read facet labels from new BinaryDocValues field, falling back to old StoredField if BinaryDocValues field does not exist or has no value for the docId. The performance penalty of doing so should be acceptable. > > Yeah +1 to, on a hit by hit basis, try BinaryDocValues first, and then fallback to the StoredField. This is the cost of backwards compatibility ... though, for a fully new (all BinaryDocValues) index, the performance should be fine. Also, note that in Lucene 10.x we can remove that back-compat fallback. > > Alternatively we can implement a special merge policy that takes care of moving data from old Stored field to BinaryDocValues field at the time of merge but that might be tricky to implement. > > I think this would indeed be tricky. Andrzej and I spent quite a bit of time trying to get something similar to work for adding docValues on the fly using a custom merge policy. We realized that you could create a docValues field from an indexed field for primitive types since all the information was already in the index. We never could get it working if there was active indexing happening, so resorted to a batch process that rewrote all segments doing the transformation along the way that had to be run on a quiescent index, the client decided that was good enough and didn’t want to spend more time on it. Our best guess was that there was a race condition that we somehow couldn’t find in the time allowed… Mostly just FYI... FWIW, Erick > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or unsubscribe. > This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe merged pull request #1861: SOLR-10391: Add overwrite option to UPLOAD ConfigSet action
tflobbe merged pull request #1861: URL: https://github.com/apache/lucene-solr/pull/1861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #1901: SOLR-14883 Add a Muse (Continuous assurance platform) configuration
tflobbe commented on a change in pull request #1901: URL: https://github.com/apache/lucene-solr/pull/1901#discussion_r492911126 ## File path: .muse/config.toml ## @@ -0,0 +1 @@ +jdk11 = true Review comment: Ah, I noticed you added the comments on the commit. Thanks for that, but I was thinking we could put some comment in the file itself, something like: ``` # This file can configure Muse PR static analysis. # Docs for adding, removing or changing checks can be found here: https://docs.muse.dev/docs/repository-configuration/ ``` or something of that sort, mostly to save some googling. WDYT? @dsmiley, do you have any concerns with this? (since you commented on the Jira issue). I seems like a great thing to try, and if Apache infra installed the app for us, I see no reason not to do it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul edited a comment on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
noblepaul edited a comment on pull request #1863: URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-697054163 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] gus-asf commented on pull request #1877: SOLR-13181: param macro expansion could throw
gus-asf commented on pull request #1877: URL: https://github.com/apache/lucene-solr/pull/1877#issuecomment-696867296 Did see your review request, but my life is doing that whole "it never rains but it pours" thing right now and I keep not getting to this. My primary concern wrt to this class is the need to ensure that default installations do not expand streaming expression parameters (but folks who need that for existing installs can turn it on with knowledge of the risks). I saw your comment on the ticket that fixed that originally, and yeah calling out a single parameter is yucky, but it was the least back compat breaking thing I could come up with. A more sensible, less backwards compatible solution for 9x is certainly a possibility. I may not have time to do any deep review here soon enough, so don't wait on me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1908: LUCENE-9539: Use more compact datastructures for sorting doc-values
dweiss commented on a change in pull request #1908:
URL: https://github.com/apache/lucene-solr/pull/1908#discussion_r492648887
##
File path:
lucene/core/src/java/org/apache/lucene/index/SortedSetDocValuesWriter.java
##
@@ -379,4 +352,31 @@ public long getValueCount() {
return in.getValueCount();
}
}
+
+ static final class DocOrds {
+final long[] offsets;
+final PackedLongValues ords;
+
+DocOrds(int maxDoc, Sorter.DocMap sortMap, SortedSetDocValues oldValues,
float acceptableOverheadRatio) throws IOException {
+ offsets = new long[maxDoc];
+ PackedLongValues.Builder builder =
PackedLongValues.packedBuilder(acceptableOverheadRatio);
+ long ordOffset = 1; // 0 marks docs with no values
+ int docID;
+ while ((docID = oldValues.nextDoc()) != NO_MORE_DOCS) {
+int newDocID = sortMap.oldToNew(docID);
+long startOffset = ordOffset;
+long ord;
+while ((ord = oldValues.nextOrd()) != NO_MORE_ORDS) {
+ builder.add(ord + 1);
+ ordOffset++;
+}
+if (startOffset != ordOffset) { // do we have any values?
+ offsets[newDocID] = startOffset;
+ builder.add(0); // 0 ord marks next value
Review comment:
Technically you could also have a sign switch as boundary (~value marks
first value) but I don't think it'd result in any savings and would only
complicate the value stream.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1908: LUCENE-9539: Use more compact datastructures for sorting doc-values
s1monw commented on a change in pull request #1908:
URL: https://github.com/apache/lucene-solr/pull/1908#discussion_r492719968
##
File path:
lucene/core/src/java/org/apache/lucene/index/SortedSetDocValuesWriter.java
##
@@ -379,4 +352,31 @@ public long getValueCount() {
return in.getValueCount();
}
}
+
+ static final class DocOrds {
+final long[] offsets;
+final PackedLongValues ords;
+
+DocOrds(int maxDoc, Sorter.DocMap sortMap, SortedSetDocValues oldValues,
float acceptableOverheadRatio) throws IOException {
+ offsets = new long[maxDoc];
+ PackedLongValues.Builder builder =
PackedLongValues.packedBuilder(acceptableOverheadRatio);
+ long ordOffset = 1; // 0 marks docs with no values
+ int docID;
+ while ((docID = oldValues.nextDoc()) != NO_MORE_DOCS) {
+int newDocID = sortMap.oldToNew(docID);
+long startOffset = ordOffset;
+long ord;
+while ((ord = oldValues.nextOrd()) != NO_MORE_ORDS) {
+ builder.add(ord + 1);
+ ordOffset++;
+}
+if (startOffset != ordOffset) { // do we have any values?
+ offsets[newDocID] = startOffset;
+ builder.add(0); // 0 ord marks next value
Review comment:
agreed, I didn't do it since it would have impacted readability too much
IMO
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] arafalov merged pull request #1904: SOLR-14878: Expose solr.xml's coreRootDirectory property via the System Settings API (part 2)
arafalov merged pull request #1904: URL: https://github.com/apache/lucene-solr/pull/1904 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc merged pull request #1902: SOLR-14840: Overseer doc in asciidoc
murblanc merged pull request #1902: URL: https://github.com/apache/lucene-solr/pull/1902 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on pull request #1861: SOLR-10391: Add overwrite option to UPLOAD ConfigSet action
tflobbe commented on pull request #1861: URL: https://github.com/apache/lucene-solr/pull/1861#issuecomment-696456676 I plan to merge this tomorrow This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw merged pull request #1908: LUCENE-9539: Use more compact datastructures for sorting doc-values
s1monw merged pull request #1908: URL: https://github.com/apache/lucene-solr/pull/1908 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] goankur commented on pull request #1733: LUCENE-9450 Use BinaryDocValues in the taxonomy writer
goankur commented on pull request #1733: URL: https://github.com/apache/lucene-solr/pull/1733#issuecomment-696420384 Thanks @gautamworah96 for this impactful change and @mikemccand for reviewing it. A few thoughts 1. This change disables STORED fields part but keeps the POSTINGS part here `fullPathField = new StringField(Consts.FULL, "", Field.Store.NO); ` which is unnecessary as postings are already enabled for facet labels in [FacetsConfig#L364-L399](https://github.com/apache/lucene-solr/blob/master/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L364-L366) including [dimension drill-down](https://github.com/apache/lucene-solr/blob/master/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L88). So I propose we get rid of the `fullPathField` altogether. 2. For maintaining backwards compatibility, we can read facet labels from new BinaryDocValues field, falling back to old StoredField if BinaryDocValues field does not exist or has no value for the docId. The performance penalty of doing so should be acceptable. Alternatively we can implement a special merge policy that takes care of moving data from old Stored field to BinaryDocValues field at the time of merge but that might be tricky to implement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1905: LUCENE-9488 Release with Gradle Part 2
dweiss commented on a change in pull request #1905:
URL: https://github.com/apache/lucene-solr/pull/1905#discussion_r492507019
##
File path: lucene/build.gradle
##
@@ -15,8 +15,56 @@
* limitations under the License.
*/
+// Should we do this as :lucene:packaging similar to how Solr does it?
Review comment:
Yes, please move it to :lucene:packaging. It makes things much cleaner
in the long run.
##
File path: lucene/build.gradle
##
@@ -15,8 +15,56 @@
* limitations under the License.
*/
+// Should we do this as :lucene:packaging similar to how Solr does it?
+// Or is this fine here?
+
+plugins {
+ id 'distribution'
+}
+
description = 'Parent project for Apache Lucene Core'
subprojects {
group "org.apache.lucene"
-}
\ No newline at end of file
+}
+
+distributions {
+ main {
+ // This is empirically wrong, but it is mostly a copy from `ant
package-zip`
Review comment:
This is wrong. It should rely on published artifacts from other
subprojects (set up proper dependencies) and assemble them in the right way
into the distribution layout.
##
File path: lucene/build.gradle
##
@@ -15,8 +15,56 @@
* limitations under the License.
*/
+// Should we do this as :lucene:packaging similar to how Solr does it?
+// Or is this fine here?
+
+plugins {
+ id 'distribution'
+}
+
description = 'Parent project for Apache Lucene Core'
subprojects {
group "org.apache.lucene"
-}
\ No newline at end of file
+}
+
+distributions {
+ main {
+ // This is empirically wrong, but it is mostly a copy from `ant
package-zip`
Review comment:
Sure but you'll have to be patient - can't do it right away.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova edited a comment on pull request #1903: Fix bug in sort optimization
mayya-sharipova edited a comment on pull request #1903: URL: https://github.com/apache/lucene-solr/pull/1903#issuecomment-696976768 @jimczi I was able to reproduce a bug, and in aaab92a added a test that reproducibly fails for me without this patch. It is a very particular combination of docs and queries to have a `BitSetIterator` as a `scorerIterator` that also ensures to go through `DefaultBulkSorter::scoreRange` section for scoring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ctargett commented on pull request #1869: SOLR-14866 autowidth tables in ref guide
ctargett commented on pull request #1869: URL: https://github.com/apache/lucene-solr/pull/1869#issuecomment-696366569 I'm a little confused what's going on here at this point. It seems the latest commits increase the mixing of styles, which as I said is fine with me in general, but sort of counter to what you said you wanted to do. There are 3 tables in `charfilterfactories.adoc` for example, and two are now changed to remove the `[options="header"]` and add a blank line between header row and data rows, but the header param remains on the 3rd table which also includes the `%autowidth.spread,width="100%"` parameters which I thought I showed aren't necessary and are already the default. Despite all that, the `asciidoc-syntax.adoc` is updated to imply the best practice is defining the autospread, width, and header params explicitly, even though other tables in this PR were changed to definitely not do that. It stops short of saying it's a best practice, though, it just points out that one might see it (when before it was more direct in what it recommended writers to do). So, I'm no longer sure what's really being achieved here. Removing TODOs that aren't needed anymore because they're holdovers from PDF days is a laudable goal and helpful cleanup, but is it any more clear now what both the internal docs and examples within the content would recommend someone do to define tables? I know that wasn't the original intent, but it got turned into a goal and now it's not doing that. At this point I'm not sure if I should approve the PR (since I don't think it's ultimately that big a deal) or if I should look to hold the PR to some level of internal consistency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta edited a comment on pull request #1836: LUCENE-9317: Clean up split package in analyzers-common
mocobeta edited a comment on pull request #1836: URL: https://github.com/apache/lucene-solr/pull/1836#issuecomment-696590916 It should be able to create a subproject for classes I moved to core here. > this would make the core really separate from analysis. I don't fully understand that... we cannot completely remove `o.a.l.analysis` from lucene-core in any case (e.g., `o.a.l.document.Field` uses `o.a.l.analysis.Analyzer`). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493148293
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException {
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493137058
##
File path:
lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestTaxonomyLabels.java
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.facet.FacetField;
+import org.apache.lucene.facet.FacetTestCase;
+import org.apache.lucene.facet.FacetsCollector;
+import org.apache.lucene.facet.FacetsCollector.MatchingDocs;
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyReader;
+import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.RandomIndexWriter;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.IOUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class TestTaxonomyLabels extends FacetTestCase {
+
+ private List prepareDocuments() {
+List docs = new ArrayList<>();
+
+Document doc = new Document();
+doc.add(new FacetField("Author", "Bob"));
+doc.add(new FacetField("Publish Date", "2010", "10", "15"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Lisa"));
+doc.add(new FacetField("Publish Date", "2010", "10", "20"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Tom"));
+doc.add(new FacetField("Publish Date", "2012", "1", "1"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Susan"));
+doc.add(new FacetField("Publish Date", "2012", "1", "7"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Frank"));
+doc.add(new FacetField("Publish Date", "1999", "5", "5"));
+docs.add(doc);
+
+return docs;
+ }
+
+ private List allDocIds(MatchingDocs m, boolean decreasingDocIds)
throws IOException {
+DocIdSetIterator disi = m.bits.iterator();
+List docIds = new ArrayList<>();
+while (disi.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {
+ docIds.add(disi.docID());
+}
+
+if (decreasingDocIds == true) {
+ Collections.reverse(docIds);
+}
+return docIds;
+ }
+
+ private List lookupFacetLabels(TaxonomyFacetLabels taxoLabels,
+ List matchingDocs)
throws IOException {
+return lookupFacetLabels(taxoLabels, matchingDocs, null, false);
+ }
+
+ private List lookupFacetLabels(TaxonomyFacetLabels taxoLabels,
+ List matchingDocs,
+ String dimension) throws
IOException {
+return lookupFacetLabels(taxoLabels, matchingDocs, dimension, false);
+ }
+
+ private List lookupFacetLabels(TaxonomyFacetLabels taxoLabels,
List matchingDocs, String dimension,
+ boolean decreasingDocIds) throws
IOException {
+List facetLabels = new ArrayList<>();
+
+for (MatchingDocs m : matchingDocs) {
+ TaxonomyFacetLabels.FacetLabelReader facetLabelReader =
taxoLabels.getFacetLabelReader(m.context);
+ List docIds = allDocIds(m, decreasingDocIds);
+ FacetLabel facetLabel;
+ for (Integer docId : docIds) {
+while (true) {
+ if (dimension != null) {
+facetLabel = facetLabelReader.nextFacetLabel(docId, dimension);
+ } else {
+facetLabel = facetLabelReader.nextFacetLabel(docId);
+ }
+
+ if (facetLabel == null) {
+break;
+ }
+ facetLabels.add(facetLabel);
+}
+ }
+}
+
+return facetLabels;
+ }
+
+
+ public void testBasic() throws Exception {
+Directory dir = newDirectory();
+Directo
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493137058
##
File path:
lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestTaxonomyLabels.java
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.facet.FacetField;
+import org.apache.lucene.facet.FacetTestCase;
+import org.apache.lucene.facet.FacetsCollector;
+import org.apache.lucene.facet.FacetsCollector.MatchingDocs;
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyReader;
+import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.RandomIndexWriter;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.IOUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class TestTaxonomyLabels extends FacetTestCase {
+
+ private List prepareDocuments() {
+List docs = new ArrayList<>();
+
+Document doc = new Document();
+doc.add(new FacetField("Author", "Bob"));
+doc.add(new FacetField("Publish Date", "2010", "10", "15"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Lisa"));
+doc.add(new FacetField("Publish Date", "2010", "10", "20"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Tom"));
+doc.add(new FacetField("Publish Date", "2012", "1", "1"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Susan"));
+doc.add(new FacetField("Publish Date", "2012", "1", "7"));
+docs.add(doc);
+
+doc = new Document();
+doc.add(new FacetField("Author", "Frank"));
+doc.add(new FacetField("Publish Date", "1999", "5", "5"));
+docs.add(doc);
+
+return docs;
+ }
+
+ private List allDocIds(MatchingDocs m, boolean decreasingDocIds)
throws IOException {
+DocIdSetIterator disi = m.bits.iterator();
+List docIds = new ArrayList<>();
+while (disi.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {
+ docIds.add(disi.docID());
+}
+
+if (decreasingDocIds == true) {
+ Collections.reverse(docIds);
+}
+return docIds;
+ }
+
+ private List lookupFacetLabels(TaxonomyFacetLabels taxoLabels,
+ List matchingDocs)
throws IOException {
+return lookupFacetLabels(taxoLabels, matchingDocs, null, false);
+ }
+
+ private List lookupFacetLabels(TaxonomyFacetLabels taxoLabels,
+ List matchingDocs,
+ String dimension) throws
IOException {
+return lookupFacetLabels(taxoLabels, matchingDocs, dimension, false);
+ }
+
+ private List lookupFacetLabels(TaxonomyFacetLabels taxoLabels,
List matchingDocs, String dimension,
+ boolean decreasingDocIds) throws
IOException {
+List facetLabels = new ArrayList<>();
+
+for (MatchingDocs m : matchingDocs) {
+ TaxonomyFacetLabels.FacetLabelReader facetLabelReader =
taxoLabels.getFacetLabelReader(m.context);
+ List docIds = allDocIds(m, decreasingDocIds);
+ FacetLabel facetLabel;
+ for (Integer docId : docIds) {
+while (true) {
+ if (dimension != null) {
+facetLabel = facetLabelReader.nextFacetLabel(docId, dimension);
+ } else {
+facetLabel = facetLabelReader.nextFacetLabel(docId);
+ }
+
+ if (facetLabel == null) {
+break;
+ }
+ facetLabels.add(facetLabel);
+}
+ }
+}
+
+return facetLabels;
+ }
+
+
+ public void testBasic() throws Exception {
+Directory dir = newDirectory();
+Directo
[GitHub] [lucene-solr] noblepaul edited a comment on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
noblepaul edited a comment on pull request #1863:
URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-697054163
> At the point, the schema is created (based on all data) but nothing is
actually indexed; the user can review and adjust the schema without any issues
How do you expect the schema to be adjusted? User has no idea what just
happened
I'm assuming Step 2 won't be required as it will be built into the end
point.
Is it possible to just return the commands to create a schema?
example
```
bin/post -c guess -params "guess-schema=true" example/films/films.json
```
the response can be
```
curl -X POST -H 'Content-type: application/json' -d '{"add-field":[
{
"name":"id",
"type":"string",
"stored":true },
{
"name":"desc",
"type":"text",
"stored":true
}
]}' http://localhost:8983/solr/guess/schema
```
The user has the following choices
1. Run the command as is
2. edit the command and post it
My point is , our guess schema is worse than useless and anything that it
creates will be WRONG for 99% of users. Why bother creating the fields in
schema when we know fully well that user will have to edit the schema in ZK
soon after
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-7520) Post filter DelegatingCollector.finish not called for multi-shard queries specifying grouping
[
https://issues.apache.org/jira/browse/SOLR-7520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200451#comment-17200451
]
Erick Erickson commented on SOLR-7520:
--
These seem identical
> Post filter DelegatingCollector.finish not called for multi-shard queries
> specifying grouping
> --
>
> Key: SOLR-7520
> URL: https://issues.apache.org/jira/browse/SOLR-7520
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: 4.8, 5.2.1
>Reporter: Eric Wheeler
>Priority: Major
> Attachments: SolrGroupTest-5.2.1.tgz, solrGroupTestEx.tgz
>
>
> The finish method for a DelegatingCollector post filter is not being called
> for queries specifying a fq filter, multiple shards, and grouping options.
> Recommend adding the following code to
> org.apache.solr.search.grouping.CommandHandler.searchWithTimeLimiter to call
> finish (if a DelegatingCollector is available) after calling search, but
> before obtaining totalHitCount:
> {code: title=org.apache.solr.search.grouping.CommandHandler line ~218}
> try {
> searcher.search(query, luceneFilter, collector);
> } catch (TimeLimitingCollector.TimeExceededException x) {
> partialResults = true;
> logger.warn( "Query: " + query + "; " + x.getMessage() );
> }
> /* Call finish of DelegatingCollector to complete post processing
>for multi-shard queries */
> if (filter.postFilter != null) {
> filter.postFilter.finish();
> }
> /* END: Call finish of DelegatingCollector to complete post processing
>for multi-shard queries */
> if (includeHitCount) {
> totalHitCount = hitCountCollector.getTotalHits();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493117522
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException {
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493114208
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException {
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493114139
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
Review comment:
done in next revision
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493113512
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException {
[GitHub] [lucene-solr] goankur commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
goankur commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493112985
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException {
[GitHub] [lucene-solr] noblepaul edited a comment on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
noblepaul edited a comment on pull request #1863:
URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-697054163
> At the point, the schema is created (based on all data) but nothing is
actually indexed; the user can review and adjust the schema without any issues
How do you expect the schema to be adjusted? User has no idea what just
happened
I'm assuming Step 2 won't be required as it will be built into the end
point.
Is it possible to just return the commands to create a schema?
example
```
bin/post -c guess -params "guess-schema=true" example/films/films.json
```
the response can be
```
curl -X POST -H 'Content-type: application/json' -d '{"add-field":[
{
"name":"id",
"type":"string",
"stored":true },
{
"name":"desc",
"type":"text",
"stored":true
}
]}' http://localhost:8983/solr/guess/config
```
The user has the following choices
1. Run the command as is
2. edit the command and post it
My point is , our guess schema is worse than useless and anything that it
creates will be WRONG for 99% of users. Why bother creating the fields in
schema when we know fully well that user will have to edit the schema in ZK
soon after
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul edited a comment on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
noblepaul edited a comment on pull request #1863: URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-697054163 > At the point, the schema is created (based on all data) but nothing is actually indexed; the user can review and adjust the schema without any issues How do you expect the schema to be adjusted? User has no idea what just happened This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #1863: SOLR-14701: GuessSchemaFields URP to replace AddSchemaFields URP in schemaless mode
noblepaul commented on pull request #1863: URL: https://github.com/apache/lucene-solr/pull/1863#issuecomment-697054163 ># 6 At the point, the schema is created (based on all data) but nothing is actually indexed; the user can review and adjust the schema without any issues How do you expect the schema to be adjusted? User has no idea what just happened This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9540) Investigate double indexing of the fullPathField in the DirectoryTaxonomyWriter
[
https://issues.apache.org/jira/browse/LUCENE-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200408#comment-17200408
]
Gautam Worah edited comment on LUCENE-9540 at 9/22/20, 10:30 PM:
-
This should ideally be a simple change to use {{StoredField}} instead of
{{StringField}} and should not break any tests.
was (Author: gworah):
This should ideally be a simple change to use `StoredField` instead of
`StringField` and should not break any tests.
> Investigate double indexing of the fullPathField in the
> DirectoryTaxonomyWriter
> ---
>
> Key: LUCENE-9540
> URL: https://issues.apache.org/jira/browse/LUCENE-9540
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Gautam Worah
>Priority: Minor
>
> We may have reason to believe that we are double indexing the fullPathField
> postings item in the DirectoryTaxonomyWriter constructor.
> This should ideally be a StoredField.
> See related discussion in PR https://github.com/apache/lucene-solr/pull/1733/
> Postings are already enabled for facet labels in
> [FacetsConfig#L364-L399|https://github.com/apache/lucene-solr/blob/master/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L364-L366]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9540) Investigate double indexing of the fullPathField in the DirectoryTaxonomyWriter
[ https://issues.apache.org/jira/browse/LUCENE-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200408#comment-17200408 ] Gautam Worah commented on LUCENE-9540: -- This should ideally be a simple change to use `StoredField` instead of `StringField` and should not break any tests. > Investigate double indexing of the fullPathField in the > DirectoryTaxonomyWriter > --- > > Key: LUCENE-9540 > URL: https://issues.apache.org/jira/browse/LUCENE-9540 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Gautam Worah >Priority: Minor > > We may have reason to believe that we are double indexing the fullPathField > postings item in the DirectoryTaxonomyWriter constructor. > This should ideally be a StoredField. > See related discussion in PR https://github.com/apache/lucene-solr/pull/1733/ > Postings are already enabled for facet labels in > [FacetsConfig#L364-L399|https://github.com/apache/lucene-solr/blob/master/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L364-L366] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14768) Multipart POST is broken; a regression since 8.6
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley resolved SOLR-14768.
-
Fix Version/s: 8.7
Resolution: Fixed
> Multipart POST is broken; a regression since 8.6
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Assignee: David Smiley
>Priority: Major
> Fix For: 8.7
>
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml,
> Solr v8.6.x fails with multipart MIME in commands.eml, Solr v8.6.x fails with
> multipart MIME in commands.eml
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
>
[jira] [Created] (LUCENE-9540) Investigate double indexing of the fullPathField in the DirectoryTaxonomyWriter
Gautam Worah created LUCENE-9540: Summary: Investigate double indexing of the fullPathField in the DirectoryTaxonomyWriter Key: LUCENE-9540 URL: https://issues.apache.org/jira/browse/LUCENE-9540 Project: Lucene - Core Issue Type: Improvement Reporter: Gautam Worah We may have reason to believe that we are double indexing the fullPathField postings item in the DirectoryTaxonomyWriter constructor. This should ideally be a StoredField. See related discussion in PR https://github.com/apache/lucene-solr/pull/1733/ Postings are already enabled for facet labels in [FacetsConfig#L364-L399|https://github.com/apache/lucene-solr/blob/master/lucene/facet/src/java/org/apache/lucene/facet/FacetsConfig.java#L364-L366] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14768) Multipart POST is broken; a regression since 8.6
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200399#comment-17200399
]
ASF subversion and git services commented on SOLR-14768:
Commit 08dcd24b17b6caa7bd49452dd30a66cfafbaaf89 in lucene-solr's branch
refs/heads/branch_8x from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=08dcd24 ]
SOLR-14768: Fix multipart POST to Solr. (#1838)
Regression from 8.6
Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart.
Solr cannot access many Jetty classes, which is not noticeable in our tests.
(cherry picked from commit 7b53671920840bb6fe6906436260d6bf906156f9)
> Multipart POST is broken; a regression since 8.6
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Assignee: David Smiley
>Priority: Major
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml,
> Solr v8.6.x fails with multipart MIME in commands.eml, Solr v8.6.x fails with
> multipart MIME in commands.eml
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v2020
[jira] [Commented] (SOLR-14768) Multipart POST is broken; a regression since 8.6
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200398#comment-17200398
]
ASF subversion and git services commented on SOLR-14768:
Commit 7b53671920840bb6fe6906436260d6bf906156f9 in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7b53671 ]
SOLR-14768: Fix multipart POST to Solr. (#1838)
Regression from 8.6
Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart.
Solr cannot access many Jetty classes, which is not noticeable in our tests.
> Multipart POST is broken; a regression since 8.6
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Assignee: David Smiley
>Priority: Major
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml,
> Solr v8.6.x fails with multipart MIME in commands.eml, Solr v8.6.x fails with
> multipart MIME in commands.eml
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.do
[GitHub] [lucene-solr] dsmiley merged pull request #1838: SOLR-14768: Fix multipart POST to Solr.
dsmiley merged pull request #1838: URL: https://github.com/apache/lucene-solr/pull/1838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1893: LUCENE-9444 Utility class to get facet labels from taxonomy for a fac…
mikemccand commented on a change in pull request #1893:
URL: https://github.com/apache/lucene-solr/pull/1893#discussion_r493041886
##
File path:
lucene/facet/src/java/org/apache/lucene/facet/taxonomy/TaxonomyFacetLabels.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy;
+
+import org.apache.lucene.facet.FacetsConfig;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.IntsRef;
+
+import java.io.IOException;
+
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.INVALID_ORDINAL;
+import static org.apache.lucene.facet.taxonomy.TaxonomyReader.ROOT_ORDINAL;
+
+/**
+ * Utility class to easily retrieve previously indexed facet labels, allowing
you to skip also adding stored fields for these values,
+ * reducing your index size.
+ *
+ * @lucene.experimental
+ **/
+public class TaxonomyFacetLabels {
+
+ /**
+ * Index field name provided to the constructor
+ */
+ private final String indexFieldName;
+
+ /**
+ * {@code TaxonomyReader} provided to the constructor
+ */
+ private final TaxonomyReader taxoReader;
+
+ /**
+ * {@code FacetsConfig} provided to the constructor
+ */
+ private final FacetsConfig config;
+
+ /**
+ * {@code OrdinalsReader} to decode ordinals previously indexed into the
{@code BinaryDocValues} facet field
+ */
+ private final OrdinalsReader ordsReader;
+
+ /**
+ * Sole constructor. Do not close the provided {@link TaxonomyReader} while
still using this instance!
+ */
+ public TaxonomyFacetLabels(TaxonomyReader taxoReader, FacetsConfig config,
String indexFieldName) throws IOException {
+this.taxoReader = taxoReader;
+this.config = config;
+this.indexFieldName = indexFieldName;
+this.ordsReader = new DocValuesOrdinalsReader(indexFieldName);
+ }
+
+ /**
+ * Create and return an instance of {@link FacetLabelReader} to retrieve
facet labels for
+ * multiple documents and (optionally) for a specific dimension. You must
create this per-segment,
+ * and then step through all hits, in order, for that segment.
+ *
+ * NOTE: This class is not thread-safe, so you must use a new
instance of this
+ * class for each thread.
+ *
+ * @param readerContext LeafReaderContext used to access the {@code
BinaryDocValues} facet field
+ * @return an instance of {@link FacetLabelReader}
+ * @throws IOException when a low-level IO issue occurs
+ */
+ public FacetLabelReader getFacetLabelReader(LeafReaderContext readerContext)
throws IOException {
+return new FacetLabelReader(ordsReader, readerContext);
+ }
+
+ /**
+ * Utility class to retrieve facet labels for multiple documents.
+ *
+ * @lucene.experimental
+ */
+ public class FacetLabelReader {
+private final OrdinalsReader.OrdinalsSegmentReader ordinalsSegmentReader;
+private final IntsRef decodedOrds = new IntsRef();
+private int currentDocId = -1;
+private int currentPos = -1;
+
+// Lazily set when nextFacetLabel(int docId, String facetDimension) is
first called
+private int[] parents;
+
+/**
+ * Sole constructor.
+ */
+public FacetLabelReader(OrdinalsReader ordsReader, LeafReaderContext
readerContext) throws IOException {
+ ordinalsSegmentReader = ordsReader.getReader(readerContext);
+}
+
+/**
+ * Retrieves the next {@link FacetLabel} for the specified {@code docId},
or {@code null} if there are no more.
+ * This method has state: if the provided {@code docId} is the same as the
previous invocation, it returns the
+ * next {@link FacetLabel} for that document. Otherwise, it advances to
the new {@code docId} and provides the
+ * first {@link FacetLabel} for that document, or {@code null} if that
document has no indexed facets. Each
+ * new {@code docId} must be in strictly monotonic (increasing) order.
+ *
+ * @param docId input docId provided in monotonic (non-decreasing) order
+ * @return the first or next {@link FacetLabel}, or {@code null} if there
are no more
+ * @throws IOException when a low-level IO issue occurs
+ */
+public FacetLabel nextFacetLabel(int docId) throws IOException
[jira] [Closed] (SOLR-9827) Make ConcurrentUpdateSolrClient create RemoteSolrException instead of just SolrException for remote errors
[ https://issues.apache.org/jira/browse/SOLR-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomas Eduardo Fernandez Lobbe closed SOLR-9827. --- > Make ConcurrentUpdateSolrClient create RemoteSolrException instead of just > SolrException for remote errors > -- > > Key: SOLR-9827 > URL: https://issues.apache.org/jira/browse/SOLR-9827 > Project: Solr > Issue Type: Improvement >Reporter: Tomas Eduardo Fernandez Lobbe >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Minor > Fix For: 6.4, 7.0 > > Attachments: SOLR-9827.patch > > > Also, improve the exception message to include the remote error message when > present. Specially when Solr is logging these errors (e.g. > DistributedUpdateProcessor), this should make it easier to understand that > the error was in the remote host and not in the one logging this exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova edited a comment on pull request #1903: Fix bug in sort optimization
mayya-sharipova edited a comment on pull request #1903: URL: https://github.com/apache/lucene-solr/pull/1903#issuecomment-696976768 @jimczi I was able to reproduce a bug, and in aaab92a added a test that reproducibly fails for me without this patch. It is a very particular combination of docs and queries to have a `BitSetIterator` as a `scorerIterator` that also ensures to go through `DefaultBulkSorter::scoreRange` section for scoring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on pull request #1903: Fix bug in sort optimization
mayya-sharipova commented on pull request #1903: URL: https://github.com/apache/lucene-solr/pull/1903#issuecomment-696976768 @jimczi I was able to reproduce a bug, and in aaab92a added a test that reproducibly fails for me without this patch. It is a very particular combination of docs and queries to have a `BitSetIterator` as a `scorerIterator` and ensure that we go through `DefaultBulkSorter::scoreRange` section for scoring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9536) Optimize OrdinalMap when one segment contains all distinct values?
[ https://issues.apache.org/jira/browse/LUCENE-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200371#comment-17200371 ] Adrien Grand commented on LUCENE-9536: -- +1 > Optimize OrdinalMap when one segment contains all distinct values? > -- > > Key: LUCENE-9536 > URL: https://issues.apache.org/jira/browse/LUCENE-9536 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Julie Tibshirani >Priority: Minor > > For doc values that are not too high cardinality, it seems common to have > some large segments that contain all distinct values (plus many small > segments who are missing some values). In this case, we could check if the > first segment ords map perfectly to global ords and if so store > `globalOrdDeltas` and `firstSegments` as `LongValues.ZEROES`. This could save > a small amount of space. > I don’t think it would help a huge amount, especially since the optimization > might only kick in with small/ medium cardinalities, which don’t create huge > `OrdinalMap` instances anyways? But it is simple and seemed worth mentioning. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9537) Add Indri Search Engine Functionality to Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200370#comment-17200370 ] Adrien Grand commented on LUCENE-9537: -- Sorry, this might be a very naive question, but would associating a smoothing score to documents when they don't match a term essentially have the same effect as subtracting this smoothing score whenever a document contains a term, and the absolute scores would be different but the ordering of hits would remain the same? I'm asking because the fact that terms that don't match don't contribute a score is hardcoded into Lucene so if there were other ways we could achieve the same effect, it might be easier to integrate. > Add Indri Search Engine Functionality to Lucene > --- > > Key: LUCENE-9537 > URL: https://issues.apache.org/jira/browse/LUCENE-9537 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Cameron VandenBerg >Priority: Major > Labels: patch > Attachments: LUCENE-INDRI.patch > > > Indri ([http://lemurproject.org/indri.php]) is an academic search engine > developed by The University of Massachusetts and Carnegie Mellon University. > The major difference between Lucene and Indri is that Indri will give a > document a "smoothing score" to a document that does not contain the search > term, which has improved the search ranking accuracy in our experiments. I > have created an Indri patch, which adds the search code needed to implement > the Indri AND logic as well as Indri's implementation of Dirichlet Smoothing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-10391) ConfigSet overwrite
[ https://issues.apache.org/jira/browse/SOLR-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomas Eduardo Fernandez Lobbe resolved SOLR-10391. -- Fix Version/s: 8.7 master (9.0) Assignee: Tomas Eduardo Fernandez Lobbe Resolution: Fixed > ConfigSet overwrite > --- > > Key: SOLR-10391 > URL: https://issues.apache.org/jira/browse/SOLR-10391 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Major > Fix For: master (9.0), 8.7 > > Time Spent: 3h > Remaining Estimate: 0h > > As a follow up to SOLR-6736, we need a way for a user to overwrite an > existing configset and/or delete an existing configset. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-10391) ConfigSet overwrite
[ https://issues.apache.org/jira/browse/SOLR-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200368#comment-17200368 ] ASF subversion and git services commented on SOLR-10391: Commit aec5292e572434ccff1176a057aa6c44295d7c76 in lucene-solr's branch refs/heads/branch_8x from Tomas Eduardo Fernandez Lobbe [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=aec5292 ] SOLR-10391: Add overwrite option to UPLOAD ConfigSet action (#1861) When set to true, Solr will overwrite an existing configset in ZooKeeper in an UPLOAD. A new cleanup parameter can also be passed to let Solr know what to do with the files that existed in the old configset, but no longer exist in the new configset (remove or keep) > ConfigSet overwrite > --- > > Key: SOLR-10391 > URL: https://issues.apache.org/jira/browse/SOLR-10391 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > As a follow up to SOLR-6736, we need a way for a user to overwrite an > existing configset and/or delete an existing configset. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley resolved SOLR-13181.
-
Fix Version/s: 8.7
Resolution: Fixed
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Fix For: 8.7
>
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Issue Comment Deleted] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-13181:
Comment: was deleted
(was: Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke )
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
--
[jira] [Issue Comment Deleted] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-13181:
Comment: was deleted
(was: Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke )
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
--
[jira] [Issue Comment Deleted] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-13181:
Comment: was deleted
(was: Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke )
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
--
[jira] [Commented] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200358#comment-17200358
]
ASF subversion and git services commented on SOLR-13181:
Commit 05b7a4a40d33403210a278c14ee951cfc2b7103f in lucene-solr's branch
refs/heads/branch_8x from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=05b7a4a ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke
(cherry picked from commit 2197776be67384d628f12a5def8663db9e4220cf)
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a
[jira] [Commented] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200349#comment-17200349
]
ASF subversion and git services commented on SOLR-13181:
Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sen
[jira] [Commented] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200346#comment-17200346
]
ASF subversion and git services commented on SOLR-13181:
Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sen
[jira] [Commented] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200344#comment-17200344
]
ASF subversion and git services commented on SOLR-13181:
Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sen
[GitHub] [lucene-solr] TomMD commented on a change in pull request #1901: SOLR-14883 Add a Muse (Continuous assurance platform) configuration
TomMD commented on a change in pull request #1901: URL: https://github.com/apache/lucene-solr/pull/1901#discussion_r492995357 ## File path: .muse/config.toml ## @@ -0,0 +1 @@ +jdk11 = true Review comment: I've added the comment to the file as well now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13181) NullPointerException in org.apache.solr.request.macro.MacroExpander
[
https://issues.apache.org/jira/browse/SOLR-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200342#comment-17200342
]
ASF subversion and git services commented on SOLR-13181:
Commit 2197776be67384d628f12a5def8663db9e4220cf in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2197776 ]
SOLR-13181: param macro expansion could throw (#1877)
...StringIndexOutOfBoundsException on bad syntax
* failOnMissingParams: should have been returning null (failing) on bad syntax
cases
Co-authored-by: Christine Poerschke
> NullPointerException in org.apache.solr.request.macro.MacroExpander
> ---
>
> Key: SOLR-13181
> URL: https://issues.apache.org/jira/browse/SOLR-13181
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html]. The
> attached file ({{home.zip}}) gives the contents of folder {{/tmp/home}} that
> you will obtain by following the steps below:
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> {noformat}
>Reporter: Cesar Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13181.patch, home.zip
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?a=${${b}}
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -4
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:150)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:101)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:65)
> at
> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:51)
> at
> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:159)
> at
> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:167)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:196)
> [...]
> {noformat}
> Parameter [macro
> expansion|http://yonik.com/solr-query-parameter-substitution/] seems to take
> place in {{org.apache.solr.request.macro.MacroExpander._expand(String val)}}.
> From reading the code of the function it seems that macros are not expanded
> inside curly brackets {{${...}}}, and so the {{${b}}} inside
> {noformat}
> ${${b}}
> {noformat}
> should not be expanded. But the function seems to fail to detect this
> specific case and graciously refuse to expand it.
> A possible fix could be updating the {{idx}} variable when the {{StrParser}}
> detects that no valid identifier can be found inside the brackets. See
> attached file
> {{0001-Macro-expander-fail-gracefully-on-unsupported-syntax.patch}}.
> We found this bug using [Diffblue Microservices
> Testing|https://www.diffblue.com/labs/]. Find more information on this [fuzz
> testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results].
--
This message was sen
[GitHub] [lucene-solr] dsmiley commented on pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on pull request #1877: URL: https://github.com/apache/lucene-solr/pull/1877#issuecomment-696943683 @gus-asf (in reference to you're fix in SOLR-12891): I can appreciate you didn't want to break back-compat. In this issue/PR, I'm not getting at that matter at all, it's a separate bug fix when the syntax is given wrong. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley merged pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley merged pull request #1877: URL: https://github.com/apache/lucene-solr/pull/1877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1877: SOLR-13181: param macro expansion could throw
dsmiley commented on a change in pull request #1877:
URL: https://github.com/apache/lucene-solr/pull/1877#discussion_r492990280
##
File path:
solr/core/src/test/org/apache/solr/request/macro/TestMacroExpander.java
##
@@ -143,15 +142,31 @@ public void testMapExprExpandOn() {
String oldVal = System.getProperty("StreamingExpressionMacros","false");
System.setProperty("StreamingExpressionMacros", "true");
try {
- @SuppressWarnings({"rawtypes"})
- Map expanded = MacroExpander.expand(request);
- assertEquals("zero", ((String[])expanded.get("fq"))[0]);
- assertEquals("one", ((String[])expanded.get("fq"))[1]);
- assertEquals("two", ((String[]) expanded.get("fq"))[2]);
- assertEquals("three", ((String[]) expanded.get("fq"))[3]);
- assertEquals("one", ((String[])expanded.get("expr"))[0]);
+ Map expanded = MacroExpander.expand(request);
+ assertEquals("zero", expanded.get("fq")[0]);
+ assertEquals("one", expanded.get("fq")[1]);
+ assertEquals("two", expanded.get("fq")[2]);
+ assertEquals("three", expanded.get("fq")[3]);
+ assertEquals("one", expanded.get("expr")[0]);
} finally {
System.setProperty("StreamingExpressionMacros", oldVal);
}
}
+
+ @Test
+ public void testUnbalanced() { // SOLR-13181
+final MacroExpander meSkipOnMissingParams = new
MacroExpander(Collections.emptyMap());
+final MacroExpander meFailOnMissingParams = new
MacroExpander(Collections.emptyMap(), true);
+assertEquals("${noClose", meSkipOnMissingParams.expand("${noClose"));
Review comment:
Ah; thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14597) Advanced Query Parser
[ https://issues.apache.org/jira/browse/SOLR-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200304#comment-17200304 ] Gus Heck commented on SOLR-14597: - Right, agreed, Lucene stuff also should be broken out to Lucene tickets. All initially here to keep the donation process simple. > Advanced Query Parser > - > > Key: SOLR-14597 > URL: https://issues.apache.org/jira/browse/SOLR-14597 > Project: Solr > Issue Type: New Feature > Components: query parsers >Reporter: Mike Nibeck >Assignee: Gus Heck >Priority: Major > Attachments: aqp_patch.patch > > > This JIRA ticket tracks the progress of SIP-9, the Advanced Query Parser that > is being donated by the Library of Congress. Full description of the feature > can be found on the SIP Page. > [https://cwiki.apache.org/confluence/display/SOLR/SIP-9+Advanced+Query+Parser] > Briefly, this parser provides a comprehensive syntax for users that use > search on a daily basis. It also reserves a smaller set of punctuators than > other parsers. This facilitates easier handling of acronyms and punctuated > patterns with meaning ( such as C++ or 401(k) ). The new syntax opens up some > advanced features while also preventing access to arbitrary features via > local parameters. This parser will be safe for accepting user queries > directly with minimal pre-parsing, but for use cases beyond it's established > features alternate query paths (using other parsers) will need to be supplied. > The code drop is being prepared and will be supplied as soon as we receive > guidance from the PMC regarding the proper process. Given that the Library > already has a signed CCLA we need to understand which of these (or other > processes) apply: > [http://incubator.apache.org/ip-clearance/ip-clearance-template.html] > and > [https://www.apache.org/licenses/contributor-agreements.html#grants] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1905: LUCENE-9488 Release with Gradle Part 2
dweiss commented on a change in pull request #1905:
URL: https://github.com/apache/lucene-solr/pull/1905#discussion_r492962137
##
File path: lucene/build.gradle
##
@@ -15,8 +15,56 @@
* limitations under the License.
*/
+// Should we do this as :lucene:packaging similar to how Solr does it?
+// Or is this fine here?
+
+plugins {
+ id 'distribution'
+}
+
description = 'Parent project for Apache Lucene Core'
subprojects {
group "org.apache.lucene"
-}
\ No newline at end of file
+}
+
+distributions {
+ main {
+ // This is empirically wrong, but it is mostly a copy from `ant
package-zip`
Review comment:
Sure but you'll have to be patient - can't do it right away.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on a change in pull request #1905: LUCENE-9488 Release with Gradle Part 2
madrob commented on a change in pull request #1905:
URL: https://github.com/apache/lucene-solr/pull/1905#discussion_r492937930
##
File path: lucene/build.gradle
##
@@ -15,8 +15,56 @@
* limitations under the License.
*/
+// Should we do this as :lucene:packaging similar to how Solr does it?
+// Or is this fine here?
+
+plugins {
+ id 'distribution'
+}
+
description = 'Parent project for Apache Lucene Core'
subprojects {
group "org.apache.lucene"
-}
\ No newline at end of file
+}
+
+distributions {
+ main {
+ // This is empirically wrong, but it is mostly a copy from `ant
package-zip`
Review comment:
I tried looking at the solr packaging for figuring this out, but kept
getting duplicate entries in the archive. I left the TODOs for what still
remains to be added to the zip, can you give me a little more direction on what
the implementation would look like?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob merged pull request #1910: Use github actions cache
madrob merged pull request #1910: URL: https://github.com/apache/lucene-solr/pull/1910 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9827) Make ConcurrentUpdateSolrClient create RemoteSolrException instead of just SolrException for remote errors
[ https://issues.apache.org/jira/browse/SOLR-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomas Eduardo Fernandez Lobbe resolved SOLR-9827. - Fix Version/s: 6.4 7.0 Resolution: Fixed > Make ConcurrentUpdateSolrClient create RemoteSolrException instead of just > SolrException for remote errors > -- > > Key: SOLR-9827 > URL: https://issues.apache.org/jira/browse/SOLR-9827 > Project: Solr > Issue Type: Improvement >Reporter: Tomas Eduardo Fernandez Lobbe >Assignee: Tomas Eduardo Fernandez Lobbe >Priority: Minor > Fix For: 7.0, 6.4 > > Attachments: SOLR-9827.patch > > > Also, improve the exception message to include the remote error message when > present. Specially when Solr is logging these errors (e.g. > DistributedUpdateProcessor), this should make it easier to understand that > the error was in the remote host and not in the one logging this exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14582) Expose IWC.setMaxCommitMergeWaitMillis as an expert feature in Solr's index config
[
https://issues.apache.org/jira/browse/SOLR-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomas Eduardo Fernandez Lobbe resolved SOLR-14582.
--
Resolution: Fixed
> Expose IWC.setMaxCommitMergeWaitMillis as an expert feature in Solr's index
> config
> --
>
> Key: SOLR-14582
> URL: https://issues.apache.org/jira/browse/SOLR-14582
> Project: Solr
> Issue Type: Improvement
>Reporter: Tomas Eduardo Fernandez Lobbe
>Assignee: Tomas Eduardo Fernandez Lobbe
>Priority: Trivial
> Fix For: master (9.0), 8.7
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> LUCENE-8962 added the ability to merge segments synchronously on commit. This
> isn't done by default and the default {{MergePolicy}} won't do it, but custom
> merge policies can take advantage of this. Solr allows plugging in custom
> merge policies, so if someone wants to make use of this feature they could,
> however, they need to set {{IndexWriterConfig.maxCommitMergeWaitSeconds}} to
> something greater than 0.
> Since this is an expert feature, I plan to document it only in javadoc and
> not the ref guide.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1909: LUCENE-9539: Remove caches from SortingCodecReader
jimczi commented on a change in pull request #1909:
URL: https://github.com/apache/lucene-solr/pull/1909#discussion_r492921270
##
File path: lucene/core/src/java/org/apache/lucene/index/SortingCodecReader.java
##
@@ -41,21 +41,13 @@
* {@link Sort}. This can be used to re-sort and index after it's been created
by wrapping all
* readers of the index with this reader and adding it to a fresh IndexWriter
via
* {@link IndexWriter#addIndexes(CodecReader...)}.
+ * NOTE: This reader should only be used for merging. Pulling fields from this
ready might be very costly and memory
Review comment:
nit:
```suggestion
* NOTE: This reader should only be used for merging. Pulling fields from
this reader might be very costly and memory
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-10391) ConfigSet overwrite
[ https://issues.apache.org/jira/browse/SOLR-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200261#comment-17200261 ] ASF subversion and git services commented on SOLR-10391: Commit 7b7a5a16cee8dcac19aa935ca916e9bc271acd25 in lucene-solr's branch refs/heads/master from Tomas Eduardo Fernandez Lobbe [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7b7a5a1 ] SOLR-10391: Add overwrite option to UPLOAD ConfigSet action (#1861) When set to true, Solr will overwrite an existing configset in ZooKeeper in an UPLOAD. A new cleanup parameter can also be passed to let Solr know what to do with the files that existed in the old configset, but no longer exist in the new configset (remove or keep) > ConfigSet overwrite > --- > > Key: SOLR-10391 > URL: https://issues.apache.org/jira/browse/SOLR-10391 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > As a follow up to SOLR-6736, we need a way for a user to overwrite an > existing configset and/or delete an existing configset. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe merged pull request #1861: SOLR-10391: Add overwrite option to UPLOAD ConfigSet action
tflobbe merged pull request #1861: URL: https://github.com/apache/lucene-solr/pull/1861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14889) improve templated variable escaping in ref-guide _config.yml
[
https://issues.apache.org/jira/browse/SOLR-14889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200259#comment-17200259
]
Uwe Schindler commented on SOLR-14889:
--
+1
That's a good plan.
> improve templated variable escaping in ref-guide _config.yml
>
>
> Key: SOLR-14889
> URL: https://issues.apache.org/jira/browse/SOLR-14889
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
>
> SOLR-14824 ran into windows failures when we switching from using a hardcoded
> "relative" path to the solrRootPath to using groovy/project variables to get
> the path. the reason for the failures was that the path us used as a
> variable tempted into {{_config.yml.template}} to build the {{_config.yml}}
> file, but on windows the path seperater of '\' was being parsed by
> jekyll/YAML as a string escape character.
> (This wasn't a problem we ran into before, even on windows, prior to the
> SOLR-14824 changes, because the hardcoded relative path only used '/'
> delimiters, which (j)ruby was happy to work with, even on windows.
> As Uwe pointed out when hotfixing this...
> {quote}Problem was that backslashes are used to escape strings, but windows
> paths also have those. Fix was to add StringEscapeUtils, but I don't like
> this too much. Maybe we find a better solution to make special characters in
> those properties escaped correctly when used in strings inside templates.
> {quote}
> ...the current fix of using {{StringEscapeUtils.escapeJava}} - only for this
> one variable -- doesn't really protect other variables that might have
> special charactes in them down the road, and while "escapeJava" work ok for
> the "\" issue, it isn't neccessarily consistent with all YAML escapse, which
> could lead to even weird bugs/cofusion down the road.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] gus-asf commented on pull request #1877: SOLR-13181: param macro expansion could throw
gus-asf commented on pull request #1877: URL: https://github.com/apache/lucene-solr/pull/1877#issuecomment-696867296 Did see your review request, but my life is doing that whole "it never rains but it pours" thing right now and I keep not getting to this. My primary concern wrt to this class is the need to ensure that default installations do not expand streaming expression parameters (but folks who need that for existing installs can turn it on with knowledge of the risks). I saw your comment on the ticket that fixed that originally, and yeah calling out a single parameter is yucky, but it was the least back compat breaking thing I could come up with. A more sensible, less backwards compatible solution for 9x is certainly a possibility. I may not have time to do any deep review here soon enough, so don't wait on me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
