jenkins-bot has submitted this change and it was merged. Change subject: Add a native script for detecting noop updates ......................................................................
Add a native script for detecting noop updates The builtin Elasticsearch detect_noop feature doesn't allow any slop when detecting noops. This super_detect_noop script does. You configure it like so: curl -XPOST localhost:9200/test/test/1/_update -d'{ "script": "super_detect_noop", "lang": "native", "params": { "source": { "foo": { "bar": 5 }, }, "detectors": { "foo.bar": "within 20%" } } }' Change-Id: I027ad6fa841cd33382dfbf3c1ea5eedb2cf9d169 (cherry picked from commit a319cf14144360e6f5169df925a8818497381cab) --- M README.md M docs/safer.md A docs/super_detect_noop.md M src/main/java/org/wikimedia/search/extra/ExtraPlugin.java A src/main/java/org/wikimedia/search/extra/superdetectnoop/CloseEnoughDetector.java A src/main/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScript.java A src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinAbsoluteDetector.java A src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinPercentageDetector.java A src/test/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScriptTest.java 9 files changed, 695 insertions(+), 5 deletions(-) Approvals: Manybubbles: Looks good to me, approved jenkins-bot: Verified diff --git a/README.md b/README.md index ded6150..3bacd1d 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@ Extra Queries and Filters ========================= -The plan is for this to include any extra queries and filters we end up -creating to make search nice for Wikimedia. At this point it only contains: +The plan is for this to include any extra queries, filters, and native scripts +we end up creating to make search nice for Wikimedia. At this point it only +contains: Filters: * [source_regex](docs/source_regex.md) - An nGram accelerated regular @@ -18,6 +19,10 @@ potentially expensive constructs. Expensive constructs either cause errors to be sent back to the user or are degraded into cheaper, less precise constructs. +Native Scripts: +* [super_detect_noop](docs/super_detect_noop.md) - Like ```detect_noop``` but +supports configurable sloppiness. New in 1.5.0, 1.4.1, and 1.3.1. + | Extra Queries and Filters Plugin | ElasticSearch | |----------------------------------|-----------------| | master | 1.3.4 -> 1.3.X | diff --git a/docs/safer.md b/docs/safer.md index 14c8ba1..4f0d864 100644 --- a/docs/safer.md +++ b/docs/safer.md @@ -126,7 +126,7 @@ } public static class MySafeifierActionsModule extends AbstractModule { - public SafeifierActionsModule(Settings settings) { + public MySafeifierActionsModule(Settings settings) { } @Override diff --git a/docs/super_detect_noop.md b/docs/super_detect_noop.md new file mode 100644 index 0000000..af4b7b0 --- /dev/null +++ b/docs/super_detect_noop.md @@ -0,0 +1,97 @@ +super_detect_noop +================= + +The ```super_detect_noop``` native script is just like Elasticsearch's +```detect_noop``` but it allows configurable sloppiness. + +Options +------- + +```super_detect_noop``` supports only the following options: +* ```source``` The source to merge into the existing source. Required. +* ```detectors``` Configures sloppiness detectors. Optional, defaults to +behaving exactly as Elasticsearch's ```detect_noop```. Possible field values: + * ```equals``` If the new value isn't equal to the old value then the new + value is written to the source. This is the default if no value is + specified in the ```detectors``` object. + * ```within nnn%``` If the new value isn't within nnn percent of the old + value then its written to the source. nnn is parsed as a double and all + math is performed with doubles. + * ```within nnn``` If the new value isn't within nnn of the old value then + its written to the source. nnn is parsed as a double and all math is + performed with doubles. + +Examples +------- +```bash +curl -XDELETE localhost:9200/test?pretty +curl -XPUT localhost:9200/test?pretty +curl -XGET 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty' +curl -XPUT localhost:9200/test/test/1?pretty -d'{ + "foo": 6 +}' +curl -XPOST localhost:9200/test/test/1/_update?pretty -d'{ + "script": "super_detect_noop", + "lang": "native", + "params": { + "source": { + "foo": 5 + }, + "detectors": { + "foo": "within 20%" + } + } +}' +``` + +```bash +curl -XDELETE localhost:9200/test?pretty +curl -XPUT localhost:9200/test?pretty +curl -XGET 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty' +curl -XPUT localhost:9200/test/test/1 -d'{ + "foo": { + "bar": 6 + } +}' +curl -XPOST localhost:9200/test/test/1/_update -d'{ + "script": "super_detect_noop", + "lang": "native", + "params": { + "source": { + "foo": { + "bar": 5 + }, + }, + "detectors": { + "foo.bar": "within 20%" + } + } +}' +``` + +Integrating +----------- +This native script was designed to allow other plugins to hook into it. Doing +so looks like this: +```java +public class MyPlugin extends AbstractPlugin { + @Override + public Collection<Class<? extends Module>> modules() { + return ImmutableList.<Class<? extends Module>>of(MyCloseEnoughDetectorsModule.class); + } + + public static class MyCloseEnoughDetectorsModule extends AbstractModule { + public MyCloseEnoughDetectorsModule(Settings settings) { + } + + @Override + @SuppressWarnings("rawtypes") + protected void configure() { + Multibinder<CloseEnoughDetector.Recognizer> detectors = Multibinder + .newSetBinder(binder(), CloseEnoughDetector.Recognizer.class); + detectors.addBinding().toInstance(new WithinPercentageDetector.Factory()); + } + } +} +``` + diff --git a/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java b/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java index 232abfa..418ceca 100644 --- a/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java +++ b/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java @@ -10,12 +10,17 @@ import org.elasticsearch.index.query.QueryParser; import org.elasticsearch.indices.query.IndicesQueriesModule; import org.elasticsearch.plugins.AbstractPlugin; +import org.elasticsearch.script.ScriptModule; import org.wikimedia.search.extra.idhashmod.IdHashModFilterParser; import org.wikimedia.search.extra.regex.SourceRegexFilterParser; import org.wikimedia.search.extra.safer.ActionModuleParser; import org.wikimedia.search.extra.safer.SaferQueryParser; import org.wikimedia.search.extra.safer.phrase.PhraseTooLargeActionModuleParser; import org.wikimedia.search.extra.safer.simple.SimpleActionModuleParser; +import org.wikimedia.search.extra.superdetectnoop.CloseEnoughDetector; +import org.wikimedia.search.extra.superdetectnoop.SuperDetectNoopScript; +import org.wikimedia.search.extra.superdetectnoop.WithinAbsoluteDetector; +import org.wikimedia.search.extra.superdetectnoop.WithinPercentageDetector; /** * Setup the Elasticsearch plugin. @@ -38,12 +43,19 @@ public void onModule(IndicesQueriesModule module) { module.addFilter(new SourceRegexFilterParser()); module.addFilter(new IdHashModFilterParser()); - module.addQuery((Class<QueryParser>) (Class<?>)SaferQueryParser.class); + module.addQuery((Class<QueryParser>) (Class<?>) SaferQueryParser.class); + } + + /** + * Register our scripts. + */ + public void onModule(ScriptModule module) { + module.registerScript("super_detect_noop", SuperDetectNoopScript.Factory.class); } @Override public Collection<Class<? extends Module>> modules() { - return ImmutableList.<Class<? extends Module>>of(SafeifierActionsModule.class); + return ImmutableList.<Class<? extends Module>> of(SafeifierActionsModule.class, CloseEnoughDetectorsModule.class); } public static class SafeifierActionsModule extends AbstractModule { @@ -58,4 +70,18 @@ moduleParsers.addBinding().to(SimpleActionModuleParser.class).asEagerSingleton(); } } + + public static class CloseEnoughDetectorsModule extends AbstractModule { + public CloseEnoughDetectorsModule(Settings settings) { + } + + @Override + protected void configure() { + Multibinder<CloseEnoughDetector.Recognizer> detectors = Multibinder + .newSetBinder(binder(), CloseEnoughDetector.Recognizer.class); + detectors.addBinding().toInstance(new CloseEnoughDetector.Equal.Factory()); + detectors.addBinding().toInstance(new WithinPercentageDetector.Factory()); + detectors.addBinding().toInstance(new WithinAbsoluteDetector.Factory()); + } + } } diff --git a/src/main/java/org/wikimedia/search/extra/superdetectnoop/CloseEnoughDetector.java b/src/main/java/org/wikimedia/search/extra/superdetectnoop/CloseEnoughDetector.java new file mode 100644 index 0000000..a27f873 --- /dev/null +++ b/src/main/java/org/wikimedia/search/extra/superdetectnoop/CloseEnoughDetector.java @@ -0,0 +1,115 @@ +package org.wikimedia.search.extra.superdetectnoop; + +/** + * Detects if two values are different enough to be changed. + * + * @param <T> type of the thin being checked + */ +public interface CloseEnoughDetector<T> { + /** + * Two objects are never close enough. + */ + static final CloseEnoughDetector<Object> EQUALS = new NullSafe<>(Equal.INSTANCE); + + boolean isCloseEnough(T oldValue, T newValue); + + /** + * Builds CloseEnoughDetectors from the string description sent in the + * script parameters. Returning null from build just means that this recognizer + * doesn't recognize that parameter. + */ + static interface Recognizer { + CloseEnoughDetector<Object> build(String description); + } + + /** + * Wraps another detector and only delegates to it if both values aren't + * null. If both values are null returns true, if only one is null then + * returns false. + * + * @param <T> type on which the wrapped detector operates + */ + class NullSafe<T> implements CloseEnoughDetector<T> { + private final CloseEnoughDetector<T> delegate; + + public NullSafe(CloseEnoughDetector<T> delegate) { + this.delegate = delegate; + } + + @Override + public boolean isCloseEnough(T oldValue, T newValue) { + if (oldValue == null) { + return newValue == null; + } + if (newValue == null) { + return false; + } + return delegate.isCloseEnough(oldValue, newValue); + } + } + + /** + * Objects are only close enough if they are {@link Object#equals(Object)} + * to each other. Doesn't do any null checking - wrap in NullSafe or just + * use CloseEnoughDetector.EQUALS if you need it. + */ + class Equal implements CloseEnoughDetector<Object> { + public static CloseEnoughDetector<Object> INSTANCE = new Equal(); + + public static class Factory implements CloseEnoughDetector.Recognizer { + @Override + public CloseEnoughDetector<Object> build(String description) { + if (description.equals("equals")) { + return INSTANCE; + } + return null; + } + } + + private Equal() { + } + + @Override + public boolean isCloseEnough(Object oldValue, Object newValue) { + return oldValue.equals(newValue); + } + } + + /** + * Wraps another detector and only delegates to it if both values are of a + * certain type. If they aren't then it delegates to Equal. Doesn't perform + * null checking - wrap me in NullSafe or just use the nullAndTypeSafe + * static method to build me. + * + * @param <T> type on which the wrapped detector operates + */ + class TypeSafe<T> implements CloseEnoughDetector<Object> { + /** + * Wraps a CloseEnoughDetector in a null-safe, type-safe way. + */ + static <T> CloseEnoughDetector<Object> nullAndTypeSafe(Class<T> type, CloseEnoughDetector<T> delegate) { + return new CloseEnoughDetector.NullSafe<>(new CloseEnoughDetector.TypeSafe<>(type, delegate)); + } + + private final Class<T> type; + private final CloseEnoughDetector<T> delegate; + + public TypeSafe(Class<T> type, CloseEnoughDetector<T> delegate) { + this.type = type; + this.delegate = delegate; + } + + @Override + public boolean isCloseEnough(Object oldValue, Object newValue) { + T oldValueCast; + T newValueCast; + try { + oldValueCast = type.cast(oldValue); + newValueCast = type.cast(newValue); + } catch (ClassCastException e) { + return Equal.INSTANCE.isCloseEnough(oldValue, newValue); + } + return delegate.isCloseEnough(oldValueCast, newValueCast); + } + } +} diff --git a/src/main/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScript.java b/src/main/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScript.java new file mode 100644 index 0000000..2735982 --- /dev/null +++ b/src/main/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScript.java @@ -0,0 +1,137 @@ +package org.wikimedia.search.extra.superdetectnoop; + +import static org.elasticsearch.common.base.Preconditions.checkNotNull; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.elasticsearch.common.collect.ImmutableList; +import org.elasticsearch.common.collect.ImmutableMap; +import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.script.AbstractExecutableScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +/** + * Like the detect_noop option on updates but with pluggable "close enough" + * detectors! So much power! + */ +public class SuperDetectNoopScript extends AbstractExecutableScript { + public static class Factory implements NativeScriptFactory { + private final List<CloseEnoughDetector.Recognizer> closeEnoughFactories; + + @Inject + public Factory(Set<CloseEnoughDetector.Recognizer> factories) { + // Note that detectors are tried in a random order.... + this.closeEnoughFactories = ImmutableList.copyOf(factories); + } + + @Override + public ExecutableScript newScript(Map<String, Object> params) { + @SuppressWarnings("unchecked") + Map<String, Object> source = (Map<String, Object>) params.get("source"); + return new SuperDetectNoopScript(source, detectors(params)); + } + + private Map<String, CloseEnoughDetector<Object>> detectors(Map<String, Object> params) { + @SuppressWarnings("unchecked") + Map<String, String> detectorConfigs = (Map<String, String>) params.get("detectors"); + if (detectorConfigs == null) { + return Collections.emptyMap(); + } + ImmutableMap.Builder<String, CloseEnoughDetector<Object>> detectors = ImmutableMap.builder(); + for (Map.Entry<String, String> detectorConfig : detectorConfigs.entrySet()) { + detectors.put(detectorConfig.getKey(), detector(detectorConfig.getValue())); + } + return detectors.build(); + } + + private CloseEnoughDetector<Object> detector(String config) { + for (CloseEnoughDetector.Recognizer factory : closeEnoughFactories) { + CloseEnoughDetector<Object> detector = factory.build(config); + if (detector != null) { + return detector; + } + } + throw new IllegalArgumentException("Don't recognize this type of detector: " + config); + } + } + + private final Map<String, Object> source; + private final Map<String, CloseEnoughDetector<Object>> pathToDetector; + private Map<String, Object> ctx; + + public SuperDetectNoopScript(Map<String, Object> source, Map<String, CloseEnoughDetector<Object>> pathToDetector) { + this.source = checkNotNull(source, "source must be specified"); + this.pathToDetector = checkNotNull(pathToDetector, "detectors must be specified"); + } + + @Override + public Object run() { + @SuppressWarnings("unchecked") + Map<String, Object> oldSource = (Map<String, Object>) ctx.get("_source"); + boolean changed = update(oldSource, source, ""); + if (!changed) { + ctx.put("op", "none"); + } + // The return value is ignored + return null; + } + + @Override + @SuppressWarnings("unchecked") + public void setNextVar(String name, Object value) { + if (name.equals("ctx")) { + ctx = (Map<String, Object>) value; + } + } + + /** + * Update old with the source and detector configuration of this script. + */ + boolean update(Map<String, Object> old, Map<String, Object> updateSource, String path) { + boolean modified = false; + for (Map.Entry<String, Object> sourceEntry : updateSource.entrySet()) { + String nextPath = path + sourceEntry.getKey(); + Object oldValue = old.get(sourceEntry.getKey()); + if (oldValue instanceof Map && sourceEntry.getValue() instanceof Map) { + // recursive merge maps + @SuppressWarnings("unchecked") + Map<String, Object> nextOld = (Map<String, Object>) old.get(sourceEntry.getKey()); + @SuppressWarnings("unchecked") + Map<String, Object> nextUpdateSource = (Map<String, Object>) sourceEntry.getValue(); + modified |= update(nextOld, nextUpdateSource, nextPath); + continue; + } + if (detector(nextPath).isCloseEnough(oldValue, sourceEntry.getValue())) { + continue; + } + if (sourceEntry.getValue() == null) { + old.remove(sourceEntry.getKey()); + } else { + old.put(sourceEntry.getKey(), sourceEntry.getValue()); + } + modified = true; + continue; + } + /* + * Right now if a field isn't in the source passed to the script the + * close enough detectors never get a chance to look at it - the field + * is never changed. + */ + return modified; + } + + /** + * Get the close enough detector for a path, defaulting to EQUALS. + */ + private CloseEnoughDetector<Object> detector(String path) { + CloseEnoughDetector<Object> d = pathToDetector.get(path); + if (d == null) { + return CloseEnoughDetector.EQUALS; + } + return d; + } +} diff --git a/src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinAbsoluteDetector.java b/src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinAbsoluteDetector.java new file mode 100644 index 0000000..887ee0b --- /dev/null +++ b/src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinAbsoluteDetector.java @@ -0,0 +1,36 @@ +package org.wikimedia.search.extra.superdetectnoop; + +import static org.wikimedia.search.extra.superdetectnoop.CloseEnoughDetector.TypeSafe.nullAndTypeSafe; + +/** + * Checks if a number is different by some absolute amount. + */ +public class WithinAbsoluteDetector implements CloseEnoughDetector<Number> { + public static class Factory implements CloseEnoughDetector.Recognizer { + private static final String PREFIX = "within "; + + @Override + public CloseEnoughDetector<Object> build(String description) { + if (!description.startsWith(PREFIX)) { + return null; + } + try { + double absoluteDifference = Double.parseDouble(description.substring(PREFIX.length(), description.length())); + return nullAndTypeSafe(Number.class, new WithinAbsoluteDetector(absoluteDifference)); + } catch (NumberFormatException e) { + return null; + } + } + } + + private final double absoluteDifference; + + public WithinAbsoluteDetector(double absoluteDifference) { + this.absoluteDifference = absoluteDifference; + } + + @Override + public boolean isCloseEnough(Number oldValue, Number newValue) { + return Math.abs(newValue.doubleValue() - oldValue.doubleValue()) < absoluteDifference; + } +} diff --git a/src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinPercentageDetector.java b/src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinPercentageDetector.java new file mode 100644 index 0000000..8729d46 --- /dev/null +++ b/src/main/java/org/wikimedia/search/extra/superdetectnoop/WithinPercentageDetector.java @@ -0,0 +1,42 @@ +package org.wikimedia.search.extra.superdetectnoop; + +import static java.lang.Math.abs; +import static org.wikimedia.search.extra.superdetectnoop.CloseEnoughDetector.TypeSafe.nullAndTypeSafe; + +/** + * Checks if a number is different by some percentage. + */ +public class WithinPercentageDetector implements CloseEnoughDetector<Number> { + public static class Factory implements CloseEnoughDetector.Recognizer { + private static final String PREFIX = "within "; + private static final String SUFFIX = "%"; + + @Override + public CloseEnoughDetector<Object> build(String description) { + if (!description.startsWith(PREFIX)) { + return null; + } + if (!description.endsWith(SUFFIX)) { + return null; + } + try { + double percentage = Double.parseDouble(description.substring(PREFIX.length(), description.length() - SUFFIX.length())); + return nullAndTypeSafe(Number.class, new WithinPercentageDetector(percentage / 100)); + } catch (NumberFormatException e) { + // Not a valid number even with the % sign.... + return null; + } + } + } + + private final double absoluteDifference; + + public WithinPercentageDetector(double absoluteDifference) { + this.absoluteDifference = absoluteDifference; + } + + @Override + public boolean isCloseEnough(Number oldValue, Number newValue) { + return abs((newValue.doubleValue() - oldValue.doubleValue()) / oldValue.doubleValue()) < absoluteDifference; + } +} diff --git a/src/test/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScriptTest.java b/src/test/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScriptTest.java new file mode 100644 index 0000000..688b7ad --- /dev/null +++ b/src/test/java/org/wikimedia/search/extra/superdetectnoop/SuperDetectNoopScriptTest.java @@ -0,0 +1,232 @@ +package org.wikimedia.search.extra.superdetectnoop; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertThrows; +import static org.hamcrest.Matchers.anything; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.greaterThan; +import static org.hamcrest.Matchers.hasEntry; +import static org.hamcrest.Matchers.not; + +import java.io.IOException; +import java.util.Map; + +import org.elasticsearch.action.index.IndexResponse; +import org.elasticsearch.action.update.UpdateRequestBuilder; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentHelper; +import org.elasticsearch.rest.RestStatus; +import org.elasticsearch.script.ScriptService.ScriptType; +import org.junit.Test; +import org.wikimedia.search.extra.AbstractPluginIntegrationTest; + +public class SuperDetectNoopScriptTest extends AbstractPluginIntegrationTest { + @Test + public void newField() throws IOException { + indexSeedData(); + XContentBuilder b = x("bar", 2); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) 3)); + assertThat(r, hasEntry("bar", (Object) 2)); + } + + @Test + public void notModified() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", 3); + Map<String, Object> r = update(b, false); + assertThat(r, hasEntry("int", (Object) 3)); + } + + @Test + public void setToNull() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", null); + Map<String, Object> r = update(b, true); + assertThat(r, not(hasEntry(equalTo("int"), anything()))); + } + + @Test + public void newValue() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", 2); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) 2)); + } + + @Test + public void withinPercentage() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", 5, "within 200%"); + Map<String, Object> r = update(b, false); + assertThat(r, hasEntry("int", (Object) 3)); + } + + @Test + public void withinPercentageNegative() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", -1, "within 200%"); + Map<String, Object> r = update(b, false); + assertThat(r, hasEntry("int", (Object) 3)); + } + + @Test + public void outsidePercentage() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", 9, "within 200%"); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) 9)); + } + + @Test + public void outsidePercentageNegative() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", -3, "within 200%"); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) (-3))); + } + + @Test + public void percentageOnString() throws IOException { + indexSeedData(); + XContentBuilder b = x("string", "cat", "within 200%"); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) 3)); + assertThat(r, hasEntry("string", (Object) "cat")); + } + + @Test + public void withinAbsolute() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", 4, "within 2"); + Map<String, Object> r = update(b, false); + assertThat(r, hasEntry("int", (Object) 3)); + } + + @Test + public void withinAbsoluteNegative() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", -1, "within 7"); + Map<String, Object> r = update(b, false); + assertThat(r, hasEntry("int", (Object) 3)); + } + + @Test + public void outsideAbsolute() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", 5, "within 2"); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) 5)); + } + + @Test + public void outsideAbsoluteNegative() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", -4, "within 7"); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) (-4))); + } + + @Test + public void absoluteOnString() throws IOException { + indexSeedData(); + XContentBuilder b = x("string", "cat", "within 2"); + Map<String, Object> r = update(b, true); + assertThat(r, hasEntry("int", (Object) 3)); + assertThat(r, hasEntry("string", (Object) "cat")); + } + + @Test + public void garbageDetector() throws IOException { + indexSeedData(); + XContentBuilder b = x("int", "cat", "not a valid detector"); + assertThrows(toUpdateRequest(b), RestStatus.BAD_REQUEST); + } + + /** + * Tests path matching. + */ + @Test + public void path() throws IOException { + indexSeedData(); + XContentBuilder b = jsonBuilder().startObject(); + b.startObject("source"); + { + b.startObject("foo"); + { + b.field("bar", 10); + } + b.endObject(); + } + b.endObject(); + b.startObject("detectors"); + { + b.field("foo.bar", "within 10"); + } + b.endObject(); + b.endObject(); + Map<String, Object> r = update(b, false); + assertThat(r, hasEntry("int", (Object) 3)); + } + + private XContentBuilder x(String field, Object value) throws IOException { + return x(field, value, null); + } + + /** + * Builds the xcontent for the request parameters for a single field. + */ + private XContentBuilder x(String field, Object value, String detector) throws IOException { + XContentBuilder b = jsonBuilder().startObject(); + b.startObject("source"); + { + b.field(field, value); + } + b.endObject(); + if (detector != null) { + b.startObject("detectors"); + { + b.field(field, detector); + } + b.endObject(); + } + b.endObject(); + return b; + } + + private void indexSeedData() throws IOException { + XContentBuilder b = jsonBuilder().startObject(); + { + b.field("int", 3); + b.field("string", "cake"); + b.startObject("foo"); + { + b.field("bar", 10); + } + b.endObject(); + } + b.endObject(); + IndexResponse ir = client().prepareIndex("test", "test", "1").setSource(b).setRefresh(true).get(); + assertTrue("Test data is newly created", ir.isCreated()); + } + + private Map<String, Object> update(XContentBuilder b, boolean shouldUpdate) { + long beforeNoops = client().admin().indices().prepareStats("test").get().getTotal().indexing.getTotal().getIndexCount(); + toUpdateRequest(b).get(); + long afterNoops = client().admin().indices().prepareStats("test").get().getTotal().indexing.getTotal().getIndexCount(); + if (shouldUpdate) { + assertThat("there have not been updates but we expected some", afterNoops, greaterThan(beforeNoops)); + } else { + assertEquals("there have been updates and we don't expect them", beforeNoops, afterNoops); + } + return client().prepareGet("test", "test", "1").get().getSource(); + } + + private UpdateRequestBuilder toUpdateRequest(XContentBuilder b) { + b.close(); + Map<String, Object> m = XContentHelper.convertToMap(b.bytes(), true).v2(); + return client().prepareUpdate("test", "test", "1").setScript("super_detect_noop", ScriptType.INLINE).setScriptLang("native") + .setScriptParams(m).setRefresh(true); + + } +} -- To view, visit https://gerrit.wikimedia.org/r/207278 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I027ad6fa841cd33382dfbf3c1ea5eedb2cf9d169 Gerrit-PatchSet: 2 Gerrit-Project: search/extra Gerrit-Branch: 1.3 Gerrit-Owner: Manybubbles <never...@wikimedia.org> Gerrit-Reviewer: Chad <ch...@wikimedia.org> Gerrit-Reviewer: Jdouglas <jdoug...@wikimedia.org> Gerrit-Reviewer: Manybubbles <never...@wikimedia.org> Gerrit-Reviewer: Smalyshev <smalys...@wikimedia.org> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits