[
https://issues.apache.org/jira/browse/SOLR-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282702#comment-15282702
]
Steve Rowe commented on SOLR-8208:
----------------------------------
Here's a reproducible failure of TestSubQueryTransformerDistrib on my Jenkins:
{noformat}
Checking out Revision 9d5b834b09d4ff23e89755e5d1af407a2bd96c16
(refs/remotes/origin/master)
[...]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestSubQueryTransformerDistrib -Dtests.method=test
-Dtests.seed=A6B6D43AC01C202D -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true
-Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt
-Dtests.locale=de-LU -Dtests.timezone=Pacific/Tongatapu -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[junit4] ERROR 55.9s J7 | TestSubQueryTransformerDistrib.test <<<
[junit4] > Throwable #1:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:54181: Cannot create collection departments.
Value of maxShardsPerNode is 12, and the number of nodes currently live or live
and part of your createNodeSet is 5. This allows a maximum of 60 to be created.
Value of numShards is 6 and value of replicationFactor is 12. This requires 72
shards to be created (higher than the allowed number)
[junit4] > at
__randomizedtesting.SeedInfo.seed([A6B6D43AC01C202D:2EE2EBE06EE04DD5]:0)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:606)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
[junit4] > at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1592)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1549)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1604)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1545)
[junit4] > at
org.apache.solr.response.transform.TestSubQueryTransformerDistrib.test(TestSubQueryTransformerDistrib.java:64)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 743644 INFO
(SUITE-TestSubQueryTransformerDistrib-seed#[A6B6D43AC01C202D]-worker) [ ]
o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at:
/var/lib/jenkins/jobs/Lucene-Solr-Nightly-master/workspace/solr/build/solr-core/test/J7/temp/solr.response.transform.TestSubQueryTransformerDistrib_A6B6D43AC01C202D-001
[junit4] 2> May 13, 2016 7:06:27 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked
thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {},
docValues:{}, maxPointsInLeafNode=772, maxMBSortInHeap=6.297414713628615,
sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=de-LU,
timezone=Pacific/Tongatapu
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation
1.8.0_77 (64-bit)/cpus=16,threads=1,free=267803120,total=527433728
[junit4] 2> NOTE: All tests run in this JVM:
[CoreMergeIndexesAdminHandlerTest, TestIBSimilarityFactory,
AnalyticsMergeStrategyTest, SolrIndexSplitterTest, SolrPluginUtilsTest,
DocumentBuilderTest, TestQueryTypes, BlockJoinFacetDistribTest,
TestReplicationHandlerBackup, BadIndexSchemaTest, DistanceUnitsTest,
CleanupOldIndexTest, OverseerRolesTest, DocValuesTest,
DistributedFacetPivotSmallAdvancedTest, TestReloadAndDeleteDocs, TestSolrJ,
TestPHPSerializedResponseWriter, TlogReplayBufferedWhileIndexingTest,
TestSha256AuthenticationProvider, TestFaceting, DeleteStatusTest,
TestSubQueryTransformerDistrib]
[junit4] Completed [218/597 (3!)] on J7 in 56.17s, 1 test, 1 error <<<
FAILURES!
{noformat}
and another one from Policeman Jenkins a few days back:
{noformat}
Checking out Revision 470ba0794ecddd6375db3da521272dde46ed6761
(refs/remotes/origin/master)
[...]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestSubQueryTransformerDistrib -Dtests.method=test
-Dtests.seed=594D23296C3F97B8 -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true
-Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt
-Dtests.locale=es-US -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[junit4] ERROR 59.8s J0 | TestSubQueryTransformerDistrib.test <<<
[junit4] > Throwable #1:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:43474/ldvl/gb: Cannot create collection
departments. Value of maxShardsPerNode is 10, and the number of nodes currently
live or live and part of your createNodeSet is 5. This allows a maximum of 50
to be created. Value of numShards is 6 and value of replicationFactor is 10.
This requires 60 shards to be created (higher than the allowed number)
[junit4] > at
__randomizedtesting.SeedInfo.seed([594D23296C3F97B8:D1191CF3C2C3FA40]:0)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:606)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
[junit4] > at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1592)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1549)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1604)
[junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1545)
[junit4] > at
org.apache.solr.response.transform.TestSubQueryTransformerDistrib.test(TestSubQueryTransformerDistrib.java:64)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 4199324 INFO
(SUITE-TestSubQueryTransformerDistrib-seed#[594D23296C3F97B8]-worker) [ ]
o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.response.transform.TestSubQueryTransformerDistrib_594D23296C3F97B8-001
[junit4] 2> May 11, 2016 5:31:49 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked
thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60): {},
docValues:{}, maxPointsInLeafNode=2044, maxMBSortInHeap=7.8998054031288465,
sim=ClassicSimilarity, locale=es-US, timezone=Etc/GMT-1
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=275515608,total=532676608
[junit4] 2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength,
DistributedSpellCheckComponentTest, TestComponentsName,
TestSha256AuthenticationProvider, TestGraphTermsQParserPlugin,
AddSchemaFieldsUpdateProcessorFactoryTest, TestFunctionQuery,
TestConfigSetImmutable, CursorPagingTest, XsltUpdateRequestHandlerTest,
JsonLoaderTest, TestRawResponseWriter, TestPseudoReturnFields, PrimUtilsTest,
LeaderElectionIntegrationTest, ParsingFieldUpdateProcessorsTest,
TestInitParams, TestIBSimilarityFactory, TestCustomDocTransformer,
SegmentsInfoRequestHandlerTest, TestTrackingShardHandlerFactory,
PreAnalyzedFieldTest, CloneFieldUpdateProcessorFactoryTest, TestLRUStatsCache,
CurrencyFieldOpenExchangeTest, TestLRUCache, TestConfigReload,
DistributedQueryComponentOptimizationTest, TestConfigSetsAPIExclusivity,
BlockDirectoryTest, JSONWriterTest, TestDistributedSearch,
TolerantUpdateProcessorTest, TestLuceneMatchVersion, TestRandomFaceting,
HdfsChaosMonkeyNothingIsSafeTest, RequiredFieldsTest,
TestLMDirichletSimilarityFactory, TestUseDocValuesAsStored, RulesTest,
TestRandomDVFaceting, PeerSyncTest, TestSSLRandomization, TestRangeQuery,
TestOrdValues, SmileWriterTest, PathHierarchyTokenizerFactoryTest,
LeaderInitiatedRecoveryOnCommitTest, TestSchemaVersionResource,
TestCloudBackupRestore, TestCustomSort, SharedFSAutoReplicaFailoverTest,
SolrRequestParserTest, SolrCoreTest, AlternateDirectoryTest,
StandardRequestHandlerTest, CachingDirectoryFactoryTest,
TestShardHandlerFactory, TestPhraseSuggestions, TestReplicationHandler,
TestHighlightDedupGrouping, TestSolrQueryParserResource,
DistributedQueryElevationComponentTest, CdcrVersionReplicationTest, TestConfig,
DistribJoinFromCollectionTest, TestCSVLoader,
ConcurrentDeleteAndCreateCollectionTest, SOLR749Test, OverseerTaskQueueTest,
OutputWriterTest, TestJoin, AnalyticsQueryTest, ResponseLogComponentTest,
TestRTGBase, ForceLeaderTest, TestJsonRequest,
TestOverriddenPrefixQueryForCustomFieldType, IndexSchemaTest,
DateRangeFieldTest, ActionThrottleTest, TestBinaryField, DeleteStatusTest,
BasicZkTest, CdcrReplicationDistributedZkTest, IndexSchemaRuntimeFieldTest,
BlockJoinFacetDistribTest, TestBadConfig, PolyFieldTest,
TestDynamicFieldCollectionResource, SpatialRPTFieldTypeTest,
CollectionTooManyReplicasTest, TestStressLucene, TestDistribDocBasedVersion,
TestMergePolicyConfig, CoreMergeIndexesAdminHandlerTest,
DistributedFacetPivotLargeTest, SSLMigrationTest, SaslZkACLProviderTest,
ShardSplitTest, TestRequestForwarding, TestRequestStatusCollectionAPI,
TestTolerantUpdateProcessorCloud, TlogReplayBufferedWhileIndexingTest,
HdfsTlogReplayBufferedWhileIndexingTest, TestRestoreCore,
DistributedDebugComponentTest, DistributedFacetPivotSmallTest,
DistributedSuggestComponentTest, FacetPivotSmallTest, SpatialHeatmapFacetsTest,
TestExpandComponent, WrapperMergePolicyFactoryTest, TestIntervalFaceting,
TestGraphMLResponseWriter, TestSortingResponseWriter, TestChildDocTransformer,
TestSubQueryTransformerDistrib]
[junit4] Completed [549/597 (8!)] on J0 in 60.44s, 1 test, 1 error <<<
FAILURES!
{noformat}
> DocTransformer executes sub-queries
> -----------------------------------
>
> Key: SOLR-8208
> URL: https://issues.apache.org/jira/browse/SOLR-8208
> Project: Solr
> Issue Type: Improvement
> Components: Response Writers
> Reporter: Mikhail Khludnev
> Assignee: Mikhail Khludnev
> Labels: features, newbie
> Fix For: 6.1, master (7.0)
>
> Attachments: SOLR-8208.diff, SOLR-8208.patch, SOLR-8208.patch,
> SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch,
> SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch,
> SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch,
> SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch, SOLR-8208.patch
>
>
> The initial idea was to return "from" side of query time join via
> doctransformer. I suppose it isn't query-time join specific, thus let to
> specify any query and parameters for them, let's call it sub-query. But it
> might be problematic to escape subquery parameters, including local ones,
> e.g. what if subquery needs to specify own doctransformer in &fl=\[..\] ?
> I suppose we can specify subquery parameter prefix:
> {code}
> ..&q=name_s:john&fl=*,depts:[subquery fromIndex=departments]&
> depts.q={!term f=dept_id_s
> v=$row.dept_ss_dv}&depts.fl=text_t,dept_id_s_dv&depts.rows=12&depts.sort=id
> desc
> {code}
> response is like
> {code}
> <response>
> ...
> <result name="response" numFound="1" start="0">
> <doc>
> <str name="id">1</str>
> <str name="name_s_dv">john</str>
> ..
> <result name="depts" numFound="2" start="0">
> <doc>
> <str name="dept_id_s_dv">Engineering</str>
> <str name="text_t">These guys develop stuff</str>
> </doc>
> <doc>
> <str name="dept_id_s_dv">Support</str>
> <str name="text_t">These guys help users</str>
> </doc>
> </result>
> </doc>
> </result>
> </response>
> {code}
> * {{fl=depts:\[subquery]}} executes a separate request for every query result
> row, and adds it into a document as a separate result list. The given field
> name (here it's 'depts') is used as a prefix to shift subquery parameters
> from main query parameter, eg {{depts.q}} turns to {{q}} for subquery,
> {{depts.rows}} to {{rows}}.
> * document fields are available as implicit parameters with prefix {{row.}}
> eg. if result document has a field {{dept_id}} it can be referred as
> {{v=$row.dept_id}} this combines well with \{!terms} query parser
> * {{separator=','}} is used when multiple field values are combined in
> parameter. eg. a document has multivalue field {code}dept_ids={2,3}{code},
> thus referring to it via {code}..&dept.q={!terms f=id
> v=$row.dept_ids}&..{code} executes a subquery {code}{!terms f=id}2,3{code}.
> When omitted it's a comma.
> * {{fromIndex=othercore}} optional param allows to run subquery on other
> core, like it works on query time join
> However, it doesn't work on cloud setup (and will let you know), but it's
> proposed to use regular params ({{collection}}, {{shards}} - whatever, with
> subquery prefix as below ) to issue subquery to a collection
> {code}
> q=name_s:dave&indent=true&fl=*,depts:[subquery]&rows=20&
> depts.q={!terms f=dept_id_s v=$row.dept_ss_dv}&depts.fl=text_t&
> depts.indent=true&
> depts.collection=departments&
> depts.rows=10&depts.logParamsList=q,fl,rows,row.dept_ss_dv
> {code}
> Caveat: it should be a way slow; it handles only search result page, not
> entire result set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
