[
https://issues.apache.org/jira/browse/CAMEL-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano Machado updated CAMEL-24396:
------------------------------------
Description:
Search results on camel.apache.org are not de-duplicated: a query like
{{timer}} returns 19 usable hits spanning a single document, so the result list
fills with near-identical anchors from one page. The site currently works
around this in the browser, in {{antora-ui-camel/src/js/08-docsearch.js}}, by
over-fetching and capping hits per page. The durable fix is at the index level.
h2. The repository already has a config file, but its settings are not in effect
{{.docsearch.config.json}} exists at the repository root and already attempts
both of the settings needed here:
{code:json}
"attributesForFaceting": ["version"],
"attributeForDistinctResults": "url"
{code}
Neither is applied to the live index. Verified against the public search key:
* {{{"facets": ["version"], "hitsPerPage": 0}}} returns {{"facets": \{\}}}. If
{{version}} were in {{attributesForFaceting}}, this would return facet counts.
* Adding {{"distinct": true}} to a query changes nothing, which means
{{attributeForDistinct}} is unset.
Two problems compound here.
h3. 1. {{attributeForDistinctResults}} is not a real Algolia setting
The setting is named {{attributeForDistinct}}, and it must be paired with
{{distinct}}. Per the Algolia API reference for the {{distinct}} parameter:
"{{attributeForDistinct}} defines how records are grouped, {{distinct}} defines
how many records per group are shown. If {{attributeForDistinct}} isn't set,
{{distinct}} is ignored."
So {{attributeForDistinctResults}} is silently ignored regardless of what
consumes the file, and {{distinct}} is missing entirely.
h3. 2. The file matches neither documented DocSearch config format
The current DocSearch / Algolia Crawler configuration is JavaScript, not JSON,
and is structured as:
{code:javascript}
new Crawler({
appId, apiKey,
startUrls: [...],
sitemaps: [...],
actions: [{ indexName, pathsToMatch, recordExtractor }],
initialIndexSettings: { INDEX_NAME: { distinct: true, attributeForDistinct:
'url', ... } },
})
{code}
{{.docsearch.config.json}} instead nests crawler options under {{index}} and
{{crawler}} objects and puts index settings under {{custom_settings}}. The
{{start_urls}} / {{selectors}} / {{selectors_exclude}} / {{custom_settings}} /
{{min_indexed_level}} / {{stop_urls}} keys resemble the deprecated
docsearch-scraper JSON format, while {{index.pathsToMatch}},
{{index.includeHeadingLevels}} and {{crawler.sitemapUrls}} resemble the Crawler
format. The result matches neither.
Nothing in the repository references the file outside documentation
({{README.md}} and {{.docsearch.README.md}}); there is no CI job or script that
applies it. It was added in a single commit, 77e25115 (#1473).
*This is the key open question:* someone with Algolia dashboard access needs to
confirm whether {{.docsearch.config.json}} actually feeds the crawler. That
determines whether this is fixed by a PR against camel-website or by editing
the crawler config in the Algolia dashboard.
h2. The records are missing the attribute the fix would normally use
DocSearch's standard {{helpers.docsearch()}} extractor emits {{url}},
{{url_without_anchor}}, {{anchor}}, {{type}} and {{weight}}. Retrieving all
attributes from the live index returns only:
{code}
content, hierarchy, keywords, objectID, pageRank, url, version
{code}
So {{url_without_anchor}}, {{anchor}}, {{type}} and {{weight}} are all absent,
and {{pageRank}} is flat rather than nested under {{weight}}. The
{{apache_camel}} index is therefore not being built by the standard DocSearch
record extractor.
This matters because {{url}} on these records *includes* the anchor:
{code}
https://camel.apache.org/components/4.22.x/timer-component.html
https://camel.apache.org/components/4.22.x/timer-component.html#_endpoint_query_option_time
https://camel.apache.org/components/4.22.x/timer-component.html#_component_option_includeMetadata
{code}
Setting {{attributeForDistinct: "url"}} would group by anchor, which does not
de-duplicate anything. The fix needs an anchor-free attribute, and one does not
currently exist on the records.
h2. Proposed change
# Confirm whether {{.docsearch.config.json}} is actually consumed by the
crawler, or whether the live config lives only in the Algolia dashboard. Fix or
remove the file accordingly, since a config file that looks authoritative but
is inert is worse than none.
# Have the crawler emit an anchor-free URL attribute ({{url_without_anchor}},
matching DocSearch convention).
# Then set, in {{initialIndexSettings}} for {{apache_camel}}:
{code:json}
{
"attributeForDistinct": "url_without_anchor",
"distinct": true
}
{code}
# Optionally add {{attributesForFaceting}} so sub-project exclusion can move
server side as a {{facetFilters}} entry rather than running in the browser.
h2. Impact on the client
Distinct parent pages surviving the client-side sub-project filter, measured
against the live index:
|| query || hitsPerPage 20 (DocSearch default) || hitsPerPage 50 || hitsPerPage
100 ||
| kamelet | 2 pages | 3 pages | 3 pages |
| timer | 1 page | 7 pages | 16 pages |
| rest dsl | 7 pages | 16 pages | 25 pages |
With server-side {{distinct}}, de-duplication happens before the hit window is
applied, so the DocSearch default of 20 would return 20 genuinely distinct
pages. The following could then be deleted from
{{antora-ui-camel/src/js/08-docsearch.js}}:
* {{limitHitsPerPage}} / {{MAX_HITS_PER_PAGE}}, replaced by
{{attributeForDistinct}}
* {{HITS_PER_PAGE = 50}}, back to the DocSearch default
* {{isSubProjectUrl}} / {{EXCLUDED_SUBPROJECTS}}, replaced by {{facetFilters}},
if step 4 is done
{{sortByCoreDocs}} would remain client side, or better, move into the index as
a custom ranking attribute.
h2. Possibly related
DocSearch v5's UI expects {{type}} and {{anchor}} on records; it uses
{{item.type === 'lvl1'}} to associate sub-results with their parent heading.
Since neither attribute exists on this index, that parent grouping cannot
engage. Search still functions, as {{hierarchy}}, {{url}} and {{content}} are
all present, but the result list will not render the nested parent/child
structure DocSearch normally shows. Worth confirming when the crawler output is
reviewed.
h2. Context
This came out of the DocSearch v5 migration in camel-website PR #1729. The
previous implementation ({{src/js/vendor/algoliasearch.bundle.js}}) hand-rolled
de-duplication, sub-project filtering and ranking against the raw Algolia
client. The v5 rewrite initially dropped all of it; it has since been
reinstated in adapted form, but the index-level fix is the durable one.
was:
The {{apache\_camel}} Algolia index has neither {{attributeForDistinct}} nor
{{attributesForFaceting}} configured. As a result, search result
de\-duplication and sub\-project exclusion both have to run in the browser,
after Algolia has already chosen which hits to return. Configuring them at the
index level would remove that constraint and let us delete most of the
client\-side logic in {{antora\-ui\-camel/src/js/08\-docsearch.js}}.
This requires Algolia admin access \(or DocSearch crawler config access\), so
it cannot be done from the camel\-website repository.
h2. Evidence
Queried against the live index with the public search key:
{code}
POST /1/indexes/apache\_camel/query {"query":"kamelet","hitsPerPage":20}
\-> nbHits 3121, 20 hits returned, spanning only 4 distinct parent pages
\(16 of the 20 were anchors on components/4.22.x/kamelet\-component.html\)
{code}
* Adding {{"distinct": true}} to the query changed nothing, which confirms
{{attributeForDistinct}} is unset. {{distinct}} is a no\-op without it.
* A {{{"facets": \["\*"\]}}} query returned no facets at all, confirming
{{attributesForFaceting}} is unset, so {{facetFilters}} cannot be used to
exclude sub\-projects server side.
h2. Why this matters
Because both filters run client side, they operate on an already\-truncated
window of hits. Distinct parent pages remaining after applying the sub\-project
exclusion:
|| query || hitsPerPage 20 || hitsPerPage 50 || hitsPerPage 100 ||
| kamelet | 2 pages | 3 pages | 3 pages |
| timer | 1 page | 7 pages | 16 pages |
| rest dsl | 7 pages | 16 pages | 25 pages |
The site currently over\-fetches \({{hitsPerPage: 50}}\) purely to compensate.
With server\-side {{distinct}} the de\-duplication happens before the window is
applied, so the default of 20 would return 20 genuinely distinct pages and the
over\-fetch could be removed.
h2. Proposed change
In the DocSearch crawler config / index settings:
{code:json}
{
"attributeForDistinct": "url\_without\_anchor",
"distinct": true,
"attributesForFaceting": \["filterOnly\(url\)"\]
}
{code}
Note that {{url\_without\_anchor}} needs to be confirmed present on the
records; the hits currently expose {{url}}, {{hierarchy}}, {{content}},
{{version}} and {{objectID}}. If it is absent, the crawler {{recordProps}} need
to emit it, or {{attributeForDistinct}} should point at an equivalent
anchor\-free attribute.
h2. Client\-side code this would replace
In {{antora\-ui\-camel/src/js/08\-docsearch.js}}:
* {{limitHitsPerPage}} / {{MAX\_HITS\_PER\_PAGE}} \- fully replaced by
{{attributeForDistinct}}.
* {{HITS\_PER\_PAGE = 50}} \- can drop back to the DocSearch default.
* {{isSubProjectUrl}} / {{EXCLUDED\_SUBPROJECTS}} \- replaced by a
{{facetFilters}} entry in {{indices\[0\].searchParameters}}.
{{sortByCoreDocs}} would remain client side, or better, move into the index as
a custom ranking attribute.
h2. Context
This came out of the DocSearch v5 migration in camel\-website PR #1729. The
previous implementation \({{src/js/vendor/algoliasearch.bundle.js}}\)
hand\-rolled all of this against the raw Algolia client. The v5 rewrite
initially dropped it entirely; it has since been reinstated in adapted form,
but the index\-level fix is the durable one.
Summary: website: Algolia index has no working deduplication;
.docsearch.config.json settings are not in effect (was: website: configure
Algolia index for search result deduplication and faceting)
> website: Algolia index has no working deduplication; .docsearch.config.json
> settings are not in effect
> ------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24396
> URL: https://issues.apache.org/jira/browse/CAMEL-24396
> Project: Camel
> Issue Type: Improvement
> Reporter: Adriano Machado
> Priority: Major
>
> Search results on camel.apache.org are not de-duplicated: a query like
> {{timer}} returns 19 usable hits spanning a single document, so the result
> list fills with near-identical anchors from one page. The site currently
> works around this in the browser, in
> {{antora-ui-camel/src/js/08-docsearch.js}}, by over-fetching and capping hits
> per page. The durable fix is at the index level.
> h2. The repository already has a config file, but its settings are not in
> effect
> {{.docsearch.config.json}} exists at the repository root and already attempts
> both of the settings needed here:
> {code:json}
> "attributesForFaceting": ["version"],
> "attributeForDistinctResults": "url"
> {code}
> Neither is applied to the live index. Verified against the public search key:
> * {{{"facets": ["version"], "hitsPerPage": 0}}} returns {{"facets": \{\}}}.
> If {{version}} were in {{attributesForFaceting}}, this would return facet
> counts.
> * Adding {{"distinct": true}} to a query changes nothing, which means
> {{attributeForDistinct}} is unset.
> Two problems compound here.
> h3. 1. {{attributeForDistinctResults}} is not a real Algolia setting
> The setting is named {{attributeForDistinct}}, and it must be paired with
> {{distinct}}. Per the Algolia API reference for the {{distinct}} parameter:
> "{{attributeForDistinct}} defines how records are grouped, {{distinct}}
> defines how many records per group are shown. If {{attributeForDistinct}}
> isn't set, {{distinct}} is ignored."
> So {{attributeForDistinctResults}} is silently ignored regardless of what
> consumes the file, and {{distinct}} is missing entirely.
> h3. 2. The file matches neither documented DocSearch config format
> The current DocSearch / Algolia Crawler configuration is JavaScript, not
> JSON, and is structured as:
> {code:javascript}
> new Crawler({
> appId, apiKey,
> startUrls: [...],
> sitemaps: [...],
> actions: [{ indexName, pathsToMatch, recordExtractor }],
> initialIndexSettings: { INDEX_NAME: { distinct: true, attributeForDistinct:
> 'url', ... } },
> })
> {code}
> {{.docsearch.config.json}} instead nests crawler options under {{index}} and
> {{crawler}} objects and puts index settings under {{custom_settings}}. The
> {{start_urls}} / {{selectors}} / {{selectors_exclude}} / {{custom_settings}}
> / {{min_indexed_level}} / {{stop_urls}} keys resemble the deprecated
> docsearch-scraper JSON format, while {{index.pathsToMatch}},
> {{index.includeHeadingLevels}} and {{crawler.sitemapUrls}} resemble the
> Crawler format. The result matches neither.
> Nothing in the repository references the file outside documentation
> ({{README.md}} and {{.docsearch.README.md}}); there is no CI job or script
> that applies it. It was added in a single commit, 77e25115 (#1473).
> *This is the key open question:* someone with Algolia dashboard access needs
> to confirm whether {{.docsearch.config.json}} actually feeds the crawler.
> That determines whether this is fixed by a PR against camel-website or by
> editing the crawler config in the Algolia dashboard.
> h2. The records are missing the attribute the fix would normally use
> DocSearch's standard {{helpers.docsearch()}} extractor emits {{url}},
> {{url_without_anchor}}, {{anchor}}, {{type}} and {{weight}}. Retrieving all
> attributes from the live index returns only:
> {code}
> content, hierarchy, keywords, objectID, pageRank, url, version
> {code}
> So {{url_without_anchor}}, {{anchor}}, {{type}} and {{weight}} are all
> absent, and {{pageRank}} is flat rather than nested under {{weight}}. The
> {{apache_camel}} index is therefore not being built by the standard DocSearch
> record extractor.
> This matters because {{url}} on these records *includes* the anchor:
> {code}
> https://camel.apache.org/components/4.22.x/timer-component.html
> https://camel.apache.org/components/4.22.x/timer-component.html#_endpoint_query_option_time
> https://camel.apache.org/components/4.22.x/timer-component.html#_component_option_includeMetadata
> {code}
> Setting {{attributeForDistinct: "url"}} would group by anchor, which does not
> de-duplicate anything. The fix needs an anchor-free attribute, and one does
> not currently exist on the records.
> h2. Proposed change
> # Confirm whether {{.docsearch.config.json}} is actually consumed by the
> crawler, or whether the live config lives only in the Algolia dashboard. Fix
> or remove the file accordingly, since a config file that looks authoritative
> but is inert is worse than none.
> # Have the crawler emit an anchor-free URL attribute ({{url_without_anchor}},
> matching DocSearch convention).
> # Then set, in {{initialIndexSettings}} for {{apache_camel}}:
> {code:json}
> {
> "attributeForDistinct": "url_without_anchor",
> "distinct": true
> }
> {code}
> # Optionally add {{attributesForFaceting}} so sub-project exclusion can move
> server side as a {{facetFilters}} entry rather than running in the browser.
> h2. Impact on the client
> Distinct parent pages surviving the client-side sub-project filter, measured
> against the live index:
> || query || hitsPerPage 20 (DocSearch default) || hitsPerPage 50 ||
> hitsPerPage 100 ||
> | kamelet | 2 pages | 3 pages | 3 pages |
> | timer | 1 page | 7 pages | 16 pages |
> | rest dsl | 7 pages | 16 pages | 25 pages |
> With server-side {{distinct}}, de-duplication happens before the hit window
> is applied, so the DocSearch default of 20 would return 20 genuinely distinct
> pages. The following could then be deleted from
> {{antora-ui-camel/src/js/08-docsearch.js}}:
> * {{limitHitsPerPage}} / {{MAX_HITS_PER_PAGE}}, replaced by
> {{attributeForDistinct}}
> * {{HITS_PER_PAGE = 50}}, back to the DocSearch default
> * {{isSubProjectUrl}} / {{EXCLUDED_SUBPROJECTS}}, replaced by
> {{facetFilters}}, if step 4 is done
> {{sortByCoreDocs}} would remain client side, or better, move into the index
> as a custom ranking attribute.
> h2. Possibly related
> DocSearch v5's UI expects {{type}} and {{anchor}} on records; it uses
> {{item.type === 'lvl1'}} to associate sub-results with their parent heading.
> Since neither attribute exists on this index, that parent grouping cannot
> engage. Search still functions, as {{hierarchy}}, {{url}} and {{content}} are
> all present, but the result list will not render the nested parent/child
> structure DocSearch normally shows. Worth confirming when the crawler output
> is reviewed.
> h2. Context
> This came out of the DocSearch v5 migration in camel-website PR #1729. The
> previous implementation ({{src/js/vendor/algoliasearch.bundle.js}})
> hand-rolled de-duplication, sub-project filtering and ranking against the raw
> Algolia client. The v5 rewrite initially dropped all of it; it has since been
> reinstated in adapted form, but the index-level fix is the durable one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
