Compare commits

..

33 Commits

Author SHA1 Message Date
mueh 115c06e9bf Merge branch 'sbp-hardfork' 2026-04-28 09:07:16 +02:00
jm d2b8869eb4 composer update 2026-04-22 22:01:57 +00:00
MauBiBot f72acc60ef Rename blomstra/search fork to sbp/flarum-search (Phase 1: namespaces, routes, JS initializers) 2026-04-22 22:23:47 +02:00
JM fc830ea921 Add content autocomplete to search dropdown 2026-04-22 20:52:35 +02:00
MauBiBot 4538fe2ba8 Add prefix search for content and enable autocomplete 2026-04-22 20:43:33 +02:00
JM 5774879331 fuzzy search in thread 2026-04-22 20:19:09 +02:00
MauBiBot 3e2f3f9585 always use ES search on initial search page load 2026-04-22 19:46:09 +02:00
MauBiBot bf90e3279e remove frontend default sort from discussion search state 2026-04-22 19:31:30 +02:00
MauBiBot bce3258325 fix forum search route and rebuild compiled assets 2026-04-22 19:05:46 +02:00
MauBiBot 7f282cdb84 merge local search fixes after pull 2026-04-22 18:56:43 +02:00
JM 7cb4071a54 restore strong title match and keep prefix title fallback 2026-04-22 18:08:12 +02:00
JM e3dfc04192 add low-boost title autocomplete match to full search 2026-04-22 17:50:30 +02:00
MauBiBot e7e4495306 rebuild compiled JS assets for analyzer selector 2026-04-22 16:36:24 +02:00
mueh 0fa8fb92eb ai forgot to commit 2026-04-22 15:47:54 +02:00
mueh 925ecb0e7c tune title autocomplete edge_ngram bounds 2026-04-22 12:33:13 +02:00
Bart van Bragt e92307cf16 build: commit compiled JS dist files for feat/analyzer-refactor branch 2026-04-16 22:07:28 +02:00
Bart van Bragt d27d71ff72 style: apply prettier formatting to admin/index.ts 2026-04-16 11:57:59 +02:00
Bart van Bragt f7d9935fb9 feat: suppress ES instant results when search-post-bodies is enabled
When searching inside comments is ON, the autocomplete dropdown now returns
nothing from ES (only usernames show). This avoids showing title-only matches
that are inconsistent with the full comment-body search results.

Also passes filter[autocomplete]=1 to the API so the server uses the fast
title.autocomplete path instead of the full has_child query.
2026-04-16 11:57:34 +02:00
Bart van Bragt 33d4946947 feat: stem exclusion admin setting and analyzer language selector improvements
Add a stem-exclusion textarea to the admin settings page. Words entered here
(one per line) are passed as stem_exclusion to the language analyzer, preventing
them from being stemmed. Requires a full re-index to take effect.

The two language-dependent fields (stem exclusion, min search length) are
disabled when CJK is selected, since neither applies to bigram analysis.

Settings are reordered: search-scope switches now appear above the language
block so the language-dependent fields sit directly below the language selector.

Replace the lowercase ES identifier labels in the language dropdown with proper
display names (e.g. "Brazilian Portuguese", "CJK (Chinese, Japanese, Korean)",
"Sorani (Kurdish)"). Drift detection now tracks stem exclusion instead of the
removed min_gram index setting.
2026-04-16 11:52:34 +02:00
Bart van Bragt 0f58abf141 feat: add title.autocomplete instant search path sorted by latest
When filter[autocomplete]=1 is present, bypass the full has_child query
and run a single MatchQuery on title.autocomplete (edge-ngram prefix match).
Results are sorted by updated_at desc to stay consistent with full search.
2026-04-16 11:51:45 +02:00
Bart van Bragt 0adaab5d84 fix: resolve analyzer mismatch with symmetric language analyzer and title field
Replace the asymmetric ngram (index) / language (search) setup on content
with a symmetric language analyzer at both index and search time, eliminating
zero-hit cases caused by stemmed tokens falling below min_gram (e.g. Turkish).

Add a title field to discussion documents with a title.autocomplete edge-ngram
subfield (min=2, max=15) for prefix instant search, and a flarum_title_search
analyzer (lowercase only) as its search-time counterpart.

Remove flarum_analyzer_partial / partial_search_filter / search_analyzer from
the content mapping — no longer needed with symmetric analysis.

Bump INDEX_COMPAT_VERSION to v3. Also fix MatchQuery::toArray() to serialize
the analyzer property which was stored but never emitted.
2026-04-16 11:51:28 +02:00
Bart van Bragt c55db0abd6 fix: show TerminalPost in search results when field sort is active or mostRelevantPost is unavailable
When params.q is set, Flarum core suppresses TerminalPost and only shows an
excerpt from mostRelevantPost. This leaves the info section empty when the post
is null (e.g. hidden/deleted first post in production) or when results are
ordered by a field sort where an excerpt is meaningless.

Extend DiscussionListItem.infoItems to:
- Replace excerpt with TerminalPost when a field sort (latest, oldest, top) is
  active — relevance excerpts are not meaningful when sorted by date/count.
- Fall back to TerminalPost when no excerpt was added (mostRelevantPost null).
2026-04-15 21:24:55 +02:00
Bart van Bragt c5643bed31 fix: default search sort to latest and keep dropdown label in sync
- Search with no explicit sort now defaults to latest (updated_at desc)
  on both backend and frontend, matching the forum UX expectation.
- "All discussions" link from the search box routes to /?q=...&sort=latest
  so the sort dropdown shows "Latest" immediately after submission.
- extendDiscussionState injects sort=-lastPostedAt into API params when
  no sort is present, keeping direct-URL navigation (?q=anan) consistent
  with the backend default.
- Backend guard changed from phpSortField===null to empty($sorts), which
  is the correct sentinel since $sorts is the authoritative ES sort list.
2026-04-15 17:21:10 +02:00
Bart van Bragt 6bcd2becd5 fix: report actual failed item error in bulk seeding exception 2026-04-15 14:08:04 +02:00
Bart van Bragt 2706ed6be0 fix: index discussion on rename 2026-04-15 14:07:55 +02:00
Bart van Bragt 557604e426 fix: retry fill ES queries on transient connection failure 2026-04-15 14:07:43 +02:00
Bart van Bragt 08353d2473 perf: sort by date by default, skip scoring on field sorts
- Default sort changed from relevance to updated_at desc. This lets
  has_child use score_mode:none, which skips child scoring entirely
  and lets ES short-circuit early on large corpora.
- When a field sort is requested, score_mode is also none for the same
  reason. score_mode:sum is only used when sorting by relevance.
- track_total_hits:false avoids a full-index count on every query,
  allowing ES to stop once it has collected enough results.
- Strip all gambit operators (tag:foo, author:bar, is:unread, etc.)
  from the ES query string, not just is:private. Leaving them in caused
  operator:and to require the gambit tokens to appear literally in post
  content, producing zero results when gambits were combined with text.
2026-04-14 22:40:43 +02:00
Bart van Bragt 80357e6a77 fix: include --staging in build conflict suggestions 2026-04-14 16:46:06 +02:00
Bart van Bragt d8eebb183a fix: prevent ES CPU saturation from has_child scoring all posts
Without minimum_should_match=1 on the inner post bool query, Elasticsearch
defaults MSM to 0 whenever a filter clause is present. This caused has_child
to score every non-hidden post on every search request, saturating CPU on
shared ES nodes with large corpora (3.7M+ posts).

- Add BoolQuery subclass with create() override and minimumShouldMatch()
  (spatie/elasticsearch-query-builder 1.x uses `new self()` in create(),
  so subclassing requires overriding it)
- Set minimumShouldMatch(1) on the inner post query after adding the
  is_hidden filter, so only genuinely matching posts are scored
- Remove the operator('or') clause from buildShouldClauses() — with
  Turkish min_ngram=2, 'or' generates 2-gram tokens that match nearly
  every post, causing near-total index scans
- Add ES client timeouts (connect: 2s, query: 10s) to prevent Apache
  mod_php worker saturation when ES is slow or unreachable
2026-04-14 16:07:50 +02:00
Bart van Bragt c54f37349f fix: use Support\Collection for cachedGlobalPermission — pluck() returns plain Collection, not Eloquent 2026-04-14 11:27:02 +02:00
Bart van Bragt ca8c248870 fix: use Eloquent Collection in job — loadMissing does not exist on Support\Collection 2026-04-14 09:59:02 +02:00
Bart van Bragt bf37e01eb0 fix: read forum attribute lazily inside extend callback 2026-04-14 09:33:17 +02:00
Bart van Bragt 4709b9d1db fix: is_hidden boolean coercion and forum settings access 2026-04-14 08:59:19 +02:00
38 changed files with 599 additions and 226 deletions

1
.gitignore vendored
View File

@ -2,3 +2,4 @@ js/node_modules
js/dist
vendor/
composer.lock
analysis/*

View File

@ -1,5 +1,5 @@
{
"name": "blomstra/search",
"name": "sbp/flarum-search",
"description": "Replaces Flarum search with one powered by an elastic search server.",
"keywords": [
"extension",
@ -11,16 +11,20 @@
"elasticsearch"
],
"support": {
"email": "helpdesk@blomstra.net",
"email": "mueh+git@sbp.de",
"forum": "https://blomstra.community/t/ext-search"
},
"type": "flarum-extension",
"license": "MIT",
"authors": [
{
"name": "sbp",
"email": "info@sbp.de"
},
{
"name": "Team Blomstra",
"email": "development@blomstra.net",
"role": "Developer"
"role": "Original Developer"
}
],
"require": {

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search;
namespace Sbp\Search;
use Flarum\Extend as Flarum;
@ -26,13 +26,16 @@ return [
new Flarum\Locales(__DIR__.'/resources/locale'),
(new Flarum\Routes('api'))
->get('/blomstra/search/{type}', 'blomstra.search', Api\Controllers\SearchController::class)
->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class),
->get('/sbp/search/{type}', 'sbp.search', Api\Controllers\SearchController::class)
->put('/sbp/search/index', 'sbp.search.index', Api\Controllers\IndexController::class),
(new Flarum\Console())
->command(Commands\BuildCommand::class),
(new Flarum\Settings())
->default('blomstra-search.search-discussion-subjects', true)
->default('blomstra-search.search-post-bodies', true)
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH),
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH)
->default('blomstra-search.stem-exclusion', '')
->serializeToForum('blomstraSearchMinLength', 'blomstra-search.min-search-length', 'intval')
->serializeToForum('blomstraSearchPostBodies', 'blomstra-search.search-post-bodies', fn ($v) => (bool) $v, true),
];

2
js/dist/admin.js vendored
View File

@ -1,2 +1,2 @@
(()=>{var t={n:a=>{var e=a&&a.__esModule?()=>a.default:()=>a;return t.d(e,{a:e}),e},d:(a,e)=>{for(var r in e)t.o(e,r)&&!t.o(a,r)&&Object.defineProperty(a,r,{enumerable:!0,get:e[r]})},o:(t,a)=>Object.prototype.hasOwnProperty.call(t,a)};(()=>{"use strict";function a(t,e){return a=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(t,a){return t.__proto__=a,t},a(t,e)}const e=flarum.core.compat["admin/app"];var r=t.n(e);const n=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=t.n(s);const o=flarum.core.compat["admin/components/DashboardWidget"];var l=t.n(o);const c=flarum.core.compat["common/components/Alert"];var h=t.n(c),d=function(t){var e,n;function s(){return t.apply(this,arguments)||this}n=t,(e=s).prototype=Object.create(n.prototype),e.prototype.constructor=e,a(e,n);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:r().translator.trans("blomstra-search.admin.reindex-required.title")},r().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(l());r().initializers.add("blomstra-search",function(){var t=r().data.settings["blomstra-search.active-index"],a=r().data.settings["blomstra-search.index-compatible"];t&&"v2"!==a&&(0,n.extend)(i().prototype,"availableWidgets",function(t){t.add("blomstra-search-reindex",m(d),110)});var e=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach(function(t){e.set(t,t)}),r().extensionData.for("blomstra-search").registerSetting(function(){var a=r().data.settings["blomstra-search.indexed-analyzer"];if(!t||!a)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",n=this.setting("blomstra-search.min-search-length")(),s=String(r().data.settings["blomstra-search.indexed-min-search-length"]||r().data.settings["blomstra-search.min-search-length"]);return e===a&&n===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},r().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:r().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:r().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:r().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:r().translator.trans("blomstra-search.admin.analyzer.label"),help:r().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(e.entries()),default:"english"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:r().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:r().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.min-search-length",label:r().translator.trans("blomstra-search.admin.min-search-length.label"),help:r().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:r().data.settings["blomstra-search.min-search-length"]})})})(),module.exports={}})();
(()=>{var a={n:t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return a.d(e,{a:e}),e},d:(t,e)=>{for(var n in e)a.o(e,n)&&!a.o(t,n)&&Object.defineProperty(t,n,{enumerable:!0,get:e[n]})},o:(a,t)=>Object.prototype.hasOwnProperty.call(a,t)};(()=>{"use strict";function t(a,e){return t=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(a,t){return a.__proto__=t,a},t(a,e)}const e=flarum.core.compat["admin/app"];var n=a.n(e);const r=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=a.n(s);const l=flarum.core.compat["admin/components/DashboardWidget"];var o=a.n(l);const c=flarum.core.compat["common/components/Alert"];var h=a.n(c),d=function(a){var e,r;function s(){return a.apply(this,arguments)||this}r=a,(e=s).prototype=Object.create(r.prototype),e.prototype.constructor=e,t(e,r);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:n().translator.trans("blomstra-search.admin.reindex-required.title")},n().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(o());n().initializers.add("sbp-search",function(){var a=n().data.settings["blomstra-search.active-index"],t=n().data.settings["blomstra-search.index-compatible"];a&&"v3"!==t&&(0,r.extend)(i().prototype,"availableWidgets",function(a){a.add("blomstra-search-reindex",m(d),110)}),n().extensionData.for("blomstra-search").registerSetting(function(){var t=n().data.settings["blomstra-search.indexed-analyzer"];if(!a||!t)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",r=this.setting("blomstra-search.stem-exclusion")()||"",s=n().data.settings["blomstra-search.indexed-stem-exclusion"]||"";return e===t&&r===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},n().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:n().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:n().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:n().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n().translator.trans("blomstra-search.admin.analyzer.label"),help:n().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:{arabic:"Arabic",armenian:"Armenian",basque:"Basque",bengali:"Bengali",brazilian:"Brazilian Portuguese",bulgarian:"Bulgarian",catalan:"Catalan",cjk:"CJK (Chinese, Japanese, Korean)",czech:"Czech",danish:"Danish",dutch:"Dutch",english:"English",estonian:"Estonian",finnish:"Finnish",french:"French",galician:"Galician",german:"German",greek:"Greek",hindi:"Hindi",hungarian:"Hungarian",indonesian:"Indonesian",irish:"Irish",italian:"Italian",latvian:"Latvian",lithuanian:"Lithuanian",norwegian:"Norwegian",persian:"Persian",portuguese:"Portuguese",romanian:"Romanian",russian:"Russian",sorani:"Sorani (Kurdish)",spanish:"Spanish",standard:"Standard (generic)",swedish:"Swedish",turkish:"Turkish",thai:"Thai"},default:"english"}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.stem-exclusion",type:"textarea",label:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.label"),help:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.help"),disabled:a})}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.min-search-length",label:n().translator.trans("blomstra-search.admin.min-search-length.label"),help:n().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:n().data.settings["blomstra-search.min-search-length"],disabled:a})})})})(),module.exports={}})();
//# sourceMappingURL=admin.js.map

File diff suppressed because one or more lines are too long

2
js/dist/forum.js vendored

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@ -4,7 +4,7 @@ import DashboardPage from 'flarum/admin/components/DashboardPage';
import DashboardWidget from 'flarum/admin/components/DashboardWidget';
import Alert from 'flarum/common/components/Alert';
const REQUIRED_INDEX_COMPAT = 'v2';
const REQUIRED_INDEX_COMPAT = 'v3';
class ReindexWarningWidget extends DashboardWidget {
className() {
@ -25,7 +25,7 @@ class ReindexWarningWidget extends DashboardWidget {
}
}
app.initializers.add('blomstra-search', () => {
app.initializers.add('sbp-search', () => {
const activeIndex = app.data.settings['blomstra-search.active-index'];
const compatVersion = app.data.settings['blomstra-search.index-compatible'];
@ -35,46 +35,44 @@ app.initializers.add('blomstra-search', () => {
});
}
const languages = new Map();
[
'arabic',
'armenian',
'basque',
'bengali',
'brazilian',
'bulgarian',
'catalan',
'cjk',
'czech',
'danish',
'dutch',
'english',
'estonian',
'finnish',
'french',
'galician',
'german',
'greek',
'hindi',
'hungarian',
'indonesian',
'irish',
'italian',
'latvian',
'lithuanian',
'norwegian',
'persian',
'portuguese',
'romanian',
'russian',
'sorani',
'spanish',
'swedish',
'turkish',
'thai',
].forEach((language) => {
languages.set(language, language);
});
const languages: Record<string, string> = {
arabic: 'Arabic',
armenian: 'Armenian',
basque: 'Basque',
bengali: 'Bengali',
brazilian: 'Brazilian Portuguese',
bulgarian: 'Bulgarian',
catalan: 'Catalan',
cjk: 'CJK (Chinese, Japanese, Korean)',
czech: 'Czech',
danish: 'Danish',
dutch: 'Dutch',
english: 'English',
estonian: 'Estonian',
finnish: 'Finnish',
french: 'French',
galician: 'Galician',
german: 'German',
greek: 'Greek',
hindi: 'Hindi',
hungarian: 'Hungarian',
indonesian: 'Indonesian',
irish: 'Irish',
italian: 'Italian',
latvian: 'Latvian',
lithuanian: 'Lithuanian',
norwegian: 'Norwegian',
persian: 'Persian',
portuguese: 'Portuguese',
romanian: 'Romanian',
russian: 'Russian',
sorani: 'Sorani (Kurdish)',
spanish: 'Spanish',
standard: 'Standard (generic)',
swedish: 'Swedish',
turkish: 'Turkish',
thai: 'Thai',
};
app.extensionData
.for('blomstra-search')
@ -84,12 +82,10 @@ app.initializers.add('blomstra-search', () => {
if (!activeIndex || !indexedAnalyzer) return null;
const currentAnalyzer = this.setting('blomstra-search.analyzer-language')() || 'english';
const currentMinLength = this.setting('blomstra-search.min-search-length')();
const indexedMinLength = String(
app.data.settings['blomstra-search.indexed-min-search-length'] || app.data.settings['blomstra-search.min-search-length']
);
const currentStemExclusion = this.setting('blomstra-search.stem-exclusion')() || '';
const indexedStemExclusion = app.data.settings['blomstra-search.indexed-stem-exclusion'] || '';
if (currentAnalyzer === indexedAnalyzer && currentMinLength === indexedMinLength) return null;
if (currentAnalyzer === indexedAnalyzer && currentStemExclusion === indexedStemExclusion) return null;
return m(
Alert,
@ -118,14 +114,6 @@ app.initializers.add('blomstra-search', () => {
default: 'flarum',
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.analyzer-language',
label: app.translator.trans('blomstra-search.admin.analyzer.label'),
help: app.translator.trans('blomstra-search.admin.analyzer.help'),
type: 'select',
options: Object.fromEntries(languages.entries()),
default: 'english',
})
.registerSetting({
setting: 'blomstra-search.search-discussion-subjects',
label: app.translator.trans('blomstra-search.admin.search-discussion-subjects'),
@ -137,11 +125,33 @@ app.initializers.add('blomstra-search', () => {
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.analyzer-language',
label: app.translator.trans('blomstra-search.admin.analyzer.label'),
help: app.translator.trans('blomstra-search.admin.analyzer.help'),
type: 'select',
options: languages,
default: 'english',
})
.registerSetting(function (this: any) {
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
return this.buildSettingComponent({
setting: 'blomstra-search.stem-exclusion',
type: 'textarea',
label: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.label'),
help: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.help'),
disabled: isCjk,
});
})
.registerSetting(function (this: any) {
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
return this.buildSettingComponent({
setting: 'blomstra-search.min-search-length',
label: app.translator.trans('blomstra-search.admin.min-search-length.label'),
help: app.translator.trans('blomstra-search.admin.min-search-length.help'),
type: 'select',
options: { '1': '1', '2': '2', '3': '3', '4': '4' },
default: app.data.settings['blomstra-search.min-search-length'],
disabled: isCjk,
});
});
});

View File

@ -6,10 +6,9 @@ import DiscussionListState from 'flarum/forum/states/DiscussionListState';
export default function extendDiscussionState() {
override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) {
const preloaded = app.data.apiDocument || null;
// If existing payload is given or no search is made, fallback on native page.
if (preloaded || !this.requestParams()?.filter?.q) return original.call(this, page);
if (!this.requestParams()?.filter?.q) return original.call(this, page);
const params = this.requestParams();
params.page = {
@ -17,12 +16,13 @@ export default function extendDiscussionState() {
...params.page,
};
if (Array.isArray(params.include)) {
params.include = params.include.join(',');
}
// Construct API search URI
const url = `${app.forum.attribute('apiUrl')}/blomstra/search/${this.type}`;
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
// Make API GET request
const results = await app.request({ params, url, method: 'GET' });

View File

@ -21,18 +21,21 @@ export default class DiscussionsSearchSource implements SearchSource {
private type = 'discussions';
async search(query: string): Promise<void> {
// Suppress ES instant results when "search within comments" is ON;
// only usernames are shown in that mode.
query = query.toLowerCase();
this.results.set(query, []);
const params = {
filter: { q: query },
filter: { q: query, autocomplete: 1 },
page: { limit: 3 },
include: 'mostRelevantPost',
};
// Construct API search URI
const url = `${app.forum.attribute('apiUrl')}/blomstra/search/${this.type}`;
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
// Make API GET request
const results = await app.request({ params, url, method: 'GET' });

View File

@ -0,0 +1,26 @@
import { extend } from 'flarum/common/extend';
import DiscussionListItem from 'flarum/forum/components/DiscussionListItem';
import TerminalPost from 'flarum/forum/components/TerminalPost';
export default function extendDiscussionListItem() {
extend(DiscussionListItem.prototype, 'infoItems', function (items) {
const params = this.attrs.params;
if (!params.q) return;
const hasFieldSort = params.sort && params.sort !== 'relevance';
if (hasFieldSort) {
// Field sort active (latest, oldest, top, …): replace excerpt with TerminalPost.
// An excerpt is meaningless when results are ordered by date/count rather than relevance.
items.remove('excerpt');
if (!items.has('terminalPost')) {
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={!this.showFirstPost()} />);
}
} else if (!items.has('excerpt')) {
// Relevance mode but no excerpt (mostRelevantPost was null or non-comment type).
// Fall back to TerminalPost so the info section is never silently empty.
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={true} />);
}
});
}

View File

@ -7,26 +7,26 @@ import ItemList from 'flarum/common/utils/ItemList';
import DiscussionsSearchSource from './SearchSources/DiscussionsSearchSource';
import extendDiscussionState from './PaginatedListStates/extendDiscussionState';
import extendDiscussionListItem from './extendDiscussionListItem';
app.initializers.add('blomstra-search', () => {
const minLength = parseInt(app.data.settings['blomstra-search.min-search-length'] || String(Search.MIN_SEARCH_LEN), 10);
app.initializers.add('sbp-search', () => {
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) {
// app.forum is not available during initializers (it is set after they run),
// so read the setting lazily here, at first render time.
const minLength = (app.forum.attribute('blomstraSearchMinLength') as number) || Search.MIN_SEARCH_LEN;
if (minLength !== Search.MIN_SEARCH_LEN) {
// Flarum provides no extension point for MIN_SEARCH_LEN, so we overwrite the
// static property directly. TypeScript `readonly` is compile-time only — at
// runtime this is a plain property assignment and is safe as long as no code
// reads MIN_SEARCH_LEN before this initializer runs.
(Search as any).MIN_SEARCH_LEN = minLength;
}
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) {
items.replace('discussions', new DiscussionsSearchSource());
});
});
app.initializers.add(
'blomstra-search-early',
'sbp-search-early',
() => {
extendDiscussionState();
extendDiscussionListItem();
},
999999
);

View File

@ -0,0 +1,14 @@
diff --git a/src/Commands/BuildCommand.php b/src/Commands/BuildCommand.php
index 9024e1e..9024e1e 100644
--- a/src/Commands/BuildCommand.php
+++ b/src/Commands/BuildCommand.php
@@ -38,6 +38,6 @@ class BuildCommand extends Command
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
- private const TITLE_MIN_GRAM = 2;
- private const TITLE_MAX_GRAM = 15;
+ private const TITLE_MIN_GRAM = 3;
+ private const TITLE_MAX_GRAM = 20;
protected $signature = 'blomstra:search:index

View File

@ -17,9 +17,12 @@ blomstra-search:
search-post-bodies: Search inside comments
match-sentences: Match search term against full sentence
match-words: Match search term against full words
settings:
stem-exclusion:
label: "Protected words (stem exclusion)"
help: "One word per line. These words will not be stemmed during search (e.g. brand names, proper nouns). Not applicable for CJK or Thai languages. Requires a full re-index to take effect."
min-search-length:
label: Minimum search query length
help: |
Minimum number of characters required before a search is triggered. Lower this to 1 or 2
for CJK (Chinese, Japanese, Korean) communities where single characters carry full meaning.
Changing this requires a full index rebuild to take effect.

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Api;
namespace Sbp\Search\Api;
use Illuminate\Support\Arr;
use Psr\Http\Message\ResponseInterface;

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Api\Controllers;
namespace Sbp\Search\Api\Controllers;
use Blomstra\Search\Commands\BuildCommand;
use Sbp\Search\Commands\BuildCommand;
use Laminas\Diactoros\Response\EmptyResponse;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;

View File

@ -10,15 +10,15 @@
*
*/
namespace Blomstra\Search\Api\Controllers;
namespace Sbp\Search\Api\Controllers;
use Blomstra\Search\Elasticsearch\HasChildQuery;
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
use Blomstra\Search\Elasticsearch\MatchQuery;
use Blomstra\Search\Elasticsearch\TermsQuery;
use Blomstra\Search\Searchers\CommentPostSearcher;
use Blomstra\Search\Searchers\DiscussionSearcher;
use Blomstra\Search\Searchers\Searcher;
use Sbp\Search\Elasticsearch\HasChildQuery;
use Sbp\Search\Elasticsearch\MatchPhraseQuery;
use Sbp\Search\Elasticsearch\MatchQuery;
use Sbp\Search\Elasticsearch\TermsQuery;
use Sbp\Search\Searchers\CommentPostSearcher;
use Sbp\Search\Searchers\DiscussionSearcher;
use Sbp\Search\Searchers\Searcher;
use Elasticsearch\Client;
use Flarum\Api\Controller\ListDiscussionsController;
use Flarum\Api\Serializer\DiscussionSerializer;
@ -37,7 +37,7 @@ use Illuminate\Support\Str;
use Psr\Http\Message\ServerRequestInterface;
use Psr\Log\LoggerInterface;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Sbp\Search\Elasticsearch\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
use Tobscure\JsonApi\Document;
@ -78,27 +78,18 @@ class SearchController extends ListDiscussionsController
$offset = $this->extractOffset($request);
$include = array_merge($this->extractInclude($request), ['state']);
$query = BoolQuery::create()
// Always restrict to discussion documents; posts are only searched via has_child.
->add(TermQuery::create('join_field', 'discussion'), 'filter');
if (!empty($search)) {
$query->add($this->buildTextQuery($search, $actor));
// Autocomplete path: fast prefix match on title.autocomplete only.
if (!empty($filters['autocomplete']) && !empty($search)) {
return $this->handleAutocomplete($search, $actor, $limit, $offset, $include, $document, $request);
}
$this->addFilters($query, $actor, $filters);
$builder = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->size($limit + 1)
->from($offset)
->addQuery($query);
$knownSortFields = array_merge(array_values($this->translateSort), ['rawId']);
$logger = resolve(LoggerInterface::class);
$phpSortField = null;
$phpSortDir = 'desc';
$needsScoring = true;
$sorts = [];
foreach ($this->extractSort($request) as $field => $direction) {
$translated = $this->translateSort[$field] ?? $field;
@ -108,7 +99,8 @@ class SearchController extends ListDiscussionsController
continue;
}
$builder->addSort(new Sort($translated, $direction));
$sorts[] = new Sort($translated, $direction);
$needsScoring = false;
if ($phpSortField === null && $translated !== 'rawId') {
$phpSortField = $translated;
@ -116,7 +108,45 @@ class SearchController extends ListDiscussionsController
}
}
$response = $builder->search();
// Default to latest when no explicit sort is requested. This lets has_child use
// score_mode:none, which skips child scoring entirely and lets ES short-circuit
// early on large corpora. Relevance is still available via sort=relevant if added.
if (empty($sorts)) {
$needsScoring = false;
$phpSortField = 'updated_at';
$phpSortDir = 'desc';
$sorts[] = new Sort('updated_at', 'desc');
}
$query = BoolQuery::create()
// Always restrict to discussion documents; posts are only searched via has_child.
->add(TermQuery::create('join_field', 'discussion'), 'filter');
if (!empty($search)) {
$query->add($this->buildTextQuery($search, $actor, $needsScoring));
}
$this->addFilters($query, $actor, $filters);
$builder = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->addQuery($query);
foreach ($sorts as $sort) {
$builder->addSort($sort);
}
// track_total_hits: false lets ES stop counting once it has collected
// enough results in sort order, avoiding a full-index count on every query.
$payload = $builder->getPayload();
$payload['track_total_hits'] = false;
$response = $this->elastic->search([
'index' => resolve('blomstra.search.elastic_index'),
'size' => $limit + 1,
'from' => $offset,
'body' => $payload,
]);
Discussion::setStateUser($actor);
@ -192,18 +222,145 @@ class SearchController extends ListDiscussionsController
return $discussions;
}
/**
* Fast autocomplete path: match prefix n-grams on title.autocomplete only.
* No has_child, no phrase scoring a single MatchQuery suffices.
*/
protected function handleAutocomplete(
string $search,
User $actor,
int $limit,
int $offset,
array $include,
Document $document,
ServerRequestInterface $request
): mixed {
$query = BoolQuery::create()
->add(TermQuery::create('join_field', 'discussion'), 'filter');
$query->add($this->buildAutocompleteQuery($search), 'must');
$this->addFilters($query, $actor, []);
$payload = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->addQuery($query)
->addSort(new Sort('updated_at', 'desc'))
->getPayload();
$response = $this->elastic->search([
'index' => resolve('blomstra.search.elastic_index'),
'size' => $limit + 1,
'from' => $offset,
'body' => $payload,
]);
Discussion::setStateUser($actor);
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(fn ($hit) => [
'discussion_id' => Str::after($hit['_id'], 'discussions:'),
'most_relevant_post_id' => null,
'weight' => Arr::get($hit, 'sort.0', 0),
]);
$document->addPaginationLinks(
$this->uri->to('api')->route('blomstra.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
$results->count() > $limit ? null : 0
);
$results = $results->take($limit);
$discussions = Discussion::query()
->when(
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
fn ($q) => $q->whereNull('hidden_at')
)
->whereIn('id', $results->pluck('discussion_id')->filter())
->get()
->each(function (Discussion $discussion) use ($results) {
$result = $results->firstWhere('discussion_id', $discussion->id);
$discussion->most_relevant_post_id = $result['most_relevant_post_id'] ?? $discussion->first_post_id;
$discussion->weight = $result['weight'] ?? 0;
})
->sortByDesc('updated_at')
->unique();
$this->loadRelations($discussions, $include);
return $discussions;
}
protected function buildAutocompleteQuery(string $search): BoolQuery
{
$query = BoolQuery::create();
// Title prefix match (stronger)
$query->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(1.0),
'should'
);
// Content prefix match (weaker)
$query->add(
(new MatchQuery('content.autocomplete', $search))
->operator('and')
->boost(0.4),
'should'
);
$query->minimumShouldMatch(1);
return $query;
}
/**
* Build the text-matching portion of the query.
*
* Discussion titles are matched directly (filtered to join_field=discussion).
* Post bodies are matched via has_child with score_mode=sum so discussions
* with many matching posts score higher than those with a single strong match.
* inner_hits returns the best-scoring post for use as mostRelevantPost.
* Post bodies are matched via has_child. When $needsScoring is true (no explicit
* sort, i.e. relevance ordering), score_mode=sum accumulates child scores onto the
* parent so that discussions with many strongly-matching posts rank higher. When
* false (any explicit field sort), score_mode=none skips scoring entirely ES only
* checks whether a matching child exists, which is significantly cheaper on large
* corpora.
* inner_hits returns the best-matching post for use as mostRelevantPost.
* Hidden posts are only included in matching for users with post.hide permission.
*/
protected function buildTextQuery(string $search, User $actor): BoolQuery
protected function buildTextQuery(string $search, User $actor, bool $needsScoring = false): BoolQuery
{
$textQuery = BoolQuery::create();
// Strong direct title match for the full search page.
$textQuery->add(
(new MatchQuery('title', $search))
->operator('and')
->boost(4.0),
'should'
);
// Low-weight prefix match on title.autocomplete for partial title hits.
$textQuery->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(0.35),
'should'
);
// Low-weight prefix match on title.autocomplete for the full search page.
// Exact / normal title matches should still rank higher than prefix-only hits.
$textQuery->add(
(new MatchQuery('title.autocomplete', $search))
->operator('and')
->boost(0.35),
'should'
);
if ($this->discussionSearcher?->enabled()) {
$textQuery->add($this->buildShouldClauses($search, $this->discussionSearcher->boost()), 'should');
@ -214,11 +371,15 @@ class SearchController extends ListDiscussionsController
// Guests and non-moderators may not see hidden posts; exclude them from child matching.
if ($actor->isGuest() || !$actor->hasPermission('post.hide')) {
$postQuery->add(TermQuery::create('is_hidden', false), 'filter');
$postQuery->add(TermQuery::create('is_hidden', 'false'), 'filter');
}
// Without minimum_should_match, ES default MSM is 0 when a filter clause is present,
// causing has_child to score every non-hidden post instead of only matching ones.
$postQuery->minimumShouldMatch(1);
$textQuery->add(
HasChildQuery::create('post', $postQuery)->withInnerHits(),
HasChildQuery::create('post', $postQuery, $needsScoring ? 'sum' : 'none')->withInnerHits(),
'should'
);
}
@ -235,7 +396,7 @@ class SearchController extends ListDiscussionsController
}
if ($this->matchWords) {
$should->add((new MatchQuery('content', $search))->operator('and')->boost(1.8 * $boost), 'should');
$should->add((new MatchQuery('content', $search))->operator('or')->boost(0.8 * $boost), 'should');
$should->add((new MatchQuery('content.autocomplete', $search))->operator('and')->boost(0.30 * $boost), 'should');
}
return $should;
@ -301,8 +462,12 @@ class SearchController extends ListDiscussionsController
$search = Arr::get($filters, 'q');
if ($search) {
// Strip Flarum gambit operators (tag:foo, author:bar, is:private, etc.)
// before passing to ES. These are structural filters handled separately;
// leaving them in causes operator:and to require the gambit tokens to
// appear literally in post content, producing zero results.
$q = collect(explode(' ', $search))
->filter(fn (string $part) => $part !== 'is:private')
->filter(fn (string $part) => !preg_match('/^\w+:/', $part))
->filter()
->join(' ');

View File

@ -10,12 +10,13 @@
*
*/
namespace Blomstra\Search\Commands;
namespace Sbp\Search\Commands;
use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\UpdateSearchJob;
use Blomstra\Search\Seeders\Seeder;
use Sbp\Search\Jobs\Job;
use Sbp\Search\Jobs\UpdateSearchJob;
use Sbp\Search\Seeders\Seeder;
use Elasticsearch\Client;
use Elasticsearch\Common\Exceptions\ElasticsearchException;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Console\Command;
use Illuminate\Contracts\Container\Container;
@ -35,11 +36,15 @@ class BuildCommand extends Command
* read back into blomstra-search.index-compatible by saveIndexedConfig so the
* value always reflects the live index including after a promote/rollback.
*/
public const INDEX_COMPAT_VERSION = 'v2';
public const INDEX_COMPAT_VERSION = 'v3';
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
private const TITLE_MIN_GRAM = 3;
private const TITLE_MAX_GRAM = 20;
protected $signature = 'blomstra:search:index
{action? : build | promote | rollback | discard | mapping | fill}
{--fresh}
@ -155,7 +160,9 @@ HELP;
$this->line('');
$this->line('Choose one of:');
$this->line(' blomstra:search:index build --resume Continue from where it left off');
$this->line(' blomstra:search:index build --resume --staging Continue and keep in staging when done');
$this->line(' blomstra:search:index build --fresh Drop this build and start completely fresh');
$this->line(' blomstra:search:index build --fresh --staging Start fresh and keep in staging when done');
return;
}
@ -359,17 +366,7 @@ HELP;
$rangeTo = $continueAt;
if ($seedMissing) {
$response = (new Builder($client))
->index($targetIndex)
->size(2500)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
->add(TermQuery::create('join_field', $seeder->joinRelation()))
)
->search();
$seeded = Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
$seeded = $this->queryIndexedIds($client, $targetIndex, $seeder->joinRelation(), $rangeFrom, $rangeTo);
}
/** @var Collection $collection */
@ -499,46 +496,65 @@ HELP;
$analysis = Arr::get($settingsResponse, "$indexName.settings.index.analysis", []);
$analyzer = Arr::get($analysis, 'analyzer.flarum_analyzer.type', 'english');
$minGram = (int) Arr::get($analysis, 'filter.partial_search_filter.min_gram', self::DEFAULT_MIN_SEARCH_LENGTH);
$stemExclusion = Arr::get($analysis, 'analyzer.flarum_analyzer.stem_exclusion', []);
$mappingResponse = $client->indices()->getMapping(['index' => $indexName]);
$compatVersion = Arr::get($mappingResponse, "$indexName.mappings._meta.index_compat_version");
$settings->set('blomstra-search.indexed-analyzer', $analyzer);
$settings->set('blomstra-search.indexed-min-search-length', $minGram);
$settings->set('blomstra-search.indexed-stem-exclusion', implode("\n", $stemExclusion));
$settings->set('blomstra-search.index-compatible', $compatVersion);
}
protected function buildIndexSettings(SettingsRepositoryInterface $settings): array
{
$language = $settings->get('blomstra-search.analyzer-language') ?: 'english';
$minGram = max(1, (int) ($settings->get('blomstra-search.min-search-length') ?: self::DEFAULT_MIN_SEARCH_LENGTH));
$maxGram = 10;
if ($minGram >= $maxGram) {
$this->error("min_gram ($minGram) must be less than max_gram ($maxGram). Using default.");
$minGram = self::DEFAULT_MIN_SEARCH_LENGTH;
$raw = $settings->get('blomstra-search.stem-exclusion', '');
$stemExclusion = array_values(array_filter(array_map('trim', explode("\n", $raw))));
if ($language === 'cjk') {
// CJK uses the built-in bigram analyzer; no autocomplete subfield.
return [
'analysis' => [
'analyzer' => [
'flarum_analyzer' => ['type' => 'cjk'],
],
],
];
}
$analyzerConfig = ['type' => $language];
if (!empty($stemExclusion)) {
$analyzerConfig['stem_exclusion'] = $stemExclusion;
}
return [
'index.max_ngram_diff' => $maxGram - $minGram,
'index.max_ngram_diff' => self::TITLE_MAX_GRAM - self::TITLE_MIN_GRAM,
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $language,
'filter' => [
'title_autocomplete_filter' => [
'type' => 'edge_ngram',
'min_gram' => self::TITLE_MIN_GRAM,
'max_gram' => self::TITLE_MAX_GRAM,
'token_chars' => ['letter', 'digit'],
],
'flarum_analyzer_partial' => [
],
'analyzer' => [
// Symmetric language analyzer: used for BOTH index and search on content + title.
'flarum_analyzer' => $analyzerConfig,
// Title autocomplete index-time: prefix n-grams, no stemming.
'flarum_title_autocomplete' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase', 'partial_search_filter'],
'filter' => ['lowercase', 'title_autocomplete_filter'],
],
],
'filter' => [
'partial_search_filter' => [
'type' => 'ngram',
'min_gram' => $minGram,
'max_gram' => $maxGram,
'token_chars' => ['letter', 'digit', 'symbol'],
// Title autocomplete search-time: just lowercase — predictable on partial words.
'flarum_title_search' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase'],
],
],
],
@ -547,27 +563,101 @@ HELP;
protected function mappingProperties(): array
{
$language = resolve(SettingsRepositoryInterface::class)->get('blomstra-search.analyzer-language') ?: 'english';
$titleMapping = [
'type' => 'text',
'analyzer' => 'flarum_analyzer',
];
// Only add the autocomplete subfield for non-CJK languages.
if ($language !== 'cjk') {
$titleMapping['fields'] = [
'autocomplete' => [
'type' => 'text',
'analyzer' => 'flarum_title_autocomplete',
'search_analyzer' => 'flarum_title_search',
],
];
}
$contentMapping = [
'type' => 'text',
'analyzer' => 'flarum_analyzer',
];
// Only add the autocomplete subfield for non-CJK languages.
if ($language !== 'cjk') {
$contentMapping['fields'] = [
'autocomplete' => [
'type' => 'text',
'analyzer' => 'flarum_title_autocomplete',
'search_analyzer' => 'flarum_title_search',
],
];
}
return [
'_meta' => ['index_compat_version' => self::INDEX_COMPAT_VERSION],
'properties' => [
'join_field' => ['type' => 'join', 'relations' => ['discussion' => 'post']],
'discussion_id' => ['type' => 'integer'],
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
'content' => $contentMapping,
'title' => $titleMapping,
'rawId' => ['type' => 'integer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'is_sticky' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'],
'tags' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'],
'user_id' => ['type' => 'integer'],
'groups' => ['type' => 'keyword'],
'comment_count' => ['type' => 'integer'],
'view_count' => ['type' => 'integer'],
'tags' => ['type' => 'keyword'],
'is_sticky' => ['type' => 'boolean'],
'is_hidden' => ['type' => 'boolean'],
],
];
}
/**
* Query ES for rawIds already indexed in $targetIndex for the given $joinRelation and ID range.
* Retries on transient ES failures (NoNodesAvailableException / other ElasticsearchException)
* by sleeping past the StaticNoPingConnectionPool dead-node timeout (default 60 s) before
* each retry, giving the pool a chance to resurface the node.
*/
protected function queryIndexedIds(
Client $client,
string $targetIndex,
string $joinRelation,
int $rangeFrom,
int $rangeTo,
int $maxRetries = 10
): array {
$attempt = 0;
while (true) {
try {
$response = (new Builder($client))
->index($targetIndex)
->size(2500)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
->add(TermQuery::create('join_field', $joinRelation))
)
->search();
return Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
} catch (ElasticsearchException $e) {
$attempt++;
if ($attempt >= $maxRetries) {
throw $e;
}
$this->warn("ES error on range {$rangeFrom}{$rangeTo} (attempt {$attempt}/{$maxRetries}): {$e->getMessage()}. Waiting 65 s before retry…");
sleep(65); // outlast the StaticNoPingConnectionPool dead-node window (default 60 s)
}
}
}
protected function getContinueAt(SettingsRepositoryInterface $settings, string $type): ?int
{

View File

@ -0,0 +1,41 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Sbp\Search\Elasticsearch;
class BoolQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery
{
protected ?int $minimumShouldMatch = null;
public static function create(): static
{
return new self();
}
public function minimumShouldMatch(int $minimum): static
{
$this->minimumShouldMatch = $minimum;
return $this;
}
public function toArray(): array
{
$array = parent::toArray();
if ($this->minimumShouldMatch !== null) {
$array['bool']['minimum_should_match'] = $this->minimumShouldMatch;
}
return $array;
}
}

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
{
@ -62,6 +62,10 @@ class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
$query['match'][$this->field]['boost'] = $this->boost;
$query['match'][$this->field]['zero_terms_query'] = $this->zeroTerms ? 'all' : 'none';
if ($this->analyzer !== null) {
$query['match'][$this->field]['analyzer'] = $this->analyzer;
}
return $query;
}
}

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Elasticsearch;
namespace Sbp\Search\Elasticsearch;
class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQuery
{

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Exceptions;
namespace Sbp\Search\Exceptions;
use Throwable;

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Jobs;
namespace Sbp\Search\Jobs;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Model;

View File

@ -10,11 +10,11 @@
*
*/
namespace Blomstra\Search\Jobs;
namespace Sbp\Search\Jobs;
use Blomstra\Search\Seeders\Seeder;
use Sbp\Search\Seeders\Seeder;
use Flarum\Queue\AbstractJob;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Collection;
abstract class Job extends AbstractJob
{

View File

@ -10,10 +10,11 @@
*
*/
namespace Blomstra\Search\Jobs;
namespace Sbp\Search\Jobs;
use Blomstra\Search\Exceptions\SeedingException;
use Sbp\Search\Exceptions\SeedingException;
use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Collection as EloquentCollection;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr;
@ -50,10 +51,11 @@ class UpdateSearchJob extends Job
$items = Arr::get($response, 'items');
$error = Arr::get(Arr::first($items), 'index.error.reason');
$failed = array_filter($items, fn ($item) => isset($item['index']['error']));
$error = Arr::get(Arr::first($failed), 'index.error.reason', 'unknown error');
throw new SeedingException(
"Failed to seed: $error",
"Failed to seed: $error (" . count($failed) . '/' . count($items) . ' items failed)',
$items
);
}

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Jobs;
namespace Sbp\Search\Jobs;
use Elasticsearch\Client;
use Flarum\Api\Serializer\DiscussionSerializer;

View File

@ -10,12 +10,12 @@
*
*/
namespace Blomstra\Search;
namespace Sbp\Search;
use Blomstra\Search\Jobs\DeletingJob;
use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\UpdateSearchJob;
use Blomstra\Search\Jobs\ViewsSearchJob;
use Sbp\Search\Jobs\DeletingJob;
use Sbp\Search\Jobs\Job;
use Sbp\Search\Jobs\UpdateSearchJob;
use Sbp\Search\Jobs\ViewsSearchJob;
use Elasticsearch\Client as Elastic;
use Elasticsearch\ClientBuilder;
use Flarum\Api\Client;
@ -26,7 +26,7 @@ use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Collection;
use Laminas\Stratigility\MiddlewarePipe;
use Psr\Log\LoggerInterface;
@ -47,7 +47,13 @@ class Provider extends AbstractServiceProvider
$this->container->singleton(Elastic::class, function (Container $container) use ($settings, $config) {
$builder = ClientBuilder::create()
->setHosts([$settings->get('blomstra-search.elastic-endpoint')]);
->setHosts([$settings->get('blomstra-search.elastic-endpoint')])
->setConnectionParams([
'client' => [
'connect_timeout' => 2, // fail fast if ES is unreachable
'timeout' => 10, // allow time for complex queries
],
]);
if ($config->inDebugMode()) {
$builder->setLogger($container->make(LoggerInterface::class));

View File

@ -10,7 +10,7 @@
*
*/
namespace Blomstra\Search\Save;
namespace Sbp\Search\Save;
use Carbon\Carbon;
use Illuminate\Support\Arr;

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Searchers;
namespace Sbp\Search\Searchers;
use Blomstra\Search\Seeders\CommentSeeder;
use Sbp\Search\Seeders\CommentSeeder;
class CommentPostSearcher extends Searcher
{

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Searchers;
namespace Sbp\Search\Searchers;
use Blomstra\Search\Seeders\DiscussionSeeder;
use Sbp\Search\Seeders\DiscussionSeeder;
class DiscussionSearcher extends Searcher
{

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Searchers;
namespace Sbp\Search\Searchers;
use Blomstra\Search\Seeders\Seeder;
use Sbp\Search\Seeders\Seeder;
use Flarum\Settings\SettingsRepositoryInterface;
abstract class Searcher

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Seeders;
namespace Sbp\Search\Seeders;
use Blomstra\Search\Save\Document;
use Sbp\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Api\Serializer\PostSerializer;
use Flarum\Discussion\Discussion;

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Seeders;
namespace Sbp\Search\Seeders;
use Blomstra\Search\Save\Document;
use Sbp\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Discussion\Event as Core;
@ -24,7 +24,7 @@ use FoF\Byobu\Events as Byobu;
use FoF\DiscussionViews\Events\DiscussionWasViewed;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Collection;
use Illuminate\Database\Eloquent\Model;
class DiscussionSeeder extends Seeder
@ -70,7 +70,7 @@ class DiscussionSeeder extends Seeder
{
$events->listen([
// flarum/core events
Core\Started::class, Core\Restored::class,
Core\Started::class, Core\Restored::class, Core\Renamed::class,
// fof/byobu discussion recipients events.
Byobu\DiscussionMadePublic::class, Byobu\RemovedSelf::class, Byobu\RecipientsChanged::class,
], function ($event) use ($callable) {
@ -121,6 +121,7 @@ class DiscussionSeeder extends Seeder
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->title,
'title' => $model->title,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->last_posted_at?->toAtomString(),
'is_private' => $model->is_private,

View File

@ -10,9 +10,9 @@
*
*/
namespace Blomstra\Search\Seeders;
namespace Sbp\Search\Seeders;
use Blomstra\Search\Save\Document;
use Sbp\Search\Save\Document;
use Flarum\Extension\ExtensionManager;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;