Compare commits
76 Commits
| Author | SHA1 | Date |
|---|---|---|
|
|
e4590f3ea9 | |
|
|
83c0b0e7ec | |
|
|
3a8559ecca | |
|
|
082892f413 | |
|
|
0b9586aee2 | |
|
|
c54d916361 | |
|
|
09c01b3445 | |
|
|
a089698247 | |
|
|
7344fc33b7 | |
|
|
2696fa4dfd | |
|
|
a6d6230553 | |
|
|
f537506166 | |
|
|
a9bb20dfe7 | |
|
|
6417e8e9f4 | |
|
|
22e2796a65 | |
|
|
52168bb5c3 | |
|
|
7f4befd73b | |
|
|
115c06e9bf | |
|
|
d2b8869eb4 | |
|
|
f72acc60ef | |
|
|
fc830ea921 | |
|
|
4538fe2ba8 | |
|
|
5774879331 | |
|
|
3e2f3f9585 | |
|
|
bf90e3279e | |
|
|
bce3258325 | |
|
|
7f282cdb84 | |
|
|
7cb4071a54 | |
|
|
e3dfc04192 | |
|
|
e7e4495306 | |
|
|
0fa8fb92eb | |
|
|
925ecb0e7c | |
|
|
e92307cf16 | |
|
|
d27d71ff72 | |
|
|
f7d9935fb9 | |
|
|
33d4946947 | |
|
|
0f58abf141 | |
|
|
0adaab5d84 | |
|
|
c55db0abd6 | |
|
|
c5643bed31 | |
|
|
6bcd2becd5 | |
|
|
2706ed6be0 | |
|
|
557604e426 | |
|
|
08353d2473 | |
|
|
80357e6a77 | |
|
|
d8eebb183a | |
|
|
c54f37349f | |
|
|
ca8c248870 | |
|
|
bf37e01eb0 | |
|
|
4709b9d1db | |
|
|
bf19eed20f | |
|
|
2fc66b9bf7 | |
|
|
439c74f0a3 | |
|
|
f486292f15 | |
|
|
4412583cd9 | |
|
|
81e7b6502d | |
|
|
3039b65946 | |
|
|
e2e2d1428f | |
|
|
80d66c98c4 | |
|
|
ff2a252e99 | |
|
|
5929a191f6 | |
|
|
6699c8cb58 | |
|
|
369ee504f3 | |
|
|
eeb091273d | |
|
|
d252a17586 | |
|
|
a9dc8bfb16 | |
|
|
82e6ec786f | |
|
|
f981c27e00 | |
|
|
f6503f5e6a | |
|
|
ad92d8856b | |
|
|
8b519619e5 | |
|
|
83afe95d61 | |
|
|
846cc348ff | |
|
|
9401b677a3 | |
|
|
4bc89949ec | |
|
|
29ba864849 |
|
|
@ -2,3 +2,4 @@ js/node_modules
|
|||
js/dist
|
||||
vendor/
|
||||
composer.lock
|
||||
analysis/*
|
||||
|
|
|
|||
166
README.md
166
README.md
|
|
@ -5,47 +5,177 @@ fulltext search with one that is completely relying on the proven elasticsearch
|
|||
|
||||
## Features
|
||||
|
||||
- Sync discussions to elastic search using your queue, unobtrusively for the user.
|
||||
- Reduces search loading times to well below 400ms (local tests with 50.000 discussion **260ms**)
|
||||
- Sync discussions and posts to Elasticsearch using your queue, unobtrusively for the user.
|
||||
- Reduces search loading times to well below 400ms (local tests with 50,000 discussions: **260ms**)
|
||||
- Uses Flarum's group permissions and tags system.
|
||||
- Compatible with Friends of Flarum Byōbu.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Elasticsearch 7.x or OpenSearch 1.x+
|
||||
- A non-sync queue driver with a running worker (`php flarum queue:work`) is strongly recommended for production. The extension works with the default sync driver, but index jobs run inline which adds latency to user-facing changes like posting.
|
||||
|
||||
## Installation
|
||||
|
||||
Use composer:
|
||||
```bash
|
||||
composer require sbp/flarum-search
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Notes
|
||||
|
||||
This fork is now maintained as `sbp/flarum-search`.
|
||||
|
||||
Intentional compatibility leftovers:
|
||||
- settings keys remain `blomstra-search.*`
|
||||
- some translation keys remain `blomstra-search.*`
|
||||
- CLI command names may still remain `blomstra:*` until a later migration
|
||||
|
||||
Search behavior:
|
||||
- title autocomplete enabled
|
||||
- fuzzy title search enabled
|
||||
- exact + fuzzy content search on full search page
|
||||
- no content autocomplete in dropdown (performance decision)
|
||||
``
|
||||
|
||||
Repository can be found here: https://gitea.muehlberger.net/sbp-jm/sbp-flarum-search
|
||||
|
||||
|
||||
|
||||
|
||||
Enable the extension in the admin area and configure the Elasticsearch endpoint and index name in the extension settings.
|
||||
|
||||
## Setting up the index
|
||||
|
||||
### First install
|
||||
|
||||
Run the build command once. It creates a timestamped concrete index, immediately aliases your configured index name to it, and begins queuing documents. Search is available and improves as the queue processes:
|
||||
|
||||
```bash
|
||||
composer require blomstra/search:*
|
||||
php flarum blomstra:search:index build
|
||||
php flarum queue:work
|
||||
```
|
||||
|
||||
Enable the extension inside the admin area and configure the settings.
|
||||
### Subsequent rebuilds (zero-downtime)
|
||||
|
||||
### Set up
|
||||
When you need to rebuild the full index (e.g. after a mapping change):
|
||||
|
||||
Enable the extension in your admin area. Now to seed your existing discussions use the following command:
|
||||
```bash
|
||||
# Simple rebuild — promotes automatically once all jobs are queued
|
||||
php flarum blomstra:search:index build
|
||||
|
||||
```
|
||||
php flarum blomstra:search:index
|
||||
# Or keep a backup of the old index in case you need to roll back
|
||||
php flarum blomstra:search:index build --keep-backup
|
||||
```
|
||||
|
||||
All mutations to discussions are automatically added and removed from the elasticsearch index.
|
||||
After the queue drains, fill any gaps from content posted during the build:
|
||||
|
||||
### FAQ
|
||||
```bash
|
||||
php flarum blomstra:search:index fill
|
||||
```
|
||||
|
||||
*I have another question.*
|
||||
Reach out to us via https://helpdesk.blomstra.net. We will get back to you as soon as we can. If you have a running subscription please mention when you started your plan and/or which plan you are on. Always add sufficient information when reporting errors. We prefer errors being reported here, but understand that sometimes you can't.
|
||||
If you kept a backup and want to roll back:
|
||||
|
||||
```bash
|
||||
php flarum blomstra:search:index rollback
|
||||
```
|
||||
|
||||
Once satisfied with the new index, drop the backup:
|
||||
|
||||
```bash
|
||||
php flarum blomstra:search:index discard --backup
|
||||
```
|
||||
|
||||
### Blue-green rebuild (manual promotion)
|
||||
|
||||
Use `--staging` to keep the old index live until you explicitly promote:
|
||||
|
||||
```bash
|
||||
# 1. Build into a staging index — live index is untouched
|
||||
php flarum blomstra:search:index build --staging
|
||||
|
||||
# 2. Drain the queue
|
||||
php flarum queue:work --stop-when-empty
|
||||
|
||||
# 3. Promote the staging index to live
|
||||
php flarum blomstra:search:index promote
|
||||
# Or keep the old index as a backup:
|
||||
php flarum blomstra:search:index promote --keep-backup
|
||||
```
|
||||
|
||||
### Resuming or cancelling an interrupted build
|
||||
|
||||
If a build is interrupted, re-run it with the appropriate flag:
|
||||
|
||||
```bash
|
||||
# Resume each seeder from where it left off
|
||||
php flarum blomstra:search:index build --resume
|
||||
|
||||
# Drop the staging index and start completely fresh
|
||||
php flarum blomstra:search:index build --fresh
|
||||
|
||||
# Cancel the build without starting a new one
|
||||
php flarum blomstra:search:index discard --pending
|
||||
```
|
||||
|
||||
### Filling gaps in an existing index
|
||||
|
||||
If documents are missing from the live index (e.g. due to queue failures):
|
||||
|
||||
```bash
|
||||
php flarum blomstra:search:index fill
|
||||
```
|
||||
|
||||
### Updating the mapping only
|
||||
|
||||
To push a mapping change to the live index without rebuilding:
|
||||
|
||||
```bash
|
||||
php flarum blomstra:search:index mapping
|
||||
```
|
||||
|
||||
## Command reference
|
||||
|
||||
| Command | Description |
|
||||
|---|---|
|
||||
| `build` | Rebuild the index and promote automatically once all jobs are queued. On first install, aliases immediately so search is live during seeding. |
|
||||
| `build --keep-backup` | Rebuild and promote, retaining the old index as a backup for rollback. |
|
||||
| `build --staging` | Build into a staging index without promoting — use `promote` when ready (blue-green workflow). |
|
||||
| `build --resume` | Resume an interrupted build from where each seeder left off. |
|
||||
| `build --fresh` | Drop the staging index and start completely fresh. |
|
||||
| `promote` | Atomically swap the alias to the staging index. Prompts for confirmation (blue-green workflow). |
|
||||
| `promote --keep-backup` | Promote and retain the replaced live index as a backup for rollback. |
|
||||
| `rollback` | Restore the backup index to live. Deletes the index that was live. |
|
||||
| `discard --pending` | Drop the staging index without promoting (cancels an in-progress build). |
|
||||
| `discard --backup` | Drop the backup index (cleanup after `--keep-backup`). |
|
||||
| `mapping` | Push updated mapping to the live index without rebuilding or reseeding. |
|
||||
| `fill` | Seed only documents missing from the live index. |
|
||||
| `build --only=discussions` | Seed only the specified document type (`discussions` or `posts`). |
|
||||
| `build --throttle=N` | Wait N seconds between batches (reduces queue pressure). |
|
||||
| `build --max-id=N` | Limit seeding to documents with ID ≤ N. |
|
||||
| `promote --i-am-sure` | Skip the promotion confirmation prompt (for scripts and CI). |
|
||||
|
||||
## FAQ
|
||||
|
||||
## Queue configuration
|
||||
|
||||
*"Can I dispatch indexing jobs to a specific queue?"*
|
||||
|
||||
*Can I dispatch the sync jobs to another queue?*
|
||||
Yes:
|
||||
|
||||
```php
|
||||
\Blomstra\Search\Observe\Job::$onQueue = 'sync';
|
||||
\Blomstra\Search\Jobs\Job::$onQueue = 'search';
|
||||
```
|
||||
|
||||
*"I have a different question"*
|
||||
|
||||
Reach out ot us via https://support.on-floxum.com/t/ext-search . If you have an active subscription, please mention what plan you are on.
|
||||
|
||||
---
|
||||
|
||||
- Blomstra provides managed Flarum hosting.
|
||||
- https://blomstra.net
|
||||
- https://blomstra.community/t/ext-search
|
||||
- Floxum provides managed Flarum hosting.
|
||||
- https://floxum.com
|
||||
- https://support.on-floxum.com/t/ext-search
|
||||
|
||||
Icon made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/).
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"name": "blomstra/search",
|
||||
"name": "sbp/flarum-search",
|
||||
"description": "Replaces Flarum search with one powered by an elastic search server.",
|
||||
"keywords": [
|
||||
"extension",
|
||||
|
|
@ -11,16 +11,20 @@
|
|||
"elasticsearch"
|
||||
],
|
||||
"support": {
|
||||
"email": "helpdesk@blomstra.net",
|
||||
"email": "mueh+git@sbp.de",
|
||||
"forum": "https://blomstra.community/t/ext-search"
|
||||
},
|
||||
"type": "flarum-extension",
|
||||
"license": "MIT",
|
||||
"authors": [
|
||||
{
|
||||
"name": "sbp",
|
||||
"email": "info@sbp.de"
|
||||
},
|
||||
{
|
||||
"name": "Team Blomstra",
|
||||
"email": "development@blomstra.net",
|
||||
"role": "Developer"
|
||||
"role": "Original Developer"
|
||||
}
|
||||
],
|
||||
"require": {
|
||||
|
|
@ -43,7 +47,7 @@
|
|||
},
|
||||
"autoload": {
|
||||
"psr-4": {
|
||||
"Blomstra\\Search\\": "src/"
|
||||
"Sbp\\Search\\": "src/"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
20
extend.php
20
extend.php
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search;
|
||||
namespace Sbp\Search;
|
||||
|
||||
use Flarum\Extend as Flarum;
|
||||
|
||||
|
|
@ -20,14 +20,22 @@ return [
|
|||
(new Flarum\Frontend('forum'))
|
||||
->js(__DIR__.'/js/dist/forum.js'),
|
||||
(new Flarum\Frontend('admin'))
|
||||
->js(__DIR__.'/js/dist/admin.js'),
|
||||
->js(__DIR__.'/js/dist/admin.js')
|
||||
->css(__DIR__.'/resources/less/admin.less'),
|
||||
|
||||
(new Flarum\Locales(__DIR__.'/resources/locale')),
|
||||
new Flarum\Locales(__DIR__.'/resources/locale'),
|
||||
|
||||
(new Flarum\Routes('api'))
|
||||
->get('/blomstra/search/{type}', 'blomstra.search', Api\Controllers\SearchController::class)
|
||||
->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class),
|
||||
|
||||
->get('/sbp/search/{type}', 'sbp.search', Api\Controllers\SearchController::class)
|
||||
->put('/sbp/search/index', 'sbp.search.index', Api\Controllers\IndexController::class),
|
||||
(new Flarum\Console())
|
||||
->command(Commands\BuildCommand::class),
|
||||
|
||||
(new Flarum\Settings())
|
||||
->default('blomstra-search.search-discussion-subjects', true)
|
||||
->default('blomstra-search.search-post-bodies', true)
|
||||
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH)
|
||||
->default('blomstra-search.stem-exclusion', '')
|
||||
->serializeToForum('blomstraSearchMinLength', 'blomstra-search.min-search-length', 'intval')
|
||||
->serializeToForum('blomstraSearchPostBodies', 'blomstra-search.search-post-bodies', fn ($v) => (bool) $v, true),
|
||||
];
|
||||
|
|
|
|||
|
|
@ -1,2 +1,2 @@
|
|||
(()=>{var a={n:e=>{var t=e&&e.__esModule?()=>e.default:()=>e;return a.d(t,{a:t}),t},d:(e,t)=>{for(var r in t)a.o(t,r)&&!a.o(e,r)&&Object.defineProperty(e,r,{enumerable:!0,get:t[r]})},o:(a,e)=>Object.prototype.hasOwnProperty.call(a,e),r:a=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(a,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(a,"__esModule",{value:!0})}},e={};(()=>{"use strict";a.r(e);const t=flarum.core.compat["admin/app"];var r=a.n(t);r().initializers.add("blomstra-search",(function(){var a=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach((function(e){a.set(e,e)})),r().extensionData.for("blomstra-search").registerSetting({setting:"blomstra-search.elastic-endpoint",label:r().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:r().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:r().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:r().translator.trans("blomstra-search.admin.analyzer.label"),help:r().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(a.entries()),default:"english"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"})}))})(),module.exports=e})();
|
||||
(()=>{var a={n:t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return a.d(e,{a:e}),e},d:(t,e)=>{for(var n in e)a.o(e,n)&&!a.o(t,n)&&Object.defineProperty(t,n,{enumerable:!0,get:e[n]})},o:(a,t)=>Object.prototype.hasOwnProperty.call(a,t)};(()=>{"use strict";function t(a,e){return t=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(a,t){return a.__proto__=t,a},t(a,e)}const e=flarum.core.compat["admin/app"];var n=a.n(e);const r=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=a.n(s);const l=flarum.core.compat["admin/components/DashboardWidget"];var o=a.n(l);const c=flarum.core.compat["common/components/Alert"];var h=a.n(c),d=function(a){var e,r;function s(){return a.apply(this,arguments)||this}r=a,(e=s).prototype=Object.create(r.prototype),e.prototype.constructor=e,t(e,r);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:n().translator.trans("blomstra-search.admin.reindex-required.title")},n().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(o());n().initializers.add("sbp-search",function(){var a=n().data.settings["blomstra-search.active-index"],t=n().data.settings["blomstra-search.index-compatible"];a&&"v3"!==t&&(0,r.extend)(i().prototype,"availableWidgets",function(a){a.add("blomstra-search-reindex",m(d),110)}),n().extensionData.for("sbp-search").registerSetting(function(){var t=n().data.settings["blomstra-search.indexed-analyzer"];if(!a||!t)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",r=this.setting("blomstra-search.stem-exclusion")()||"",s=n().data.settings["blomstra-search.indexed-stem-exclusion"]||"";return e===t&&r===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},n().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:n().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:n().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:n().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n().translator.trans("blomstra-search.admin.analyzer.label"),help:n().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:{arabic:"Arabic",armenian:"Armenian",basque:"Basque",bengali:"Bengali",brazilian:"Brazilian Portuguese",bulgarian:"Bulgarian",catalan:"Catalan",cjk:"CJK (Chinese, Japanese, Korean)",czech:"Czech",danish:"Danish",dutch:"Dutch",english:"English",estonian:"Estonian",finnish:"Finnish",french:"French",galician:"Galician",german:"German",greek:"Greek",hindi:"Hindi",hungarian:"Hungarian",indonesian:"Indonesian",irish:"Irish",italian:"Italian",latvian:"Latvian",lithuanian:"Lithuanian",norwegian:"Norwegian",persian:"Persian",portuguese:"Portuguese",romanian:"Romanian",russian:"Russian",sorani:"Sorani (Kurdish)",spanish:"Spanish",standard:"Standard (generic)",swedish:"Swedish",turkish:"Turkish",thai:"Thai"},default:"english"}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.stem-exclusion",type:"textarea",label:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.label"),help:n().translator.trans("blomstra-search.admin.settings.stem-exclusion.help"),disabled:a})}).registerSetting(function(){var a="cjk"===(this.setting("blomstra-search.analyzer-language")()||"english");return this.buildSettingComponent({setting:"blomstra-search.min-search-length",label:n().translator.trans("blomstra-search.admin.min-search-length.label"),help:n().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:n().data.settings["blomstra-search.min-search-length"],disabled:a})})})})(),module.exports={}})();
|
||||
//# sourceMappingURL=admin.js.map
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1 @@
|
|||
/*! regenerator-runtime -- Copyright (c) 2014-present, Facebook, Inc. -- license (MIT): https://github.com/babel/babel/blob/main/packages/babel-helpers/LICENSE */
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
|
|
@ -1,49 +1,98 @@
|
|||
import app from 'flarum/admin/app';
|
||||
import { extend } from 'flarum/common/extend';
|
||||
import DashboardPage from 'flarum/admin/components/DashboardPage';
|
||||
import DashboardWidget from 'flarum/admin/components/DashboardWidget';
|
||||
import Alert from 'flarum/common/components/Alert';
|
||||
|
||||
app.initializers.add('blomstra-search', () => {
|
||||
const languages = new Map();
|
||||
[
|
||||
'arabic',
|
||||
'armenian',
|
||||
'basque',
|
||||
'bengali',
|
||||
'brazilian',
|
||||
'bulgarian',
|
||||
'catalan',
|
||||
'cjk',
|
||||
'czech',
|
||||
'danish',
|
||||
'dutch',
|
||||
'english',
|
||||
'estonian',
|
||||
'finnish',
|
||||
'french',
|
||||
'galician',
|
||||
'german',
|
||||
'greek',
|
||||
'hindi',
|
||||
'hungarian',
|
||||
'indonesian',
|
||||
'irish',
|
||||
'italian',
|
||||
'latvian',
|
||||
'lithuanian',
|
||||
'norwegian',
|
||||
'persian',
|
||||
'portuguese',
|
||||
'romanian',
|
||||
'russian',
|
||||
'sorani',
|
||||
'spanish',
|
||||
'swedish',
|
||||
'turkish',
|
||||
'thai',
|
||||
].forEach((language) => {
|
||||
languages.set(language, language);
|
||||
const REQUIRED_INDEX_COMPAT = 'v3';
|
||||
|
||||
class ReindexWarningWidget extends DashboardWidget {
|
||||
className() {
|
||||
return 'ReindexWarningWidget';
|
||||
}
|
||||
|
||||
content() {
|
||||
return m(
|
||||
Alert,
|
||||
{
|
||||
type: 'warning',
|
||||
dismissible: false,
|
||||
icon: 'fas fa-exclamation-triangle',
|
||||
title: app.translator.trans('blomstra-search.admin.reindex-required.title'),
|
||||
},
|
||||
app.translator.trans('blomstra-search.admin.reindex-required.detail')
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
app.initializers.add('sbp-search', () => {
|
||||
const activeIndex = app.data.settings['blomstra-search.active-index'];
|
||||
const compatVersion = app.data.settings['blomstra-search.index-compatible'];
|
||||
|
||||
if (activeIndex && compatVersion !== REQUIRED_INDEX_COMPAT) {
|
||||
extend(DashboardPage.prototype, 'availableWidgets', function (items) {
|
||||
items.add('blomstra-search-reindex', m(ReindexWarningWidget), 110);
|
||||
});
|
||||
}
|
||||
|
||||
const languages: Record<string, string> = {
|
||||
arabic: 'Arabic',
|
||||
armenian: 'Armenian',
|
||||
basque: 'Basque',
|
||||
bengali: 'Bengali',
|
||||
brazilian: 'Brazilian Portuguese',
|
||||
bulgarian: 'Bulgarian',
|
||||
catalan: 'Catalan',
|
||||
cjk: 'CJK (Chinese, Japanese, Korean)',
|
||||
czech: 'Czech',
|
||||
danish: 'Danish',
|
||||
dutch: 'Dutch',
|
||||
english: 'English',
|
||||
estonian: 'Estonian',
|
||||
finnish: 'Finnish',
|
||||
french: 'French',
|
||||
galician: 'Galician',
|
||||
german: 'German',
|
||||
greek: 'Greek',
|
||||
hindi: 'Hindi',
|
||||
hungarian: 'Hungarian',
|
||||
indonesian: 'Indonesian',
|
||||
irish: 'Irish',
|
||||
italian: 'Italian',
|
||||
latvian: 'Latvian',
|
||||
lithuanian: 'Lithuanian',
|
||||
norwegian: 'Norwegian',
|
||||
persian: 'Persian',
|
||||
portuguese: 'Portuguese',
|
||||
romanian: 'Romanian',
|
||||
russian: 'Russian',
|
||||
sorani: 'Sorani (Kurdish)',
|
||||
spanish: 'Spanish',
|
||||
standard: 'Standard (generic)',
|
||||
swedish: 'Swedish',
|
||||
turkish: 'Turkish',
|
||||
thai: 'Thai',
|
||||
};
|
||||
|
||||
app.extensionData
|
||||
.for('blomstra-search')
|
||||
.for('sbp-search')
|
||||
.registerSetting(function (this: any) {
|
||||
// No index yet, or first build predates this tracking — stay silent.
|
||||
const indexedAnalyzer = app.data.settings['blomstra-search.indexed-analyzer'];
|
||||
if (!activeIndex || !indexedAnalyzer) return null;
|
||||
|
||||
const currentAnalyzer = this.setting('blomstra-search.analyzer-language')() || 'english';
|
||||
const currentStemExclusion = this.setting('blomstra-search.stem-exclusion')() || '';
|
||||
const indexedStemExclusion = app.data.settings['blomstra-search.indexed-stem-exclusion'] || '';
|
||||
|
||||
if (currentAnalyzer === indexedAnalyzer && currentStemExclusion === indexedStemExclusion) return null;
|
||||
|
||||
return m(
|
||||
Alert,
|
||||
{ type: 'warning', dismissible: false, icon: 'fas fa-exclamation-triangle' },
|
||||
app.translator.trans('blomstra-search.admin.index-settings-changed')
|
||||
);
|
||||
})
|
||||
.registerSetting({
|
||||
setting: 'blomstra-search.elastic-endpoint',
|
||||
label: app.translator.trans('blomstra-search.admin.elastic-endpoint'),
|
||||
|
|
@ -65,18 +114,44 @@ app.initializers.add('blomstra-search', () => {
|
|||
default: 'flarum',
|
||||
type: 'input',
|
||||
})
|
||||
.registerSetting({
|
||||
setting: 'blomstra-search.search-discussion-subjects',
|
||||
label: app.translator.trans('blomstra-search.admin.search-discussion-subjects'),
|
||||
type: 'switch',
|
||||
})
|
||||
.registerSetting({
|
||||
setting: 'blomstra-search.search-post-bodies',
|
||||
label: app.translator.trans('blomstra-search.admin.search-post-bodies'),
|
||||
type: 'switch',
|
||||
})
|
||||
.registerSetting({
|
||||
setting: 'blomstra-search.analyzer-language',
|
||||
label: app.translator.trans('blomstra-search.admin.analyzer.label'),
|
||||
help: app.translator.trans('blomstra-search.admin.analyzer.help'),
|
||||
type: 'select',
|
||||
options: Object.fromEntries(languages.entries()),
|
||||
options: languages,
|
||||
default: 'english',
|
||||
})
|
||||
.registerSetting({
|
||||
setting: 'blomstra-search.elastic-index',
|
||||
label: app.translator.trans('blomstra-search.admin.elastic-index'),
|
||||
default: 'flarum',
|
||||
type: 'input',
|
||||
.registerSetting(function (this: any) {
|
||||
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
|
||||
return this.buildSettingComponent({
|
||||
setting: 'blomstra-search.stem-exclusion',
|
||||
type: 'textarea',
|
||||
label: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.label'),
|
||||
help: app.translator.trans('blomstra-search.admin.settings.stem-exclusion.help'),
|
||||
disabled: isCjk,
|
||||
});
|
||||
})
|
||||
.registerSetting(function (this: any) {
|
||||
const isCjk = (this.setting('blomstra-search.analyzer-language')() || 'english') === 'cjk';
|
||||
return this.buildSettingComponent({
|
||||
setting: 'blomstra-search.min-search-length',
|
||||
label: app.translator.trans('blomstra-search.admin.min-search-length.label'),
|
||||
help: app.translator.trans('blomstra-search.admin.min-search-length.help'),
|
||||
type: 'select',
|
||||
options: { '1': '1', '2': '2', '3': '3', '4': '4' },
|
||||
default: app.data.settings['blomstra-search.min-search-length'],
|
||||
disabled: isCjk,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -6,10 +6,9 @@ import DiscussionListState from 'flarum/forum/states/DiscussionListState';
|
|||
|
||||
export default function extendDiscussionState() {
|
||||
override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) {
|
||||
const preloaded = app.data.apiDocument || null;
|
||||
|
||||
// If existing payload is given or no search is made, fallback on native page.
|
||||
if (preloaded || !this.requestParams()?.filter?.q) return original.call(this, page);
|
||||
if (!this.requestParams()?.filter?.q) return original.call(this, page);
|
||||
|
||||
const params = this.requestParams();
|
||||
params.page = {
|
||||
|
|
@ -17,12 +16,13 @@ export default function extendDiscussionState() {
|
|||
...params.page,
|
||||
};
|
||||
|
||||
|
||||
if (Array.isArray(params.include)) {
|
||||
params.include = params.include.join(',');
|
||||
}
|
||||
|
||||
// Construct API search URI
|
||||
const url = `${app.forum.attribute('apiUrl')}/blomstra/search/${this.type}`;
|
||||
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
|
||||
|
||||
// Make API GET request
|
||||
const results = await app.request({ params, url, method: 'GET' });
|
||||
|
|
|
|||
|
|
@ -21,18 +21,21 @@ export default class DiscussionsSearchSource implements SearchSource {
|
|||
private type = 'discussions';
|
||||
|
||||
async search(query: string): Promise<void> {
|
||||
// Suppress ES instant results when "search within comments" is ON;
|
||||
// only usernames are shown in that mode.
|
||||
|
||||
query = query.toLowerCase();
|
||||
|
||||
this.results.set(query, []);
|
||||
|
||||
const params = {
|
||||
filter: { q: query },
|
||||
filter: { q: query, autocomplete: 1 },
|
||||
page: { limit: 3 },
|
||||
include: 'mostRelevantPost',
|
||||
};
|
||||
|
||||
// Construct API search URI
|
||||
const url = `${app.forum.attribute('apiUrl')}/blomstra/search/${this.type}`;
|
||||
const url = `${app.forum.attribute('apiUrl')}/sbp/search/${this.type}`;
|
||||
|
||||
// Make API GET request
|
||||
const results = await app.request({ params, url, method: 'GET' });
|
||||
|
|
|
|||
|
|
@ -0,0 +1,26 @@
|
|||
import { extend } from 'flarum/common/extend';
|
||||
import DiscussionListItem from 'flarum/forum/components/DiscussionListItem';
|
||||
import TerminalPost from 'flarum/forum/components/TerminalPost';
|
||||
|
||||
export default function extendDiscussionListItem() {
|
||||
extend(DiscussionListItem.prototype, 'infoItems', function (items) {
|
||||
const params = this.attrs.params;
|
||||
|
||||
if (!params.q) return;
|
||||
|
||||
const hasFieldSort = params.sort && params.sort !== 'relevance';
|
||||
|
||||
if (hasFieldSort) {
|
||||
// Field sort active (latest, oldest, top, …): replace excerpt with TerminalPost.
|
||||
// An excerpt is meaningless when results are ordered by date/count rather than relevance.
|
||||
items.remove('excerpt');
|
||||
if (!items.has('terminalPost')) {
|
||||
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={!this.showFirstPost()} />);
|
||||
}
|
||||
} else if (!items.has('excerpt')) {
|
||||
// Relevance mode but no excerpt (mostRelevantPost was null or non-comment type).
|
||||
// Fall back to TerminalPost so the info section is never silently empty.
|
||||
items.add('terminalPost', <TerminalPost discussion={this.attrs.discussion} lastPost={true} />);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
|
@ -7,10 +7,18 @@ import ItemList from 'flarum/common/utils/ItemList';
|
|||
|
||||
import DiscussionsSearchSource from './SearchSources/DiscussionsSearchSource';
|
||||
import extendDiscussionState from './PaginatedListStates/extendDiscussionState';
|
||||
import extendDiscussionListItem from './extendDiscussionListItem';
|
||||
|
||||
app.initializers.add('blomstra-search', () => {
|
||||
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) {
|
||||
items.setContent('discussions', new DiscussionsSearchSource());
|
||||
// app.forum is not available during initializers (it is set after they run),
|
||||
// so read the setting lazily here, at first render time.
|
||||
const minLength = (app.forum.attribute('blomstraSearchMinLength') as number) || Search.MIN_SEARCH_LEN;
|
||||
if (minLength !== Search.MIN_SEARCH_LEN) {
|
||||
(Search as any).MIN_SEARCH_LEN = minLength;
|
||||
}
|
||||
|
||||
items.replace('discussions', new DiscussionsSearchSource());
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -18,6 +26,7 @@ app.initializers.add(
|
|||
'blomstra-search-early',
|
||||
() => {
|
||||
extendDiscussionState();
|
||||
extendDiscussionListItem();
|
||||
},
|
||||
999999
|
||||
);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
diff --git a/src/Commands/BuildCommand.php b/src/Commands/BuildCommand.php
|
||||
index 9024e1e..9024e1e 100644
|
||||
--- a/src/Commands/BuildCommand.php
|
||||
+++ b/src/Commands/BuildCommand.php
|
||||
@@ -38,6 +38,6 @@ class BuildCommand extends Command
|
||||
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
|
||||
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
|
||||
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
|
||||
- private const TITLE_MIN_GRAM = 2;
|
||||
- private const TITLE_MAX_GRAM = 15;
|
||||
+ private const TITLE_MIN_GRAM = 3;
|
||||
+ private const TITLE_MAX_GRAM = 20;
|
||||
|
||||
protected $signature = 'blomstra:search:index
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
.ReindexWarningWidget {
|
||||
padding: 0;
|
||||
}
|
||||
|
|
@ -8,4 +8,21 @@ blomstra-search:
|
|||
label: Analyzer language
|
||||
help: |
|
||||
The analyzer makes search understand stop words and undertakes language
|
||||
specific improvements for indexing.
|
||||
specific improvements for indexing. Changing this requires a full index rebuild to take effect.
|
||||
index-settings-changed: "Search index settings have changed. Run: php flarum blomstra:search:index build"
|
||||
reindex-required:
|
||||
title: Search index rebuild required
|
||||
detail: "The search index is not compatible with this version of the extension. Run: php flarum blomstra:search:index build"
|
||||
search-discussion-subjects: Search inside discussion titles
|
||||
search-post-bodies: Search inside comments
|
||||
match-sentences: Match search term against full sentence
|
||||
match-words: Match search term against full words
|
||||
settings:
|
||||
stem-exclusion:
|
||||
label: "Protected words (stem exclusion)"
|
||||
help: "One word per line. These words will not be stemmed during search (e.g. brand names, proper nouns). Not applicable for CJK or Thai languages. Requires a full re-index to take effect."
|
||||
min-search-length:
|
||||
label: Minimum search query length
|
||||
help: |
|
||||
Minimum number of characters required before a search is triggered. Lower this to 1 or 2
|
||||
for CJK (Chinese, Japanese, Korean) communities where single characters carry full meaning.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Api;
|
||||
namespace Sbp\Search\Api;
|
||||
|
||||
use Illuminate\Support\Arr;
|
||||
use Psr\Http\Message\ResponseInterface;
|
||||
|
|
|
|||
|
|
@ -10,9 +10,9 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Api\Controllers;
|
||||
namespace Sbp\Search\Api\Controllers;
|
||||
|
||||
use Blomstra\Search\Commands\BuildCommand;
|
||||
use Sbp\Search\Commands\BuildCommand;
|
||||
use Laminas\Diactoros\Response\EmptyResponse;
|
||||
use Psr\Http\Message\ResponseInterface;
|
||||
use Psr\Http\Message\ServerRequestInterface;
|
||||
|
|
|
|||
|
|
@ -10,12 +10,15 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Api\Controllers;
|
||||
namespace Sbp\Search\Api\Controllers;
|
||||
|
||||
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
|
||||
use Blomstra\Search\Elasticsearch\MatchQuery;
|
||||
use Blomstra\Search\Elasticsearch\TermsQuery;
|
||||
use Blomstra\Search\Save\Document as ElasticDocument;
|
||||
use Sbp\Search\Elasticsearch\HasChildQuery;
|
||||
use Sbp\Search\Elasticsearch\MatchPhraseQuery;
|
||||
use Sbp\Search\Elasticsearch\MatchQuery;
|
||||
use Sbp\Search\Elasticsearch\TermsQuery;
|
||||
use Sbp\Search\Searchers\CommentPostSearcher;
|
||||
use Sbp\Search\Searchers\DiscussionSearcher;
|
||||
use Sbp\Search\Searchers\Searcher;
|
||||
use Elasticsearch\Client;
|
||||
use Flarum\Api\Controller\ListDiscussionsController;
|
||||
use Flarum\Api\Serializer\DiscussionSerializer;
|
||||
|
|
@ -24,15 +27,17 @@ use Flarum\Extension\ExtensionManager;
|
|||
use Flarum\Group\Group;
|
||||
use Flarum\Http\RequestUtil;
|
||||
use Flarum\Http\UrlGenerator;
|
||||
use Flarum\Settings\SettingsRepositoryInterface;
|
||||
use Flarum\Tags\Tag;
|
||||
use Flarum\User\User;
|
||||
use Illuminate\Contracts\Container\Container;
|
||||
use Illuminate\Support\Arr;
|
||||
use Illuminate\Support\Collection;
|
||||
use Illuminate\Support\Str;
|
||||
use Psr\Http\Message\ServerRequestInterface;
|
||||
use Psr\Log\LoggerInterface;
|
||||
use Spatie\ElasticsearchQueryBuilder\Builder;
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
|
||||
use Sbp\Search\Elasticsearch\BoolQuery;
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
|
||||
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
|
||||
use Tobscure\JsonApi\Document;
|
||||
|
|
@ -45,92 +50,133 @@ class SearchController extends ListDiscussionsController
|
|||
'lastPostedAt' => 'updated_at',
|
||||
'createdAt' => 'created_at',
|
||||
'commentCount' => 'comment_count',
|
||||
'view_count' => 'view_count',
|
||||
];
|
||||
|
||||
public function __construct(protected Client $elastic, protected UrlGenerator $uri)
|
||||
protected bool $matchSentences;
|
||||
protected bool $matchWords;
|
||||
protected ?Searcher $discussionSearcher;
|
||||
protected ?Searcher $postSearcher;
|
||||
|
||||
public function __construct(protected Client $elastic, protected UrlGenerator $uri, Container $container, SettingsRepositoryInterface $settings)
|
||||
{
|
||||
$this->matchSentences = true;
|
||||
$this->matchWords = true;
|
||||
|
||||
$searchers = collect($container->tagged('blomstra.search.searchers'));
|
||||
|
||||
$this->discussionSearcher = $searchers->first(fn ($s) => $s instanceof DiscussionSearcher);
|
||||
$this->postSearcher = $searchers->first(fn ($s) => $s instanceof CommentPostSearcher);
|
||||
}
|
||||
|
||||
protected function data(ServerRequestInterface $request, Document $document)
|
||||
{
|
||||
// Not used for now.
|
||||
$type = Arr::get($request->getQueryParams(), 'type');
|
||||
|
||||
$actor = RequestUtil::getActor($request);
|
||||
|
||||
$filters = $this->extractFilter($request);
|
||||
|
||||
$search = $this->getSearch($filters);
|
||||
|
||||
$limit = $this->extractLimit($request);
|
||||
$offset = $this->extractOffset($request);
|
||||
|
||||
$include = array_merge($this->extractInclude($request), ['state']);
|
||||
|
||||
$filterQuery = BoolQuery::create();
|
||||
// Autocomplete path: fast prefix match on title.autocomplete only.
|
||||
if (!empty($filters['autocomplete']) && !empty($search)) {
|
||||
return $this->handleAutocomplete($search, $actor, $limit, $offset, $include, $document, $request);
|
||||
}
|
||||
|
||||
$knownSortFields = array_merge(array_values($this->translateSort), ['rawId']);
|
||||
$logger = resolve(LoggerInterface::class);
|
||||
|
||||
$phpSortField = null;
|
||||
$phpSortDir = 'desc';
|
||||
$needsScoring = true;
|
||||
$sorts = [];
|
||||
|
||||
foreach ($this->extractSort($request) as $field => $direction) {
|
||||
$translated = $this->translateSort[$field] ?? $field;
|
||||
|
||||
if (!in_array($translated, $knownSortFields)) {
|
||||
$logger->warning("blomstra/search: unknown sort field \"{$field}\", ignoring.");
|
||||
continue;
|
||||
}
|
||||
|
||||
$sorts[] = new Sort($translated, $direction);
|
||||
$needsScoring = false;
|
||||
|
||||
if ($phpSortField === null && $translated !== 'rawId') {
|
||||
$phpSortField = $translated;
|
||||
$phpSortDir = $direction;
|
||||
}
|
||||
}
|
||||
|
||||
// Default to latest when no explicit sort is requested. This lets has_child use
|
||||
// score_mode:none, which skips child scoring entirely and lets ES short-circuit
|
||||
// early on large corpora. Relevance is still available via sort=relevant if added.
|
||||
if (empty($sorts)) {
|
||||
$needsScoring = false;
|
||||
$phpSortField = 'updated_at';
|
||||
$phpSortDir = 'desc';
|
||||
$sorts[] = new Sort('updated_at', 'desc');
|
||||
}
|
||||
|
||||
$query = BoolQuery::create()
|
||||
// Always restrict to discussion documents; posts are only searched via has_child.
|
||||
->add(TermQuery::create('join_field', 'discussion'), 'filter');
|
||||
|
||||
if (!empty($search)) {
|
||||
$filterQuery
|
||||
// @todo commented out to use only partial matching for now
|
||||
->add($this->sentenceMatch($search))
|
||||
->add($this->wordMatch($search, 'and'))
|
||||
->add($this->wordMatch($search, 'or'))
|
||||
// ->add($this->partialMatch($search))
|
||||
;
|
||||
$query->add($this->buildTextQuery($search, $actor, $needsScoring));
|
||||
}
|
||||
|
||||
$this->addFilters($query, $actor, $filters);
|
||||
|
||||
$builder = (new Builder($this->elastic))
|
||||
->index(resolve('blomstra.search.elastic_index'))
|
||||
->size($limit + 1)
|
||||
->from($this->extractOffset($request))
|
||||
->addQuery(
|
||||
$this->addFilters($filterQuery, $actor, $filters)
|
||||
);
|
||||
->addQuery($query);
|
||||
|
||||
foreach ($this->extractSort($request) as $field => $direction) {
|
||||
$field = $this->translateSort[$field] ?? $field;
|
||||
$builder->addSort(new Sort($field, $direction));
|
||||
foreach ($sorts as $sort) {
|
||||
$builder->addSort($sort);
|
||||
}
|
||||
|
||||
$response = $builder->search();
|
||||
// track_total_hits: false lets ES stop counting once it has collected
|
||||
// enough results in sort order, avoiding a full-index count on every query.
|
||||
$payload = $builder->getPayload();
|
||||
$payload['track_total_hits'] = false;
|
||||
|
||||
$response = $this->elastic->search([
|
||||
'index' => resolve('blomstra.search.elastic_index'),
|
||||
'size' => $limit + 1,
|
||||
'from' => $offset,
|
||||
'body' => $payload,
|
||||
]);
|
||||
|
||||
Discussion::setStateUser($actor);
|
||||
|
||||
// Eager load groups for use in the policies (isAdmin check)
|
||||
if (in_array('mostRelevantPost.user', $include)) {
|
||||
$include[] = 'mostRelevantPost.user.groups';
|
||||
|
||||
// If the first level of the relationship wasn't explicitly included,
|
||||
// add it so the code below can look for it
|
||||
if (!in_array('mostRelevantPost', $include)) {
|
||||
$include[] = 'mostRelevantPost';
|
||||
}
|
||||
}
|
||||
|
||||
// we need to retrieve all discussion ids and when the results are posts,
|
||||
// their ids as most relevant post id
|
||||
// All hits are discussion documents. Extract the best-matching post ID from
|
||||
// inner_hits when a has_child clause matched (i.e. the match came from a post).
|
||||
$results = Collection::make(Arr::get($response, 'hits.hits'))
|
||||
->map(function ($hit) {
|
||||
$type = $hit['_source']['type'];
|
||||
$id = Str::after($hit['_source']['id'], "$type:");
|
||||
// _id is "discussions:123" — parse the numeric part directly.
|
||||
$discussionId = Str::after($hit['_id'], 'discussions:');
|
||||
|
||||
// rawId on the inner hit gives us the integer post ID.
|
||||
$bestPostId = Arr::get($hit, 'inner_hits.best_post.hits.hits.0._source.rawId');
|
||||
|
||||
if ($type === 'posts') {
|
||||
return [
|
||||
'most_relevant_post_id' => $id,
|
||||
'weight' => Arr::get($hit, 'sort.0'),
|
||||
'discussion_id' => $discussionId,
|
||||
'most_relevant_post_id' => $bestPostId,
|
||||
'weight' => Arr::get($hit, 'sort.0', Arr::get($hit, '_score', 0)),
|
||||
];
|
||||
} else {
|
||||
return [
|
||||
'discussion_id' => $id,
|
||||
'weight' => Arr::get($hit, 'sort.0'),
|
||||
];
|
||||
}
|
||||
});
|
||||
|
||||
$document->addPaginationLinks(
|
||||
$this->uri->to('api')->route('blomstra.search', [
|
||||
'type' => 'discussions',
|
||||
]),
|
||||
$this->uri->to('api')->route('sbp.search', ['type' => 'discussions']),
|
||||
$request->getQueryParams(),
|
||||
$offset,
|
||||
$limit,
|
||||
|
|
@ -140,28 +186,25 @@ class SearchController extends ListDiscussionsController
|
|||
$results = $results->take($limit);
|
||||
|
||||
$discussions = Discussion::query()
|
||||
->select('discussions.*')
|
||||
->join('posts', 'posts.discussion_id', 'discussions.id')
|
||||
// Extra safety to prevent leaking hidden discussion (titles) towards search results.
|
||||
->when($actor->isGuest() || !$actor->hasPermission('discussion.hide'), fn ($query) => $query->whereNull('discussions.hidden_at'))
|
||||
->where(function ($query) use ($results) {
|
||||
$query
|
||||
->whereIn('discussions.id', $results->pluck('discussion_id')->filter())
|
||||
->orWhereIn('posts.id', $results->pluck('most_relevant_post_id')->filter());
|
||||
})
|
||||
->when(
|
||||
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
|
||||
fn ($q) => $q->whereNull('hidden_at')
|
||||
)
|
||||
->whereIn('id', $results->pluck('discussion_id')->filter())
|
||||
->get()
|
||||
->each(function (Discussion $discussion) use ($results) {
|
||||
if (in_array($discussion->id, $results->pluck('discussion_id')->toArray())) {
|
||||
$discussion->most_relevant_post_id = $discussion->first_post_id;
|
||||
$discussion->weight = $results->firstWhere('discussion_id', $discussion->id)['weight'] ?? 0;
|
||||
} else {
|
||||
$post = $discussion->posts()->whereIn('id', $results->pluck('most_relevant_post_id'))->first();
|
||||
$discussion->most_relevant_post_id = $post?->id ?? $discussion->first_post_id;
|
||||
$discussion->weight = $results->firstWhere('most_relevant_post_id', $post?->id)['weight'] ?? 0;
|
||||
}
|
||||
$result = $results->firstWhere('discussion_id', $discussion->id);
|
||||
|
||||
$discussion->most_relevant_post_id = $result['most_relevant_post_id']
|
||||
?? $discussion->first_post_id;
|
||||
$discussion->weight = $result['weight'] ?? 0;
|
||||
})
|
||||
->keyBy('id')
|
||||
->sortByDesc('weight')
|
||||
->when(
|
||||
$phpSortField,
|
||||
fn ($c) => $phpSortDir === 'desc' ? $c->sortByDesc($phpSortField) : $c->sortBy($phpSortField),
|
||||
fn ($c) => $c->sortByDesc('weight')
|
||||
)
|
||||
->unique();
|
||||
|
||||
$this->loadRelations($discussions, $include);
|
||||
|
|
@ -179,33 +222,208 @@ class SearchController extends ListDiscussionsController
|
|||
return $discussions;
|
||||
}
|
||||
|
||||
protected function getDocument(string $type): ?ElasticDocument
|
||||
{
|
||||
$documents = resolve(Container::class)->tagged('blomstra.search.documents');
|
||||
/**
|
||||
* Fast autocomplete path: match prefix n-grams on title.autocomplete only.
|
||||
* No has_child, no phrase scoring — a single MatchQuery suffices.
|
||||
*/
|
||||
protected function handleAutocomplete(
|
||||
string $search,
|
||||
User $actor,
|
||||
int $limit,
|
||||
int $offset,
|
||||
array $include,
|
||||
Document $document,
|
||||
ServerRequestInterface $request
|
||||
): mixed {
|
||||
$query = BoolQuery::create()
|
||||
->add(TermQuery::create('join_field', 'discussion'), 'filter');
|
||||
|
||||
return collect($documents)->first(function (ElasticDocument $document) use ($type) {
|
||||
return $document->type() === $type;
|
||||
});
|
||||
$query->add($this->buildAutocompleteQuery($search), 'must');
|
||||
|
||||
$this->addFilters($query, $actor, []);
|
||||
|
||||
$payload = (new Builder($this->elastic))
|
||||
->index(resolve('blomstra.search.elastic_index'))
|
||||
->addQuery($query)
|
||||
->addSort(new Sort('updated_at', 'desc'))
|
||||
->getPayload();
|
||||
|
||||
$response = $this->elastic->search([
|
||||
'index' => resolve('blomstra.search.elastic_index'),
|
||||
'size' => $limit + 1,
|
||||
'from' => $offset,
|
||||
'body' => $payload,
|
||||
]);
|
||||
|
||||
Discussion::setStateUser($actor);
|
||||
|
||||
$results = Collection::make(Arr::get($response, 'hits.hits'))
|
||||
->map(fn ($hit) => [
|
||||
'discussion_id' => Str::after($hit['_id'], 'discussions:'),
|
||||
'most_relevant_post_id' => null,
|
||||
'weight' => Arr::get($hit, 'sort.0', 0),
|
||||
]);
|
||||
|
||||
$document->addPaginationLinks(
|
||||
$this->uri->to('api')->route('sbp.search', ['type' => 'discussions']),
|
||||
$request->getQueryParams(),
|
||||
$offset,
|
||||
$limit,
|
||||
$results->count() > $limit ? null : 0
|
||||
);
|
||||
|
||||
$results = $results->take($limit);
|
||||
|
||||
$discussions = Discussion::query()
|
||||
->when(
|
||||
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
|
||||
fn ($q) => $q->whereNull('hidden_at')
|
||||
)
|
||||
->whereIn('id', $results->pluck('discussion_id')->filter())
|
||||
->get()
|
||||
->each(function (Discussion $discussion) use ($results) {
|
||||
$result = $results->firstWhere('discussion_id', $discussion->id);
|
||||
$discussion->most_relevant_post_id = $result['most_relevant_post_id'] ?? $discussion->first_post_id;
|
||||
$discussion->weight = $result['weight'] ?? 0;
|
||||
})
|
||||
->sortByDesc('updated_at')
|
||||
->unique();
|
||||
|
||||
$this->loadRelations($discussions, $include);
|
||||
|
||||
return $discussions;
|
||||
}
|
||||
|
||||
|
||||
protected function buildAutocompleteQuery(string $search): BoolQuery
|
||||
{
|
||||
$query = BoolQuery::create();
|
||||
|
||||
// Title prefix match (stronger)
|
||||
$query->add(
|
||||
(new MatchQuery('title.autocomplete', $search))
|
||||
->operator('and')
|
||||
->boost(1.0),
|
||||
'should'
|
||||
);
|
||||
|
||||
// Content prefix match (weaker)
|
||||
$query->add(
|
||||
(new MatchQuery('content.autocomplete', $search))
|
||||
->operator('and')
|
||||
->boost(0.4),
|
||||
'should'
|
||||
);
|
||||
|
||||
$query->minimumShouldMatch(1);
|
||||
|
||||
return $query;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the text-matching portion of the query.
|
||||
*
|
||||
* Discussion titles are matched directly (filtered to join_field=discussion).
|
||||
* Post bodies are matched via has_child. When $needsScoring is true (no explicit
|
||||
* sort, i.e. relevance ordering), score_mode=sum accumulates child scores onto the
|
||||
* parent so that discussions with many strongly-matching posts rank higher. When
|
||||
* false (any explicit field sort), score_mode=none skips scoring entirely — ES only
|
||||
* checks whether a matching child exists, which is significantly cheaper on large
|
||||
* corpora.
|
||||
* inner_hits returns the best-matching post for use as mostRelevantPost.
|
||||
* Hidden posts are only included in matching for users with post.hide permission.
|
||||
*/
|
||||
protected function buildTextQuery(string $search, User $actor, bool $needsScoring = false): BoolQuery
|
||||
{
|
||||
$textQuery = BoolQuery::create();
|
||||
// Strong direct title match for the full search page.
|
||||
$textQuery->add(
|
||||
(new MatchQuery('title', $search))
|
||||
->operator('and')
|
||||
->boost(4.0),
|
||||
'should'
|
||||
);
|
||||
|
||||
// Low-weight prefix match on title.autocomplete for partial title hits.
|
||||
$textQuery->add(
|
||||
(new MatchQuery('title.autocomplete', $search))
|
||||
->operator('and')
|
||||
->boost(0.35),
|
||||
'should'
|
||||
);
|
||||
|
||||
// Low-weight prefix match on title.autocomplete for the full search page.
|
||||
// Exact / normal title matches should still rank higher than prefix-only hits.
|
||||
$textQuery->add(
|
||||
(new MatchQuery('title.autocomplete', $search))
|
||||
->operator('and')
|
||||
->boost(0.35),
|
||||
'should'
|
||||
);
|
||||
|
||||
|
||||
if ($this->discussionSearcher?->enabled()) {
|
||||
$textQuery->add($this->buildShouldClauses($search, $this->discussionSearcher->boost()), 'should');
|
||||
}
|
||||
|
||||
if ($this->postSearcher?->enabled()) {
|
||||
$postQuery = $this->buildShouldClauses($search, $this->postSearcher->boost());
|
||||
|
||||
// Guests and non-moderators may not see hidden posts; exclude them from child matching.
|
||||
if ($actor->isGuest() || !$actor->hasPermission('post.hide')) {
|
||||
$postQuery->add(TermQuery::create('is_hidden', 'false'), 'filter');
|
||||
}
|
||||
|
||||
// Without minimum_should_match, ES default MSM is 0 when a filter clause is present,
|
||||
// causing has_child to score every non-hidden post instead of only matching ones.
|
||||
$postQuery->minimumShouldMatch(1);
|
||||
|
||||
$textQuery->add(
|
||||
HasChildQuery::create('post', $postQuery, $needsScoring ? 'sum' : 'none')->withInnerHits(),
|
||||
'should'
|
||||
);
|
||||
}
|
||||
|
||||
return $textQuery;
|
||||
}
|
||||
|
||||
protected function buildShouldClauses(string $search, float $boost): BoolQuery
|
||||
{
|
||||
$should = BoolQuery::create();
|
||||
|
||||
if ($this->matchSentences) {
|
||||
$should->add((new MatchPhraseQuery('content', $search))->boost(2 * $boost), 'should');
|
||||
}
|
||||
if ($this->matchWords) {
|
||||
$should->add((new MatchQuery('content', $search))->operator('and')->boost(1.8 * $boost), 'should');
|
||||
$should->add((new MatchQuery('content.autocomplete', $search))->operator('and')->boost(0.30 * $boost), 'should');
|
||||
}
|
||||
|
||||
return $should;
|
||||
}
|
||||
|
||||
protected function extensionEnabled(string $extension): bool
|
||||
{
|
||||
/** @var ExtensionManager $manager */
|
||||
$manager = resolve(ExtensionManager::class);
|
||||
|
||||
return $manager->isEnabled($extension);
|
||||
return resolve(ExtensionManager::class)->isEnabled($extension);
|
||||
}
|
||||
|
||||
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): BoolQuery
|
||||
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): void
|
||||
{
|
||||
$groups = $this->getGroups($actor);
|
||||
|
||||
$onlyPrivate = Str::contains($filters['q'] ?? '', 'is:private');
|
||||
|
||||
$subQuery = BoolQuery::create()
|
||||
->add(TermQuery::create('is_private', 'false'))
|
||||
->add(TermsQuery::create('groups', $groups->toArray()));
|
||||
|
||||
if ($this->extensionEnabled('flarum-tags') && !empty($filters['tag'])) {
|
||||
$slugs = is_array($filters['tag']) ? $filters['tag'] : explode(',', $filters['tag']);
|
||||
$tagIds = Tag::query()->whereIn('slug', $slugs)->pluck('id')->toArray();
|
||||
if (!empty($tagIds)) {
|
||||
$query->add(TermsQuery::create('tags', $tagIds), 'filter');
|
||||
}
|
||||
}
|
||||
|
||||
if ($this->extensionEnabled('fof-byobu') && $actor->exists) {
|
||||
$byobuQuery = BoolQuery::create()
|
||||
->add(TermQuery::create('is_private', 'true'))
|
||||
|
|
@ -224,85 +442,12 @@ class SearchController extends ListDiscussionsController
|
|||
}
|
||||
}
|
||||
|
||||
$query->add(
|
||||
$subQuery,
|
||||
'filter'
|
||||
);
|
||||
|
||||
return $query;
|
||||
}
|
||||
|
||||
protected function sentenceMatch(string $q): Query
|
||||
{
|
||||
$query = (new MatchPhraseQuery('content', $q));
|
||||
|
||||
return BoolQuery::create()
|
||||
// Discussion titles
|
||||
->add(
|
||||
BoolQuery::create()
|
||||
->add(TermQuery::create('type', 'discussions'), 'filter')
|
||||
->add($query->boost(2)),
|
||||
'should'
|
||||
)
|
||||
// Post bodies
|
||||
->add(
|
||||
BoolQuery::create()
|
||||
->add(TermQuery::create('type', 'posts'), 'filter')
|
||||
->add($query->boost(1.9)),
|
||||
'should'
|
||||
);
|
||||
}
|
||||
|
||||
protected function wordMatch(string $q, string $operator = 'or')
|
||||
{
|
||||
$query = (new MatchQuery('content', $q))
|
||||
->operator($operator);
|
||||
|
||||
$boost = $operator === 'and' ? 1 : .8;
|
||||
|
||||
return BoolQuery::create()
|
||||
// Discussion titles
|
||||
->add(
|
||||
BoolQuery::create()
|
||||
->add(TermQuery::create('type', 'discussions'), 'filter')
|
||||
->add($query->boost($boost * 1.8)),
|
||||
'should'
|
||||
)
|
||||
// Post bodies
|
||||
->add(
|
||||
BoolQuery::create()
|
||||
->add(TermQuery::create('type', 'posts'), 'filter')
|
||||
->add($query->boost($boost * 1.8)),
|
||||
'should'
|
||||
);
|
||||
}
|
||||
|
||||
protected function partialMatch(string $q)
|
||||
{
|
||||
$query = (new MatchQuery('content', $q));
|
||||
|
||||
return BoolQuery::create()
|
||||
// Discussion titles
|
||||
->add(
|
||||
BoolQuery::create()
|
||||
->add(TermQuery::create('type', 'discussions'), 'filter')
|
||||
->add(clone $query->boost(1.6)),
|
||||
'should'
|
||||
)
|
||||
// Post bodies
|
||||
->add(
|
||||
BoolQuery::create()
|
||||
->add(TermQuery::create('type', 'posts'), 'filter')
|
||||
->add(clone $query->boost(1.3)),
|
||||
'should'
|
||||
);
|
||||
$query->add($subQuery, 'filter');
|
||||
}
|
||||
|
||||
protected function getGroups(User $actor): Collection
|
||||
{
|
||||
/** @var Collection $groups */
|
||||
$groups = $actor->groups->pluck('id');
|
||||
|
||||
$groups->add(Group::GUEST_ID);
|
||||
|
||||
if ($actor->is_email_confirmed) {
|
||||
|
|
@ -317,10 +462,12 @@ class SearchController extends ListDiscussionsController
|
|||
$search = Arr::get($filters, 'q');
|
||||
|
||||
if ($search) {
|
||||
// Strip Flarum gambit operators (tag:foo, author:bar, is:private, etc.)
|
||||
// before passing to ES. These are structural filters handled separately;
|
||||
// leaving them in causes operator:and to require the gambit tokens to
|
||||
// appear literally in post content, producing zero results.
|
||||
$q = collect(explode(' ', $search))
|
||||
->filter(function (string $part) {
|
||||
return $part !== 'is:private';
|
||||
})
|
||||
->filter(fn (string $part) => !preg_match('/^\w+:/', $part))
|
||||
->filter()
|
||||
->join(' ');
|
||||
|
||||
|
|
|
|||
|
|
@ -10,12 +10,13 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Commands;
|
||||
namespace Sbp\Search\Commands;
|
||||
|
||||
use Blomstra\Search\Jobs\Job;
|
||||
use Blomstra\Search\Jobs\SavingJob;
|
||||
use Blomstra\Search\Seeders\Seeder;
|
||||
use Sbp\Search\Jobs\Job;
|
||||
use Sbp\Search\Jobs\UpdateSearchJob;
|
||||
use Sbp\Search\Seeders\Seeder;
|
||||
use Elasticsearch\Client;
|
||||
use Elasticsearch\Common\Exceptions\ElasticsearchException;
|
||||
use Flarum\Settings\SettingsRepositoryInterface;
|
||||
use Illuminate\Console\Command;
|
||||
use Illuminate\Contracts\Container\Container;
|
||||
|
|
@ -29,102 +30,312 @@ use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
|
|||
|
||||
class BuildCommand extends Command
|
||||
{
|
||||
/**
|
||||
* Bump this when a mapping change requires a full reindex.
|
||||
* Written into the mapping's _meta.index_compat_version on every build and
|
||||
* read back into blomstra-search.index-compatible by saveIndexedConfig so the
|
||||
* value always reflects the live index — including after a promote/rollback.
|
||||
*/
|
||||
public const INDEX_COMPAT_VERSION = 'v3';
|
||||
|
||||
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
|
||||
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
|
||||
|
||||
/** N-gram bounds for the title autocomplete subfield (not admin-configurable). */
|
||||
private const TITLE_MIN_GRAM = 3;
|
||||
private const TITLE_MAX_GRAM = 20;
|
||||
|
||||
protected $signature = 'blomstra:search:index
|
||||
{--max-id= : Limits for each object the number of items to seed}
|
||||
{--throttle= : Number of seconds to wait between pushing to the queue}
|
||||
{--only= : type to run seeder for, eg discussions or posts}
|
||||
{--recreate : create or recreate the index}
|
||||
{--mapping : recreate the mapping}
|
||||
{--continue : continue each object type where you left off}
|
||||
{--seed-missing : attempt to seed only objects that are missing in the index}';
|
||||
{action? : build | promote | rollback | discard | mapping | fill}
|
||||
{--fresh}
|
||||
{--resume}
|
||||
{--staging}
|
||||
{--keep-backup}
|
||||
{--pending}
|
||||
{--backup}
|
||||
{--only=}
|
||||
{--max-id=}
|
||||
{--throttle=}
|
||||
{--i-am-sure}';
|
||||
|
||||
protected $description = 'Rebuilds the complete search server with its documents.';
|
||||
protected $description = 'Build and manage the Elasticsearch search index.';
|
||||
|
||||
public function handle(Container $container)
|
||||
protected $help = <<<'HELP'
|
||||
<comment>Actions and their options:</comment>
|
||||
|
||||
<info>build</info> Queue documents and promote automatically.
|
||||
<comment>--resume</comment> Resume an interrupted build from where it left off
|
||||
<comment>--fresh</comment> Drop the staging index and start completely fresh
|
||||
<comment>--staging</comment> Keep in staging — requires explicit <info>promote</info> (blue-green)
|
||||
<comment>--keep-backup</comment> Retain the replaced index for rollback
|
||||
|
||||
<info>promote</info> Swap alias to staging index (blue-green workflow).
|
||||
<comment>--keep-backup</comment> Retain the replaced index for rollback
|
||||
<comment>--i-am-sure</comment> Skip the confirmation prompt
|
||||
|
||||
<info>rollback</info> Restore the backup index to live.
|
||||
|
||||
<info>discard</info> Drop an index without promoting.
|
||||
<comment>--pending</comment> Drop the staging index (cancel a build)
|
||||
<comment>--backup</comment> Drop the backup index (cleanup after --keep-backup)
|
||||
|
||||
<info>mapping</info> Push updated mapping to the live index only.
|
||||
|
||||
<info>fill</info> Seed only documents missing from the live index.
|
||||
|
||||
<comment>Shared seeding options (build / fill):</comment>
|
||||
<comment>--only=TYPE</comment> Seed only this type: <info>discussions</info> or <info>posts</info>
|
||||
<comment>--throttle=N</comment> Seconds to wait between batches
|
||||
<comment>--max-id=N</comment> Limit seeding to IDs up to this value
|
||||
HELP;
|
||||
|
||||
public function handle(Container $container): void
|
||||
{
|
||||
/** @var string $index */
|
||||
$index = $container->make('blomstra.search.elastic_index');
|
||||
|
||||
/** @var array|string[] $seeders */
|
||||
$seeders = $container->tagged('blomstra.search.seeders');
|
||||
|
||||
/** @var Queue $queue */
|
||||
$queue = $container->make(Queue::class);
|
||||
|
||||
/** @var Client $client */
|
||||
$client = $container->make(Client::class);
|
||||
|
||||
/** @var SettingsRepositoryInterface $settings */
|
||||
$settings = $container->make(SettingsRepositoryInterface::class);
|
||||
|
||||
// Elastic scheme definition.
|
||||
$properties = [
|
||||
'properties' => [
|
||||
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
|
||||
'rawId' => ['type' => 'integer'],
|
||||
'created_at' => ['type' => 'date'],
|
||||
'updated_at' => ['type' => 'date'],
|
||||
'is_private' => ['type' => 'boolean'],
|
||||
'is_sticky' => ['type' => 'boolean'],
|
||||
'groups' => ['type' => 'integer'],
|
||||
'tags' => ['type' => 'integer'],
|
||||
'recipient_groups' => ['type' => 'integer'],
|
||||
'recipient_users' => ['type' => 'integer'],
|
||||
'comment_count' => ['type' => 'integer'],
|
||||
],
|
||||
];
|
||||
/** @var string $alias */
|
||||
$alias = $container->make('blomstra.search.elastic_index');
|
||||
|
||||
// Remove/delete the whole index.
|
||||
if ($this->option('recreate')) {
|
||||
// Flush the index.
|
||||
$client->indices()->delete([
|
||||
'index' => $index,
|
||||
'ignore_unavailable' => true,
|
||||
]);
|
||||
|
||||
// Create a new index.
|
||||
$client->indices()->create([
|
||||
'index' => $index,
|
||||
'body' => [
|
||||
'settings' => [
|
||||
'index.max_ngram_diff' => 10,
|
||||
'analysis' => [
|
||||
'analyzer' => [
|
||||
'flarum_analyzer' => [
|
||||
'type' => $settings->get('blomstra-search.analyzer-language') ?: 'english',
|
||||
],
|
||||
'flarum_analyzer_partial' => [
|
||||
'type' => 'custom',
|
||||
'tokenizer' => 'standard',
|
||||
'filter' => [
|
||||
'lowercase',
|
||||
'partial_search_filter',
|
||||
],
|
||||
],
|
||||
],
|
||||
'filter' => [
|
||||
'partial_search_filter' => [
|
||||
'type' => 'ngram',
|
||||
'min_gram' => 1,
|
||||
'max_gram' => 10,
|
||||
'token_chars' => ['letter', 'digit', 'symbol'],
|
||||
],
|
||||
],
|
||||
],
|
||||
],
|
||||
],
|
||||
]);
|
||||
if (!$this->argument('action')) {
|
||||
$this->call('help', ['command_name' => $this->getName()]);
|
||||
return;
|
||||
}
|
||||
|
||||
// Create the index.
|
||||
if ($this->option('recreate') || $this->option('mapping')) {
|
||||
switch ($this->argument('action')) {
|
||||
case 'build':
|
||||
$this->runBuild($client, $alias, $settings, $container);
|
||||
break;
|
||||
|
||||
case 'promote':
|
||||
$this->runPromote($client, $alias, $settings);
|
||||
break;
|
||||
|
||||
case 'rollback':
|
||||
$this->runRollback($client, $alias, $settings);
|
||||
break;
|
||||
|
||||
case 'discard':
|
||||
$this->runDiscard($client, $settings);
|
||||
break;
|
||||
|
||||
case 'mapping':
|
||||
$client->indices()->putMapping([
|
||||
'index' => $index,
|
||||
'body' => $properties,
|
||||
'index' => $alias,
|
||||
'body' => $this->mappingProperties(),
|
||||
]);
|
||||
$this->info('Mapping updated on live index.');
|
||||
break;
|
||||
|
||||
case 'fill':
|
||||
$this->runSeeders(
|
||||
collect($container->tagged('blomstra.search.seeders')),
|
||||
$container->make(Queue::class),
|
||||
$client,
|
||||
$settings,
|
||||
$alias,
|
||||
seedMissing: true
|
||||
);
|
||||
break;
|
||||
|
||||
default:
|
||||
$this->error("Unknown action '{$this->argument('action')}'. Valid actions: build, promote, rollback, discard, mapping, fill.");
|
||||
}
|
||||
}
|
||||
|
||||
// Seed only a specific resource.
|
||||
protected function runBuild(Client $client, string $alias, SettingsRepositoryInterface $settings, Container $container): void
|
||||
{
|
||||
/** @var Seeder[] $seeders */
|
||||
$seeders = collect($container->tagged('blomstra.search.seeders'));
|
||||
|
||||
// Guard: staging build exists without explicit intent given.
|
||||
$staging = $settings->get('blomstra-search.staging-index');
|
||||
if ($staging && $client->indices()->exists(['index' => $staging])
|
||||
&& !$this->option('resume') && !$this->option('fresh')
|
||||
) {
|
||||
$this->warn("An in-progress build already exists: $staging");
|
||||
$this->line('');
|
||||
$this->line('Choose one of:');
|
||||
$this->line(' blomstra:search:index build --resume Continue from where it left off');
|
||||
$this->line(' blomstra:search:index build --resume --staging Continue and keep in staging when done');
|
||||
$this->line(' blomstra:search:index build --fresh Drop this build and start completely fresh');
|
||||
$this->line(' blomstra:search:index build --fresh --staging Start fresh and keep in staging when done');
|
||||
return;
|
||||
}
|
||||
|
||||
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
|
||||
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
|
||||
|
||||
if (!$aliasExists && !$indexExists) {
|
||||
// First install: create a timestamped index, alias it immediately, seed live.
|
||||
$targetIndex = $this->prepareFirstInstall($client, $alias, $settings, $seeders);
|
||||
$stagingBuild = false;
|
||||
} else {
|
||||
// Live system: build into a staging index without touching the live alias.
|
||||
$targetIndex = $this->prepareStagingIndex($client, $alias, $settings, $seeders);
|
||||
$stagingBuild = true;
|
||||
}
|
||||
|
||||
$client->indices()->putMapping([
|
||||
'index' => $targetIndex,
|
||||
'body' => $this->mappingProperties(),
|
||||
]);
|
||||
|
||||
$this->runSeeders($seeders, $container->make(Queue::class), $client, $settings, $targetIndex);
|
||||
|
||||
if ($stagingBuild) {
|
||||
if ($this->option('staging')) {
|
||||
$staging = $settings->get('blomstra-search.staging-index');
|
||||
$this->info("Staging build complete. Drain the queue, then promote '$staging' to live:");
|
||||
$this->line(' php flarum queue:work --stop-when-empty');
|
||||
$this->line(' php flarum blomstra:search:index promote');
|
||||
} else {
|
||||
$this->info('Jobs queued. Promoting now (search improves as the queue drains)...');
|
||||
$this->performPromote($client, $alias, $settings);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
protected function runPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
|
||||
{
|
||||
if (!$this->option('i-am-sure')) {
|
||||
$this->warn('promote switches the live index. Ensure all queued jobs have finished first:');
|
||||
$this->line(' php flarum queue:work --stop-when-empty');
|
||||
$this->line('');
|
||||
|
||||
if (!$this->confirm('Queue drained? Proceed?')) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
$this->performPromote($client, $alias, $settings);
|
||||
}
|
||||
|
||||
protected function performPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
|
||||
{
|
||||
$staging = $settings->get('blomstra-search.staging-index');
|
||||
|
||||
if (!$staging || !$client->indices()->exists(['index' => $staging])) {
|
||||
$this->error('No staging index ready to promote. Run: blomstra:search:index build');
|
||||
return;
|
||||
}
|
||||
|
||||
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
|
||||
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
|
||||
|
||||
if ($aliasExists) {
|
||||
$result = $client->indices()->getAlias(['name' => $alias]);
|
||||
$activeIndex = array_key_first($result);
|
||||
|
||||
$client->indices()->updateAliases([
|
||||
'body' => ['actions' => [
|
||||
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
|
||||
['add' => ['index' => $staging, 'alias' => $alias]],
|
||||
]],
|
||||
]);
|
||||
|
||||
$this->info("'$alias' now points to '$staging'. Promote complete.");
|
||||
|
||||
if ($this->option('keep-backup')) {
|
||||
$settings->set('blomstra-search.backup-index', $activeIndex);
|
||||
$this->info("Kept '$activeIndex' as backup — run rollback to revert, discard --backup to drop it.");
|
||||
} else {
|
||||
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
|
||||
$this->info("Deleted old index: $activeIndex");
|
||||
}
|
||||
} else {
|
||||
// One-time migration: concrete index → alias (legacy installs only).
|
||||
if ($indexExists) {
|
||||
$client->indices()->delete(['index' => $alias]);
|
||||
$this->info("Deleted legacy concrete index: $alias");
|
||||
}
|
||||
|
||||
$client->indices()->putAlias(['index' => $staging, 'name' => $alias]);
|
||||
$this->info("'$alias' now points to '$staging'. Promote complete.");
|
||||
}
|
||||
|
||||
$settings->set('blomstra-search.active-index', $staging);
|
||||
$settings->set('blomstra-search.staging-index', null);
|
||||
$this->saveIndexedConfig($client, $settings, $staging);
|
||||
}
|
||||
|
||||
protected function runRollback(Client $client, string $alias, SettingsRepositoryInterface $settings): void
|
||||
{
|
||||
$backup = $settings->get('blomstra-search.backup-index');
|
||||
|
||||
if (!$backup || !$client->indices()->exists(['index' => $backup])) {
|
||||
$this->error('No backup index to roll back to. Was promote run with --keep-backup?');
|
||||
return;
|
||||
}
|
||||
|
||||
$result = $client->indices()->getAlias(['name' => $alias]);
|
||||
$activeIndex = array_key_first($result);
|
||||
|
||||
$client->indices()->updateAliases([
|
||||
'body' => ['actions' => [
|
||||
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
|
||||
['add' => ['index' => $backup, 'alias' => $alias]],
|
||||
]],
|
||||
]);
|
||||
|
||||
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
|
||||
$this->info("Rolled back: '$alias' now points to '$backup'. Deleted '$activeIndex'.");
|
||||
|
||||
$settings->set('blomstra-search.active-index', $backup);
|
||||
$settings->set('blomstra-search.backup-index', null);
|
||||
$this->saveIndexedConfig($client, $settings, $backup);
|
||||
}
|
||||
|
||||
protected function runDiscard(Client $client, SettingsRepositoryInterface $settings): void
|
||||
{
|
||||
$pending = $this->option('pending');
|
||||
$backup = $this->option('backup');
|
||||
|
||||
if (!$pending && !$backup) {
|
||||
$this->error('Specify what to discard: --pending (staging build) or --backup (rollback index).');
|
||||
return;
|
||||
}
|
||||
|
||||
if ($pending) {
|
||||
$staging = $settings->get('blomstra-search.staging-index');
|
||||
|
||||
if (!$staging) {
|
||||
$this->warn('No staging index to discard.');
|
||||
} else {
|
||||
if ($client->indices()->exists(['index' => $staging])) {
|
||||
$client->indices()->delete(['index' => $staging]);
|
||||
}
|
||||
$settings->set('blomstra-search.staging-index', null);
|
||||
$this->info("Discarded staging index: $staging");
|
||||
}
|
||||
}
|
||||
|
||||
if ($backup) {
|
||||
$backupIndex = $settings->get('blomstra-search.backup-index');
|
||||
|
||||
if (!$backupIndex) {
|
||||
$this->warn('No backup index to discard.');
|
||||
} else {
|
||||
if ($client->indices()->exists(['index' => $backupIndex])) {
|
||||
$client->indices()->delete(['index' => $backupIndex]);
|
||||
}
|
||||
$settings->set('blomstra-search.backup-index', null);
|
||||
$this->info("Discarded backup index: $backupIndex");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
protected function runSeeders(
|
||||
iterable $seeders,
|
||||
Queue $queue,
|
||||
Client $client,
|
||||
SettingsRepositoryInterface $settings,
|
||||
string $targetIndex,
|
||||
bool $seedMissing = false
|
||||
): void {
|
||||
$only = $this->option('only');
|
||||
|
||||
/** @var Seeder $seeder */
|
||||
|
|
@ -135,52 +346,53 @@ class BuildCommand extends Command
|
|||
|
||||
$total = 0;
|
||||
|
||||
$continueAt = $this->option('continue')
|
||||
? ($this->continueAt($seeder->type()) ?? $seeder->query()->max('id'))
|
||||
: $seeder->query()->max('id');
|
||||
if ($this->option('resume')) {
|
||||
$saved = $this->getContinueAt($settings, $seeder->type());
|
||||
|
||||
if ($saved === 0) {
|
||||
$this->info("Seeder '{$seeder->type()}' already completed in a previous run, skipping.");
|
||||
continue;
|
||||
}
|
||||
|
||||
$continueAt = $saved ?? $seeder->query()->max('id');
|
||||
} else {
|
||||
$continueAt = $seeder->query()->max('id');
|
||||
}
|
||||
|
||||
$seeded = null;
|
||||
|
||||
while ($continueAt !== null) {
|
||||
if ($this->option('seed-missing')) {
|
||||
$response = (new Builder($client))
|
||||
->index($index)
|
||||
->size(1000)
|
||||
->addQuery(
|
||||
(new BoolQuery())
|
||||
->add((new RangeQuery('rawId'))
|
||||
->gte($continueAt - 1000)
|
||||
->lte($continueAt))
|
||||
->add(TermQuery::create('type', $seeder->type()))
|
||||
)
|
||||
->search();
|
||||
$rangeFrom = max(1, $continueAt - 2500);
|
||||
$rangeTo = $continueAt;
|
||||
|
||||
$seeded = Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
|
||||
if ($seedMissing) {
|
||||
$seeded = $this->queryIndexedIds($client, $targetIndex, $seeder->joinRelation(), $rangeFrom, $rangeTo);
|
||||
}
|
||||
|
||||
/** @var Collection $collection */
|
||||
$collection = $seeder->query()
|
||||
->latest('id')
|
||||
->whereBetween('id', [$continueAt - 1000, $continueAt])
|
||||
->when($this->option('max-id'), function ($query, $id) {
|
||||
$query->where('id', '<=', $id);
|
||||
})
|
||||
->when($seeded, fn ($query, $seeded) => $query->whereNotIn('id', $seeded))
|
||||
->whereBetween('id', [$rangeFrom, $rangeTo])
|
||||
->when($this->option('max-id'), fn ($q, $id) => $q->where('id', '<=', $id))
|
||||
->when($seeded, fn ($q, $seeded) => $q->whereNotIn('id', $seeded))
|
||||
->get();
|
||||
|
||||
$min = $collection->min('id');
|
||||
|
||||
if ($seedMissing && $collection->isEmpty()) {
|
||||
$continueAt = $rangeFrom > 2 ? $rangeFrom - 1 : null;
|
||||
} else {
|
||||
$continueAt = $min && $min > 2 ? $min - 1 : null;
|
||||
}
|
||||
|
||||
$queue->pushOn(Job::$onQueue, new SavingJob($collection, $seeder));
|
||||
if ($collection->isNotEmpty()) {
|
||||
$queue->pushOn(Job::$onQueue, new UpdateSearchJob($collection, $seeder, $targetIndex));
|
||||
}
|
||||
|
||||
$this->info("Pushed into the index, type: {$seeder->type()}, amount: {$collection->count()}.");
|
||||
$this->info("IDs {$rangeFrom}–{$rangeTo} | type: {$seeder->type()} | queued: {$collection->count()}.");
|
||||
|
||||
$total += $collection->count();
|
||||
|
||||
$this->continueAt(
|
||||
$seeder->type(),
|
||||
$continueAt
|
||||
);
|
||||
$this->setContinueAt($settings, $seeder->type(), $continueAt);
|
||||
|
||||
if ($throttle = $this->option('throttle')) {
|
||||
$this->info("Throttling for $throttle seconds");
|
||||
|
|
@ -188,21 +400,274 @@ class BuildCommand extends Command
|
|||
}
|
||||
}
|
||||
|
||||
$this->info("Pushed a total of $total into the index.");
|
||||
$this->setContinueAt($settings, $seeder->type(), 0);
|
||||
$this->info("Queued a total of $total {$seeder->type()} for indexing.");
|
||||
}
|
||||
}
|
||||
|
||||
protected function continueAt(string $type, int $at = null)
|
||||
/**
|
||||
* First install: create a timestamped concrete index, alias the configured name to it
|
||||
* immediately, and return the alias so seeding goes through it and is live from the start.
|
||||
*/
|
||||
protected function prepareFirstInstall(
|
||||
Client $client,
|
||||
string $alias,
|
||||
SettingsRepositoryInterface $settings,
|
||||
iterable $seeders
|
||||
): string {
|
||||
$concrete = $alias . '_' . date('YmdHis');
|
||||
|
||||
$client->indices()->create([
|
||||
'index' => $concrete,
|
||||
'body' => ['settings' => $this->buildIndexSettings($settings)],
|
||||
]);
|
||||
|
||||
$client->indices()->putAlias(['index' => $concrete, 'name' => $alias]);
|
||||
|
||||
$settings->set('blomstra-search.active-index', $concrete);
|
||||
$this->saveIndexedConfig($client, $settings, $concrete);
|
||||
|
||||
foreach ($seeders as $seeder) {
|
||||
$this->setContinueAt($settings, $seeder->type(), null);
|
||||
}
|
||||
|
||||
$this->info("Created '$concrete', aliased '$alias' → '$concrete'.");
|
||||
$this->info("Index is live — documents become searchable as the queue processes.");
|
||||
|
||||
return $alias;
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepare the staging index for a blue-green build.
|
||||
*
|
||||
* - If a staging build exists and --fresh is not set, resume it.
|
||||
* - Otherwise create a fresh timestamped index and save it as staging.
|
||||
*/
|
||||
protected function prepareStagingIndex(
|
||||
Client $client,
|
||||
string $alias,
|
||||
SettingsRepositoryInterface $settings,
|
||||
iterable $seeders
|
||||
): string {
|
||||
$staging = $settings->get('blomstra-search.staging-index');
|
||||
|
||||
if ($this->option('fresh') && $staging) {
|
||||
if ($client->indices()->exists(['index' => $staging])) {
|
||||
$client->indices()->delete(['index' => $staging]);
|
||||
$this->info("Dropped staging index: $staging");
|
||||
}
|
||||
$staging = null;
|
||||
$settings->set('blomstra-search.staging-index', null);
|
||||
}
|
||||
|
||||
if ($staging && $client->indices()->exists(['index' => $staging])) {
|
||||
$this->info("Resuming staging index build: $staging");
|
||||
return $staging;
|
||||
}
|
||||
|
||||
$staging = $alias . '_' . date('YmdHis');
|
||||
|
||||
$client->indices()->create([
|
||||
'index' => $staging,
|
||||
'body' => ['settings' => $this->buildIndexSettings($settings)],
|
||||
]);
|
||||
|
||||
$settings->set('blomstra-search.staging-index', $staging);
|
||||
|
||||
foreach ($seeders as $seeder) {
|
||||
$this->setContinueAt($settings, $seeder->type(), null);
|
||||
}
|
||||
|
||||
$this->info("Created staging index: $staging");
|
||||
|
||||
return $staging;
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist the analysis config and compat version that are actually live in ES for the
|
||||
* given index. Reading from ES (rather than from Flarum settings) means rollbacks are
|
||||
* also covered: the stored values always reflect the index that is currently aliased,
|
||||
* not the settings at the time the command ran. Indexes built before _meta tracking
|
||||
* existed will yield a null compat version, which correctly triggers the reindex warning.
|
||||
*/
|
||||
protected function saveIndexedConfig(Client $client, SettingsRepositoryInterface $settings, string $indexName): void
|
||||
{
|
||||
/** @var SettingsRepositoryInterface $settings */
|
||||
$settings = resolve(SettingsRepositoryInterface::class);
|
||||
$settingsResponse = $client->indices()->getSettings(['index' => $indexName]);
|
||||
$analysis = Arr::get($settingsResponse, "$indexName.settings.index.analysis", []);
|
||||
|
||||
$key = "blomstra-search.continued-at.$type";
|
||||
$analyzer = Arr::get($analysis, 'analyzer.flarum_analyzer.type', 'english');
|
||||
$stemExclusion = Arr::get($analysis, 'analyzer.flarum_analyzer.stem_exclusion', []);
|
||||
|
||||
if ($at) {
|
||||
$settings->set($key, $at);
|
||||
} else {
|
||||
return $settings->get($key);
|
||||
$mappingResponse = $client->indices()->getMapping(['index' => $indexName]);
|
||||
$compatVersion = Arr::get($mappingResponse, "$indexName.mappings._meta.index_compat_version");
|
||||
|
||||
$settings->set('blomstra-search.indexed-analyzer', $analyzer);
|
||||
$settings->set('blomstra-search.indexed-stem-exclusion', implode("\n", $stemExclusion));
|
||||
$settings->set('blomstra-search.index-compatible', $compatVersion);
|
||||
}
|
||||
|
||||
protected function buildIndexSettings(SettingsRepositoryInterface $settings): array
|
||||
{
|
||||
$language = $settings->get('blomstra-search.analyzer-language') ?: 'english';
|
||||
|
||||
$raw = $settings->get('blomstra-search.stem-exclusion', '');
|
||||
$stemExclusion = array_values(array_filter(array_map('trim', explode("\n", $raw))));
|
||||
|
||||
if ($language === 'cjk') {
|
||||
// CJK uses the built-in bigram analyzer; no autocomplete subfield.
|
||||
return [
|
||||
'analysis' => [
|
||||
'analyzer' => [
|
||||
'flarum_analyzer' => ['type' => 'cjk'],
|
||||
],
|
||||
],
|
||||
];
|
||||
}
|
||||
|
||||
$analyzerConfig = ['type' => $language];
|
||||
|
||||
if (!empty($stemExclusion)) {
|
||||
$analyzerConfig['stem_exclusion'] = $stemExclusion;
|
||||
}
|
||||
|
||||
return [
|
||||
'index.max_ngram_diff' => self::TITLE_MAX_GRAM - self::TITLE_MIN_GRAM,
|
||||
'analysis' => [
|
||||
'filter' => [
|
||||
'title_autocomplete_filter' => [
|
||||
'type' => 'edge_ngram',
|
||||
'min_gram' => self::TITLE_MIN_GRAM,
|
||||
'max_gram' => self::TITLE_MAX_GRAM,
|
||||
'token_chars' => ['letter', 'digit'],
|
||||
],
|
||||
],
|
||||
'analyzer' => [
|
||||
// Symmetric language analyzer: used for BOTH index and search on content + title.
|
||||
'flarum_analyzer' => $analyzerConfig,
|
||||
// Title autocomplete index-time: prefix n-grams, no stemming.
|
||||
'flarum_title_autocomplete' => [
|
||||
'type' => 'custom',
|
||||
'tokenizer' => 'standard',
|
||||
'filter' => ['lowercase', 'title_autocomplete_filter'],
|
||||
],
|
||||
// Title autocomplete search-time: just lowercase — predictable on partial words.
|
||||
'flarum_title_search' => [
|
||||
'type' => 'custom',
|
||||
'tokenizer' => 'standard',
|
||||
'filter' => ['lowercase'],
|
||||
],
|
||||
],
|
||||
],
|
||||
];
|
||||
}
|
||||
|
||||
protected function mappingProperties(): array
|
||||
{
|
||||
$language = resolve(SettingsRepositoryInterface::class)->get('blomstra-search.analyzer-language') ?: 'english';
|
||||
|
||||
$titleMapping = [
|
||||
'type' => 'text',
|
||||
'analyzer' => 'flarum_analyzer',
|
||||
];
|
||||
|
||||
// Only add the autocomplete subfield for non-CJK languages.
|
||||
if ($language !== 'cjk') {
|
||||
$titleMapping['fields'] = [
|
||||
'autocomplete' => [
|
||||
'type' => 'text',
|
||||
'analyzer' => 'flarum_title_autocomplete',
|
||||
'search_analyzer' => 'flarum_title_search',
|
||||
],
|
||||
];
|
||||
}
|
||||
|
||||
$contentMapping = [
|
||||
'type' => 'text',
|
||||
'analyzer' => 'flarum_analyzer',
|
||||
];
|
||||
|
||||
// Only add the autocomplete subfield for non-CJK languages.
|
||||
if ($language !== 'cjk') {
|
||||
$contentMapping['fields'] = [
|
||||
'autocomplete' => [
|
||||
'type' => 'text',
|
||||
'analyzer' => 'flarum_title_autocomplete',
|
||||
'search_analyzer' => 'flarum_title_search',
|
||||
],
|
||||
];
|
||||
}
|
||||
|
||||
return [
|
||||
'_meta' => ['index_compat_version' => self::INDEX_COMPAT_VERSION],
|
||||
'properties' => [
|
||||
'join_field' => ['type' => 'join', 'relations' => ['discussion' => 'post']],
|
||||
'discussion_id' => ['type' => 'integer'],
|
||||
'content' => $contentMapping,
|
||||
'title' => $titleMapping,
|
||||
'rawId' => ['type' => 'integer'],
|
||||
'created_at' => ['type' => 'date'],
|
||||
'updated_at' => ['type' => 'date'],
|
||||
'is_private' => ['type' => 'boolean'],
|
||||
'user_id' => ['type' => 'integer'],
|
||||
'groups' => ['type' => 'keyword'],
|
||||
'comment_count' => ['type' => 'integer'],
|
||||
'tags' => ['type' => 'keyword'],
|
||||
'is_sticky' => ['type' => 'boolean'],
|
||||
'is_hidden' => ['type' => 'boolean'],
|
||||
],
|
||||
];
|
||||
}
|
||||
/**
|
||||
* Query ES for rawIds already indexed in $targetIndex for the given $joinRelation and ID range.
|
||||
* Retries on transient ES failures (NoNodesAvailableException / other ElasticsearchException)
|
||||
* by sleeping past the StaticNoPingConnectionPool dead-node timeout (default 60 s) before
|
||||
* each retry, giving the pool a chance to resurface the node.
|
||||
*/
|
||||
protected function queryIndexedIds(
|
||||
Client $client,
|
||||
string $targetIndex,
|
||||
string $joinRelation,
|
||||
int $rangeFrom,
|
||||
int $rangeTo,
|
||||
int $maxRetries = 10
|
||||
): array {
|
||||
$attempt = 0;
|
||||
|
||||
while (true) {
|
||||
try {
|
||||
$response = (new Builder($client))
|
||||
->index($targetIndex)
|
||||
->size(2500)
|
||||
->addQuery(
|
||||
(new BoolQuery())
|
||||
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
|
||||
->add(TermQuery::create('join_field', $joinRelation))
|
||||
)
|
||||
->search();
|
||||
|
||||
return Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
|
||||
} catch (ElasticsearchException $e) {
|
||||
$attempt++;
|
||||
|
||||
if ($attempt >= $maxRetries) {
|
||||
throw $e;
|
||||
}
|
||||
|
||||
$this->warn("ES error on range {$rangeFrom}–{$rangeTo} (attempt {$attempt}/{$maxRetries}): {$e->getMessage()}. Waiting 65 s before retry…");
|
||||
sleep(65); // outlast the StaticNoPingConnectionPool dead-node window (default 60 s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
protected function getContinueAt(SettingsRepositoryInterface $settings, string $type): ?int
|
||||
{
|
||||
$raw = $settings->get("blomstra-search.continued-at.$type");
|
||||
|
||||
return $raw !== null ? (int) $raw : null;
|
||||
}
|
||||
|
||||
protected function setContinueAt(SettingsRepositoryInterface $settings, string $type, ?int $at): void
|
||||
{
|
||||
$settings->set("blomstra-search.continued-at.$type", $at);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,41 @@
|
|||
<?php
|
||||
|
||||
/*
|
||||
* This file is part of blomstra/search.
|
||||
*
|
||||
* Copyright (c) 2022 Blomstra Ltd.
|
||||
*
|
||||
* For the full copyright and license information, please view the LICENSE.md
|
||||
* file that was distributed with this source code.
|
||||
*
|
||||
*/
|
||||
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
class BoolQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery
|
||||
{
|
||||
protected ?int $minimumShouldMatch = null;
|
||||
|
||||
public static function create(): static
|
||||
{
|
||||
return new self();
|
||||
}
|
||||
|
||||
public function minimumShouldMatch(int $minimum): static
|
||||
{
|
||||
$this->minimumShouldMatch = $minimum;
|
||||
|
||||
return $this;
|
||||
}
|
||||
|
||||
public function toArray(): array
|
||||
{
|
||||
$array = parent::toArray();
|
||||
|
||||
if ($this->minimumShouldMatch !== null) {
|
||||
$array['bool']['minimum_should_match'] = $this->minimumShouldMatch;
|
||||
}
|
||||
|
||||
return $array;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
<?php
|
||||
|
||||
/*
|
||||
* This file is part of blomstra/search.
|
||||
*
|
||||
* Copyright (c) 2022 Blomstra Ltd.
|
||||
*
|
||||
* For the full copyright and license information, please view the LICENSE.md
|
||||
* file that was distributed with this source code.
|
||||
*
|
||||
*/
|
||||
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
|
||||
|
||||
class HasChildQuery implements Query
|
||||
{
|
||||
protected bool $innerHits = false;
|
||||
|
||||
public function __construct(
|
||||
protected string $type,
|
||||
protected Query $query,
|
||||
protected string $scoreMode = 'sum'
|
||||
) {}
|
||||
|
||||
public static function create(string $type, Query $query, string $scoreMode = 'sum'): static
|
||||
{
|
||||
return new static($type, $query, $scoreMode);
|
||||
}
|
||||
|
||||
/**
|
||||
* Include inner_hits so the best-matching post ID is available in the response.
|
||||
* ES returns the top-scoring child document per parent hit under inner_hits.best_post.
|
||||
*/
|
||||
public function withInnerHits(): static
|
||||
{
|
||||
$this->innerHits = true;
|
||||
|
||||
return $this;
|
||||
}
|
||||
|
||||
public function toArray(): array
|
||||
{
|
||||
$hasChild = [
|
||||
'type' => $this->type,
|
||||
'score_mode' => $this->scoreMode,
|
||||
'query' => $this->query->toArray(),
|
||||
];
|
||||
|
||||
if ($this->innerHits) {
|
||||
$hasChild['inner_hits'] = ['name' => 'best_post', 'size' => 1];
|
||||
}
|
||||
|
||||
return ['has_child' => $hasChild];
|
||||
}
|
||||
}
|
||||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Elasticsearch;
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Elasticsearch;
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
|
||||
{
|
||||
|
|
@ -62,6 +62,10 @@ class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
|
|||
$query['match'][$this->field]['boost'] = $this->boost;
|
||||
$query['match'][$this->field]['zero_terms_query'] = $this->zeroTerms ? 'all' : 'none';
|
||||
|
||||
if ($this->analyzer !== null) {
|
||||
$query['match'][$this->field]['analyzer'] = $this->analyzer;
|
||||
}
|
||||
|
||||
return $query;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Elasticsearch;
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Elasticsearch;
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Elasticsearch;
|
||||
namespace Sbp\Search\Elasticsearch;
|
||||
|
||||
class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQuery
|
||||
{
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Exceptions;
|
||||
namespace Sbp\Search\Exceptions;
|
||||
|
||||
use Throwable;
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Jobs;
|
||||
namespace Sbp\Search\Jobs;
|
||||
|
||||
use Elasticsearch\Client;
|
||||
use Illuminate\Database\Eloquent\Model;
|
||||
|
|
@ -26,9 +26,10 @@ class DeletingJob extends Job
|
|||
// Preparing body for storing.
|
||||
$body = $this->models->map(function (Model $model) {
|
||||
$document = $this->seeder->toDocument($model);
|
||||
$routing = $this->seeder->routing($model);
|
||||
|
||||
return [
|
||||
['delete' => ['_index' => $this->index, '_id' => $document->id]],
|
||||
['delete' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
|
||||
];
|
||||
})->flatten(1);
|
||||
|
||||
|
|
|
|||
|
|
@ -10,11 +10,11 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Jobs;
|
||||
namespace Sbp\Search\Jobs;
|
||||
|
||||
use Blomstra\Search\Seeders\Seeder;
|
||||
use Sbp\Search\Seeders\Seeder;
|
||||
use Flarum\Queue\AbstractJob;
|
||||
use Illuminate\Support\Collection;
|
||||
use Illuminate\Database\Eloquent\Collection;
|
||||
|
||||
abstract class Job extends AbstractJob
|
||||
{
|
||||
|
|
@ -22,9 +22,13 @@ abstract class Job extends AbstractJob
|
|||
|
||||
public static ?string $onQueue = null;
|
||||
|
||||
public function __construct(protected Collection $models, protected Seeder $seeder)
|
||||
/**
|
||||
* @param string|null $targetIndex Explicit index name for blue-green builds.
|
||||
* Defaults to the configured alias when null.
|
||||
*/
|
||||
public function __construct(protected Collection $models, protected Seeder $seeder, ?string $targetIndex = null)
|
||||
{
|
||||
$this->index = resolve('blomstra.search.elastic_index');
|
||||
$this->index = $targetIndex ?? resolve('blomstra.search.elastic_index');
|
||||
|
||||
if (static::$onQueue) {
|
||||
$this->onQueue(static::$onQueue);
|
||||
|
|
|
|||
|
|
@ -10,14 +10,15 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Jobs;
|
||||
namespace Sbp\Search\Jobs;
|
||||
|
||||
use Blomstra\Search\Exceptions\SeedingException;
|
||||
use Sbp\Search\Exceptions\SeedingException;
|
||||
use Elasticsearch\Client;
|
||||
use Illuminate\Database\Eloquent\Collection as EloquentCollection;
|
||||
use Illuminate\Database\Eloquent\Model;
|
||||
use Illuminate\Support\Arr;
|
||||
|
||||
class SavingJob extends Job
|
||||
class UpdateSearchJob extends Job
|
||||
{
|
||||
public function handle(Client $client)
|
||||
{
|
||||
|
|
@ -25,21 +26,23 @@ class SavingJob extends Job
|
|||
return;
|
||||
}
|
||||
|
||||
$this->models->loadMissing($this->seeder->relationships());
|
||||
|
||||
// Preparing body for storing.
|
||||
$body = $this->models->map(function (Model $model) {
|
||||
$document = $this->seeder->toDocument($model);
|
||||
$routing = $this->seeder->routing($model);
|
||||
|
||||
return [
|
||||
['index' => ['_index' => $this->index, '_id' => $document->id]],
|
||||
['index' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
|
||||
$document->toArray(),
|
||||
];
|
||||
})
|
||||
->flatten(1);
|
||||
dump($body);
|
||||
|
||||
$response = $client->bulk([
|
||||
'index' => $this->index,
|
||||
'body' => $body->toArray(),
|
||||
'refresh' => true,
|
||||
]);
|
||||
|
||||
if (Arr::get($response, 'errors') !== true) {
|
||||
|
|
@ -48,10 +51,11 @@ class SavingJob extends Job
|
|||
|
||||
$items = Arr::get($response, 'items');
|
||||
|
||||
$error = Arr::get(Arr::first($items), 'index.error.reason');
|
||||
$failed = array_filter($items, fn ($item) => isset($item['index']['error']));
|
||||
$error = Arr::get(Arr::first($failed), 'index.error.reason', 'unknown error');
|
||||
|
||||
throw new SeedingException(
|
||||
"Failed to seed: $error",
|
||||
"Failed to seed: $error (" . count($failed) . '/' . count($items) . ' items failed)',
|
||||
$items
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
<?php
|
||||
|
||||
/*
|
||||
* This file is part of blomstra/search.
|
||||
*
|
||||
* Copyright (c) 2022 Blomstra Ltd.
|
||||
*
|
||||
* For the full copyright and license information, please view the LICENSE.md
|
||||
* file that was distributed with this source code.
|
||||
*
|
||||
*/
|
||||
|
||||
namespace Sbp\Search\Jobs;
|
||||
|
||||
use Elasticsearch\Client;
|
||||
use Flarum\Api\Serializer\DiscussionSerializer;
|
||||
use Flarum\Discussion\Discussion;
|
||||
use Flarum\Queue\AbstractJob;
|
||||
|
||||
class ViewsSearchJob extends AbstractJob
|
||||
{
|
||||
protected string $index;
|
||||
protected string $documentType;
|
||||
|
||||
public function __construct(protected int $discussionId)
|
||||
{
|
||||
$this->index = resolve('blomstra.search.elastic_index');
|
||||
$this->documentType = resolve(DiscussionSerializer::class)->getType(new Discussion());
|
||||
|
||||
if (Job::$onQueue) {
|
||||
$this->onQueue(Job::$onQueue);
|
||||
}
|
||||
}
|
||||
|
||||
public function handle(Client $client): void
|
||||
{
|
||||
$discussion = Discussion::find($this->discussionId);
|
||||
|
||||
if (!$discussion) {
|
||||
return;
|
||||
}
|
||||
|
||||
$type = $this->documentType;
|
||||
|
||||
try {
|
||||
$client->update([
|
||||
'index' => $this->index,
|
||||
'id' => "$type:{$this->discussionId}",
|
||||
'routing' => (string) $this->discussionId,
|
||||
'retry_on_conflict' => 3,
|
||||
'body' => [
|
||||
'doc' => ['view_count' => (int) $discussion->view_count],
|
||||
],
|
||||
]);
|
||||
} catch (\Elasticsearch\Common\Exceptions\Missing404Exception $e) {
|
||||
// Document not yet indexed; will be picked up on next: blomstra:search:index fill
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -10,11 +10,12 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search;
|
||||
namespace Sbp\Search;
|
||||
|
||||
use Blomstra\Search\Jobs\DeletingJob;
|
||||
use Blomstra\Search\Jobs\Job;
|
||||
use Blomstra\Search\Jobs\SavingJob;
|
||||
use Sbp\Search\Jobs\DeletingJob;
|
||||
use Sbp\Search\Jobs\Job;
|
||||
use Sbp\Search\Jobs\UpdateSearchJob;
|
||||
use Sbp\Search\Jobs\ViewsSearchJob;
|
||||
use Elasticsearch\Client as Elastic;
|
||||
use Elasticsearch\ClientBuilder;
|
||||
use Flarum\Api\Client;
|
||||
|
|
@ -25,7 +26,7 @@ use Flarum\Settings\SettingsRepositoryInterface;
|
|||
use Illuminate\Contracts\Container\Container;
|
||||
use Illuminate\Contracts\Events\Dispatcher;
|
||||
use Illuminate\Contracts\Queue\Queue;
|
||||
use Illuminate\Support\Collection;
|
||||
use Illuminate\Database\Eloquent\Collection;
|
||||
use Laminas\Stratigility\MiddlewarePipe;
|
||||
use Psr\Log\LoggerInterface;
|
||||
|
||||
|
|
@ -46,7 +47,13 @@ class Provider extends AbstractServiceProvider
|
|||
|
||||
$this->container->singleton(Elastic::class, function (Container $container) use ($settings, $config) {
|
||||
$builder = ClientBuilder::create()
|
||||
->setHosts([$settings->get('blomstra-search.elastic-endpoint')]);
|
||||
->setHosts([$settings->get('blomstra-search.elastic-endpoint')])
|
||||
->setConnectionParams([
|
||||
'client' => [
|
||||
'connect_timeout' => 2, // fail fast if ES is unreachable
|
||||
'timeout' => 10, // allow time for complex queries
|
||||
],
|
||||
]);
|
||||
|
||||
if ($config->inDebugMode()) {
|
||||
$builder->setLogger($container->make(LoggerInterface::class));
|
||||
|
|
@ -87,6 +94,11 @@ class Provider extends AbstractServiceProvider
|
|||
return new Api\Client($pipe);
|
||||
}
|
||||
);
|
||||
|
||||
$this->container->tag([
|
||||
Searchers\DiscussionSearcher::class,
|
||||
Searchers\CommentPostSearcher::class,
|
||||
], 'blomstra.search.searchers');
|
||||
}
|
||||
|
||||
public function boot()
|
||||
|
|
@ -103,12 +115,16 @@ class Provider extends AbstractServiceProvider
|
|||
/** @var string|Seeders\Seeder $seeder */
|
||||
foreach ($seeders as $seeder) {
|
||||
$seeder::savingOn($events, function ($model) use ($queue, $seeder) {
|
||||
$queue->pushOn(Job::$onQueue, new SavingJob(Collection::make([$model]), $seeder));
|
||||
$queue->pushOn(Job::$onQueue, new UpdateSearchJob(Collection::make([$model]), $seeder));
|
||||
});
|
||||
|
||||
$seeder::deletingOn($events, function ($model) use ($queue, $seeder) {
|
||||
$queue->pushOn(Job::$onQueue, new DeletingJob(Collection::make([$model]), $seeder));
|
||||
});
|
||||
|
||||
$seeder::viewingOn($events, function (int $discussionId) use ($queue) {
|
||||
$queue->pushOn(Job::$onQueue, new ViewsSearchJob($discussionId));
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,14 +10,15 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Save;
|
||||
namespace Sbp\Search\Save;
|
||||
|
||||
use Carbon\Carbon;
|
||||
use Illuminate\Support\Arr;
|
||||
use Illuminate\Support\Fluent;
|
||||
|
||||
/**
|
||||
* @property string $type
|
||||
* @property string $id
|
||||
* @property int $rawId
|
||||
* @property string $content
|
||||
* @property Carbon $created_at
|
||||
* @property Carbon $updated_at
|
||||
|
|
@ -30,4 +31,13 @@ use Illuminate\Support\Fluent;
|
|||
*/
|
||||
class Document extends Fluent
|
||||
{
|
||||
/**
|
||||
* Exclude `id` from the body sent to Elasticsearch. The document ID is
|
||||
* already stored as `_id` by the bulk API action; duplicating it in
|
||||
* `_source` wastes space and is redundant.
|
||||
*/
|
||||
public function toArray(): array
|
||||
{
|
||||
return Arr::except($this->attributes, ['id']);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,27 @@
|
|||
<?php
|
||||
|
||||
/*
|
||||
* This file is part of blomstra/search.
|
||||
*
|
||||
* Copyright (c) 2022 Blomstra Ltd.
|
||||
*
|
||||
* For the full copyright and license information, please view the LICENSE.md
|
||||
* file that was distributed with this source code.
|
||||
*
|
||||
*/
|
||||
|
||||
namespace Sbp\Search\Searchers;
|
||||
|
||||
use Sbp\Search\Seeders\CommentSeeder;
|
||||
|
||||
class CommentPostSearcher extends Searcher
|
||||
{
|
||||
protected string|null $seeder = CommentSeeder::class;
|
||||
|
||||
public function enabled(): bool
|
||||
{
|
||||
$enabled = $this->setting('blomstra-search.search-post-bodies', true);
|
||||
|
||||
return boolval($enabled);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
<?php
|
||||
|
||||
/*
|
||||
* This file is part of blomstra/search.
|
||||
*
|
||||
* Copyright (c) 2022 Blomstra Ltd.
|
||||
*
|
||||
* For the full copyright and license information, please view the LICENSE.md
|
||||
* file that was distributed with this source code.
|
||||
*
|
||||
*/
|
||||
|
||||
namespace Sbp\Search\Searchers;
|
||||
|
||||
use Sbp\Search\Seeders\DiscussionSeeder;
|
||||
|
||||
class DiscussionSearcher extends Searcher
|
||||
{
|
||||
protected string|null $seeder = DiscussionSeeder::class;
|
||||
|
||||
public function enabled(): bool
|
||||
{
|
||||
$enabled = $this->setting('blomstra-search.search-discussion-subjects', true);
|
||||
|
||||
return boolval($enabled);
|
||||
}
|
||||
|
||||
public function boost(): float
|
||||
{
|
||||
return 1.5;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
<?php
|
||||
|
||||
/*
|
||||
* This file is part of blomstra/search.
|
||||
*
|
||||
* Copyright (c) 2022 Blomstra Ltd.
|
||||
*
|
||||
* For the full copyright and license information, please view the LICENSE.md
|
||||
* file that was distributed with this source code.
|
||||
*
|
||||
*/
|
||||
|
||||
namespace Sbp\Search\Searchers;
|
||||
|
||||
use Sbp\Search\Seeders\Seeder;
|
||||
use Flarum\Settings\SettingsRepositoryInterface;
|
||||
|
||||
abstract class Searcher
|
||||
{
|
||||
protected string|null $seeder = null;
|
||||
|
||||
public function type(): string
|
||||
{
|
||||
/** @var Seeder $seeder */
|
||||
$seeder = $this->seeder;
|
||||
|
||||
if (empty($seeder)) {
|
||||
throw new \InvalidArgumentException('Implement type or add $seeder');
|
||||
}
|
||||
|
||||
return (new $seeder())->type();
|
||||
}
|
||||
|
||||
public function enabled(): bool
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
public function boost(): float
|
||||
{
|
||||
return 1;
|
||||
}
|
||||
|
||||
protected function setting(string $key, $default = null)
|
||||
{
|
||||
return resolve(SettingsRepositoryInterface::class)->get($key, $default);
|
||||
}
|
||||
}
|
||||
|
|
@ -10,13 +10,14 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Seeders;
|
||||
namespace Sbp\Search\Seeders;
|
||||
|
||||
use Blomstra\Search\Save\Document;
|
||||
use Sbp\Search\Save\Document;
|
||||
use Flarum\Api\Serializer\DiscussionSerializer;
|
||||
use Flarum\Api\Serializer\PostSerializer;
|
||||
use Flarum\Discussion\Discussion;
|
||||
use Flarum\Post\CommentPost;
|
||||
use Flarum\Post\Event\Deleted;
|
||||
use Flarum\Post\Event\Posted;
|
||||
use Flarum\Post\Event as Core;
|
||||
use Illuminate\Contracts\Events\Dispatcher;
|
||||
use Illuminate\Database\Eloquent\Builder;
|
||||
use Illuminate\Database\Eloquent\Model;
|
||||
|
|
@ -28,69 +29,68 @@ class CommentSeeder extends Seeder
|
|||
return resolve(PostSerializer::class)->getType(new CommentPost());
|
||||
}
|
||||
|
||||
public function joinRelation(): string
|
||||
{
|
||||
return 'post';
|
||||
}
|
||||
|
||||
public function routing(Model $model): string
|
||||
{
|
||||
return (string) $model->discussion_id;
|
||||
}
|
||||
|
||||
public function query(): Builder
|
||||
{
|
||||
$includes = ['discussion'];
|
||||
|
||||
if ($this->extensionEnabled('flarum-tags')) {
|
||||
$includes[] = 'discussion.tags';
|
||||
}
|
||||
|
||||
if ($this->extensionEnabled('fof-byobu')) {
|
||||
$includes[] = 'discussion.recipientUsers';
|
||||
$includes[] = 'discussion.recipientGroups';
|
||||
}
|
||||
|
||||
return CommentPost::query()
|
||||
->whereNull('hidden_at')
|
||||
->where('type', CommentPost::$type)
|
||||
->with($includes);
|
||||
->where('type', CommentPost::$type);
|
||||
}
|
||||
|
||||
public static function savingOn(Dispatcher $events, callable $callable)
|
||||
{
|
||||
$events->listen(Posted::class, function (Posted $event) use ($callable) {
|
||||
$events->listen([
|
||||
Core\Posted::class,
|
||||
Core\Revised::class,
|
||||
Core\Hidden::class,
|
||||
Core\Restored::class,
|
||||
], function ($event) use ($callable) {
|
||||
$callable($event->post);
|
||||
});
|
||||
}
|
||||
|
||||
public static function deletingOn(Dispatcher $events, callable $callable)
|
||||
{
|
||||
$events->listen(Deleted::class, function (Deleted $event) use ($callable) {
|
||||
$events->listen([
|
||||
Core\Deleted::class
|
||||
], function ($event) use ($callable) {
|
||||
$callable($event->post);
|
||||
});
|
||||
}
|
||||
|
||||
/** Cached discussion type string (e.g. "discussions") — resolved once per job. */
|
||||
private ?string $discussionType = null;
|
||||
|
||||
private function discussionType(): string
|
||||
{
|
||||
return $this->discussionType ??= resolve(DiscussionSerializer::class)->getType(new Discussion());
|
||||
}
|
||||
|
||||
/**
|
||||
* @param CommentPost $model
|
||||
* Post documents only need content for has_child matching and the join
|
||||
* field to establish the parent-child relationship. All discussion-level
|
||||
* fields (groups, tags, comment_count, etc.) live on the discussion
|
||||
* document and are irrelevant here.
|
||||
*
|
||||
* @return Document
|
||||
* @param CommentPost $model
|
||||
*/
|
||||
public function toDocument(Model $model): Document
|
||||
{
|
||||
$document = new Document([
|
||||
'type' => $this->type(),
|
||||
return new Document([
|
||||
'join_field' => ['name' => $this->joinRelation(), 'parent' => "{$this->discussionType()}:{$model->discussion_id}"],
|
||||
'discussion_id' => $model->discussion_id,
|
||||
'id' => $this->type().':'.$model->id,
|
||||
'rawId' => $model->id,
|
||||
'content' => $model->content,
|
||||
'content_partial' => $model->content,
|
||||
'created_at' => $model->created_at?->toAtomString(),
|
||||
'updated_at' => $model->edited_at?->toAtomString(),
|
||||
'is_private' => $model->is_private,
|
||||
'user_id' => $model->user_id,
|
||||
'groups' => $this->groupsForDiscussion($model->discussion),
|
||||
'comment_count' => $model->discussion->comment_count,
|
||||
'is_hidden' => $model->hidden_at !== null,
|
||||
]);
|
||||
|
||||
if ($this->extensionEnabled('flarum-tags')) {
|
||||
$document['tags'] = $model->discussion->tags->pluck('id')->toArray();
|
||||
}
|
||||
|
||||
if ($this->extensionEnabled('fof-byobu')) {
|
||||
$document['recipient_users'] = $model->discussion->recipientUsers->pluck('id')->toArray();
|
||||
$document['recipient_groups'] = $model->discussion->recipientGroups->pluck('id')->toArray();
|
||||
}
|
||||
|
||||
return $document;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,17 +10,21 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Seeders;
|
||||
namespace Sbp\Search\Seeders;
|
||||
|
||||
use Blomstra\Search\Save\Document;
|
||||
use Sbp\Search\Save\Document;
|
||||
use Flarum\Api\Serializer\DiscussionSerializer;
|
||||
use Flarum\Discussion\Discussion;
|
||||
use Flarum\Discussion\Event\Deleted;
|
||||
use Flarum\Discussion\Event\Hidden;
|
||||
use Flarum\Discussion\Event\Restored;
|
||||
use Flarum\Discussion\Event\Started;
|
||||
use Flarum\Discussion\Event as Core;
|
||||
use Flarum\Extension\ExtensionManager;
|
||||
use Flarum\Group\Group;
|
||||
use Flarum\Group\Permission;
|
||||
use Flarum\Tags\Tag;
|
||||
use FoF\Byobu\Events as Byobu;
|
||||
use FoF\DiscussionViews\Events\DiscussionWasViewed;
|
||||
use Illuminate\Contracts\Events\Dispatcher;
|
||||
use Illuminate\Database\Eloquent\Builder;
|
||||
use Illuminate\Support\Collection;
|
||||
use Illuminate\Database\Eloquent\Model;
|
||||
|
||||
class DiscussionSeeder extends Seeder
|
||||
|
|
@ -30,7 +34,23 @@ class DiscussionSeeder extends Seeder
|
|||
return resolve(DiscussionSerializer::class)->getType(new Discussion());
|
||||
}
|
||||
|
||||
public function joinRelation(): string
|
||||
{
|
||||
return 'discussion';
|
||||
}
|
||||
|
||||
public function routing(Model $model): string
|
||||
{
|
||||
return (string) $model->id;
|
||||
}
|
||||
|
||||
public function query(): Builder
|
||||
{
|
||||
return Discussion::query()
|
||||
->whereNull('hidden_at');
|
||||
}
|
||||
|
||||
public function relationships(): array
|
||||
{
|
||||
$includes = [];
|
||||
|
||||
|
|
@ -43,21 +63,48 @@ class DiscussionSeeder extends Seeder
|
|||
$includes[] = 'recipientGroups';
|
||||
}
|
||||
|
||||
return Discussion::query()
|
||||
->whereNull('hidden_at')
|
||||
->with($includes);
|
||||
return $includes;
|
||||
}
|
||||
|
||||
public static function savingOn(Dispatcher $events, callable $callable)
|
||||
{
|
||||
$events->listen([Started::class, Restored::class], function ($event) use ($callable) {
|
||||
$events->listen([
|
||||
// flarum/core events
|
||||
Core\Started::class, Core\Restored::class, Core\Renamed::class,
|
||||
// fof/byobu discussion recipients events.
|
||||
Byobu\DiscussionMadePublic::class, Byobu\RemovedSelf::class, Byobu\RecipientsChanged::class,
|
||||
], function ($event) use ($callable) {
|
||||
return $callable($event->discussion);
|
||||
});
|
||||
}
|
||||
|
||||
public static function viewingOn(Dispatcher $events, callable $callable): void
|
||||
{
|
||||
if (!resolve(ExtensionManager::class)->isEnabled('fof-discussion-views')) {
|
||||
return;
|
||||
}
|
||||
|
||||
$events->listen(DiscussionWasViewed::class, function (DiscussionWasViewed $event) use ($callable) {
|
||||
$viewCount = $event->discussion->view_count;
|
||||
|
||||
$shouldSync = match (true) {
|
||||
$viewCount < 15 => true,
|
||||
$viewCount < 100 => rand(1, 3) === 1,
|
||||
default => rand(1, 19) === 1,
|
||||
};
|
||||
|
||||
if ($shouldSync) {
|
||||
$callable($event->discussion->id);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
public static function deletingOn(Dispatcher $events, callable $callable)
|
||||
{
|
||||
$events->listen([Deleted::class, Hidden::class], function ($event) use ($callable) {
|
||||
$events->listen([
|
||||
// flarum/core events.
|
||||
Core\Deleted::class, Core\Hidden::class
|
||||
], function ($event) use ($callable) {
|
||||
return $callable($event->discussion);
|
||||
});
|
||||
}
|
||||
|
|
@ -70,11 +117,11 @@ class DiscussionSeeder extends Seeder
|
|||
public function toDocument(Model $model): Document
|
||||
{
|
||||
$document = new Document([
|
||||
'type' => $this->type(),
|
||||
'join_field' => $this->joinRelation(),
|
||||
'id' => $this->type().':'.$model->id,
|
||||
'rawId' => $model->id,
|
||||
'content' => $model->title,
|
||||
'content_partial' => $model->title,
|
||||
'title' => $model->title,
|
||||
'created_at' => $model->created_at?->toAtomString(),
|
||||
'updated_at' => $model->last_posted_at?->toAtomString(),
|
||||
'is_private' => $model->is_private,
|
||||
|
|
@ -88,14 +135,76 @@ class DiscussionSeeder extends Seeder
|
|||
}
|
||||
|
||||
if ($this->extensionEnabled('fof-byobu')) {
|
||||
$document['recipient_users'] = $model->recipientUsers->pluck('id')->toArray();
|
||||
$document['recipient_groups'] = $model->recipientGroups->pluck('id')->toArray();
|
||||
$document['recipient_users'] = $model->recipientUsers
|
||||
->whereNull('removed_at')
|
||||
->pluck('id')
|
||||
->toArray();
|
||||
$document['recipient_groups'] = $model->recipientGroups
|
||||
->whereNull('removed_at')
|
||||
->pluck('id')
|
||||
->toArray();
|
||||
}
|
||||
|
||||
if ($this->extensionEnabled('flarum-sticky')) {
|
||||
$document['is_sticky'] = (bool) $model->is_sticky;
|
||||
}
|
||||
|
||||
if ($this->extensionEnabled('fof-discussion-views')) {
|
||||
$document['view_count'] = (int) ($model->view_count ?? 0);
|
||||
}
|
||||
|
||||
return $document;
|
||||
}
|
||||
|
||||
/**
|
||||
* All viewForum permissions keyed by permission string, loaded once per seeder instance.
|
||||
* Avoids N×2 Permission queries inside the per-document map loop.
|
||||
*/
|
||||
private ?Collection $cachedPermissions = null;
|
||||
private ?Collection $cachedGlobalPermission = null;
|
||||
|
||||
private function allPermissions(): Collection
|
||||
{
|
||||
if ($this->cachedPermissions === null) {
|
||||
$this->cachedPermissions = Permission::query()
|
||||
->where(function ($q) {
|
||||
$q->where('permission', 'viewForum')
|
||||
->orWhere('permission', 'like', 'tag%.viewForum');
|
||||
})
|
||||
->get();
|
||||
|
||||
$this->cachedGlobalPermission = $this->cachedPermissions
|
||||
->where('permission', 'viewForum')
|
||||
->pluck('group_id');
|
||||
}
|
||||
|
||||
return $this->cachedPermissions;
|
||||
}
|
||||
|
||||
protected function groupsForDiscussion(Discussion $discussion): array
|
||||
{
|
||||
$allPerms = $this->allPermissions();
|
||||
$permissions = collect();
|
||||
|
||||
if ($this->extensionEnabled('flarum-tags')) {
|
||||
/** @var Collection $tags */
|
||||
$tags = $discussion->tags;
|
||||
|
||||
$permissions = $tags->map(function (Tag $tag) use ($allPerms) {
|
||||
$tagPerms = $allPerms->where('permission', "tag$tag->id.viewForum");
|
||||
|
||||
if ($tag->is_restricted) {
|
||||
$tagPerms = $tagPerms->add(['group_id' => Group::ADMINISTRATOR_ID]);
|
||||
}
|
||||
|
||||
return $tagPerms->pluck('group_id');
|
||||
})->flatten();
|
||||
}
|
||||
|
||||
if (!$discussion->is_private && $permissions->isEmpty()) {
|
||||
$permissions = $this->cachedGlobalPermission;
|
||||
}
|
||||
|
||||
return $permissions->toArray();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,67 +10,45 @@
|
|||
*
|
||||
*/
|
||||
|
||||
namespace Blomstra\Search\Seeders;
|
||||
namespace Sbp\Search\Seeders;
|
||||
|
||||
use Blomstra\Search\Save\Document;
|
||||
use Flarum\Discussion\Discussion;
|
||||
use Sbp\Search\Save\Document;
|
||||
use Flarum\Extension\ExtensionManager;
|
||||
use Flarum\Group\Group;
|
||||
use Flarum\Group\Permission;
|
||||
use Flarum\Tags\Tag;
|
||||
use Illuminate\Contracts\Events\Dispatcher;
|
||||
use Illuminate\Database\Eloquent\Builder;
|
||||
use Illuminate\Database\Eloquent\Collection;
|
||||
use Illuminate\Database\Eloquent\Model;
|
||||
|
||||
abstract class Seeder
|
||||
{
|
||||
abstract public function type(): string;
|
||||
|
||||
/**
|
||||
* The join relation name for this document type ('discussion' or 'post').
|
||||
* Used in the join_field mapping and for filtering in seed-missing checks.
|
||||
*/
|
||||
abstract public function joinRelation(): string;
|
||||
|
||||
/**
|
||||
* The ES routing key for this model. Parent and child documents for the same
|
||||
* discussion must share a routing key so ES places them on the same shard.
|
||||
*/
|
||||
abstract public function routing(Model $model): string;
|
||||
|
||||
abstract public function query(): Builder;
|
||||
|
||||
abstract public static function savingOn(Dispatcher $events, callable $callable);
|
||||
|
||||
abstract public static function deletingOn(Dispatcher $events, callable $callable);
|
||||
|
||||
/** No-op default; override in seeders that need to react to view events. */
|
||||
public static function viewingOn(Dispatcher $events, callable $callable): void {}
|
||||
|
||||
abstract public function toDocument(Model $model): Document;
|
||||
|
||||
protected function groupsForDiscussion(Discussion $discussion): array
|
||||
/** Relationships to eager-load on the collection before calling toDocument(). */
|
||||
public function relationships(): array
|
||||
{
|
||||
$permissions = collect();
|
||||
|
||||
$globalPermission = Permission::query()
|
||||
->where('permission', 'viewForum')
|
||||
->pluck('group_id');
|
||||
|
||||
if ($this->extensionEnabled('flarum-tags')) {
|
||||
/** @var Collection $tags */
|
||||
$tags = $discussion->tags;
|
||||
|
||||
$tagPermissions = Permission::query()
|
||||
->whereIn(
|
||||
'permission',
|
||||
$tags->pluck('id')->map(function (int $id) {
|
||||
return "tag$id.viewForum";
|
||||
})
|
||||
)->get();
|
||||
|
||||
$permissions = $tags->map(function (Tag $tag) use ($tagPermissions) {
|
||||
$permissions = $tagPermissions->where('permission', "tag$tag->id.viewForum");
|
||||
|
||||
if ($tag->is_restricted) {
|
||||
$permissions = $permissions->add(['group_id' => Group::ADMINISTRATOR_ID]);
|
||||
}
|
||||
|
||||
return $permissions->pluck('group_id');
|
||||
})->flatten();
|
||||
}
|
||||
|
||||
if (!$discussion->is_private && $permissions->isEmpty()) {
|
||||
$permissions = $globalPermission;
|
||||
}
|
||||
|
||||
return $permissions->toArray();
|
||||
return [];
|
||||
}
|
||||
|
||||
protected function extensionEnabled(string $extension): bool
|
||||
|
|
|
|||
Loading…
Reference in New Issue