Compare commits

..

No commits in common. "main" and "0.0.9" have entirely different histories.
main ... 0.0.9

40 changed files with 8698 additions and 4531 deletions

View File

@ -1,18 +1,17 @@
name: Search JS name: Build JavaScript assets
on: [workflow_dispatch, push, pull_request] on:
push:
branches:
- main
jobs: jobs:
run: build:
uses: flarum/framework/.github/workflows/REUSABLE_frontend.yml@main runs-on: ubuntu-latest
with:
enable_bundlewatch: false steps:
enable_prettier: true - uses: actions/checkout@master
enable_typescript: false - uses: flarum/action-build@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
frontend_directory: ./js
backend_directory: .
js_package_manager: npm
main_git_branch: main
secrets:
bundlewatch_github_token: ${{ secrets.BUNDLEWATCH_GITHUB_TOKEN }}

View File

@ -1,9 +0,0 @@
MIT License
Copyright (c) 2021 Blomstra Ltd
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

143
README.md
View File

@ -5,154 +5,47 @@ fulltext search with one that is completely relying on the proven elasticsearch
## Features ## Features
- Sync discussions and posts to Elasticsearch using your queue, unobtrusively for the user. - Sync discussions to elastic search using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50,000 discussions: **260ms**) - Reduces search loading times to well below 400ms (local tests with 50.000 discussion **260ms**)
- Uses Flarum's group permissions and tags system. - Uses Flarum's group permissions and tags system.
- Compatible with Friends of Flarum Byōbu. - Compatible with Friends of Flarum Byōbu.
## Requirements
- Elasticsearch 7.x or OpenSearch 1.x+
- A non-sync queue driver with a running worker (`php flarum queue:work`) is strongly recommended for production. The extension works with the default sync driver, but index jobs run inline which adds latency to user-facing changes like posting.
## Installation ## Installation
Use composer:
```bash ```bash
composer require blomstra/search:* composer require blomstra/search:*
``` ```
Enable the extension in the admin area and configure the Elasticsearch endpoint and index name in the extension settings. Enable the extension inside the admin area and configure the settings.
## Setting up the index ### Set up
### First install Enable the extension in your admin area. Now to seed your existing discussions use the following command:
Run the build command once. It creates a timestamped concrete index, immediately aliases your configured index name to it, and begins queuing documents. Search is available and improves as the queue processes: ```
php flarum blomstra:search:build
```bash
php flarum blomstra:search:index build
php flarum queue:work
``` ```
### Subsequent rebuilds (zero-downtime) All mutations to discussions are automatically added and removed from the elasticsearch index.
When you need to rebuild the full index (e.g. after a mapping change): ### FAQ
```bash *I have another question.*
# Simple rebuild — promotes automatically once all jobs are queued Reach out to us via https://helpdesk.blomstra.net. We will get back to you as soon as we can. If you have a running subscription please mention when you started your plan and/or which plan you are on. Always add sufficient information when reporting errors. We prefer errors being reported here, but understand that sometimes you can't.
php flarum blomstra:search:index build
# Or keep a backup of the old index in case you need to roll back
php flarum blomstra:search:index build --keep-backup
```
After the queue drains, fill any gaps from content posted during the build:
```bash
php flarum blomstra:search:index fill
```
If you kept a backup and want to roll back:
```bash
php flarum blomstra:search:index rollback
```
Once satisfied with the new index, drop the backup:
```bash
php flarum blomstra:search:index discard --backup
```
### Blue-green rebuild (manual promotion)
Use `--staging` to keep the old index live until you explicitly promote:
```bash
# 1. Build into a staging index — live index is untouched
php flarum blomstra:search:index build --staging
# 2. Drain the queue
php flarum queue:work --stop-when-empty
# 3. Promote the staging index to live
php flarum blomstra:search:index promote
# Or keep the old index as a backup:
php flarum blomstra:search:index promote --keep-backup
```
### Resuming or cancelling an interrupted build
If a build is interrupted, re-run it with the appropriate flag:
```bash
# Resume each seeder from where it left off
php flarum blomstra:search:index build --resume
# Drop the staging index and start completely fresh
php flarum blomstra:search:index build --fresh
# Cancel the build without starting a new one
php flarum blomstra:search:index discard --pending
```
### Filling gaps in an existing index
If documents are missing from the live index (e.g. due to queue failures):
```bash
php flarum blomstra:search:index fill
```
### Updating the mapping only
To push a mapping change to the live index without rebuilding:
```bash
php flarum blomstra:search:index mapping
```
## Command reference
| Command | Description |
|---|---|
| `build` | Rebuild the index and promote automatically once all jobs are queued. On first install, aliases immediately so search is live during seeding. |
| `build --keep-backup` | Rebuild and promote, retaining the old index as a backup for rollback. |
| `build --staging` | Build into a staging index without promoting — use `promote` when ready (blue-green workflow). |
| `build --resume` | Resume an interrupted build from where each seeder left off. |
| `build --fresh` | Drop the staging index and start completely fresh. |
| `promote` | Atomically swap the alias to the staging index. Prompts for confirmation (blue-green workflow). |
| `promote --keep-backup` | Promote and retain the replaced live index as a backup for rollback. |
| `rollback` | Restore the backup index to live. Deletes the index that was live. |
| `discard --pending` | Drop the staging index without promoting (cancels an in-progress build). |
| `discard --backup` | Drop the backup index (cleanup after `--keep-backup`). |
| `mapping` | Push updated mapping to the live index without rebuilding or reseeding. |
| `fill` | Seed only documents missing from the live index. |
| `build --only=discussions` | Seed only the specified document type (`discussions` or `posts`). |
| `build --throttle=N` | Wait N seconds between batches (reduces queue pressure). |
| `build --max-id=N` | Limit seeding to documents with ID ≤ N. |
| `promote --i-am-sure` | Skip the promotion confirmation prompt (for scripts and CI). |
## FAQ
## Queue configuration
*"Can I dispatch indexing jobs to a specific queue?"*
*Can I dispatch the sync jobs to another queue?*
Yes: Yes:
```php ```php
\Blomstra\Search\Jobs\Job::$onQueue = 'search'; \Blomstra\Search\Observe\Job::$onQueue = 'sync';
``` ```
*"I have a different question"*
Reach out ot us via https://support.on-floxum.com/t/ext-search . If you have an active subscription, please mention what plan you are on.
--- ---
- Floxum provides managed Flarum hosting. - Blomstra provides managed Flarum hosting.
- https://floxum.com - https://blomstra.net
- https://support.on-floxum.com/t/ext-search - https://blomstra.community/t/ext-search
Icon made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/). Icon made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/).

View File

@ -3,12 +3,8 @@
"description": "Replaces Flarum search with one powered by an elastic search server.", "description": "Replaces Flarum search with one powered by an elastic search server.",
"keywords": [ "keywords": [
"extension", "extension",
"flarum", "flarum", "blomstra",
"blomstra", "search", "fulltext", "elastic", "elasticsearch"
"search",
"fulltext",
"elastic",
"elasticsearch"
], ],
"support": { "support": {
"email": "helpdesk@blomstra.net", "email": "helpdesk@blomstra.net",
@ -18,14 +14,14 @@
"license": "MIT", "license": "MIT",
"authors": [ "authors": [
{ {
"name": "Team Blomstra", "name": "Daniël Klabbers",
"email": "development@blomstra.net", "email": "daniel@klabbers.email",
"role": "Developer" "homepage": "http://luceos.com"
} }
], ],
"require": { "require": {
"php": ">= 8.0", "php": ">= 8.0",
"flarum/core": "^1.2.0", "flarum/core": "^1.0.0",
"elasticsearch/elasticsearch": "7.*", "elasticsearch/elasticsearch": "7.*",
"spatie/elasticsearch-query-builder": "^1.3.0" "spatie/elasticsearch-query-builder": "^1.3.0"
}, },

View File

@ -1,38 +1,23 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search; namespace Blomstra\Search;
use Flarum\Extend as Flarum; use Flarum\Extend as Flarum;
return [ return [
(new Flarum\ServiceProvider())->register(Provider::class), (new Flarum\ServiceProvider)->register(Provider::class),
(new Flarum\Frontend('forum')) (new Flarum\Frontend('forum'))
->js(__DIR__.'/js/dist/forum.js'), ->js(__DIR__ . '/js/dist/forum.js'),
(new Flarum\Frontend('admin')) (new Flarum\Frontend('admin'))
->js(__DIR__.'/js/dist/admin.js') ->js(__DIR__ . '/js/dist/admin.js'),
->css(__DIR__.'/resources/less/admin.less'),
new Flarum\Locales(__DIR__.'/resources/locale'), (new Flarum\Locales(__DIR__ . '/resources/locale')),
(new Flarum\Routes('api')) (new Flarum\Routes('api'))
->get('/blomstra/search/{type}', 'blomstra.search', Api\Controllers\SearchController::class) ->get('/blomstra/search/{type}', 'blomstra.search', Api\Controllers\SearchController::class)
->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class), ->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class),
(new Flarum\Console())
->command(Commands\BuildCommand::class),
(new Flarum\Settings()) (new Flarum\Console)
->default('blomstra-search.search-discussion-subjects', true) ->command(Commands\BuildCommand::class)
->default('blomstra-search.search-post-bodies', true)
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH),
]; ];

2
js/dist/admin.js vendored
View File

@ -1,2 +1,2 @@
(()=>{var t={n:a=>{var e=a&&a.__esModule?()=>a.default:()=>a;return t.d(e,{a:e}),e},d:(a,e)=>{for(var r in e)t.o(e,r)&&!t.o(a,r)&&Object.defineProperty(a,r,{enumerable:!0,get:e[r]})},o:(t,a)=>Object.prototype.hasOwnProperty.call(t,a)};(()=>{"use strict";function a(t,e){return a=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(t,a){return t.__proto__=a,t},a(t,e)}const e=flarum.core.compat["admin/app"];var r=t.n(e);const n=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=t.n(s);const o=flarum.core.compat["admin/components/DashboardWidget"];var l=t.n(o);const c=flarum.core.compat["common/components/Alert"];var h=t.n(c),d=function(t){var e,n;function s(){return t.apply(this,arguments)||this}n=t,(e=s).prototype=Object.create(n.prototype),e.prototype.constructor=e,a(e,n);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:r().translator.trans("blomstra-search.admin.reindex-required.title")},r().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(l());r().initializers.add("blomstra-search",function(){var t=r().data.settings["blomstra-search.active-index"],a=r().data.settings["blomstra-search.index-compatible"];t&&"v2"!==a&&(0,n.extend)(i().prototype,"availableWidgets",function(t){t.add("blomstra-search-reindex",m(d),110)});var e=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach(function(t){e.set(t,t)}),r().extensionData.for("blomstra-search").registerSetting(function(){var a=r().data.settings["blomstra-search.indexed-analyzer"];if(!t||!a)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",n=this.setting("blomstra-search.min-search-length")(),s=String(r().data.settings["blomstra-search.indexed-min-search-length"]||r().data.settings["blomstra-search.min-search-length"]);return e===a&&n===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},r().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:r().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:r().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:r().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:r().translator.trans("blomstra-search.admin.analyzer.label"),help:r().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(e.entries()),default:"english"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:r().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:r().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.min-search-length",label:r().translator.trans("blomstra-search.admin.min-search-length.label"),help:r().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:r().data.settings["blomstra-search.min-search-length"]})})})(),module.exports={}})(); module.exports=function(e){var t={};function a(r){if(t[r])return t[r].exports;var n=t[r]={i:r,l:!1,exports:{}};return e[r].call(n.exports,n,n.exports,a),n.l=!0,n.exports}return a.m=e,a.c=t,a.d=function(e,t,r){a.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},a.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},a.t=function(e,t){if(1&t&&(e=a(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(a.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)a.d(r,n,function(t){return e[t]}.bind(null,n));return r},a.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return a.d(t,"a",t),t},a.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},a.p="",a(a.s=11)}({1:function(e,t){e.exports=flarum.core.compat["admin/app"]},11:function(e,t,a){"use strict";a.r(t);var r=a(1),n=a.n(r);n.a.initializers.add("blomstra-search",(function(){var e=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach((function(t){e.set(t,t)})),n.a.extensionData.for("blomstra-search").registerSetting({setting:"blomstra-search.elastic-endpoint",label:n.a.translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n.a.translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n.a.translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n.a.translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n.a.translator.trans("blomstra-search.admin.analyzer.label"),help:n.a.translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(e.entries()),default:"english"})}))}});
//# sourceMappingURL=admin.js.map //# sourceMappingURL=admin.js.map

File diff suppressed because one or more lines are too long

3
js/dist/forum.js vendored

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

10831
js/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -1,14 +1,16 @@
{ {
"name": "@blomstra/search", "name": "@blomstra/realtime",
"private": true, "private": true,
"version": "0.0.0", "version": "0.0.0",
"prettier": "@flarum/prettier-config", "prettier": "@flarum/prettier-config",
"dependencies": { "dependencies": {
"@flarum/prettier-config": "^1.0.0", "@flarum/prettier-config": "^1.0.0",
"build": "^0.1.4",
"flarum-tsconfig": "^1.0.2", "flarum-tsconfig": "^1.0.2",
"flarum-webpack-config": "^2.0.0", "flarum-webpack-config": "^1.0.0",
"webpack": "^5.75.0", "pusher-js": "^7.0.3",
"webpack-cli": "^5.0.0" "webpack": "^4.46.0",
"webpack-cli": "^4.9.1"
}, },
"scripts": { "scripts": {
"dev": "webpack --mode development --watch", "dev": "webpack --mode development --watch",
@ -17,6 +19,6 @@
"format-check": "prettier --check src" "format-check": "prettier --check src"
}, },
"devDependencies": { "devDependencies": {
"prettier": "^2.7.1" "prettier": "^2.4.1"
} }
} }

View File

@ -1,122 +1,37 @@
import app from 'flarum/admin/app'; import app from 'flarum/admin/app';
import { extend } from 'flarum/common/extend';
import DashboardPage from 'flarum/admin/components/DashboardPage';
import DashboardWidget from 'flarum/admin/components/DashboardWidget';
import Alert from 'flarum/common/components/Alert';
const REQUIRED_INDEX_COMPAT = 'v2';
class ReindexWarningWidget extends DashboardWidget {
className() {
return 'ReindexWarningWidget';
}
content() {
return m(
Alert,
{
type: 'warning',
dismissible: false,
icon: 'fas fa-exclamation-triangle',
title: app.translator.trans('blomstra-search.admin.reindex-required.title'),
},
app.translator.trans('blomstra-search.admin.reindex-required.detail')
);
}
}
app.initializers.add('blomstra-search', () => { app.initializers.add('blomstra-search', () => {
const activeIndex = app.data.settings['blomstra-search.active-index']; const languages = new Map;
const compatVersion = app.data.settings['blomstra-search.index-compatible']; ['arabic', 'armenian', 'basque', 'bengali', 'brazilian', 'bulgarian', 'catalan', 'cjk', 'czech',
'danish', 'dutch', 'english', 'estonian', 'finnish', 'french', 'galician', 'german', 'greek',
if (activeIndex && compatVersion !== REQUIRED_INDEX_COMPAT) { 'hindi', 'hungarian', 'indonesian', 'irish', 'italian', 'latvian', 'lithuanian', 'norwegian',
extend(DashboardPage.prototype, 'availableWidgets', function (items) { 'persian', 'portuguese', 'romanian', 'russian', 'sorani', 'spanish', 'swedish', 'turkish', 'thai']
items.add('blomstra-search-reindex', m(ReindexWarningWidget), 110); .forEach(language => {
});
}
const languages = new Map();
[
'arabic',
'armenian',
'basque',
'bengali',
'brazilian',
'bulgarian',
'catalan',
'cjk',
'czech',
'danish',
'dutch',
'english',
'estonian',
'finnish',
'french',
'galician',
'german',
'greek',
'hindi',
'hungarian',
'indonesian',
'irish',
'italian',
'latvian',
'lithuanian',
'norwegian',
'persian',
'portuguese',
'romanian',
'russian',
'sorani',
'spanish',
'swedish',
'turkish',
'thai',
].forEach((language) => {
languages.set(language, language); languages.set(language, language);
}); });
app.extensionData app.extensionData
.for('blomstra-search') .for('blomstra-search')
.registerSetting(function (this: any) {
// No index yet, or first build predates this tracking — stay silent.
const indexedAnalyzer = app.data.settings['blomstra-search.indexed-analyzer'];
if (!activeIndex || !indexedAnalyzer) return null;
const currentAnalyzer = this.setting('blomstra-search.analyzer-language')() || 'english';
const currentMinLength = this.setting('blomstra-search.min-search-length')();
const indexedMinLength = String(
app.data.settings['blomstra-search.indexed-min-search-length'] || app.data.settings['blomstra-search.min-search-length']
);
if (currentAnalyzer === indexedAnalyzer && currentMinLength === indexedMinLength) return null;
return m(
Alert,
{ type: 'warning', dismissible: false, icon: 'fas fa-exclamation-triangle' },
app.translator.trans('blomstra-search.admin.index-settings-changed')
);
})
.registerSetting({ .registerSetting({
setting: 'blomstra-search.elastic-endpoint', setting: 'blomstra-search.elastic-endpoint',
label: app.translator.trans('blomstra-search.admin.elastic-endpoint'), label: app.translator.trans('blomstra-search.admin.elastic-endpoint'),
type: 'input', type: 'input'
}) })
.registerSetting({ .registerSetting({
setting: 'blomstra-search.elastic-username', setting: 'blomstra-search.elastic-username',
label: app.translator.trans('blomstra-search.admin.elastic-username'), label: app.translator.trans('blomstra-search.admin.elastic-username'),
type: 'input', type: 'input'
}) })
.registerSetting({ .registerSetting({
setting: 'blomstra-search.elastic-password', setting: 'blomstra-search.elastic-password',
label: app.translator.trans('blomstra-search.admin.elastic-password'), label: app.translator.trans('blomstra-search.admin.elastic-password'),
type: 'password', type: 'password'
}) })
.registerSetting({ .registerSetting({
setting: 'blomstra-search.elastic-index', setting: 'blomstra-search.elastic-index',
label: app.translator.trans('blomstra-search.admin.elastic-index'), label: app.translator.trans('blomstra-search.admin.elastic-index'),
default: 'flarum', default: 'flarum',
type: 'input', type: 'input'
}) })
.registerSetting({ .registerSetting({
setting: 'blomstra-search.analyzer-language', setting: 'blomstra-search.analyzer-language',
@ -126,22 +41,4 @@ app.initializers.add('blomstra-search', () => {
options: Object.fromEntries(languages.entries()), options: Object.fromEntries(languages.entries()),
default: 'english', default: 'english',
}) })
.registerSetting({
setting: 'blomstra-search.search-discussion-subjects',
label: app.translator.trans('blomstra-search.admin.search-discussion-subjects'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.search-post-bodies',
label: app.translator.trans('blomstra-search.admin.search-post-bodies'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.min-search-length',
label: app.translator.trans('blomstra-search.admin.min-search-length.label'),
help: app.translator.trans('blomstra-search.admin.min-search-length.help'),
type: 'select',
options: { '1': '1', '2': '2', '3': '3', '4': '4' },
default: app.data.settings['blomstra-search.min-search-length'],
});
}); });

View File

@ -6,10 +6,9 @@ import DiscussionListState from 'flarum/forum/states/DiscussionListState';
export default function extendDiscussionState() { export default function extendDiscussionState() {
override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) { override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) {
const preloaded = app.data.apiDocument || null; console.log(this.requestParams());
// If existing payload is given or no search is made, fallback on native page. if (!this.requestParams()?.filter?.q) return original.call(this, page);
if (preloaded || !this.requestParams()?.filter?.q) return original.call(this, page);
const params = this.requestParams(); const params = this.requestParams();
params.page = { params.page = {
@ -28,6 +27,8 @@ export default function extendDiscussionState() {
const results = await app.request({ params, url, method: 'GET' }); const results = await app.request({ params, url, method: 'GET' });
// Parse API response into models and push to store // Parse API response into models and push to store
return app.store.pushPayload(results); const models = app.store.pushPayload(results);
return models;
}); });
} }

View File

@ -44,7 +44,7 @@ export default class DiscussionsSearchSource implements SearchSource {
this.results.set(query, models); this.results.set(query, models);
} }
view(query: string): Array<Mithril.Vnode> { view(query: string): Mithril.Children {
query = query.toLowerCase(); query = query.toLowerCase();
// Get results from map // Get results from map
@ -54,7 +54,7 @@ export default class DiscussionsSearchSource implements SearchSource {
const mostRelevantPost = discussion.mostRelevantPost(); const mostRelevantPost = discussion.mostRelevantPost();
return ( return (
<li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`}> <li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`} key={`${this.type}${discussion.id()}`}>
<Link href={app.route.discussion(discussion, mostRelevantPost && mostRelevantPost.number())}> <Link href={app.route.discussion(discussion, mostRelevantPost && mostRelevantPost.number())}>
<div className="DiscussionSearchResult-title">{highlight(discussion.title(), query)}</div> <div className="DiscussionSearchResult-title">{highlight(discussion.title(), query)}</div>
{!!mostRelevantPost && <div className="DiscussionSearchResult-excerpt">{highlight(mostRelevantPost.contentPlain(), query, 100)}</div>} {!!mostRelevantPost && <div className="DiscussionSearchResult-excerpt">{highlight(mostRelevantPost.contentPlain(), query, 100)}</div>}
@ -63,14 +63,16 @@ export default class DiscussionsSearchSource implements SearchSource {
); );
}); });
return [ return (
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>, <>
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>
<li> <li>
<LinkButton icon="fas fa-search" href={app.route('index', { q: query })}> <LinkButton icon="fas fa-search" href={app.route('index', { q: query })}>
{app.translator.trans('core.forum.search.all_discussions_button', { query })} {app.translator.trans('core.forum.search.all_discussions_button', { query })}
</LinkButton> </LinkButton>
</li>, </li>
...results, {results}
]; </>
);
} }
} }

View File

@ -1,6 +1,6 @@
import app from 'flarum/forum/app'; import app from 'flarum/forum/app';
import Search, { SearchAttrs, SearchSource } from 'flarum/forum/components/Search'; import Search from 'flarum/forum/components/Search';
import { extend } from 'flarum/common/extend'; import { extend } from 'flarum/common/extend';
import ItemList from 'flarum/common/utils/ItemList'; import ItemList from 'flarum/common/utils/ItemList';
@ -9,24 +9,14 @@ import DiscussionsSearchSource from './SearchSources/DiscussionsSearchSource';
import extendDiscussionState from './PaginatedListStates/extendDiscussionState'; import extendDiscussionState from './PaginatedListStates/extendDiscussionState';
app.initializers.add('blomstra-search', () => { app.initializers.add('blomstra-search', () => {
const minLength = parseInt(app.data.settings['blomstra-search.min-search-length'] || String(Search.MIN_SEARCH_LEN), 10); extend(Search.prototype, 'sourceItems', function (this: Search, items: ItemList) {
if (minLength !== Search.MIN_SEARCH_LEN) { items.remove('users');
// Flarum provides no extension point for MIN_SEARCH_LEN, so we overwrite the items.remove('discussions');
// static property directly. TypeScript `readonly` is compile-time only — at
// runtime this is a plain property assignment and is safe as long as no code
// reads MIN_SEARCH_LEN before this initializer runs.
(Search as any).MIN_SEARCH_LEN = minLength;
}
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) { /* if (app.forum.attribute('canViewForum')) */ items.add('discussions', new DiscussionsSearchSource());
items.replace('discussions', new DiscussionsSearchSource());
// console.log(items);
}); });
});
app.initializers.add(
'blomstra-search-early',
() => {
extendDiscussionState(); extendDiscussionState();
}, });
999999
);

View File

@ -1,3 +0,0 @@
.ReindexWarningWidget {
padding: 0;
}

View File

@ -8,18 +8,4 @@ blomstra-search:
label: Analyzer language label: Analyzer language
help: | help: |
The analyzer makes search understand stop words and undertakes language The analyzer makes search understand stop words and undertakes language
specific improvements for indexing. Changing this requires a full index rebuild to take effect. specific improvements for indexing.
index-settings-changed: "Search index settings have changed. Run: php flarum blomstra:search:index build"
reindex-required:
title: Search index rebuild required
detail: "The search index is not compatible with this version of the extension. Run: php flarum blomstra:search:index build"
search-discussion-subjects: Search inside discussion titles
search-post-bodies: Search inside comments
match-sentences: Match search term against full sentence
match-words: Match search term against full words
min-search-length:
label: Minimum search query length
help: |
Minimum number of characters required before a search is triggered. Lower this to 1 or 2
for CJK (Chinese, Japanese, Korean) communities where single characters carry full meaning.
Changing this requires a full index rebuild to take effect.

View File

@ -1,28 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Api;
use Illuminate\Support\Arr;
use Psr\Http\Message\ResponseInterface;
class Client extends \Flarum\Api\Client
{
public function get(string $path): ResponseInterface
{
if ($path === '/discussions' && Arr::has($this->queryParams, 'filter.q')) {
return parent::get('/blomstra/search/discussions');
}
return parent::get($path);
}
}

View File

@ -1,15 +1,5 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Api\Controllers; namespace Blomstra\Search\Api\Controllers;
use Blomstra\Search\Commands\BuildCommand; use Blomstra\Search\Commands\BuildCommand;
@ -33,8 +23,7 @@ class IndexController implements RequestHandlerInterface
$command->run( $command->run(
new ArrayInput([]), new ArrayInput([]),
new ConsoleOutput() new ConsoleOutput);
);
return new EmptyResponse(); return new EmptyResponse();
} }

View File

@ -1,24 +1,9 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Api\Controllers; namespace Blomstra\Search\Api\Controllers;
use Blomstra\Search\Elasticsearch\HasChildQuery; use Blomstra\Search\Save\Document as ElasticDocument;
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
use Blomstra\Search\Elasticsearch\MatchQuery;
use Blomstra\Search\Elasticsearch\TermsQuery; use Blomstra\Search\Elasticsearch\TermsQuery;
use Blomstra\Search\Searchers\CommentPostSearcher;
use Blomstra\Search\Searchers\DiscussionSearcher;
use Blomstra\Search\Searchers\Searcher;
use Elasticsearch\Client; use Elasticsearch\Client;
use Flarum\Api\Controller\ListDiscussionsController; use Flarum\Api\Controller\ListDiscussionsController;
use Flarum\Api\Serializer\DiscussionSerializer; use Flarum\Api\Serializer\DiscussionSerializer;
@ -26,18 +11,15 @@ use Flarum\Discussion\Discussion;
use Flarum\Extension\ExtensionManager; use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group; use Flarum\Group\Group;
use Flarum\Http\RequestUtil; use Flarum\Http\RequestUtil;
use Flarum\Http\UrlGenerator;
use Flarum\Settings\SettingsRepositoryInterface;
use Flarum\Tags\Tag;
use Flarum\User\User; use Flarum\User\User;
use Illuminate\Contracts\Container\Container; use Illuminate\Contracts\Container\Container;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Arr; use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Illuminate\Support\Str; use Illuminate\Support\Str;
use Psr\Http\Message\ServerRequestInterface; use Psr\Http\Message\ServerRequestInterface;
use Psr\Log\LoggerInterface;
use Spatie\ElasticsearchQueryBuilder\Builder; use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery; use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery; use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort; use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
use Tobscure\JsonApi\Document; use Tobscure\JsonApi\Document;
@ -48,133 +30,88 @@ class SearchController extends ListDiscussionsController
protected array $translateSort = [ protected array $translateSort = [
'lastPostedAt' => 'updated_at', 'lastPostedAt' => 'updated_at',
'createdAt' => 'created_at', 'createdAt' => 'created_at'
'commentCount' => 'comment_count',
'view_count' => 'view_count',
]; ];
protected bool $matchSentences; public function __construct(protected Client $elastic)
protected bool $matchWords; {}
protected ?Searcher $discussionSearcher;
protected ?Searcher $postSearcher;
public function __construct(protected Client $elastic, protected UrlGenerator $uri, Container $container, SettingsRepositoryInterface $settings)
{
$this->matchSentences = true;
$this->matchWords = true;
$searchers = collect($container->tagged('blomstra.search.searchers'));
$this->discussionSearcher = $searchers->first(fn ($s) => $s instanceof DiscussionSearcher);
$this->postSearcher = $searchers->first(fn ($s) => $s instanceof CommentPostSearcher);
}
protected function data(ServerRequestInterface $request, Document $document) protected function data(ServerRequestInterface $request, Document $document)
{ {
$type = Arr::get($request->getQueryParams(), 'type');
$actor = RequestUtil::getActor($request); $actor = RequestUtil::getActor($request);
$filters = $this->extractFilter($request); $filters = $this->extractFilter($request);
$search = $this->getSearch($filters);
$limit = $this->extractLimit($request);
$offset = $this->extractOffset($request);
$include = array_merge($this->extractInclude($request), ['state']); $include = array_merge($this->extractInclude($request), ['state']);
$query = BoolQuery::create() $filterQuery = (BoolQuery::create())
// Always restrict to discussion documents; posts are only searched via has_child. ->add(
->add(TermQuery::create('join_field', 'discussion'), 'filter'); BoolQuery::create()
->add(MatchQuery::create('content', $filters['q']))
if (!empty($search)) { );
$query->add($this->buildTextQuery($search, $actor));
}
$this->addFilters($query, $actor, $filters);
$builder = (new Builder($this->elastic)) $builder = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index')) ->index(resolve('blomstra.search.elastic_index'))
->size($limit + 1) ->size($this->extractLimit($request))
->from($offset) ->from($this->extractOffset($request))
->addQuery($query); ->addQuery(
$this->addFilters($filterQuery, $actor)
$knownSortFields = array_merge(array_values($this->translateSort), ['rawId']); );
$logger = resolve(LoggerInterface::class);
$phpSortField = null;
$phpSortDir = 'desc';
foreach ($this->extractSort($request) as $field => $direction) { foreach ($this->extractSort($request) as $field => $direction) {
$translated = $this->translateSort[$field] ?? $field; $field = $this->translateSort[$field] ?? $field;
$builder->addSort(new Sort($field, $direction));
if (!in_array($translated, $knownSortFields)) {
$logger->warning("blomstra/search: unknown sort field \"{$field}\", ignoring.");
continue;
}
$builder->addSort(new Sort($translated, $direction));
if ($phpSortField === null && $translated !== 'rawId') {
$phpSortField = $translated;
$phpSortDir = $direction;
}
} }
$response = $builder->search(); $response = $builder->search();
Discussion::setStateUser($actor); Discussion::setStateUser($actor);
// Eager load groups for use in the policies (isAdmin check)
if (in_array('mostRelevantPost.user', $include)) { if (in_array('mostRelevantPost.user', $include)) {
$include[] = 'mostRelevantPost.user.groups'; $include[] = 'mostRelevantPost.user.groups';
if (!in_array('mostRelevantPost', $include)) { // If the first level of the relationship wasn't explicitly included,
// add it so the code below can look for it
if (! in_array('mostRelevantPost', $include)) {
$include[] = 'mostRelevantPost'; $include[] = 'mostRelevantPost';
} }
} }
// All hits are discussion documents. Extract the best-matching post ID from // we need to retrieve all discussion ids and when the results are posts,
// inner_hits when a has_child clause matched (i.e. the match came from a post). // their ids as most relevant post id
$results = Collection::make(Arr::get($response, 'hits.hits')) $results = Collection::make(Arr::get($response, 'hits.hits'))
->map(function ($hit) { ->map(function ($hit) {
// _id is "discussions:123" — parse the numeric part directly. $id = Str::after($hit['_source']['id'], ':');
$discussionId = Str::after($hit['_id'], 'discussions:'); $type = $hit['_source']['type'];
// rawId on the inner hit gives us the integer post ID.
$bestPostId = Arr::get($hit, 'inner_hits.best_post.hits.hits.0._source.rawId');
if ($type === 'posts') {
return [ return [
'discussion_id' => $discussionId, 'most_relevant_post_id' => $id,
'most_relevant_post_id' => $bestPostId,
'weight' => Arr::get($hit, 'sort.0', Arr::get($hit, '_score', 0)),
]; ];
} else {
return [
'discussion_id' => $id
];
}
}); });
$document->addPaginationLinks(
$this->uri->to('api')->route('blomstra.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
$results->count() > $limit ? null : 0
);
$results = $results->take($limit);
$discussions = Discussion::query() $discussions = Discussion::query()
->when( ->whereIn('id', $results->pluck('discussion_id'))
$actor->isGuest() || !$actor->hasPermission('discussion.hide'), ->orWhereHas('posts', function ($query) use ($results) {
fn ($q) => $q->whereNull('hidden_at') $query->whereIn('id', $results->pluck('most_relevant_post_id'));
) })
->whereIn('id', $results->pluck('discussion_id')->filter())
->get() ->get()
->each(function (Discussion $discussion) use ($results) { ->each(function (Discussion $discussion) use ($results) {
$result = $results->firstWhere('discussion_id', $discussion->id); if (in_array($discussion->id, $results->pluck('discussion_id')->toArray())) {
$discussion->most_relevant_post_id = $discussion->first_post_id;
$discussion->most_relevant_post_id = $result['most_relevant_post_id'] } else {
?? $discussion->first_post_id; $post = $discussion->posts()->whereIn('id', $results->pluck('most_relevant_post_id'))->first();
$discussion->weight = $result['weight'] ?? 0; $discussion->most_relevant_post_id = $post?->id ?? $discussion->first_post_id;
}
}) })
->keyBy('id') ->keyBy('id')
->when(
$phpSortField,
fn ($c) => $phpSortDir === 'desc' ? $c->sortByDesc($phpSortField) : $c->sortBy($phpSortField),
fn ($c) => $c->sortByDesc('weight')
)
->unique(); ->unique();
$this->loadRelations($discussions, $include); $this->loadRelations($discussions, $include);
@ -192,123 +129,56 @@ class SearchController extends ListDiscussionsController
return $discussions; return $discussions;
} }
/** protected function getDocument(string $type): ?ElasticDocument
* Build the text-matching portion of the query.
*
* Discussion titles are matched directly (filtered to join_field=discussion).
* Post bodies are matched via has_child with score_mode=sum so discussions
* with many matching posts score higher than those with a single strong match.
* inner_hits returns the best-scoring post for use as mostRelevantPost.
* Hidden posts are only included in matching for users with post.hide permission.
*/
protected function buildTextQuery(string $search, User $actor): BoolQuery
{ {
$textQuery = BoolQuery::create(); $documents = resolve(Container::class)->tagged('blomstra.search.documents');
if ($this->discussionSearcher?->enabled()) { return collect($documents)->first(function (ElasticDocument $document) use ($type) {
$textQuery->add($this->buildShouldClauses($search, $this->discussionSearcher->boost()), 'should'); return $document->type() === $type;
} });
if ($this->postSearcher?->enabled()) {
$postQuery = $this->buildShouldClauses($search, $this->postSearcher->boost());
// Guests and non-moderators may not see hidden posts; exclude them from child matching.
if ($actor->isGuest() || !$actor->hasPermission('post.hide')) {
$postQuery->add(TermQuery::create('is_hidden', false), 'filter');
}
$textQuery->add(
HasChildQuery::create('post', $postQuery)->withInnerHits(),
'should'
);
}
return $textQuery;
}
protected function buildShouldClauses(string $search, float $boost): BoolQuery
{
$should = BoolQuery::create();
if ($this->matchSentences) {
$should->add((new MatchPhraseQuery('content', $search))->boost(2 * $boost), 'should');
}
if ($this->matchWords) {
$should->add((new MatchQuery('content', $search))->operator('and')->boost(1.8 * $boost), 'should');
$should->add((new MatchQuery('content', $search))->operator('or')->boost(0.8 * $boost), 'should');
}
return $should;
} }
protected function extensionEnabled(string $extension): bool protected function extensionEnabled(string $extension): bool
{ {
return resolve(ExtensionManager::class)->isEnabled($extension); /** @var ExtensionManager $manager */
$manager = resolve(ExtensionManager::class);
return $manager->isEnabled($extension);
} }
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): void protected function addFilters(BoolQuery $query, User $actor): BoolQuery
{ {
$groups = $this->getGroups($actor); /** @var Collection $groups */
$onlyPrivate = Str::contains($filters['q'] ?? '', 'is:private'); $groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) $groups->add(Group::MEMBER_ID);
$subQuery = BoolQuery::create() $subQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'false')) ->add(TermQuery::create('is_private', 'false'))
->add(TermsQuery::create('groups', $groups->toArray())); ->add(TermsQuery::create('groups', $groups->toArray()));
if ($this->extensionEnabled('flarum-tags') && !empty($filters['tag'])) {
$slugs = is_array($filters['tag']) ? $filters['tag'] : explode(',', $filters['tag']);
$tagIds = Tag::query()->whereIn('slug', $slugs)->pluck('id')->toArray();
if (!empty($tagIds)) {
$query->add(TermsQuery::create('tags', $tagIds), 'filter');
}
}
if ($this->extensionEnabled('fof-byobu') && $actor->exists) { if ($this->extensionEnabled('fof-byobu') && $actor->exists) {
$byobuQuery = BoolQuery::create() $byobuQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'true')) ->add(TermQuery::create('is_private', 'true'), 'should')
->add( ->add(
BoolQuery::create() BoolQuery::create()
->add(TermsQuery::create('recipient_groups', $groups->toArray()), 'should') ->add(TermsQuery::create('recipient-groups', $groups->toArray()))
->add(TermsQuery::create('recipient_users', [$actor->id]), 'should'), ->add(TermQuery::create('recipient-users', $actor->id)),
'should'
); );
if ($onlyPrivate) {
$subQuery = $byobuQuery;
} else {
$subQuery = BoolQuery::create() $subQuery = BoolQuery::create()
->add($subQuery, 'should') ->add($subQuery, 'should')
->add($byobuQuery, 'should'); ->add($byobuQuery, 'should');
} }
}
$query->add($subQuery, 'filter'); $query->add(
} $subQuery,
'filter'
);
protected function getGroups(User $actor): Collection return $query;
{
$groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) {
$groups->add(Group::MEMBER_ID);
}
return $groups;
}
protected function getSearch(array $filters): ?string
{
$search = Arr::get($filters, 'q');
if ($search) {
$q = collect(explode(' ', $search))
->filter(fn (string $part) => $part !== 'is:private')
->filter()
->join(' ');
return empty($q) ? null : $q;
}
return null;
} }
} }

View File

@ -1,19 +1,9 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Commands; namespace Blomstra\Search\Commands;
use Blomstra\Search\Jobs\Job; use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\UpdateSearchJob; use Blomstra\Search\Jobs\SavingJob;
use Blomstra\Search\Seeders\Seeder; use Blomstra\Search\Seeders\Seeder;
use Elasticsearch\Client; use Elasticsearch\Client;
use Flarum\Settings\SettingsRepositoryInterface; use Flarum\Settings\SettingsRepositoryInterface;
@ -21,563 +11,84 @@ use Illuminate\Console\Command;
use Illuminate\Contracts\Container\Container; use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Queue\Queue; use Illuminate\Contracts\Queue\Queue;
use Illuminate\Database\Eloquent\Collection; use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Arr;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\RangeQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
class BuildCommand extends Command class BuildCommand extends Command
{ {
/**
* Bump this when a mapping change requires a full reindex.
* Written into the mapping's _meta.index_compat_version on every build and
* read back into blomstra-search.index-compatible by saveIndexedConfig so the
* value always reflects the live index including after a promote/rollback.
*/
public const INDEX_COMPAT_VERSION = 'v2';
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
protected $signature = 'blomstra:search:index protected $signature = 'blomstra:search:index
{action? : build | promote | rollback | discard | mapping | fill} {--max-id= : Limits for each object the number of items to seed}';
{--fresh} protected $description = 'Rebuilds the complete search server with its documents.';
{--resume}
{--staging}
{--keep-backup}
{--pending}
{--backup}
{--only=}
{--max-id=}
{--throttle=}
{--i-am-sure}';
protected $description = 'Build and manage the Elasticsearch search index.'; public function handle(Container $container)
protected $help = <<<'HELP'
<comment>Actions and their options:</comment>
<info>build</info> Queue documents and promote automatically.
<comment>--resume</comment> Resume an interrupted build from where it left off
<comment>--fresh</comment> Drop the staging index and start completely fresh
<comment>--staging</comment> Keep in staging requires explicit <info>promote</info> (blue-green)
<comment>--keep-backup</comment> Retain the replaced index for rollback
<info>promote</info> Swap alias to staging index (blue-green workflow).
<comment>--keep-backup</comment> Retain the replaced index for rollback
<comment>--i-am-sure</comment> Skip the confirmation prompt
<info>rollback</info> Restore the backup index to live.
<info>discard</info> Drop an index without promoting.
<comment>--pending</comment> Drop the staging index (cancel a build)
<comment>--backup</comment> Drop the backup index (cleanup after --keep-backup)
<info>mapping</info> Push updated mapping to the live index only.
<info>fill</info> Seed only documents missing from the live index.
<comment>Shared seeding options (build / fill):</comment>
<comment>--only=TYPE</comment> Seed only this type: <info>discussions</info> or <info>posts</info>
<comment>--throttle=N</comment> Seconds to wait between batches
<comment>--max-id=N</comment> Limit seeding to IDs up to this value
HELP;
public function handle(Container $container): void
{ {
$index = $container->make('blomstra.search.elastic_index');
/** @var array $seeders */
$seeders = $container->tagged('blomstra.search.seeders');
/** @var Queue $queue */
$queue = $container->make(Queue::class);
/** @var Client $client */ /** @var Client $client */
$client = $container->make(Client::class); $client = $container->make(Client::class);
/** @var SettingsRepositoryInterface $settings */ /** @var SettingsRepositoryInterface $settings */
$settings = $container->make(SettingsRepositoryInterface::class); $settings = $container->make(SettingsRepositoryInterface::class);
/** @var string $alias */ $properties = [
$alias = $container->make('blomstra.search.elastic_index');
if (!$this->argument('action')) {
$this->call('help', ['command_name' => $this->getName()]);
return;
}
switch ($this->argument('action')) {
case 'build':
$this->runBuild($client, $alias, $settings, $container);
break;
case 'promote':
$this->runPromote($client, $alias, $settings);
break;
case 'rollback':
$this->runRollback($client, $alias, $settings);
break;
case 'discard':
$this->runDiscard($client, $settings);
break;
case 'mapping':
$client->indices()->putMapping([
'index' => $alias,
'body' => $this->mappingProperties(),
]);
$this->info('Mapping updated on live index.');
break;
case 'fill':
$this->runSeeders(
collect($container->tagged('blomstra.search.seeders')),
$container->make(Queue::class),
$client,
$settings,
$alias,
seedMissing: true
);
break;
default:
$this->error("Unknown action '{$this->argument('action')}'. Valid actions: build, promote, rollback, discard, mapping, fill.");
}
}
protected function runBuild(Client $client, string $alias, SettingsRepositoryInterface $settings, Container $container): void
{
/** @var Seeder[] $seeders */
$seeders = collect($container->tagged('blomstra.search.seeders'));
// Guard: staging build exists without explicit intent given.
$staging = $settings->get('blomstra-search.staging-index');
if ($staging && $client->indices()->exists(['index' => $staging])
&& !$this->option('resume') && !$this->option('fresh')
) {
$this->warn("An in-progress build already exists: $staging");
$this->line('');
$this->line('Choose one of:');
$this->line(' blomstra:search:index build --resume Continue from where it left off');
$this->line(' blomstra:search:index build --fresh Drop this build and start completely fresh');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if (!$aliasExists && !$indexExists) {
// First install: create a timestamped index, alias it immediately, seed live.
$targetIndex = $this->prepareFirstInstall($client, $alias, $settings, $seeders);
$stagingBuild = false;
} else {
// Live system: build into a staging index without touching the live alias.
$targetIndex = $this->prepareStagingIndex($client, $alias, $settings, $seeders);
$stagingBuild = true;
}
$client->indices()->putMapping([
'index' => $targetIndex,
'body' => $this->mappingProperties(),
]);
$this->runSeeders($seeders, $container->make(Queue::class), $client, $settings, $targetIndex);
if ($stagingBuild) {
if ($this->option('staging')) {
$staging = $settings->get('blomstra-search.staging-index');
$this->info("Staging build complete. Drain the queue, then promote '$staging' to live:");
$this->line(' php flarum queue:work --stop-when-empty');
$this->line(' php flarum blomstra:search:index promote');
} else {
$this->info('Jobs queued. Promoting now (search improves as the queue drains)...');
$this->performPromote($client, $alias, $settings);
}
}
}
protected function runPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
if (!$this->option('i-am-sure')) {
$this->warn('promote switches the live index. Ensure all queued jobs have finished first:');
$this->line(' php flarum queue:work --stop-when-empty');
$this->line('');
if (!$this->confirm('Queue drained? Proceed?')) {
return;
}
}
$this->performPromote($client, $alias, $settings);
}
protected function performPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging || !$client->indices()->exists(['index' => $staging])) {
$this->error('No staging index ready to promote. Run: blomstra:search:index build');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if ($aliasExists) {
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $staging, 'alias' => $alias]],
]],
]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
if ($this->option('keep-backup')) {
$settings->set('blomstra-search.backup-index', $activeIndex);
$this->info("Kept '$activeIndex' as backup — run rollback to revert, discard --backup to drop it.");
} else {
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Deleted old index: $activeIndex");
}
} else {
// One-time migration: concrete index → alias (legacy installs only).
if ($indexExists) {
$client->indices()->delete(['index' => $alias]);
$this->info("Deleted legacy concrete index: $alias");
}
$client->indices()->putAlias(['index' => $staging, 'name' => $alias]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
}
$settings->set('blomstra-search.active-index', $staging);
$settings->set('blomstra-search.staging-index', null);
$this->saveIndexedConfig($client, $settings, $staging);
}
protected function runRollback(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$backup = $settings->get('blomstra-search.backup-index');
if (!$backup || !$client->indices()->exists(['index' => $backup])) {
$this->error('No backup index to roll back to. Was promote run with --keep-backup?');
return;
}
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $backup, 'alias' => $alias]],
]],
]);
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Rolled back: '$alias' now points to '$backup'. Deleted '$activeIndex'.");
$settings->set('blomstra-search.active-index', $backup);
$settings->set('blomstra-search.backup-index', null);
$this->saveIndexedConfig($client, $settings, $backup);
}
protected function runDiscard(Client $client, SettingsRepositoryInterface $settings): void
{
$pending = $this->option('pending');
$backup = $this->option('backup');
if (!$pending && !$backup) {
$this->error('Specify what to discard: --pending (staging build) or --backup (rollback index).');
return;
}
if ($pending) {
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging) {
$this->warn('No staging index to discard.');
} else {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
}
$settings->set('blomstra-search.staging-index', null);
$this->info("Discarded staging index: $staging");
}
}
if ($backup) {
$backupIndex = $settings->get('blomstra-search.backup-index');
if (!$backupIndex) {
$this->warn('No backup index to discard.');
} else {
if ($client->indices()->exists(['index' => $backupIndex])) {
$client->indices()->delete(['index' => $backupIndex]);
}
$settings->set('blomstra-search.backup-index', null);
$this->info("Discarded backup index: $backupIndex");
}
}
}
protected function runSeeders(
iterable $seeders,
Queue $queue,
Client $client,
SettingsRepositoryInterface $settings,
string $targetIndex,
bool $seedMissing = false
): void {
$only = $this->option('only');
/** @var Seeder $seeder */
foreach ($seeders as $seeder) {
if ($only && $seeder->type() !== $only) {
continue;
}
$total = 0;
if ($this->option('resume')) {
$saved = $this->getContinueAt($settings, $seeder->type());
if ($saved === 0) {
$this->info("Seeder '{$seeder->type()}' already completed in a previous run, skipping.");
continue;
}
$continueAt = $saved ?? $seeder->query()->max('id');
} else {
$continueAt = $seeder->query()->max('id');
}
$seeded = null;
while ($continueAt !== null) {
$rangeFrom = max(1, $continueAt - 2500);
$rangeTo = $continueAt;
if ($seedMissing) {
$response = (new Builder($client))
->index($targetIndex)
->size(2500)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
->add(TermQuery::create('join_field', $seeder->joinRelation()))
)
->search();
$seeded = Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
}
/** @var Collection $collection */
$collection = $seeder->query()
->latest('id')
->whereBetween('id', [$rangeFrom, $rangeTo])
->when($this->option('max-id'), fn ($q, $id) => $q->where('id', '<=', $id))
->when($seeded, fn ($q, $seeded) => $q->whereNotIn('id', $seeded))
->get();
$min = $collection->min('id');
if ($seedMissing && $collection->isEmpty()) {
$continueAt = $rangeFrom > 2 ? $rangeFrom - 1 : null;
} else {
$continueAt = $min && $min > 2 ? $min - 1 : null;
}
if ($collection->isNotEmpty()) {
$queue->pushOn(Job::$onQueue, new UpdateSearchJob($collection, $seeder, $targetIndex));
}
$this->info("IDs {$rangeFrom}{$rangeTo} | type: {$seeder->type()} | queued: {$collection->count()}.");
$total += $collection->count();
$this->setContinueAt($settings, $seeder->type(), $continueAt);
if ($throttle = $this->option('throttle')) {
$this->info("Throttling for $throttle seconds");
sleep($throttle);
}
}
$this->setContinueAt($settings, $seeder->type(), 0);
$this->info("Queued a total of $total {$seeder->type()} for indexing.");
}
}
/**
* First install: create a timestamped concrete index, alias the configured name to it
* immediately, and return the alias so seeding goes through it and is live from the start.
*/
protected function prepareFirstInstall(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$concrete = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $concrete,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$client->indices()->putAlias(['index' => $concrete, 'name' => $alias]);
$settings->set('blomstra-search.active-index', $concrete);
$this->saveIndexedConfig($client, $settings, $concrete);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created '$concrete', aliased '$alias' → '$concrete'.");
$this->info("Index is live — documents become searchable as the queue processes.");
return $alias;
}
/**
* Prepare the staging index for a blue-green build.
*
* - If a staging build exists and --fresh is not set, resume it.
* - Otherwise create a fresh timestamped index and save it as staging.
*/
protected function prepareStagingIndex(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$staging = $settings->get('blomstra-search.staging-index');
if ($this->option('fresh') && $staging) {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
$this->info("Dropped staging index: $staging");
}
$staging = null;
$settings->set('blomstra-search.staging-index', null);
}
if ($staging && $client->indices()->exists(['index' => $staging])) {
$this->info("Resuming staging index build: $staging");
return $staging;
}
$staging = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $staging,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$settings->set('blomstra-search.staging-index', $staging);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created staging index: $staging");
return $staging;
}
/**
* Persist the analysis config and compat version that are actually live in ES for the
* given index. Reading from ES (rather than from Flarum settings) means rollbacks are
* also covered: the stored values always reflect the index that is currently aliased,
* not the settings at the time the command ran. Indexes built before _meta tracking
* existed will yield a null compat version, which correctly triggers the reindex warning.
*/
protected function saveIndexedConfig(Client $client, SettingsRepositoryInterface $settings, string $indexName): void
{
$settingsResponse = $client->indices()->getSettings(['index' => $indexName]);
$analysis = Arr::get($settingsResponse, "$indexName.settings.index.analysis", []);
$analyzer = Arr::get($analysis, 'analyzer.flarum_analyzer.type', 'english');
$minGram = (int) Arr::get($analysis, 'filter.partial_search_filter.min_gram', self::DEFAULT_MIN_SEARCH_LENGTH);
$mappingResponse = $client->indices()->getMapping(['index' => $indexName]);
$compatVersion = Arr::get($mappingResponse, "$indexName.mappings._meta.index_compat_version");
$settings->set('blomstra-search.indexed-analyzer', $analyzer);
$settings->set('blomstra-search.indexed-min-search-length', $minGram);
$settings->set('blomstra-search.index-compatible', $compatVersion);
}
protected function buildIndexSettings(SettingsRepositoryInterface $settings): array
{
$language = $settings->get('blomstra-search.analyzer-language') ?: 'english';
$minGram = max(1, (int) ($settings->get('blomstra-search.min-search-length') ?: self::DEFAULT_MIN_SEARCH_LENGTH));
$maxGram = 10;
if ($minGram >= $maxGram) {
$this->error("min_gram ($minGram) must be less than max_gram ($maxGram). Using default.");
$minGram = self::DEFAULT_MIN_SEARCH_LENGTH;
}
return [
'index.max_ngram_diff' => $maxGram - $minGram,
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $language,
],
'flarum_analyzer_partial' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase', 'partial_search_filter'],
],
],
'filter' => [
'partial_search_filter' => [
'type' => 'ngram',
'min_gram' => $minGram,
'max_gram' => $maxGram,
'token_chars' => ['letter', 'digit', 'symbol'],
],
],
],
];
}
protected function mappingProperties(): array
{
return [
'_meta' => ['index_compat_version' => self::INDEX_COMPAT_VERSION],
'properties' => [ 'properties' => [
'join_field' => ['type' => 'join', 'relations' => ['discussion' => 'post']], 'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer'],
'discussion_id' => ['type' => 'integer'],
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
'rawId' => ['type' => 'integer'],
'created_at' => ['type' => 'date'], 'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'], 'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'], 'is_private' => ['type' => 'boolean'],
'is_sticky' => ['type' => 'boolean'], 'is_sticky' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'], 'groups' => ['type' => 'integer'],
'tags' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'], 'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'], 'recipient_users' => ['type' => 'integer'],
'comment_count' => ['type' => 'integer'], ]
'view_count' => ['type' => 'integer'],
'is_hidden' => ['type' => 'boolean'],
],
]; ];
// Flush the index.
$client->indices()->delete([
'index' => $index,
'ignore_unavailable' => true
]);
// Create a new index.
$client->indices()->create([
'index' => $index,
'body' => [
'settings' => [
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $settings->get('blomstra-search.analyzer-language') ?: 'english'
]
]
]
]
]
]);
$client->indices()->putMapping([
'index' => $index,
'body' => $properties
]);
/** @var Seeder $seeder */
foreach ($seeders as $seeder) {
$seeder->query()
->when($this->option('max-id'), function ($query, $id) {
$query->where('id', '<=', $id);
})
->chunk(50, function (Collection $collection) use ($queue, &$total, $seeder) {
$queue->pushOn(Job::$onQueue, new SavingJob($collection, $seeder));
$this->info("Pushed into the index, type: {$seeder->type()}, amount: {$collection->count()}.");
$total += $collection->count();
});
$this->info("Pushed a total of $total into the index.");
} }
protected function getContinueAt(SettingsRepositoryInterface $settings, string $type): ?int
{
$raw = $settings->get("blomstra-search.continued-at.$type");
return $raw !== null ? (int) $raw : null;
}
protected function setContinueAt(SettingsRepositoryInterface $settings, string $type, ?int $at): void
{
$settings->set("blomstra-search.continued-at.$type", $at);
} }
} }

View File

@ -1,57 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class HasChildQuery implements Query
{
protected bool $innerHits = false;
public function __construct(
protected string $type,
protected Query $query,
protected string $scoreMode = 'sum'
) {}
public static function create(string $type, Query $query, string $scoreMode = 'sum'): static
{
return new static($type, $query, $scoreMode);
}
/**
* Include inner_hits so the best-matching post ID is available in the response.
* ES returns the top-scoring child document per parent hit under inner_hits.best_post.
*/
public function withInnerHits(): static
{
$this->innerHits = true;
return $this;
}
public function toArray(): array
{
$hasChild = [
'type' => $this->type,
'score_mode' => $this->scoreMode,
'query' => $this->query->toArray(),
];
if ($this->innerHits) {
$hasChild['inner_hits'] = ['name' => 'best_post', 'size' => 1];
}
return ['has_child' => $hasChild];
}
}

View File

@ -1,38 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;
class MatchPhraseQuery extends MatchQuery
{
protected float $boost = 1;
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function toArray(): array
{
$query = parent::toArray()['match'];
$query[$this->field]['boost'] = $this->boost;
return [
'match_phrase' => $query,
];
}
}

View File

@ -1,67 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
{
protected string $operator = 'or';
protected float $boost = 1;
protected ?string $analyzer = null;
protected bool $zeroTerms = false;
public function and()
{
$this->operator = 'and';
return $this;
}
public function operator(string $operator)
{
$this->operator = $operator;
return $this;
}
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function analyzer(string $analyzer)
{
$this->analyzer = $analyzer;
return $this;
}
public function zeroTerms(bool $zeroTerms = true)
{
$this->zeroTerms = $zeroTerms;
return $this;
}
public function toArray(): array
{
$query = parent::toArray();
$query['match'][$this->field]['operator'] = $this->operator;
$query['match'][$this->field]['boost'] = $this->boost;
$query['match'][$this->field]['zero_terms_query'] = $this->zeroTerms ? 'all' : 'none';
return $query;
}
}

View File

@ -1,59 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class SimpleSearchQuery implements Query
{
protected float $boost = 1;
protected ?string $analyzer = null;
public static function create(array $field, string $value)
{
return new self($field, $value);
}
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function analyzer(string $analyzer)
{
$this->analyzer = $analyzer;
return $this;
}
public function __construct(
protected array $fields,
protected string $value
) {
}
public function toArray(): array
{
return [
'simple_query_string' => [
'query' => $this->value,
'fields' => $this->fields,
'analyzer' => $this->analyzer,
'default_operator' => 'AND',
'boost' => $this->boost,
],
];
}
}

View File

@ -1,15 +1,5 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch; namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query; use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -1,55 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQuery
{
protected float $boost = 1;
protected bool $sensitivity = true;
protected ?string $rewrite = null;
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function caseSensitivity(bool $sensitivity = true)
{
$this->sensitivity = $sensitivity;
return $this;
}
public function rewrite(string $rewrite = null)
{
$this->rewrite = $rewrite;
return $this;
}
public function toArray(): array
{
$query = parent::toArray();
$query['wildcard'][$this->field]['boost'] = $this->boost;
$query['wildcard'][$this->field]['case_insensitive'] = !$this->sensitivity;
if ($this->rewrite) {
$query['wildcard'][$this->field]['rewrite'] = $this->rewrite;
}
return $query;
}
}

View File

@ -1,22 +1,13 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Exceptions; namespace Blomstra\Search\Exceptions;
use Throwable; use Throwable;
class SeedingException extends \Exception class SeedingException extends \Exception
{ {
public function __construct($message = '', public array $items, $code = 0, Throwable $previous = null)
public function __construct($message = "", public array $items, $code = 0, Throwable $previous = null)
{ {
parent::__construct($message, $code, $previous); parent::__construct($message, $code, $previous);
} }

View File

@ -1,15 +1,5 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs; namespace Blomstra\Search\Jobs;
use Elasticsearch\Client; use Elasticsearch\Client;
@ -19,24 +9,21 @@ class DeletingJob extends Job
{ {
public function handle(Client $client) public function handle(Client $client)
{ {
if ($this->models->isEmpty()) { if ($this->models->isEmpty()) return;
return;
}
// Preparing body for storing. // Preparing body for storing.
$body = $this->models->map(function (Model $model) { $body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model); $document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [ return [
['delete' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]], ['delete' => ['_index' => $this->index, '_id' => $document->id]]
]; ];
})->flatten(1); })->flatten(1);
$response = $client->bulk([ $response = $client->bulk([
'index' => $this->index, 'index' => $this->index,
'body' => $body->toArray(), 'body' => $body->toArray(),
'refresh' => true, 'refresh' => true
]); ]);
} }
} }

View File

@ -1,15 +1,5 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs; namespace Blomstra\Search\Jobs;
use Blomstra\Search\Seeders\Seeder; use Blomstra\Search\Seeders\Seeder;
@ -22,16 +12,10 @@ abstract class Job extends AbstractJob
public static ?string $onQueue = null; public static ?string $onQueue = null;
/** public function __construct(protected Collection $models, protected Seeder $seeder)
* @param string|null $targetIndex Explicit index name for blue-green builds.
* Defaults to the configured alias when null.
*/
public function __construct(protected Collection $models, protected Seeder $seeder, ?string $targetIndex = null)
{ {
$this->index = $targetIndex ?? resolve('blomstra.search.elastic_index'); $this->index = resolve('blomstra.search.elastic_index');
if (static::$onQueue) { if (static::$onQueue) $this->onQueue(static::$onQueue);
$this->onQueue(static::$onQueue);
}
} }
} }

View File

@ -1,15 +1,5 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs; namespace Blomstra\Search\Jobs;
use Blomstra\Search\Exceptions\SeedingException; use Blomstra\Search\Exceptions\SeedingException;
@ -17,24 +7,19 @@ use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Model; use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr; use Illuminate\Support\Arr;
class UpdateSearchJob extends Job class SavingJob extends Job
{ {
public function handle(Client $client) public function handle(Client $client)
{ {
if ($this->models->isEmpty()) { if ($this->models->isEmpty()) return;
return;
}
$this->models->loadMissing($this->seeder->relationships());
// Preparing body for storing. // Preparing body for storing.
$body = $this->models->map(function (Model $model) { $body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model); $document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [ return [
['index' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]], ['index' => ['_index' => $this->index, '_id' => $document->id]],
$document->toArray(), $document->toArray()
]; ];
}) })
->flatten(1); ->flatten(1);
@ -42,11 +27,10 @@ class UpdateSearchJob extends Job
$response = $client->bulk([ $response = $client->bulk([
'index' => $this->index, 'index' => $this->index,
'body' => $body->toArray(), 'body' => $body->toArray(),
'refresh' => true
]); ]);
if (Arr::get($response, 'errors') !== true) { if (Arr::get($response, 'errors') !== true) return true;
return true;
}
$items = Arr::get($response, 'items'); $items = Arr::get($response, 'items');

View File

@ -1,59 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs;
use Elasticsearch\Client;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Queue\AbstractJob;
class ViewsSearchJob extends AbstractJob
{
protected string $index;
protected string $documentType;
public function __construct(protected int $discussionId)
{
$this->index = resolve('blomstra.search.elastic_index');
$this->documentType = resolve(DiscussionSerializer::class)->getType(new Discussion());
if (Job::$onQueue) {
$this->onQueue(Job::$onQueue);
}
}
public function handle(Client $client): void
{
$discussion = Discussion::find($this->discussionId);
if (!$discussion) {
return;
}
$type = $this->documentType;
try {
$client->update([
'index' => $this->index,
'id' => "$type:{$this->discussionId}",
'routing' => (string) $this->discussionId,
'retry_on_conflict' => 3,
'body' => [
'doc' => ['view_count' => (int) $discussion->view_count],
],
]);
} catch (\Elasticsearch\Common\Exceptions\Missing404Exception $e) {
// Document not yet indexed; will be picked up on next: blomstra:search:index fill
}
}
}

View File

@ -1,33 +1,19 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search; namespace Blomstra\Search;
use Blomstra\Search\Jobs\DeletingJob; use Blomstra\Search\Jobs\DeletingJob;
use Blomstra\Search\Jobs\Job; use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\UpdateSearchJob; use Blomstra\Search\Jobs\SavingJob;
use Blomstra\Search\Jobs\ViewsSearchJob; use Blomstra\Search\Seeders;
use Elasticsearch\Client as Elastic; use Elasticsearch\Client;
use Elasticsearch\ClientBuilder; use Elasticsearch\ClientBuilder;
use Flarum\Api\Client;
use Flarum\Foundation\AbstractServiceProvider; use Flarum\Foundation\AbstractServiceProvider;
use Flarum\Foundation\Config;
use Flarum\Http\Middleware\ExecuteRoute;
use Flarum\Settings\SettingsRepositoryInterface; use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Contracts\Container\Container; use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Events\Dispatcher; use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Contracts\Queue\Queue; use Illuminate\Contracts\Queue\Queue;
use Illuminate\Support\Collection; use Illuminate\Support\Collection;
use Laminas\Stratigility\MiddlewarePipe;
use Psr\Log\LoggerInterface; use Psr\Log\LoggerInterface;
class Provider extends AbstractServiceProvider class Provider extends AbstractServiceProvider
@ -42,16 +28,10 @@ class Provider extends AbstractServiceProvider
/** @var SettingsRepositoryInterface $settings */ /** @var SettingsRepositoryInterface $settings */
$settings = $this->container->make(SettingsRepositoryInterface::class); $settings = $this->container->make(SettingsRepositoryInterface::class);
/** @var Config $config */ $this->container->singleton(Client::class, function (Container $container) use ($settings) {
$config = $this->container->make(Config::class);
$this->container->singleton(Elastic::class, function (Container $container) use ($settings, $config) {
$builder = ClientBuilder::create() $builder = ClientBuilder::create()
->setHosts([$settings->get('blomstra-search.elastic-endpoint')]); ->setHosts([$settings->get('blomstra-search.elastic-endpoint')])
->setLogger($container->make(LoggerInterface::class));
if ($config->inDebugMode()) {
$builder->setLogger($container->make(LoggerInterface::class));
}
if ($settings->get('blomstra-search.elastic-username')) { if ($settings->get('blomstra-search.elastic-username')) {
$builder->setBasicAuthentication( $builder->setBasicAuthentication(
@ -63,36 +43,11 @@ class Provider extends AbstractServiceProvider
return $builder->build(); return $builder->build();
}); });
$this->container->instance( $this->container->instance(
'blomstra.search.elastic_index', 'blomstra.search.elastic_index',
$settings->get('blomstra-search.elastic-index', 'flarum') $settings->get('blomstra-search.elastic-index', 'flarum')
); );
$this->container->extend(
Client::class,
function () {
$pipe = new MiddlewarePipe();
$exclude = resolve('flarum.api_client.exclude_middleware');
$middlewareStack = array_filter(resolve('flarum.api.middleware'), function ($middlewareClass) use ($exclude) {
return !in_array($middlewareClass, $exclude);
});
foreach ($middlewareStack as $middleware) {
$pipe->pipe(resolve($middleware));
}
$pipe->pipe(new ExecuteRoute());
return new Api\Client($pipe);
}
);
$this->container->tag([
Searchers\DiscussionSearcher::class,
Searchers\CommentPostSearcher::class,
], 'blomstra.search.searchers');
} }
public function boot() public function boot()
@ -109,16 +64,12 @@ class Provider extends AbstractServiceProvider
/** @var string|Seeders\Seeder $seeder */ /** @var string|Seeders\Seeder $seeder */
foreach ($seeders as $seeder) { foreach ($seeders as $seeder) {
$seeder::savingOn($events, function ($model) use ($queue, $seeder) { $seeder::savingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new UpdateSearchJob(Collection::make([$model]), $seeder)); $queue->pushOn(Job::$onQueue, new SavingJob(Collection::make([$model]), $seeder));
}); });
$seeder::deletingOn($events, function ($model) use ($queue, $seeder) { $seeder::deletingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new DeletingJob(Collection::make([$model]), $seeder)); $queue->pushOn(Job::$onQueue, new DeletingJob(Collection::make([$model]), $seeder));
}); });
$seeder::viewingOn($events, function (int $discussionId) use ($queue) {
$queue->pushOn(Job::$onQueue, new ViewsSearchJob($discussionId));
});
} }
} }
} }

View File

@ -1,24 +1,13 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Save; namespace Blomstra\Search\Save;
use Carbon\Carbon; use Carbon\Carbon;
use Illuminate\Support\Arr;
use Illuminate\Support\Fluent; use Illuminate\Support\Fluent;
/** /**
* @property string $type
* @property string $id * @property string $id
* @property int $rawId
* @property string $content * @property string $content
* @property Carbon $created_at * @property Carbon $created_at
* @property Carbon $updated_at * @property Carbon $updated_at
@ -31,13 +20,4 @@ use Illuminate\Support\Fluent;
*/ */
class Document extends Fluent class Document extends Fluent
{ {
/**
* Exclude `id` from the body sent to Elasticsearch. The document ID is
* already stored as `_id` by the bulk API action; duplicating it in
* `_source` wastes space and is redundant.
*/
public function toArray(): array
{
return Arr::except($this->attributes, ['id']);
}
} }

View File

@ -1,27 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Searchers;
use Blomstra\Search\Seeders\CommentSeeder;
class CommentPostSearcher extends Searcher
{
protected string|null $seeder = CommentSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-post-bodies', true);
return boolval($enabled);
}
}

View File

@ -1,32 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Searchers;
use Blomstra\Search\Seeders\DiscussionSeeder;
class DiscussionSearcher extends Searcher
{
protected string|null $seeder = DiscussionSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-discussion-subjects', true);
return boolval($enabled);
}
public function boost(): float
{
return 1.5;
}
}

View File

@ -1,48 +0,0 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Searchers;
use Blomstra\Search\Seeders\Seeder;
use Flarum\Settings\SettingsRepositoryInterface;
abstract class Searcher
{
protected string|null $seeder = null;
public function type(): string
{
/** @var Seeder $seeder */
$seeder = $this->seeder;
if (empty($seeder)) {
throw new \InvalidArgumentException('Implement type or add $seeder');
}
return (new $seeder())->type();
}
public function enabled(): bool
{
return true;
}
public function boost(): float
{
return 1;
}
protected function setting(string $key, $default = null)
{
return resolve(SettingsRepositoryInterface::class)->get($key, $default);
}
}

View File

@ -1,23 +1,12 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Seeders; namespace Blomstra\Search\Seeders;
use Blomstra\Search\Save\Document; use Blomstra\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Api\Serializer\PostSerializer; use Flarum\Api\Serializer\PostSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Post\CommentPost; use Flarum\Post\CommentPost;
use Flarum\Post\Event as Core; use Flarum\Post\Event\Deleted;
use Flarum\Post\Event\Posted;
use Illuminate\Contracts\Events\Dispatcher; use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder; use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Model; use Illuminate\Database\Eloquent\Model;
@ -26,71 +15,52 @@ class CommentSeeder extends Seeder
{ {
public function type(): string public function type(): string
{ {
return resolve(PostSerializer::class)->getType(new CommentPost()); return resolve(PostSerializer::class)->getType(new CommentPost);
}
public function joinRelation(): string
{
return 'post';
}
public function routing(Model $model): string
{
return (string) $model->discussion_id;
} }
public function query(): Builder public function query(): Builder
{ {
return CommentPost::query() return CommentPost::query()
->where('type', CommentPost::$type); ->where('type', CommentPost::$type)
->with('discussion');
} }
public static function savingOn(Dispatcher $events, callable $callable) public static function savingOn(Dispatcher $events, callable $callable)
{ {
$events->listen([ $events->listen(Posted::class, function (Posted $event) use ($callable) {
Core\Posted::class,
Core\Revised::class,
Core\Hidden::class,
Core\Restored::class,
], function ($event) use ($callable) {
$callable($event->post); $callable($event->post);
}); });
} }
public static function deletingOn(Dispatcher $events, callable $callable) public static function deletingOn(Dispatcher $events, callable $callable)
{ {
$events->listen([ $events->listen(Deleted::class, function (Deleted $event) use ($callable) {
Core\Deleted::class
], function ($event) use ($callable) {
$callable($event->post); $callable($event->post);
}); });
} }
/** Cached discussion type string (e.g. "discussions") — resolved once per job. */
private ?string $discussionType = null;
private function discussionType(): string
{
return $this->discussionType ??= resolve(DiscussionSerializer::class)->getType(new Discussion());
}
/** /**
* Post documents only need content for has_child matching and the join
* field to establish the parent-child relationship. All discussion-level
* fields (groups, tags, comment_count, etc.) live on the discussion
* document and are irrelevant here.
*
* @param CommentPost $model * @param CommentPost $model
* @return Document
*/ */
public function toDocument(Model $model): Document public function toDocument(Model $model): Document
{ {
return new Document([ $document = new Document([
'join_field' => ['name' => $this->joinRelation(), 'parent' => "{$this->discussionType()}:{$model->discussion_id}"], 'type' => $this->type(),
'discussion_id' => $model->discussion_id, 'id' => $this->type() . ':' . $model->id,
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->content, 'content' => $model->content,
'is_hidden' => $model->hidden_at !== null, 'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->edited_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model->discussion)
]); ]);
if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->discussion->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->discussion->recipientGroups->pluck('id')->toArray();
}
return $document;
} }
} }

View File

@ -1,209 +1,70 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Seeders; namespace Blomstra\Search\Seeders;
use Blomstra\Search\Save\Document; use Blomstra\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer; use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion; use Flarum\Discussion\Discussion;
use Flarum\Discussion\Event as Core; use Flarum\Discussion\Event\Deleted;
use Flarum\Extension\ExtensionManager; use Flarum\Discussion\Event\Hidden;
use Flarum\Group\Group; use Flarum\Discussion\Event\Restored;
use Flarum\Group\Permission; use Flarum\Discussion\Event\Started;
use Flarum\Tags\Tag;
use FoF\Byobu\Events as Byobu;
use FoF\DiscussionViews\Events\DiscussionWasViewed;
use Illuminate\Contracts\Events\Dispatcher; use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder; use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Database\Eloquent\Model; use Illuminate\Database\Eloquent\Model;
class DiscussionSeeder extends Seeder class DiscussionSeeder extends Seeder
{ {
public function type(): string public function type(): string
{ {
return resolve(DiscussionSerializer::class)->getType(new Discussion()); return resolve(DiscussionSerializer::class)->getType(new Discussion);
}
public function joinRelation(): string
{
return 'discussion';
}
public function routing(Model $model): string
{
return (string) $model->id;
} }
public function query(): Builder public function query(): Builder
{ {
return Discussion::query() return Discussion::query();
->whereNull('hidden_at');
}
public function relationships(): array
{
$includes = [];
if ($this->extensionEnabled('flarum-tags')) {
$includes[] = 'tags';
}
if ($this->extensionEnabled('fof-byobu')) {
$includes[] = 'recipientUsers';
$includes[] = 'recipientGroups';
}
return $includes;
} }
public static function savingOn(Dispatcher $events, callable $callable) public static function savingOn(Dispatcher $events, callable $callable)
{ {
$events->listen([ $events->listen([Started::class, Restored::class], function ($event) use ($callable) {
// flarum/core events
Core\Started::class, Core\Restored::class,
// fof/byobu discussion recipients events.
Byobu\DiscussionMadePublic::class, Byobu\RemovedSelf::class, Byobu\RecipientsChanged::class,
], function ($event) use ($callable) {
return $callable($event->discussion); return $callable($event->discussion);
}); });
} }
public static function viewingOn(Dispatcher $events, callable $callable): void
{
if (!resolve(ExtensionManager::class)->isEnabled('fof-discussion-views')) {
return;
}
$events->listen(DiscussionWasViewed::class, function (DiscussionWasViewed $event) use ($callable) {
$viewCount = $event->discussion->view_count;
$shouldSync = match (true) {
$viewCount < 15 => true,
$viewCount < 100 => rand(1, 3) === 1,
default => rand(1, 19) === 1,
};
if ($shouldSync) {
$callable($event->discussion->id);
}
});
}
public static function deletingOn(Dispatcher $events, callable $callable) public static function deletingOn(Dispatcher $events, callable $callable)
{ {
$events->listen([ $events->listen([Deleted::class, Hidden::class], function ($event) use ($callable) {
// flarum/core events.
Core\Deleted::class, Core\Hidden::class
], function ($event) use ($callable) {
return $callable($event->discussion); return $callable($event->discussion);
}); });
} }
/** /**
* @param Discussion $model * @param Discussion $model
*
* @return Document * @return Document
*/ */
public function toDocument(Model $model): Document public function toDocument(Model $model): Document
{ {
$document = new Document([ $document = new Document([
'join_field' => $this->joinRelation(), 'type' => $this->type(),
'id' => $this->type().':'.$model->id, 'id' => $this->type() . ':' . $model->id,
'rawId' => $model->id,
'content' => $model->title, 'content' => $model->title,
'created_at' => $model->created_at?->toAtomString(), 'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->last_posted_at?->toAtomString(), 'updated_at' => $model->last_posted_at?->toAtomString(),
'is_private' => $model->is_private, 'is_private' => $model->is_private,
'user_id' => $model->user_id, 'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model), 'groups' => $this->groupsForDiscussion($model)
'comment_count' => $model->comment_count,
]); ]);
if ($this->extensionEnabled('flarum-tags')) {
$document['tags'] = $model->tags->pluck('id')->toArray();
}
if ($this->extensionEnabled('fof-byobu')) { if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->recipientUsers $document['recipient_users'] = $model->recipientUsers->pluck('id')->toArray();
->whereNull('removed_at') $document['recipient_groups'] = $model->recipientGroups->pluck('id')->toArray();
->pluck('id')
->toArray();
$document['recipient_groups'] = $model->recipientGroups
->whereNull('removed_at')
->pluck('id')
->toArray();
} }
if ($this->extensionEnabled('flarum-sticky')) { if ($this->extensionEnabled('flarum-sticky')) {
$document['is_sticky'] = (bool) $model->is_sticky; $document['is_sticky'] = (bool) $model->is_sticky;
} }
if ($this->extensionEnabled('fof-discussion-views')) {
$document['view_count'] = (int) ($model->view_count ?? 0);
}
return $document; return $document;
} }
/**
* All viewForum permissions keyed by permission string, loaded once per seeder instance.
* Avoids N×2 Permission queries inside the per-document map loop.
*/
private ?Collection $cachedPermissions = null;
private ?Collection $cachedGlobalPermission = null;
private function allPermissions(): Collection
{
if ($this->cachedPermissions === null) {
$this->cachedPermissions = Permission::query()
->where(function ($q) {
$q->where('permission', 'viewForum')
->orWhere('permission', 'like', 'tag%.viewForum');
})
->get();
$this->cachedGlobalPermission = $this->cachedPermissions
->where('permission', 'viewForum')
->pluck('group_id');
}
return $this->cachedPermissions;
}
protected function groupsForDiscussion(Discussion $discussion): array
{
$allPerms = $this->allPermissions();
$permissions = collect();
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$permissions = $tags->map(function (Tag $tag) use ($allPerms) {
$tagPerms = $allPerms->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$tagPerms = $tagPerms->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $tagPerms->pluck('group_id');
})->flatten();
}
if (!$discussion->is_private && $permissions->isEmpty()) {
$permissions = $this->cachedGlobalPermission;
}
return $permissions->toArray();
}
} }

View File

@ -1,54 +1,66 @@
<?php <?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Seeders; namespace Blomstra\Search\Seeders;
use Blomstra\Search\Save\Document; use Blomstra\Search\Save\Document;
use Flarum\Discussion\Discussion;
use Flarum\Extension\ExtensionManager; use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use Illuminate\Contracts\Events\Dispatcher; use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder; use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Database\Eloquent\Model; use Illuminate\Database\Eloquent\Model;
abstract class Seeder abstract class Seeder
{ {
abstract public function type(): string; abstract public function type(): string;
/**
* The join relation name for this document type ('discussion' or 'post').
* Used in the join_field mapping and for filtering in seed-missing checks.
*/
abstract public function joinRelation(): string;
/**
* The ES routing key for this model. Parent and child documents for the same
* discussion must share a routing key so ES places them on the same shard.
*/
abstract public function routing(Model $model): string;
abstract public function query(): Builder; abstract public function query(): Builder;
abstract public static function savingOn(Dispatcher $events, callable $callable); abstract public static function savingOn(Dispatcher $events, callable $callable);
abstract public static function deletingOn(Dispatcher $events, callable $callable); abstract public static function deletingOn(Dispatcher $events, callable $callable);
/** No-op default; override in seeders that need to react to view events. */
public static function viewingOn(Dispatcher $events, callable $callable): void {}
abstract public function toDocument(Model $model): Document; abstract public function toDocument(Model $model): Document;
/** Relationships to eager-load on the collection before calling toDocument(). */ protected function groupsForDiscussion(Discussion $discussion): array
public function relationships(): array
{ {
return []; $permissions = collect();
$globalPermission = Permission::query()
->where('permission', 'viewForum')
->pluck('group_id');
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$filters['tags'] = $tags->pluck('id')->toArray();
$tagPermissions = Permission::query()
->whereIn(
'permission',
$tags->pluck('id')->map(function (int $id) {
return "tag$id.viewForum";
})
)->get();
$permissions = $tags->map(function (Tag $tag) use ($tagPermissions) {
$permissions = $tagPermissions->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$permissions = $permissions->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $permissions->pluck('group_id');
})->flatten();
}
if (! $discussion->is_private && $permissions->isEmpty()) {
$permissions = $globalPermission;
}
return $permissions->toArray();
} }
protected function extensionEnabled(string $extension): bool protected function extensionEnabled(string $extension): bool