Compare commits

..

90 Commits
0.0.15 ... main

Author SHA1 Message Date
flarum-bot bf19eed20f Bundled output for commit 2fc66b9bf7
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2026-04-13 20:22:53 +00:00
Bart van Bragt 2fc66b9bf7 perf: eager-load relationships in UpdateSearchJob and cache permissions in DiscussionSeeder
- Add relationships() to Seeder base class; DiscussionSeeder implements it to
  return tags/recipientUsers/recipientGroups for eager loading
- UpdateSearchJob calls loadMissing(relationships()) before the map loop,
  eliminating N×R lazy-load queries per batch
- Remove refresh:true from bulk call — synchronous ES refresh was the primary
  throughput bottleneck during indexing
- Cache viewForum permissions in DiscussionSeeder per job instance, replacing
  N×2 Permission queries per document with a single cached query
- Increase seeder batch size from 1000 to 2500
2026-04-13 22:18:14 +02:00
Bart van Bragt 439c74f0a3 fix: store index compat version in ES mapping _meta for correct rollback behaviour
- Write INDEX_COMPAT_VERSION into mapping _meta.index_compat_version on every
  build/promote so the version travels with the index rather than being a
  separate settings value
- saveIndexedConfig now reads compat version back from _meta alongside the
  existing analyzer/min_gram sync, so rollback automatically restores the
  correct value (or null for pre-feature indexes, triggering the reindex warning)
- Replace InvalidArgumentException in buildIndexSettings with $this->error()
  fallback to avoid stack traces in the console
- Fix stale comment in ViewsSearchJob referencing removed CLI flags
2026-04-10 13:46:08 +02:00
Bart van Bragt f486292f15 feat: make ngram min_gram configurable via admin setting
Adds a 'Minimum search query length' setting (1–4, default 3). The admin
shows a warning when the analyzer or min-search-length have changed since
the last index build.
2026-04-10 13:19:22 +02:00
Bart van Bragt 4412583cd9 chore: fix prettier formatting and npm audit vulnerabilities
Reformat admin/index.ts to satisfy prettier checks. Update package-lock.json
via npm audit fix to resolve 17 vulnerabilities in transitive dev dependencies
(webpack, babel, lodash, ws, semver, etc.).
2026-04-10 11:47:38 +02:00
Bart van Bragt 81e7b6502d feat: Show admin dashboard notice when search index needs rebuilding
After a mapping change (e.g. the parent-child restructure), existing installs
need to rebuild their index. A DashboardWidget appears on the admin page
warning the admin and showing the command to run. Compatibility is tracked
via the blomstra-search.index-compatible setting.
2026-04-10 11:10:53 +02:00
Bart van Bragt 3039b65946 refactor: Simplify index command with better naming/options
The flag-based interface (--rebuild etc) became rather convoluted with the changes. Replace this with commands with more straight forward naming.

Also added an explicit 'rollback' action and updated the README.md
2026-04-10 09:35:49 +02:00
Bart van Bragt e2e2d1428f feat: restructure index as parent-child documents with blue-green rebuilds
- Switch to ES parent/child join_field: discussion docs hold all metadata,
  post docs hold content only. Routing ensures parent and child land on the
  same shard. Removes the old flat-document approach where post data was
  duplicated onto every comment document.

- Add HasChildQuery with inner_hits so the best-matching post ID is
  surfaced as mostRelevantPost without a second DB query.

- Blue-green index rebuilds: --recreate writes into a timestamped pending
  index, --swap atomically promotes it via alias. Interrupted builds are
  resumable with --recreate --continue. --swap requires confirmation before
  proceeding.

- Index hidden posts; non-moderators are filtered at query time via
  is_hidden on the has_child clause. Core\Hidden and Core\Restored trigger
  re-indexing so moderators with post.hide can search hidden posts.
2026-04-09 23:30:57 +02:00
Bart van Bragt 80d66c98c4 feat: index and sync view_count for fof/discussion-views sort support
- Rename SavingJob to UpdateSearchJob
- Add ViewsSearchJob for lightweight partial view_count updates
- Add probabilistic sync throttle in DiscussionSeeder::viewingOn()
  to avoid an ES update on every page view
- Add view_count to mapping, DiscussionSeeder::toDocument(), and
  translateSort
- Log a warning and skip instead of 500ing on unknown sort fields
2026-04-09 12:05:28 +02:00
Bart van Bragt ff2a252e99 fix: editing a post does not update the search index
CommentSeeder only listened to Post\Event\Posted, so post edits
left the Elasticsearch document stale.
2026-04-09 11:17:17 +02:00
Bart van Bragt 5929a191f6 fix: --seed-missing stops early when a full range is already indexed
When all documents in a range were already in the index, the empty
collection caused continueAt to be set to null, aborting before
scanning lower ID ranges. Now advances past the range bottom instead.
2026-04-09 11:17:17 +02:00
Bart van Bragt 6699c8cb58 fix: ngram min_gram=1 indexes tokens that can never be matched
Flarum's search UI enforces a minimum query length of 3 characters
(MIN_SEARCH_LEN in Search.tsx), so 1- and 2-character ngram tokens
are indexed but can never be queried. Changed min_gram from 1 to 3
and updated max_ngram_diff accordingly.
2026-04-09 11:04:27 +02:00
Bart van Bragt 369ee504f3 fix: remove unused content_partial field from seeders
content_partial was introduced alongside a partialMatch() query that
targeted it. That query was later switched to use the content field
instead, but content_partial was never removed from the seeders,
causing every document to store and index its content twice.
2026-04-09 08:39:58 +02:00
Bart van Bragt eeb091273d fix: tag filter produces empty results when browsing a tag
When navigating to /t/{tag}?q=query the tag filter from the request
was silently ignored. Added a TermsQuery on the tags field using the
tag IDs resolved from the provided slugs.
2026-04-09 08:20:42 +02:00
Daniël Klabbers d252a17586 fix: not changing recipients visibility when recipients change 2024-06-13 19:27:14 +02:00
Daniël Klabbers a9dc8bfb16 fix: byobu discussions open up while using this extension 2024-06-12 11:34:18 +02:00
StyleCI Bot 82e6ec786f
Apply fixes from StyleCI 2023-10-18 13:57:41 +00:00
flarum-bot f981c27e00 Bundled output for commit f6503f5e6a
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2023-01-30 22:17:40 +00:00
Daniël Klabbers f6503f5e6a fix: #10, private forums have no discussions key, throws error 2023-01-30 23:17:01 +01:00
flarum-bot ad92d8856b Bundled output for commit 8b519619e5
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-24 09:39:12 +00:00
StyleCI Bot 8b519619e5
Apply fixes from StyleCI 2022-11-24 09:38:21 +00:00
Daniël Klabbers 83afe95d61 chore: fix search and seeding 2022-11-24 10:37:57 +01:00
Daniël Klabbers 846cc348ff Merge branch 'dk/search-improvements' into main 2022-11-24 10:29:18 +01:00
Daniël Klabbers 508bb42ac9 chore: clean up buildcommand 2022-11-23 12:46:04 +01:00
StyleCI Bot db74da3e37
Apply fixes from StyleCI 2022-11-23 11:29:48 +00:00
Daniël Klabbers a02d35394b feat: store tags, rawId, support indexing missing objects 2022-11-23 12:09:10 +01:00
flarum-bot b60563f2ec Bundled output for commit 703cbbdd48
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-21 15:33:18 +00:00
David Wheatley 703cbbdd48
fix: incorrect JS package name 2022-11-21 15:32:22 +00:00
flarum-bot 769b6cd278 Bundled output for commit f500d7f956
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-21 12:38:34 +00:00
Ian Morland f500d7f956 fix: revert back to not using the elastic endpoint for filtering, at least until we support tags in the search database 2022-11-21 12:37:52 +00:00
flarum-bot 7257d5fb34 Bundled output for commit 09fb4e8b7b
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-18 09:39:40 +00:00
Daniël Klabbers 09fb4e8b7b
Merge pull request #9 from blomstra/im/fix-search-override
fix: override search not fired correctly
2022-11-18 10:38:47 +01:00
Ian Morland 21170fbe1b Merge remote-tracking branch 'origin/main' into im/fix-search-override 2022-11-18 09:37:37 +00:00
StyleCI Bot 1cd3dac177
Apply fixes from StyleCI 2022-11-18 09:34:30 +00:00
Ian Morland 5815144d3a Revert "Apply fixes from StyleCI"
This reverts commit a26f88a9b6.
2022-11-18 09:34:16 +00:00
StyleCI Bot f8aa3dcf13
Apply fixes from StyleCI 2022-11-18 09:33:13 +00:00
StyleCI Bot a26f88a9b6
Apply fixes from StyleCI 2022-11-18 09:29:46 +00:00
Ian Morland 23dd403417 fix: override endpoint not used correctly 2022-11-18 09:29:19 +00:00
Ian Morland ed718e1458 chore: fix incorect command 2022-11-18 09:28:52 +00:00
flarum-bot 4240269b7f Bundled output for commit c186535bbb
Includes transpiled JS/TS, and Typescript declaration files (typings).

[skip ci]
2022-11-18 08:31:43 +00:00
Ian Morland c186535bbb chore: use npm 2022-11-18 08:30:44 +00:00
Ian Morland fa90def995 chore: update frontend workflow 2022-11-18 08:29:19 +00:00
Ian Morland 4d2c5cb22f chore: bump deps, npm audit fix 2022-11-18 08:26:26 +00:00
Daniël Klabbers 9401b677a3 change weighing for exact words match 2022-11-10 12:57:46 +01:00
Daniël Klabbers 4bc89949ec complete testing 2022-11-10 12:54:26 +01:00
Daniël Klabbers 29ba864849 settings configurations 2022-11-09 18:06:48 +01:00
flarum-bot aabba9c2d0 Bundled output for commit 886e78d6fb
Includes transpiled JS/TS.

[skip ci]
2022-05-03 08:15:09 +00:00
Ian Morland 886e78d6fb Format js 2022-05-03 09:13:50 +01:00
Ian Morland af85834a5a Webpack 5 2022-05-03 09:13:10 +01:00
Daniël Klabbers e26d90d7b6 fix ambiguous 2022-03-04 12:37:43 +01:00
Daniël Klabbers cb3d4f5225 fix typo 2022-02-28 20:33:01 +01:00
Daniël Klabbers c6fd942bf1 prevent leaking deleted discussions 2022-02-28 20:32:27 +01:00
Daniël Klabbers cc8145c236 drop partial search 2022-02-21 10:20:46 +01:00
Robert Korulczyk 8ab60b2fb8
Create LICENSE.md (#7) 2022-02-17 22:24:42 +00:00
Ian Morland 14833fbe8f Update author meta 2022-02-17 19:25:53 +00:00
Daniel Klabbers ccfb964aff improve search results 2022-02-14 16:51:55 +01:00
Daniël Klabbers 133ded0684 fixes #5 2022-01-11 10:10:39 +01:00
Daniel Klabbers 803d1ecfc5 fix partial matching and weighting 2022-01-10 14:16:42 +01:00
Daniel Klabbers 502f7a9f86 use partial match with weighting 2022-01-10 13:47:05 +01:00
Daniel Klabbers 9108553f09 add and vs or distinction, but partial isnt working in combination with the others yet 2022-01-06 12:58:50 +01:00
Daniel Klabbers b6ac018a56 fix partial matching 2022-01-03 16:53:45 +01:00
Daniel Klabbers 88b3e57d8f added weighting properly, also different for titles vs posts 2022-01-03 15:13:13 +01:00
Daniel Klabbers f180d66815 Revert "feat: add partial word match with low weighting"
This reverts commit 0d51e5d912.
2021-12-23 13:07:04 +01:00
Daniel Klabbers 013f5162c5 Revert "introduce wildcard, needs further testing"
This reverts commit 877533b43d.
2021-12-23 13:07:03 +01:00
Daniel Klabbers 23074a3a0c Revert "attempt to make partial match filter"
This reverts commit f21dc04a00.
2021-12-23 13:07:00 +01:00
Daniel Klabbers 0ee902b7b7 Revert "allow minimum should match config"
This reverts commit d5eebf6c65.
2021-12-23 13:06:56 +01:00
Daniel Klabbers d5eebf6c65 allow minimum should match config 2021-12-16 15:05:22 +01:00
Daniel Klabbers f21dc04a00 attempt to make partial match filter 2021-12-16 15:02:18 +01:00
Daniel Klabbers 877533b43d introduce wildcard, needs further testing 2021-12-14 16:35:16 +01:00
Daniel Klabbers 0d51e5d912 feat: add partial word match with low weighting 2021-12-14 12:05:32 +01:00
flarum-bot 2d989c7bbc Bundled output for commit ca229c8af8
Includes transpiled JS/TS.

[skip ci]
2021-12-13 11:07:39 +00:00
Daniel Klabbers ca229c8af8 possibly reduce logging 2021-12-13 12:06:26 +01:00
Daniel Klabbers 68fce10d56 support patching the mapping without recreating 2021-11-16 20:33:17 +01:00
Daniel Klabbers cb7da5d3f8 fix search for the right fields 2021-11-16 20:23:07 +01:00
Daniel Klabbers 868713da11 fix: search for byobu page
Fixes the index for byobu.
2021-11-16 20:22:22 +01:00
Daniël Klabbers 8b1a32af8d add comment count to mapping 2021-11-11 21:46:17 +01:00
Daniël Klabbers cadfc7b9bd seed and use comment count for sorting 2021-11-11 21:40:34 +01:00
Daniel Klabbers 87f695484b compiled 2021-11-11 13:12:42 +01:00
Daniel Klabbers 5600236bb8 exclude byobu from elastic for now 2021-11-11 13:12:36 +01:00
Daniel Klabbers cc9c57308e re enable search, cleanup 2021-11-11 13:12:17 +01:00
Daniel Klabbers 77c406c07f improve scoring by boosting titles, now also major 2021-11-10 16:52:53 +01:00
Daniel Klabbers 050482dd4a fix calling xhr still 2021-11-10 14:39:47 +01:00
Daniel Klabbers 8576579aa6 use correct path on index 2021-11-10 13:55:20 +01:00
Daniel Klabbers bad10ef03d only override the search on the frontend if its not already been handled by the backend to prevent duplicate queries 2021-11-10 13:36:34 +01:00
Daniel Klabbers 8111c214cc override api client on the backend to retrieve discussions from elastic always, this prevents all hits to the MySQL db 2021-11-10 12:40:32 +01:00
Daniel Klabbers a03de74f0b clean up js 2021-11-10 12:39:41 +01:00
Daniel Klabbers 6369637145 fix issue with selection of columns 2021-11-08 20:25:02 +01:00
Daniel Klabbers fd11442a76 optimize search query with join 2021-11-08 16:46:15 +01:00
Daniel Klabbers 8977d3d161 fix for issue with search 2021-11-08 16:41:37 +01:00
Daniël Klabbers 3106667519 attempt to improve results loading with eloquent 2021-11-08 16:04:42 +01:00
40 changed files with 4512 additions and 8745 deletions

View File

@ -1,17 +1,18 @@
name: Build JavaScript assets
name: Search JS
on:
push:
branches:
- main
on: [workflow_dispatch, push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- uses: flarum/action-build@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run:
uses: flarum/framework/.github/workflows/REUSABLE_frontend.yml@main
with:
enable_bundlewatch: false
enable_prettier: true
enable_typescript: false
frontend_directory: ./js
backend_directory: .
js_package_manager: npm
main_git_branch: main
secrets:
bundlewatch_github_token: ${{ secrets.BUNDLEWATCH_GITHUB_TOKEN }}

9
LICENSE.md Normal file
View File

@ -0,0 +1,9 @@
MIT License
Copyright (c) 2021 Blomstra Ltd
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

143
README.md
View File

@ -5,47 +5,154 @@ fulltext search with one that is completely relying on the proven elasticsearch
## Features
- Sync discussions to elastic search using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50.000 discussion **260ms**)
- Sync discussions and posts to Elasticsearch using your queue, unobtrusively for the user.
- Reduces search loading times to well below 400ms (local tests with 50,000 discussions: **260ms**)
- Uses Flarum's group permissions and tags system.
- Compatible with Friends of Flarum Byōbu.
## Installation
## Requirements
Use composer:
- Elasticsearch 7.x or OpenSearch 1.x+
- A non-sync queue driver with a running worker (`php flarum queue:work`) is strongly recommended for production. The extension works with the default sync driver, but index jobs run inline which adds latency to user-facing changes like posting.
## Installation
```bash
composer require blomstra/search:*
```
Enable the extension inside the admin area and configure the settings.
Enable the extension in the admin area and configure the Elasticsearch endpoint and index name in the extension settings.
### Set up
## Setting up the index
Enable the extension in your admin area. Now to seed your existing discussions use the following command:
### First install
```
php flarum blomstra:search:build
Run the build command once. It creates a timestamped concrete index, immediately aliases your configured index name to it, and begins queuing documents. Search is available and improves as the queue processes:
```bash
php flarum blomstra:search:index build
php flarum queue:work
```
All mutations to discussions are automatically added and removed from the elasticsearch index.
### Subsequent rebuilds (zero-downtime)
### FAQ
When you need to rebuild the full index (e.g. after a mapping change):
*I have another question.*
Reach out to us via https://helpdesk.blomstra.net. We will get back to you as soon as we can. If you have a running subscription please mention when you started your plan and/or which plan you are on. Always add sufficient information when reporting errors. We prefer errors being reported here, but understand that sometimes you can't.
```bash
# Simple rebuild — promotes automatically once all jobs are queued
php flarum blomstra:search:index build
# Or keep a backup of the old index in case you need to roll back
php flarum blomstra:search:index build --keep-backup
```
After the queue drains, fill any gaps from content posted during the build:
```bash
php flarum blomstra:search:index fill
```
If you kept a backup and want to roll back:
```bash
php flarum blomstra:search:index rollback
```
Once satisfied with the new index, drop the backup:
```bash
php flarum blomstra:search:index discard --backup
```
### Blue-green rebuild (manual promotion)
Use `--staging` to keep the old index live until you explicitly promote:
```bash
# 1. Build into a staging index — live index is untouched
php flarum blomstra:search:index build --staging
# 2. Drain the queue
php flarum queue:work --stop-when-empty
# 3. Promote the staging index to live
php flarum blomstra:search:index promote
# Or keep the old index as a backup:
php flarum blomstra:search:index promote --keep-backup
```
### Resuming or cancelling an interrupted build
If a build is interrupted, re-run it with the appropriate flag:
```bash
# Resume each seeder from where it left off
php flarum blomstra:search:index build --resume
# Drop the staging index and start completely fresh
php flarum blomstra:search:index build --fresh
# Cancel the build without starting a new one
php flarum blomstra:search:index discard --pending
```
### Filling gaps in an existing index
If documents are missing from the live index (e.g. due to queue failures):
```bash
php flarum blomstra:search:index fill
```
### Updating the mapping only
To push a mapping change to the live index without rebuilding:
```bash
php flarum blomstra:search:index mapping
```
## Command reference
| Command | Description |
|---|---|
| `build` | Rebuild the index and promote automatically once all jobs are queued. On first install, aliases immediately so search is live during seeding. |
| `build --keep-backup` | Rebuild and promote, retaining the old index as a backup for rollback. |
| `build --staging` | Build into a staging index without promoting — use `promote` when ready (blue-green workflow). |
| `build --resume` | Resume an interrupted build from where each seeder left off. |
| `build --fresh` | Drop the staging index and start completely fresh. |
| `promote` | Atomically swap the alias to the staging index. Prompts for confirmation (blue-green workflow). |
| `promote --keep-backup` | Promote and retain the replaced live index as a backup for rollback. |
| `rollback` | Restore the backup index to live. Deletes the index that was live. |
| `discard --pending` | Drop the staging index without promoting (cancels an in-progress build). |
| `discard --backup` | Drop the backup index (cleanup after `--keep-backup`). |
| `mapping` | Push updated mapping to the live index without rebuilding or reseeding. |
| `fill` | Seed only documents missing from the live index. |
| `build --only=discussions` | Seed only the specified document type (`discussions` or `posts`). |
| `build --throttle=N` | Wait N seconds between batches (reduces queue pressure). |
| `build --max-id=N` | Limit seeding to documents with ID ≤ N. |
| `promote --i-am-sure` | Skip the promotion confirmation prompt (for scripts and CI). |
## FAQ
## Queue configuration
*"Can I dispatch indexing jobs to a specific queue?"*
*Can I dispatch the sync jobs to another queue?*
Yes:
```php
\Blomstra\Search\Observe\Job::$onQueue = 'sync';
\Blomstra\Search\Jobs\Job::$onQueue = 'search';
```
*"I have a different question"*
Reach out ot us via https://support.on-floxum.com/t/ext-search . If you have an active subscription, please mention what plan you are on.
---
- Blomstra provides managed Flarum hosting.
- https://blomstra.net
- https://blomstra.community/t/ext-search
- Floxum provides managed Flarum hosting.
- https://floxum.com
- https://support.on-floxum.com/t/ext-search
Icon made by [Freepik](https://www.freepik.com) from [Flaticon](https://www.flaticon.com/).

View File

@ -3,8 +3,12 @@
"description": "Replaces Flarum search with one powered by an elastic search server.",
"keywords": [
"extension",
"flarum", "blomstra",
"search", "fulltext", "elastic", "elasticsearch"
"flarum",
"blomstra",
"search",
"fulltext",
"elastic",
"elasticsearch"
],
"support": {
"email": "helpdesk@blomstra.net",
@ -14,14 +18,14 @@
"license": "MIT",
"authors": [
{
"name": "Daniël Klabbers",
"email": "daniel@klabbers.email",
"homepage": "http://luceos.com"
"name": "Team Blomstra",
"email": "development@blomstra.net",
"role": "Developer"
}
],
"require": {
"php": ">= 8.0",
"flarum/core": "^1.0.0",
"flarum/core": "^1.2.0",
"elasticsearch/elasticsearch": "7.*",
"spatie/elasticsearch-query-builder": "^1.3.0"
},

View File

@ -1,23 +1,38 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search;
use Flarum\Extend as Flarum;
return [
(new Flarum\ServiceProvider)->register(Provider::class),
(new Flarum\ServiceProvider())->register(Provider::class),
(new Flarum\Frontend('forum'))
->js(__DIR__ . '/js/dist/forum.js'),
->js(__DIR__.'/js/dist/forum.js'),
(new Flarum\Frontend('admin'))
->js(__DIR__ . '/js/dist/admin.js'),
->js(__DIR__.'/js/dist/admin.js')
->css(__DIR__.'/resources/less/admin.less'),
(new Flarum\Locales(__DIR__ . '/resources/locale')),
new Flarum\Locales(__DIR__.'/resources/locale'),
(new Flarum\Routes('api'))
->get('/blomstra/search/{type}', 'blomstra.search', Api\Controllers\SearchController::class)
->put('/blomstra/search/index', 'blomstra.search.index', Api\Controllers\IndexController::class),
(new Flarum\Console())
->command(Commands\BuildCommand::class),
(new Flarum\Console)
->command(Commands\BuildCommand::class)
(new Flarum\Settings())
->default('blomstra-search.search-discussion-subjects', true)
->default('blomstra-search.search-post-bodies', true)
->default('blomstra-search.min-search-length', Commands\BuildCommand::DEFAULT_MIN_SEARCH_LENGTH),
];

2
js/dist/admin.js vendored
View File

@ -1,2 +1,2 @@
module.exports=function(e){var t={};function a(r){if(t[r])return t[r].exports;var n=t[r]={i:r,l:!1,exports:{}};return e[r].call(n.exports,n,n.exports,a),n.l=!0,n.exports}return a.m=e,a.c=t,a.d=function(e,t,r){a.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},a.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},a.t=function(e,t){if(1&t&&(e=a(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(a.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)a.d(r,n,function(t){return e[t]}.bind(null,n));return r},a.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return a.d(t,"a",t),t},a.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},a.p="",a(a.s=11)}({1:function(e,t){e.exports=flarum.core.compat["admin/app"]},11:function(e,t,a){"use strict";a.r(t);var r=a(1),n=a.n(r);n.a.initializers.add("blomstra-search",(function(){var e=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach((function(t){e.set(t,t)})),n.a.extensionData.for("blomstra-search").registerSetting({setting:"blomstra-search.elastic-endpoint",label:n.a.translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:n.a.translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:n.a.translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:n.a.translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:n.a.translator.trans("blomstra-search.admin.analyzer.label"),help:n.a.translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(e.entries()),default:"english"})}))}});
(()=>{var t={n:a=>{var e=a&&a.__esModule?()=>a.default:()=>a;return t.d(e,{a:e}),e},d:(a,e)=>{for(var r in e)t.o(e,r)&&!t.o(a,r)&&Object.defineProperty(a,r,{enumerable:!0,get:e[r]})},o:(t,a)=>Object.prototype.hasOwnProperty.call(t,a)};(()=>{"use strict";function a(t,e){return a=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(t,a){return t.__proto__=a,t},a(t,e)}const e=flarum.core.compat["admin/app"];var r=t.n(e);const n=flarum.core.compat["common/extend"],s=flarum.core.compat["admin/components/DashboardPage"];var i=t.n(s);const o=flarum.core.compat["admin/components/DashboardWidget"];var l=t.n(o);const c=flarum.core.compat["common/components/Alert"];var h=t.n(c),d=function(t){var e,n;function s(){return t.apply(this,arguments)||this}n=t,(e=s).prototype=Object.create(n.prototype),e.prototype.constructor=e,a(e,n);var i=s.prototype;return i.className=function(){return"ReindexWarningWidget"},i.content=function(){return m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle",title:r().translator.trans("blomstra-search.admin.reindex-required.title")},r().translator.trans("blomstra-search.admin.reindex-required.detail"))},s}(l());r().initializers.add("blomstra-search",function(){var t=r().data.settings["blomstra-search.active-index"],a=r().data.settings["blomstra-search.index-compatible"];t&&"v2"!==a&&(0,n.extend)(i().prototype,"availableWidgets",function(t){t.add("blomstra-search-reindex",m(d),110)});var e=new Map;["arabic","armenian","basque","bengali","brazilian","bulgarian","catalan","cjk","czech","danish","dutch","english","estonian","finnish","french","galician","german","greek","hindi","hungarian","indonesian","irish","italian","latvian","lithuanian","norwegian","persian","portuguese","romanian","russian","sorani","spanish","swedish","turkish","thai"].forEach(function(t){e.set(t,t)}),r().extensionData.for("blomstra-search").registerSetting(function(){var a=r().data.settings["blomstra-search.indexed-analyzer"];if(!t||!a)return null;var e=this.setting("blomstra-search.analyzer-language")()||"english",n=this.setting("blomstra-search.min-search-length")(),s=String(r().data.settings["blomstra-search.indexed-min-search-length"]||r().data.settings["blomstra-search.min-search-length"]);return e===a&&n===s?null:m(h(),{type:"warning",dismissible:!1,icon:"fas fa-exclamation-triangle"},r().translator.trans("blomstra-search.admin.index-settings-changed"))}).registerSetting({setting:"blomstra-search.elastic-endpoint",label:r().translator.trans("blomstra-search.admin.elastic-endpoint"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-username",label:r().translator.trans("blomstra-search.admin.elastic-username"),type:"input"}).registerSetting({setting:"blomstra-search.elastic-password",label:r().translator.trans("blomstra-search.admin.elastic-password"),type:"password"}).registerSetting({setting:"blomstra-search.elastic-index",label:r().translator.trans("blomstra-search.admin.elastic-index"),default:"flarum",type:"input"}).registerSetting({setting:"blomstra-search.analyzer-language",label:r().translator.trans("blomstra-search.admin.analyzer.label"),help:r().translator.trans("blomstra-search.admin.analyzer.help"),type:"select",options:Object.fromEntries(e.entries()),default:"english"}).registerSetting({setting:"blomstra-search.search-discussion-subjects",label:r().translator.trans("blomstra-search.admin.search-discussion-subjects"),type:"switch"}).registerSetting({setting:"blomstra-search.search-post-bodies",label:r().translator.trans("blomstra-search.admin.search-post-bodies"),type:"switch"}).registerSetting({setting:"blomstra-search.min-search-length",label:r().translator.trans("blomstra-search.admin.min-search-length.label"),help:r().translator.trans("blomstra-search.admin.min-search-length.help"),type:"select",options:{1:"1",2:"2",3:"3",4:"4"},default:r().data.settings["blomstra-search.min-search-length"]})})})(),module.exports={}})();
//# sourceMappingURL=admin.js.map

File diff suppressed because one or more lines are too long

3
js/dist/forum.js vendored

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

10811
js/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -1,16 +1,14 @@
{
"name": "@blomstra/realtime",
"name": "@blomstra/search",
"private": true,
"version": "0.0.0",
"prettier": "@flarum/prettier-config",
"dependencies": {
"@flarum/prettier-config": "^1.0.0",
"build": "^0.1.4",
"flarum-tsconfig": "^1.0.2",
"flarum-webpack-config": "^1.0.0",
"pusher-js": "^7.0.3",
"webpack": "^4.46.0",
"webpack-cli": "^4.9.1"
"flarum-webpack-config": "^2.0.0",
"webpack": "^5.75.0",
"webpack-cli": "^5.0.0"
},
"scripts": {
"dev": "webpack --mode development --watch",
@ -19,6 +17,6 @@
"format-check": "prettier --check src"
},
"devDependencies": {
"prettier": "^2.4.1"
"prettier": "^2.7.1"
}
}

View File

@ -1,37 +1,122 @@
import app from 'flarum/admin/app';
import { extend } from 'flarum/common/extend';
import DashboardPage from 'flarum/admin/components/DashboardPage';
import DashboardWidget from 'flarum/admin/components/DashboardWidget';
import Alert from 'flarum/common/components/Alert';
const REQUIRED_INDEX_COMPAT = 'v2';
class ReindexWarningWidget extends DashboardWidget {
className() {
return 'ReindexWarningWidget';
}
content() {
return m(
Alert,
{
type: 'warning',
dismissible: false,
icon: 'fas fa-exclamation-triangle',
title: app.translator.trans('blomstra-search.admin.reindex-required.title'),
},
app.translator.trans('blomstra-search.admin.reindex-required.detail')
);
}
}
app.initializers.add('blomstra-search', () => {
const languages = new Map;
['arabic', 'armenian', 'basque', 'bengali', 'brazilian', 'bulgarian', 'catalan', 'cjk', 'czech',
'danish', 'dutch', 'english', 'estonian', 'finnish', 'french', 'galician', 'german', 'greek',
'hindi', 'hungarian', 'indonesian', 'irish', 'italian', 'latvian', 'lithuanian', 'norwegian',
'persian', 'portuguese', 'romanian', 'russian', 'sorani', 'spanish', 'swedish', 'turkish', 'thai']
.forEach(language => {
const activeIndex = app.data.settings['blomstra-search.active-index'];
const compatVersion = app.data.settings['blomstra-search.index-compatible'];
if (activeIndex && compatVersion !== REQUIRED_INDEX_COMPAT) {
extend(DashboardPage.prototype, 'availableWidgets', function (items) {
items.add('blomstra-search-reindex', m(ReindexWarningWidget), 110);
});
}
const languages = new Map();
[
'arabic',
'armenian',
'basque',
'bengali',
'brazilian',
'bulgarian',
'catalan',
'cjk',
'czech',
'danish',
'dutch',
'english',
'estonian',
'finnish',
'french',
'galician',
'german',
'greek',
'hindi',
'hungarian',
'indonesian',
'irish',
'italian',
'latvian',
'lithuanian',
'norwegian',
'persian',
'portuguese',
'romanian',
'russian',
'sorani',
'spanish',
'swedish',
'turkish',
'thai',
].forEach((language) => {
languages.set(language, language);
});
app.extensionData
.for('blomstra-search')
.registerSetting(function (this: any) {
// No index yet, or first build predates this tracking — stay silent.
const indexedAnalyzer = app.data.settings['blomstra-search.indexed-analyzer'];
if (!activeIndex || !indexedAnalyzer) return null;
const currentAnalyzer = this.setting('blomstra-search.analyzer-language')() || 'english';
const currentMinLength = this.setting('blomstra-search.min-search-length')();
const indexedMinLength = String(
app.data.settings['blomstra-search.indexed-min-search-length'] || app.data.settings['blomstra-search.min-search-length']
);
if (currentAnalyzer === indexedAnalyzer && currentMinLength === indexedMinLength) return null;
return m(
Alert,
{ type: 'warning', dismissible: false, icon: 'fas fa-exclamation-triangle' },
app.translator.trans('blomstra-search.admin.index-settings-changed')
);
})
.registerSetting({
setting: 'blomstra-search.elastic-endpoint',
label: app.translator.trans('blomstra-search.admin.elastic-endpoint'),
type: 'input'
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.elastic-username',
label: app.translator.trans('blomstra-search.admin.elastic-username'),
type: 'input'
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.elastic-password',
label: app.translator.trans('blomstra-search.admin.elastic-password'),
type: 'password'
type: 'password',
})
.registerSetting({
setting: 'blomstra-search.elastic-index',
label: app.translator.trans('blomstra-search.admin.elastic-index'),
default: 'flarum',
type: 'input'
type: 'input',
})
.registerSetting({
setting: 'blomstra-search.analyzer-language',
@ -41,4 +126,22 @@ app.initializers.add('blomstra-search', () => {
options: Object.fromEntries(languages.entries()),
default: 'english',
})
.registerSetting({
setting: 'blomstra-search.search-discussion-subjects',
label: app.translator.trans('blomstra-search.admin.search-discussion-subjects'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.search-post-bodies',
label: app.translator.trans('blomstra-search.admin.search-post-bodies'),
type: 'switch',
})
.registerSetting({
setting: 'blomstra-search.min-search-length',
label: app.translator.trans('blomstra-search.admin.min-search-length.label'),
help: app.translator.trans('blomstra-search.admin.min-search-length.help'),
type: 'select',
options: { '1': '1', '2': '2', '3': '3', '4': '4' },
default: app.data.settings['blomstra-search.min-search-length'],
});
});

View File

@ -6,9 +6,10 @@ import DiscussionListState from 'flarum/forum/states/DiscussionListState';
export default function extendDiscussionState() {
override(DiscussionListState.prototype, 'loadPage', async function (this: DiscussionListState, original, page: number = 1) {
console.log(this.requestParams());
const preloaded = app.data.apiDocument || null;
if (!this.requestParams()?.filter?.q) return original.call(this, page);
// If existing payload is given or no search is made, fallback on native page.
if (preloaded || !this.requestParams()?.filter?.q) return original.call(this, page);
const params = this.requestParams();
params.page = {
@ -27,8 +28,6 @@ export default function extendDiscussionState() {
const results = await app.request({ params, url, method: 'GET' });
// Parse API response into models and push to store
const models = app.store.pushPayload(results);
return models;
return app.store.pushPayload(results);
});
}

View File

@ -44,7 +44,7 @@ export default class DiscussionsSearchSource implements SearchSource {
this.results.set(query, models);
}
view(query: string): Mithril.Children {
view(query: string): Array<Mithril.Vnode> {
query = query.toLowerCase();
// Get results from map
@ -54,7 +54,7 @@ export default class DiscussionsSearchSource implements SearchSource {
const mostRelevantPost = discussion.mostRelevantPost();
return (
<li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`} key={`${this.type}${discussion.id()}`}>
<li className="DiscussionSearchResult" data-index={`${this.type}${discussion.id()}`}>
<Link href={app.route.discussion(discussion, mostRelevantPost && mostRelevantPost.number())}>
<div className="DiscussionSearchResult-title">{highlight(discussion.title(), query)}</div>
{!!mostRelevantPost && <div className="DiscussionSearchResult-excerpt">{highlight(mostRelevantPost.contentPlain(), query, 100)}</div>}
@ -63,16 +63,14 @@ export default class DiscussionsSearchSource implements SearchSource {
);
});
return (
<>
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>
return [
<li className="Dropdown-header">{app.translator.trans('core.forum.search.discussions_heading')}</li>,
<li>
<LinkButton icon="fas fa-search" href={app.route('index', { q: query })}>
{app.translator.trans('core.forum.search.all_discussions_button', { query })}
</LinkButton>
</li>
{results}
</>
);
</li>,
...results,
];
}
}

View File

@ -1,6 +1,6 @@
import app from 'flarum/forum/app';
import Search from 'flarum/forum/components/Search';
import Search, { SearchAttrs, SearchSource } from 'flarum/forum/components/Search';
import { extend } from 'flarum/common/extend';
import ItemList from 'flarum/common/utils/ItemList';
@ -9,14 +9,24 @@ import DiscussionsSearchSource from './SearchSources/DiscussionsSearchSource';
import extendDiscussionState from './PaginatedListStates/extendDiscussionState';
app.initializers.add('blomstra-search', () => {
extend(Search.prototype, 'sourceItems', function (this: Search, items: ItemList) {
items.remove('users');
items.remove('discussions');
const minLength = parseInt(app.data.settings['blomstra-search.min-search-length'] || String(Search.MIN_SEARCH_LEN), 10);
if (minLength !== Search.MIN_SEARCH_LEN) {
// Flarum provides no extension point for MIN_SEARCH_LEN, so we overwrite the
// static property directly. TypeScript `readonly` is compile-time only — at
// runtime this is a plain property assignment and is safe as long as no code
// reads MIN_SEARCH_LEN before this initializer runs.
(Search as any).MIN_SEARCH_LEN = minLength;
}
/* if (app.forum.attribute('canViewForum')) */ items.add('discussions', new DiscussionsSearchSource());
// console.log(items);
extend(Search.prototype, 'sourceItems', function (this: Search<SearchAttrs>, items: ItemList<SearchSource>) {
items.replace('discussions', new DiscussionsSearchSource());
});
extendDiscussionState();
});
app.initializers.add(
'blomstra-search-early',
() => {
extendDiscussionState();
},
999999
);

View File

@ -0,0 +1,3 @@
.ReindexWarningWidget {
padding: 0;
}

View File

@ -8,4 +8,18 @@ blomstra-search:
label: Analyzer language
help: |
The analyzer makes search understand stop words and undertakes language
specific improvements for indexing.
specific improvements for indexing. Changing this requires a full index rebuild to take effect.
index-settings-changed: "Search index settings have changed. Run: php flarum blomstra:search:index build"
reindex-required:
title: Search index rebuild required
detail: "The search index is not compatible with this version of the extension. Run: php flarum blomstra:search:index build"
search-discussion-subjects: Search inside discussion titles
search-post-bodies: Search inside comments
match-sentences: Match search term against full sentence
match-words: Match search term against full words
min-search-length:
label: Minimum search query length
help: |
Minimum number of characters required before a search is triggered. Lower this to 1 or 2
for CJK (Chinese, Japanese, Korean) communities where single characters carry full meaning.
Changing this requires a full index rebuild to take effect.

28
src/Api/Client.php Normal file
View File

@ -0,0 +1,28 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Api;
use Illuminate\Support\Arr;
use Psr\Http\Message\ResponseInterface;
class Client extends \Flarum\Api\Client
{
public function get(string $path): ResponseInterface
{
if ($path === '/discussions' && Arr::has($this->queryParams, 'filter.q')) {
return parent::get('/blomstra/search/discussions');
}
return parent::get($path);
}
}

View File

@ -1,5 +1,15 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Api\Controllers;
use Blomstra\Search\Commands\BuildCommand;
@ -23,7 +33,8 @@ class IndexController implements RequestHandlerInterface
$command->run(
new ArrayInput([]),
new ConsoleOutput);
new ConsoleOutput()
);
return new EmptyResponse();
}

View File

@ -1,9 +1,24 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Api\Controllers;
use Blomstra\Search\Save\Document as ElasticDocument;
use Blomstra\Search\Elasticsearch\HasChildQuery;
use Blomstra\Search\Elasticsearch\MatchPhraseQuery;
use Blomstra\Search\Elasticsearch\MatchQuery;
use Blomstra\Search\Elasticsearch\TermsQuery;
use Blomstra\Search\Searchers\CommentPostSearcher;
use Blomstra\Search\Searchers\DiscussionSearcher;
use Blomstra\Search\Searchers\Searcher;
use Elasticsearch\Client;
use Flarum\Api\Controller\ListDiscussionsController;
use Flarum\Api\Serializer\DiscussionSerializer;
@ -11,15 +26,18 @@ use Flarum\Discussion\Discussion;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Http\RequestUtil;
use Flarum\Http\UrlGenerator;
use Flarum\Settings\SettingsRepositoryInterface;
use Flarum\Tags\Tag;
use Flarum\User\User;
use Illuminate\Contracts\Container\Container;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Psr\Http\Message\ServerRequestInterface;
use Psr\Log\LoggerInterface;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
use Spatie\ElasticsearchQueryBuilder\Sorts\Sort;
use Tobscure\JsonApi\Document;
@ -30,88 +48,133 @@ class SearchController extends ListDiscussionsController
protected array $translateSort = [
'lastPostedAt' => 'updated_at',
'createdAt' => 'created_at'
'createdAt' => 'created_at',
'commentCount' => 'comment_count',
'view_count' => 'view_count',
];
public function __construct(protected Client $elastic)
{}
protected bool $matchSentences;
protected bool $matchWords;
protected ?Searcher $discussionSearcher;
protected ?Searcher $postSearcher;
public function __construct(protected Client $elastic, protected UrlGenerator $uri, Container $container, SettingsRepositoryInterface $settings)
{
$this->matchSentences = true;
$this->matchWords = true;
$searchers = collect($container->tagged('blomstra.search.searchers'));
$this->discussionSearcher = $searchers->first(fn ($s) => $s instanceof DiscussionSearcher);
$this->postSearcher = $searchers->first(fn ($s) => $s instanceof CommentPostSearcher);
}
protected function data(ServerRequestInterface $request, Document $document)
{
$type = Arr::get($request->getQueryParams(), 'type');
$actor = RequestUtil::getActor($request);
$filters = $this->extractFilter($request);
$search = $this->getSearch($filters);
$limit = $this->extractLimit($request);
$offset = $this->extractOffset($request);
$include = array_merge($this->extractInclude($request), ['state']);
$filterQuery = (BoolQuery::create())
->add(
BoolQuery::create()
->add(MatchQuery::create('content', $filters['q']))
);
$query = BoolQuery::create()
// Always restrict to discussion documents; posts are only searched via has_child.
->add(TermQuery::create('join_field', 'discussion'), 'filter');
if (!empty($search)) {
$query->add($this->buildTextQuery($search, $actor));
}
$this->addFilters($query, $actor, $filters);
$builder = (new Builder($this->elastic))
->index(resolve('blomstra.search.elastic_index'))
->size($this->extractLimit($request))
->from($this->extractOffset($request))
->addQuery(
$this->addFilters($filterQuery, $actor)
);
->size($limit + 1)
->from($offset)
->addQuery($query);
$knownSortFields = array_merge(array_values($this->translateSort), ['rawId']);
$logger = resolve(LoggerInterface::class);
$phpSortField = null;
$phpSortDir = 'desc';
foreach ($this->extractSort($request) as $field => $direction) {
$field = $this->translateSort[$field] ?? $field;
$builder->addSort(new Sort($field, $direction));
$translated = $this->translateSort[$field] ?? $field;
if (!in_array($translated, $knownSortFields)) {
$logger->warning("blomstra/search: unknown sort field \"{$field}\", ignoring.");
continue;
}
$builder->addSort(new Sort($translated, $direction));
if ($phpSortField === null && $translated !== 'rawId') {
$phpSortField = $translated;
$phpSortDir = $direction;
}
}
$response = $builder->search();
Discussion::setStateUser($actor);
// Eager load groups for use in the policies (isAdmin check)
if (in_array('mostRelevantPost.user', $include)) {
$include[] = 'mostRelevantPost.user.groups';
// If the first level of the relationship wasn't explicitly included,
// add it so the code below can look for it
if (! in_array('mostRelevantPost', $include)) {
if (!in_array('mostRelevantPost', $include)) {
$include[] = 'mostRelevantPost';
}
}
// we need to retrieve all discussion ids and when the results are posts,
// their ids as most relevant post id
// All hits are discussion documents. Extract the best-matching post ID from
// inner_hits when a has_child clause matched (i.e. the match came from a post).
$results = Collection::make(Arr::get($response, 'hits.hits'))
->map(function ($hit) {
$id = Str::after($hit['_source']['id'], ':');
$type = $hit['_source']['type'];
// _id is "discussions:123" — parse the numeric part directly.
$discussionId = Str::after($hit['_id'], 'discussions:');
// rawId on the inner hit gives us the integer post ID.
$bestPostId = Arr::get($hit, 'inner_hits.best_post.hits.hits.0._source.rawId');
if ($type === 'posts') {
return [
'most_relevant_post_id' => $id,
'discussion_id' => $discussionId,
'most_relevant_post_id' => $bestPostId,
'weight' => Arr::get($hit, 'sort.0', Arr::get($hit, '_score', 0)),
];
} else {
return [
'discussion_id' => $id
];
}
});
$document->addPaginationLinks(
$this->uri->to('api')->route('blomstra.search', ['type' => 'discussions']),
$request->getQueryParams(),
$offset,
$limit,
$results->count() > $limit ? null : 0
);
$results = $results->take($limit);
$discussions = Discussion::query()
->whereIn('id', $results->pluck('discussion_id'))
->orWhereHas('posts', function ($query) use ($results) {
$query->whereIn('id', $results->pluck('most_relevant_post_id'));
})
->when(
$actor->isGuest() || !$actor->hasPermission('discussion.hide'),
fn ($q) => $q->whereNull('hidden_at')
)
->whereIn('id', $results->pluck('discussion_id')->filter())
->get()
->each(function (Discussion $discussion) use ($results) {
if (in_array($discussion->id, $results->pluck('discussion_id')->toArray())) {
$discussion->most_relevant_post_id = $discussion->first_post_id;
} else {
$post = $discussion->posts()->whereIn('id', $results->pluck('most_relevant_post_id'))->first();
$discussion->most_relevant_post_id = $post?->id ?? $discussion->first_post_id;
}
$result = $results->firstWhere('discussion_id', $discussion->id);
$discussion->most_relevant_post_id = $result['most_relevant_post_id']
?? $discussion->first_post_id;
$discussion->weight = $result['weight'] ?? 0;
})
->keyBy('id')
->when(
$phpSortField,
fn ($c) => $phpSortDir === 'desc' ? $c->sortByDesc($phpSortField) : $c->sortBy($phpSortField),
fn ($c) => $c->sortByDesc('weight')
)
->unique();
$this->loadRelations($discussions, $include);
@ -129,56 +192,123 @@ class SearchController extends ListDiscussionsController
return $discussions;
}
protected function getDocument(string $type): ?ElasticDocument
/**
* Build the text-matching portion of the query.
*
* Discussion titles are matched directly (filtered to join_field=discussion).
* Post bodies are matched via has_child with score_mode=sum so discussions
* with many matching posts score higher than those with a single strong match.
* inner_hits returns the best-scoring post for use as mostRelevantPost.
* Hidden posts are only included in matching for users with post.hide permission.
*/
protected function buildTextQuery(string $search, User $actor): BoolQuery
{
$documents = resolve(Container::class)->tagged('blomstra.search.documents');
$textQuery = BoolQuery::create();
return collect($documents)->first(function (ElasticDocument $document) use ($type) {
return $document->type() === $type;
});
if ($this->discussionSearcher?->enabled()) {
$textQuery->add($this->buildShouldClauses($search, $this->discussionSearcher->boost()), 'should');
}
if ($this->postSearcher?->enabled()) {
$postQuery = $this->buildShouldClauses($search, $this->postSearcher->boost());
// Guests and non-moderators may not see hidden posts; exclude them from child matching.
if ($actor->isGuest() || !$actor->hasPermission('post.hide')) {
$postQuery->add(TermQuery::create('is_hidden', false), 'filter');
}
$textQuery->add(
HasChildQuery::create('post', $postQuery)->withInnerHits(),
'should'
);
}
return $textQuery;
}
protected function buildShouldClauses(string $search, float $boost): BoolQuery
{
$should = BoolQuery::create();
if ($this->matchSentences) {
$should->add((new MatchPhraseQuery('content', $search))->boost(2 * $boost), 'should');
}
if ($this->matchWords) {
$should->add((new MatchQuery('content', $search))->operator('and')->boost(1.8 * $boost), 'should');
$should->add((new MatchQuery('content', $search))->operator('or')->boost(0.8 * $boost), 'should');
}
return $should;
}
protected function extensionEnabled(string $extension): bool
{
/** @var ExtensionManager $manager */
$manager = resolve(ExtensionManager::class);
return $manager->isEnabled($extension);
return resolve(ExtensionManager::class)->isEnabled($extension);
}
protected function addFilters(BoolQuery $query, User $actor): BoolQuery
protected function addFilters(BoolQuery $query, User $actor, array $filters = []): void
{
/** @var Collection $groups */
$groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) $groups->add(Group::MEMBER_ID);
$groups = $this->getGroups($actor);
$onlyPrivate = Str::contains($filters['q'] ?? '', 'is:private');
$subQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'false'))
->add(TermsQuery::create('groups', $groups->toArray()));
if ($this->extensionEnabled('flarum-tags') && !empty($filters['tag'])) {
$slugs = is_array($filters['tag']) ? $filters['tag'] : explode(',', $filters['tag']);
$tagIds = Tag::query()->whereIn('slug', $slugs)->pluck('id')->toArray();
if (!empty($tagIds)) {
$query->add(TermsQuery::create('tags', $tagIds), 'filter');
}
}
if ($this->extensionEnabled('fof-byobu') && $actor->exists) {
$byobuQuery = BoolQuery::create()
->add(TermQuery::create('is_private', 'true'), 'should')
->add(TermQuery::create('is_private', 'true'))
->add(
BoolQuery::create()
->add(TermsQuery::create('recipient-groups', $groups->toArray()))
->add(TermQuery::create('recipient-users', $actor->id)),
'should'
->add(TermsQuery::create('recipient_groups', $groups->toArray()), 'should')
->add(TermsQuery::create('recipient_users', [$actor->id]), 'should'),
);
if ($onlyPrivate) {
$subQuery = $byobuQuery;
} else {
$subQuery = BoolQuery::create()
->add($subQuery, 'should')
->add($byobuQuery, 'should');
}
}
$query->add(
$subQuery,
'filter'
);
$query->add($subQuery, 'filter');
}
return $query;
protected function getGroups(User $actor): Collection
{
$groups = $actor->groups->pluck('id');
$groups->add(Group::GUEST_ID);
if ($actor->is_email_confirmed) {
$groups->add(Group::MEMBER_ID);
}
return $groups;
}
protected function getSearch(array $filters): ?string
{
$search = Arr::get($filters, 'q');
if ($search) {
$q = collect(explode(' ', $search))
->filter(fn (string $part) => $part !== 'is:private')
->filter()
->join(' ');
return empty($q) ? null : $q;
}
return null;
}
}

View File

@ -1,136 +1,583 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Commands;
use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\SavingJob;
use Blomstra\Search\Jobs\UpdateSearchJob;
use Blomstra\Search\Seeders\Seeder;
use Carbon\Carbon;
use Elasticsearch\Client;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Console\Command;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Arr;
use Spatie\ElasticsearchQueryBuilder\Builder;
use Spatie\ElasticsearchQueryBuilder\Queries\BoolQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\RangeQuery;
use Spatie\ElasticsearchQueryBuilder\Queries\TermQuery;
class BuildCommand extends Command
{
/**
* Bump this when a mapping change requires a full reindex.
* Written into the mapping's _meta.index_compat_version on every build and
* read back into blomstra-search.index-compatible by saveIndexedConfig so the
* value always reflects the live index including after a promote/rollback.
*/
public const INDEX_COMPAT_VERSION = 'v2';
/** Matches Flarum's Search::MIN_SEARCH_LEN — the default minimum query length. */
public const DEFAULT_MIN_SEARCH_LENGTH = 3;
protected $signature = 'blomstra:search:index
{--max-id= : Limits for each object the number of items to seed}
{--chunk-size= : Size of the chunks to dispatch into jobs}
{--throttle= : Number of seconds to wait between pushing to the queue}
{--only= : type to run seeder for, eg discussions or posts}
{--recreate : create or recreate the index}
{--continue : continue each object type where you left off}';
protected $description = 'Rebuilds the complete search server with its documents.';
{action? : build | promote | rollback | discard | mapping | fill}
{--fresh}
{--resume}
{--staging}
{--keep-backup}
{--pending}
{--backup}
{--only=}
{--max-id=}
{--throttle=}
{--i-am-sure}';
public function handle(Container $container)
protected $description = 'Build and manage the Elasticsearch search index.';
protected $help = <<<'HELP'
<comment>Actions and their options:</comment>
<info>build</info> Queue documents and promote automatically.
<comment>--resume</comment> Resume an interrupted build from where it left off
<comment>--fresh</comment> Drop the staging index and start completely fresh
<comment>--staging</comment> Keep in staging requires explicit <info>promote</info> (blue-green)
<comment>--keep-backup</comment> Retain the replaced index for rollback
<info>promote</info> Swap alias to staging index (blue-green workflow).
<comment>--keep-backup</comment> Retain the replaced index for rollback
<comment>--i-am-sure</comment> Skip the confirmation prompt
<info>rollback</info> Restore the backup index to live.
<info>discard</info> Drop an index without promoting.
<comment>--pending</comment> Drop the staging index (cancel a build)
<comment>--backup</comment> Drop the backup index (cleanup after --keep-backup)
<info>mapping</info> Push updated mapping to the live index only.
<info>fill</info> Seed only documents missing from the live index.
<comment>Shared seeding options (build / fill):</comment>
<comment>--only=TYPE</comment> Seed only this type: <info>discussions</info> or <info>posts</info>
<comment>--throttle=N</comment> Seconds to wait between batches
<comment>--max-id=N</comment> Limit seeding to IDs up to this value
HELP;
public function handle(Container $container): void
{
$index = $container->make('blomstra.search.elastic_index');
/** @var array $seeders */
$seeders = $container->tagged('blomstra.search.seeders');
/** @var Queue $queue */
$queue = $container->make(Queue::class);
/** @var Client $client */
$client = $container->make(Client::class);
/** @var SettingsRepositoryInterface $settings */
$settings = $container->make(SettingsRepositoryInterface::class);
$properties = [
'properties' => [
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'is_sticky' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'],
]
];
/** @var string $alias */
$alias = $container->make('blomstra.search.elastic_index');
if ($this->option('recreate')) {
// Flush the index.
$client->indices()->delete([
'index' => $index,
'ignore_unavailable' => true
]);
// Create a new index.
$client->indices()->create([
'index' => $index,
'body' => [
'settings' => [
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $settings->get('blomstra-search.analyzer-language') ?: 'english'
]
]
]
]
]
]);
$client->indices()->putMapping([
'index' => $index,
'body' => $properties
]);
if (!$this->argument('action')) {
$this->call('help', ['command_name' => $this->getName()]);
return;
}
switch ($this->argument('action')) {
case 'build':
$this->runBuild($client, $alias, $settings, $container);
break;
case 'promote':
$this->runPromote($client, $alias, $settings);
break;
case 'rollback':
$this->runRollback($client, $alias, $settings);
break;
case 'discard':
$this->runDiscard($client, $settings);
break;
case 'mapping':
$client->indices()->putMapping([
'index' => $alias,
'body' => $this->mappingProperties(),
]);
$this->info('Mapping updated on live index.');
break;
case 'fill':
$this->runSeeders(
collect($container->tagged('blomstra.search.seeders')),
$container->make(Queue::class),
$client,
$settings,
$alias,
seedMissing: true
);
break;
default:
$this->error("Unknown action '{$this->argument('action')}'. Valid actions: build, promote, rollback, discard, mapping, fill.");
}
}
protected function runBuild(Client $client, string $alias, SettingsRepositoryInterface $settings, Container $container): void
{
/** @var Seeder[] $seeders */
$seeders = collect($container->tagged('blomstra.search.seeders'));
// Guard: staging build exists without explicit intent given.
$staging = $settings->get('blomstra-search.staging-index');
if ($staging && $client->indices()->exists(['index' => $staging])
&& !$this->option('resume') && !$this->option('fresh')
) {
$this->warn("An in-progress build already exists: $staging");
$this->line('');
$this->line('Choose one of:');
$this->line(' blomstra:search:index build --resume Continue from where it left off');
$this->line(' blomstra:search:index build --fresh Drop this build and start completely fresh');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if (!$aliasExists && !$indexExists) {
// First install: create a timestamped index, alias it immediately, seed live.
$targetIndex = $this->prepareFirstInstall($client, $alias, $settings, $seeders);
$stagingBuild = false;
} else {
// Live system: build into a staging index without touching the live alias.
$targetIndex = $this->prepareStagingIndex($client, $alias, $settings, $seeders);
$stagingBuild = true;
}
$client->indices()->putMapping([
'index' => $targetIndex,
'body' => $this->mappingProperties(),
]);
$this->runSeeders($seeders, $container->make(Queue::class), $client, $settings, $targetIndex);
if ($stagingBuild) {
if ($this->option('staging')) {
$staging = $settings->get('blomstra-search.staging-index');
$this->info("Staging build complete. Drain the queue, then promote '$staging' to live:");
$this->line(' php flarum queue:work --stop-when-empty');
$this->line(' php flarum blomstra:search:index promote');
} else {
$this->info('Jobs queued. Promoting now (search improves as the queue drains)...');
$this->performPromote($client, $alias, $settings);
}
}
}
protected function runPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
if (!$this->option('i-am-sure')) {
$this->warn('promote switches the live index. Ensure all queued jobs have finished first:');
$this->line(' php flarum queue:work --stop-when-empty');
$this->line('');
if (!$this->confirm('Queue drained? Proceed?')) {
return;
}
}
$this->performPromote($client, $alias, $settings);
}
protected function performPromote(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging || !$client->indices()->exists(['index' => $staging])) {
$this->error('No staging index ready to promote. Run: blomstra:search:index build');
return;
}
$aliasExists = (bool) $client->indices()->existsAlias(['name' => $alias]);
$indexExists = !$aliasExists && (bool) $client->indices()->exists(['index' => $alias]);
if ($aliasExists) {
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $staging, 'alias' => $alias]],
]],
]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
if ($this->option('keep-backup')) {
$settings->set('blomstra-search.backup-index', $activeIndex);
$this->info("Kept '$activeIndex' as backup — run rollback to revert, discard --backup to drop it.");
} else {
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Deleted old index: $activeIndex");
}
} else {
// One-time migration: concrete index → alias (legacy installs only).
if ($indexExists) {
$client->indices()->delete(['index' => $alias]);
$this->info("Deleted legacy concrete index: $alias");
}
$client->indices()->putAlias(['index' => $staging, 'name' => $alias]);
$this->info("'$alias' now points to '$staging'. Promote complete.");
}
$settings->set('blomstra-search.active-index', $staging);
$settings->set('blomstra-search.staging-index', null);
$this->saveIndexedConfig($client, $settings, $staging);
}
protected function runRollback(Client $client, string $alias, SettingsRepositoryInterface $settings): void
{
$backup = $settings->get('blomstra-search.backup-index');
if (!$backup || !$client->indices()->exists(['index' => $backup])) {
$this->error('No backup index to roll back to. Was promote run with --keep-backup?');
return;
}
$result = $client->indices()->getAlias(['name' => $alias]);
$activeIndex = array_key_first($result);
$client->indices()->updateAliases([
'body' => ['actions' => [
['remove' => ['index' => $activeIndex, 'alias' => $alias]],
['add' => ['index' => $backup, 'alias' => $alias]],
]],
]);
$client->indices()->delete(['index' => $activeIndex, 'ignore_unavailable' => true]);
$this->info("Rolled back: '$alias' now points to '$backup'. Deleted '$activeIndex'.");
$settings->set('blomstra-search.active-index', $backup);
$settings->set('blomstra-search.backup-index', null);
$this->saveIndexedConfig($client, $settings, $backup);
}
protected function runDiscard(Client $client, SettingsRepositoryInterface $settings): void
{
$pending = $this->option('pending');
$backup = $this->option('backup');
if (!$pending && !$backup) {
$this->error('Specify what to discard: --pending (staging build) or --backup (rollback index).');
return;
}
if ($pending) {
$staging = $settings->get('blomstra-search.staging-index');
if (!$staging) {
$this->warn('No staging index to discard.');
} else {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
}
$settings->set('blomstra-search.staging-index', null);
$this->info("Discarded staging index: $staging");
}
}
if ($backup) {
$backupIndex = $settings->get('blomstra-search.backup-index');
if (!$backupIndex) {
$this->warn('No backup index to discard.');
} else {
if ($client->indices()->exists(['index' => $backupIndex])) {
$client->indices()->delete(['index' => $backupIndex]);
}
$settings->set('blomstra-search.backup-index', null);
$this->info("Discarded backup index: $backupIndex");
}
}
}
protected function runSeeders(
iterable $seeders,
Queue $queue,
Client $client,
SettingsRepositoryInterface $settings,
string $targetIndex,
bool $seedMissing = false
): void {
$only = $this->option('only');
/** @var Seeder $seeder */
foreach ($seeders as $seeder) {
if ($only && $seeder->type() !== $only) continue;
$seeder->query()
->when($this->option('max-id'), function ($query, $id) {
$query->where('id', '<=', $id);
})
->when($this->option('continue'), function ($query) use ($seeder) {
if ($continueAt = $this->continueAt($seeder->type())) {
$query->where('id', '<=', $continueAt);
if ($only && $seeder->type() !== $only) {
continue;
}
})
->latest('id')
->chunk($this->option('chunk-size') ?? 1000, function (Collection $collection) use ($queue, &$total, $seeder) {
$queue->pushOn(Job::$onQueue, new SavingJob($collection, $seeder));
$this->info("Pushed into the index, type: {$seeder->type()}, amount: {$collection->count()}.");
$total = 0;
if ($this->option('resume')) {
$saved = $this->getContinueAt($settings, $seeder->type());
if ($saved === 0) {
$this->info("Seeder '{$seeder->type()}' already completed in a previous run, skipping.");
continue;
}
$continueAt = $saved ?? $seeder->query()->max('id');
} else {
$continueAt = $seeder->query()->max('id');
}
$seeded = null;
while ($continueAt !== null) {
$rangeFrom = max(1, $continueAt - 2500);
$rangeTo = $continueAt;
if ($seedMissing) {
$response = (new Builder($client))
->index($targetIndex)
->size(2500)
->addQuery(
(new BoolQuery())
->add((new RangeQuery('rawId'))->gte($rangeFrom)->lte($rangeTo))
->add(TermQuery::create('join_field', $seeder->joinRelation()))
)
->search();
$seeded = Arr::pluck(Arr::get($response, 'hits.hits'), '_source.rawId');
}
/** @var Collection $collection */
$collection = $seeder->query()
->latest('id')
->whereBetween('id', [$rangeFrom, $rangeTo])
->when($this->option('max-id'), fn ($q, $id) => $q->where('id', '<=', $id))
->when($seeded, fn ($q, $seeded) => $q->whereNotIn('id', $seeded))
->get();
$min = $collection->min('id');
if ($seedMissing && $collection->isEmpty()) {
$continueAt = $rangeFrom > 2 ? $rangeFrom - 1 : null;
} else {
$continueAt = $min && $min > 2 ? $min - 1 : null;
}
if ($collection->isNotEmpty()) {
$queue->pushOn(Job::$onQueue, new UpdateSearchJob($collection, $seeder, $targetIndex));
}
$this->info("IDs {$rangeFrom}{$rangeTo} | type: {$seeder->type()} | queued: {$collection->count()}.");
$total += $collection->count();
$this->continueAt(
$seeder->type(),
$collection->min('id')
);
});
$this->info("Pushed a total of $total into the index.");
$this->setContinueAt($settings, $seeder->type(), $continueAt);
if ($throttle = $this->option('throttle')) {
$this->info("Throttling for $throttle seconds");
sleep($throttle);
}
}
$this->setContinueAt($settings, $seeder->type(), 0);
$this->info("Queued a total of $total {$seeder->type()} for indexing.");
}
}
protected function continueAt(string $type, int $at = null)
/**
* First install: create a timestamped concrete index, alias the configured name to it
* immediately, and return the alias so seeding goes through it and is live from the start.
*/
protected function prepareFirstInstall(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$concrete = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $concrete,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$client->indices()->putAlias(['index' => $concrete, 'name' => $alias]);
$settings->set('blomstra-search.active-index', $concrete);
$this->saveIndexedConfig($client, $settings, $concrete);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created '$concrete', aliased '$alias' → '$concrete'.");
$this->info("Index is live — documents become searchable as the queue processes.");
return $alias;
}
/**
* Prepare the staging index for a blue-green build.
*
* - If a staging build exists and --fresh is not set, resume it.
* - Otherwise create a fresh timestamped index and save it as staging.
*/
protected function prepareStagingIndex(
Client $client,
string $alias,
SettingsRepositoryInterface $settings,
iterable $seeders
): string {
$staging = $settings->get('blomstra-search.staging-index');
if ($this->option('fresh') && $staging) {
if ($client->indices()->exists(['index' => $staging])) {
$client->indices()->delete(['index' => $staging]);
$this->info("Dropped staging index: $staging");
}
$staging = null;
$settings->set('blomstra-search.staging-index', null);
}
if ($staging && $client->indices()->exists(['index' => $staging])) {
$this->info("Resuming staging index build: $staging");
return $staging;
}
$staging = $alias . '_' . date('YmdHis');
$client->indices()->create([
'index' => $staging,
'body' => ['settings' => $this->buildIndexSettings($settings)],
]);
$settings->set('blomstra-search.staging-index', $staging);
foreach ($seeders as $seeder) {
$this->setContinueAt($settings, $seeder->type(), null);
}
$this->info("Created staging index: $staging");
return $staging;
}
/**
* Persist the analysis config and compat version that are actually live in ES for the
* given index. Reading from ES (rather than from Flarum settings) means rollbacks are
* also covered: the stored values always reflect the index that is currently aliased,
* not the settings at the time the command ran. Indexes built before _meta tracking
* existed will yield a null compat version, which correctly triggers the reindex warning.
*/
protected function saveIndexedConfig(Client $client, SettingsRepositoryInterface $settings, string $indexName): void
{
/** @var SettingsRepositoryInterface $settings */
$settings = resolve(SettingsRepositoryInterface::class);
$settingsResponse = $client->indices()->getSettings(['index' => $indexName]);
$analysis = Arr::get($settingsResponse, "$indexName.settings.index.analysis", []);
$key = "blomstra-search.continued-at.$type";
$analyzer = Arr::get($analysis, 'analyzer.flarum_analyzer.type', 'english');
$minGram = (int) Arr::get($analysis, 'filter.partial_search_filter.min_gram', self::DEFAULT_MIN_SEARCH_LENGTH);
if ($at) {
$settings->set($key, $at);
} else {
return $settings->get($key);
$mappingResponse = $client->indices()->getMapping(['index' => $indexName]);
$compatVersion = Arr::get($mappingResponse, "$indexName.mappings._meta.index_compat_version");
$settings->set('blomstra-search.indexed-analyzer', $analyzer);
$settings->set('blomstra-search.indexed-min-search-length', $minGram);
$settings->set('blomstra-search.index-compatible', $compatVersion);
}
protected function buildIndexSettings(SettingsRepositoryInterface $settings): array
{
$language = $settings->get('blomstra-search.analyzer-language') ?: 'english';
$minGram = max(1, (int) ($settings->get('blomstra-search.min-search-length') ?: self::DEFAULT_MIN_SEARCH_LENGTH));
$maxGram = 10;
if ($minGram >= $maxGram) {
$this->error("min_gram ($minGram) must be less than max_gram ($maxGram). Using default.");
$minGram = self::DEFAULT_MIN_SEARCH_LENGTH;
}
return [
'index.max_ngram_diff' => $maxGram - $minGram,
'analysis' => [
'analyzer' => [
'flarum_analyzer' => [
'type' => $language,
],
'flarum_analyzer_partial' => [
'type' => 'custom',
'tokenizer' => 'standard',
'filter' => ['lowercase', 'partial_search_filter'],
],
],
'filter' => [
'partial_search_filter' => [
'type' => 'ngram',
'min_gram' => $minGram,
'max_gram' => $maxGram,
'token_chars' => ['letter', 'digit', 'symbol'],
],
],
],
];
}
protected function mappingProperties(): array
{
return [
'_meta' => ['index_compat_version' => self::INDEX_COMPAT_VERSION],
'properties' => [
'join_field' => ['type' => 'join', 'relations' => ['discussion' => 'post']],
'discussion_id' => ['type' => 'integer'],
'content' => ['type' => 'text', 'analyzer' => 'flarum_analyzer_partial', 'search_analyzer' => 'flarum_analyzer'],
'rawId' => ['type' => 'integer'],
'created_at' => ['type' => 'date'],
'updated_at' => ['type' => 'date'],
'is_private' => ['type' => 'boolean'],
'is_sticky' => ['type' => 'boolean'],
'groups' => ['type' => 'integer'],
'tags' => ['type' => 'integer'],
'recipient_groups' => ['type' => 'integer'],
'recipient_users' => ['type' => 'integer'],
'comment_count' => ['type' => 'integer'],
'view_count' => ['type' => 'integer'],
'is_hidden' => ['type' => 'boolean'],
],
];
}
protected function getContinueAt(SettingsRepositoryInterface $settings, string $type): ?int
{
$raw = $settings->get("blomstra-search.continued-at.$type");
return $raw !== null ? (int) $raw : null;
}
protected function setContinueAt(SettingsRepositoryInterface $settings, string $type, ?int $at): void
{
$settings->set("blomstra-search.continued-at.$type", $at);
}
}

View File

@ -0,0 +1,57 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class HasChildQuery implements Query
{
protected bool $innerHits = false;
public function __construct(
protected string $type,
protected Query $query,
protected string $scoreMode = 'sum'
) {}
public static function create(string $type, Query $query, string $scoreMode = 'sum'): static
{
return new static($type, $query, $scoreMode);
}
/**
* Include inner_hits so the best-matching post ID is available in the response.
* ES returns the top-scoring child document per parent hit under inner_hits.best_post.
*/
public function withInnerHits(): static
{
$this->innerHits = true;
return $this;
}
public function toArray(): array
{
$hasChild = [
'type' => $this->type,
'score_mode' => $this->scoreMode,
'query' => $this->query->toArray(),
];
if ($this->innerHits) {
$hasChild['inner_hits'] = ['name' => 'best_post', 'size' => 1];
}
return ['has_child' => $hasChild];
}
}

View File

@ -0,0 +1,38 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery;
class MatchPhraseQuery extends MatchQuery
{
protected float $boost = 1;
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function toArray(): array
{
$query = parent::toArray()['match'];
$query[$this->field]['boost'] = $this->boost;
return [
'match_phrase' => $query,
];
}
}

View File

@ -0,0 +1,67 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
class MatchQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\MatchQuery
{
protected string $operator = 'or';
protected float $boost = 1;
protected ?string $analyzer = null;
protected bool $zeroTerms = false;
public function and()
{
$this->operator = 'and';
return $this;
}
public function operator(string $operator)
{
$this->operator = $operator;
return $this;
}
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function analyzer(string $analyzer)
{
$this->analyzer = $analyzer;
return $this;
}
public function zeroTerms(bool $zeroTerms = true)
{
$this->zeroTerms = $zeroTerms;
return $this;
}
public function toArray(): array
{
$query = parent::toArray();
$query['match'][$this->field]['operator'] = $this->operator;
$query['match'][$this->field]['boost'] = $this->boost;
$query['match'][$this->field]['zero_terms_query'] = $this->zeroTerms ? 'all' : 'none';
return $query;
}
}

View File

@ -0,0 +1,59 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;
class SimpleSearchQuery implements Query
{
protected float $boost = 1;
protected ?string $analyzer = null;
public static function create(array $field, string $value)
{
return new self($field, $value);
}
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function analyzer(string $analyzer)
{
$this->analyzer = $analyzer;
return $this;
}
public function __construct(
protected array $fields,
protected string $value
) {
}
public function toArray(): array
{
return [
'simple_query_string' => [
'query' => $this->value,
'fields' => $this->fields,
'analyzer' => $this->analyzer,
'default_operator' => 'AND',
'boost' => $this->boost,
],
];
}
}

View File

@ -1,5 +1,15 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
use Spatie\ElasticsearchQueryBuilder\Queries\Query;

View File

@ -0,0 +1,55 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Elasticsearch;
class WildcardQuery extends \Spatie\ElasticsearchQueryBuilder\Queries\WildcardQuery
{
protected float $boost = 1;
protected bool $sensitivity = true;
protected ?string $rewrite = null;
public function boost(float $boost = 1)
{
$this->boost = $boost;
return $this;
}
public function caseSensitivity(bool $sensitivity = true)
{
$this->sensitivity = $sensitivity;
return $this;
}
public function rewrite(string $rewrite = null)
{
$this->rewrite = $rewrite;
return $this;
}
public function toArray(): array
{
$query = parent::toArray();
$query['wildcard'][$this->field]['boost'] = $this->boost;
$query['wildcard'][$this->field]['case_insensitive'] = !$this->sensitivity;
if ($this->rewrite) {
$query['wildcard'][$this->field]['rewrite'] = $this->rewrite;
}
return $query;
}
}

View File

@ -1,13 +1,22 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Exceptions;
use Throwable;
class SeedingException extends \Exception
{
public function __construct($message = "", public array $items, $code = 0, Throwable $previous = null)
public function __construct($message = '', public array $items, $code = 0, Throwable $previous = null)
{
parent::__construct($message, $code, $previous);
}

View File

@ -1,5 +1,15 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs;
use Elasticsearch\Client;
@ -9,21 +19,24 @@ class DeletingJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) return;
if ($this->models->isEmpty()) {
return;
}
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [
['delete' => ['_index' => $this->index, '_id' => $document->id]]
['delete' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
];
})->flatten(1);
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
'refresh' => true
'refresh' => true,
]);
}
}

View File

@ -1,5 +1,15 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs;
use Blomstra\Search\Seeders\Seeder;
@ -12,10 +22,16 @@ abstract class Job extends AbstractJob
public static ?string $onQueue = null;
public function __construct(protected Collection $models, protected Seeder $seeder)
/**
* @param string|null $targetIndex Explicit index name for blue-green builds.
* Defaults to the configured alias when null.
*/
public function __construct(protected Collection $models, protected Seeder $seeder, ?string $targetIndex = null)
{
$this->index = resolve('blomstra.search.elastic_index');
$this->index = $targetIndex ?? resolve('blomstra.search.elastic_index');
if (static::$onQueue) $this->onQueue(static::$onQueue);
if (static::$onQueue) {
$this->onQueue(static::$onQueue);
}
}
}

View File

@ -1,5 +1,15 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs;
use Blomstra\Search\Exceptions\SeedingException;
@ -7,19 +17,24 @@ use Elasticsearch\Client;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Arr;
class SavingJob extends Job
class UpdateSearchJob extends Job
{
public function handle(Client $client)
{
if ($this->models->isEmpty()) return;
if ($this->models->isEmpty()) {
return;
}
$this->models->loadMissing($this->seeder->relationships());
// Preparing body for storing.
$body = $this->models->map(function (Model $model) {
$document = $this->seeder->toDocument($model);
$routing = $this->seeder->routing($model);
return [
['index' => ['_index' => $this->index, '_id' => $document->id]],
$document->toArray()
['index' => ['_index' => $this->index, '_id' => $document->id, 'routing' => $routing]],
$document->toArray(),
];
})
->flatten(1);
@ -27,10 +42,11 @@ class SavingJob extends Job
$response = $client->bulk([
'index' => $this->index,
'body' => $body->toArray(),
'refresh' => true
]);
if (Arr::get($response, 'errors') !== true) return true;
if (Arr::get($response, 'errors') !== true) {
return true;
}
$items = Arr::get($response, 'items');

View File

@ -0,0 +1,59 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Jobs;
use Elasticsearch\Client;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Queue\AbstractJob;
class ViewsSearchJob extends AbstractJob
{
protected string $index;
protected string $documentType;
public function __construct(protected int $discussionId)
{
$this->index = resolve('blomstra.search.elastic_index');
$this->documentType = resolve(DiscussionSerializer::class)->getType(new Discussion());
if (Job::$onQueue) {
$this->onQueue(Job::$onQueue);
}
}
public function handle(Client $client): void
{
$discussion = Discussion::find($this->discussionId);
if (!$discussion) {
return;
}
$type = $this->documentType;
try {
$client->update([
'index' => $this->index,
'id' => "$type:{$this->discussionId}",
'routing' => (string) $this->discussionId,
'retry_on_conflict' => 3,
'body' => [
'doc' => ['view_count' => (int) $discussion->view_count],
],
]);
} catch (\Elasticsearch\Common\Exceptions\Missing404Exception $e) {
// Document not yet indexed; will be picked up on next: blomstra:search:index fill
}
}
}

View File

@ -1,19 +1,33 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search;
use Blomstra\Search\Jobs\DeletingJob;
use Blomstra\Search\Jobs\Job;
use Blomstra\Search\Jobs\SavingJob;
use Blomstra\Search\Seeders;
use Elasticsearch\Client;
use Blomstra\Search\Jobs\UpdateSearchJob;
use Blomstra\Search\Jobs\ViewsSearchJob;
use Elasticsearch\Client as Elastic;
use Elasticsearch\ClientBuilder;
use Flarum\Api\Client;
use Flarum\Foundation\AbstractServiceProvider;
use Flarum\Foundation\Config;
use Flarum\Http\Middleware\ExecuteRoute;
use Flarum\Settings\SettingsRepositoryInterface;
use Illuminate\Contracts\Container\Container;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Contracts\Queue\Queue;
use Illuminate\Support\Collection;
use Laminas\Stratigility\MiddlewarePipe;
use Psr\Log\LoggerInterface;
class Provider extends AbstractServiceProvider
@ -28,10 +42,16 @@ class Provider extends AbstractServiceProvider
/** @var SettingsRepositoryInterface $settings */
$settings = $this->container->make(SettingsRepositoryInterface::class);
$this->container->singleton(Client::class, function (Container $container) use ($settings) {
/** @var Config $config */
$config = $this->container->make(Config::class);
$this->container->singleton(Elastic::class, function (Container $container) use ($settings, $config) {
$builder = ClientBuilder::create()
->setHosts([$settings->get('blomstra-search.elastic-endpoint')])
->setLogger($container->make(LoggerInterface::class));
->setHosts([$settings->get('blomstra-search.elastic-endpoint')]);
if ($config->inDebugMode()) {
$builder->setLogger($container->make(LoggerInterface::class));
}
if ($settings->get('blomstra-search.elastic-username')) {
$builder->setBasicAuthentication(
@ -43,11 +63,36 @@ class Provider extends AbstractServiceProvider
return $builder->build();
});
$this->container->instance(
'blomstra.search.elastic_index',
$settings->get('blomstra-search.elastic-index', 'flarum')
);
$this->container->extend(
Client::class,
function () {
$pipe = new MiddlewarePipe();
$exclude = resolve('flarum.api_client.exclude_middleware');
$middlewareStack = array_filter(resolve('flarum.api.middleware'), function ($middlewareClass) use ($exclude) {
return !in_array($middlewareClass, $exclude);
});
foreach ($middlewareStack as $middleware) {
$pipe->pipe(resolve($middleware));
}
$pipe->pipe(new ExecuteRoute());
return new Api\Client($pipe);
}
);
$this->container->tag([
Searchers\DiscussionSearcher::class,
Searchers\CommentPostSearcher::class,
], 'blomstra.search.searchers');
}
public function boot()
@ -64,12 +109,16 @@ class Provider extends AbstractServiceProvider
/** @var string|Seeders\Seeder $seeder */
foreach ($seeders as $seeder) {
$seeder::savingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new SavingJob(Collection::make([$model]), $seeder));
$queue->pushOn(Job::$onQueue, new UpdateSearchJob(Collection::make([$model]), $seeder));
});
$seeder::deletingOn($events, function ($model) use ($queue, $seeder) {
$queue->pushOn(Job::$onQueue, new DeletingJob(Collection::make([$model]), $seeder));
});
$seeder::viewingOn($events, function (int $discussionId) use ($queue) {
$queue->pushOn(Job::$onQueue, new ViewsSearchJob($discussionId));
});
}
}
}

View File

@ -1,13 +1,24 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Save;
use Carbon\Carbon;
use Illuminate\Support\Arr;
use Illuminate\Support\Fluent;
/**
* @property string $type
* @property string $id
* @property int $rawId
* @property string $content
* @property Carbon $created_at
* @property Carbon $updated_at
@ -20,4 +31,13 @@ use Illuminate\Support\Fluent;
*/
class Document extends Fluent
{
/**
* Exclude `id` from the body sent to Elasticsearch. The document ID is
* already stored as `_id` by the bulk API action; duplicating it in
* `_source` wastes space and is redundant.
*/
public function toArray(): array
{
return Arr::except($this->attributes, ['id']);
}
}

View File

@ -0,0 +1,27 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Searchers;
use Blomstra\Search\Seeders\CommentSeeder;
class CommentPostSearcher extends Searcher
{
protected string|null $seeder = CommentSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-post-bodies', true);
return boolval($enabled);
}
}

View File

@ -0,0 +1,32 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Searchers;
use Blomstra\Search\Seeders\DiscussionSeeder;
class DiscussionSearcher extends Searcher
{
protected string|null $seeder = DiscussionSeeder::class;
public function enabled(): bool
{
$enabled = $this->setting('blomstra-search.search-discussion-subjects', true);
return boolval($enabled);
}
public function boost(): float
{
return 1.5;
}
}

View File

@ -0,0 +1,48 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Searchers;
use Blomstra\Search\Seeders\Seeder;
use Flarum\Settings\SettingsRepositoryInterface;
abstract class Searcher
{
protected string|null $seeder = null;
public function type(): string
{
/** @var Seeder $seeder */
$seeder = $this->seeder;
if (empty($seeder)) {
throw new \InvalidArgumentException('Implement type or add $seeder');
}
return (new $seeder())->type();
}
public function enabled(): bool
{
return true;
}
public function boost(): float
{
return 1;
}
protected function setting(string $key, $default = null)
{
return resolve(SettingsRepositoryInterface::class)->get($key, $default);
}
}

View File

@ -1,12 +1,23 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Seeders;
use Blomstra\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Api\Serializer\PostSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Post\CommentPost;
use Flarum\Post\Event\Deleted;
use Flarum\Post\Event\Posted;
use Flarum\Post\Event as Core;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Model;
@ -15,64 +26,71 @@ class CommentSeeder extends Seeder
{
public function type(): string
{
return resolve(PostSerializer::class)->getType(new CommentPost);
return resolve(PostSerializer::class)->getType(new CommentPost());
}
public function joinRelation(): string
{
return 'post';
}
public function routing(Model $model): string
{
return (string) $model->discussion_id;
}
public function query(): Builder
{
$includes = ['discussion'];
if ($this->extensionEnabled('flarum-tags')) {
$includes[] = 'discussion.tags';
}
if ($this->extensionEnabled('fof-byobu')) {
$includes[] = 'discussion.recipientUsers';
$includes[] = 'discussion.recipientGroups';
}
return CommentPost::query()
->whereNull('hidden_at')
->where('type', CommentPost::$type)
->with($includes);
->where('type', CommentPost::$type);
}
public static function savingOn(Dispatcher $events, callable $callable)
{
$events->listen(Posted::class, function (Posted $event) use ($callable) {
$events->listen([
Core\Posted::class,
Core\Revised::class,
Core\Hidden::class,
Core\Restored::class,
], function ($event) use ($callable) {
$callable($event->post);
});
}
public static function deletingOn(Dispatcher $events, callable $callable)
{
$events->listen(Deleted::class, function (Deleted $event) use ($callable) {
$events->listen([
Core\Deleted::class
], function ($event) use ($callable) {
$callable($event->post);
});
}
/** Cached discussion type string (e.g. "discussions") — resolved once per job. */
private ?string $discussionType = null;
private function discussionType(): string
{
return $this->discussionType ??= resolve(DiscussionSerializer::class)->getType(new Discussion());
}
/**
* Post documents only need content for has_child matching and the join
* field to establish the parent-child relationship. All discussion-level
* fields (groups, tags, comment_count, etc.) live on the discussion
* document and are irrelevant here.
*
* @param CommentPost $model
* @return Document
*/
public function toDocument(Model $model): Document
{
$document = new Document([
'type' => $this->type(),
'id' => $this->type() . ':' . $model->id,
return new Document([
'join_field' => ['name' => $this->joinRelation(), 'parent' => "{$this->discussionType()}:{$model->discussion_id}"],
'discussion_id' => $model->discussion_id,
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->content,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->edited_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model->discussion)
'is_hidden' => $model->hidden_at !== null,
]);
if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->discussion->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->discussion->recipientGroups->pluck('id')->toArray();
}
return $document;
}
}

View File

@ -1,26 +1,56 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Seeders;
use Blomstra\Search\Save\Document;
use Flarum\Api\Serializer\DiscussionSerializer;
use Flarum\Discussion\Discussion;
use Flarum\Discussion\Event\Deleted;
use Flarum\Discussion\Event\Hidden;
use Flarum\Discussion\Event\Restored;
use Flarum\Discussion\Event\Started;
use Flarum\Discussion\Event as Core;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use FoF\Byobu\Events as Byobu;
use FoF\DiscussionViews\Events\DiscussionWasViewed;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Database\Eloquent\Model;
class DiscussionSeeder extends Seeder
{
public function type(): string
{
return resolve(DiscussionSerializer::class)->getType(new Discussion);
return resolve(DiscussionSerializer::class)->getType(new Discussion());
}
public function joinRelation(): string
{
return 'discussion';
}
public function routing(Model $model): string
{
return (string) $model->id;
}
public function query(): Builder
{
return Discussion::query()
->whereNull('hidden_at');
}
public function relationships(): array
{
$includes = [];
@ -32,51 +62,148 @@ class DiscussionSeeder extends Seeder
$includes[] = 'recipientUsers';
$includes[] = 'recipientGroups';
}
return Discussion::query()
->whereNull('hidden_at')
->with($includes);
return $includes;
}
public static function savingOn(Dispatcher $events, callable $callable)
{
$events->listen([Started::class, Restored::class], function ($event) use ($callable) {
$events->listen([
// flarum/core events
Core\Started::class, Core\Restored::class,
// fof/byobu discussion recipients events.
Byobu\DiscussionMadePublic::class, Byobu\RemovedSelf::class, Byobu\RecipientsChanged::class,
], function ($event) use ($callable) {
return $callable($event->discussion);
});
}
public static function viewingOn(Dispatcher $events, callable $callable): void
{
if (!resolve(ExtensionManager::class)->isEnabled('fof-discussion-views')) {
return;
}
$events->listen(DiscussionWasViewed::class, function (DiscussionWasViewed $event) use ($callable) {
$viewCount = $event->discussion->view_count;
$shouldSync = match (true) {
$viewCount < 15 => true,
$viewCount < 100 => rand(1, 3) === 1,
default => rand(1, 19) === 1,
};
if ($shouldSync) {
$callable($event->discussion->id);
}
});
}
public static function deletingOn(Dispatcher $events, callable $callable)
{
$events->listen([Deleted::class, Hidden::class], function ($event) use ($callable) {
$events->listen([
// flarum/core events.
Core\Deleted::class, Core\Hidden::class
], function ($event) use ($callable) {
return $callable($event->discussion);
});
}
/**
* @param Discussion $model
*
* @return Document
*/
public function toDocument(Model $model): Document
{
$document = new Document([
'type' => $this->type(),
'id' => $this->type() . ':' . $model->id,
'join_field' => $this->joinRelation(),
'id' => $this->type().':'.$model->id,
'rawId' => $model->id,
'content' => $model->title,
'created_at' => $model->created_at?->toAtomString(),
'updated_at' => $model->last_posted_at?->toAtomString(),
'is_private' => $model->is_private,
'user_id' => $model->user_id,
'groups' => $this->groupsForDiscussion($model)
'groups' => $this->groupsForDiscussion($model),
'comment_count' => $model->comment_count,
]);
if ($this->extensionEnabled('flarum-tags')) {
$document['tags'] = $model->tags->pluck('id')->toArray();
}
if ($this->extensionEnabled('fof-byobu')) {
$document['recipient_users'] = $model->recipientUsers->pluck('id')->toArray();
$document['recipient_groups'] = $model->recipientGroups->pluck('id')->toArray();
$document['recipient_users'] = $model->recipientUsers
->whereNull('removed_at')
->pluck('id')
->toArray();
$document['recipient_groups'] = $model->recipientGroups
->whereNull('removed_at')
->pluck('id')
->toArray();
}
if ($this->extensionEnabled('flarum-sticky')) {
$document['is_sticky'] = (bool) $model->is_sticky;
}
if ($this->extensionEnabled('fof-discussion-views')) {
$document['view_count'] = (int) ($model->view_count ?? 0);
}
return $document;
}
/**
* All viewForum permissions keyed by permission string, loaded once per seeder instance.
* Avoids N×2 Permission queries inside the per-document map loop.
*/
private ?Collection $cachedPermissions = null;
private ?Collection $cachedGlobalPermission = null;
private function allPermissions(): Collection
{
if ($this->cachedPermissions === null) {
$this->cachedPermissions = Permission::query()
->where(function ($q) {
$q->where('permission', 'viewForum')
->orWhere('permission', 'like', 'tag%.viewForum');
})
->get();
$this->cachedGlobalPermission = $this->cachedPermissions
->where('permission', 'viewForum')
->pluck('group_id');
}
return $this->cachedPermissions;
}
protected function groupsForDiscussion(Discussion $discussion): array
{
$allPerms = $this->allPermissions();
$permissions = collect();
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$permissions = $tags->map(function (Tag $tag) use ($allPerms) {
$tagPerms = $allPerms->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$tagPerms = $tagPerms->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $tagPerms->pluck('group_id');
})->flatten();
}
if (!$discussion->is_private && $permissions->isEmpty()) {
$permissions = $this->cachedGlobalPermission;
}
return $permissions->toArray();
}
}

View File

@ -1,66 +1,54 @@
<?php
/*
* This file is part of blomstra/search.
*
* Copyright (c) 2022 Blomstra Ltd.
*
* For the full copyright and license information, please view the LICENSE.md
* file that was distributed with this source code.
*
*/
namespace Blomstra\Search\Seeders;
use Blomstra\Search\Save\Document;
use Flarum\Discussion\Discussion;
use Flarum\Extension\ExtensionManager;
use Flarum\Group\Group;
use Flarum\Group\Permission;
use Flarum\Tags\Tag;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Database\Eloquent\Model;
abstract class Seeder
{
abstract public function type(): string;
/**
* The join relation name for this document type ('discussion' or 'post').
* Used in the join_field mapping and for filtering in seed-missing checks.
*/
abstract public function joinRelation(): string;
/**
* The ES routing key for this model. Parent and child documents for the same
* discussion must share a routing key so ES places them on the same shard.
*/
abstract public function routing(Model $model): string;
abstract public function query(): Builder;
abstract public static function savingOn(Dispatcher $events, callable $callable);
abstract public static function deletingOn(Dispatcher $events, callable $callable);
/** No-op default; override in seeders that need to react to view events. */
public static function viewingOn(Dispatcher $events, callable $callable): void {}
abstract public function toDocument(Model $model): Document;
protected function groupsForDiscussion(Discussion $discussion): array
/** Relationships to eager-load on the collection before calling toDocument(). */
public function relationships(): array
{
$permissions = collect();
$globalPermission = Permission::query()
->where('permission', 'viewForum')
->pluck('group_id');
if ($this->extensionEnabled('flarum-tags')) {
/** @var Collection $tags */
$tags = $discussion->tags;
$filters['tags'] = $tags->pluck('id')->toArray();
$tagPermissions = Permission::query()
->whereIn(
'permission',
$tags->pluck('id')->map(function (int $id) {
return "tag$id.viewForum";
})
)->get();
$permissions = $tags->map(function (Tag $tag) use ($tagPermissions) {
$permissions = $tagPermissions->where('permission', "tag$tag->id.viewForum");
if ($tag->is_restricted) {
$permissions = $permissions->add(['group_id' => Group::ADMINISTRATOR_ID]);
}
return $permissions->pluck('group_id');
})->flatten();
}
if (! $discussion->is_private && $permissions->isEmpty()) {
$permissions = $globalPermission;
}
return $permissions->toArray();
return [];
}
protected function extensionEnabled(string $extension): bool